[PATCH] Fix malloc

[PATCH] Fix malloc

[Jakub Jelinek]

Hi!

#include <stdlib.h>

void main (void)
{
  malloc (0x3ffffffc);
}

crashes with new malloc on an i686 box with 256MB of RAM and
no overcommit.
The issue is that if MORECORE_FAILURE != MMAP_FAILED, then
if brk fails and mmap fails too, it would leave brk set to MMAP_FAILED
and so code below would consider it as if brk was successfully set.

2002-03-13  Jakub Jelinek  <jakub@redhat.com>

	* malloc/malloc.c (sYSMALLOc): Set brk to MORECORE_FAILURE if mmap
	failed.

--- libc/malloc/malloc.c.jj	Fri Feb  8 12:04:04 2002
+++ libc/malloc/malloc.c	Wed Mar 13 12:44:48 2002
@@ -2838,7 +2838,8 @@ static Void_t* sYSMALLOc(nb, av) INTERNA
            bridge regions.
         */
         set_noncontiguous(av);
-      }
+      } else
+	brk = (char *)(MORECORE_FAILURE);
     }
 #endif
   }

	Jakub

Re: [PATCH] Fix malloc

[Wolfram Gloger]

Hi,

> The issue is that if MORECORE_FAILURE != MMAP_FAILED, then
> if brk fails and mmap fails too, it would leave brk set to MMAP_FAILED
> and so code below would consider it as if brk was successfully set.

Clearly you're right.  I'd rather not change 'brk' at all though,
like below:

2002-03-13  Wolfram Gloger  <wg@malloc.de>

	* malloc/malloc.c (sYSMALLOc): Don't change brk if mmap
	failed.

--- malloc.c	2002/02/15 09:30:28	1.6
+++ malloc.c	2002/03/13 12:06:55
@@ -2824,11 +2824,12 @@
     /* Don't try if size wraps around 0 */
     if ((unsigned long)(size) > (unsigned long)(nb)) {
 
-      brk = (char*)(MMAP(0, size, PROT_READ|PROT_WRITE, MAP_PRIVATE));
+      char *mbrk = (char*)(MMAP(0, size, PROT_READ|PROT_WRITE, MAP_PRIVATE));
 
-      if (brk != MAP_FAILED) {
+      if (mbrk != MAP_FAILED) {
 
         /* We do not need, and cannot use, another sbrk call to find end */
+        brk = mbrk;
         snd_brk = brk + size;
 
         /*

Re: [PATCH] Fix malloc

[Ulrich Drepper]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

On Wed, 2002-03-13 at 04:11, Wolfram Gloger wrote:

> 2002-03-13  Wolfram Gloger  <wg@malloc.de>
> 
> 	* malloc/malloc.c (sYSMALLOc): Don't change brk if mmap
> 	failed.

I've applied this patch.  Thanks,

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]