From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <libc-hacker-return-9015-listarch-libc-hacker=sources.redhat.com@sourceware.org>
Received: (qmail 5269 invoked by alias); 24 Aug 2006 17:07:23 -0000
Received: (qmail 5211 invoked by uid 22791); 24 Aug 2006 17:07:22 -0000
X-Spam-Check-By: sourceware.org
Received: from sunsite.ms.mff.cuni.cz (HELO sunsite.mff.cuni.cz) (195.113.15.26)
    by sourceware.org (qpsmtpd/0.31) with ESMTP; Thu, 24 Aug 2006 17:07:14 +0000
Received: from sunsite.mff.cuni.cz (sunsite.mff.cuni.cz [127.0.0.1])
	by sunsite.mff.cuni.cz (8.13.1/8.13.1) with ESMTP id k7OH6hS1017026;
	Thu, 24 Aug 2006 19:06:43 +0200
Received: (from jj@localhost)
	by sunsite.mff.cuni.cz (8.13.1/8.13.1/Submit) id k7OH6hPS017024;
	Thu, 24 Aug 2006 19:06:43 +0200
Date: Thu, 24 Aug 2006 17:07:00 -0000
From: Jakub Jelinek <jakub@redhat.com>
To: Ulrich Drepper <drepper@redhat.com>
Cc: Glibc hackers <libc-hacker@sources.redhat.com>
Subject: [PATCH] Fix infinite loop in malloc for really large sizes
Message-ID: <20060824170642.GJ4556@sunsite.mff.cuni.cz>
Reply-To: Jakub Jelinek <jakub@redhat.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4.1i
Mailing-List: contact libc-hacker-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Subscribe: <mailto:libc-hacker-subscribe@sourceware.org>
List-Archive: <http://sourceware.org/ml/libc-hacker/>
List-Post: <mailto:libc-hacker@sourceware.org>
List-Help: <mailto:libc-hacker-help@sourceware.org>, <http://sourceware.org/ml/#faqs>
Sender: libc-hacker-owner@sourceware.org
X-SW-Source: 2006-08/txt/msg00025.txt.bz2

Hi!

As the attached testcase shows, for really large allocations where
all of mmap, heap growth and new heap alloc fail malloc loops forever,
calling mmap with the given size.

2006-08-24  Jakub Jelinek  <jakub@redhat.com>

	* malloc/malloc.c (sYSMALLOc): Avoid infinite loop if MMAP
	keeps failing and heap growth or new heap creation isn't
	successful either.
	* malloc/tst-malloc.c (main): Add new test.

--- libc/malloc/malloc.c.jj	2006-08-22 09:04:53.000000000 +0200
+++ libc/malloc/malloc.c	2006-08-24 18:48:54.000000000 +0200
@@ -2860,6 +2860,7 @@ static Void_t* sYSMALLOc(nb, av) INTERNA
   unsigned long   sum;            /* for updating stats */
 
   size_t          pagemask  = mp_.pagesize - 1;
+  bool            tried_mmap = false;
 
 
 #if HAVE_MMAP
@@ -2883,6 +2884,7 @@ static Void_t* sYSMALLOc(nb, av) INTERNA
       is no following chunk whose prev_size field could be used.
     */
     size = (nb + SIZE_SZ + MALLOC_ALIGN_MASK + pagemask) & ~pagemask;
+    tried_mmap = true;
 
     /* Don't try if size wraps around 0 */
     if ((unsigned long)(size) > (unsigned long)(nb)) {
@@ -3006,7 +3008,7 @@ static Void_t* sYSMALLOc(nb, av) INTERNA
 	set_foot(old_top, (old_size + 2*SIZE_SZ));
       }
     }
-    else
+    else if (!tried_mmap)
       /* We can at least try to use to mmap memory.  */
       goto try_mmap;
 
--- libc/malloc/tst-malloc.c.jj	2001-07-06 06:55:35.000000000 +0200
+++ libc/malloc/tst-malloc.c	2006-08-24 18:59:38.000000000 +0200
@@ -33,7 +33,7 @@ merror (const char *msg)
 int
 main (void)
 {
-  void *p;
+  void *p, *q;
   int save;
 
   errno = 0;
@@ -64,5 +64,15 @@ main (void)
   if (p != NULL)
     merror ("realloc (p, 0) failed.");
 
+  p = malloc (513 * 1024);
+  if (p == NULL)
+    merror ("malloc (513K) failed.");
+
+  q = malloc (-512 * 1024);
+  if (q != NULL)
+    merror ("malloc (-512K) succeeded.");
+
+  free (p);
+
   return errors != 0;
 }

	Jakub