[Bug libc/13276] New: assertation failure in realloc when running out of virtual mappings

[Bug libc/13276] New: assertation failure in realloc when running out of virtual mappings

[andi-bz at firstfloor dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

             Bug #: 13276
           Summary: assertation failure in realloc when running out of
                    virtual mappings
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: andi-bz@firstfloor.org
    Classification: Unclassified


When a process runs out of virtual mappings on Linux
(more mmaps than vm.max_map_count) then munmap can fail because it may 
need to split a mapping.

In this case when there is a realloc() it will get an assertation 
failure because it doesn't expect munmap to fail.

Seen with gcc with a specific input file that fragments memory badly:

lto1: malloc.c:3551: munmap_chunk: Assertion `ret == 0' failed.

realloc should return NULL in this case, not assert. free should cleanly 
return.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertation failure in realloc when running out of virtual mappings

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

Rich Felker <bugdal at aerifal dot cx> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugdal at aerifal dot cx

--- Comment #1 from Rich Felker <bugdal at aerifal dot cx> 2011-10-08 18:40:07 UTC ---
Making realloc fail in this case is not entirely a solution. The new memory has
already been allocated at this point, and the exact same error (inability to
munmap) could happen when trying to free this new allocation. (Maybe this isn't
possible, however, if the new allocation is either always on the heap or
performed atomically via mremap, i.e. if realloc never uses mmap+munmap.)

The most robust solution would be to ensure that each allocation is always its
own vma by putting guard pages between them, i.e. mmap(size+1page) with
PROT_NONE to begin with, then mmap size over top of that with MAP_FIXED.
Unfortunately this increases the kernelspace memory usage quite a bit (more
vmas) and adds an extra syscall to every allocation (probably an unacceptable
performance cost).

An alternative solution might be merge the region that munmap fails to free
into the free lists managed for non-mmap-serviced allocations.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertation failure in realloc when running out of virtual mappings

[andi-bz at firstfloor dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

--- Comment #2 from Andi Kleen <andi-bz at firstfloor dot org> 2011-10-08 19:04:31 UTC ---
I guess it would be reasonable to just leave the mapping around in this case
Normally the program will error out anyways when this happens, otherwise
it should happen rarely.

Just it shouldn't assertation failure.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertation failure in realloc when running out of virtual mappings

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

--- Comment #3 from Rich Felker <bugdal at aerifal dot cx> 2011-10-08 20:01:02 UTC ---
Created attachment 5967
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5967
test case

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertation failure in realloc when running out of virtual mappings

[bugdal at aerifal dot cx]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

--- Comment #4 from Rich Felker <bugdal at aerifal dot cx> 2011-10-08 20:01:30 UTC ---
I can't reproduce the failure for realloc (calling realloc to resize a
mmap-sized block down to size 1 results in mremap to PAGE_SIZE rather than
servicing it from the heap), but I was able to reproduce it with free instead
of realloc. Test case is attached.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertion failure in realloc when running out of virtual mappings

[ppluzhnikov at google dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

Paul Pluzhnikov <ppluzhnikov at google dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ppluzhnikov at google dot
                   |                            |com
            Summary|assertation failure in      |assertion failure in
                   |realloc when running out of |realloc when running out of
                   |virtual mappings            |virtual mappings

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertion failure in realloc when running out of virtual mappings

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13276

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #5 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-10-29 20:40:10 UTC ---
I checked in a patch.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/13276] assertion failure in realloc when running out of virtual mappings

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=13276

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.