[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

["jakub at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636

--- Comment #13 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-10-08 20:14:14 UTC ---
In the 0001-initial-madvise.patch patch I think if you subtract the size
MADV_DONTNEEDed from G.bytes_mapped, then you should add it again in
alloc_page (i.e. replace
+      p->unmapped = false;
+
you've added there with:
+      if (p->unmapped)
+        G.bytes_mapped += p->bytes;
+      p->unmapped = false;
+
Not sure if it is enough as is with that change, or if more is needed - like
also keeping track of the size of currently MADV_DONTNEED bytes and if it grows
too much together with G.bytes_mapped, consider really munmapping some of the
pages (perhaps try to sort the free list by increasing size, so that further
allocations will prefer the smaller blocks over the larger ones, and if getting
close to filled address space, consider freeing some of the largest one in case
malloc would be requested.  Or alternatively, hook into xmalloc_failed and if
malloc fails, try to free some of the largest chunks and retry.  I'm just
worried if at some point during the compilation we need huge amount of GC
memory, close to filling 32-bit address space, then we GC collect lots of it,
but on the other side would need a lot of malloced memory.  But perhaps that is
unlikely to happen.