[Bug malloc/26663] malloc() doesn't handle MALLOC_MMAP_THRESHOLD_=0 well

["toiwoton at gmail dot com" <sourceware-bugzilla@sourceware.org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=26663

--- Comment #3 from Topi Miettinen <toiwoton at gmail dot com> ---
(In reply to Carlos O'Donell from comment #1)
> (In reply to Topi Miettinen from comment #0)
> > Using environment variable MALLOC_MMAP_THRESHOLD_=0 may cause malloc() to
> > fail very easily. A case in point is startup of systemd-cryptsetup, which
> > fails when lvm2 library can't malloc() something
> > (https://github.com/lvmteam/lvm2/issues/39). This is probably a bug in
> > kernel, there's plenty of RAM available and I suppose the memory shouldn't
> > ever be too fragmented for page sized items.
> 
> In the upstream ticket I've asked "Why?" It's important we understand your
> use case so we can give better feedback about solutions.
> 
> In this bug we'll talk about the specific behaviour of malloc.
> 
> Firstly you need to be aware that you'll hit MALLOC_MMAP_MAX_ at some point
> which is 65,536 mappings before malloc stops allocating anything.

I suppose that this limit will not be an issue, since other services and user
applications worked just fine.

> Secondly, the MALLOC_MMAP_THRESHOLD_ value is the threshold at which
> requests to extend the heap will go directly to mmap() instead of growing
> the current arena (also done via mmap()).
> 
> Thus even if you set MALLOC_MMAP_THRESHOLD_ to a low value, you will still
> use the process heap, it is only when you run out of heap that you will
> switch to trying to mmap() all subsequent allocations via single mmap()
> calls (instead of trying to extend the main arena with another larger single
> mmap to service current and future requests).
> 
> So the first thing to check is the value of MALLOC_MMAP_MAX_ and see if that
> is set high enough for your expectations.

I specifically used zero, since I wanted malloc() to always use mmap() for
allocating memory. The original idea was based on my expectation that then the
mmap()ed regions would be randomly distributed in process address space ("ASLR
for malloc()"), but sadly that does not seem to be the case as the addresses
are pretty much consecutive. Also performance seems to be worse (judging just
from noise of CPU fan).

> I would also use `strace -ff -ttt` on the process to see if mmap() is
> actually failing or not, and with what values.

I've attached a strace file. The problem happens here:

1600959989.943625 mmap(NULL, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 EAGAIN (Resource temporarily
unavailable)

My expectation for the kernel is that this should only happen if the kernel was
really unable to allocate a single page, which should never be the case when
there are gigabytes of free RAM in the system.

> If you want deeper insight into your application behaviour you can use a
> trace/dump tooling for the malloc API calls:
> https://pagure.io/glibc-malloc-trace-utils
> (not yet integrated upstream)
>  
> > But perhaps malloc() should use a better allocation strategy when mmap()ing,
> > for example mmap() larger areas at once, use them as arenas and deal them to
> > malloc() users as needed.
> 
> This suggestion is exactly what it does *normally*, but the setting of
> MALLOC_MMAP_THRESHOLD_=0 bypasses that and attempts to *always* service
> extension requirements with single mmap() calls.

I meant that instead of single mmap() calls per malloc() call, it would grab a
larger area with mmap() and then use that.

I thought that MALLOC_MMAP_THRESHOLD_=0 would just switch from sbrk() to mmap()
but otherwise things would stay the same. If MALLOC_MMAP_THRESHOLD_=0 is not so
great tool to achieve this, could there perhaps be another option to just force
mmap() use?

> > Maybe also when mmap() specifically fails with EAGAIN it could be retried a
> > few times. It looks like sysmalloc() only checks for MAP_FAILED
> > (https://sourceware.org/git?p=glibc.git;a=blob;f=malloc/malloc.c;
> > h=cd9933b4e580a58a694ebf34e76ac6fecee29c14;hb=HEAD#l2330) but not errno.
> > Perhaps also some unused memory could be released by munmap()ping unused
> > parts of arenas etc. before retrying.
> 
> Calls to malloc() should not block retrying mmap(), they should fail
> immediately and report that failure to higher software layers that have
> better visbility into overall allocation strategy. The retry should happen
> at higher levels. From the allocators perspective there is no difference
> between EAGAIN and ENOMEM.

OK. I think most callers just look at return value of NULL and don't look at
errno code.

> There is one place where we can improve though, and it's in tracking the
> currently cached amount of memory that the process is using, and keeping
> that at a constant percentage by walking the cached chunk lists and freeing
> down to an acceptable percentage e.g. calling malloc_trim(0) but terminating
> when a certain percentage has been freed.

That could also work. Though if the problem is at kernel side, maybe mmap()
would eventually fail again later. 

> 
> Does this answer your questions?
> 
> If it does then I'll mark this RESOLVED/NOTABUG

My only remaining question is that can a new flag could be introduced to
instruct malloc() to never use sbrk() and the process heap but otherwise
perform normally while using mmap() for allocating? That way, if one day mmap()
could be randomized, the memory arenas would be at random locations.

I think also the current feature would work to a degree (minus the possible
performance issue) with improvements to kernel (avoid failing and randomize the
address).

Maybe I could also force the randomization with a special allocator by plugging
into malloc internals and using LD_PRELOAD to force this to critical
applications. I suppose then nothing would need changing, but it's a hack.
Perhaps also glibc could do this, for example another option to malloc()?

-Topi

-- 
You are receiving this mail because:
You are on the CC list for the bug.