Re: [PATCH] malloc: Deprecate hook variables, __default_morecore, <mcheck.h>

[Steve Vormwald <sdvormwa@cray.com>]

> On 11/18/2016 04:13 AM, Florian Weimer wrote:
> I've been considering for a while to remove the use of sbrk from the
> allocator, both to simplify it and to take away the possibility for
> attacks to tweak the NON_MAIN_ARENA flag to trick the allocator into
> doing something bad.  Some embedded targets appear to strongly prefer
> the sbrk as well, so maybe this approach is doomed.

For us anyway, the use of sbrk isn't as important as its semantics (eg,
heap is contiguous in virtual address space and only grows/shrinks on
one end).  Thus, things like libhugetlbfs that provide the same semantics
via __morecore work just as well.

> Would your configuration have problems if we handed all allocations
> which are larger than 1 MiB or so straight to mmap?

That would probably not work out for us.  The underlying problem we have
is that unmapping pages can be extremely expensive (on the order of tens
of seconds or worse in certain circumstances).  We use segments with
fixed base addresses and maximum length to transparently make portions
of a process's virtual address space (notably including all user allocations)
available for hardware DMA.  Updating page mappings within a segment is
extremely cheap.  Adding a new segment is also relatively cheap, but they
are a limited resource.  Invalidating a segment so it can be reused can get
very expensive since everything that has used it needs to be notified of the
invalidation.  We can take a fast path if the allocator has sbrk-like semantics
as described above because we know it only requires updating page
mappings, assuming the existing segment is large enough.  We have to be
much more conservative for allocations that are satisfied with mmap,
leading to a lot of expensive and potentially unnecessary segment creation
and invalidation.

> We would also be interested in general patterns of fragmentation you
> have encountered, if any.  Maybe there's something we can do about them.

The usual culprit is something "caching" small (<128byte) allocations for
future reuse rather than freeing them when they become unused, which
prevents the allocator from coalescing free regions to satisfy larger allocations.
Changing the code either to allocate in bulk instead of one object at a time or
to simply free them without caching is generally sufficient to fix the
fragmentation issue.

Steven Vormwald