Excessive memory consumption when using malloc()

[Christian Hoff <christian_hoff@gmx.net>]

Hello all,

we are facing the a problem with the memory allocator in glibc 2.17 on
RHEL 7.9. Or application allocates about 10 GB of memory (split into
chunks that are each around 512 KB large). This memory is used for some
computations and released afterwards. After a while, the application is
running the same computations again, but this time in different threads.
The first issue we are seeing is that - after the computations are done
- the 10 GB of memory is not released back to the operating system. Only
after calling malloc_trim() manually with GDB, the size of the process
shrinks dramatically from ~10GB to 400 MB. So, at this point, the unused
memory from the computations is finally returned to the OS.

Our wish would be that the memory is returned to the OS without us
having to call malloc_trim(). And I understand that glibc also trims the
heap when there is sufficient free space in top of it (the
M_TRIM_THRESHOLD in mallopt() controls when this should happen). What
could be the reason why this is not working in our case? Could it be
related to heap fragmentation? But assuming that is the reason, why is
malloc_trim() nevertheless able to free this memory?

And then we also have one other problem. The first run of the
computations is always fine: we allocate 10 GB of memory and the
application grows to 10 GB. Afterwards, we release those 10 GB of memory
since the computations are now done and at this point the freed memory
is returned back to the allocator (however, the size of the process
remains 10 GB unless we call malloc_trim()). But if we now re-run the
same computations again a second time (this time using different
threads), a problem occurs. In this case, the size of the application
grows well beyond 10 GB. It can get 20 GB or larger and the process is
eventually killed because the system runs out of memory.

Do you have any idea why this happens? To me it seems like the threads
are assigned to different arenas and therefore the previously freed 10
GB of memory can not be re-used as they are in different arenas. Is that
possible?

A workaround I have found is to set M_MMAP_THRESHOLD to 128 KB - then
the memory for the computations is always allocated using mmap() and
returned back to the system immediately when it is free()'ed. This
solves both of the issues. But I am afraid that this workaround could
degrade the performance of our application. So, we are grateful for any
better solution to this problem.

Kind regards,

    Christian Hoff