[Bug malloc/29125] New: allocations in pthread arena when low on virtual memory result in very high allocator overhead

[Bug malloc/29125] New: allocations in pthread arena when low on virtual memory result in very high allocator overhead

[niedzejkob at invisiblethingslab dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29125

            Bug ID: 29125
           Summary: allocations in pthread arena when low on virtual
                    memory result in very high allocator overhead
           Product: glibc
           Version: 2.35
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: malloc
          Assignee: unassigned at sourceware dot org
          Reporter: niedzejkob at invisiblethingslab dot com
  Target Milestone: ---

Consider the following program:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>

void * f(void * x) {
    (void) x;

    for(int i = 0; i < 100000; i++) {
        void * buf = malloc(16);
        if (!buf) {
            fprintf(stderr, "Allocation %d failed\n", i);
            return NULL;
        }
    }

    fprintf(stderr, "All allocations succeeded\n");
    return NULL;
}

int main(int argc, char *argv[]) {
    if (argc == 2 && strcmp(argv[1], "--threaded") == 0) {
        fprintf(stderr, "Running on a pthread...\n");
        pthread_t thread;
        pthread_create(&thread, NULL, f, NULL);
        pthread_join(thread, NULL);
    } else {
        fprintf(stderr, "Running on the main thread...\n");
        f(NULL);
    }
}

Run the following:

$ gcc repro.c -o repro -pthread
$ ./repro
Running on the main thread...
All allocations succeeded
$ ./repro --threaded
Running on a pthread...
All allocations succeeded
$ ulimit -v 65536
$ ./repro
Running on the main thread...
All allocations succeeded
$ ./repro --threaded
Running on a pthread...
Allocation 13674 failed

All together, it attempts to allocate 1.6 MB of memory. Even when taking
malloc's block headers into account, there should be no problem with fitting it
into 64 MB of virtual memory. Indeed, it works fine when the sbrk-allocated
arena is used. However, as you can see, we quickly run out of memory when
performing the allocations on a thread.

Debugging reveals that glibc attempts to allocate either 128 or 64 MB for a
per-thread arena. When this fails, it allocates a single page to serve only the
current allocation. This means that most of the returned memory is wasted, as
no attempt is made to make use of the rest of the page.

Interestingly, glibc retries the arena allocation each time malloc gets called.
There are way more calls to mmap than you'd expect from a healthy malloc.


Okay, but why am I running my program with so little virtual memory, you might
ask. The issue that lead me to find this behavior occurred within an SGX
enclave, where you need to decide how much memory your process is allowed to
use, and reserve it all in advance. There's also no way to back the mmaps on
demand — if you're using virtual memory, you're using physical memory.

With a typical application getting started with 256 MB, a few threads consuming
64 MB each for their arena can quickly get you into a situation where even
virtual memory becomes a scarce resource.

We're aware of the various malloc tunables that can help work around this
issue, such as MALLOC_ARENA_MAX=1. However I can't shake the feeling that what
we're observing is not what the author of this part of malloc's strategy had in
mind.

The bug was initially observed on glibc 2.31, and I've reproduced it on glibc
2.35, which is the most recent available on ftp.gnu.org at the time of writing.

-- 
You are receiving this mail because:
You are on the CC list for the bug.