[Bug malloc/30579] New: trim_threshold in realloc lead to high memory usage

["nicolas at freedelity dot be" <sourceware-bugzilla@sourceware.org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=30579

            Bug ID: 30579
           Summary: trim_threshold in realloc lead to high memory usage
           Product: glibc
           Version: 2.37
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: malloc
          Assignee: unassigned at sourceware dot org
          Reporter: nicolas at freedelity dot be
  Target Milestone: ---

The recent usage of trim_threshold in realloc (commit
f4f2ca1509288f6f780af50659693a89949e7e46:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=f4f2ca1509288f6f780af50659693a89949e7e46)
is preventing to reclaim unused memory in some use case scenario.

The scenario that is affecting us now is that we need to normalize a lot of
strings (around ~20M) and their final size after normalization is unknown (just
that we have a upperbound of 1K bytes). So the memory is preallocated to the
maximum size and then sized down to the final size, so that we avoid many
reallocation during string constructions.

So during normalization, the memory for a given string is preallocated to 1024
and then reallocated back to its final size. The default value for
trim_threshold is 128K so this memory is never reclaimed.The process ends up
eating much more memory than necessary.

The following test program reproduce the issue:

```
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main() {
  int i;
  int count = 1000000;
  char ** strings = malloc(sizeof(char*) * count);

  for(i=0; i<count; i++){
    strings[i] = malloc(sizeof(char) * 1000);
    strcpy(strings[i], "hello");
    strings[i] = realloc(strings[i], sizeof(char) * 6);
  }


  while(1) {
    sleep(100);
  }

  return 0;
}
```

It constructs 1M strings set to "hello" by preallocating a 1000-bytes array for
each string to emulate the fact that we do not know its size in advance. The
correct size is set back with a call to realloc.

Until GLIBC 2.36, the memory was correctly reclaimed when calling realloc and
we end up consuming around 40 MB.
Starting from GLIBC 2.37 and the mentionned commit, the memory goes up until
1GB.

I'm not an expert but reusing trim_threshold as an heuristic to decide to
reclaim memory or not does not seem a good fit for heap memory. 
By reading the comments, I would be tempted to set glibc.malloc.trim_threshold
(via GLIBC_TUNABLES) to a value high enough for performance reason but now, we
are somewhat forced to set it to a very low value to avoid consuming too much
memory.

In my opinion, the heuristics should probably be based on an independent value.

-- 
You are receiving this mail because:
You are on the CC list for the bug.