From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <sourceware-bugzilla@sourceware.org>
Received: by sourceware.org (Postfix, from userid 48)
	id D990F3857803; Thu, 22 Jun 2023 13:54:08 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D990F3857803
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org;
	s=default; t=1687442048;
	bh=eY8YX+IejauCmm5YTsilqMCln6GZNmwwWtUgX1WuNxE=;
	h=From:To:Subject:Date:From;
	b=TVJVWDu0K5tN29UFYVHGBrvKb7zMTorK/19rQp7l30edzzVkfce3FRWEGWiumB1Th
	 22IK4hDAzNCqTOaF7pgLuDmaAAvsKLEBASdvclugNwjBibhUo/IlJ77iEfNWq2UY7y
	 aM3DHStlMtKdi5fGvWC6KpkJQ15ieT3IMLWk7PIQ=
From: "nicolas at freedelity dot be" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug malloc/30579] New: trim_threshold in realloc lead to high
 memory usage
Date: Thu, 22 Jun 2023 13:54:08 +0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: malloc
X-Bugzilla-Version: 2.37
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: normal
X-Bugzilla-Who: nicolas at freedelity dot be
X-Bugzilla-Status: UNCONFIRMED
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: unassigned at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status
 bug_severity priority component assigned_to reporter target_milestone
Message-ID: <bug-30579-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
List-Id: <glibc-bugs.sourceware.org>

https://sourceware.org/bugzilla/show_bug.cgi?id=3D30579

            Bug ID: 30579
           Summary: trim_threshold in realloc lead to high memory usage
           Product: glibc
           Version: 2.37
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: malloc
          Assignee: unassigned at sourceware dot org
          Reporter: nicolas at freedelity dot be
  Target Milestone: ---

The recent usage of trim_threshold in realloc (commit
f4f2ca1509288f6f780af50659693a89949e7e46:
https://sourceware.org/git/?p=3Dglibc.git;a=3Dcommitdiff;h=3Df4f2ca1509288f=
6f780af50659693a89949e7e46)
is preventing to reclaim unused memory in some use case scenario.

The scenario that is affecting us now is that we need to normalize a lot of
strings (around ~20M) and their final size after normalization is unknown (=
just
that we have a upperbound of 1K bytes). So the memory is preallocated to the
maximum size and then sized down to the final size, so that we avoid many
reallocation during string constructions.

So during normalization, the memory for a given string is preallocated to 1=
024
and then reallocated back to its final size. The default value for
trim_threshold is 128K so this memory is never reclaimed.The process ends up
eating much more memory than necessary.

The following test program reproduce the issue:

```
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main() {
  int i;
  int count =3D 1000000;
  char ** strings =3D malloc(sizeof(char*) * count);

  for(i=3D0; i<count; i++){
    strings[i] =3D malloc(sizeof(char) * 1000);
    strcpy(strings[i], "hello");
    strings[i] =3D realloc(strings[i], sizeof(char) * 6);
  }


  while(1) {
    sleep(100);
  }

  return 0;
}
```

It constructs 1M strings set to "hello" by preallocating a 1000-bytes array=
 for
each string to emulate the fact that we do not know its size in advance. The
correct size is set back with a call to realloc.

Until GLIBC 2.36, the memory was correctly reclaimed when calling realloc a=
nd
we end up consuming around 40 MB.
Starting from GLIBC 2.37 and the mentionned commit, the memory goes up until
1GB.

I'm not an expert but reusing trim_threshold as an heuristic to decide to
reclaim memory or not does not seem a good fit for heap memory.=20
By reading the comments, I would be tempted to set glibc.malloc.trim_thresh=
old
(via GLIBC_TUNABLES) to a value high enough for performance reason but now,=
 we
are somewhat forced to set it to a very low value to avoid consuming too mu=
ch
memory.

In my opinion, the heuristics should probably be based on an independent va=
lue.

--=20
You are receiving this mail because:
You are on the CC list for the bug.=