From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id D33383844019; Mon, 1 Feb 2021 08:52:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D33383844019 From: "keyid.w at qq dot com" To: glibc-bugs@sourceware.org Subject: [Bug malloc/26969] A common malloc pattern can make memory not given back to OS Date: Mon, 01 Feb 2021 08:52:54 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: glibc X-Bugzilla-Component: malloc X-Bugzilla-Version: 2.27 X-Bugzilla-Keywords: X-Bugzilla-Severity: minor X-Bugzilla-Who: keyid.w at qq dot com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P1 X-Bugzilla-Assigned-To: unassigned at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: security- X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: glibc-bugs@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Glibc-bugs mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Feb 2021 08:52:54 -0000 https://sourceware.org/bugzilla/show_bug.cgi?id=3D26969 --- Comment #4 from XY Wen --- (In reply to Dmitry from comment #3) > This is a really common problem especially for long-running multithreaded > processes with a lot of arenas. Calling malloc_trim - not always an optio= n, > if a service using C bindings and is written on high level languages. Als= o, > malloc_trim would cause unnecessary overhead, since it will be trimming a= nd > locking all arenas. Additional confusion comes from man page for malloc_t= rim > which says it's called sometimes during free. >=20 > I was looking at the code and have the following suggestion for improveme= nt: > the main idea is to call mtrim for arena in _int_free. To amortize > performance overhead: call it only if free is called for a chunk with siz= e > > than FASTBIN_CONSOLIDATION_THRESHOLD, and we have a chance to free more > memory than let's say 3*TRIM_THRESHOLD. To understand how much we can ret= urn > to OS we can add 1 bit flag to chunk, set it to 0 after returning to OS, = set > to 1 otherwise. This would not give 100% accurate result, but should give= a > good estimate. >=20 > I'd be glad to work on this functionality, if you think it makes sense. If > you have other suggestions also happy to discuss this. The main point - > mtrim should be called during free some times, otherwise current malloc is > unfortunately hard to use for long-running multithreaded workload, since > it's using 2-3x more RSS in comparison with jemalloc or tcmalloc. I think your solution will be helpful in this situation. I have no better i= dea right now(and I think it's hard to design a trade-off strategy since it probably need lots of tests). But jemalloc may have some different methods = to solve this problem, to which you can refer, because I noticed that the virt= ual memory is also reduced using jemalloc. By the way, tcmalloc doesn't do bett= er in my usage. --=20 You are receiving this mail because: You are on the CC list for the bug.=