From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id 5D56D3858C41; Wed, 16 Aug 2023 11:55:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5D56D3858C41 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1692186924; bh=/SySdgw3Ik2sfQhZiI68KPfBaPhy/moBBL1j/EuBM7o=; h=From:To:Subject:Date:From; b=DdT8I7pYN7aKiqZPKHvWcXFNvGQaTyW9KB9yJPZ1iVvcbE5FCmtVEb+3eRovrJsqu XEmS7th4Ex4jOlSicfMVROEWgD317AZ4A5mIG5olNEpfbllfemphkmwWyzS7VBcbQh rb4JU855DOsmbRfjjTdWb/1mGquO03wAfOVV2GrQ= From: "sascha.zorn at sap dot com" To: glibc-bugs@sourceware.org Subject: [Bug malloc/30769] New: malloc_trim is not working correctly for arenas other than arena 0 Date: Wed, 16 Aug 2023 11:55:22 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: glibc X-Bugzilla-Component: malloc X-Bugzilla-Version: 2.35 X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: sascha.zorn at sap dot com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: unassigned at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status bug_severity priority component assigned_to reporter target_milestone attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 List-Id: https://sourceware.org/bugzilla/show_bug.cgi?id=3D30769 Bug ID: 30769 Summary: malloc_trim is not working correctly for arenas other than arena 0 Product: glibc Version: 2.35 Status: UNCONFIRMED Severity: normal Priority: P2 Component: malloc Assignee: unassigned at sourceware dot org Reporter: sascha.zorn at sap dot com Target Milestone: --- Created attachment 15068 --> https://sourceware.org/bugzilla/attachment.cgi?id=3D15068&action=3Ded= it mem.cpp This bug is a bit harder to explain and we stumbled upon it via multiple long-running multi-threaded applications in a sidecar docker container, that has only 300MB hard limit after which the container is killed. We periodica= lly reached this limit even though we have no memory leaks in our code (verifie= d by multiple tools and own allocator frameworks on top) and even manually tried= to trim the malloc areans with malloc_trim(). We need to free the reserved mem= ory as other processes might need the scarce 300MB. Upon research I stumbled ov= er a few articles that MIGHT have similar issues and this lead me to experiment = with malloc_stat/malloc_info and limiting the number of arenas (to 1) which fina= lly worked around the issue. Examples being: https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-= precious-memory/ https://thehftguy.com/2020/05/21/major-bug-in-glibc-is-killing-applications= -with-a-memory-limit/ Although the second article mentions there are already plans to: "Throttle number of arenas based on process rlimit on process startup and/or everytime RLIMIT_AS is modified" - https://sourceware.org/glibc/wiki/Development_Todo/Enhancing_malloc"=20 I don't think this is being worked on, would not consider cgroups (docker container limits) or would even completely solve our issue, as all areans except Arena 0 is properly trimmed via malloc_trim(). Unfortunately I'm not really good at writing multi-threaded, synchronized C code and only have a C++20 sample that can reproduce this issue reliably (I compiled with 'g++ --std=3Dc++20 -o mem -O3 -g mem.cpp'). Basically it spaw= ns a configurable amount of threads that all wait in a barrier until all threads have been started and the tries to call malloc() as concurrent as possible. This in my experience triggers the creation of the most amount of arenas as quickly as possible. Then it does this concurrent malloc() a few times more, every time with a little bit increasing malloc sizes, to fragment and popul= ate the arenas. I hope you can work with this example to reproduce this issue. I tried this with glibc 2.35 and see the following behaviour. First number is the thread= id. First iteration looks good: 11: memory allocated (0x7f2681bfe010, size=3D18.000MB, rss=3D217.875MB)...= =20 1: memory allocated (0x7f2682dff010, size=3D18.000MB, rss=3D226.875MB)...= =20 3: memory allocated (0x7f266adff010, size=3D18.000MB, rss=3D232.875MB)...= =20 9: memory allocated (0x7f26903ee010, size=3D18.000MB, rss=3D271.500MB)...= =20 5: memory allocated (0x7f2689bfe010, size=3D18.000MB, rss=3D276.750MB)...= =20 15: memory allocated (0x7f2672dff010, size=3D18.000MB, rss=3D280.688MB)...= =20 10: memory allocated (0x7f26889fd010, size=3D18.000MB, rss=3D287.250MB)...= =20 13: memory allocated (0x7f26789fd010, size=3D18.000MB, rss=3D288.188MB)...= =20 6: memory allocated (0x7f26915ef010, size=3D18.000MB, rss=3D288.750MB)...= =20 4: memory allocated (0x7f2679bfe010, size=3D18.000MB, rss=3D289.125MB)...= =20 14: memory allocated (0x7f2669bfe010, size=3D18.000MB, rss=3D289.500MB)...= =20 8: memory allocated (0x7f267adff010, size=3D18.000MB, rss=3D289.688MB)...= =20 2: memory allocated (0x7f268adff010, size=3D18.000MB, rss=3D290.250MB)...= =20 16: memory allocated (0x7f26709fd010, size=3D18.000MB, rss=3D290.438MB)...= =20 7: memory allocated (0x7f2671bfe010, size=3D18.000MB, rss=3D290.625MB)...= =20 12: memory allocated (0x7f26809fd010, size=3D18.000MB, rss=3D290.625MB)...= =20 Arena 0: system bytes =3D 135168 in use bytes =3D 80624 Arena 1: system bytes =3D 135168 in use bytes =3D 3392 Arena 2: system bytes =3D 135168 in use bytes =3D 3392 Arena 3: system bytes =3D 135168 in use bytes =3D 3392 Arena 4: system bytes =3D 135168 in use bytes =3D 3392 Arena 5: system bytes =3D 135168 in use bytes =3D 3392 Arena 6: system bytes =3D 135168 in use bytes =3D 3392 Arena 7: system bytes =3D 135168 in use bytes =3D 3392 Arena 8: system bytes =3D 135168 in use bytes =3D 3392 Arena 9: system bytes =3D 135168 in use bytes =3D 3392 Arena 10: system bytes =3D 135168 in use bytes =3D 3392 Arena 11: system bytes =3D 135168 in use bytes =3D 3392 Arena 12: system bytes =3D 135168 in use bytes =3D 3392 Arena 13: system bytes =3D 135168 in use bytes =3D 3392 Arena 14: system bytes =3D 135168 in use bytes =3D 3392 Arena 15: system bytes =3D 135168 in use bytes =3D 3392 Arena 16: system bytes =3D 135168 in use bytes =3D 3392 Total (incl. mmap): system bytes =3D 304353280 in use bytes =3D 302190320 max mmap regions =3D 16 max mmap bytes =3D 302055424 3: deallocated (rss=3D272.805MB)... trimmed (rss=3D272.805MB) 14: deallocated (rss=3D254.816MB)... trimmed (rss=3D254.816MB) 12: deallocated (rss=3D236.824MB)... trimmed (rss=3D236.824MB) 8: deallocated (rss=3D218.832MB)... trimmed (rss=3D218.832MB) 7: deallocated (rss=3D201.000MB)... trimmed (rss=3D201.000MB) 1: deallocated (rss=3D183.043MB)... trimmed (rss=3D183.043MB) 2: deallocated (rss=3D165.062MB)... trimmed (rss=3D165.062MB) 16: deallocated (rss=3D147.094MB)... trimmed (rss=3D147.094MB) 5: deallocated (rss=3D129.207MB)... trimmed (rss=3D129.207MB) 9: deallocated (rss=3D111.250MB)... trimmed (rss=3D111.250MB) 6: deallocated (rss=3D93.352MB)... trimmed (rss=3D93.352MB) 4: deallocated (rss=3D75.449MB)... trimmed (rss=3D75.449MB) 10: deallocated (rss=3D57.457MB)... trimmed (rss=3D57.457MB) 15: deallocated (rss=3D39.605MB)... trimmed (rss=3D39.605MB) 13: deallocated (rss=3D21.746MB)... trimmed (rss=3D21.746MB) 11: deallocated (rss=3D3.746MB)... trimmed (rss=3D3.746MB) Arena 0: system bytes =3D 81920 in use bytes =3D 80624 Arena 1: system bytes =3D 135168 in use bytes =3D 3392 Arena 2: system bytes =3D 135168 in use bytes =3D 3392 Arena 3: system bytes =3D 135168 in use bytes =3D 3392 Arena 4: system bytes =3D 135168 in use bytes =3D 3392 Arena 5: system bytes =3D 135168 in use bytes =3D 3392 Arena 6: system bytes =3D 135168 in use bytes =3D 3392 Arena 7: system bytes =3D 135168 in use bytes =3D 3392 Arena 8: system bytes =3D 135168 in use bytes =3D 3392 Arena 9: system bytes =3D 135168 in use bytes =3D 3392 Arena 10: system bytes =3D 135168 in use bytes =3D 3392 Arena 11: system bytes =3D 135168 in use bytes =3D 3392 Arena 12: system bytes =3D 135168 in use bytes =3D 3392 Arena 13: system bytes =3D 135168 in use bytes =3D 3392 Arena 14: system bytes =3D 135168 in use bytes =3D 3392 Arena 15: system bytes =3D 135168 in use bytes =3D 3392 Arena 16: system bytes =3D 135168 in use bytes =3D 3392 Total (incl. mmap): system bytes =3D 2244608 in use bytes =3D 134896 max mmap regions =3D 16 max mmap bytes =3D 302055424 I don't really understand why the areans only show 3392 bytes of memory "in use", even after the allocations took place, but I guess that all the allocations took place in mmap, and not in the areans. After the last deallocation RSS is falling down to 3.746MB, which is=20 Fun starts after the second iteration, where the areans > Arena 0 are "resi= zed" to 18890752 bytes: 15: memory allocated (0x7f2624000d50, size=3D18.001MB, rss=3D234.371MB)...= =20 11: memory allocated (0x7f265c000d50, size=3D18.001MB, rss=3D244.684MB)...= =20 14: memory allocated (0x7f267c000d50, size=3D18.001MB, rss=3D276.559MB)...= =20 1: memory allocated (0x7f2664000d50, size=3D18.001MB, rss=3D279.559MB)...= =20 9: memory allocated (0x7f2634000d50, size=3D18.001MB, rss=3D281.621MB)...= =20 5: memory allocated (0x7f2684000d50, size=3D18.001MB, rss=3D285.184MB)...= =20 6: memory allocated (0x7f268c000d50, size=3D18.001MB, rss=3D285.934MB)...= =20 8: memory allocated (0x7f2644000d50, size=3D18.001MB, rss=3D288.746MB)...= =20 4: memory allocated (0x7f263c000d50, size=3D18.001MB, rss=3D289.496MB)...= =20 3: memory allocated (0x7f2614000d50, size=3D18.001MB, rss=3D290.246MB)...= =20 2: memory allocated (0x7f2674000d50, size=3D18.001MB, rss=3D290.621MB)...= =20 16: memory allocated (0x7f2654000d50, size=3D18.001MB, rss=3D290.996MB)...= =20 10: memory allocated (0x7f266c000d50, size=3D18.001MB, rss=3D291.371MB)...= =20 7: memory allocated (0x7f261c000d50, size=3D18.001MB, rss=3D291.559MB)...= =20 13: memory allocated (0x7f262c000d50, size=3D18.001MB, rss=3D291.559MB)...= =20 12: memory allocated (0x7f264c000d50, size=3D18.001MB, rss=3D291.559MB)...= =20 Arena 0: system bytes =3D 81920 in use bytes =3D 80624 Arena 1: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 2: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 3: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 4: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 5: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 6: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 7: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 8: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 9: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 10: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 11: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 12: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 13: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 14: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 15: system bytes =3D 18890752 in use bytes =3D 18878800 Arena 16: system bytes =3D 18890752 in use bytes =3D 18878800 Total (incl. mmap): system bytes =3D 302333952 in use bytes =3D 302141424 max mmap regions =3D 16 max mmap bytes =3D 302055424 12: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 9: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 15: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 5: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 11: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 6: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 8: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 4: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 13: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 1: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 3: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 2: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 16: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 10: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 7: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) 14: deallocated (rss=3D291.559MB)... trimmed (rss=3D291.559MB) Arena 0: system bytes =3D 81920 in use bytes =3D 80624 Arena 1: system bytes =3D 18890752 in use bytes =3D 3392 Arena 2: system bytes =3D 18890752 in use bytes =3D 3392 Arena 3: system bytes =3D 18890752 in use bytes =3D 3392 Arena 4: system bytes =3D 18890752 in use bytes =3D 3392 Arena 5: system bytes =3D 18890752 in use bytes =3D 3392 Arena 6: system bytes =3D 18890752 in use bytes =3D 3392 Arena 7: system bytes =3D 18890752 in use bytes =3D 3392 Arena 8: system bytes =3D 18890752 in use bytes =3D 3392 Arena 9: system bytes =3D 18890752 in use bytes =3D 3392 Arena 10: system bytes =3D 18890752 in use bytes =3D 3392 Arena 11: system bytes =3D 18890752 in use bytes =3D 3392 Arena 12: system bytes =3D 18890752 in use bytes =3D 3392 Arena 13: system bytes =3D 18890752 in use bytes =3D 3392 Arena 14: system bytes =3D 18890752 in use bytes =3D 3392 Arena 15: system bytes =3D 18890752 in use bytes =3D 3392 Arena 16: system bytes =3D 18890752 in use bytes =3D 3392 Total (incl. mmap): system bytes =3D 302333952 in use bytes =3D 134896 max mmap regions =3D 16 max mmap bytes =3D 302055424 Now, RSS does not fall back, but keeps being at rss=3D291.559MB. Now if you change ALLOC_INCREMENT from 1024 to 16*1024, now the arenas are properly trimmed: 7: deallocated (rss=3D296.152MB)... trimmed (rss=3D296.152MB) 8: deallocated (rss=3D276.652MB)... trimmed (rss=3D276.652MB) 3: deallocated (rss=3D257.152MB)... trimmed (rss=3D257.152MB) 10: deallocated (rss=3D237.652MB)... trimmed (rss=3D237.652MB) 4: deallocated (rss=3D218.184MB)... trimmed (rss=3D218.184MB) 11: deallocated (rss=3D198.684MB)... trimmed (rss=3D198.684MB) 12: deallocated (rss=3D179.184MB)... trimmed (rss=3D179.184MB) 1: deallocated (rss=3D159.684MB)... trimmed (rss=3D159.684MB) 2: deallocated (rss=3D140.184MB)... trimmed (rss=3D140.184MB) 5: deallocated (rss=3D120.684MB)... trimmed (rss=3D120.684MB) 13: deallocated (rss=3D101.184MB)... trimmed (rss=3D101.184MB) 15: deallocated (rss=3D81.684MB)... trimmed (rss=3D81.684MB) 14: deallocated (rss=3D62.184MB)... trimmed (rss=3D62.184MB) 6: deallocated (rss=3D42.684MB)... trimmed (rss=3D42.684MB) 16: deallocated (rss=3D23.137MB)... trimmed (rss=3D23.137MB) 9: deallocated (rss=3D3.590MB)... trimmed (rss=3D3.590MB) dealloc stat: Arena 0: system bytes =3D 81920 in use bytes =3D 80624 Arena 1: system bytes =3D 135168 in use bytes =3D 3392 Arena 2: system bytes =3D 135168 in use bytes =3D 3392 Arena 3: system bytes =3D 135168 in use bytes =3D 3392 Arena 4: system bytes =3D 135168 in use bytes =3D 3392 Arena 5: system bytes =3D 135168 in use bytes =3D 3392 Arena 6: system bytes =3D 135168 in use bytes =3D 3392 Arena 7: system bytes =3D 135168 in use bytes =3D 3392 Arena 8: system bytes =3D 135168 in use bytes =3D 3392 Arena 9: system bytes =3D 135168 in use bytes =3D 3392 Arena 10: system bytes =3D 135168 in use bytes =3D 3392 Arena 11: system bytes =3D 135168 in use bytes =3D 3392 Arena 12: system bytes =3D 135168 in use bytes =3D 3392 Arena 13: system bytes =3D 135168 in use bytes =3D 3392 Arena 14: system bytes =3D 135168 in use bytes =3D 3392 Arena 15: system bytes =3D 135168 in use bytes =3D 3392 Arena 16: system bytes =3D 135168 in use bytes =3D 3392 Total (incl. mmap): system bytes =3D 2244608 in use bytes =3D 134896 max mmap regions =3D 16 max mmap bytes =3D 328007680 Also MALLOC_TOP_PAD_=3D0 helps. Or setting MALLOC_ARENA_MAX=3D1 9: deallocated (rss=3D291.422MB)... trimmed (rss=3D273.426MB) 16: deallocated (rss=3D273.426MB)... trimmed (rss=3D255.426MB) 7: deallocated (rss=3D255.426MB)... trimmed (rss=3D237.426MB) 14: deallocated (rss=3D237.426MB)... trimmed (rss=3D219.430MB) 8: deallocated (rss=3D219.430MB)... trimmed (rss=3D201.430MB) 4: deallocated (rss=3D201.430MB)... trimmed (rss=3D183.438MB) 1: deallocated (rss=3D183.438MB)... trimmed (rss=3D165.438MB) 5: deallocated (rss=3D165.438MB)... trimmed (rss=3D147.344MB) 10: deallocated (rss=3D147.344MB)... trimmed (rss=3D129.340MB) 11: deallocated (rss=3D129.340MB)... trimmed (rss=3D111.348MB) 6: deallocated (rss=3D93.348MB)... trimmed (rss=3D93.348MB) 12: deallocated (rss=3D93.348MB)... trimmed (rss=3D75.355MB) 15: deallocated (rss=3D75.355MB)... trimmed (rss=3D57.352MB) 2: deallocated (rss=3D57.352MB)... trimmed (rss=3D39.410MB) 3: deallocated (rss=3D39.410MB)... trimmed (rss=3D21.309MB) 13: deallocated (rss=3D3.375MB)... trimmed (rss=3D3.375MB) dealloc stat: Arena 0: system bytes =3D 241664 in use bytes =3D 98800 Total (incl. mmap): system bytes =3D 241664 in use bytes =3D 98800 max mmap regions =3D 16 max mmap bytes =3D 303628288 --=20 You are receiving this mail because: You are on the CC list for the bug.=