From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) by sourceware.org (Postfix) with ESMTPS id C73493858D35 for ; Thu, 25 Nov 2021 17:20:42 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C73493858D35 X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.178.34] ([77.180.249.156]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MSKu0-1nET6P0Cwi-00Scf3 for ; Thu, 25 Nov 2021 18:20:41 +0100 To: libc-help@sourceware.org From: Christian Hoff Subject: Excessive memory consumption when using malloc() Message-ID: Date: Thu, 25 Nov 2021 18:20:39 +0100 User-Agent: Mozilla/5.0 (X11; Linux armv7l; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-GB X-Provags-ID: V03:K1:VtkxnVC76bphX+Z6uLeM9RHEBtxfwWwkjGLBjHq/COE1YgpLbJt fEhrbaGQw1k2ZmsplOHJ7vDWAJ5DoaAyxaaQgTOw/H9MooJ9cVMPJI/IJDabJf0k3md2nWW 0T18yO/f6dVmg9aI4INfhzsFMNlAaPiYjkE7ZasoIPlNN0mqDELKXAlUnrY5wTlhPzHXOIe R4uWfOrW9i+ifXYJPEtyg== X-UI-Out-Filterresults: notjunk:1;V03:K0:t90YEhFC3co=:V9aodOE68DxCcL3Ea43eIm GQMgBrZIsucT8b/ppdb1WPUkgTOETgSDnrVvWIFFlPCWIcLcrb+Xvsu1CREbsh14oJoT1jT4W OnUabLQpox8CvLaV0v1mObMnFP5skSUsVleRJNozYE31M5/dYaPsIOj59T4EXo8HA0xrIivW7 5DTe15vhJB7duzzZSPSwmfTkN8XdwgvBp4WzwN2VS9u1eqx7Oeb1vLTEeezY7YGosUuZgMul8 SRe/PPpj/IFV0HCr8v915iCcLRqbkag/WrkgNhytHBfhnS6DFMk1vq5zfvI3DRypoc191mSWI iRGkSPnjVKLE6KOsNAG3jvX+idXEzg+L1mojBC+CzH9VP/tKEdVAoeJyVspgPq0TQ0CenpWd7 b3bOeArgLA51MCNUmfBhMGIMTJ0V5yf2IYgda+pP1Ywq4BTSYUdP6R0YHOcercmxX6uRkurqH TLnNZr3JG8Nl1/gvxyO7LpcLHJe4cBGLyoriYgS2OTI+HqEn5LL1J+rMFXJuFJ05V4XE5h8co Ytk7YIQlhoQWD1fUpCh/n55j7XgkMw36PqvY3nVOF9Sd3PAbaPPkNsM8dnaDnfpyF0xz+FgUw hxwt3YCOsFiYrLvytU5gzmneKhvc00fhh5pUT7mkhpOUdmCHF5VI10iirCNbW1Qum6evKoW83 piDCyGGhtaRXT1/VYyxthQ0zXX6UYABRa9t88oTyVE8BPeasbyYZkYhq2r9KqHCDiihVMaYEO +IxSy2haKjKs9zFtw6qeTowJART8Q9ErtPnn4ukPtInC1GmlYypcRU5DBzBpMoJpeBnUfwSP2 xgRKC6EEqUJk4xZkz7R9s8TNvNcHQATdK0ssnIsNYOlysLoDXFs9v+IrPwM8w6+n+hxaSKxYi tCiDElAGWqtMMCscrE531w1SwQiGjeR9RK2Yjf/JSUSZIj6yC1BTx3xluRm2scrM4iuQa/phz qAJgF7AK3zuU4jbL0cPoaooB5YJtWXCTYqlB25ehqUj9PfFqHAkwiyeq2kx91k7SEZOqa0Yr7 l6QEuIlKG03oIpfE+X8kmsyecvz7r/stCM0hIdZk0eMftX1t3DOQShuf52ImpyNx4pi4fXD/I Pm1/uap1SXbakg= X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-help@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-help mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Nov 2021 17:20:44 -0000 Hello all, we are facing the a problem with the memory allocator in glibc 2.17 on RHEL 7.9. Or application allocates about 10 GB of memory (split into chunks that are each around 512 KB large). This memory is used for some computations and released afterwards. After a while, the application is running the same computations again, but this time in different threads. The first issue we are seeing is that - after the computations are done - the 10 GB of memory is not released back to the operating system. Only after calling malloc_trim() manually with GDB, the size of the process shrinks dramatically from ~10GB to 400 MB. So, at this point, the unused memory from the computations is finally returned to the OS. Our wish would be that the memory is returned to the OS without us having to call malloc_trim(). And I understand that glibc also trims the heap when there is sufficient free space in top of it (the M_TRIM_THRESHOLD in mallopt() controls when this should happen). What could be the reason why this is not working in our case? Could it be related to heap fragmentation? But assuming that is the reason, why is malloc_trim() nevertheless able to free this memory? And then we also have one other problem. The first run of the computations is always fine: we allocate 10 GB of memory and the application grows to 10 GB. Afterwards, we release those 10 GB of memory since the computations are now done and at this point the freed memory is returned back to the allocator (however, the size of the process remains 10 GB unless we call malloc_trim()). But if we now re-run the same computations again a second time (this time using different threads), a problem occurs. In this case, the size of the application grows well beyond 10 GB. It can get 20 GB or larger and the process is eventually killed because the system runs out of memory. Do you have any idea why this happens? To me it seems like the threads are assigned to different arenas and therefore the previously freed 10 GB of memory can not be re-used as they are in different arenas. Is that possible? A workaround I have found is to set M_MMAP_THRESHOLD to 128 KB - then the memory for the computations is always allocated using mmap() and returned back to the system immediately when it is free()'ed. This solves both of the issues. But I am afraid that this workaround could degrade the performance of our application. So, we are grateful for any better solution to this problem. Kind regards, =C2=A0=C2=A0 Christian Hoff