From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from forward501j.mail.yandex.net (forward501j.mail.yandex.net [5.45.198.251]) by sourceware.org (Postfix) with ESMTPS id D1CB23858D35 for ; Thu, 25 Nov 2021 17:46:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D1CB23858D35 Received: from myt6-0c96ceb81798.qloud-c.yandex.net (myt6-0c96ceb81798.qloud-c.yandex.net [IPv6:2a02:6b8:c12:2ca1:0:640:c96:ceb8]) by forward501j.mail.yandex.net (Yandex) with ESMTP id 959B3623272; Thu, 25 Nov 2021 20:46:32 +0300 (MSK) Received: from myt5-89cdf5c4a3a5.qloud-c.yandex.net (myt5-89cdf5c4a3a5.qloud-c.yandex.net [2a02:6b8:c12:289b:0:640:89cd:f5c4]) by myt6-0c96ceb81798.qloud-c.yandex.net (mxback/Yandex) with ESMTP id yx9lJO7njc-kWCqet6s; Thu, 25 Nov 2021 20:46:32 +0300 Received: by myt5-89cdf5c4a3a5.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id EGx95KytmI-kWLqs8js; Thu, 25 Nov 2021 20:46:32 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) X-Yandex-Fwd: 2 Message-ID: <560ed6888a62b21362cda5385655c3a84fd354b9.camel@yandex.ru> Subject: Re: Excessive memory consumption when using malloc() From: Konstantin Kharlamov To: Christian Hoff , libc-help@sourceware.org Date: Thu, 25 Nov 2021 20:46:31 +0300 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.40.4 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-help@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-help mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Nov 2021 17:46:37 -0000 On Thu, 2021-11-25 at 18:20 +0100, Christian Hoff via Libc-help wrote: > Hello all, > > we are facing the a problem with the memory allocator in glibc 2.17 on > RHEL 7.9. Or application allocates about 10 GB of memory (split into > chunks that are each around 512 KB large). This memory is used for some > computations and released afterwards. After a while, the application is > running the same computations again, but this time in different threads. > The first issue we are seeing is that - after the computations are done > - the 10 GB of memory is not released back to the operating system. Only > after calling malloc_trim() manually with GDB, the size of the process > shrinks dramatically from ~10GB to 400 MB. So, at this point, the unused > memory from the computations is finally returned to the OS. > > Our wish would be that the memory is returned to the OS without us > having to call malloc_trim(). And I understand that glibc also trims the > heap when there is sufficient free space in top of it (the > M_TRIM_THRESHOLD in mallopt() controls when this should happen). What > could be the reason why this is not working in our case? Could it be > related to heap fragmentation? But assuming that is the reason, why is > malloc_trim() nevertheless able to free this memory? > I assume the bug you stumbled upon is this one https://sourceware.org/bugzilla/show_bug.cgi?id=27103 I'm not sure there is anything more to say except that it is a long-standing glibc bug (it seems to have been known long before I reported it in 2020), and malloc_trim is the official workaround to it. For you purposes though you could perhaps try other malloc implementations such as jemalloc. Try and see if it fixes these problems. Please report back if you try it, I am curious if that can be used as another workaround. > And then we also have one other problem. The first run of the > computations is always fine: we allocate 10 GB of memory and the > application grows to 10 GB. Afterwards, we release those 10 GB of memory > since the computations are now done and at this point the freed memory > is returned back to the allocator (however, the size of the process > remains 10 GB unless we call malloc_trim()). But if we now re-run the > same computations again a second time (this time using different > threads), a problem occurs. In this case, the size of the application > grows well beyond 10 GB. It can get 20 GB or larger and the process is > eventually killed because the system runs out of memory. > > Do you have any idea why this happens? To me it seems like the threads > are assigned to different arenas and therefore the previously freed 10 > GB of memory can not be re-used as they are in different arenas. Is that > possible? > > A workaround I have found is to set M_MMAP_THRESHOLD to 128 KB - then > the memory for the computations is always allocated using mmap() and > returned back to the system immediately when it is free()'ed. This > solves both of the issues. But I am afraid that this workaround could > degrade the performance of our application. So, we are grateful for any > better solution to this problem. > > Kind regards, > >     Christian Hoff >