From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) by sourceware.org (Postfix) with ESMTPS id 9FCFE385801F for ; Tue, 18 Jan 2022 12:45:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 9FCFE385801F X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.178.34] ([77.182.222.192]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N1wlv-1mCaZi2wfZ-012IFK; Tue, 18 Jan 2022 13:45:19 +0100 Subject: Re: expecations of glibcs (de)allocator To: Dominik Csapak , libc-help@sourceware.org Cc: Wolfgang Bumiller References: From: Christian Hoff Message-ID: <7a0294c9-1ff0-95fb-0231-4366ba014c4a@gmx.net> Date: Tue, 18 Jan 2022 13:45:17 +0100 User-Agent: Mozilla/5.0 (X11; Linux armv7l; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-GB X-Provags-ID: V03:K1:3xm4fgho8TJgGvcUS3H+TmoMxVjTP06MNsrVRN6jZ9gN+TsPrmd RvVi8QK+Xttug+myLt84W72UCBOZt8UTAUNVcyLkIQndbI6s1RAyXN0F9IE0ymSHdVFdtPN CDnspIbqQcqoedk7Va5QQ1bPoEc4hfZz/AfsAnbLM/H3N4JvkrH/LP0T9Bztw2AUtnO4NyN whUUvTLHzQa4jD3eJ5nTA== X-UI-Out-Filterresults: notjunk:1;V03:K0:DQ3fnM7eeE8=:HvoIQZfYLqt9/1L9Kggst2 nbsxf0DoqUMWLtUdaXmwf4FfieBSYyk8ugJvKGzRkduhCOm7d/UehlQRpNvf/EmW2t+nKapz7 eQKIz6yk3dgSOFGG3X4etq30FkGKVkSU5FvueWCtESBjUMpG0PPVF5b3bJ3uaKPw2lvr5jVV8 0CDLdK9ROif08sL2w14rnQR92PNdKpxZOX53tB1cTlrAzqbiZMy2VroNPotR5231N3sazfiky 8U4juacYCrr/6cXMacY8PELO4FnmWSePw1i8XCjgkQZRTs+anc38uh+17AUT8l5LwM2ghJ4xn 1ZaAfcDrnyhW8CHnM3QZP9hfMaoBcUpoEdNmrQKHMlWCzgfUIDb3I2THMC79T0Oabde2ITZ0B 4cfTiNuj27fLp4Jh089OuQQ3F4lQRf1mXk7EiBaKb7bqIcGh/oYlvrlnvLhcCanlZ/VPveIdN FLl7pVVl09qZBsTMBIdwAME8NbLAI2P97LY06hzyhunARdzo/j6VKEJcleC0w8XGUmI1A2JiJ xqIOY6+QxqcX5Oc78admjrFHZy61dLd8wV5ynVtrT8yAffQdzMQ9qKoA2S8k7BCg7D7WptJ5w zYBomffHl5dW1IhN/f00oR/1TjDWsVPw6vJ5oOe70YX0LD+U9NvCVEvoPS9qRRHTiWTKoNoGK H0zHG40qZLOA0Nc1tQPfBo+PYTHTgk3AS12uyfN+fVTu838200tEg+BRI4c0ghk2vOLPj16r5 dbrNazjeANXmj94stz5E4OeTSH4SsUyBN8FNA3UbyhaOT3uhhLWCkRC4MwRue748qjgWdeut9 FVrru7+RuGTs9fg4JMgryBMZ6goTmSdkph8b42n7rMexkqML2DQeFRxBj1HL+f1J+dRkUjR1+ KK6yorddk+e0HJaG6/mlJMk17GrI5ORePqL9tYnxkN7Kb/E7k/dStTxxEE+0t7Dwz73+TF8gQ pxNzHZBLe11AhfOV3kUgxlakZMjE8UKVLdU9XBaqXaiJVHdHWj3kXfUXGJ13Bx5ht6CqOP44l 7lg0TUQ43/pnAv72w3exgiW5S7cowg6jRYDp3LHHyMnV4ACuSXH5sbIKuLYICc8GpGaoSULFr ZCg/G1MQDAmhn0= X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00, BODY_8BITS, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-help@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-help mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jan 2022 12:45:32 -0000 Hello Dominik, this looks a lot like the same issue I faced in December, and because of which I also contacted the Glibc mailing list. You can read the e-mail thread in the mailing list archives here: https://sourceware.org/pipermail/libc-help/2021-November/006048.html . I described two separate issues in that mail - the first issue is the one you are also facing. See especially the very helpful reply from Carlos O'Donnell,which explains this behaviour. Basically, this a known problem in Glibc. If there is a memory chunk with a long lifetime on top of the heap, it prevents the whole heap from being trimmed. malloc_trim() however is able to reclaim the memory, because it walks all the way down the heap. So, currently, the only available solution for you is to call malloc_trim() or to use a different memory allocator (e.g. jemalloc). There has been a discussion on this list if glibc should call malloc_trim() internally if there is a lot of free memory on the heap that cannot be freed because a chunk with a long lifetime is keeping the heap from being trimmed. But I think no one followed up on this. This issue becomes even worse if you need to do the memory-intensive computations multiple times in different threads. In that case, the memory usage of the program continues to grow far beyond what the actual needed peak memory of your program is. This is because the threads are assigned to different arenas and each arena gets larger and larger over the lifetime of the program. In my opinion, glibc's memory allocator has an architectural issue. Larger allocations shouldn't be served by the individual arenas, but by one central allocation area - just as tcmalloc redirects allocations larger than 256 KB to the backend. That way, previously free()'ed large memory chunks can easily be reused by another thread, even though that other thread is served by a different arena. We are in the process of changing our software to use tcmalloc instead of glibc malloc because of this issue. I also noticed that jemalloc is doing a better job than tcmalloc to return memory back to the OS, so this may also be an option. Best regards, =C2=A0=C2=A0 Christian On 1/18/22 12:32 PM, Dominik Csapak wrote: > Hi, > > i am sorry in advance for the wall of text, but maybe this list can help > me, or at least shed some light on issues we have had regarding memory > (de)allocation. > > The setup: we have a long running daemon written in rust for x86_64 > Linux (especially Debian, currently bullseye) that uses the default rust > malloc/free which is AFAIK glibcs malloc/free (de)allocator. > > The daemon makes heavy use of the async rust frameworks tokio and hyper. > Our problem is the following: during the lifetime of the daemon, > there are some memory heavy operations (network traffic/disk io/etc.) > and thus it allocates quite a bit of memory, but at the end of the > operation, this memory is still allocated to the program > (we checked with e.g. htop/ps for the RSS/resident memory) > and even letting it run for extended periods of time does not really > releases the memory. (We had customers where it retained over 5GiB > of memory basically doing nothing). > > There are some things we tried (e.g. by tuning the options mentioned in > mallopt(3)): > > * calling malloc_trim(0) at the end of the program released > =C2=A0 the memory (see the reproducer at the end) > > * Changing M_TRIM_THRESHOLD did not change anything, the > =C2=A0 memory is still allocated to the program (even setting > =C2=A0 it to 0 or 1 did not make a difference) > =C2=A0 This surprised us quite a bit since the documentation reads > =C2=A0 like this would release the memory sooner and because > =C2=A0 malloc_trim also released it. > > * Setting M_MMAP_THRESHOLD option to a very low value fixed > =C2=A0 the behavior (but not really surprising). > > (ofc changing the allocator altogether e.g. jemalloc or musl > also changed the behaviour, but we'd like to avoid that if > possible) > > We have some small reproducer that trigger this behavior too. > (i paste it at the end of the mail). It starts a large number > of async tasks, and waits for them to finish, then drops the > async runtime completely (at this point the program cannot really > have any memory in use, but uses still memory according to htop). > to debug we added a 'malloc_trim' at the end which actually releases > the memory to the os. > > So from our side it looks like that either > > * we (or the frameworks) trigger some bad/worst case in the > =C2=A0 memory allocation pattern. In this case it would be interesting > =C2=A0 how we could check/debug that and, if possible, how to fix > =C2=A0 it in our program > > * glibcs allocator has some bug regarding releasing memory to the > =C2=A0 os. while i personally doubt that, it's curious that tuning the > =C2=A0 M_TRIM_THRESHOLD does not seem to do anything. It would also > =C2=A0 be interesting how to debug/check that ofc. > > I hope that this list is not completely wrong, but if it is, just say > so (and maybe point me in the right direction) > > thanks > Dominik > > > ---- below is the reproducer (note: uses about 1.4GiB peak memory) ---- > use std::io; > use std::time::Duration; > use tokio::task; > > extern "C" { > =C2=A0=C2=A0=C2=A0 fn malloc_trim(pad: libc::size_t) -> i32; > } > > async fn wait(_i: usize) { > =C2=A0=C2=A0=C2=A0 let delay_in_seconds =3D Duration::new(2, 0); > =C2=A0=C2=A0=C2=A0 tokio::time::sleep(delay_in_seconds).await; > } > > fn main() { > =C2=A0=C2=A0=C2=A0 let rt =3D tokio::runtime::Runtime::new().unwrap(); > =C2=A0=C2=A0=C2=A0 rt.block_on(async move { > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 let num =3D 1_000_000; > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for i in 0..num { > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 task:= :spawn(async move { > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 wait(i).await; > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }); > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wait(0).await; > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wait(0).await; > =C2=A0=C2=A0=C2=A0 }); > > =C2=A0=C2=A0=C2=A0 println!("all tasks should be finished"); > =C2=A0=C2=A0=C2=A0 let mut buffer =3D String::new(); > =C2=A0=C2=A0=C2=A0 io::stdin().read_line(&mut buffer).expect("error"); > > =C2=A0=C2=A0=C2=A0 drop(rt); > =C2=A0=C2=A0=C2=A0 println!("dropped runtime"); > > =C2=A0=C2=A0=C2=A0 let mut buffer =3D String::new(); > =C2=A0=C2=A0=C2=A0 io::stdin().read_line(&mut buffer).expect("error"); > > =C2=A0=C2=A0=C2=A0 unsafe { malloc_trim(0); }; > =C2=A0=C2=A0=C2=A0 println!("called malloc_trim"); > > =C2=A0=C2=A0=C2=A0 let mut buffer =3D String::new(); > =C2=A0=C2=A0=C2=A0 io::stdin().read_line(&mut buffer).expect("error"); > } >