From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 102492 invoked by alias); 31 Aug 2018 19:01:09 -0000 Mailing-List: contact libc-help-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Post: List-Help: , Sender: libc-help-owner@sourceware.org Received: (qmail 102104 invoked by uid 89); 31 Aug 2018 19:00:49 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00,KAM_LAZY_DOMAIN_SECURITY,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.2 spammy=million, measuring, H*M:8796, unbounded X-HELO: mail-qt0-f171.google.com Received: from mail-qt0-f171.google.com (HELO mail-qt0-f171.google.com) (209.85.216.171) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 31 Aug 2018 19:00:47 +0000 Received: by mail-qt0-f171.google.com with SMTP id x7-v6so15698245qtk.5 for ; Fri, 31 Aug 2018 12:00:26 -0700 (PDT) Return-Path: Received: from [10.150.73.190] (169.sub-174-192-2.myvzw.com. [174.192.2.169]) by smtp.gmail.com with ESMTPSA id g14-v6sm7551793qtc.66.2018.08.31.12.00.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Aug 2018 12:00:23 -0700 (PDT) Subject: Re: Query regarding malloc_trim To: Sainath Latkar , libc-maintainers@gnu.org, libc-help@sourceware.org References: From: Carlos O'Donell Openpgp: preference=signencrypt Message-ID: Date: Fri, 31 Aug 2018 19:01:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-IsSubscribed: yes X-SW-Source: 2018-08/txt/msg00039.txt.bz2 On 08/30/2018 09:55 AM, Sainath Latkar wrote: > I observed weird behavior with memory utilization of our application. The > issue narrows down to the usage of *std::queue* putting small chunks of > data in the queue and emptying the queue completely after some time. How small? > In the life cycle of our application, a particular queue gets populated > with more than a million records and after around 7 to 10 minutes, we empty > the queue completely. At this point, logically the heap memory held by > these objects should be freed but it doesn’t happen. One more observation > is that if we put enough memory usage load on our system this surplus > memory goes into swap. Can you confirm that you are measuring and observing RSS usage (not VSZ)? > So the program is never freeing up the memory. This memory gets freed at > the end of .org applications life cycle. This indeed is not a memory leak, > because if we try to acquire more heap memory, application reuses the > surplus memory and doesn’t allocate more. Correct, this is a performance optimization. > As I understood after reading up a bit, as part of optimization, Glibc > allocators cache this memory for later use. Using *malloc_trim* we can > return this heap memory to the system. I tried using *malloc_trim* and it > works as expected OK. > My question is, is there any other way around this optimization, apart from > using malloc trim and *M_TRIM_THRESHOLD*? Also note that we are not using > c++ 11 so using Shriya to fit is out of the picture. I assume you mean "shrink_to_fit"? The system allocator has many tunable parameters that you must come to understand, and set based on your application workload. If you want optimal allocator performance you need to tune it based on the upcoming workload usage (or as soon as you can). If your objects fit into fastbins it may be the case that fastbins is growing unbounded. You can test this theory by trying mallopt and set M_MXFAST to 0 to disable fastbins (but leave tcache enabled which has a fixed limit). -- Cheers, Carlos.