From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 73509 invoked by alias); 27 Sep 2018 18:16:17 -0000 Mailing-List: contact libc-help-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Post: List-Help: , Sender: libc-help-owner@sourceware.org Received: (qmail 73317 invoked by uid 89); 27 Sep 2018 18:15:56 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE,TIME_LIMIT_EXCEEDED autolearn=unavailable version=3.3.2 spammy=HTo:U*fw, student, terrible, door X-HELO: mail-qt1-f181.google.com Received: from mail-qt1-f181.google.com (HELO mail-qt1-f181.google.com) (209.85.160.181) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 27 Sep 2018 18:15:31 +0000 Received: by mail-qt1-f181.google.com with SMTP id z13-v6so3801383qts.5 for ; Thu, 27 Sep 2018 11:15:10 -0700 (PDT) Return-Path: Received: from [10.150.73.190] (247.sub-174-196-128.myvzw.com. [174.196.128.247]) by smtp.gmail.com with ESMTPSA id o185-v6sm326396qke.34.2018.09.27.11.15.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Sep 2018 11:15:07 -0700 (PDT) Subject: Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). To: Florian Weimer , ritesh sonawane Cc: libc-help@sourceware.org References: <87bm8kefql.fsf@oldenburg.str.redhat.com> <87sh1uj0l8.fsf@mid.deneb.enyo.de> From: Carlos O'Donell Openpgp: preference=signencrypt Message-ID: Date: Thu, 27 Sep 2018 18:16:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <87sh1uj0l8.fsf@mid.deneb.enyo.de> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2018-09/txt/msg00024.txt.bz2 On 9/27/18 1:45 PM, Florian Weimer wrote: > * ritesh sonawane: > >> Yes it is 64 Bit target. >> >> Fragmentation means when malloc() request is more than threshold >> value, memory is allocated using mmap(). Due to size alignment with >> page size, there is memory wastage per request. e.g. Our system is >> having 48GB memory, then total Number of malloc() requests (each >> 64MB) will be 380 and total used memory is 23 GB out of 48GB. > > I see. I agree that's a problem, and changing the malloc threshold > could be a solution. > >> The system (NEC SX-Aurora TSUBASA) on which we are currently working >> is having huge page size and each process can have 16 GB and 512 GB >> address space for 2 MB and 64 MB page size respectively. > > I'm not familiar with that system and haven't seen the glibc port, > sorry. I used to work next door to a NEC SX-6 as a student, but > that's it. > >> Also there is no particular limit on maximum heap size and Each >> thread can easily consumes 2 GB of address space and thats why we >> want to increase the M_MMAP_THRESHOLD to 1 GB. > > If you do that, you also have to increase the heap size to something > like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c). The default of 2 * > DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that > DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB). Otherwise you will have > substantial fragmentation for allocation requests between 2 GiB and > HEAP_MAX_SIZE. Right. * Change malloc/malloc.c DEFAULT_MMAP_TRESHOLD_MAX to 16GiB. * This in turn sets HEAP_MAX_SIZE to 32GiB. * Now you can set M_MMAP_THRESHOLD to any value 0 > x <= (32GiB - a couple of pages). The key here is that an arena is made up of discontiguous heaps in a chain which are logically "the heap" for the attached thread (non-main arena). You can't have a request that crosses heap boundaries, so if you set M_MMAP_THRESHOLD larger than the heap size it is impossible to service the request except through mmap. All of this requires a custom glibc. I *had* some patches to set HEAP_MAX_SIZE for testing, but it can only be set at startup via a tunable because once set all allocations have to use it for masking to compute chunk->heap mapping. I never had a reason to change it... but this example is such a reason. I don't think it's a terrible idea to allow a tunable for HEAP_MAX_SIZE or DEFAULT_MMAP_THRESHOLD_MAX, but can only be set early and not late, which is fine. > I expect each heap will only allocate two pages via page faults, but > will reserve HEAP_MAX_SIZE bytes of address space. Agreed. > If that's a problem, you could also make these changes, but set the > maximum arena count to 1, then only the main arena will be used, which > is sbrk-based. The main arena doesn't need the coarse-grained > mappings with large power-of-two sizes. Right. -- Cheers, Carlos.