* Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). @ 2018-09-26 9:46 ritesh sonawane 2018-09-26 10:05 ` Florian Weimer 0 siblings, 1 reply; 9+ messages in thread From: ritesh sonawane @ 2018-09-26 9:46 UTC (permalink / raw) To: libc-help Hi All, I am working on system with page size 2MB and 64MB and use of malloc() leads to fragmentation. To avoid this I want to increase the size of M_MMAP_THRESHOLD to 1GB. Is there any impact of this on bin index calculation ? Need some help. Thanks and Regards, Ritesh ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-26 9:46 Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc() ritesh sonawane @ 2018-09-26 10:05 ` Florian Weimer 2018-09-27 16:55 ` ritesh sonawane 0 siblings, 1 reply; 9+ messages in thread From: Florian Weimer @ 2018-09-26 10:05 UTC (permalink / raw) To: ritesh sonawane; +Cc: libc-help * ritesh sonawane: > I am working on system with page size 2MB and 64MB and use of malloc() > leads to fragmentation. What kind of fragmentation? > To avoid this I want to increase the size of M_MMAP_THRESHOLD to > 1GB. Is there any impact of this on bin index calculation ? I assume this is for a 64-bit target. You will also have to increase the heap size (the amount by which thread arenas grow), otherwise you will end up with much more fragmenation for certain allocation sizes. This will mean that every thread consumes at least 2 GiB of address space. Thanks, Florian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-26 10:05 ` Florian Weimer @ 2018-09-27 16:55 ` ritesh sonawane 2018-09-27 17:46 ` Florian Weimer 0 siblings, 1 reply; 9+ messages in thread From: ritesh sonawane @ 2018-09-27 16:55 UTC (permalink / raw) To: Florian Weimer; +Cc: libc-help ++libc-help Thanks for quick reply . Yes it is 64 Bit target. Fragmentation means when malloc() request is more than threshold value, memory is allocated using mmap(). Due to size alignment with page size, there is memory wastage per request. e.g. Our system is having 48GB memory, then total Number of malloc() requests (each 64MB) will be 380 and total used memory is 23 GB out of 48GB. The system (NEC SX-Aurora TSUBASA) on which we are currently working is having huge page size and each process can have 16 GB and 512 GB address space for 2 MB and 64 MB page size respectively. Also there is no particular limit on maximum heap size and Each thread can easily consumes 2 GB of address space and thats why we want to increase the M_MMAP_THRESHOLD to 1 GB. Best Regards, Ritesh On Wed, Sep 26, 2018 at 3:35 PM Florian Weimer <fweimer@redhat.com> wrote: > * ritesh sonawane: > > > I am working on system with page size 2MB and 64MB and use of malloc() > > leads to fragmentation. > > What kind of fragmentation? > > > To avoid this I want to increase the size of M_MMAP_THRESHOLD to > > 1GB. Is there any impact of this on bin index calculation ? > > I assume this is for a 64-bit target. > > You will also have to increase the heap size (the amount by which thread > arenas grow), otherwise you will end up with much more fragmenation for > certain allocation sizes. This will mean that every thread consumes at > least 2 GiB of address space. > > Thanks, > Florian > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-27 16:55 ` ritesh sonawane @ 2018-09-27 17:46 ` Florian Weimer 2018-09-27 18:16 ` Carlos O'Donell 2018-09-28 10:03 ` ritesh sonawane 0 siblings, 2 replies; 9+ messages in thread From: Florian Weimer @ 2018-09-27 17:46 UTC (permalink / raw) To: ritesh sonawane; +Cc: libc-help * ritesh sonawane: > Yes it is 64 Bit target. > > Fragmentation means when malloc() request is more than threshold > value, memory is allocated using mmap(). Due to size alignment with > page size, there is memory wastage per request. e.g. Our system is > having 48GB memory, then total Number of malloc() requests (each > 64MB) will be 380 and total used memory is 23 GB out of 48GB. I see. I agree that's a problem, and changing the malloc threshold could be a solution. > The system (NEC SX-Aurora TSUBASA) on which we are currently working > is having huge page size and each process can have 16 GB and 512 GB > address space for 2 MB and 64 MB page size respectively. I'm not familiar with that system and haven't seen the glibc port, sorry. I used to work next door to a NEC SX-6 as a student, but that's it. > Also there is no particular limit on maximum heap size and Each > thread can easily consumes 2 GB of address space and thats why we > want to increase the M_MMAP_THRESHOLD to 1 GB. If you do that, you also have to increase the heap size to something like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c). The default of 2 * DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB). Otherwise you will have substantial fragmentation for allocation requests between 2 GiB and HEAP_MAX_SIZE. I expect each heap will only allocate two pages via page faults, but will reserve HEAP_MAX_SIZE bytes of address space. If that's a problem, you could also make these changes, but set the maximum arena count to 1, then only the main arena will be used, which is sbrk-based. The main arena doesn't need the coarse-grained mappings with large power-of-two sizes. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-27 17:46 ` Florian Weimer @ 2018-09-27 18:16 ` Carlos O'Donell 2018-09-28 10:22 ` ritesh sonawane 2018-09-28 10:03 ` ritesh sonawane 1 sibling, 1 reply; 9+ messages in thread From: Carlos O'Donell @ 2018-09-27 18:16 UTC (permalink / raw) To: Florian Weimer, ritesh sonawane; +Cc: libc-help On 9/27/18 1:45 PM, Florian Weimer wrote: > * ritesh sonawane: > >> Yes it is 64 Bit target. >> >> Fragmentation means when malloc() request is more than threshold >> value, memory is allocated using mmap(). Due to size alignment with >> page size, there is memory wastage per request. e.g. Our system is >> having 48GB memory, then total Number of malloc() requests (each >> 64MB) will be 380 and total used memory is 23 GB out of 48GB. > > I see. I agree that's a problem, and changing the malloc threshold > could be a solution. > >> The system (NEC SX-Aurora TSUBASA) on which we are currently working >> is having huge page size and each process can have 16 GB and 512 GB >> address space for 2 MB and 64 MB page size respectively. > > I'm not familiar with that system and haven't seen the glibc port, > sorry. I used to work next door to a NEC SX-6 as a student, but > that's it. > >> Also there is no particular limit on maximum heap size and Each >> thread can easily consumes 2 GB of address space and thats why we >> want to increase the M_MMAP_THRESHOLD to 1 GB. > > If you do that, you also have to increase the heap size to something > like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c). The default of 2 * > DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that > DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB). Otherwise you will have > substantial fragmentation for allocation requests between 2 GiB and > HEAP_MAX_SIZE. Right. * Change malloc/malloc.c DEFAULT_MMAP_TRESHOLD_MAX to 16GiB. * This in turn sets HEAP_MAX_SIZE to 32GiB. * Now you can set M_MMAP_THRESHOLD to any value 0 > x <= (32GiB - a couple of pages). The key here is that an arena is made up of discontiguous heaps in a chain which are logically "the heap" for the attached thread (non-main arena). You can't have a request that crosses heap boundaries, so if you set M_MMAP_THRESHOLD larger than the heap size it is impossible to service the request except through mmap. All of this requires a custom glibc. I *had* some patches to set HEAP_MAX_SIZE for testing, but it can only be set at startup via a tunable because once set all allocations have to use it for masking to compute chunk->heap mapping. I never had a reason to change it... but this example is such a reason. I don't think it's a terrible idea to allow a tunable for HEAP_MAX_SIZE or DEFAULT_MMAP_THRESHOLD_MAX, but can only be set early and not late, which is fine. > I expect each heap will only allocate two pages via page faults, but > will reserve HEAP_MAX_SIZE bytes of address space. Agreed. > If that's a problem, you could also make these changes, but set the > maximum arena count to 1, then only the main arena will be used, which > is sbrk-based. The main arena doesn't need the coarse-grained > mappings with large power-of-two sizes. Right. -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-27 18:16 ` Carlos O'Donell @ 2018-09-28 10:22 ` ritesh sonawane 2018-09-28 10:40 ` Florian Weimer 0 siblings, 1 reply; 9+ messages in thread From: ritesh sonawane @ 2018-09-28 10:22 UTC (permalink / raw) To: carlos; +Cc: fw, libc-help On Thu, Sep 27, 2018 at 11:45 PM Carlos O'Donell <carlos@redhat.com> wrote: > On 9/27/18 1:45 PM, Florian Weimer wrote: > > * ritesh sonawane: > > > >> Yes it is 64 Bit target. > >> > >> Fragmentation means when malloc() request is more than threshold > >> value, memory is allocated using mmap(). Due to size alignment with > >> page size, there is memory wastage per request. e.g. Our system is > >> having 48GB memory, then total Number of malloc() requests (each > >> 64MB) will be 380 and total used memory is 23 GB out of 48GB. > > > > I see. I agree that's a problem, and changing the malloc threshold > > could be a solution. > > > >> The system (NEC SX-Aurora TSUBASA) on which we are currently working > >> is having huge page size and each process can have 16 GB and 512 GB > >> address space for 2 MB and 64 MB page size respectively. > > > > I'm not familiar with that system and haven't seen the glibc port, > > sorry. I used to work next door to a NEC SX-6 as a student, but > > that's it. > > > >> Also there is no particular limit on maximum heap size and Each > >> thread can easily consumes 2 GB of address space and thats why we > >> want to increase the M_MMAP_THRESHOLD to 1 GB. > > > > If you do that, you also have to increase the heap size to something > > like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c). The default of 2 * > > DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that > > DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB). Otherwise you will have > > substantial fragmentation for allocation requests between 2 GiB and > > HEAP_MAX_SIZE. > > Right. > > * Change malloc/malloc.c DEFAULT_MMAP_TRESHOLD_MAX to 16GiB. > * This in turn sets HEAP_MAX_SIZE to 32GiB. > * Now you can set M_MMAP_THRESHOLD to any value 0 > x <= (32GiB - a couple > of pages). > > Thanks for suggestions, Right now requirement of M_MMAP_THRESHOLD is Up to 1GB only. So I set DEFAULT_MMAP_TRESHOLD_MAX to 1GB ( for 2MB and 64MB page size) and testing the performance. But I will check with above limits also. Apart from changing threshold value do I need to take care of bin index ? I have tried to understand the binning of malloc(), and found that its not necessary. The key here is that an arena is made up of discontiguous heaps in > a chain which are logically "the heap" for the attached thread (non-main > arena). You can't have a request that crosses heap boundaries, so if > you set M_MMAP_THRESHOLD larger than the heap size it is impossible to > service the request except through mmap. > > All of this requires a custom glibc. > > I *had* some patches to set HEAP_MAX_SIZE for testing, but it can only > be set at startup via a tunable because once set all allocations have > to use it for masking to compute chunk->heap mapping. > > I never had a reason to change it... but this example is such a reason. > > I don't think it's a terrible idea to allow a tunable for HEAP_MAX_SIZE > or DEFAULT_MMAP_THRESHOLD_MAX, but can only be set early and not late, > which is fine. > > > I expect each heap will only allocate two pages via page faults, but > > will reserve HEAP_MAX_SIZE bytes of address space. > > Agreed. > > > If that's a problem, you could also make these changes, but set the > > maximum arena count to 1, then only the main arena will be used, which > > is sbrk-based. The main arena doesn't need the coarse-grained > > mappings with large power-of-two sizes. > > Right. > > -- > Cheers, > Carlos. > Thanks and Best Regards, Ritesh Sonawane ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-28 10:22 ` ritesh sonawane @ 2018-09-28 10:40 ` Florian Weimer 2018-09-28 11:07 ` ritesh sonawane 0 siblings, 1 reply; 9+ messages in thread From: Florian Weimer @ 2018-09-28 10:40 UTC (permalink / raw) To: ritesh sonawane; +Cc: carlos, libc-help * ritesh sonawane: > Apart from changing threshold value do I need to take care of bin index ? > I have tried to understand the binning of malloc(), and found that its not > necessary. The binning is for small allocations only. No changes will be needed for correctness. There are scalability issues if you have many allocations of sizes that are not tracked in bins, but that is a preexisting problem in the allocator, unrelated to changing the mmap threshold or the heap size. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-28 10:40 ` Florian Weimer @ 2018-09-28 11:07 ` ritesh sonawane 0 siblings, 0 replies; 9+ messages in thread From: ritesh sonawane @ 2018-09-28 11:07 UTC (permalink / raw) To: Florian Weimer; +Cc: carlos, libc-help Thanks a lot. On Fri, 28 Sep 2018 at 4:10 PM, Florian Weimer <fw@deneb.enyo.de> wrote: > * ritesh sonawane: > > > Apart from changing threshold value do I need to take care of bin index ? > > I have tried to understand the binning of malloc(), and found that its > not > > necessary. > > The binning is for small allocations only. No changes will be needed > for correctness. There are scalability issues if you have many > allocations of sizes that are not tracked in bins, but that is a > preexisting problem in the allocator, unrelated to changing the mmap > threshold or the heap size. > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc(). 2018-09-27 17:46 ` Florian Weimer 2018-09-27 18:16 ` Carlos O'Donell @ 2018-09-28 10:03 ` ritesh sonawane 1 sibling, 0 replies; 9+ messages in thread From: ritesh sonawane @ 2018-09-28 10:03 UTC (permalink / raw) To: fw; +Cc: libc-help On Thu, Sep 27, 2018 at 11:15 PM Florian Weimer <fw@deneb.enyo.de> wrote: > * ritesh sonawane: > > > Yes it is 64 Bit target. > > > > Fragmentation means when malloc() request is more than threshold > > value, memory is allocated using mmap(). Due to size alignment with > > page size, there is memory wastage per request. e.g. Our system is > > having 48GB memory, then total Number of malloc() requests (each > > 64MB) will be 380 and total used memory is 23 GB out of 48GB. > > I see. I agree that's a problem, and changing the malloc threshold > could be a solution. > > > The system (NEC SX-Aurora TSUBASA) on which we are currently working > > is having huge page size and each process can have 16 GB and 512 GB > > address space for 2 MB and 64 MB page size respectively. > > I'm not familiar with that system and haven't seen the glibc port, > sorry. I used to work next door to a NEC SX-6 as a student, but > that's it. > > > Also there is no particular limit on maximum heap size and Each > > thread can easily consumes 2 GB of address space and thats why we > > want to increase the M_MMAP_THRESHOLD to 1 GB. > > If you do that, you also have to increase the heap size to something > like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c). The default of 2 * > DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that > DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB). Otherwise you will have > substantial fragmentation for allocation requests between 2 GiB and > HEAP_MAX_SIZE. > > Thank you for more clarification. I have set DEFAULT_MMAP_THRESHOLD_MAX to 1GB and HEAP_MAX_SIZE kept default (2 * DEFAULT_MMAP_THRESHOLD_MAX) and DEFAULT_MMAP_THRESHOLD is also set to DEFAULT_MMAP_THRESHOLD_MAX instead of DEFAULT_MMAP_THRESHOLD_MIN, So that default value for M_MMAP_THRESHOLD will be 1GB. Because we want default value of M_MMAP_THRESHOLD to be 1GB. With this changes we got the improvement in memory utilization (less fragmentation). But having some doubt regarding 32GiB HEAP_MAX_SIZE. Because we want malloc() request above 1GB to be done by mmap() to keep minimum memory locked by arena. I expect each heap will only allocate two pages via page faults, but > will reserve HEAP_MAX_SIZE bytes of address space > Currently we are not supporting the demand paging. If that's a problem, you could also make these changes, but set the > maximum arena count to 1, then only the main arena will be used, which > is sbrk-based. The main arena doesn't need the coarse-grained > mappings with large power-of-two sizes. > Sure, I will try this also, ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-09-28 11:07 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-26 9:46 Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc() ritesh sonawane 2018-09-26 10:05 ` Florian Weimer 2018-09-27 16:55 ` ritesh sonawane 2018-09-27 17:46 ` Florian Weimer 2018-09-27 18:16 ` Carlos O'Donell 2018-09-28 10:22 ` ritesh sonawane 2018-09-28 10:40 ` Florian Weimer 2018-09-28 11:07 ` ritesh sonawane 2018-09-28 10:03 ` ritesh sonawane
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).