Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[ritesh sonawane]

Hi All,

I am working on system with page size 2MB and 64MB and use of malloc()
leads to fragmentation.
To avoid this I want to increase the size of M_MMAP_THRESHOLD to 1GB. Is
there any impact of
this on bin index calculation ? Need some help.

Thanks and Regards,
Ritesh

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[Florian Weimer]

* ritesh sonawane:

> I am working on system with page size 2MB and 64MB and use of malloc()
> leads to fragmentation.

What kind of fragmentation?

> To avoid this I want to increase the size of M_MMAP_THRESHOLD to
> 1GB. Is there any impact of this on bin index calculation ?

I assume this is for a 64-bit target.

You will also have to increase the heap size (the amount by which thread
arenas grow), otherwise you will end up with much more fragmenation for
certain allocation sizes.  This will mean that every thread consumes at
least 2 GiB of address space.

Thanks,
Florian

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[ritesh sonawane]

++libc-help

Thanks for quick reply .

Yes it is 64 Bit target.

Fragmentation means when malloc() request is more than threshold value,
memory is allocated using mmap().
Due to size alignment with page size, there is memory wastage per request.
e.g. Our system is having 48GB
memory, then  total Number of malloc() requests (each 64MB) will be 380 and
total used memory is 23 GB out
of 48GB.

The system (NEC SX-Aurora TSUBASA) on which we are currently working is
having huge page size and each
process  can have  16 GB and 512 GB address space for 2 MB and 64 MB page
size respectively.

Also there is no particular limit on maximum heap size and Each thread can
easily consumes 2 GB of address
space  and thats why we want to increase the M_MMAP_THRESHOLD to 1 GB.

Best Regards,
Ritesh

On Wed, Sep 26, 2018 at 3:35 PM Florian Weimer <fweimer@redhat.com> wrote:

> * ritesh sonawane:
>
> > I am working on system with page size 2MB and 64MB and use of malloc()
> > leads to fragmentation.
>
> What kind of fragmentation?
>
> > To avoid this I want to increase the size of M_MMAP_THRESHOLD to
> > 1GB. Is there any impact of this on bin index calculation ?
>
> I assume this is for a 64-bit target.
>
> You will also have to increase the heap size (the amount by which thread
> arenas grow), otherwise you will end up with much more fragmenation for
> certain allocation sizes.  This will mean that every thread consumes at
> least 2 GiB of address space.
>
> Thanks,
> Florian
>

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[Florian Weimer]

* ritesh sonawane:

> Yes it is 64 Bit target.
>
> Fragmentation means when malloc() request is more than threshold
> value, memory is allocated using mmap().  Due to size alignment with
> page size, there is memory wastage per request.  e.g. Our system is
> having 48GB memory, then total Number of malloc() requests (each
> 64MB) will be 380 and total used memory is 23 GB out of 48GB.

I see.  I agree that's a problem, and changing the malloc threshold
could be a solution.

> The system (NEC SX-Aurora TSUBASA) on which we are currently working
> is having huge page size and each process can have 16 GB and 512 GB
> address space for 2 MB and 64 MB page size respectively.

I'm not familiar with that system and haven't seen the glibc port,
sorry.  I used to work next door to a NEC SX-6 as a student, but
that's it.

> Also there is no particular limit on maximum heap size and Each
> thread can easily consumes 2 GB of address space and thats why we
> want to increase the M_MMAP_THRESHOLD to 1 GB.

If you do that, you also have to increase the heap size to something
like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c).  The default of 2 *
DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that
DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB).  Otherwise you will have
substantial fragmentation for allocation requests between 2 GiB and
HEAP_MAX_SIZE.

I expect each heap will only allocate two pages via page faults, but
will reserve HEAP_MAX_SIZE bytes of address space.

If that's a problem, you could also make these changes, but set the
maximum arena count to 1, then only the main arena will be used, which
is sbrk-based.  The main arena doesn't need the coarse-grained
mappings with large power-of-two sizes.

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[Carlos O'Donell]

On 9/27/18 1:45 PM, Florian Weimer wrote:
> * ritesh sonawane:
> 
>> Yes it is 64 Bit target.
>>
>> Fragmentation means when malloc() request is more than threshold
>> value, memory is allocated using mmap().  Due to size alignment with
>> page size, there is memory wastage per request.  e.g. Our system is
>> having 48GB memory, then total Number of malloc() requests (each
>> 64MB) will be 380 and total used memory is 23 GB out of 48GB.
> 
> I see.  I agree that's a problem, and changing the malloc threshold
> could be a solution.
> 
>> The system (NEC SX-Aurora TSUBASA) on which we are currently working
>> is having huge page size and each process can have 16 GB and 512 GB
>> address space for 2 MB and 64 MB page size respectively.
> 
> I'm not familiar with that system and haven't seen the glibc port,
> sorry.  I used to work next door to a NEC SX-6 as a student, but
> that's it.
> 
>> Also there is no particular limit on maximum heap size and Each
>> thread can easily consumes 2 GB of address space and thats why we
>> want to increase the M_MMAP_THRESHOLD to 1 GB.
> 
> If you do that, you also have to increase the heap size to something
> like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c).  The default of 2 *
> DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that
> DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB).  Otherwise you will have
> substantial fragmentation for allocation requests between 2 GiB and
> HEAP_MAX_SIZE.

Right.

* Change malloc/malloc.c DEFAULT_MMAP_TRESHOLD_MAX to 16GiB.
* This in turn sets HEAP_MAX_SIZE to 32GiB.
* Now you can set M_MMAP_THRESHOLD to any value 0 > x <= (32GiB - a couple of pages).

The key here is that an arena is made up of discontiguous heaps in
a chain which are logically "the heap" for the attached thread (non-main
arena). You can't have a request that crosses heap boundaries, so if
you set M_MMAP_THRESHOLD larger than the heap size it is impossible to
service the request except through mmap.

All of this requires a custom glibc.

I *had* some patches to set HEAP_MAX_SIZE for testing, but it can only
be set at startup via a tunable because once set all allocations have
to use it for masking to compute chunk->heap mapping.

I never had a reason to change it... but this example is such a reason.

I don't think it's a terrible idea to allow a tunable for HEAP_MAX_SIZE
or DEFAULT_MMAP_THRESHOLD_MAX, but can only be set early and not late,
which is fine.

> I expect each heap will only allocate two pages via page faults, but
> will reserve HEAP_MAX_SIZE bytes of address space.

Agreed.

> If that's a problem, you could also make these changes, but set the
> maximum arena count to 1, then only the main arena will be used, which
> is sbrk-based.  The main arena doesn't need the coarse-grained
> mappings with large power-of-two sizes.

Right.

-- 
Cheers,
Carlos.

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[ritesh sonawane]

On Thu, Sep 27, 2018 at 11:15 PM Florian Weimer <fw@deneb.enyo.de> wrote:

> * ritesh sonawane:
>
> > Yes it is 64 Bit target.
> >
> > Fragmentation means when malloc() request is more than threshold
> > value, memory is allocated using mmap().  Due to size alignment with
> > page size, there is memory wastage per request.  e.g. Our system is
> > having 48GB memory, then total Number of malloc() requests (each
> > 64MB) will be 380 and total used memory is 23 GB out of 48GB.
>
> I see.  I agree that's a problem, and changing the malloc threshold
> could be a solution.
>
> > The system (NEC SX-Aurora TSUBASA) on which we are currently working
> > is having huge page size and each process can have 16 GB and 512 GB
> > address space for 2 MB and 64 MB page size respectively.
>
> I'm not familiar with that system and haven't seen the glibc port,
> sorry.  I used to work next door to a NEC SX-6 as a student, but
> that's it.
>
> > Also there is no particular limit on maximum heap size and Each
> > thread can easily consumes 2 GB of address space and thats why we
> > want to increase the M_MMAP_THRESHOLD to 1 GB.
>
> If you do that, you also have to increase the heap size to something
> like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c).  The default of 2 *
> DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that
> DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB).  Otherwise you will have
> substantial fragmentation for allocation requests between 2 GiB and
> HEAP_MAX_SIZE.
>
>
Thank you for more clarification. I have set DEFAULT_MMAP_THRESHOLD_MAX  to
1GB
and HEAP_MAX_SIZE kept default (2 * DEFAULT_MMAP_THRESHOLD_MAX) and
DEFAULT_MMAP_THRESHOLD is also set to DEFAULT_MMAP_THRESHOLD_MAX instead of
DEFAULT_MMAP_THRESHOLD_MIN, So that default value for M_MMAP_THRESHOLD will
be 1GB.
Because we want default value of M_MMAP_THRESHOLD to be 1GB. With this
changes we got the
improvement in memory utilization (less fragmentation).
But having some doubt regarding 32GiB HEAP_MAX_SIZE. Because we want
malloc() request above
1GB to be done by mmap()  to keep minimum memory locked by arena.

I expect each heap will only allocate two pages via page faults, but
> will reserve HEAP_MAX_SIZE bytes of address space
>

Currently we are not supporting the demand paging.

If that's a problem, you could also make these changes, but set the
> maximum arena count to 1, then only the main arena will be used, which
> is sbrk-based.  The main arena doesn't need the coarse-grained
> mappings with large power-of-two sizes.
>

Sure, I will try this also,

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[ritesh sonawane]

On Thu, Sep 27, 2018 at 11:45 PM Carlos O'Donell <carlos@redhat.com> wrote:

> On 9/27/18 1:45 PM, Florian Weimer wrote:
> > * ritesh sonawane:
> >
> >> Yes it is 64 Bit target.
> >>
> >> Fragmentation means when malloc() request is more than threshold
> >> value, memory is allocated using mmap().  Due to size alignment with
> >> page size, there is memory wastage per request.  e.g. Our system is
> >> having 48GB memory, then total Number of malloc() requests (each
> >> 64MB) will be 380 and total used memory is 23 GB out of 48GB.
> >
> > I see.  I agree that's a problem, and changing the malloc threshold
> > could be a solution.
> >
> >> The system (NEC SX-Aurora TSUBASA) on which we are currently working
> >> is having huge page size and each process can have 16 GB and 512 GB
> >> address space for 2 MB and 64 MB page size respectively.
> >
> > I'm not familiar with that system and haven't seen the glibc port,
> > sorry.  I used to work next door to a NEC SX-6 as a student, but
> > that's it.
> >
> >> Also there is no particular limit on maximum heap size and Each
> >> thread can easily consumes 2 GB of address space and thats why we
> >> want to increase the M_MMAP_THRESHOLD to 1 GB.
> >
> > If you do that, you also have to increase the heap size to something
> > like 32 GiB (HEAP_MAX_SIZE in malloc/arena.c).  The default of 2 *
> > DEFAULT_MMAP_THRESHOLD_MAX is probably too small (assuming that
> > DEFAULT_MMAP_THRESHOLD_MAX will be 2 GiB).  Otherwise you will have
> > substantial fragmentation for allocation requests between 2 GiB and
> > HEAP_MAX_SIZE.
>
> Right.
>
> * Change malloc/malloc.c DEFAULT_MMAP_TRESHOLD_MAX to 16GiB.
> * This in turn sets HEAP_MAX_SIZE to 32GiB.
> * Now you can set M_MMAP_THRESHOLD to any value 0 > x <= (32GiB - a couple
> of pages).
>
> Thanks for suggestions,  Right now requirement of M_MMAP_THRESHOLD is Up
to 1GB only.
So I set DEFAULT_MMAP_TRESHOLD_MAX to 1GB ( for 2MB and 64MB page size) and
testing
the performance. But I will check with above limits also.

Apart from changing threshold value do I need to take care of bin index ?
I have tried to understand the binning of malloc(), and found that its not
necessary.

The key here is that an arena is made up of discontiguous heaps in
> a chain which are logically "the heap" for the attached thread (non-main
> arena). You can't have a request that crosses heap boundaries, so if
> you set M_MMAP_THRESHOLD larger than the heap size it is impossible to
> service the request except through mmap.
>
> All of this requires a custom glibc.
>
> I *had* some patches to set HEAP_MAX_SIZE for testing, but it can only
> be set at startup via a tunable because once set all allocations have
> to use it for masking to compute chunk->heap mapping.
>
> I never had a reason to change it... but this example is such a reason.
>
> I don't think it's a terrible idea to allow a tunable for HEAP_MAX_SIZE
> or DEFAULT_MMAP_THRESHOLD_MAX, but can only be set early and not late,
> which is fine.
>
> > I expect each heap will only allocate two pages via page faults, but
> > will reserve HEAP_MAX_SIZE bytes of address space.
>
> Agreed.
>
> > If that's a problem, you could also make these changes, but set the
> > maximum arena count to 1, then only the main arena will be used, which
> > is sbrk-based.  The main arena doesn't need the coarse-grained
> > mappings with large power-of-two sizes.
>
> Right.
>
> --
> Cheers,
> Carlos.
>

Thanks and Best Regards,
Ritesh Sonawane

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[Florian Weimer]

* ritesh sonawane:

> Apart from changing threshold value do I need to take care of bin index ?
> I have tried to understand the binning of malloc(), and found that its not
> necessary.

The binning is for small allocations only.  No changes will be needed
for correctness.  There are scalability issues if you have many
allocations of sizes that are not tracked in bins, but that is a
preexisting problem in the allocator, unrelated to changing the mmap
threshold or the heap size.

Re: Impact of Increasing M_MMAP_THRESHOLD to 1GB in malloc().

[ritesh sonawane]

Thanks a lot.

On Fri, 28 Sep 2018 at 4:10 PM, Florian Weimer <fw@deneb.enyo.de> wrote:

> * ritesh sonawane:
>
> > Apart from changing threshold value do I need to take care of bin index ?
> > I have tried to understand the binning of malloc(), and found that its
> not
> > necessary.
>
> The binning is for small allocations only.  No changes will be needed
> for correctness.  There are scalability issues if you have many
> allocations of sizes that are not tracked in bins, but that is a
> preexisting problem in the allocator, unrelated to changing the mmap
> threshold or the heap size.
>