[PATCH 0/2] Malloc manual cleanups

[PATCH 0/2] Malloc manual cleanups

[Siddhesh Poyarekar]

Here's take 2 of the cleanup of descriptions in the manual description in
malloc.  As suggested in previous review, I have made the following changes:

 - Moved up "Efficiency considerations for malloc" into a new subsection called
   "The GNU allocator" that gives a high level description of the allocator
   design and introduces terms like arenas and chunks.  I did not go into too
   much detail here and have instead pointed interested readers to the wiki page
   DJ wrote.

 - Added default values of all of the mallopt parameters

Siddhesh Poyarekar (2):
  Add note on MALLOC_MMAP_* environment variables
  Document the M_ARENA_* mallopt parameters

 malloc/malloc.c    |   1 -
 manual/memory.texi | 153 ++++++++++++++++++++++++++++++++---------------------
 2 files changed, 93 insertions(+), 61 deletions(-)

-- 
2.7.4

[PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

The M_ARENA_* mallopt parameters are in wide use in production to
control the number of arenas that a long lived process creates and
hence there is no point in stating that this interface is non-public.
Document this interface and remove the obsolete comment.

	* manual/memory.texi (M_ARENA_TEST): Add documentation.
	(M_ARENA_MAX): Likewise.
	* malloc/malloc.c: Remove obsolete comment.
---
 malloc/malloc.c    |   1 -
 manual/memory.texi | 122 +++++++++++++++++++++++++++--------------------------
 2 files changed, 62 insertions(+), 61 deletions(-)

diff --git a/malloc/malloc.c b/malloc/malloc.c
index ef04360..a849901 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -1718,7 +1718,6 @@ static struct malloc_par mp_ =
 };
 
 
-/*  Non public mallopt parameters.  */
 #define M_ARENA_TEST -7
 #define M_ARENA_MAX  -8
 
diff --git a/manual/memory.texi b/manual/memory.texi
index 6f33455..198a933 100644
--- a/manual/memory.texi
+++ b/manual/memory.texi
@@ -162,6 +162,8 @@ special to @theglibc{} and GNU Compiler.
 
 @menu
 * Memory Allocation and C::     How to get different kinds of allocation in C.
+* The GNU allocator::		An overview of the GNU @code{malloc}
+				implementation.
 * Unconstrained Allocation::    The @code{malloc} facility allows fully general
 		 		 dynamic allocation.
 * Allocation Debugging::        Finding memory leaks and not freed memory.
@@ -258,6 +260,43 @@ address of the space.  Then you can use the operators @samp{*} and
 @}
 @end smallexample
 
+@node The GNU allocator
+@subsection The GNU allocator
+@cindex gnu allocator
+
+The @code{malloc} implementation in @theglibc{} is derived from ptmalloc
+(pthreads malloc), which in turn is derived from dlmalloc (Doug Lea malloc).
+This malloc may allocate memory in two different ways depending on their size
+and certain parameters that may be controlled by users. The most common way is
+to allocate portions of memory (called chunks) from a large contiguous area of
+memory and manage these areas to optimize their use and reduce wastage in the
+form of unusable chunks. Traditionally the system heap was set up to be the one
+large memory area but @theglibc{} @code{malloc} implementation maintains
+multiple such areas to optimize their use in multi-threaded applications.  Each
+such area is internally referred to as an @code{arena}.
+
+As opposed to other versions, the @code{malloc} in @theglibc{} does not round
+up chunk sizes to powers of two, neither for large nor for small sizes.
+Neighboring chunks can be coalesced on a @code{free} no matter what their size
+is.  This makes the implementation suitable for all kinds of allocation
+patterns without generally incurring high memory waste through fragmentation.
+The presence of multiple @code{arenas} allows multiple threads to allocate
+memory simultaneously in their own separate arenas, thus improving performance.
+
+The other way of memory allocation is for very large blocks, i.e. much larger
+than a page. These requests are allocated with @code{mmap} (anonymous or via
+@code{/dev/zero}). This has the great advantage that these chunks are returned
+to the system immediately when they are freed.  Therefore, it cannot happen
+that a large chunk becomes ``locked'' in between smaller ones and even after
+calling @code{free} wastes memory.  The size threshold for @code{mmap} to be
+used is dynamic and gets adjusted according to allocation patterns of the
+program.  This can also be statically adjusted with @code{mallopt}.  The use of
+@code{mmap} can also be disabled completely.
+
+A more detailed technical description of the GNU allocator is maintained in
+@theglibc{} wiki. See
+@uref{https://sourceware.org/glibc/wiki/MallocInternals}.
+
 @node Unconstrained Allocation
 @subsection Unconstrained Allocation
 @cindex unconstrained memory allocation
@@ -278,8 +317,6 @@ any time (or never).
 				 bigger or smaller.
 * Allocating Cleared Space::    Use @code{calloc} to allocate a
 				 block and clear it.
-* Efficiency and Malloc::       Efficiency considerations in use of
-				 these functions.
 * Aligned Memory Blocks::       Allocating specially aligned memory.
 * Malloc Tunable Parameters::   Use @code{mallopt} to adjust allocation
                                  parameters.
@@ -867,59 +904,6 @@ But in general, it is not guaranteed that @code{calloc} calls
 @code{malloc}/@code{realloc}/@code{free} outside the C library, it
 should always define @code{calloc}, too.
 
-@node Efficiency and Malloc
-@subsubsection Efficiency Considerations for @code{malloc}
-@cindex efficiency and @code{malloc}
-
-
-
-
-@ignore
-
-@c No longer true, see below instead.
-To make the best use of @code{malloc}, it helps to know that the GNU
-version of @code{malloc} always dispenses small amounts of memory in
-blocks whose sizes are powers of two.  It keeps separate pools for each
-power of two.  This holds for sizes up to a page size.  Therefore, if
-you are free to choose the size of a small block in order to make
-@code{malloc} more efficient, make it a power of two.
-@c !!! xref getpagesize
-
-Once a page is split up for a particular block size, it can't be reused
-for another size unless all the blocks in it are freed.  In many
-programs, this is unlikely to happen.  Thus, you can sometimes make a
-program use memory more efficiently by using blocks of the same size for
-many different purposes.
-
-When you ask for memory blocks of a page or larger, @code{malloc} uses a
-different strategy; it rounds the size up to a multiple of a page, and
-it can coalesce and split blocks as needed.
-
-The reason for the two strategies is that it is important to allocate
-and free small blocks as fast as possible, but speed is less important
-for a large block since the program normally spends a fair amount of
-time using it.  Also, large blocks are normally fewer in number.
-Therefore, for large blocks, it makes sense to use a method which takes
-more time to minimize the wasted space.
-
-@end ignore
-
-As opposed to other versions, the @code{malloc} in @theglibc{}
-does not round up block sizes to powers of two, neither for large nor
-for small sizes.  Neighboring chunks can be coalesced on a @code{free}
-no matter what their size is.  This makes the implementation suitable
-for all kinds of allocation patterns without generally incurring high
-memory waste through fragmentation.
-
-Very large blocks (much larger than a page) are allocated with
-@code{mmap} (anonymous or via @code{/dev/zero}) by this implementation.
-This has the great advantage that these chunks are returned to the
-system immediately when they are freed.  Therefore, it cannot happen
-that a large chunk becomes ``locked'' in between smaller ones and even
-after calling @code{free} wastes memory.  The size threshold for
-@code{mmap} to be used can be adjusted with @code{mallopt}.  The use of
-@code{mmap} can also be disabled completely.
-
 @node Aligned Memory Blocks
 @subsubsection Allocating Aligned Memory Blocks
 
@@ -1105,10 +1089,6 @@ parameter to be set, and @var{value} the new value to be set.  Possible
 choices for @var{param}, as defined in @file{malloc.h}, are:
 
 @table @code
-@comment TODO: @item M_ARENA_MAX
-@comment       - Document ARENA_MAX env var.
-@comment TODO: @item M_ARENA_TEST
-@comment       - Document ARENA_TEST env var.
 @comment TODO: @item M_CHECK_ACTION
 @item M_MMAP_MAX
 The maximum number of chunks to allocate with @code{mmap}.  Setting this
@@ -1169,6 +1149,28 @@ value is set statically to the provided input.
 
 This parameter can also be set for the process at startup by setting the
 environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
+
+@item M_ARENA_TEST
+This parameter specifies the number of arenas that can be created before the
+test on the limit to the number of arenas is conducted. The value is ignored if
+@code{M_ARENA_MAX} is set.
+
+The default value of this parameter is 2 on 32-bit systems and 8 on 64-bit
+systems.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_ARENA_TEST} to the desired value.
+@item M_ARENA_MAX
+This parameter sets the number of arenas to use regardless of the number of
+cores in the system.
+
+The default value of this tunable is @code{0}, meaning that the limit on the
+number of arenas is determined by the number of CPU cores online. For 32-bit
+systems the limit is twice the number of cores online and on 64-bit systems, it
+is eight times the number of cores online.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_ARENA_MAX} to the desired value.
 @end table
 
 @end deftypefun
@@ -1511,7 +1513,7 @@ This is the total size of memory allocated with @code{sbrk} by
 This is the number of chunks not in use.  (The memory allocator
 internally gets chunks of memory from the operating system, and then
 carves them up to satisfy individual @code{malloc} requests; see
-@ref{Efficiency and Malloc}.)
+@ref{The GNU allocator}.)
 
 @item int smblks
 This field is unused.
-- 
2.7.4

[PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Siddhesh Poyarekar]

The mallopt parameters manual does not mention the environment
variables that can be used to set these parameters at program startup.
Mention those environment variables for completeness.

	* manual/memory.texi: Add environment variable alternatives to
	setting mallopt parameters.
---
 manual/memory.texi | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/manual/memory.texi b/manual/memory.texi
index 5383105..6f33455 100644
--- a/manual/memory.texi
+++ b/manual/memory.texi
@@ -1113,12 +1113,25 @@ choices for @var{param}, as defined in @file{malloc.h}, are:
 @item M_MMAP_MAX
 The maximum number of chunks to allocate with @code{mmap}.  Setting this
 to zero disables all use of @code{mmap}.
+
+The default value of this parameter is @code{65536}.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
 @item M_MMAP_THRESHOLD
 All chunks larger than this value are allocated outside the normal
 heap, using the @code{mmap} system call.  This way it is guaranteed
 that the memory for these chunks can be returned to the system on
 @code{free}.  Note that requests smaller than this threshold might still
 be allocated via @code{mmap}.
+
+If this parameter is not set, the default value is set as 128 KB and the
+threshold is adjusted dynamically to suit the allocation patterns of the
+program. If the parameter is set, the dynamic adjustment is disabled and the
+value is set statically to the input value.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value.
 @comment TODO: @item M_MXFAST
 @item M_PERTURB
 If non-zero, memory blocks are filled with values depending on some
@@ -1128,16 +1141,34 @@ use of uninitialized or freed heap memory.  Note that this option does not
 guarantee that the freed block will have any specific values.  It only
 guarantees that the content the block had before it was freed will be
 overwritten.
+
+The default value of this parameter is @code{0}.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value.
 @item M_TOP_PAD
 This parameter determines the amount of extra memory to obtain from the
 system when a call to @code{sbrk} is required.  It also specifies the
 number of bytes to retain when shrinking the heap by calling @code{sbrk}
 with a negative argument.  This provides the necessary hysteresis in
 heap size such that excessive amounts of system calls can be avoided.
+
+The default value of this parameter is @code{0}.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_TOP_PAD_} to the desired value.
 @item M_TRIM_THRESHOLD
 This is the minimum size (in bytes) of the top-most, releasable chunk
 that will cause @code{sbrk} to be called with a negative argument in
 order to return memory to the system.
+
+If this parameter is not set, the default value is set as 128 KB and the
+threshold is adjusted dynamically to suit the allocation patterns of the
+program. If the parameter is set, the dynamic adjustment is disabled and the
+value is set statically to the provided input.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
 @end table
 
 @end deftypefun
-- 
2.7.4

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Rical Jasan]

Hi Siddhesh,

Most of this is formatting or cleanup/improvement of the pre-existing
chunks that were moved, but the comment on M_ARENA_MAX regards content.

On 10/24/2016 07:07 AM, Siddhesh Poyarekar wrote:
> The M_ARENA_* mallopt parameters are in wide use in production to
> control the number of arenas that a long lived process creates and
> hence there is no point in stating that this interface is non-public.
> Document this interface and remove the obsolete comment.
> 
> 	* manual/memory.texi (M_ARENA_TEST): Add documentation.
> 	(M_ARENA_MAX): Likewise.
> 	* malloc/malloc.c: Remove obsolete comment.
> ---
>  malloc/malloc.c    |   1 -
>  manual/memory.texi | 122 +++++++++++++++++++++++++++--------------------------
>  2 files changed, 62 insertions(+), 61 deletions(-)
> 
> diff --git a/malloc/malloc.c b/malloc/malloc.c
> index ef04360..a849901 100644
> --- a/malloc/malloc.c
> +++ b/malloc/malloc.c
> @@ -1718,7 +1718,6 @@ static struct malloc_par mp_ =
>  };
>  
>  
> -/*  Non public mallopt parameters.  */
>  #define M_ARENA_TEST -7
>  #define M_ARENA_MAX  -8
>  
> diff --git a/manual/memory.texi b/manual/memory.texi
> index 6f33455..198a933 100644
> --- a/manual/memory.texi
> +++ b/manual/memory.texi
> @@ -162,6 +162,8 @@ special to @theglibc{} and GNU Compiler.
>  
>  @menu
>  * Memory Allocation and C::     How to get different kinds of allocation in C.
> +* The GNU allocator::		An overview of the GNU @code{malloc}

I think "Allocator" should be capitalized.  The manual is inconsistent
on @subsection capitalization, but the majority of it is (capitalized).
Also, all the other info menu entries visible in the context here are
fully capitalized.

> +				implementation.
>  * Unconstrained Allocation::    The @code{malloc} facility allows fully general
>  		 		 dynamic allocation.
>  * Allocation Debugging::        Finding memory leaks and not freed memory.
> @@ -258,6 +260,43 @@ address of the space.  Then you can use the operators @samp{*} and
>  @}
>  @end smallexample
>  
> +@node The GNU allocator
> +@subsection The GNU allocator

Allocator

> +@cindex gnu allocator
> +
> +The @code{malloc} implementation in @theglibc{} is derived from ptmalloc
> +(pthreads malloc), which in turn is derived from dlmalloc (Doug Lea malloc).
> +This malloc may allocate memory in two different ways depending on their size
> +and certain parameters that may be controlled by users. The most common way is
> +to allocate portions of memory (called chunks) from a large contiguous area of
> +memory and manage these areas to optimize their use and reduce wastage in the
> +form of unusable chunks. Traditionally the system heap was set up to be the one
> +large memory area but @theglibc{} @code{malloc} implementation maintains

This should be "the @glibcadj{} @code{malloc}".

> +multiple such areas to optimize their use in multi-threaded applications.  Each
> +such area is internally referred to as an @code{arena}.

@dfn{arena}

Unless this is a function name (or literal string one would be using in
code, for example), this is simply a term we use to describe the
concept.  On first use, where we define the term, it should have @dfn{},
and otherwise it doesn't need to be stylized at all, such as...

> +
> +As opposed to other versions, the @code{malloc} in @theglibc{} does not round
> +up chunk sizes to powers of two, neither for large nor for small sizes.
> +Neighboring chunks can be coalesced on a @code{free} no matter what their size
> +is.  This makes the implementation suitable for all kinds of allocation
> +patterns without generally incurring high memory waste through fragmentation.
> +The presence of multiple @code{arenas} allows multiple threads to allocate

here (no @code{}).

> +memory simultaneously in their own separate arenas, thus improving performance.
> +
> +The other way of memory allocation is for very large blocks, i.e. much larger
> +than a page. These requests are allocated with @code{mmap} (anonymous or via
> +@code{/dev/zero}). This has the great advantage that these chunks are returned

Should be @file{}.

A reference to mmap would be good.  Something like, "... (anonymous or
via @file{/dev/zero}; @pxref{Memory-mapped I/O})."

> +to the system immediately when they are freed.  Therefore, it cannot happen
> +that a large chunk becomes ``locked'' in between smaller ones and even after
> +calling @code{free} wastes memory.  The size threshold for @code{mmap} to be
> +used is dynamic and gets adjusted according to allocation patterns of the
> +program.  This can also be statically adjusted with @code{mallopt}.  The use of

A reference to mallopt would be good here.

> +@code{mmap} can also be disabled completely.

Should briefly say how and/or give a reference.  I believe mallopt
applies to both, so maybe something like, "@code{mallopt} can be used to
statically adjust the threshold using @code{M_MMAP_THRESHOLD}, and the
use of @code{mmap} can be disabled completely with @code{M_MMAP_MAX};
@pxref{Malloc Tunable Parameters}."

> +
> +A more detailed technical description of the GNU allocator is maintained in
> +@theglibc{} wiki. See

the @glibcadj{}

> +@uref{https://sourceware.org/glibc/wiki/MallocInternals}.
> +
>  @node Unconstrained Allocation
>  @subsection Unconstrained Allocation
>  @cindex unconstrained memory allocation
> @@ -278,8 +317,6 @@ any time (or never).
>  				 bigger or smaller.
>  * Allocating Cleared Space::    Use @code{calloc} to allocate a
>  				 block and clear it.
> -* Efficiency and Malloc::       Efficiency considerations in use of
> -				 these functions.
>  * Aligned Memory Blocks::       Allocating specially aligned memory.
>  * Malloc Tunable Parameters::   Use @code{mallopt} to adjust allocation
>                                   parameters.
> @@ -867,59 +904,6 @@ But in general, it is not guaranteed that @code{calloc} calls
>  @code{malloc}/@code{realloc}/@code{free} outside the C library, it
>  should always define @code{calloc}, too.
>  
> -@node Efficiency and Malloc
> -@subsubsection Efficiency Considerations for @code{malloc}
> -@cindex efficiency and @code{malloc}
> -
> -
> -
> -
> -@ignore
> -
> -@c No longer true, see below instead.
> -To make the best use of @code{malloc}, it helps to know that the GNU
> -version of @code{malloc} always dispenses small amounts of memory in
> -blocks whose sizes are powers of two.  It keeps separate pools for each
> -power of two.  This holds for sizes up to a page size.  Therefore, if
> -you are free to choose the size of a small block in order to make
> -@code{malloc} more efficient, make it a power of two.
> -@c !!! xref getpagesize
> -
> -Once a page is split up for a particular block size, it can't be reused
> -for another size unless all the blocks in it are freed.  In many
> -programs, this is unlikely to happen.  Thus, you can sometimes make a
> -program use memory more efficiently by using blocks of the same size for
> -many different purposes.
> -
> -When you ask for memory blocks of a page or larger, @code{malloc} uses a
> -different strategy; it rounds the size up to a multiple of a page, and
> -it can coalesce and split blocks as needed.
> -
> -The reason for the two strategies is that it is important to allocate
> -and free small blocks as fast as possible, but speed is less important
> -for a large block since the program normally spends a fair amount of
> -time using it.  Also, large blocks are normally fewer in number.
> -Therefore, for large blocks, it makes sense to use a method which takes
> -more time to minimize the wasted space.
> -
> -@end ignore
> -
> -As opposed to other versions, the @code{malloc} in @theglibc{}
> -does not round up block sizes to powers of two, neither for large nor
> -for small sizes.  Neighboring chunks can be coalesced on a @code{free}
> -no matter what their size is.  This makes the implementation suitable
> -for all kinds of allocation patterns without generally incurring high
> -memory waste through fragmentation.
> -
> -Very large blocks (much larger than a page) are allocated with
> -@code{mmap} (anonymous or via @code{/dev/zero}) by this implementation.
> -This has the great advantage that these chunks are returned to the
> -system immediately when they are freed.  Therefore, it cannot happen
> -that a large chunk becomes ``locked'' in between smaller ones and even
> -after calling @code{free} wastes memory.  The size threshold for
> -@code{mmap} to be used can be adjusted with @code{mallopt}.  The use of
> -@code{mmap} can also be disabled completely.
> -
>  @node Aligned Memory Blocks
>  @subsubsection Allocating Aligned Memory Blocks
>  
> @@ -1105,10 +1089,6 @@ parameter to be set, and @var{value} the new value to be set.  Possible
>  choices for @var{param}, as defined in @file{malloc.h}, are:
>  
>  @table @code
> -@comment TODO: @item M_ARENA_MAX
> -@comment       - Document ARENA_MAX env var.
> -@comment TODO: @item M_ARENA_TEST
> -@comment       - Document ARENA_TEST env var.
>  @comment TODO: @item M_CHECK_ACTION
>  @item M_MMAP_MAX
>  The maximum number of chunks to allocate with @code{mmap}.  Setting this
> @@ -1169,6 +1149,28 @@ value is set statically to the provided input.
>  
>  This parameter can also be set for the process at startup by setting the
>  environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
> +
> +@item M_ARENA_TEST
> +This parameter specifies the number of arenas that can be created before the
> +test on the limit to the number of arenas is conducted. The value is ignored if
> +@code{M_ARENA_MAX} is set.
> +
> +The default value of this parameter is 2 on 32-bit systems and 8 on 64-bit
> +systems.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_ARENA_TEST} to the desired value.
> +@item M_ARENA_MAX
> +This parameter sets the number of arenas to use regardless of the number of
> +cores in the system.
> +
> +The default value of this tunable is @code{0}, meaning that the limit on the
> +number of arenas is determined by the number of CPU cores online. For 32-bit
> +systems the limit is twice the number of cores online and on 64-bit systems, it
> +is eight times the number of cores online.

Even though I had followed the thread, I immediately jumped to the same
confusion with M_ARENA_TEST that was resolved in [1].  Explicitly
stating here that 2 and 8 are not derived from M_ARENA_TEST defaults
might save a lot of general confusion down the road.

> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_ARENA_MAX} to the desired value.
>  @end table
>  
>  @end deftypefun
> @@ -1511,7 +1513,7 @@ This is the total size of memory allocated with @code{sbrk} by
>  This is the number of chunks not in use.  (The memory allocator
>  internally gets chunks of memory from the operating system, and then
>  carves them up to satisfy individual @code{malloc} requests; see
> -@ref{Efficiency and Malloc}.)
> +@ref{The GNU allocator}.)

Allocator

Also, making this an @pxref{} now would be nice, since this renders
wrong in info as-is.

>  
>  @item int smblks
>  This field is unused.
> 

Rical

[1] https://sourceware.org/ml/libc-alpha/2016-10/msg00310.html

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[DJ Delorie]

Siddhesh Poyarekar <siddhesh@sourceware.org> writes:
> -/*  Non public mallopt parameters.  */
>  #define M_ARENA_TEST -7
>  #define M_ARENA_MAX  -8

Hmmm... if these are now public, do they need to be moved elsewhere?
Like, malloc.h ?

> +The presence of multiple @code{arenas} allows multiple threads to allocate
> +memory simultaneously in their own separate arenas, thus improving performance.

Not quite true - there isn't one arena per thread, there are N arenas
per M threads.  Probably better to say "... simultaneously in separate
arenas ..."

> +environment variable @code{MALLOC_ARENA_TEST} to the desired value.
> +@item M_ARENA_MAX

Missing blank line.

Otherwise, the content looks OK to me from a technical point of view.

Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Carlos O'Donell]

On 10/24/2016 10:07 AM, Siddhesh Poyarekar wrote:
> The mallopt parameters manual does not mention the environment
> variables that can be used to set these parameters at program startup.
> Mention those environment variables for completeness.
> 
> 	* manual/memory.texi: Add environment variable alternatives to
> 	setting mallopt parameters.

Two nits. Otherwise OK.

> ---
>  manual/memory.texi | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/manual/memory.texi b/manual/memory.texi
> index 5383105..6f33455 100644
> --- a/manual/memory.texi
> +++ b/manual/memory.texi
> @@ -1113,12 +1113,25 @@ choices for @var{param}, as defined in @file{malloc.h}, are:
>  @item M_MMAP_MAX
>  The maximum number of chunks to allocate with @code{mmap}.  Setting this
>  to zero disables all use of @code{mmap}.
> +
> +The default value of this parameter is @code{65536}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
>  @item M_MMAP_THRESHOLD
>  All chunks larger than this value are allocated outside the normal
>  heap, using the @code{mmap} system call.  This way it is guaranteed
>  that the memory for these chunks can be returned to the system on
>  @code{free}.  Note that requests smaller than this threshold might still
>  be allocated via @code{mmap}.
> +
> +If this parameter is not set, the default value is set as 128 KB and the

Should this be 128KiB?

> +threshold is adjusted dynamically to suit the allocation patterns of the
> +program. If the parameter is set, the dynamic adjustment is disabled and the
> +value is set statically to the input value.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value.
>  @comment TODO: @item M_MXFAST
>  @item M_PERTURB
>  If non-zero, memory blocks are filled with values depending on some
> @@ -1128,16 +1141,34 @@ use of uninitialized or freed heap memory.  Note that this option does not
>  guarantee that the freed block will have any specific values.  It only
>  guarantees that the content the block had before it was freed will be
>  overwritten.
> +
> +The default value of this parameter is @code{0}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value.
>  @item M_TOP_PAD
>  This parameter determines the amount of extra memory to obtain from the
>  system when a call to @code{sbrk} is required.  It also specifies the
>  number of bytes to retain when shrinking the heap by calling @code{sbrk}
>  with a negative argument.  This provides the necessary hysteresis in
>  heap size such that excessive amounts of system calls can be avoided.
> +
> +The default value of this parameter is @code{0}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_TOP_PAD_} to the desired value.
>  @item M_TRIM_THRESHOLD
>  This is the minimum size (in bytes) of the top-most, releasable chunk
>  that will cause @code{sbrk} to be called with a negative argument in
>  order to return memory to the system.
> +
> +If this parameter is not set, the default value is set as 128 KB and the

Likewise.

> +threshold is adjusted dynamically to suit the allocation patterns of the
> +program. If the parameter is set, the dynamic adjustment is disabled and the
> +value is set statically to the provided input.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
>  @end table
>  
>  @end deftypefun
> 


-- 
Cheers,
Carlos.

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

On Wednesday 26 October 2016 03:25 AM, DJ Delorie wrote:
> Siddhesh Poyarekar <siddhesh@sourceware.org> writes:
>> -/*  Non public mallopt parameters.  */
>>  #define M_ARENA_TEST -7
>>  #define M_ARENA_MAX  -8
> 
> Hmmm... if these are now public, do they need to be moved elsewhere?
> Like, malloc.h ?

They're already there in malloc.h.  I'll push a follow-up patch if they
can be removed.

>> +The presence of multiple @code{arenas} allows multiple threads to allocate
>> +memory simultaneously in their own separate arenas, thus improving performance.
> 
> Not quite true - there isn't one arena per thread, there are N arenas
> per M threads.  Probably better to say "... simultaneously in separate
> arenas ..."

OK.

>> +environment variable @code{MALLOC_ARENA_TEST} to the desired value.
>> +@item M_ARENA_MAX
> 
> Missing blank line.
> 
> Otherwise, the content looks OK to me from a technical point of view.

Thanks, I'll fix up formatting comments from Rical and push this.

Siddhesh

Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Michael Kerrisk]

I've nothing to add beyond Carlos's comments. I looked this over and
it looks good to me.

Cheers,

Michael


On Wed, Oct 26, 2016 at 6:15 AM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/24/2016 10:07 AM, Siddhesh Poyarekar wrote:
>> The mallopt parameters manual does not mention the environment
>> variables that can be used to set these parameters at program startup.
>> Mention those environment variables for completeness.
>>
>>       * manual/memory.texi: Add environment variable alternatives to
>>       setting mallopt parameters.
>
> Two nits. Otherwise OK.
>
>> ---
>>  manual/memory.texi | 31 +++++++++++++++++++++++++++++++
>>  1 file changed, 31 insertions(+)
>>
>> diff --git a/manual/memory.texi b/manual/memory.texi
>> index 5383105..6f33455 100644
>> --- a/manual/memory.texi
>> +++ b/manual/memory.texi
>> @@ -1113,12 +1113,25 @@ choices for @var{param}, as defined in @file{malloc.h}, are:
>>  @item M_MMAP_MAX
>>  The maximum number of chunks to allocate with @code{mmap}.  Setting this
>>  to zero disables all use of @code{mmap}.
>> +
>> +The default value of this parameter is @code{65536}.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
>>  @item M_MMAP_THRESHOLD
>>  All chunks larger than this value are allocated outside the normal
>>  heap, using the @code{mmap} system call.  This way it is guaranteed
>>  that the memory for these chunks can be returned to the system on
>>  @code{free}.  Note that requests smaller than this threshold might still
>>  be allocated via @code{mmap}.
>> +
>> +If this parameter is not set, the default value is set as 128 KB and the
>
> Should this be 128KiB?
>
>> +threshold is adjusted dynamically to suit the allocation patterns of the
>> +program. If the parameter is set, the dynamic adjustment is disabled and the
>> +value is set statically to the input value.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value.
>>  @comment TODO: @item M_MXFAST
>>  @item M_PERTURB
>>  If non-zero, memory blocks are filled with values depending on some
>> @@ -1128,16 +1141,34 @@ use of uninitialized or freed heap memory.  Note that this option does not
>>  guarantee that the freed block will have any specific values.  It only
>>  guarantees that the content the block had before it was freed will be
>>  overwritten.
>> +
>> +The default value of this parameter is @code{0}.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value.
>>  @item M_TOP_PAD
>>  This parameter determines the amount of extra memory to obtain from the
>>  system when a call to @code{sbrk} is required.  It also specifies the
>>  number of bytes to retain when shrinking the heap by calling @code{sbrk}
>>  with a negative argument.  This provides the necessary hysteresis in
>>  heap size such that excessive amounts of system calls can be avoided.
>> +
>> +The default value of this parameter is @code{0}.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_TOP_PAD_} to the desired value.
>>  @item M_TRIM_THRESHOLD
>>  This is the minimum size (in bytes) of the top-most, releasable chunk
>>  that will cause @code{sbrk} to be called with a negative argument in
>>  order to return memory to the system.
>> +
>> +If this parameter is not set, the default value is set as 128 KB and the
>
> Likewise.
>
>> +threshold is adjusted dynamically to suit the allocation patterns of the
>> +program. If the parameter is set, the dynamic adjustment is disabled and the
>> +value is set statically to the provided input.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
>>  @end table
>>
>>  @end deftypefun
>>
>
>
> --
> Cheers,
> Carlos.

Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Siddhesh Poyarekar]

On Wednesday 26 October 2016 09:45 AM, Carlos O'Donell wrote:
> On 10/24/2016 10:07 AM, Siddhesh Poyarekar wrote:
>> The mallopt parameters manual does not mention the environment
>> variables that can be used to set these parameters at program startup.
>> Mention those environment variables for completeness.
>>
>> 	* manual/memory.texi: Add environment variable alternatives to
>> 	setting mallopt parameters.
> 
> Two nits. Otherwise OK.

Thanks, pushed.

Siddhesh

Re: Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Rical Jasan]

On 10/25/2016 09:15 PM, Carlos O'Donell wrote:
>> > +If this parameter is not set, the default value is set as 128 KB and the
> Should this be 128KiB?

I don't think binary computers count with 10 fingers.  At any rate, we
don't have any ibi's AFAICS:

$ grep -Pi '\d[kmgt]i?b\b' manual/*.texi

So maybe "128kB"?  ;)

Rical

Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Siddhesh Poyarekar]

On Wednesday 26 October 2016 04:56 PM, Rical Jasan wrote:
> On 10/25/2016 09:15 PM, Carlos O'Donell wrote:
>>>> +If this parameter is not set, the default value is set as 128 KB and the
>> Should this be 128KiB?
> 
> I don't think binary computers count with 10 fingers.  At any rate, we
> don't have any ibi's AFAICS:
> 
> $ grep -Pi '\d[kmgt]i?b\b' manual/*.texi

It's the other way around AFAIK, i.e. KiB == 1024 and KB == 1000.  Looks
like there are references to kB in code comments, so I'm fine with
changing it to that, or if you want to be absolutely clear, I could just
make it 131072 bytes ;)

Siddhesh

Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables

[Rical Jasan]

On 10/26/2016 04:30 AM, Siddhesh Poyarekar wrote:
> On Wednesday 26 October 2016 04:56 PM, Rical Jasan wrote:
>> On 10/25/2016 09:15 PM, Carlos O'Donell wrote:
>>>>> +If this parameter is not set, the default value is set as 128 KB and the
>>> Should this be 128KiB?
>>
>> I don't think binary computers count with 10 fingers.  At any rate, we
>> don't have any ibi's AFAICS:
>>
>> $ grep -Pi '\d[kmgt]i?b\b' manual/*.texi
> 
> It's the other way around AFAIK, i.e. KiB == 1024 and KB == 1000.  Looks
> like there are references to kB in code comments, so I'm fine with
> changing it to that, or if you want to be absolutely clear, I could just
> make it 131072 bytes ;)

As unambiguous as possible is best, so if you could fix the 2147483648-
and 4294967296-byte references too, that'd be great.  lol

Rical

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

On Wednesday 19 October 2016 12:23 PM, Michael Kerrisk (man-pages) wrote:
> I agree. But it's also not what I said ;-). I said "determined according
> to the value of..."

Ahh, I wouldn't have caught that if you hadn't pointed it out.

> Your details following are a little more precise. But I believe Carlos,
> when he wrote the text about how the limit was calculated, deliberately
> chose to be more vague. I'm okay with including this level of detail 
> in the man page, but, if I recall correctly, Carlos wanted to avoid it.

I'll let Carlos take the call on that.  He asked for detail in the glibc
manual, so I'll propose that level and detail for the manual and see
what he thinks.

Siddhesh

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Michael Kerrisk (man-pages)]

Hi Siddhesh,

On 10/18/2016 06:46 PM, Siddhesh Poyarekar wrote:
> On Tuesday 18 October 2016 09:33 PM, Michael Kerrisk (man-pages) wrote:
>>>> I don't think you're correct here. 'arena_max' is a field in a static
>>>> structure that is not otherwise initialized, AFAICT. So, it has the
>>>> value zero. (Some dirty hacking with a program that uses
>>>> malloc_get_state() and inspects the internal data structure seems
>>>> to confirm this.)
>>>>
>>>> And then in malloc/arena.c we have
>>>>
>>>>           if (mp_.arena_max != 0)
>>>>             narenas_limit = mp_.arena_max;
>>>>           else if (narenas > mp_.arena_test)
>>>>             {
>>>>               int n = __get_nprocs ();
>>>>               
>>>>               if (n >= 1)
>>>>                 narenas_limit = NARENAS_FROM_NCORES (n);
>>>>               else
>>>>                 /* We have no information about the system.  Assume two
>>>>                    cores.  */ 
>>>>                 narenas_limit = NARENAS_FROM_NCORES (2);
>>>>
>>>> So, I believe my original statement about M_ARENA_MAX is correct.
>>>> Have I missed something?
>>>
>>> You're right in that the variable arena_max is initialized to 0.
>>
>> Okay.
>>
>>> However you also concluded that there is no limit to the number of
>>> arenas that can be created when arena_max is 0, which is incorrect.  As
>>> the code snippet you pasted above shows that if arena_max is 0, once we
>>> cross arena_test arenas, the narenas_limit static variable is set to a
>>> default upper limit based on the number of cores.  That acts as the
>>> upper limit to the number of arenas that can be created when arena_max is 0.
>>
>> D'oh!  Yes, of course. Thanks for that. Not sure how I managed to 
>> misread that code :-}. So a better formulation would be something like:
>>
>>     The default value of this parameter is 0, meaning that the limit on 
>>     the number of arenas is determined according to the setting of
>>     M_ARENA_TEST.
> 
> No, it is not the value of M_ARENA_TEST :)

I agree. But it's also not what I said ;-). I said "determined according
to the value of..."

Currently, the M_ARENA_TEST text in mallopt(3) says:

       M_ARENA_TEST
              This  parameter specifies a value, in number of arenas cre‐
              ated, at which point the system configuration will be exam‐
              ined  to  determine  a  hard limit on the number of created
              arenas.  (See M_ARENA_MAX for the definition of an arena.)

              The computation of the arena hard limit is  implementation-
              defined and is usually calculated as a multiple of the num‐
              ber of available CPUs.  Once the hard  limit  is  computed,
              the result is final and constrains the total number of are‐
              nas.

              The default value for the M_ARENA_TEST parameter  is  2  on
              systems  where  sizeof(long)  is  4;  otherwise the default
              value is 8.

              This parameter has been  available  since  glibc  2.10  via
              --enable-experimental-malloc,   and  since  glibc  2.15  by
              default.

              The value of M_ARENA_TEST is not used when M_ARENA_MAX  has
              a nonzero value.

> M_ARENA_TEST is set by default to NARENAS_FROM_CORES(1).  The limit on
> the number of arenas is decided after M_ARENA_TEST number of arenas have
> already created, but it is actually set to NARENAS_FROM_CORES(n), where
> n is the number of cores.  If we can't find the number of cores for some
> reason, then we set the limit to NARENAS_FROM_CORES(2).
> 
> So something like this is more accurate:
> 
>     The default value of this parameter is 0, meaning that the limit on
>     the number of arenas is determined by the number of CPU cores
>     online and the size of the '''long''' datatype.  For 32-bit systems
>     the limit of on the number of arenas is 2 * '''number of CPU cores
>     online''' while 64-bit systems, the limit on the number of arenas
>     is 8 * '''number of CPU cores online'''.  If information on CPU
>     cores is not available, it is assumed that there are 2 CPU cores
>     online.

Your details following are a little more precise. But I believe Carlos,
when he wrote the text about how the limit was calculated, deliberately
chose to be more vague. I'm okay with including this level of detail 
in the man page, but, if I recall correctly, Carlos wanted to avoid it.

Cheers,

Michael

> I have implicitly stated here that 32-bit long == 32-bit pointers since
> I don't think there are Unix systems that have 32-bit long and 64-bit
> pointers and also because the choice of multipliers for cores (2 and 8)
> correlates better to the size of the address space than to size of long.
> 
> Siddhesh
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Andreas Schwab]

On Okt 18 2016, Siddhesh Poyarekar <siddhesh@sourceware.org> wrote:

> So something like this is more accurate:
>
>     The default value of this parameter is 0, meaning that the limit on
>     the number of arenas is determined by the number of CPU cores
>     online and the size of the '''long''' datatype.  For 32-bit systems
>     the limit of on the number of arenas is 2 * '''number of CPU cores
                                              twice the
>     online''' while 64-bit systems, the limit on the number of arenas
>     is 8 * '''number of CPU cores online'''.  If information on CPU
         eight times the
>     cores is not available, it is assumed that there are 2 CPU cores
>     online.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

On Tuesday 18 October 2016 09:33 PM, Michael Kerrisk (man-pages) wrote:
>>> I don't think you're correct here. 'arena_max' is a field in a static
>>> structure that is not otherwise initialized, AFAICT. So, it has the
>>> value zero. (Some dirty hacking with a program that uses
>>> malloc_get_state() and inspects the internal data structure seems
>>> to confirm this.)
>>>
>>> And then in malloc/arena.c we have
>>>
>>>           if (mp_.arena_max != 0)
>>>             narenas_limit = mp_.arena_max;
>>>           else if (narenas > mp_.arena_test)
>>>             {
>>>               int n = __get_nprocs ();
>>>               
>>>               if (n >= 1)
>>>                 narenas_limit = NARENAS_FROM_NCORES (n);
>>>               else
>>>                 /* We have no information about the system.  Assume two
>>>                    cores.  */ 
>>>                 narenas_limit = NARENAS_FROM_NCORES (2);
>>>
>>> So, I believe my original statement about M_ARENA_MAX is correct.
>>> Have I missed something?
>>
>> You're right in that the variable arena_max is initialized to 0.
> 
> Okay.
> 
>> However you also concluded that there is no limit to the number of
>> arenas that can be created when arena_max is 0, which is incorrect.  As
>> the code snippet you pasted above shows that if arena_max is 0, once we
>> cross arena_test arenas, the narenas_limit static variable is set to a
>> default upper limit based on the number of cores.  That acts as the
>> upper limit to the number of arenas that can be created when arena_max is 0.
> 
> D'oh!  Yes, of course. Thanks for that. Not sure how I managed to 
> misread that code :-}. So a better formulation would be something like:
> 
>     The default value of this parameter is 0, meaning that the limit on 
>     the number of arenas is determined according to the setting of
>     M_ARENA_TEST.

No, it is not the value of M_ARENA_TEST :)

M_ARENA_TEST is set by default to NARENAS_FROM_CORES(1).  The limit on
the number of arenas is decided after M_ARENA_TEST number of arenas have
already created, but it is actually set to NARENAS_FROM_CORES(n), where
n is the number of cores.  If we can't find the number of cores for some
reason, then we set the limit to NARENAS_FROM_CORES(2).

So something like this is more accurate:

    The default value of this parameter is 0, meaning that the limit on
    the number of arenas is determined by the number of CPU cores
    online and the size of the '''long''' datatype.  For 32-bit systems
    the limit of on the number of arenas is 2 * '''number of CPU cores
    online''' while 64-bit systems, the limit on the number of arenas
    is 8 * '''number of CPU cores online'''.  If information on CPU
    cores is not available, it is assumed that there are 2 CPU cores
    online.

I have implicitly stated here that 32-bit long == 32-bit pointers since
I don't think there are Unix systems that have 32-bit long and 64-bit
pointers and also because the choice of multipliers for cores (2 and 8)
correlates better to the size of the address space than to size of long.

Siddhesh

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Michael Kerrisk (man-pages)]

Hi Siddhesh,

On 10/18/2016 04:30 PM, Siddhesh Poyarekar wrote:
> On Tuesday 18 October 2016 07:20 PM, Michael Kerrisk (man-pages) wrote:
>> Hi Siddhesh,
>>
>> On 10/18/2016 12:07 PM, Siddhesh Poyarekar wrote:
>>> On Tuesday 18 October 2016 12:45 PM, Michael Kerrisk wrote:
>>>> So my reading from malloc/malloc.c:
>>>>
>>>> #define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))
>>>>   .arena_test = NARENAS_FROM_NCORES (1)
>>>>
>>>> So, the default value for this parameter is 2 on systems where
>>>> sizeof(long) is 4; otherwise the default value is 8.
>>>
>>> That is correct.
>>
>> Thanks for the confirmation.
>>
>> [You've over trimmed the mail for this next piece. To be clear, below,
>> we are talking about M_ARENA_MAX; or at least I was.]
>>
>>>> So, IIUC, the default value of this parameter is 0, meaning that there
>>>> is no limit on the number of arenas that can be created. Do you
>>>> confirm, Siddhesh?
>>>
>>> No, the default is a function of the number of cores using the
>>> NARENAS_FROM_NCORES macro.  That is, 2 * (number of cores) where
>>> sizeof(long) == 4 and 8 * (number of cores) otherwise.  If the number of
>>> cores is not available (i.e. we can't read this info for some reason)
>>> then we default to 4 and 16 respectively, i.e. assume 2 cores.
>>>
>>> Note that this default only comes into force once the number of arenas
>>> cross arena_test.
>>
>> I don't think you're correct here. 'arena_max' is a field in a static
>> structure that is not otherwise initialized, AFAICT. So, it has the
>> value zero. (Some dirty hacking with a program that uses
>> malloc_get_state() and inspects the internal data structure seems
>> to confirm this.)
>>
>> And then in malloc/arena.c we have
>>
>>           if (mp_.arena_max != 0)
>>             narenas_limit = mp_.arena_max;
>>           else if (narenas > mp_.arena_test)
>>             {
>>               int n = __get_nprocs ();
>>               
>>               if (n >= 1)
>>                 narenas_limit = NARENAS_FROM_NCORES (n);
>>               else
>>                 /* We have no information about the system.  Assume two
>>                    cores.  */ 
>>                 narenas_limit = NARENAS_FROM_NCORES (2);
>>
>> So, I believe my original statement about M_ARENA_MAX is correct.
>> Have I missed something?
> 
> You're right in that the variable arena_max is initialized to 0.

Okay.

> However you also concluded that there is no limit to the number of
> arenas that can be created when arena_max is 0, which is incorrect.  As
> the code snippet you pasted above shows that if arena_max is 0, once we
> cross arena_test arenas, the narenas_limit static variable is set to a
> default upper limit based on the number of cores.  That acts as the
> upper limit to the number of arenas that can be created when arena_max is 0.

D'oh!  Yes, of course. Thanks for that. Not sure how I managed to 
misread that code :-}. So a better formulation would be something like:

    The default value of this parameter is 0, meaning that the limit on 
    the number of arenas is determined according to the setting of
    M_ARENA_TEST.

Seem okay?

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

On Tuesday 18 October 2016 07:20 PM, Michael Kerrisk (man-pages) wrote:
> Hi Siddhesh,
> 
> On 10/18/2016 12:07 PM, Siddhesh Poyarekar wrote:
>> On Tuesday 18 October 2016 12:45 PM, Michael Kerrisk wrote:
>>> So my reading from malloc/malloc.c:
>>>
>>> #define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))
>>>   .arena_test = NARENAS_FROM_NCORES (1)
>>>
>>> So, the default value for this parameter is 2 on systems where
>>> sizeof(long) is 4; otherwise the default value is 8.
>>
>> That is correct.
> 
> Thanks for the confirmation.
> 
> [You've over trimmed the mail for this next piece. To be clear, below,
> we are talking about M_ARENA_MAX; or at least I was.]
> 
>>> So, IIUC, the default value of this parameter is 0, meaning that there
>>> is no limit on the number of arenas that can be created. Do you
>>> confirm, Siddhesh?
>>
>> No, the default is a function of the number of cores using the
>> NARENAS_FROM_NCORES macro.  That is, 2 * (number of cores) where
>> sizeof(long) == 4 and 8 * (number of cores) otherwise.  If the number of
>> cores is not available (i.e. we can't read this info for some reason)
>> then we default to 4 and 16 respectively, i.e. assume 2 cores.
>>
>> Note that this default only comes into force once the number of arenas
>> cross arena_test.
> 
> I don't think you're correct here. 'arena_max' is a field in a static
> structure that is not otherwise initialized, AFAICT. So, it has the
> value zero. (Some dirty hacking with a program that uses
> malloc_get_state() and inspects the internal data structure seems
> to confirm this.)
> 
> And then in malloc/arena.c we have
> 
>           if (mp_.arena_max != 0)
>             narenas_limit = mp_.arena_max;
>           else if (narenas > mp_.arena_test)
>             {
>               int n = __get_nprocs ();
>               
>               if (n >= 1)
>                 narenas_limit = NARENAS_FROM_NCORES (n);
>               else
>                 /* We have no information about the system.  Assume two
>                    cores.  */ 
>                 narenas_limit = NARENAS_FROM_NCORES (2);
> 
> So, I believe my original statement about M_ARENA_MAX is correct.
> Have I missed something?

You're right in that the variable arena_max is initialized to 0.
However you also concluded that there is no limit to the number of
arenas that can be created when arena_max is 0, which is incorrect.  As
the code snippet you pasted above shows that if arena_max is 0, once we
cross arena_test arenas, the narenas_limit static variable is set to a
default upper limit based on the number of cores.  That acts as the
upper limit to the number of arenas that can be created when arena_max is 0.

Siddhesh

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Michael Kerrisk (man-pages)]

Hi Siddhesh,

On 10/18/2016 12:07 PM, Siddhesh Poyarekar wrote:
> On Tuesday 18 October 2016 12:45 PM, Michael Kerrisk wrote:
>> So my reading from malloc/malloc.c:
>>
>> #define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))
>>   .arena_test = NARENAS_FROM_NCORES (1)
>>
>> So, the default value for this parameter is 2 on systems where
>> sizeof(long) is 4; otherwise the default value is 8.
> 
> That is correct.

Thanks for the confirmation.

[You've over trimmed the mail for this next piece. To be clear, below,
we are talking about M_ARENA_MAX; or at least I was.]

>> So, IIUC, the default value of this parameter is 0, meaning that there
>> is no limit on the number of arenas that can be created. Do you
>> confirm, Siddhesh?
> 
> No, the default is a function of the number of cores using the
> NARENAS_FROM_NCORES macro.  That is, 2 * (number of cores) where
> sizeof(long) == 4 and 8 * (number of cores) otherwise.  If the number of
> cores is not available (i.e. we can't read this info for some reason)
> then we default to 4 and 16 respectively, i.e. assume 2 cores.
> 
> Note that this default only comes into force once the number of arenas
> cross arena_test.

I don't think you're correct here. 'arena_max' is a field in a static
structure that is not otherwise initialized, AFAICT. So, it has the
value zero. (Some dirty hacking with a program that uses
malloc_get_state() and inspects the internal data structure seems
to confirm this.)

And then in malloc/arena.c we have

          if (mp_.arena_max != 0)
            narenas_limit = mp_.arena_max;
          else if (narenas > mp_.arena_test)
            {
              int n = __get_nprocs ();
              
              if (n >= 1)
                narenas_limit = NARENAS_FROM_NCORES (n);
              else
                /* We have no information about the system.  Assume two
                   cores.  */ 
                narenas_limit = NARENAS_FROM_NCORES (2);

So, I believe my original statement about M_ARENA_MAX is correct.
Have I missed something?

Cheers,

Michael

>> Siddhesh, in case you want to use any of my wordings (assuming they
>> are correct), I place them in the public domain. (The text will also
>> land in the mallopt(3) manual page.)
> 
> Thanks.
> 
> Siddhesh
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

On Tuesday 18 October 2016 12:45 PM, Michael Kerrisk wrote:
> So my reading from malloc/malloc.c:
> 
> #define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))
>   .arena_test = NARENAS_FROM_NCORES (1)
> 
> So, the default value for this parameter is 2 on systems where
> sizeof(long) is 4; otherwise the default value is 8.

That is correct.

> So, IIUC, the default value of this parameter is 0, meaning that there
> is no limit on the number of arenas that can be created. Do you
> confirm, Siddhesh?

No, the default is a function of the number of cores using the
NARENAS_FROM_NCORES macro.  That is, 2 * (number of cores) where
sizeof(long) == 4 and 8 * (number of cores) otherwise.  If the number of
cores is not available (i.e. we can't read this info for some reason)
then we default to 4 and 16 respectively, i.e. assume 2 cores.

Note that this default only comes into force once the number of arenas
cross arena_test.

> Siddhesh, in case you want to use any of my wordings (assuming they
> are correct), I place them in the public domain. (The text will also
> land in the mallopt(3) manual page.)

Thanks.

Siddhesh

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Michael Kerrisk]

On Mon, Oct 17, 2016 at 6:08 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/10/2016 01:26 PM, Siddhesh Poyarekar wrote:
>> The M_ARENA_* mallopt parameters are in wide use in production to
>> control the number of arenas that a long lived process creates and
>> hence there is no point in stating that this interface is non-public.
>> Document this interface and remove the obsolete comment.
>>
>>       * manual/memory.texi (M_ARENA_TEST): Add documentation.
>>       (M_ARENA_MAX): Likewise.
>>       * malloc/malloc.c: Remove obsolete comment.
>> ---
>>  malloc/malloc.c    |  1 -
>>  manual/memory.texi | 21 +++++++++++++++++++++
>>  2 files changed, 21 insertions(+), 1 deletion(-)
>>
>> diff --git a/malloc/malloc.c b/malloc/malloc.c
>> index 09e004b..0011a6d 100644
>> --- a/malloc/malloc.c
>> +++ b/malloc/malloc.c
>> @@ -1718,7 +1718,6 @@ static struct malloc_par mp_ =
>>  };
>>
>>
>> -/*  Non public mallopt parameters.  */
>>  #define M_ARENA_TEST -7
>>  #define M_ARENA_MAX  -8
>>
>> diff --git a/manual/memory.texi b/manual/memory.texi
>> index 222f126..b98dcf2 100644
>> --- a/manual/memory.texi
>> +++ b/manual/memory.texi
>> @@ -1153,6 +1153,27 @@ order to return memory to the system.
>>
>>  This parameter can also be set for the process at startup by setting the
>>  environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
>> +
>> +@item M_ARENA_TEST
>
> This description of an arena doesn't belong here, it belongs higher up in the
> manual, and in fact I would suggest the following:
>
> - Move 3.2.2.6 "Efficiency considerations for malloc" up to just under
>   "Dynamic Memory allocation" and put it under a new subsection "The GNU allocator".
>
> - Under "The GNU allocator" give a brief 1 paragraph description of how
>   the allocator works e.g. https://sourceware.org/glibc/wiki/MallocInternals
>   and talk about arenas.
>
>> +An arena is a memory pool that is allocated to act as a heap alongside the
>> +system heap that the kernel creates for the process.  This is to provide
>> +multiple heap structures for multiple threads of a process to reduce contention
>> +between them.  The limit to the number of such arenas per process is determined
>> +by the number of cores of the system and whether it is a 32-bit or a 64-bit
>> +processor.
>
> You don't say what M_ARENA_TEST is for?
>
>> +
>> +The value is ignored if @code{M_ARENA_MAX} is set either via @code{mallopt} or
>> +via the @code{MALLOC_ARENA_MAX} environment variable.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_ARENA_TEST} to the desired value.
>
> What is the default?

So my reading from malloc/malloc.c:

#define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))
  .arena_test = NARENAS_FROM_NCORES (1)

So, the default value for this parameter is 2 on systems where
sizeof(long) is 4; otherwise the default value is 8.

>> +
>> +@item M_ARENA_MAX
>> +This parameter sets the number of arenas to use regardless of the number of
>> +cores in the system.
>> +
>> +This parameter can also be set for the process at startup by setting the
>> +environment variable @code{MALLOC_ARENA_MAX} to the desired value.
>
> What is the default?

So, IIUC, the default value of this parameter is 0, meaning that there
is no limit on the number of arenas that can be created. Do you
confirm, Siddhesh?

Siddhesh, in case you want to use any of my wordings (assuming they
are correct), I place them in the public domain. (The text will also
land in the mallopt(3) manual page.)

Cheers,

Michael

Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Carlos O'Donell]

On 10/10/2016 01:26 PM, Siddhesh Poyarekar wrote:
> The M_ARENA_* mallopt parameters are in wide use in production to
> control the number of arenas that a long lived process creates and
> hence there is no point in stating that this interface is non-public.
> Document this interface and remove the obsolete comment.
> 
> 	* manual/memory.texi (M_ARENA_TEST): Add documentation.
> 	(M_ARENA_MAX): Likewise.
> 	* malloc/malloc.c: Remove obsolete comment.
> ---
>  malloc/malloc.c    |  1 -
>  manual/memory.texi | 21 +++++++++++++++++++++
>  2 files changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/malloc/malloc.c b/malloc/malloc.c
> index 09e004b..0011a6d 100644
> --- a/malloc/malloc.c
> +++ b/malloc/malloc.c
> @@ -1718,7 +1718,6 @@ static struct malloc_par mp_ =
>  };
>  
>  
> -/*  Non public mallopt parameters.  */
>  #define M_ARENA_TEST -7
>  #define M_ARENA_MAX  -8
>  
> diff --git a/manual/memory.texi b/manual/memory.texi
> index 222f126..b98dcf2 100644
> --- a/manual/memory.texi
> +++ b/manual/memory.texi
> @@ -1153,6 +1153,27 @@ order to return memory to the system.
>  
>  This parameter can also be set for the process at startup by setting the
>  environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
> +
> +@item M_ARENA_TEST

This description of an arena doesn't belong here, it belongs higher up in the
manual, and in fact I would suggest the following:

- Move 3.2.2.6 "Efficiency considerations for malloc" up to just under
  "Dynamic Memory allocation" and put it under a new subsection "The GNU allocator".

- Under "The GNU allocator" give a brief 1 paragraph description of how
  the allocator works e.g. https://sourceware.org/glibc/wiki/MallocInternals
  and talk about arenas.

> +An arena is a memory pool that is allocated to act as a heap alongside the
> +system heap that the kernel creates for the process.  This is to provide
> +multiple heap structures for multiple threads of a process to reduce contention
> +between them.  The limit to the number of such arenas per process is determined
> +by the number of cores of the system and whether it is a 32-bit or a 64-bit
> +processor.

You don't say what M_ARENA_TEST is for?

> +
> +The value is ignored if @code{M_ARENA_MAX} is set either via @code{mallopt} or
> +via the @code{MALLOC_ARENA_MAX} environment variable.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_ARENA_TEST} to the desired value.

What is the default?

> +
> +@item M_ARENA_MAX
> +This parameter sets the number of arenas to use regardless of the number of
> +cores in the system.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_ARENA_MAX} to the desired value.

What is the default?

>  @end table
>  
>  @end deftypefun
> 


-- 
Cheers,
Carlos.

[PATCH 2/2] Document the M_ARENA_* mallopt parameters

[Siddhesh Poyarekar]

The M_ARENA_* mallopt parameters are in wide use in production to
control the number of arenas that a long lived process creates and
hence there is no point in stating that this interface is non-public.
Document this interface and remove the obsolete comment.

	* manual/memory.texi (M_ARENA_TEST): Add documentation.
	(M_ARENA_MAX): Likewise.
	* malloc/malloc.c: Remove obsolete comment.
---
 malloc/malloc.c    |  1 -
 manual/memory.texi | 21 +++++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/malloc/malloc.c b/malloc/malloc.c
index 09e004b..0011a6d 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -1718,7 +1718,6 @@ static struct malloc_par mp_ =
 };
 
 
-/*  Non public mallopt parameters.  */
 #define M_ARENA_TEST -7
 #define M_ARENA_MAX  -8
 
diff --git a/manual/memory.texi b/manual/memory.texi
index 222f126..b98dcf2 100644
--- a/manual/memory.texi
+++ b/manual/memory.texi
@@ -1153,6 +1153,27 @@ order to return memory to the system.
 
 This parameter can also be set for the process at startup by setting the
 environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
+
+@item M_ARENA_TEST
+An arena is a memory pool that is allocated to act as a heap alongside the
+system heap that the kernel creates for the process.  This is to provide
+multiple heap structures for multiple threads of a process to reduce contention
+between them.  The limit to the number of such arenas per process is determined
+by the number of cores of the system and whether it is a 32-bit or a 64-bit
+processor.
+
+The value is ignored if @code{M_ARENA_MAX} is set either via @code{mallopt} or
+via the @code{MALLOC_ARENA_MAX} environment variable.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_ARENA_TEST} to the desired value.
+
+@item M_ARENA_MAX
+This parameter sets the number of arenas to use regardless of the number of
+cores in the system.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_ARENA_MAX} to the desired value.
 @end table
 
 @end deftypefun
-- 
2.7.4