* [PATCH 0/2] Malloc manual cleanups @ 2016-10-24 14:08 Siddhesh Poyarekar 2016-10-24 14:08 ` [PATCH 2/2] Document the M_ARENA_* mallopt parameters Siddhesh Poyarekar 2016-10-24 14:08 ` [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables Siddhesh Poyarekar 0 siblings, 2 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-24 14:08 UTC (permalink / raw) To: libc-alpha Here's take 2 of the cleanup of descriptions in the manual description in malloc. As suggested in previous review, I have made the following changes: - Moved up "Efficiency considerations for malloc" into a new subsection called "The GNU allocator" that gives a high level description of the allocator design and introduces terms like arenas and chunks. I did not go into too much detail here and have instead pointed interested readers to the wiki page DJ wrote. - Added default values of all of the mallopt parameters Siddhesh Poyarekar (2): Add note on MALLOC_MMAP_* environment variables Document the M_ARENA_* mallopt parameters malloc/malloc.c | 1 - manual/memory.texi | 153 ++++++++++++++++++++++++++++++++--------------------- 2 files changed, 93 insertions(+), 61 deletions(-) -- 2.7.4 ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 2/2] Document the M_ARENA_* mallopt parameters 2016-10-24 14:08 [PATCH 0/2] Malloc manual cleanups Siddhesh Poyarekar @ 2016-10-24 14:08 ` Siddhesh Poyarekar 2016-10-25 7:51 ` Rical Jasan 2016-10-25 21:55 ` DJ Delorie 2016-10-24 14:08 ` [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables Siddhesh Poyarekar 1 sibling, 2 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-24 14:08 UTC (permalink / raw) To: libc-alpha The M_ARENA_* mallopt parameters are in wide use in production to control the number of arenas that a long lived process creates and hence there is no point in stating that this interface is non-public. Document this interface and remove the obsolete comment. * manual/memory.texi (M_ARENA_TEST): Add documentation. (M_ARENA_MAX): Likewise. * malloc/malloc.c: Remove obsolete comment. --- malloc/malloc.c | 1 - manual/memory.texi | 122 +++++++++++++++++++++++++++-------------------------- 2 files changed, 62 insertions(+), 61 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index ef04360..a849901 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -1718,7 +1718,6 @@ static struct malloc_par mp_ = }; -/* Non public mallopt parameters. */ #define M_ARENA_TEST -7 #define M_ARENA_MAX -8 diff --git a/manual/memory.texi b/manual/memory.texi index 6f33455..198a933 100644 --- a/manual/memory.texi +++ b/manual/memory.texi @@ -162,6 +162,8 @@ special to @theglibc{} and GNU Compiler. @menu * Memory Allocation and C:: How to get different kinds of allocation in C. +* The GNU allocator:: An overview of the GNU @code{malloc} + implementation. * Unconstrained Allocation:: The @code{malloc} facility allows fully general dynamic allocation. * Allocation Debugging:: Finding memory leaks and not freed memory. @@ -258,6 +260,43 @@ address of the space. Then you can use the operators @samp{*} and @} @end smallexample +@node The GNU allocator +@subsection The GNU allocator +@cindex gnu allocator + +The @code{malloc} implementation in @theglibc{} is derived from ptmalloc +(pthreads malloc), which in turn is derived from dlmalloc (Doug Lea malloc). +This malloc may allocate memory in two different ways depending on their size +and certain parameters that may be controlled by users. The most common way is +to allocate portions of memory (called chunks) from a large contiguous area of +memory and manage these areas to optimize their use and reduce wastage in the +form of unusable chunks. Traditionally the system heap was set up to be the one +large memory area but @theglibc{} @code{malloc} implementation maintains +multiple such areas to optimize their use in multi-threaded applications. Each +such area is internally referred to as an @code{arena}. + +As opposed to other versions, the @code{malloc} in @theglibc{} does not round +up chunk sizes to powers of two, neither for large nor for small sizes. +Neighboring chunks can be coalesced on a @code{free} no matter what their size +is. This makes the implementation suitable for all kinds of allocation +patterns without generally incurring high memory waste through fragmentation. +The presence of multiple @code{arenas} allows multiple threads to allocate +memory simultaneously in their own separate arenas, thus improving performance. + +The other way of memory allocation is for very large blocks, i.e. much larger +than a page. These requests are allocated with @code{mmap} (anonymous or via +@code{/dev/zero}). This has the great advantage that these chunks are returned +to the system immediately when they are freed. Therefore, it cannot happen +that a large chunk becomes ``locked'' in between smaller ones and even after +calling @code{free} wastes memory. The size threshold for @code{mmap} to be +used is dynamic and gets adjusted according to allocation patterns of the +program. This can also be statically adjusted with @code{mallopt}. The use of +@code{mmap} can also be disabled completely. + +A more detailed technical description of the GNU allocator is maintained in +@theglibc{} wiki. See +@uref{https://sourceware.org/glibc/wiki/MallocInternals}. + @node Unconstrained Allocation @subsection Unconstrained Allocation @cindex unconstrained memory allocation @@ -278,8 +317,6 @@ any time (or never). bigger or smaller. * Allocating Cleared Space:: Use @code{calloc} to allocate a block and clear it. -* Efficiency and Malloc:: Efficiency considerations in use of - these functions. * Aligned Memory Blocks:: Allocating specially aligned memory. * Malloc Tunable Parameters:: Use @code{mallopt} to adjust allocation parameters. @@ -867,59 +904,6 @@ But in general, it is not guaranteed that @code{calloc} calls @code{malloc}/@code{realloc}/@code{free} outside the C library, it should always define @code{calloc}, too. -@node Efficiency and Malloc -@subsubsection Efficiency Considerations for @code{malloc} -@cindex efficiency and @code{malloc} - - - - -@ignore - -@c No longer true, see below instead. -To make the best use of @code{malloc}, it helps to know that the GNU -version of @code{malloc} always dispenses small amounts of memory in -blocks whose sizes are powers of two. It keeps separate pools for each -power of two. This holds for sizes up to a page size. Therefore, if -you are free to choose the size of a small block in order to make -@code{malloc} more efficient, make it a power of two. -@c !!! xref getpagesize - -Once a page is split up for a particular block size, it can't be reused -for another size unless all the blocks in it are freed. In many -programs, this is unlikely to happen. Thus, you can sometimes make a -program use memory more efficiently by using blocks of the same size for -many different purposes. - -When you ask for memory blocks of a page or larger, @code{malloc} uses a -different strategy; it rounds the size up to a multiple of a page, and -it can coalesce and split blocks as needed. - -The reason for the two strategies is that it is important to allocate -and free small blocks as fast as possible, but speed is less important -for a large block since the program normally spends a fair amount of -time using it. Also, large blocks are normally fewer in number. -Therefore, for large blocks, it makes sense to use a method which takes -more time to minimize the wasted space. - -@end ignore - -As opposed to other versions, the @code{malloc} in @theglibc{} -does not round up block sizes to powers of two, neither for large nor -for small sizes. Neighboring chunks can be coalesced on a @code{free} -no matter what their size is. This makes the implementation suitable -for all kinds of allocation patterns without generally incurring high -memory waste through fragmentation. - -Very large blocks (much larger than a page) are allocated with -@code{mmap} (anonymous or via @code{/dev/zero}) by this implementation. -This has the great advantage that these chunks are returned to the -system immediately when they are freed. Therefore, it cannot happen -that a large chunk becomes ``locked'' in between smaller ones and even -after calling @code{free} wastes memory. The size threshold for -@code{mmap} to be used can be adjusted with @code{mallopt}. The use of -@code{mmap} can also be disabled completely. - @node Aligned Memory Blocks @subsubsection Allocating Aligned Memory Blocks @@ -1105,10 +1089,6 @@ parameter to be set, and @var{value} the new value to be set. Possible choices for @var{param}, as defined in @file{malloc.h}, are: @table @code -@comment TODO: @item M_ARENA_MAX -@comment - Document ARENA_MAX env var. -@comment TODO: @item M_ARENA_TEST -@comment - Document ARENA_TEST env var. @comment TODO: @item M_CHECK_ACTION @item M_MMAP_MAX The maximum number of chunks to allocate with @code{mmap}. Setting this @@ -1169,6 +1149,28 @@ value is set statically to the provided input. This parameter can also be set for the process at startup by setting the environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. + +@item M_ARENA_TEST +This parameter specifies the number of arenas that can be created before the +test on the limit to the number of arenas is conducted. The value is ignored if +@code{M_ARENA_MAX} is set. + +The default value of this parameter is 2 on 32-bit systems and 8 on 64-bit +systems. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_ARENA_TEST} to the desired value. +@item M_ARENA_MAX +This parameter sets the number of arenas to use regardless of the number of +cores in the system. + +The default value of this tunable is @code{0}, meaning that the limit on the +number of arenas is determined by the number of CPU cores online. For 32-bit +systems the limit is twice the number of cores online and on 64-bit systems, it +is eight times the number of cores online. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_ARENA_MAX} to the desired value. @end table @end deftypefun @@ -1511,7 +1513,7 @@ This is the total size of memory allocated with @code{sbrk} by This is the number of chunks not in use. (The memory allocator internally gets chunks of memory from the operating system, and then carves them up to satisfy individual @code{malloc} requests; see -@ref{Efficiency and Malloc}.) +@ref{The GNU allocator}.) @item int smblks This field is unused. -- 2.7.4 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters 2016-10-24 14:08 ` [PATCH 2/2] Document the M_ARENA_* mallopt parameters Siddhesh Poyarekar @ 2016-10-25 7:51 ` Rical Jasan 2016-10-25 21:55 ` DJ Delorie 1 sibling, 0 replies; 23+ messages in thread From: Rical Jasan @ 2016-10-25 7:51 UTC (permalink / raw) To: Siddhesh Poyarekar; +Cc: libc-alpha Hi Siddhesh, Most of this is formatting or cleanup/improvement of the pre-existing chunks that were moved, but the comment on M_ARENA_MAX regards content. On 10/24/2016 07:07 AM, Siddhesh Poyarekar wrote: > The M_ARENA_* mallopt parameters are in wide use in production to > control the number of arenas that a long lived process creates and > hence there is no point in stating that this interface is non-public. > Document this interface and remove the obsolete comment. > > * manual/memory.texi (M_ARENA_TEST): Add documentation. > (M_ARENA_MAX): Likewise. > * malloc/malloc.c: Remove obsolete comment. > --- > malloc/malloc.c | 1 - > manual/memory.texi | 122 +++++++++++++++++++++++++++-------------------------- > 2 files changed, 62 insertions(+), 61 deletions(-) > > diff --git a/malloc/malloc.c b/malloc/malloc.c > index ef04360..a849901 100644 > --- a/malloc/malloc.c > +++ b/malloc/malloc.c > @@ -1718,7 +1718,6 @@ static struct malloc_par mp_ = > }; > > > -/* Non public mallopt parameters. */ > #define M_ARENA_TEST -7 > #define M_ARENA_MAX -8 > > diff --git a/manual/memory.texi b/manual/memory.texi > index 6f33455..198a933 100644 > --- a/manual/memory.texi > +++ b/manual/memory.texi > @@ -162,6 +162,8 @@ special to @theglibc{} and GNU Compiler. > > @menu > * Memory Allocation and C:: How to get different kinds of allocation in C. > +* The GNU allocator:: An overview of the GNU @code{malloc} I think "Allocator" should be capitalized. The manual is inconsistent on @subsection capitalization, but the majority of it is (capitalized). Also, all the other info menu entries visible in the context here are fully capitalized. > + implementation. > * Unconstrained Allocation:: The @code{malloc} facility allows fully general > dynamic allocation. > * Allocation Debugging:: Finding memory leaks and not freed memory. > @@ -258,6 +260,43 @@ address of the space. Then you can use the operators @samp{*} and > @} > @end smallexample > > +@node The GNU allocator > +@subsection The GNU allocator Allocator > +@cindex gnu allocator > + > +The @code{malloc} implementation in @theglibc{} is derived from ptmalloc > +(pthreads malloc), which in turn is derived from dlmalloc (Doug Lea malloc). > +This malloc may allocate memory in two different ways depending on their size > +and certain parameters that may be controlled by users. The most common way is > +to allocate portions of memory (called chunks) from a large contiguous area of > +memory and manage these areas to optimize their use and reduce wastage in the > +form of unusable chunks. Traditionally the system heap was set up to be the one > +large memory area but @theglibc{} @code{malloc} implementation maintains This should be "the @glibcadj{} @code{malloc}". > +multiple such areas to optimize their use in multi-threaded applications. Each > +such area is internally referred to as an @code{arena}. @dfn{arena} Unless this is a function name (or literal string one would be using in code, for example), this is simply a term we use to describe the concept. On first use, where we define the term, it should have @dfn{}, and otherwise it doesn't need to be stylized at all, such as... > + > +As opposed to other versions, the @code{malloc} in @theglibc{} does not round > +up chunk sizes to powers of two, neither for large nor for small sizes. > +Neighboring chunks can be coalesced on a @code{free} no matter what their size > +is. This makes the implementation suitable for all kinds of allocation > +patterns without generally incurring high memory waste through fragmentation. > +The presence of multiple @code{arenas} allows multiple threads to allocate here (no @code{}). > +memory simultaneously in their own separate arenas, thus improving performance. > + > +The other way of memory allocation is for very large blocks, i.e. much larger > +than a page. These requests are allocated with @code{mmap} (anonymous or via > +@code{/dev/zero}). This has the great advantage that these chunks are returned Should be @file{}. A reference to mmap would be good. Something like, "... (anonymous or via @file{/dev/zero}; @pxref{Memory-mapped I/O})." > +to the system immediately when they are freed. Therefore, it cannot happen > +that a large chunk becomes ``locked'' in between smaller ones and even after > +calling @code{free} wastes memory. The size threshold for @code{mmap} to be > +used is dynamic and gets adjusted according to allocation patterns of the > +program. This can also be statically adjusted with @code{mallopt}. The use of A reference to mallopt would be good here. > +@code{mmap} can also be disabled completely. Should briefly say how and/or give a reference. I believe mallopt applies to both, so maybe something like, "@code{mallopt} can be used to statically adjust the threshold using @code{M_MMAP_THRESHOLD}, and the use of @code{mmap} can be disabled completely with @code{M_MMAP_MAX}; @pxref{Malloc Tunable Parameters}." > + > +A more detailed technical description of the GNU allocator is maintained in > +@theglibc{} wiki. See the @glibcadj{} > +@uref{https://sourceware.org/glibc/wiki/MallocInternals}. > + > @node Unconstrained Allocation > @subsection Unconstrained Allocation > @cindex unconstrained memory allocation > @@ -278,8 +317,6 @@ any time (or never). > bigger or smaller. > * Allocating Cleared Space:: Use @code{calloc} to allocate a > block and clear it. > -* Efficiency and Malloc:: Efficiency considerations in use of > - these functions. > * Aligned Memory Blocks:: Allocating specially aligned memory. > * Malloc Tunable Parameters:: Use @code{mallopt} to adjust allocation > parameters. > @@ -867,59 +904,6 @@ But in general, it is not guaranteed that @code{calloc} calls > @code{malloc}/@code{realloc}/@code{free} outside the C library, it > should always define @code{calloc}, too. > > -@node Efficiency and Malloc > -@subsubsection Efficiency Considerations for @code{malloc} > -@cindex efficiency and @code{malloc} > - > - > - > - > -@ignore > - > -@c No longer true, see below instead. > -To make the best use of @code{malloc}, it helps to know that the GNU > -version of @code{malloc} always dispenses small amounts of memory in > -blocks whose sizes are powers of two. It keeps separate pools for each > -power of two. This holds for sizes up to a page size. Therefore, if > -you are free to choose the size of a small block in order to make > -@code{malloc} more efficient, make it a power of two. > -@c !!! xref getpagesize > - > -Once a page is split up for a particular block size, it can't be reused > -for another size unless all the blocks in it are freed. In many > -programs, this is unlikely to happen. Thus, you can sometimes make a > -program use memory more efficiently by using blocks of the same size for > -many different purposes. > - > -When you ask for memory blocks of a page or larger, @code{malloc} uses a > -different strategy; it rounds the size up to a multiple of a page, and > -it can coalesce and split blocks as needed. > - > -The reason for the two strategies is that it is important to allocate > -and free small blocks as fast as possible, but speed is less important > -for a large block since the program normally spends a fair amount of > -time using it. Also, large blocks are normally fewer in number. > -Therefore, for large blocks, it makes sense to use a method which takes > -more time to minimize the wasted space. > - > -@end ignore > - > -As opposed to other versions, the @code{malloc} in @theglibc{} > -does not round up block sizes to powers of two, neither for large nor > -for small sizes. Neighboring chunks can be coalesced on a @code{free} > -no matter what their size is. This makes the implementation suitable > -for all kinds of allocation patterns without generally incurring high > -memory waste through fragmentation. > - > -Very large blocks (much larger than a page) are allocated with > -@code{mmap} (anonymous or via @code{/dev/zero}) by this implementation. > -This has the great advantage that these chunks are returned to the > -system immediately when they are freed. Therefore, it cannot happen > -that a large chunk becomes ``locked'' in between smaller ones and even > -after calling @code{free} wastes memory. The size threshold for > -@code{mmap} to be used can be adjusted with @code{mallopt}. The use of > -@code{mmap} can also be disabled completely. > - > @node Aligned Memory Blocks > @subsubsection Allocating Aligned Memory Blocks > > @@ -1105,10 +1089,6 @@ parameter to be set, and @var{value} the new value to be set. Possible > choices for @var{param}, as defined in @file{malloc.h}, are: > > @table @code > -@comment TODO: @item M_ARENA_MAX > -@comment - Document ARENA_MAX env var. > -@comment TODO: @item M_ARENA_TEST > -@comment - Document ARENA_TEST env var. > @comment TODO: @item M_CHECK_ACTION > @item M_MMAP_MAX > The maximum number of chunks to allocate with @code{mmap}. Setting this > @@ -1169,6 +1149,28 @@ value is set statically to the provided input. > > This parameter can also be set for the process at startup by setting the > environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. > + > +@item M_ARENA_TEST > +This parameter specifies the number of arenas that can be created before the > +test on the limit to the number of arenas is conducted. The value is ignored if > +@code{M_ARENA_MAX} is set. > + > +The default value of this parameter is 2 on 32-bit systems and 8 on 64-bit > +systems. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_ARENA_TEST} to the desired value. > +@item M_ARENA_MAX > +This parameter sets the number of arenas to use regardless of the number of > +cores in the system. > + > +The default value of this tunable is @code{0}, meaning that the limit on the > +number of arenas is determined by the number of CPU cores online. For 32-bit > +systems the limit is twice the number of cores online and on 64-bit systems, it > +is eight times the number of cores online. Even though I had followed the thread, I immediately jumped to the same confusion with M_ARENA_TEST that was resolved in [1]. Explicitly stating here that 2 and 8 are not derived from M_ARENA_TEST defaults might save a lot of general confusion down the road. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_ARENA_MAX} to the desired value. > @end table > > @end deftypefun > @@ -1511,7 +1513,7 @@ This is the total size of memory allocated with @code{sbrk} by > This is the number of chunks not in use. (The memory allocator > internally gets chunks of memory from the operating system, and then > carves them up to satisfy individual @code{malloc} requests; see > -@ref{Efficiency and Malloc}.) > +@ref{The GNU allocator}.) Allocator Also, making this an @pxref{} now would be nice, since this renders wrong in info as-is. > > @item int smblks > This field is unused. > Rical [1] https://sourceware.org/ml/libc-alpha/2016-10/msg00310.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters 2016-10-24 14:08 ` [PATCH 2/2] Document the M_ARENA_* mallopt parameters Siddhesh Poyarekar 2016-10-25 7:51 ` Rical Jasan @ 2016-10-25 21:55 ` DJ Delorie 2016-10-26 5:42 ` Siddhesh Poyarekar 1 sibling, 1 reply; 23+ messages in thread From: DJ Delorie @ 2016-10-25 21:55 UTC (permalink / raw) To: Siddhesh Poyarekar; +Cc: libc-alpha Siddhesh Poyarekar <siddhesh@sourceware.org> writes: > -/* Non public mallopt parameters. */ > #define M_ARENA_TEST -7 > #define M_ARENA_MAX -8 Hmmm... if these are now public, do they need to be moved elsewhere? Like, malloc.h ? > +The presence of multiple @code{arenas} allows multiple threads to allocate > +memory simultaneously in their own separate arenas, thus improving performance. Not quite true - there isn't one arena per thread, there are N arenas per M threads. Probably better to say "... simultaneously in separate arenas ..." > +environment variable @code{MALLOC_ARENA_TEST} to the desired value. > +@item M_ARENA_MAX Missing blank line. Otherwise, the content looks OK to me from a technical point of view. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] Document the M_ARENA_* mallopt parameters 2016-10-25 21:55 ` DJ Delorie @ 2016-10-26 5:42 ` Siddhesh Poyarekar 0 siblings, 0 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-26 5:42 UTC (permalink / raw) To: DJ Delorie; +Cc: libc-alpha On Wednesday 26 October 2016 03:25 AM, DJ Delorie wrote: > Siddhesh Poyarekar <siddhesh@sourceware.org> writes: >> -/* Non public mallopt parameters. */ >> #define M_ARENA_TEST -7 >> #define M_ARENA_MAX -8 > > Hmmm... if these are now public, do they need to be moved elsewhere? > Like, malloc.h ? They're already there in malloc.h. I'll push a follow-up patch if they can be removed. >> +The presence of multiple @code{arenas} allows multiple threads to allocate >> +memory simultaneously in their own separate arenas, thus improving performance. > > Not quite true - there isn't one arena per thread, there are N arenas > per M threads. Probably better to say "... simultaneously in separate > arenas ..." OK. >> +environment variable @code{MALLOC_ARENA_TEST} to the desired value. >> +@item M_ARENA_MAX > > Missing blank line. > > Otherwise, the content looks OK to me from a technical point of view. Thanks, I'll fix up formatting comments from Rical and push this. Siddhesh ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-24 14:08 [PATCH 0/2] Malloc manual cleanups Siddhesh Poyarekar 2016-10-24 14:08 ` [PATCH 2/2] Document the M_ARENA_* mallopt parameters Siddhesh Poyarekar @ 2016-10-24 14:08 ` Siddhesh Poyarekar 2016-10-26 4:18 ` Carlos O'Donell 1 sibling, 1 reply; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-24 14:08 UTC (permalink / raw) To: libc-alpha The mallopt parameters manual does not mention the environment variables that can be used to set these parameters at program startup. Mention those environment variables for completeness. * manual/memory.texi: Add environment variable alternatives to setting mallopt parameters. --- manual/memory.texi | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/manual/memory.texi b/manual/memory.texi index 5383105..6f33455 100644 --- a/manual/memory.texi +++ b/manual/memory.texi @@ -1113,12 +1113,25 @@ choices for @var{param}, as defined in @file{malloc.h}, are: @item M_MMAP_MAX The maximum number of chunks to allocate with @code{mmap}. Setting this to zero disables all use of @code{mmap}. + +The default value of this parameter is @code{65536}. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_MMAP_MAX_} to the desired value. @item M_MMAP_THRESHOLD All chunks larger than this value are allocated outside the normal heap, using the @code{mmap} system call. This way it is guaranteed that the memory for these chunks can be returned to the system on @code{free}. Note that requests smaller than this threshold might still be allocated via @code{mmap}. + +If this parameter is not set, the default value is set as 128 KB and the +threshold is adjusted dynamically to suit the allocation patterns of the +program. If the parameter is set, the dynamic adjustment is disabled and the +value is set statically to the input value. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value. @comment TODO: @item M_MXFAST @item M_PERTURB If non-zero, memory blocks are filled with values depending on some @@ -1128,16 +1141,34 @@ use of uninitialized or freed heap memory. Note that this option does not guarantee that the freed block will have any specific values. It only guarantees that the content the block had before it was freed will be overwritten. + +The default value of this parameter is @code{0}. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value. @item M_TOP_PAD This parameter determines the amount of extra memory to obtain from the system when a call to @code{sbrk} is required. It also specifies the number of bytes to retain when shrinking the heap by calling @code{sbrk} with a negative argument. This provides the necessary hysteresis in heap size such that excessive amounts of system calls can be avoided. + +The default value of this parameter is @code{0}. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_TOP_PAD_} to the desired value. @item M_TRIM_THRESHOLD This is the minimum size (in bytes) of the top-most, releasable chunk that will cause @code{sbrk} to be called with a negative argument in order to return memory to the system. + +If this parameter is not set, the default value is set as 128 KB and the +threshold is adjusted dynamically to suit the allocation patterns of the +program. If the parameter is set, the dynamic adjustment is disabled and the +value is set statically to the provided input. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. @end table @end deftypefun -- 2.7.4 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-24 14:08 ` [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables Siddhesh Poyarekar @ 2016-10-26 4:18 ` Carlos O'Donell 2016-10-26 7:58 ` Michael Kerrisk ` (2 more replies) 0 siblings, 3 replies; 23+ messages in thread From: Carlos O'Donell @ 2016-10-26 4:18 UTC (permalink / raw) To: Siddhesh Poyarekar, libc-alpha On 10/24/2016 10:07 AM, Siddhesh Poyarekar wrote: > The mallopt parameters manual does not mention the environment > variables that can be used to set these parameters at program startup. > Mention those environment variables for completeness. > > * manual/memory.texi: Add environment variable alternatives to > setting mallopt parameters. Two nits. Otherwise OK. > --- > manual/memory.texi | 31 +++++++++++++++++++++++++++++++ > 1 file changed, 31 insertions(+) > > diff --git a/manual/memory.texi b/manual/memory.texi > index 5383105..6f33455 100644 > --- a/manual/memory.texi > +++ b/manual/memory.texi > @@ -1113,12 +1113,25 @@ choices for @var{param}, as defined in @file{malloc.h}, are: > @item M_MMAP_MAX > The maximum number of chunks to allocate with @code{mmap}. Setting this > to zero disables all use of @code{mmap}. > + > +The default value of this parameter is @code{65536}. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_MAX_} to the desired value. > @item M_MMAP_THRESHOLD > All chunks larger than this value are allocated outside the normal > heap, using the @code{mmap} system call. This way it is guaranteed > that the memory for these chunks can be returned to the system on > @code{free}. Note that requests smaller than this threshold might still > be allocated via @code{mmap}. > + > +If this parameter is not set, the default value is set as 128 KB and the Should this be 128KiB? > +threshold is adjusted dynamically to suit the allocation patterns of the > +program. If the parameter is set, the dynamic adjustment is disabled and the > +value is set statically to the input value. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value. > @comment TODO: @item M_MXFAST > @item M_PERTURB > If non-zero, memory blocks are filled with values depending on some > @@ -1128,16 +1141,34 @@ use of uninitialized or freed heap memory. Note that this option does not > guarantee that the freed block will have any specific values. It only > guarantees that the content the block had before it was freed will be > overwritten. > + > +The default value of this parameter is @code{0}. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value. > @item M_TOP_PAD > This parameter determines the amount of extra memory to obtain from the > system when a call to @code{sbrk} is required. It also specifies the > number of bytes to retain when shrinking the heap by calling @code{sbrk} > with a negative argument. This provides the necessary hysteresis in > heap size such that excessive amounts of system calls can be avoided. > + > +The default value of this parameter is @code{0}. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_TOP_PAD_} to the desired value. > @item M_TRIM_THRESHOLD > This is the minimum size (in bytes) of the top-most, releasable chunk > that will cause @code{sbrk} to be called with a negative argument in > order to return memory to the system. > + > +If this parameter is not set, the default value is set as 128 KB and the Likewise. > +threshold is adjusted dynamically to suit the allocation patterns of the > +program. If the parameter is set, the dynamic adjustment is disabled and the > +value is set statically to the provided input. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. > @end table > > @end deftypefun > -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-26 4:18 ` Carlos O'Donell @ 2016-10-26 7:58 ` Michael Kerrisk 2016-10-26 9:47 ` Siddhesh Poyarekar 2016-10-26 11:26 ` Rical Jasan 2 siblings, 0 replies; 23+ messages in thread From: Michael Kerrisk @ 2016-10-26 7:58 UTC (permalink / raw) To: Carlos O'Donell Cc: Siddhesh Poyarekar, libc-alpha, Michael Kerrisk-manpages I've nothing to add beyond Carlos's comments. I looked this over and it looks good to me. Cheers, Michael On Wed, Oct 26, 2016 at 6:15 AM, Carlos O'Donell <carlos@redhat.com> wrote: > On 10/24/2016 10:07 AM, Siddhesh Poyarekar wrote: >> The mallopt parameters manual does not mention the environment >> variables that can be used to set these parameters at program startup. >> Mention those environment variables for completeness. >> >> * manual/memory.texi: Add environment variable alternatives to >> setting mallopt parameters. > > Two nits. Otherwise OK. > >> --- >> manual/memory.texi | 31 +++++++++++++++++++++++++++++++ >> 1 file changed, 31 insertions(+) >> >> diff --git a/manual/memory.texi b/manual/memory.texi >> index 5383105..6f33455 100644 >> --- a/manual/memory.texi >> +++ b/manual/memory.texi >> @@ -1113,12 +1113,25 @@ choices for @var{param}, as defined in @file{malloc.h}, are: >> @item M_MMAP_MAX >> The maximum number of chunks to allocate with @code{mmap}. Setting this >> to zero disables all use of @code{mmap}. >> + >> +The default value of this parameter is @code{65536}. >> + >> +This parameter can also be set for the process at startup by setting the >> +environment variable @code{MALLOC_MMAP_MAX_} to the desired value. >> @item M_MMAP_THRESHOLD >> All chunks larger than this value are allocated outside the normal >> heap, using the @code{mmap} system call. This way it is guaranteed >> that the memory for these chunks can be returned to the system on >> @code{free}. Note that requests smaller than this threshold might still >> be allocated via @code{mmap}. >> + >> +If this parameter is not set, the default value is set as 128 KB and the > > Should this be 128KiB? > >> +threshold is adjusted dynamically to suit the allocation patterns of the >> +program. If the parameter is set, the dynamic adjustment is disabled and the >> +value is set statically to the input value. >> + >> +This parameter can also be set for the process at startup by setting the >> +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value. >> @comment TODO: @item M_MXFAST >> @item M_PERTURB >> If non-zero, memory blocks are filled with values depending on some >> @@ -1128,16 +1141,34 @@ use of uninitialized or freed heap memory. Note that this option does not >> guarantee that the freed block will have any specific values. It only >> guarantees that the content the block had before it was freed will be >> overwritten. >> + >> +The default value of this parameter is @code{0}. >> + >> +This parameter can also be set for the process at startup by setting the >> +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value. >> @item M_TOP_PAD >> This parameter determines the amount of extra memory to obtain from the >> system when a call to @code{sbrk} is required. It also specifies the >> number of bytes to retain when shrinking the heap by calling @code{sbrk} >> with a negative argument. This provides the necessary hysteresis in >> heap size such that excessive amounts of system calls can be avoided. >> + >> +The default value of this parameter is @code{0}. >> + >> +This parameter can also be set for the process at startup by setting the >> +environment variable @code{MALLOC_TOP_PAD_} to the desired value. >> @item M_TRIM_THRESHOLD >> This is the minimum size (in bytes) of the top-most, releasable chunk >> that will cause @code{sbrk} to be called with a negative argument in >> order to return memory to the system. >> + >> +If this parameter is not set, the default value is set as 128 KB and the > > Likewise. > >> +threshold is adjusted dynamically to suit the allocation patterns of the >> +program. If the parameter is set, the dynamic adjustment is disabled and the >> +value is set statically to the provided input. >> + >> +This parameter can also be set for the process at startup by setting the >> +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. >> @end table >> >> @end deftypefun >> > > > -- > Cheers, > Carlos. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-26 4:18 ` Carlos O'Donell 2016-10-26 7:58 ` Michael Kerrisk @ 2016-10-26 9:47 ` Siddhesh Poyarekar 2016-10-26 11:26 ` Rical Jasan 2 siblings, 0 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-26 9:47 UTC (permalink / raw) To: Carlos O'Donell, libc-alpha On Wednesday 26 October 2016 09:45 AM, Carlos O'Donell wrote: > On 10/24/2016 10:07 AM, Siddhesh Poyarekar wrote: >> The mallopt parameters manual does not mention the environment >> variables that can be used to set these parameters at program startup. >> Mention those environment variables for completeness. >> >> * manual/memory.texi: Add environment variable alternatives to >> setting mallopt parameters. > > Two nits. Otherwise OK. Thanks, pushed. Siddhesh ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-26 4:18 ` Carlos O'Donell 2016-10-26 7:58 ` Michael Kerrisk 2016-10-26 9:47 ` Siddhesh Poyarekar @ 2016-10-26 11:26 ` Rical Jasan 2016-10-26 11:30 ` Siddhesh Poyarekar 2 siblings, 1 reply; 23+ messages in thread From: Rical Jasan @ 2016-10-26 11:26 UTC (permalink / raw) To: Carlos O'Donell; +Cc: Siddhesh Poyarekar, libc-alpha On 10/25/2016 09:15 PM, Carlos O'Donell wrote: >> > +If this parameter is not set, the default value is set as 128 KB and the > Should this be 128KiB? I don't think binary computers count with 10 fingers. At any rate, we don't have any ibi's AFAICS: $ grep -Pi '\d[kmgt]i?b\b' manual/*.texi So maybe "128kB"? ;) Rical ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-26 11:26 ` Rical Jasan @ 2016-10-26 11:30 ` Siddhesh Poyarekar 2016-10-26 12:04 ` Rical Jasan 0 siblings, 1 reply; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-26 11:30 UTC (permalink / raw) To: Rical Jasan, Carlos O'Donell; +Cc: libc-alpha On Wednesday 26 October 2016 04:56 PM, Rical Jasan wrote: > On 10/25/2016 09:15 PM, Carlos O'Donell wrote: >>>> +If this parameter is not set, the default value is set as 128 KB and the >> Should this be 128KiB? > > I don't think binary computers count with 10 fingers. At any rate, we > don't have any ibi's AFAICS: > > $ grep -Pi '\d[kmgt]i?b\b' manual/*.texi It's the other way around AFAIK, i.e. KiB == 1024 and KB == 1000. Looks like there are references to kB in code comments, so I'm fine with changing it to that, or if you want to be absolutely clear, I could just make it 131072 bytes ;) Siddhesh ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-26 11:30 ` Siddhesh Poyarekar @ 2016-10-26 12:04 ` Rical Jasan 0 siblings, 0 replies; 23+ messages in thread From: Rical Jasan @ 2016-10-26 12:04 UTC (permalink / raw) To: siddhesh; +Cc: Carlos O'Donell, libc-alpha On 10/26/2016 04:30 AM, Siddhesh Poyarekar wrote: > On Wednesday 26 October 2016 04:56 PM, Rical Jasan wrote: >> On 10/25/2016 09:15 PM, Carlos O'Donell wrote: >>>>> +If this parameter is not set, the default value is set as 128 KB and the >>> Should this be 128KiB? >> >> I don't think binary computers count with 10 fingers. At any rate, we >> don't have any ibi's AFAICS: >> >> $ grep -Pi '\d[kmgt]i?b\b' manual/*.texi > > It's the other way around AFAIK, i.e. KiB == 1024 and KB == 1000. Looks > like there are references to kB in code comments, so I'm fine with > changing it to that, or if you want to be absolutely clear, I could just > make it 131072 bytes ;) As unambiguous as possible is best, so if you could fix the 2147483648- and 4294967296-byte references too, that'd be great. lol Rical ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables @ 2016-10-10 17:26 Siddhesh Poyarekar 2016-10-10 17:33 ` DJ Delorie ` (2 more replies) 0 siblings, 3 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-10 17:26 UTC (permalink / raw) To: libc-alpha The mallopt parameters manual does not mention the environment variables that can be used to set these parameters at program startup. Mention those environment variables for completeness. * manual/memory.texi: Add environment variable alternatives to setting mallopt parameters. --- manual/memory.texi | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/manual/memory.texi b/manual/memory.texi index 5383105..222f126 100644 --- a/manual/memory.texi +++ b/manual/memory.texi @@ -1113,12 +1113,18 @@ choices for @var{param}, as defined in @file{malloc.h}, are: @item M_MMAP_MAX The maximum number of chunks to allocate with @code{mmap}. Setting this to zero disables all use of @code{mmap}. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_MMAP_MAX_} to the desired value. @item M_MMAP_THRESHOLD All chunks larger than this value are allocated outside the normal heap, using the @code{mmap} system call. This way it is guaranteed that the memory for these chunks can be returned to the system on @code{free}. Note that requests smaller than this threshold might still be allocated via @code{mmap}. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value. @comment TODO: @item M_MXFAST @item M_PERTURB If non-zero, memory blocks are filled with values depending on some @@ -1128,16 +1134,25 @@ use of uninitialized or freed heap memory. Note that this option does not guarantee that the freed block will have any specific values. It only guarantees that the content the block had before it was freed will be overwritten. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value. @item M_TOP_PAD This parameter determines the amount of extra memory to obtain from the system when a call to @code{sbrk} is required. It also specifies the number of bytes to retain when shrinking the heap by calling @code{sbrk} with a negative argument. This provides the necessary hysteresis in heap size such that excessive amounts of system calls can be avoided. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_TOP_PAD_} to the desired value. @item M_TRIM_THRESHOLD This is the minimum size (in bytes) of the top-most, releasable chunk that will cause @code{sbrk} to be called with a negative argument in order to return memory to the system. + +This parameter can also be set for the process at startup by setting the +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. @end table @end deftypefun -- 2.7.4 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-10 17:26 Siddhesh Poyarekar @ 2016-10-10 17:33 ` DJ Delorie 2016-10-10 17:42 ` Siddhesh Poyarekar 2016-10-11 6:20 ` Michael Kerrisk 2016-10-11 19:35 ` Kalle Olavi Niemitalo 2016-10-17 16:13 ` Carlos O'Donell 2 siblings, 2 replies; 23+ messages in thread From: DJ Delorie @ 2016-10-10 17:33 UTC (permalink / raw) To: Siddhesh Poyarekar; +Cc: libc-alpha These variables all end with "_", most likely the original intention is that they are not official and may be removed or changed at any time. By making them official, we lock in that ABI. Is that your intention? (likewise your other patch). ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-10 17:33 ` DJ Delorie @ 2016-10-10 17:42 ` Siddhesh Poyarekar 2016-10-11 6:20 ` Michael Kerrisk 1 sibling, 0 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-10 17:42 UTC (permalink / raw) To: DJ Delorie; +Cc: libc-alpha On Monday 10 October 2016 11:03 PM, DJ Delorie wrote: > These variables all end with "_", most likely the original intention is > that they are not official and may be removed or changed at any time. > > By making them official, we lock in that ABI. Is that your intention? > > (likewise your other patch). I don't see the point in not doing so because they're too widely in use for us to take them back anyway. Siddhesh ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-10 17:33 ` DJ Delorie 2016-10-10 17:42 ` Siddhesh Poyarekar @ 2016-10-11 6:20 ` Michael Kerrisk 2016-10-11 18:19 ` DJ Delorie 2016-10-17 19:40 ` Carlos O'Donell 1 sibling, 2 replies; 23+ messages in thread From: Michael Kerrisk @ 2016-10-11 6:20 UTC (permalink / raw) To: DJ Delorie; +Cc: Siddhesh Poyarekar, libc-alpha On Mon, Oct 10, 2016 at 7:33 PM, DJ Delorie <dj@redhat.com> wrote: > > These variables all end with "_", most likely the original intention is > that they are not official and may be removed or changed at any time. > > By making them official, we lock in that ABI. Is that your intention? I think the notion that because one does not document something, it's not an official part of the ABI is at best highly dubious. Especially when so many "official" parts of the ABI are also not documented. The only way that documentation might be able to help in such situations is where pieces are clearly and loudly documented right from the beginning, in the official documentation, as "not supported, may disappear at any moment in the future, use at your own risk", but even then people are likely to ignore the documentation or be unaware of it. And in any case, these environment vars have long been documented in various "unofficial" places including (since 2012) http://man7.org/linux/man-pages/man3/mallopt.3.html Cheers, Michael ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-11 6:20 ` Michael Kerrisk @ 2016-10-11 18:19 ` DJ Delorie 2016-10-11 19:03 ` Michael Kerrisk (man-pages) 2016-10-12 11:57 ` Siddhesh Poyarekar 2016-10-17 19:40 ` Carlos O'Donell 1 sibling, 2 replies; 23+ messages in thread From: DJ Delorie @ 2016-10-11 18:19 UTC (permalink / raw) To: Michael Kerrisk; +Cc: siddhesh, libc-alpha Michael Kerrisk <mtk.manpages@gmail.com> writes: > I think the notion that because one does not document something, it's > not an official part of the ABI is at best highly dubious. I wasn't here for the origin of those variables, I'm just thinking that we use a leading underscore elsewhere to mean "not official" so the trailing underscore here might have similar intent. FTR I have no problem with documenting unsupported behavior, I just want to make sure we understand how this patch affects the officialness of this ABI. > The only way that documentation might be able to help in such > situations is where pieces are clearly and loudly documented right > from the beginning, in the official documentation, as "not supported, > may disappear at any moment in the future, use at your own risk", but > even then people are likely to ignore the documentation or be unaware > of it. That would be fine too. My concern is: if we intended for those variables to be unofficial before, do we want to (1) make them official now, or (2) be careful not to *accidentally* make them official? And if we decide they're official (and/or always have been), should we add in variants without underscores to be the official ones? So I guess the next step is to have someone in authority (or consensus?) decide if those variables are "official" or not, and if they should become so if not. Based on that we may want to tweak the documentation patch, or the code. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-11 18:19 ` DJ Delorie @ 2016-10-11 19:03 ` Michael Kerrisk (man-pages) 2016-10-12 11:57 ` Siddhesh Poyarekar 1 sibling, 0 replies; 23+ messages in thread From: Michael Kerrisk (man-pages) @ 2016-10-11 19:03 UTC (permalink / raw) To: DJ Delorie; +Cc: mtk.manpages, siddhesh, libc-alpha On 10/11/2016 08:18 PM, DJ Delorie wrote: > > Michael Kerrisk <mtk.manpages@gmail.com> writes: >> I think the notion that because one does not document something, it's >> not an official part of the ABI is at best highly dubious. > > I wasn't here for the origin of those variables, I'm just thinking that > we use a leading underscore elsewhere to mean "not official" so the > trailing underscore here might have similar intent. > > FTR I have no problem with documenting unsupported behavior, I just want > to make sure we understand how this patch affects the officialness of > this ABI. My thought is: not very much. The ABI is out there and is well known. There's not really anyway to argue to users that it is not "official", no matter what the original developers might once have thought about these interfaces. Cheers, Michael -- Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-11 18:19 ` DJ Delorie 2016-10-11 19:03 ` Michael Kerrisk (man-pages) @ 2016-10-12 11:57 ` Siddhesh Poyarekar 1 sibling, 0 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-12 11:57 UTC (permalink / raw) To: DJ Delorie, Michael Kerrisk; +Cc: libc-alpha On Tuesday 11 October 2016 11:48 PM, DJ Delorie wrote: > I wasn't here for the origin of those variables, I'm just thinking that > we use a leading underscore elsewhere to mean "not official" so the > trailing underscore here might have similar intent. I agree, that may have been the intent. However what happened afterwards (i.e. their widespread use and the lack of communication to the contrary from the then maintainers) have invalidated that intent IMO. What's more, other important sources (the man pages, Red Hat product documentation) document these environment variables and that gives them as much a sense of credibility as anything else. > And if we decide they're official (and/or always have been), should we > add in variants without underscores to be the official ones? I'll be adding in tunables, which would become the 'official' versions of these envvars. Tunables come with the disclaimer that they may disappear/reappear in releases, but I don't think we will do that for the malloc tunables in practice. > So I guess the next step is to have someone in authority (or consensus?) > decide if those variables are "official" or not, and if they should > become so if not. It is usually consensus. Siddhesh ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-11 6:20 ` Michael Kerrisk 2016-10-11 18:19 ` DJ Delorie @ 2016-10-17 19:40 ` Carlos O'Donell 1 sibling, 0 replies; 23+ messages in thread From: Carlos O'Donell @ 2016-10-17 19:40 UTC (permalink / raw) To: Michael Kerrisk, DJ Delorie; +Cc: Siddhesh Poyarekar, libc-alpha On 10/11/2016 02:20 AM, Michael Kerrisk wrote: > On Mon, Oct 10, 2016 at 7:33 PM, DJ Delorie <dj@redhat.com> wrote: >> >> These variables all end with "_", most likely the original intention is >> that they are not official and may be removed or changed at any time. >> >> By making them official, we lock in that ABI. Is that your intention? > > I think the notion that because one does not document something, it's > not an official part of the ABI is at best highly dubious. Especially > when so many "official" parts of the ABI are also not documented. > > The only way that documentation might be able to help in such > situations is where pieces are clearly and loudly documented right > from the beginning, in the official documentation, as "not supported, > may disappear at any moment in the future, use at your own risk", but > even then people are likely to ignore the documentation or be unaware > of it. > > And in any case, these environment vars have long been documented in > various "unofficial" places including (since 2012) > http://man7.org/linux/man-pages/man3/mallopt.3.html The things that DJ is worried about are the arena ones e.g. M_ARENA_MAX, and M_ARENA_TEST and their env vars. In 2013 I brought up the discussion about ABI implications: https://sourceware.org/ml/libc-alpha/2013-03/msg00376.html Only Siddhesh and I commented, and we both agreed it was a forgone conclusion that these were stable parts of the ABI/API. In 2015 I documented them in the linux man pages project: https://www.sourceware.org/ml/libc-alpha/2015-08/msg00991.html ~~~ Consensus among Siddhesh and myself was that they should be public, and in fact they were already in the public header. Therefore there may already be applications uses these constants and expecting them to work. At best we could limit mallopt's acceptance of the options, but that seems like a bad solution that could lead to unexpected behaviour for user applications. A quick google search shows that there are packages relying on these constants to tune the glibc malloc implementation. ~~~ They are part of the ABI and public, and should be documented in the manual as Siddhesh's patches do. We should however be very very circumspect about adding more of these mallopt tunables without first seeing that such tunables are stable and long-term useful as generic tunables (via the tunables interface). So go review Siddhesh's patches on tunables :-) -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-10 17:26 Siddhesh Poyarekar 2016-10-10 17:33 ` DJ Delorie @ 2016-10-11 19:35 ` Kalle Olavi Niemitalo 2016-10-17 16:13 ` Carlos O'Donell 2 siblings, 0 replies; 23+ messages in thread From: Kalle Olavi Niemitalo @ 2016-10-11 19:35 UTC (permalink / raw) To: libc-alpha Siddhesh Poyarekar <siddhesh@sourceware.org> writes: > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_MAX_} to the desired value. startup.texi has e.g. "@cindex @code{TZ} environment variable". Could add similar index entries for MALLOC_MMAP_MAX_ and others. Inconsistently, @vindex is used for the PWD, LANGUAGE, and COREFILE environment variables. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-10 17:26 Siddhesh Poyarekar 2016-10-10 17:33 ` DJ Delorie 2016-10-11 19:35 ` Kalle Olavi Niemitalo @ 2016-10-17 16:13 ` Carlos O'Donell 2016-10-17 16:16 ` Siddhesh Poyarekar 2 siblings, 1 reply; 23+ messages in thread From: Carlos O'Donell @ 2016-10-17 16:13 UTC (permalink / raw) To: Siddhesh Poyarekar, libc-alpha On 10/10/2016 01:26 PM, Siddhesh Poyarekar wrote: > The mallopt parameters manual does not mention the environment > variables that can be used to set these parameters at program startup. > Mention those environment variables for completeness. Please don't get upset with me :-) I'm asking you to document the default for these env var tunables. I think it's valid for users to want to know what value they would have had if they _didn't_ set the environment variable. Throwing our hands up in the air and saying "it's implementation defined" isn't very useful IMO. > * manual/memory.texi: Add environment variable alternatives to > setting mallopt parameters. > --- > manual/memory.texi | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/manual/memory.texi b/manual/memory.texi > index 5383105..222f126 100644 > --- a/manual/memory.texi > +++ b/manual/memory.texi > @@ -1113,12 +1113,18 @@ choices for @var{param}, as defined in @file{malloc.h}, are: > @item M_MMAP_MAX > The maximum number of chunks to allocate with @code{mmap}. Setting this > to zero disables all use of @code{mmap}. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_MAX_} to the desired value. What's the default? > @item M_MMAP_THRESHOLD > All chunks larger than this value are allocated outside the normal > heap, using the @code{mmap} system call. This way it is guaranteed > that the memory for these chunks can be returned to the system on > @code{free}. Note that requests smaller than this threshold might still > be allocated via @code{mmap}. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value. What's the default? > @comment TODO: @item M_MXFAST > @item M_PERTURB > If non-zero, memory blocks are filled with values depending on some > @@ -1128,16 +1134,25 @@ use of uninitialized or freed heap memory. Note that this option does not > guarantee that the freed block will have any specific values. It only > guarantees that the content the block had before it was freed will be > overwritten. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value. What is the default? > @item M_TOP_PAD > This parameter determines the amount of extra memory to obtain from the > system when a call to @code{sbrk} is required. It also specifies the > number of bytes to retain when shrinking the heap by calling @code{sbrk} > with a negative argument. This provides the necessary hysteresis in > heap size such that excessive amounts of system calls can be avoided. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_TOP_PAD_} to the desired value. What's the default? > @item M_TRIM_THRESHOLD > This is the minimum size (in bytes) of the top-most, releasable chunk > that will cause @code{sbrk} to be called with a negative argument in > order to return memory to the system. > + > +This parameter can also be set for the process at startup by setting the > +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value. What's the default? Note: There is a limit even with dynamic trim threshold on. This disables the dynamic trim threshold, and that's a very important thing to mention. > @end table > > @end deftypefun > -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables 2016-10-17 16:13 ` Carlos O'Donell @ 2016-10-17 16:16 ` Siddhesh Poyarekar 0 siblings, 0 replies; 23+ messages in thread From: Siddhesh Poyarekar @ 2016-10-17 16:16 UTC (permalink / raw) To: Carlos O'Donell, libc-alpha On Monday 17 October 2016 09:43 PM, Carlos O'Donell wrote: > On 10/10/2016 01:26 PM, Siddhesh Poyarekar wrote: >> The mallopt parameters manual does not mention the environment >> variables that can be used to set these parameters at program startup. >> Mention those environment variables for completeness. > > Please don't get upset with me :-) Grrr... > I'm asking you to document the default for these env var tunables. > > I think it's valid for users to want to know what value they would have > had if they _didn't_ set the environment variable. > > Throwing our hands up in the air and saying "it's implementation defined" > isn't very useful IMO. No I agree that is a good point. I'll work on it based on your comments on the MALLOC_ARENA_* doc patch, i.e. first describe the GNU allocator and then use that information to explain the envvars/mallopt params and also add in defaults. Siddhesh ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2016-10-26 12:04 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-10-24 14:08 [PATCH 0/2] Malloc manual cleanups Siddhesh Poyarekar 2016-10-24 14:08 ` [PATCH 2/2] Document the M_ARENA_* mallopt parameters Siddhesh Poyarekar 2016-10-25 7:51 ` Rical Jasan 2016-10-25 21:55 ` DJ Delorie 2016-10-26 5:42 ` Siddhesh Poyarekar 2016-10-24 14:08 ` [PATCH 1/2] Add note on MALLOC_MMAP_* environment variables Siddhesh Poyarekar 2016-10-26 4:18 ` Carlos O'Donell 2016-10-26 7:58 ` Michael Kerrisk 2016-10-26 9:47 ` Siddhesh Poyarekar 2016-10-26 11:26 ` Rical Jasan 2016-10-26 11:30 ` Siddhesh Poyarekar 2016-10-26 12:04 ` Rical Jasan -- strict thread matches above, loose matches on Subject: below -- 2016-10-10 17:26 Siddhesh Poyarekar 2016-10-10 17:33 ` DJ Delorie 2016-10-10 17:42 ` Siddhesh Poyarekar 2016-10-11 6:20 ` Michael Kerrisk 2016-10-11 18:19 ` DJ Delorie 2016-10-11 19:03 ` Michael Kerrisk (man-pages) 2016-10-12 11:57 ` Siddhesh Poyarekar 2016-10-17 19:40 ` Carlos O'Donell 2016-10-11 19:35 ` Kalle Olavi Niemitalo 2016-10-17 16:13 ` Carlos O'Donell 2016-10-17 16:16 ` Siddhesh Poyarekar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).