[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[heuler at infosim dot net]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

Marius Heuler <heuler at infosim dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
                 CC|                            |heuler at infosim dot net
         Resolution|WONTFIX                     |

--- Comment #8 from Marius Heuler <heuler at infosim dot net> 2011-08-27 21:45:04 UTC ---
We have exactly the same problem with the current implementation of malloc.

The suggested solutions by Ulrich using M_ARENA_MAX does not work since the
check for number of arenas is not thread safe. In fact the limit is not working
for heay threading applications where that would be needed! 

Since the number of cores and usage of threads will increase strongly there
should be a solution for that kind of applications! If the arena limit would
work as described we would have no problem.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[rich at testardi dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

--- Comment #9 from Rich Testardi <rich at testardi dot com> 2011-08-27 22:02:03 UTC ---
Hi,

We ended up building our own memory allocator -- it's faster and more efficient
than glibc, and it works equally fast with threads and wihout.

We used the "small block allocator" concept from HP-UX where we only allocate
huge (32MB) allocations from the system (after setting M_MMAP_THRESHOLD
suitably small).

We then carve out large *naturally aligned* 1MB blocks from the huge allocation
(accepting 3% waste, since the allocation was page alogned to begin with, not
naturally aligned).

And we carve each one of those large blocks into small fixed size buckets
(which are fractional powers of 2 -- like 16 bytes, 20, 24, 28, 32, 40, 48, 56,
64, 80, etc.).

Then we put the aligned addresses into a very fast hash and have a linked list
for each bucket size.

This means our allocate routine is just a lock, linked list remove, unlock, on
average, and our free routine is just a hash lookup, lock, linked list insert,
unlock on average.

The trick here is that from any address being freed, you can get back to the
naturally aligned 1MB block that contains it with just a pointer mask, and from
there you can get the allocation's size as well as the head of the linked list
of free entries to which it should be returned...

-- Rich

  ----- Original Message ----- 
  From: heuler at infosim dot net 
  To: rich@testardi.com 
  Sent: Saturday, August 27, 2011 3:45 PM
  Subject: [Bug libc/11261] malloc uses excessive memory for multi-threaded
applications


  http://sourceware.org/bugzilla/show_bug.cgi?id=11261

  Marius Heuler <heuler at infosim dot net> changed:

             What    |Removed                     |Added
  ----------------------------------------------------------------------------
               Status|RESOLVED                    |REOPENED
                   CC|                            |heuler at infosim dot net
           Resolution|WONTFIX                     |

  --- Comment #8 from Marius Heuler <heuler at infosim dot net> 2011-08-27
21:45:04 UTC ---
  We have exactly the same problem with the current implementation of malloc.

  The suggested solutions by Ulrich using M_ARENA_MAX does not work since the
  check for number of arenas is not thread safe. In fact the limit is not
working
  for heay threading applications where that would be needed! 

  Since the number of cores and usage of threads will increase strongly there
  should be a solution for that kind of applications! If the arena limit would
  work as described we would have no problem.

  -- 
  Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
  ------- You are receiving this mail because: -------
  You reported the bug.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[heuler at infosim dot net]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

--- Comment #10 from Marius Heuler <heuler at infosim dot net> 2011-09-02 07:38:51 UTC ---
Created attachment 5917
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5917
Memory consumption with glibc malloc and jeMalloc (straight line).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[heuler at infosim dot net]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

--- Comment #11 from Marius Heuler <heuler at infosim dot net> 2011-09-02 07:44:31 UTC ---
Comment on attachment 5917
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5917
Memory consumption with glibc malloc and jeMalloc (straight line).

We now changed to another malloc implementation: jeMalloc
(http://www.canonware.com/jemalloc/) which is a magnitude superior to the glibc
malloc. A similar implementation is also used in *BSD variants!
Linux/glibc should really improve their malloc since the current implementation
is not sufficient for large applications. 
Why can't this implemenetion be used inside glibc? Is it GPL <-> BSD license
problem?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |WORKSFORME

--- Comment #12 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-09-11 15:46:13 UTC ---
Stop reopening.  There is a solution for people who are stupid enough to create
too many threads.  No implementation will be perfect for everyone.  The glibc
implementation is tuned for reasonable programs and will run much faster than
any other I tested.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[rich at testardi dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

--- Comment #13 from Rich Testardi <rich at testardi dot com> 2011-09-11 21:31:37 UTC ---
Let's all not take things so personally -- nobody here is stupid (and I'm sure
some folks here are a *lot* smarter than other folks 
give them credit for)...

There are lots of reasons to create a half dozen threads and that's all it
takes to make the glibc version perform absolutely 
horribly.

(And there can be no non-objective measurement that won't show my version of
malloc is faster than yours -- so this has been a win 
all around for us, thanks...)

If you're not interested in improving glibc, you can just say so.

But stop name calling when you feel threatened -- my 5 year old daughter has
already outgrown that.

-- Rich


-----Original Message----- 
From: drepper.fsp at gmail dot com
Sent: Sunday, September 11, 2011 9:46 AM
To: rich@testardi.com
Subject: [Bug libc/11261] malloc uses excessive memory for multi-threaded
applications

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |WORKSFORME

--- Comment #12 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-09-11
15:46:13 UTC ---
Stop reopening.  There is a solution for people who are stupid enough to create
too many threads.  No implementation will be perfect for everyone.  The glibc
implementation is tuned for reasonable programs and will run much faster than
any other I tested.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[zhannk at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

zhannk at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
                 CC|                            |zhannk at gmail dot com
         Resolution|WORKSFORME                  |
     Ever Confirmed|1                           |0

--- Comment #14 from zhannk at gmail dot com 2012-07-29 10:09:47 UTC ---
Ulrich Drepper, this huge virtual memory allocator could be a potential trouble
maker on Linux6 with 64bit JVM. 
There is already one document on hadoop regarding to this issue, while their
solution by setting MALLOC_ARENA_MAX=4 has no effect. we still found JVM with
30G virtual memory reported. 
https://issues.apache.org/jira/browse/HADOOP-7154

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|drepper.fsp at gmail dot    |unassigned at sourceware
                   |com                         |dot org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[carlos at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
                 CC|                            |carlos at redhat dot com
         Resolution|                            |FIXED

--- Comment #15 from Carlos O'Donell <carlos at redhat dot com> 2013-03-14 19:03:05 UTC ---
This should have been fixed by the following commit:

commit 41b81892f11fe1353123e892158b53de73863d62
Author: Ulrich Drepper <drepper@gmail.com>
Date:   Tue Jan 31 14:42:34 2012 -0500

    Handle ARENA_TEST correctly

I have verified that using `mallopt (M_ARENA_MAX, 1)' that the limit of memory
is bounded by the single arena.

creating 10 threads
allowing threads to contend to create preferred arenas
display preferred arenas
Arena 0:
system bytes     =     135168
in use bytes     =       2880
Total (incl. mmap):
system bytes     =     135168
in use bytes     =       2880
max mmap regions =          0
max mmap bytes   =          0
allowing threads to allocate 100MB each, sequentially in turn
thread 0 alloc 100MB
thread 0 free 100MB-20kB
thread 4 alloc 100MB
thread 4 free 100MB-20kB
thread 9 alloc 100MB
thread 9 free 100MB-20kB
thread 5 alloc 100MB
thread 5 free 100MB-20kB
thread 2 alloc 100MB
thread 2 free 100MB-20kB
thread 7 alloc 100MB
thread 7 free 100MB-20kB
thread 1 alloc 100MB
thread 1 free 100MB-20kB
thread 8 alloc 100MB
thread 8 free 100MB-20kB
thread 6 alloc 100MB
thread 6 free 100MB-20kB
thread 3 alloc 100MB
thread 3 free 100MB-20kB
Arena 0:
system bytes     =  100392960
in use bytes     =     201472
Total (incl. mmap):
system bytes     =  100392960
in use bytes     =     201472
max mmap regions =          0
max mmap bytes   =          0

Therefore the solution to a program with lots of threads is to limit the arenas
as a trade-off for memory.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[neleai at seznam dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Ondrej Bilka <neleai at seznam dot cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
   Last reconfirmed|                            |2013-12-12
                 CC|                            |neleai at seznam dot cz
         Resolution|FIXED                       |---
     Ever confirmed|0                           |1

--- Comment #16 from Ondrej Bilka <neleai at seznam dot cz> ---
> Therefore the solution to a program with lots of threads is to limit the arenas > as a trade-off for memory.

That is a bandaid not a solution. Still there is no memory returned to system
when one first does allocations and then allocates auxiliary memory like

void *calculate ()
{
  void **ary = malloc (1000000 * sizeof (void *))
  for (i = 0; i < 1000000; i++) ary[i] = malloc (100);
  for (i = 0; i <  999999; i++) free (ary [i]);
  return ary[999999];
}

When one acknowledges a bug a solution is relatively simple. Add a flag
UNMAPPED for chunks which means that all pages completely contained in chunk
were zeroed by madvise(s, n, MADV_DONTNEED).

You keep track of memory used and system and when their ratio is bigger than
two you make chunks starting from largest ones UNMAPPED to decrease system
charge.

This deals with RSS problem. A virtual space usage could still be excesive but
that is smaller problem.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[siddhesh at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Siddhesh Poyarekar <siddhesh at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |siddhesh at redhat dot com

--- Comment #17 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
(In reply to Ondrej Bilka from comment #16)
> > Therefore the solution to a program with lots of threads is to limit the arenas > as a trade-off for memory.
> 
> That is a bandaid not a solution. Still there is no memory returned to
> system when one first does allocations and then allocates auxiliary memory
> like

You have not understood the bug report.

> void *calculate ()
> {
>   void **ary = malloc (1000000 * sizeof (void *))
>   for (i = 0; i < 1000000; i++) ary[i] = malloc (100);
>   for (i = 0; i <  999999; i++) free (ary [i]);
>   return ary[999999];
> }

This is a different problem from the current bug report, which is about too
many arenas getting created resulting in excessive address space usage
 and the MALLOC_ARENA_* variables not working to limit them.  Memory holes not
being freed has nothing to do with it. 

> When one acknowledges a bug a solution is relatively simple. Add a flag
> UNMAPPED for chunks which means that all pages completely contained in chunk
> were zeroed by madvise(s, n, MADV_DONTNEED).
> 
> You keep track of memory used and system and when their ratio is bigger than
> two you make chunks starting from largest ones UNMAPPED to decrease system
> charge.
> 
> This deals with RSS problem. A virtual space usage could still be excesive
> but that is smaller problem.

The problem you've described is different and I'm sure there's a bug report
open for it too.  madvise is not sufficient to free up commit charge; there's a
mail thread on libc-alpha that discusses this problem that you can search for
and read up on.  I think vm.overcommit_memory is one of the keywords to look
for.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[neleai at seznam dot cz]

http://sourceware.org/bugzilla/show_bug.cgi?id=11261

--- Comment #18 from Ondrej Bilka <neleai at seznam dot cz> ---
On Thu, Dec 12, 2013 at 03:31:58AM +0000, siddhesh at redhat dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=11261
> 
> Siddhesh Poyarekar <siddhesh at redhat dot com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |siddhesh at redhat dot com
> 
> --- Comment #17 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
> (In reply to Ondrej Bilka from comment #16)
> > > Therefore the solution to a program with lots of threads is to limit the arenas > as a trade-off for memory.
> > 
> > That is a bandaid not a solution. Still there is no memory returned to
> > system when one first does allocations and then allocates auxiliary memory
> > like
> 
> You have not understood the bug report.
>
When you read discussion more carefully there are following posts where
this problem is mentioned:


Ulrich Drepper:

 You don't understand the difference between address space and allocated
 memory.

Rich Testardi:

Actually, I totally understand the difference and that is why I mentioned the 
fragmentation of memory...  When each arena has just a few straggling 
allocations, the maximum *committed* RAM required for the program's *working 
set* using the thread-preferred arena model is, in fact, N times that required 
for a traditional model, where N is the number of threads.  This shows up in 
real-world thrashing that could actually be avoided.  Basically, if the 
program is doing small allocations, a small percentage of stragglers can pin 
the entire allocated space -- and the allocated space is, in fact, much larger 
than it needs to be (and larger than it is in other OS's).  But thank you for

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/11261] malloc uses excessive memory for multi-threaded applications

[siddhesh at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

--- Comment #19 from Siddhesh Poyarekar <siddhesh at redhat dot com> ---
(In reply to Ondrej Bilka from comment #18)
> When you read discussion more carefully there are following posts where
> this problem is mentioned:
> 
> 
> Ulrich Drepper:
> 
>  You don't understand the difference between address space and allocated
>  memory.
> 
> Rich Testardi:
> 
> Actually, I totally understand the difference and that is why I mentioned
> the 
> fragmentation of memory...  When each arena has just a few straggling 
> allocations, the maximum *committed* RAM required for the program's *working 
> set* using the thread-preferred arena model is, in fact, N times that
> required 
> for a traditional model, where N is the number of threads.  This shows up in 
> real-world thrashing that could actually be avoided.  Basically, if the 
> program is doing small allocations, a small percentage of stragglers can pin 
> the entire allocated space -- and the allocated space is, in fact, much
> larger 
> than it needs to be (and larger than it is in other OS's).  But thank you for

Right, but most comments on the bug report (and the resolution) are in the
context of malloc creating too many arenas and the switches not working. 
Single allocations blocking an entire free space is not a multi-threaded
problem - it occurs on single-threads too and is only compounded with multiple
arenas.  I'd suggest working with a fresh bug report or an open bug report that
describes this problem exactly (which I'm pretty sure there should be).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/11261] malloc uses excessive memory for multi-threaded applications

[jsm28 at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|libc                        |malloc

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/11261] malloc uses excessive memory for multi-threaded applications

[jackie.rosen at hushmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Jackie Rosen <jackie.rosen at hushmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jackie.rosen at hushmail dot com

--- Comment #20 from Jackie Rosen <jackie.rosen at hushmail dot com> ---
*** Bug 260998 has been marked as a duplicate of this bug. ***
Seen from the domain http://volichat.com
Page where seen: http://volichat.com/adult-chat-rooms
Marked for reference. Resolved as fixed @bugzilla.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/11261] malloc uses excessive memory for multi-threaded applications

[schwab at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Andreas Schwab <schwab at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|jackie.rosen at hushmail dot com   |

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/11261] malloc uses excessive memory for multi-threaded applications

[schwab at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Andreas Schwab <schwab at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|jackie.rosen at hushmail dot com   |

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/11261] malloc uses excessive memory for multi-threaded applications

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug malloc/11261] malloc uses excessive memory for multi-threaded applications

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=11261

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #21 from Carlos O'Donell <carlos at redhat dot com> ---
I'm marking this fixed, since the tunnables that limit arena creation are
fixed. You can limit the number of arenas in your application at the cost of
thread contention during allocation (increased malloc latency). This does
however limit the total VA usage. This is particularly true of 32-bit
applications running close to the 32-bit VA limit.

-- 
You are receiving this mail because:
You are on the CC list for the bug.