public inbox for glibc-bugs@sourceware.org
help / color / mirror / Atom feed
* [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage
@ 2009-12-02 18:31 tom at atoptech dot com
2010-04-05 5:08 ` [Bug libc/11044] " drepper at redhat dot com
` (4 more replies)
0 siblings, 5 replies; 9+ messages in thread
From: tom at atoptech dot com @ 2009-12-02 18:31 UTC (permalink / raw)
To: glibc-bugs
We recently compiled and ran our application on Centos 5 which has glibc 2.5. We
found many of our benchmarks and regression runs resulted in 50-100% increase in
memory usage. We eventually pin-pointed the problem to malloc and after
examining the 2.5 malloc source code we found that malloc now adjusts the mmap
threshold dynamically. For our application, the dynamic adjustment does not
work. The end result is more fragmentation and many large malloc requests
results in more and more memory allocated.
Our application is Place-And-Route for backend chip design. It is very common
for an innovation of our application to run for several days using up to 32GB of
memory (depending on the size of the chip, i.e., multi-million gates). During an
invocation of our application many different algorithms are executed with
different memory usage patterns (many very large memory allocations and frees.)
We relied on the old malloc behavior to mmap these requests, because this tended
to reduce the fragmentation.
My guess is that this malloc change is probably just hitting people in userland,
because the Linux distributions use older versions of glibc. Again Centos 5, is
using 2.5.
I found some discussion of this change via google. And there was some
questioning as to whether this change was valid. Again, in our application the
memory usage pattern varies greatly and unless you have a good statistical model
of the memory usage pattern, it is hight unlikely to dynamically adjust the mmap
threshold and get it right.
Because we have the ability to turn off the dynamic threshold (mallopt), we have
a work around. But I thought it was best to give feedback on how this changes
was impacting our application.
Tom Geocaris
Atoptech
--
Summary: malloc dynamic mmap threshold causes 50%-100% increase
in memory usage
Product: glibc
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: libc
AssignedTo: drepper at redhat dot com
ReportedBy: tom at atoptech dot com
CC: glibc-bugs at sources dot redhat dot com
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
@ 2010-04-05 5:08 ` drepper at redhat dot com
2010-04-05 16:10 ` tom at atoptech dot com
` (3 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: drepper at redhat dot com @ 2010-04-05 5:08 UTC (permalink / raw)
To: glibc-bugs
------- Additional Comments From drepper at redhat dot com 2010-04-05 05:08 -------
No malloc can work perfectly in all situations. This is why mallopt exist. So,
use it.
Aside, you are not reporting anything about current releases.
--
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |WONTFIX
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
2010-04-05 5:08 ` [Bug libc/11044] " drepper at redhat dot com
@ 2010-04-05 16:10 ` tom at atoptech dot com
2010-04-05 18:32 ` drepper at redhat dot com
` (2 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: tom at atoptech dot com @ 2010-04-05 16:10 UTC (permalink / raw)
To: glibc-bugs
------- Additional Comments From tom at atoptech dot com 2010-04-05 16:10 -------
Ulrich,
> No malloc can work perfectly in all situations....
That is the point. This change had no statistical basis proving that it improved
the malloc behavior on average (at least I'm not aware of any papers on the
subject).
> Aside, you are not reporting anything about current releases.
Correct me in I'm wrong, but this change is in the current release.
Regards,
Tom Geocaris
--
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
2010-04-05 5:08 ` [Bug libc/11044] " drepper at redhat dot com
2010-04-05 16:10 ` tom at atoptech dot com
@ 2010-04-05 18:32 ` drepper at redhat dot com
2010-04-05 18:35 ` drepper at redhat dot com
2010-04-07 17:09 ` tom at atoptech dot com
4 siblings, 0 replies; 9+ messages in thread
From: drepper at redhat dot com @ 2010-04-05 18:32 UTC (permalink / raw)
To: glibc-bugs
------- Additional Comments From drepper at redhat dot com 2010-04-05 18:31 -------
(In reply to comment #2)
> That is the point. This change had no statistical basis proving that it improved
> the malloc behavior on average (at least I'm not aware of any papers on the
> subject).
>
> > Aside, you are not reporting anything about current releases.
>
> Correct me in I'm wrong, but this change is in the current release.
>
> Regards,
>
> Tom Geocaris
--
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
` (2 preceding siblings ...)
2010-04-05 18:32 ` drepper at redhat dot com
@ 2010-04-05 18:35 ` drepper at redhat dot com
2010-04-07 17:09 ` tom at atoptech dot com
4 siblings, 0 replies; 9+ messages in thread
From: drepper at redhat dot com @ 2010-04-05 18:35 UTC (permalink / raw)
To: glibc-bugs
------- Additional Comments From drepper at redhat dot com 2010-04-05 18:34 -------
(In reply to comment #2)
> That is the point. This change had no statistical basis proving that it improved
> the malloc behavior on average (at least I'm not aware of any papers on the
> subject).
Appropriate measurements were made. We don't require "papers" for every change
made. Go to academia if you you want that.
The changes showed improvements and there are ways to turn the changes off. You
cannot possibly ask for more.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
` (3 preceding siblings ...)
2010-04-05 18:35 ` drepper at redhat dot com
@ 2010-04-07 17:09 ` tom at atoptech dot com
4 siblings, 0 replies; 9+ messages in thread
From: tom at atoptech dot com @ 2010-04-07 17:09 UTC (permalink / raw)
To: glibc-bugs
------- Additional Comments From tom at atoptech dot com 2010-04-07 17:09 -------
Ulrich,
I filed this bug to show, at least for our application, this change did not
result in improved performance. To the contrary, this change increase memory
fragmentation and in some case, memory usage when from 32GB to 64GB when running
our application. Our application ran fine on RedHat release 3 and 4. When we
ported to release 5, memory usage increased with out reason. From our standpoint
we would like the compute platform to be stable from one release to the next.
All I wanted to point out was that after reading the patch comments for this
change, such as,
+ The threshold goes up in value when the application frees memory that was
+ allocated with the mmap allocator. The idea is that once the application
+ starts freeing memory of a certain size, it's highly probable that this is
+ a size the application uses for transient allocations. This estimator
+ is there to satisfy the new third requirement.
seem to me; weakly justifiable. Some applications may exhibit this behavior,
however, our application does not. And if one going to try and dynamically
adjust the mmap threshold, one should keep adjusting the mmap threshold over the
lifetime of the process and not clamp it so early, i.e., based upon the first
free...
Regards,
Tom Geocaris
--
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
[not found] <bug-11044-131@http.sourceware.org/bugzilla/>
2014-02-16 17:43 ` jackie.rosen at hushmail dot com
2014-05-28 19:46 ` schwab at sourceware dot org
@ 2014-06-30 20:35 ` fweimer at redhat dot com
2 siblings, 0 replies; 9+ messages in thread
From: fweimer at redhat dot com @ 2014-06-30 20:35 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=11044
Florian Weimer <fweimer at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags| |security-
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
[not found] <bug-11044-131@http.sourceware.org/bugzilla/>
2014-02-16 17:43 ` jackie.rosen at hushmail dot com
@ 2014-05-28 19:46 ` schwab at sourceware dot org
2014-06-30 20:35 ` fweimer at redhat dot com
2 siblings, 0 replies; 9+ messages in thread
From: schwab at sourceware dot org @ 2014-05-28 19:46 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=11044
Andreas Schwab <schwab at sourceware dot org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC|jackie.rosen at hushmail dot com |
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Bug libc/11044] malloc dynamic mmap threshold causes 50%-100% increase in memory usage
[not found] <bug-11044-131@http.sourceware.org/bugzilla/>
@ 2014-02-16 17:43 ` jackie.rosen at hushmail dot com
2014-05-28 19:46 ` schwab at sourceware dot org
2014-06-30 20:35 ` fweimer at redhat dot com
2 siblings, 0 replies; 9+ messages in thread
From: jackie.rosen at hushmail dot com @ 2014-02-16 17:43 UTC (permalink / raw)
To: glibc-bugs
https://sourceware.org/bugzilla/show_bug.cgi?id=11044
Jackie Rosen <jackie.rosen at hushmail dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jackie.rosen at hushmail dot com
--- Comment #6 from Jackie Rosen <jackie.rosen at hushmail dot com> ---
*** Bug 260998 has been marked as a duplicate of this bug. ***
Seen from the domain http://volichat.com
Page where seen: http://volichat.com/adult-chat-rooms
Marked for reference. Resolved as fixed @bugzilla.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-06-30 20:35 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
2010-04-05 5:08 ` [Bug libc/11044] " drepper at redhat dot com
2010-04-05 16:10 ` tom at atoptech dot com
2010-04-05 18:32 ` drepper at redhat dot com
2010-04-05 18:35 ` drepper at redhat dot com
2010-04-07 17:09 ` tom at atoptech dot com
[not found] <bug-11044-131@http.sourceware.org/bugzilla/>
2014-02-16 17:43 ` jackie.rosen at hushmail dot com
2014-05-28 19:46 ` schwab at sourceware dot org
2014-06-30 20:35 ` fweimer at redhat dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).