public inbox for glibc-bugs@sourceware.org
help / color / mirror / Atom feed
* [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage
@ 2009-12-02 18:31 tom at atoptech dot com
  2010-04-05  5:08 ` [Bug libc/11044] " drepper at redhat dot com
                   ` (4 more replies)
  0 siblings, 5 replies; 9+ messages in thread
From: tom at atoptech dot com @ 2009-12-02 18:31 UTC (permalink / raw)
  To: glibc-bugs

We recently compiled and ran our application on Centos 5 which has glibc 2.5. We
found many of our benchmarks and regression runs resulted in 50-100% increase in
memory usage. We eventually pin-pointed the problem to malloc and after
examining the 2.5 malloc source code we found that malloc now adjusts the mmap
threshold dynamically. For our application, the dynamic adjustment does not
work. The end result is more fragmentation and many large malloc requests
results in more and more memory allocated.

Our application is Place-And-Route for backend chip design. It is very common
for an innovation of our application to run for several days using up to 32GB of
memory (depending on the size of the chip, i.e., multi-million gates). During an
 invocation of our application many different algorithms are executed with
different memory usage patterns (many very large memory allocations and frees.) 

We relied on the old malloc behavior to mmap these requests, because this tended
to reduce the fragmentation.

My guess is that this malloc change is probably just hitting people in userland,
because the Linux distributions use older versions of glibc. Again Centos 5, is
using 2.5.

I found some discussion of this change via google. And there was some
questioning as to whether this change was valid. Again, in our application the
memory usage pattern varies greatly and unless you have a good statistical model
of the memory usage pattern, it is hight unlikely to dynamically adjust the mmap
threshold and get it right.

Because we have the ability to turn off the dynamic threshold (mallopt), we have
a work around. But I thought it was best to give feedback on how this changes
was impacting our application.

Tom Geocaris
Atoptech

-- 
           Summary: malloc dynamic mmap threshold causes 50%-100% increase
                    in memory usage
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper at redhat dot com
        ReportedBy: tom at atoptech dot com
                CC: glibc-bugs at sources dot redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=11044

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.


^ permalink raw reply	[flat|nested] 9+ messages in thread
[parent not found: <bug-11044-131@http.sourceware.org/bugzilla/>]
end of thread, other threads:[~2014-06-30 20:35 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-12-02 18:31 [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage tom at atoptech dot com
2010-04-05  5:08 ` [Bug libc/11044] " drepper at redhat dot com
2010-04-05 16:10 ` tom at atoptech dot com
2010-04-05 18:32 ` drepper at redhat dot com
2010-04-05 18:35 ` drepper at redhat dot com
2010-04-07 17:09 ` tom at atoptech dot com
     [not found] <bug-11044-131@http.sourceware.org/bugzilla/>
2014-02-16 17:43 ` jackie.rosen at hushmail dot com
2014-05-28 19:46 ` schwab at sourceware dot org
2014-06-30 20:35 ` fweimer at redhat dot com

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).