From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 19772 invoked by alias); 2 Dec 2009 18:31:56 -0000 Received: (qmail 19710 invoked by uid 48); 2 Dec 2009 18:31:40 -0000 Date: Wed, 02 Dec 2009 18:31:00 -0000 From: "tom at atoptech dot com" To: glibc-bugs@sources.redhat.com Message-ID: <20091202183137.11044.tom@atoptech.com> Reply-To: sourceware-bugzilla@sourceware.org Subject: [Bug libc/11044] New: malloc dynamic mmap threshold causes 50%-100% increase in memory usage X-Bugzilla-Reason: CC Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: glibc-bugs-owner@sourceware.org X-SW-Source: 2009-12/txt/msg00011.txt.bz2 We recently compiled and ran our application on Centos 5 which has glibc 2.5. We found many of our benchmarks and regression runs resulted in 50-100% increase in memory usage. We eventually pin-pointed the problem to malloc and after examining the 2.5 malloc source code we found that malloc now adjusts the mmap threshold dynamically. For our application, the dynamic adjustment does not work. The end result is more fragmentation and many large malloc requests results in more and more memory allocated. Our application is Place-And-Route for backend chip design. It is very common for an innovation of our application to run for several days using up to 32GB of memory (depending on the size of the chip, i.e., multi-million gates). During an invocation of our application many different algorithms are executed with different memory usage patterns (many very large memory allocations and frees.) We relied on the old malloc behavior to mmap these requests, because this tended to reduce the fragmentation. My guess is that this malloc change is probably just hitting people in userland, because the Linux distributions use older versions of glibc. Again Centos 5, is using 2.5. I found some discussion of this change via google. And there was some questioning as to whether this change was valid. Again, in our application the memory usage pattern varies greatly and unless you have a good statistical model of the memory usage pattern, it is hight unlikely to dynamically adjust the mmap threshold and get it right. Because we have the ability to turn off the dynamic threshold (mallopt), we have a work around. But I thought it was best to give feedback on how this changes was impacting our application. Tom Geocaris Atoptech -- Summary: malloc dynamic mmap threshold causes 50%-100% increase in memory usage Product: glibc Version: unspecified Status: NEW Severity: normal Priority: P2 Component: libc AssignedTo: drepper at redhat dot com ReportedBy: tom at atoptech dot com CC: glibc-bugs at sources dot redhat dot com http://sourceware.org/bugzilla/show_bug.cgi?id=11044 ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.