From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 11203 invoked by alias); 27 Aug 2011 22:02:41 -0000 Received: (qmail 11189 invoked by uid 22791); 27 Aug 2011 22:02:37 -0000 X-SWARE-Spam-Status: No, hits=-2.8 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 X-Spam-Check-By: sourceware.org Received: from localhost (HELO sourceware.org) (127.0.0.1) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Sat, 27 Aug 2011 22:02:24 +0000 From: "rich at testardi dot com" To: glibc-bugs@sources.redhat.com Subject: [Bug libc/11261] malloc uses excessive memory for multi-threaded applications Date: Sat, 27 Aug 2011 22:02:00 -0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: glibc X-Bugzilla-Component: libc X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: rich at testardi dot com X-Bugzilla-Status: REOPENED X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: drepper.fsp at gmail dot com X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: glibc-bugs-owner@sourceware.org X-SW-Source: 2011-08/txt/msg00104.txt.bz2 http://sourceware.org/bugzilla/show_bug.cgi?id=11261 --- Comment #9 from Rich Testardi 2011-08-27 22:02:03 UTC --- Hi, We ended up building our own memory allocator -- it's faster and more efficient than glibc, and it works equally fast with threads and wihout. We used the "small block allocator" concept from HP-UX where we only allocate huge (32MB) allocations from the system (after setting M_MMAP_THRESHOLD suitably small). We then carve out large *naturally aligned* 1MB blocks from the huge allocation (accepting 3% waste, since the allocation was page alogned to begin with, not naturally aligned). And we carve each one of those large blocks into small fixed size buckets (which are fractional powers of 2 -- like 16 bytes, 20, 24, 28, 32, 40, 48, 56, 64, 80, etc.). Then we put the aligned addresses into a very fast hash and have a linked list for each bucket size. This means our allocate routine is just a lock, linked list remove, unlock, on average, and our free routine is just a hash lookup, lock, linked list insert, unlock on average. The trick here is that from any address being freed, you can get back to the naturally aligned 1MB block that contains it with just a pointer mask, and from there you can get the allocation's size as well as the head of the linked list of free entries to which it should be returned... -- Rich ----- Original Message ----- From: heuler at infosim dot net To: rich@testardi.com Sent: Saturday, August 27, 2011 3:45 PM Subject: [Bug libc/11261] malloc uses excessive memory for multi-threaded applications http://sourceware.org/bugzilla/show_bug.cgi?id=11261 Marius Heuler changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |heuler at infosim dot net Resolution|WONTFIX | --- Comment #8 from Marius Heuler 2011-08-27 21:45:04 UTC --- We have exactly the same problem with the current implementation of malloc. The suggested solutions by Ulrich using M_ARENA_MAX does not work since the check for number of arenas is not thread safe. In fact the limit is not working for heay threading applications where that would be needed! Since the number of cores and usage of threads will increase strongly there should be a solution for that kind of applications! If the arena limit would work as described we would have no problem. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You reported the bug. -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.