From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 2601 invoked by alias); 12 Dec 2007 17:12:21 -0000 Received: (qmail 2591 invoked by uid 22791); 12 Dec 2007 17:12:20 -0000 X-Spam-Check-By: sourceware.org Received: from wa-out-1112.google.com (HELO wa-out-1112.google.com) (209.85.146.178) by sourceware.org (qpsmtpd/0.31) with ESMTP; Wed, 12 Dec 2007 17:12:12 +0000 Received: by wa-out-1112.google.com with SMTP id m16so496955waf.20 for ; Wed, 12 Dec 2007 09:12:10 -0800 (PST) Received: by 10.114.111.1 with SMTP id j1mr1006994wac.138.1197479530127; Wed, 12 Dec 2007 09:12:10 -0800 (PST) Received: by 10.114.208.17 with HTTP; Wed, 12 Dec 2007 09:12:04 -0800 (PST) Message-ID: <9e4733910712120912l342350f2i1f190c45730108f2@mail.gmail.com> Date: Wed, 12 Dec 2007 17:29:00 -0000 From: "Jon Smirl" To: "Linus Torvalds" Subject: Re: Something is broken in repack Cc: "Nicolas Pitre" , "Junio C Hamano" , gcc@gcc.gnu.org, "Git Mailing List" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <9e4733910712071505y6834f040k37261d65a2d445c4@mail.gmail.com> <9e4733910712102129v140c2affqf2e73e75855b61ea@mail.gmail.com> <9e4733910712102301p5e6c4165v6afb32d157478828@mail.gmail.com> <9e4733910712110821o7748802ag75d9df4be8b2c123@mail.gmail.com> Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org X-SW-Source: 2007-12/txt/msg00380.txt.bz2 On 12/12/07, Linus Torvalds wrote: > > > On Wed, 12 Dec 2007, Nicolas Pitre wrote: > > > > So... my conclusion is that the glibc allocator has fragmentation issues > > with this work load, given the notable difference with the Google > > allocator, which itself might not be completely immune to fragmentation > > issues of its own. > > Yes. > > Note that delta following involves patterns something like > > allocate (small) space for delta > for i in (1..depth) { > allocate large space for base > allocate large space for result > .. apply delta .. > free large space for base > free small space for delta > } Is it hard to hack up something that statically allocates a big block of memory per thread for these two and then just reuses it? allocate (small) space for delta allocate large space for base The alternating between long term and short term allocations definitely aggravates fragmentation. > > so if you have some stupid heap algorithm that doesn't try to merge and > re-use free'd spaces very aggressively (because that takes CPU time!), you > might have memory usage be horribly inflated by the heap having all those > holes for all the objects that got free'd in the chain that don't get > aggressively re-used. > > Threaded memory allocators then make this worse by probably using totally > different heaps for different threads (in order to avoid locking), so they > will *all* have the fragmentation issue. > > And if you *really* want to cause trouble for a memory allocator, what you > should try to do is to allocate the memory in one thread, and free it in > another, and then things can really explode (the freeing thread notices > that the allocation is not in its thread-local heap, so instead of really > freeing it, it puts it on a separate list of areas to be freed later by > the original thread when it needs memory - or worse, it adds it to the > local thread list, and makes it effectively totally impossible to then > ever merge different free'd allocations ever again because the freed > things will be on different heap lists!). > > I'm not saying that particular case happens in git, I'm just saying that > it's not unheard of. And with the delta cache and the object lookup, it's > not at _all_ impossible that we hit the "allocate in one thread, free in > another" case! > > Linus > -- Jon Smirl jonsmirl@gmail.com