From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from m0.truegem.net (m0.truegem.net [69.55.228.47]) by sourceware.org (Postfix) with ESMTPS id 8AF713987C30 for ; Wed, 14 Apr 2021 08:19:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 8AF713987C30 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=maxrnd.com Authentication-Results: sourceware.org; spf=none smtp.mailfrom=mark@maxrnd.com Received: (from daemon@localhost) by m0.truegem.net (8.12.11/8.12.11) id 13E8Jd9X096770 for ; Wed, 14 Apr 2021 01:19:39 -0700 (PDT) (envelope-from mark@maxrnd.com) Received: from 162-235-43-67.lightspeed.irvnca.sbcglobal.net(162.235.43.67), claiming to be "[192.168.1.20]" via SMTP by m0.truegem.net, id smtpdnXRMSr; Wed Apr 14 01:19:30 2021 Subject: Re: Maybe consider rpmalloc To: cygwin-developers@cygwin.com References: <067987e2-e958-b56c-efea-25d827568453@maxrnd.com> <6f68b10b-7fe5-4378-afb9-9001de084edf@maxrnd.com> <3adb36f3-8740-3ff7-5f8a-90cdf3dfb64d@maxrnd.com> <69159cfa-8fc5-283b-126b-740b841841cd@maxrnd.com> From: Mark Geisert Message-ID: <93809c4f-7747-3611-0d20-bde09e091f1d@maxrnd.com> Date: Wed, 14 Apr 2021 01:19:31 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.4 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2021 08:19:44 -0000 Teemu Nätkinniemi wrote: > Thanks a lot for looking into this issue! I wonder if there are any > other applications affected by this? We have several examples by now. All are (relatively) long-lasting apps, with high to very high memory allocation churn, often multi-threaded. I believe some specific rsync operations hit this. Achim reported a zstd operation that exhibited the symptoms. And I've been attempting to get a working replacement for Cygwin's malloc for some time but every malloc I've tested, several of them, exhibits similar symptoms: excessive time being spent in ntdll.dll presumably supporting the memory operations. Your rpmalloc "hack" is interesting in that you aren't using Cygwin's mmap() underneath the malloc routines; you're calling Windows VM ops directly. Not sure yet what all the implications are. I need to identify what's being hit within ntdll.dll. Is it one or two routines, or just hot locks. So that means getting the correct PDB file from the MS Symbol Server and working with Windows tools I'm unfamiliar with. Sigh, in an earlier life I had a gdb that we'd taught how to work with PDB files; dunno if I could resurrect that. Profiling the Cygwin DLL itself, call profiling I mean, might lead somewhere as well. Lots of plausible directions to go... Cheers, ..mark