From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sa-prd-fep-047.btinternet.com (mailomta7-sa.btinternet.com [213.120.69.13]) by sourceware.org (Postfix) with ESMTPS id DA16B3959C64 for ; Wed, 14 Apr 2021 18:54:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org DA16B3959C64 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=dronecode.org.uk Authentication-Results: sourceware.org; spf=none smtp.mailfrom=jon.turney@dronecode.org.uk Received: from sa-prd-rgout-005.btmx-prd.synchronoss.net ([10.2.38.8]) by sa-prd-fep-047.btinternet.com with ESMTP id <20210414185414.ZJA28013.sa-prd-fep-047.btinternet.com@sa-prd-rgout-005.btmx-prd.synchronoss.net>; Wed, 14 Apr 2021 19:54:14 +0100 Authentication-Results: btinternet.com; auth=pass (PLAIN) smtp.auth=jonturney@btinternet.com X-SNCR-Rigid: 6038718006C9E391 X-Originating-IP: [81.153.98.246] X-OWM-Source-IP: 81.153.98.246 (GB) X-OWM-Env-Sender: jonturney@btinternet.com X-VadeSecure-score: verdict=clean score=0/300, class=clean X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgeduledrudeluddgudefvdcutefuodetggdotefrodftvfcurfhrohhfihhlvgemuceutffkvffkuffjvffgnffgvefqofdpqfgfvfenuceurghilhhouhhtmecufedtudenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhepuffvfhfhkffffgggjggtgfesthekredttdefjeenucfhrhhomheplfhonhcuvfhurhhnvgihuceojhhonhdrthhurhhnvgihsegurhhonhgvtghouggvrdhorhhgrdhukheqnecuggftrfgrthhtvghrnhepheekfedvtedvfeelgfffiefguddtieegjeekveeitdeiheefgeetteeiuddtvdegnecuffhomhgrihhnpegulhhlrdhishdpghhithhhuhgsrdgtohhmnecukfhppeekuddrudehfedrleekrddvgeeinecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehhvghloheplgduledvrdduieekrddurdduuddungdpihhnvghtpeekuddrudehfedrleekrddvgeeipdhmrghilhhfrhhomhepoehjohhnrdhtuhhrnhgvhiesughrohhnvggtohguvgdrohhrghdruhhkqecuuefqffgjpeekuefkvffokffogfdprhgtphhtthhopeeotgihghifihhnqdguvghvvghlohhpvghrshestgihghifihhnrdgtohhmqedprhgtphhtthhopeeomhgrrhhksehmrgigrhhnugdrtghomheq X-RazorGate-Vade-Verdict: clean 0 X-RazorGate-Vade-Classification: clean Received: from [192.168.1.111] (81.153.98.246) by sa-prd-rgout-005.btmx-prd.synchronoss.net (5.8.340) (authenticated as jonturney@btinternet.com) id 6038718006C9E391; Wed, 14 Apr 2021 19:54:14 +0100 Subject: Re: Maybe consider rpmalloc To: cygwin-developers@cygwin.com References: <067987e2-e958-b56c-efea-25d827568453@maxrnd.com> <6f68b10b-7fe5-4378-afb9-9001de084edf@maxrnd.com> <3adb36f3-8740-3ff7-5f8a-90cdf3dfb64d@maxrnd.com> <69159cfa-8fc5-283b-126b-740b841841cd@maxrnd.com> <93809c4f-7747-3611-0d20-bde09e091f1d@maxrnd.com> From: Jon Turney Message-ID: Date: Wed, 14 Apr 2021 19:53:22 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.1 MIME-Version: 1.0 In-Reply-To: <93809c4f-7747-3611-0d20-bde09e091f1d@maxrnd.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3571.6 required=5.0 tests=BAYES_00, FORGED_SPF_HELO, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2021 18:54:17 -0000 On 14/04/2021 09:19, Mark Geisert wrote: > Teemu Nätkinniemi wrote: >> Thanks a lot for looking into this issue! I wonder if there are any >> other applications affected by this? > > We have several examples by now.  All are (relatively) long-lasting > apps, with high to very high memory allocation churn, often > multi-threaded.  I believe some specific rsync operations hit this. > Achim reported a zstd operation that exhibited the symptoms.  And I've > been attempting to get a working replacement for Cygwin's malloc for > some time but every malloc I've tested, several of them, exhibits > similar symptoms: excessive time being spent in ntdll.dll presumably > supporting the memory operations. > > Your rpmalloc "hack" is interesting in that you aren't using Cygwin's > mmap() underneath the malloc routines; you're calling Windows VM ops > directly.  Not sure yet what all the implications are. > > I need to identify what's being hit within ntdll.dll.  Is it one or two > routines, or just hot locks.  So that means getting the correct PDB file > from the MS Symbol Server and working with Windows tools I'm unfamiliar > with.  Sigh, in an earlier life I had a gdb that we'd taught how to work Yes, this would indeed be a very useful thing to have in gdb. I'm not aware of any public work in that direction, though. > with PDB files; dunno if I could resurrect that.  Profiling the Cygwin > DLL itself, call profiling I mean, might lead somewhere as well. In the past I've had some success with using the Very Sleepy profiler ([1]), which can use both PDB and DWARF symbols, on cygwin executables. [1] https://github.com/VerySleepy/verysleepy