From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by sourceware.org (Postfix) with ESMTPS id 6894C3944433 for ; Wed, 14 Apr 2021 18:36:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 6894C3944433 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=tnatkinn@gmail.com Received: by mail-ed1-x536.google.com with SMTP id bx20so23761952edb.12 for ; Wed, 14 Apr 2021 11:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=wwnQT2LtTAyOa56D+Jzo0MEFIV5af2yRk8Le+CRv5Qw=; b=Uu9gh+CU4NGipCDNVKLznOFDam+XhpmfThhyOXriYlO+X5UOIKEAg2B/Tk+Kr4yvaS bCi+bcWg45M2fdqIQp5Paf3C4tzAMsslTQfDBNrn5/Q3QbqUkPSlgRjvtYqTKhnJf6W5 mPIZErtn5+we2lZ8gLRhMGO2ToqCn7mX4VMuyHn1k6VlSnn8hMRA+f2JM3/U4KGIFS37 mUweKSVMXJoSuhDL/1mLQGZl6ISv2pfjuaHI3qawgYD7IriawwFlrXESzohohyhWhEsb voy036ENo9hDiG5T16obbl0fnAIsE5LM1P7h0hz9Zb6pPPxD4IjVLCH+mciemCgDGtyG QVLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=wwnQT2LtTAyOa56D+Jzo0MEFIV5af2yRk8Le+CRv5Qw=; b=VRlnhpvXrw6O8B1Pqj9nU3SDT6KitQo872JhARFFSbsWhtD+0IYRLQPw7ZX/Br5SsX tOw1hyNE1t+HtfyPhHG6ounOkamuJXJgyC/GvZxaL/CrYdzfzk5EX1c/s+YQ5/2KVhqP vviFUgXXIgue/EPap0k2w4Ikwh8s6+e3Jw/vrRc50+AuZ9+VtoX3egfYjLh90Yq8YzfP ysx+mVUk0X+443Nm7HHj10q/EFqEZ32G5sih7ehhilmL6SGx/xGFRJWpBHC++TB5OCWi dw9DveltipgTQHNyIip1cPXdqfX6tzMPnPGLGqOzphyXKKkYuAV1qupUpZbvM5OpS8qG aI0w== X-Gm-Message-State: AOAM530GayJgb12ZxBsMPFSmePirhbzYs2z9Rw3dPtWonI6nttxLya2e GTsLvUf0yeBjHYXL8LZO/UIpVaHFZvy8aX9xQHZPu1cH4vI= X-Google-Smtp-Source: ABdhPJzWxjOfegGNFu8Lpmm+hGH4kKftMkIrgKNrj510D/rCOzcIOCLWXwHTVwsvjWoAjd/vHKWip6Xkqo6p6LH0FA0= X-Received: by 2002:aa7:c349:: with SMTP id j9mr296543edr.41.1618425382425; Wed, 14 Apr 2021 11:36:22 -0700 (PDT) MIME-Version: 1.0 References: <067987e2-e958-b56c-efea-25d827568453@maxrnd.com> <6f68b10b-7fe5-4378-afb9-9001de084edf@maxrnd.com> <3adb36f3-8740-3ff7-5f8a-90cdf3dfb64d@maxrnd.com> <69159cfa-8fc5-283b-126b-740b841841cd@maxrnd.com> <93809c4f-7747-3611-0d20-bde09e091f1d@maxrnd.com> In-Reply-To: <93809c4f-7747-3611-0d20-bde09e091f1d@maxrnd.com> From: =?UTF-8?Q?Teemu_N=C3=A4tkinniemi?= Date: Wed, 14 Apr 2021 21:36:12 +0300 Message-ID: Subject: Re: Maybe consider rpmalloc To: "cygwin-developers@cygwin.com" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2021 18:36:24 -0000 ke 14. huhtik. 2021 klo 12.20 Mark Geisert (mark@maxrnd.com) kirjoitti: > > Teemu N=C3=A4tkinniemi wrote: > > Thanks a lot for looking into this issue! I wonder if there are any > > other applications affected by this? > > We have several examples by now. All are (relatively) long-lasting apps,= with > high to very high memory allocation churn, often multi-threaded. I belie= ve some > specific rsync operations hit this. Achim reported a zstd operation that > exhibited the symptoms. And I've been attempting to get a working replac= ement for > Cygwin's malloc for some time but every malloc I've tested, several of th= em, > exhibits similar symptoms: excessive time being spent in ntdll.dll presum= ably > supporting the memory operations. Bwa's author has a more recent program called bwa-mem2 which seems to have exact same problem as bwa. I have tried the rpmalloc trick but it did not work or I could not identify the correct routine(s). https://github.com/bwa-mem2/bwa-mem2 > Your rpmalloc "hack" is interesting in that you aren't using Cygwin's mma= p() > underneath the malloc routines; you're calling Windows VM ops directly. = Not sure > yet what all the implications are. It was not my hack. Just a third party who managed to fix the problem as bwa is actually a useful program and there was a need to get it running on Windows/Cygwin. > Lots of plausible directions to go... I am thankful that you are looking into this. Does anyone else have any input that could be useful? Corinna? Teemu