From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from m0.truegem.net (m0.truegem.net [69.55.228.47]) by sourceware.org (Postfix) with ESMTPS id D04DC3857027 for ; Fri, 2 Apr 2021 05:45:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org D04DC3857027 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=maxrnd.com Authentication-Results: sourceware.org; spf=none smtp.mailfrom=mark@maxrnd.com Received: (from daemon@localhost) by m0.truegem.net (8.12.11/8.12.11) id 1325jtRS055476 for ; Thu, 1 Apr 2021 22:45:55 -0700 (PDT) (envelope-from mark@maxrnd.com) Received: from 162-235-43-67.lightspeed.irvnca.sbcglobal.net(162.235.43.67), claiming to be "[192.168.1.20]" via SMTP by m0.truegem.net, id smtpdwPkSkD; Thu Apr 1 21:45:49 2021 Subject: Maybe consider rpmalloc -- Was: Re: Cygwin malloc tune-up status To: cygwin-developers@cygwin.com References: <067987e2-e958-b56c-efea-25d827568453@maxrnd.com> From: Mark Geisert Message-ID: Date: Thu, 1 Apr 2021 22:45:49 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.4 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Apr 2021 05:46:00 -0000 Hi Teemu, Teemu Nätkinniemi via Cygwin-developers wrote: > Hi! > > I encounter a problem with Cygwin's malloc and remembered this thread. > Sorry if this is off-topic. New topic, so new thread :-) > I have been trying to port bwa aligner to Cygwin. Initially everything > seemed to work but for some reason in some cases threading didn't seem > to work properly. I got a fix recently from a third party which was to > force bwa to use rpmalloc. > > This got me thinking if there is a problem with Cygwin's malloc in > some cases and if there were people in this list who might be > interested in knowing that the problem exists. > > Here's a link to the rpmalloc fix. > > https://github.com/WGSExtract/bwa/commit/3087fa876b079fcb6a0a58f1e01757f4820094a8 > > Here's a test case: > > bwa_original mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam > > bwa_working mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam > > Files: > > https://drive.google.com/drive/folders/1waICih51f4mHZEyWY1onyEcKqm0kj3Yt?usp=sharing I'm not aware of anything actually being broken in the current Cygwin malloc (which is just the most recent dlmalloc), even in multi-threaded workloads. It's only that there has long been interest around possibilities of making it faster. So far I've been unsuccessful: though I can get the Cygwin-level malloc operation faster by using other malloc's, it has always been overshadowed by a lot more time being spent in ntdll.dll underneath. There's no point releasing something cosmetically better but slower in practice. Thanks for mentioning rpmalloc; that's a new one to me. It does appear to be coded nicely, like most of the other malloc's I've looked at. I'll see if I can get that to work with Cygwin; the big issue tends to be fork() having to replicate the parent malloc layout, book-keeping as well as app data, in the child. I will point out that you're not using rpmalloc as it would work on Cygwin, you've got a Windows-flavored rpmalloc because you've #defined _WIN32 and such via the Makefile.cygwin. But it appears to be a clean break with Cygwin's malloc in your application, so congrats on that. I understand that you've solved what you think was a Cygwin malloc issue by using rpmalloc, but I don't see how you came to the conclusion that it was a malloc issue, as opposed to something else with threads or beyond that. Thanks again, ..mark