From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by sourceware.org (Postfix) with ESMTPS id 0418A3857C52 for ; Thu, 1 Apr 2021 09:20:06 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 0418A3857C52 Received: by mail-ed1-x531.google.com with SMTP id dm8so1162759edb.2 for ; Thu, 01 Apr 2021 02:20:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=12VEA6m/hCwWr/6oK9Lac8XP90hsg6y/KYNUYFb8zNc=; b=KwB5OdDMjiD65K4j6tmU3jVUIFaD9w6x1xYbyo/ZGwK2xbZ562zeS6IC2lIruiaXs/ 8zLShYn6KBLaJw0ALMwnxGFF8tkIrCrffxf/qOufvsMavS9/bS90nIm85M1U1C8I5nXH golagR9b5R5MGHAF/iU6OWx22qCMs2fh0E1giStPh1GZHRph5/KD/pDOeL2wIwQFg0SL Ms0m7CEPwLmQtu/S4vcBrEFa9JLkMtUOXKkceqax3P9iyBv1ATgtesITbjT72YTO5+3J FxWbUqV1xkoqYpVcktt2ycT+cfVoVnV9qzNPWD3ZZhTe1073xNDi0NYu//Yrtpvo0GcL rQ3g== X-Gm-Message-State: AOAM531J4PHqm1ay0TDYOhszrQnBhGFUU/CHT1y8xRwkSrN8zl01Nf9u yP7o6CZWME/fE6vid+H+bcO24154duwcU32QObcpDr1srP8= X-Google-Smtp-Source: ABdhPJwWzt9dJCzYp03vqRZiD+WZsvesjsXHLNLX3e/ZEXRFlzTDIPkKJ368KUyAz2hYHGrEq0QMHo/13PHMxAGExjI= X-Received: by 2002:a05:6402:b31:: with SMTP id bo17mr8678958edb.113.1617268805059; Thu, 01 Apr 2021 02:20:05 -0700 (PDT) MIME-Version: 1.0 References: <067987e2-e958-b56c-efea-25d827568453@maxrnd.com> In-Reply-To: From: =?UTF-8?Q?Teemu_N=C3=A4tkinniemi?= Date: Thu, 1 Apr 2021 12:19:53 +0300 Message-ID: Subject: Re: Cygwin malloc tune-up status To: Mark Geisert Cc: "cygwin-developers@cygwin.com" Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Apr 2021 09:20:07 -0000 ti 29. syysk. 2020 klo 5.23 Mark Geisert (mark@maxrnd.com) kirjoitti: > > Johannes Schindelin wrote: > > Hi Mark, > > On Thu, 24 Sep 2020, Mark Geisert wrote: > >> I've been looking into two potential enhancements of Cygwin's malloc operation > >> for Cygwin 3.2. The first is to replace the existing function-level locking > >> with something more fine-grained to help threaded processes; the second is to > >> implement thread-specific memory pools with the aim of lessening lock activity > >> even further. > >> > >> Although I've investigated several alternative malloc packages, including > >> ptmalloc[23], nedalloc, and Windows Heaps, only the latter seems to improve on > >> the performance of Cygwin's malloc. Unfortunately using Windows Heaps would > >> require fiddling with undocumented heap structures to enable use with fork(). > >> I also looked at BSD's jemalloc and Google's tcmalloc. Those two require much > >> more work to port to Cygwin so I've back-burnered them for the time being. > > > > I am just a lurker when it comes to your project, but I wonder whether you > > had any chance to look into mimalloc > > (https://github.com/microsoft/mimalloc)? I had investigated it in Git for > > Windows' context for a while (because nedmalloc, which is used by Git for > > Windows, is no longer actively maintained). > > Hi Johannes, > Great minds think alike! Yours is the 3rd pointer I've received on- and off-list > towards mimalloc. I had not heard of it before. I've looked into it and have now > added it to my back-burnered list. > > mimalloc looks promising. It's fairly small. It has at least one issue it shares > with jemalloc and tcmalloc: initialization code that needs to run before the > Cygwin DLL has completely set up a new process' environment. A chicken-and-egg > problem, if you will. A solution to that (which I'm pondering) will allow me to > test all three malloc alternatives in the future. > Hi! I encounter a problem with Cygwin's malloc and remembered this thread. Sorry if this is off-topic. I have been trying to port bwa aligner to Cygwin. Initially everything seemed to work but for some reason in some cases threading didn't seem to work properly. I got a fix recently from a third party which was to force bwa to use rpmalloc. This got me thinking if there is a problem with Cygwin's malloc in some cases and if there were people in this list who might be interested in knowing that the problem exists. Here's a link to the rpmalloc fix. https://github.com/WGSExtract/bwa/commit/3087fa876b079fcb6a0a58f1e01757f4820094a8 Here's a test case: bwa_original mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam bwa_working mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam Files: https://drive.google.com/drive/folders/1waICih51f4mHZEyWY1onyEcKqm0kj3Yt?usp=sharing Thanks, Teemu