From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by sourceware.org (Postfix) with ESMTPS id 32AA43857012 for ; Mon, 12 Apr 2021 08:49:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 32AA43857012 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=tnatkinn@gmail.com Received: by mail-ed1-x534.google.com with SMTP id w23so14079052edx.7 for ; Mon, 12 Apr 2021 01:49:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Intn56ILJkWJ5meAF3Jb1G7Sfn3e9mh78Mf5lVg1e/s=; b=V5eyi69yYeLVaTGP3v5zENlnv8P1u6Z1ryCOkU71mToc5NNlypOwQD+il1sGVgmE5e eQQBmuclj2ZEQmEsvtdcQMpR0XiOZb/+8f7bQ2m/xOfV8ZWO2qjGcgF3CVUodrCuG1Ap ts7T99PgkB6S1wFzR9J1e/Fd32F4zrnuY2BzNJ6JHY5uNUsItALnYDkv8uJsmv+o7bfF nR/9VEzKd+//S0BkSlck6B0aU8kh3s88SOj8ONuMPUNv2Hm7bAbe/rutTEFjZx+dRKch n05t1X4q/ZO619vqDh173N144oRD0cklLf5c9a/pRJS1q/pN3AayMdwalrGMAaS8Por9 BUaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Intn56ILJkWJ5meAF3Jb1G7Sfn3e9mh78Mf5lVg1e/s=; b=pYd490aUWCpMVcz7LlvQCTjBdFQ7wnQ57Bez4W/+KEyBC+TG3Ms37xpceTup4n3XfD zyHyk0CnTyJffzNmb2zoyeXBDxkECaBrEwcWzYJxBrM1nzmE3SL9qctwH5ClrpztcnwW Yml91iOFSmpaObAzy09KxnauHK13NaZoPLjBPJyNKnht7YLgRBAyqn0lkVABkq2R4emW WunODvn6YyGTTmba/KZE9hdcwQ2ErAv5OmqYryevtOkx7L3uy6/SwvRRsXNdpoYLyGOL Eyu8KA+4Xkqopcwlid+iqLrsCOSDod8vFlmsxAjccb3iqqSNZeLkY3xJIgQvXyW6ucJY hxNA== X-Gm-Message-State: AOAM531CuiAYpSx0Fo3aR+TUT8VEsPAwZIyCbXLeSRwOts0nPfEM3JAU eCn265xEaB9mqq2i0G4f+keG6wSSqao0vzeX4ncxWClx0PI= X-Google-Smtp-Source: ABdhPJz77zXyTd2J3czz//9i3Nbt8Dn8KXZ9KR9nUEfaiLGmaIyJ7wQ4enYwDDTJINiEeJkEugS9HD19Yx1GFFwL1o8= X-Received: by 2002:a05:6402:440d:: with SMTP id y13mr28623601eda.316.1618217339331; Mon, 12 Apr 2021 01:48:59 -0700 (PDT) MIME-Version: 1.0 References: <067987e2-e958-b56c-efea-25d827568453@maxrnd.com> <6f68b10b-7fe5-4378-afb9-9001de084edf@maxrnd.com> <3adb36f3-8740-3ff7-5f8a-90cdf3dfb64d@maxrnd.com> In-Reply-To: <3adb36f3-8740-3ff7-5f8a-90cdf3dfb64d@maxrnd.com> From: =?UTF-8?Q?Teemu_N=C3=A4tkinniemi?= Date: Mon, 12 Apr 2021 11:48:48 +0300 Message-ID: Subject: Re: Maybe consider rpmalloc To: Mark Geisert Cc: "cygwin-developers@cygwin.com" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Apr 2021 08:49:02 -0000 Hello, Thanks for testing! I found a better test case with smaller files which should clearly show the issue. https://drive.google.com/drive/folders/1jOilHtKrr6CHn7zg__DE93RCDyseoTB1?us= p=3Dsharing Here's the results. Bwa_working is the one with rpmalloc and bwa_original is unpatched. As you can see the unpatched version with several threads takes a whole lot more time to finish even when compared with unpatched exe running with a single thread. I am not the only one experiencing the issue so I doubt it is my system. $ ../bwa-working/bwa_working.exe mem chr19_KI270866v1_alt.fasta 7859_GPI.read1.fq 7859_GPI.read2.fq > test1working.sam (cut) [main] Real time: 1.744 sec; CPU: 1.624 sec $ ../bwa-working/bwa_working.exe mem -t 10 chr19_KI270866v1_alt.fasta 7859_GPI.read1.fq 7859_GPI.read2.fq > test1workingt10.sam (cut) [main] Real time: 0.354 sec; CPU: 2.218 sec $ ../bwa-test/bwa_original.exe mem chr19_KI270866v1_alt.fasta 7859_GPI.read1.fq 7859_GPI.read2.fq > test1orig.sam (cut) [main] Real time: 1.733 sec; CPU: 1.608 sec $ ../bwa-test/bwa_original.exe mem -t 10 chr19_KI270866v1_alt.fasta 7859_GPI.read1.fq 7859_GPI.read2.fq > test1origt10.sam (cut) [main] Real time: 8.131 sec; CPU: 5.265 sec Teemu su 11. huhtik. 2021 klo 12.52 Mark Geisert (mark@maxrnd.com) kirjoitti: > > Hi Teemu, > > Teemu N=C3=A4tkinniemi via Cygwin-developers wrote: > > Sorry, hurt my back yesterday and looks like I am not thinking clearly. > > Hope you are feeling better by this time. > > > ./bwa mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test1.sam > > Thanks. It was my unfamiliarity with Google Drive which prevented my fin= ding all > the data files you had stored there. After a while I did find all I need= ed. > > I rebuilt bwa.exe alternately using the provided Makefile and Makefile.cy= gwin. > When building with the latter I made sure your #ifdef patches were enable= d so that > rpmalloc was pulled in for the build. When building with the former I ma= de sure > your patches were disabled, so the Cygwin malloc would be used for this c= ase. > > I had no difficulty running either version of bwa to completion. On one = smallish > test machine the rpmalloc version finished in a bit less elapsed time but= with the > same CPU time as the Cygwin malloc version. > > I also ran on a larger system; here both versions ran with similar elapse= d and CPU > times. I also ran the Cygwin malloc version with '-t 32' to add some str= ess but > still your test case ran to successful completion. > > So I'm afraid I can't explain the results you were seeing. Is it possibl= e that > you might have given up too soon running the Cygwin malloc version, think= ing you > should be seeing output as quickly as you would on Linux? You won't, unf= ortunately. > > You might try backing out your changes, or I think, building again on you= r main > branch, to see if waiting longer proves successful. If you have any othe= r > suggestions, please let us know. > Thanks & Regards, > > ..mark > > P.S. Here's Cygwin malloc version's output from my smallish system > (i5, 2.3GHz, 2C/4T)... > ./bwa mem -t 10 bwa_reference/hs37d5.fa /tmp/ERS4238880_1.fastq > test1.s= am > [M::bwa_idx_load_from_disk] read 0 ALT contigs > [M::process] read 1712342 sequences (100000087 bp)... > [M::process] read 1103688 sequences (64503600 bp)... > [M::mem_process_seqs] Processed 1712342 reads in 2157.077 CPU sec, 2214.8= 71 real sec > [M::mem_process_seqs] Processed 1103688 reads in 1541.766 CPU sec, 1591.7= 04 real sec > [main] Version: 0.7.17-r1198-dirty > [main] CMD: ./bwa mem -t 10 bwa_reference/hs37d5.fa /tmp/ERS4238880_1.fas= tq > [main] Real time: 3831.624 sec; CPU: 3713.937 sec