Re: Maybe consider rpmalloc

["Teemu Nätkinniemi" <tnatkinn@gmail.com>]

Hello,

Thanks for testing! I found a better test case with smaller files
which should clearly show the issue.

https://drive.google.com/drive/folders/1jOilHtKrr6CHn7zg__DE93RCDyseoTB1?usp=sharing

Here's the results. Bwa_working is the one with rpmalloc and
bwa_original is unpatched. As you can see the unpatched version with
several threads takes a  whole lot more time to finish even when
compared with unpatched exe running with a single thread. I am not the
only one experiencing the issue so I doubt it is my system.

$ ../bwa-working/bwa_working.exe mem chr19_KI270866v1_alt.fasta
7859_GPI.read1.fq 7859_GPI.read2.fq > test1working.sam
(cut)
[main] Real time: 1.744 sec; CPU: 1.624 sec

$ ../bwa-working/bwa_working.exe mem -t 10 chr19_KI270866v1_alt.fasta
7859_GPI.read1.fq 7859_GPI.read2.fq > test1workingt10.sam
(cut)
[main] Real time: 0.354 sec; CPU: 2.218 sec

$ ../bwa-test/bwa_original.exe mem chr19_KI270866v1_alt.fasta
7859_GPI.read1.fq 7859_GPI.read2.fq > test1orig.sam
(cut)
[main] Real time: 1.733 sec; CPU: 1.608 sec

$ ../bwa-test/bwa_original.exe mem -t 10 chr19_KI270866v1_alt.fasta
7859_GPI.read1.fq 7859_GPI.read2.fq > test1origt10.sam
(cut)
[main] Real time: 8.131 sec; CPU: 5.265 sec

Teemu


su 11. huhtik. 2021 klo 12.52 Mark Geisert (mark@maxrnd.com) kirjoitti:
>
> Hi Teemu,
>
> Teemu Nätkinniemi via Cygwin-developers wrote:
> > Sorry, hurt my back yesterday and looks like I am not thinking clearly.
>
> Hope you are feeling better by this time.
>
> > ./bwa mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test1.sam
>
> Thanks.  It was my unfamiliarity with Google Drive which prevented my finding all
> the data files you had stored there.  After a while I did find all I needed.
>
> I rebuilt bwa.exe alternately using the provided Makefile and Makefile.cygwin.
> When building with the latter I made sure your #ifdef patches were enabled so that
> rpmalloc was pulled in for the build.  When building with the former I made sure
> your patches were disabled, so the Cygwin malloc would be used for this case.
>
> I had no difficulty running either version of bwa to completion.  On one smallish
> test machine the rpmalloc version finished in a bit less elapsed time but with the
> same CPU time as the Cygwin malloc version.
>
> I also ran on a larger system; here both versions ran with similar elapsed and CPU
> times.  I also ran the Cygwin malloc version with '-t 32' to add some stress but
> still your test case ran to successful completion.
>
> So I'm afraid I can't explain the results you were seeing.  Is it possible that
> you might have given up too soon running the Cygwin malloc version, thinking you
> should be seeing output as quickly as you would on Linux?  You won't, unfortunately.
>
> You might try backing out your changes, or I think, building again on your main
> branch, to see if waiting longer proves successful.  If you have any other
> suggestions, please let us know.
> Thanks & Regards,
>
> ..mark
>
> P.S. Here's Cygwin malloc version's output from my smallish system
> (i5, 2.3GHz, 2C/4T)...
> ./bwa mem -t 10 bwa_reference/hs37d5.fa /tmp/ERS4238880_1.fastq > test1.sam
> [M::bwa_idx_load_from_disk] read 0 ALT contigs
> [M::process] read 1712342 sequences (100000087 bp)...
> [M::process] read 1103688 sequences (64503600 bp)...
> [M::mem_process_seqs] Processed 1712342 reads in 2157.077 CPU sec, 2214.871 real sec
> [M::mem_process_seqs] Processed 1103688 reads in 1541.766 CPU sec, 1591.704 real sec
> [main] Version: 0.7.17-r1198-dirty
> [main] CMD: ./bwa mem -t 10 bwa_reference/hs37d5.fa /tmp/ERS4238880_1.fastq
> [main] Real time: 3831.624 sec; CPU: 3713.937 sec