Maybe consider rpmalloc -- Was: Re: Cygwin malloc tune-up status

[Mark Geisert <mark@maxrnd.com>]

Hi Teemu,

Teemu Nätkinniemi via Cygwin-developers wrote:
> Hi!
> 
> I encounter a problem with Cygwin's malloc and remembered this thread.
> Sorry if this is off-topic.

New topic, so new thread :-)

> I have been trying to port bwa aligner to Cygwin. Initially everything
> seemed to work but for some reason in some cases threading didn't seem
> to work properly. I got a fix recently from a third party which was to
> force bwa to use rpmalloc.
> 
> This got me thinking if there is a problem with Cygwin's malloc in
> some cases and if there were people in this list who might be
> interested in knowing that the problem exists. >
> Here's a link to the rpmalloc fix.
> 
> https://github.com/WGSExtract/bwa/commit/3087fa876b079fcb6a0a58f1e01757f4820094a8
> 
> Here's a test case:
> 
> bwa_original mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam
> 
> bwa_working mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam
> 
> Files:
> 
> https://drive.google.com/drive/folders/1waICih51f4mHZEyWY1onyEcKqm0kj3Yt?usp=sharing

I'm not aware of anything actually being broken in the current Cygwin malloc 
(which is just the most recent dlmalloc), even in multi-threaded workloads.  It's 
only that there has long been interest around possibilities of making it faster. 
So far I've been unsuccessful: though I can get the Cygwin-level malloc operation 
faster by using other malloc's, it has always been overshadowed by a lot more time 
being spent in ntdll.dll underneath.  There's no point releasing something 
cosmetically better but slower in practice.

Thanks for mentioning rpmalloc; that's a new one to me.  It does appear to be 
coded nicely, like most of the other malloc's I've looked at.  I'll see if I can 
get that to work with Cygwin; the big issue tends to be fork() having to replicate 
the parent malloc layout, book-keeping as well as app data, in the child.

I will point out that you're not using rpmalloc as it would work on Cygwin, you've 
got a Windows-flavored rpmalloc because you've #defined _WIN32 and such via the 
Makefile.cygwin.  But it appears to be a clean break with Cygwin's malloc in your 
application, so congrats on that.

I understand that you've solved what you think was a Cygwin malloc issue by using 
rpmalloc, but I don't see how you came to the conclusion that it was a malloc 
issue, as opposed to something else with threads or beyond that.
Thanks again,

..mark