From: "Teemu Nätkinniemi" <tnatkinn@gmail.com>
To: Mark Geisert <mark@maxrnd.com>
Cc: "cygwin-developers@cygwin.com" <cygwin-developers@cygwin.com>
Subject: Re: Cygwin malloc tune-up status
Date: Thu, 1 Apr 2021 12:19:53 +0300 [thread overview]
Message-ID: <CAEHcbmdj=TeUhO+MVwf6QcKZTRX6d2_bHwaPF6Yu1iWMy0fsEw@mail.gmail.com> (raw)
In-Reply-To: <d02d00af-8e6d-b3eb-7061-3ac68da7ec2c@maxrnd.com>
ti 29. syysk. 2020 klo 5.23 Mark Geisert (mark@maxrnd.com) kirjoitti:
>
> Johannes Schindelin wrote:
> > Hi Mark,
> > On Thu, 24 Sep 2020, Mark Geisert wrote:
> >> I've been looking into two potential enhancements of Cygwin's malloc operation
> >> for Cygwin 3.2. The first is to replace the existing function-level locking
> >> with something more fine-grained to help threaded processes; the second is to
> >> implement thread-specific memory pools with the aim of lessening lock activity
> >> even further.
> >>
> >> Although I've investigated several alternative malloc packages, including
> >> ptmalloc[23], nedalloc, and Windows Heaps, only the latter seems to improve on
> >> the performance of Cygwin's malloc. Unfortunately using Windows Heaps would
> >> require fiddling with undocumented heap structures to enable use with fork().
> >> I also looked at BSD's jemalloc and Google's tcmalloc. Those two require much
> >> more work to port to Cygwin so I've back-burnered them for the time being.
> >
> > I am just a lurker when it comes to your project, but I wonder whether you
> > had any chance to look into mimalloc
> > (https://github.com/microsoft/mimalloc)? I had investigated it in Git for
> > Windows' context for a while (because nedmalloc, which is used by Git for
> > Windows, is no longer actively maintained).
>
> Hi Johannes,
> Great minds think alike! Yours is the 3rd pointer I've received on- and off-list
> towards mimalloc. I had not heard of it before. I've looked into it and have now
> added it to my back-burnered list.
>
> mimalloc looks promising. It's fairly small. It has at least one issue it shares
> with jemalloc and tcmalloc: initialization code that needs to run before the
> Cygwin DLL has completely set up a new process' environment. A chicken-and-egg
> problem, if you will. A solution to that (which I'm pondering) will allow me to
> test all three malloc alternatives in the future.
>
Hi!
I encounter a problem with Cygwin's malloc and remembered this thread.
Sorry if this is off-topic.
I have been trying to port bwa aligner to Cygwin. Initially everything
seemed to work but for some reason in some cases threading didn't seem
to work properly. I got a fix recently from a third party which was to
force bwa to use rpmalloc.
This got me thinking if there is a problem with Cygwin's malloc in
some cases and if there were people in this list who might be
interested in knowing that the problem exists.
Here's a link to the rpmalloc fix.
https://github.com/WGSExtract/bwa/commit/3087fa876b079fcb6a0a58f1e01757f4820094a8
Here's a test case:
bwa_original mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam
bwa_working mem -t 10 bwa_reference/hs37d5.fa ERS4238880_1.fastq > test.sam
Files:
https://drive.google.com/drive/folders/1waICih51f4mHZEyWY1onyEcKqm0kj3Yt?usp=sharing
Thanks,
Teemu
next prev parent reply other threads:[~2021-04-01 9:20 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-25 6:01 Mark Geisert
2020-09-27 16:54 ` Johannes Schindelin
2020-09-29 2:22 ` Mark Geisert
2021-04-01 9:19 ` Teemu Nätkinniemi [this message]
2021-04-02 5:45 ` Maybe consider rpmalloc -- Was: " Mark Geisert
2021-04-03 2:53 ` Maybe consider rpmalloc Mark Geisert
2021-04-03 6:46 ` Teemu Nätkinniemi
2021-04-03 6:48 ` Teemu Nätkinniemi
2021-04-11 9:28 ` Mark Geisert
2021-04-12 8:48 ` Teemu Nätkinniemi
2021-04-13 8:24 ` Mark Geisert
2021-04-13 13:05 ` Teemu Nätkinniemi
2021-04-14 8:19 ` Mark Geisert
2021-04-14 18:36 ` Teemu Nätkinniemi
2021-04-14 18:53 ` Jon Turney
2021-04-19 5:16 ` Mark Geisert
2021-04-20 19:34 ` Jon Turney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAEHcbmdj=TeUhO+MVwf6QcKZTRX6d2_bHwaPF6Yu1iWMy0fsEw@mail.gmail.com' \
--to=tnatkinn@gmail.com \
--cc=cygwin-developers@cygwin.com \
--cc=mark@maxrnd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).