From: Achim Gratz <Stromeko@nexgo.de>
To: cygwin-apps@cygwin.com
Subject: Re: Extreme slowdown due to malloc?
Date: Mon, 18 Jan 2021 21:06:44 +0100 [thread overview]
Message-ID: <877doa57sb.fsf@Rainer.invalid> (raw)
In-Reply-To: <cf2ade9f-a2fa-ed59-f754-9f558b66ef2d@maxrnd.com> (Mark Geisert's message of "Sun, 17 Jan 2021 23:07:24 -0800")
Mark Geisert writes:
> The page fault numbers are comparable to what you've shown for Cygwin
> on your system. The long pause after zstd prints "Constructing
> partial suffix array" is because zstd is cpu-bound in qsort() for a
> long time. No paging during that time.
Then there's probably another performance bug in qsort since that step
finishes much faster on Linux too.
> Then when the statistics start being printed out, that's when the
> paging insanity starts.
OK, so you see the same thing. Good.
> What I discovered is that zstd is repeatedly asking malloc() for large
> memory blocks, presumably to mmap files in, then free()ing them.
Oh, so it's not recycling those… maybe that's not very hard to change.
> Any malloc request 256K or larger is fulfilled by mmap() rather than
> enlarging the heap for it. But crucially, there is no mechanism for
> our malloc to hang on to freed mmap()ed pages for future use. If you
> free an mmap()ed block, it is unmap()ed immediately. So for zstd's
> usage pattern you get an incredible number of page faults to satisfy
> the mmap()s and Windows seems to take a non-trivial bit of time for
> each mmap().
It probably puts backing store behind it even when there's enough
memory. Come to think of it I should try to move the page file to the
NVMe drive (off the SSD it's on currently) and see if that changes
things.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
prev parent reply other threads:[~2021-01-18 20:07 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-21 20:52 Achim Gratz
2020-12-22 4:37 ` Mark Geisert
2020-12-22 6:34 ` ASSI
2021-01-02 14:17 ` Achim Gratz
2021-01-18 7:07 ` Mark Geisert
2021-01-18 20:06 ` Achim Gratz [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=877doa57sb.fsf@Rainer.invalid \
--to=stromeko@nexgo.de \
--cc=cygwin-apps@cygwin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).