From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtpout2.vodafonemail.de (smtpout2.vodafonemail.de [145.253.239.133]) by sourceware.org (Postfix) with ESMTPS id DEC993858006 for ; Mon, 18 Jan 2021 20:07:02 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org DEC993858006 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=nexgo.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=Stromeko@nexgo.de Received: from smtp.vodafone.de (unknown [10.2.0.34]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id 48419123672 for ; Mon, 18 Jan 2021 21:07:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nexgo.de; s=vfde-smtpout-mb-15sep; t=1611000420; bh=DLBc/pbkhYLThhUaA2HRmqrWgGzb76HhlKbwF2T702c=; h=From:To:Subject:References:Date:In-Reply-To; b=aoUju6xnENenkRqaUMIC6Eskkk/84ClimDajAfAqoyEN+se3WIo8DeREt3GGxtTRU jel1WDE3akKVvN8oG5feQ1aaqsHfYGdKqn6Gc/asE8lcZ7sS4aVZfKoLewgw1rm4tB SoVjeQUO0cTzAiMFmDAOfPNXKRrkcypvaI+FiE5I= Received: from Gertrud (p54a0ca05.dip0.t-ipconnect.de [84.160.202.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp.vodafone.de (Postfix) with ESMTPSA id CB7301420A9 for ; Mon, 18 Jan 2021 20:06:59 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: Re: Extreme slowdown due to malloc? References: <87mty66fw5.fsf@Rainer.invalid> <012a9e3c-ec24-f307-a3c4-9f2589d54e34@maxrnd.com> <87k0tae4cm.fsf@Otto.invalid> <87eej3beys.fsf@Rainer.invalid> Date: Mon, 18 Jan 2021 21:06:44 +0100 In-Reply-To: (Mark Geisert's message of "Sun, 17 Jan 2021 23:07:24 -0800") Message-ID: <877doa57sb.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-purgate-size: 1640 X-purgate-ID: 155817::1611000419-00003440-9BB485C5/0/0 X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-apps@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin package maintainer discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 20:07:05 -0000 Mark Geisert writes: > The page fault numbers are comparable to what you've shown for Cygwin > on your system. The long pause after zstd prints "Constructing > partial suffix array" is because zstd is cpu-bound in qsort() for a > long time. No paging during that time. Then there's probably another performance bug in qsort since that step finishes much faster on Linux too. > Then when the statistics start being printed out, that's when the > paging insanity starts. OK, so you see the same thing. Good. > What I discovered is that zstd is repeatedly asking malloc() for large > memory blocks, presumably to mmap files in, then free()ing them. Oh, so it's not recycling those=E2=80=A6 maybe that's not very hard to chan= ge. > Any malloc request 256K or larger is fulfilled by mmap() rather than > enlarging the heap for it. But crucially, there is no mechanism for > our malloc to hang on to freed mmap()ed pages for future use. If you > free an mmap()ed block, it is unmap()ed immediately. So for zstd's > usage pattern you get an incredible number of page faults to satisfy > the mmap()s and Windows seems to take a non-trivial bit of time for > each mmap(). It probably puts backing store behind it even when there's enough memory. Come to think of it I should try to move the page file to the NVMe drive (off the SSD it's on currently) and see if that changes things. Regards, Achim. --=20 +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds