From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 34642 invoked by alias); 22 Aug 2018 19:12:28 -0000 Mailing-List: contact cygwin-apps-help@cygwin.com; run by ezmlm Precedence: bulk Sender: cygwin-apps-owner@cygwin.com List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Mail-Followup-To: cygwin-apps@cygwin.com Received: (qmail 34626 invoked by uid 89); 22 Aug 2018 19:12:27 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.9 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=backed, promptly, WAVE, peak X-HELO: vsmx009.vodafonemail.xion.oxcs.net Received: from Unknown (HELO vsmx009.vodafonemail.xion.oxcs.net) (153.92.174.87) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 22 Aug 2018 19:12:25 +0000 Received: from vsmx001.vodafonemail.xion.oxcs.net (unknown [192.168.75.191]) by mta-5-out.mta.xion.oxcs.net (Postfix) with ESMTP id 40E58D021F for ; Wed, 22 Aug 2018 19:12:17 +0000 (UTC) Received: from Gertrud (unknown [87.185.215.3]) by mta-5-out.mta.xion.oxcs.net (Postfix) with ESMTPA id 19C10300688 for ; Wed, 22 Aug 2018 19:12:14 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: Re: Zstandard support for setup References: <874lg0d6l8.fsf@Rainer.invalid> <876008u8us.fsf@Rainer.invalid> Date: Wed, 22 Aug 2018 19:12:00 -0000 In-Reply-To: <876008u8us.fsf@Rainer.invalid> (Achim Gratz's message of "Fri, 17 Aug 2018 20:49:15 +0200") Message-ID: <87lg8yfc6t.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-SW-Source: 2018-08/txt/msg00040.txt.bz2 Achim Gratz writes: > I've had 16 cores to run that on and those stayed at 2.8GHz all the > time, so it didn't take too long. Here at home I've tasked two slower > 4-core machines with it, so they were running a few hours of wall-time. > I might try long-range matching to get even better compression at work > since I've got 128GB memory now to see if there's an improvement. At > home I've run out of 8GiB memory more than once even with a single > compression for the largest debuginfo packages. I used some spare cycles today to look into that. It turns out that if you enable long-range matching, each zstd process reserves about 12GB of memory, which Windows insists to "commit", i.e. your page file must be large enough to hold all committed memory. If it's dynamically sized it will never grow larger than some fixed percentage of the free space on that drive, so I've ran into that limit promptly. It never actually uses that memory best I can tell, the actual VM used stays below 4GB per zstd process. So I just blew up the pagefile.sys manually to 384GiB (224GiB would have sufficed, but anyway) and started 20 parallel compressions. I used up 48GiB peak of the 128GiB in the machine and was done in less than 50 minutes (the four extra threads not backed by a CPU helped to keep the machine loaded while it was spinning up new compressions when it came to the smaller files which are dominated by I/O). The compression ration only improved about 1% vs. the non-long-range option to 103% vs. the original data, so using that option isn't really necessary or recommended. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves