From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 12797 invoked by alias); 17 Aug 2018 18:49:23 -0000 Mailing-List: contact cygwin-apps-help@cygwin.com; run by ezmlm Precedence: bulk Sender: cygwin-apps-owner@cygwin.com List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Mail-Followup-To: cygwin-apps@cygwin.com Received: (qmail 12762 invoked by uid 89); 17 Aug 2018 18:49:22 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=listening, mass, ultra, expense X-HELO: vsmx009.vodafonemail.xion.oxcs.net Received: from vsmx009.vodafonemail.xion.oxcs.net (HELO vsmx009.vodafonemail.xion.oxcs.net) (153.92.174.87) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Aug 2018 18:49:20 +0000 Received: from vsmx001.vodafonemail.xion.oxcs.net (unknown [192.168.75.191]) by mta-5-out.mta.xion.oxcs.net (Postfix) with ESMTP id B3077C51C9 for ; Fri, 17 Aug 2018 18:49:18 +0000 (UTC) Received: from Gertrud (unknown [87.185.210.65]) by mta-5-out.mta.xion.oxcs.net (Postfix) with ESMTPA id 894A9300304 for ; Fri, 17 Aug 2018 18:49:16 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: Re: Zstandard support for setup References: <874lg0d6l8.fsf@Rainer.invalid> Date: Fri, 17 Aug 2018 18:49:00 -0000 In-Reply-To: <874lg0d6l8.fsf@Rainer.invalid> (Achim Gratz's message of "Sat, 11 Aug 2018 21:52:03 +0200") Message-ID: <876008u8us.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-SW-Source: 2018-08/txt/msg00037.txt.bz2 While it seems like I'm talking to myself, in case anybody is listening: With the "normal" maximum compression level of 19 the package archive for my installation (including sources and debuginfo) got about 10% larger, taking about the same time as Xz for compression. Using the "ultra" compression level 22 gets that difference to about 3% at the expense of twice the CPU time for the compression. My repo is about 8GiB compressed and it took about 8 (resp. 16 hours) of single-core CPU time to re-compress (uncompress from BZip2 or Xz and recompress on-the-fly to ZStdandard). The higher compression level needs just a bit over 1.5GiB on average and around 3GiB peak. I've had 16 cores to run that on and those stayed at 2.8GHz all the time, so it didn't take too long. Here at home I've tasked two slower 4-core machines with it, so they were running a few hours of wall-time. I might try long-range matching to get even better compression at work since I've got 128GB memory now to see if there's an improvement. At home I've run out of 8GiB memory more than once even with a single compression for the largest debuginfo packages. Incidentally, these packages always take the longest time to compress, so if you want to do a parallel mass conversion it's advisable to start these first (keeping an eye on available memory) and have the faster compression of the smaller package archives fill the remaining CPU/time. I've installed and updated over a dozen of machines by now with this, so I'm reasonably certain that the implementation in setup is OK. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada