From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 1859 invoked by alias); 11 Aug 2018 19:52:11 -0000 Mailing-List: contact cygwin-apps-help@cygwin.com; run by ezmlm Precedence: bulk Sender: cygwin-apps-owner@cygwin.com List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Mail-Followup-To: cygwin-apps@cygwin.com Received: (qmail 1842 invoked by uid 89); 11 Aug 2018 19:52:10 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.7 required=5.0 tests=AWL,BAYES_00,KAM_SHORT,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=yesterday, streaming, average, fresh X-HELO: vsmx012.vodafonemail.xion.oxcs.net Received: from vsmx012.vodafonemail.xion.oxcs.net (HELO vsmx012.vodafonemail.xion.oxcs.net) (153.92.174.90) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sat, 11 Aug 2018 19:52:08 +0000 Received: from vsmx004.vodafonemail.xion.oxcs.net (unknown [192.168.75.198]) by mta-8-out.mta.xion.oxcs.net (Postfix) with ESMTP id 1085F8CDC36 for ; Sat, 11 Aug 2018 19:52:06 +0000 (UTC) Received: from Gertrud (unknown [87.185.210.65]) by mta-8-out.mta.xion.oxcs.net (Postfix) with ESMTPA id E0033CDF79 for ; Sat, 11 Aug 2018 19:52:03 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: Zstandard support for setup Date: Sat, 11 Aug 2018 19:52:00 -0000 Message-ID: <874lg0d6l8.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-SW-Source: 2018-08/txt/msg00034.txt.bz2 I've finally implemented ZStandard support for setup and tested it today with a fresh install on a Celeron 1037U dual-core machine from and to local SSD. The archive extraction part of setup went from taking 23 minutes to about 14 minutes. A (much) faster machine at work did the same install yesterday in under 12 minutes, limited mainly by the latency of requesting those files from a remote filer (found a defect trunk NIC the admin didn't know about while doing that) and writing it out to iSCSI block storage. For large writes to disk I've seen it peek in the 200MiB/s range today, which is close to the performance limit for my SSD through the file system. However, for unpacking lots of smaller files the latency of the SSD limits the average write speed to around 35MiB/s (up from 20MiB/s). It would take a major reorganization of setup to allow for overlapped / multithreaded extraction in order to improve significantly on that, minor improvements might still be had with more aggressive buffering in memory to reduce read/write turnaround and perhaps by using zero-copy data streaming. The current code is available here: http://repo.or.cz/w/cygwin-setup/local.git The debug output statements are still in the code (although commented), I'll let that sit a while and see if I find something else I want to clean up before I submit it for the upstream repo. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs