From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtpout2.vodafonemail.de (smtpout2.vodafonemail.de [145.253.239.133]) by sourceware.org (Postfix) with ESMTPS id 0B25E385700D for ; Sat, 3 Jul 2021 13:17:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 0B25E385700D Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=nexgo.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=nexgo.de Received: from smtp.vodafone.de (smtpa03.fra-mediabeam.com [10.2.0.34]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id 223541209A0 for ; Sat, 3 Jul 2021 15:17:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nexgo.de; s=vfde-smtpout-mb-15sep; t=1625318226; bh=NVbE/JKtFglRDzApQ2lc8dml9RUU28LHKaL0zx3jfbU=; h=From:To:Subject:Date; b=Eh2NF5BrwvLLNCDSp5OI3bZlJ7+Lcfy9pOO7dclKLhCgE5WNcfd01OqIOxjfczNBT 1ae0qYZpCkaA2c++eZQVg2XKPhiH0lW1xJ3jVNGoUdmO1FEAka6XVRHCeoHAqYWcsV LnyB+VRCvFKbSMA9AotMkkTKGmA4zb0GxI6If3wo= Received: from Gertrud (p5b2f356a.dip0.t-ipconnect.de [91.47.53.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.vodafone.de (Postfix) with ESMTPSA id B28C4140258 for ; Sat, 3 Jul 2021 13:17:05 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: [RFC] zchunk Date: Sat, 03 Jul 2021 15:17:04 +0200 Message-ID: <874kdb1qr3.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-purgate-size: 2529 X-purgate-ID: 155817::1625318225-00000B26-8AF3A62A/0/0 X-Spam-Status: No, score=-3032.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-apps@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin package maintainer discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2021 13:17:09 -0000 I've finally cleaned up my build of zchunk on Cygwin and MingW64 (at the current version 1.1.16 now). The latter is losing some exectuables using POSIX interfaces that are not easily emulated since I am only interested in having libzck available anyway. Upstream: https://github.com/zchunk/zchunk (further links at the end of that page) If you want to have a look at the Cygwin packages: https://cygwin.com/git-cygwin-packages/?p=3Dgit/cygwin-packages/playground.= git;a=3Dshortlog;h=3Drefs/heads/zchunk https://ci.appveyor.com/project/cygwin/scallywag/builds/39855288/job/11vtsw= 0l7p5acmxw https://ci.appveyor.com/project/cygwin/scallywag/builds/39855288/job/2e0rcx= 2d9hdw6nm5 The reason I'm looking at zchunk is that it would be helpful with reducing the amount of initial data that needs to be transferred on an update with setup. When using an external dictionary (only needed on the server) on a slightly re-arranged setup.ini file (so I can create just one chunk per source package), I'm getting the following results: original: 3,4M setup64.zst chunked: 3,8M setup64-rechunk.ini.zck dictionary: 110K setup64-rechunk.zdict Delta from 2021-06-26 to 2021-07-03 Would download 141634 of 3960442 bytes Matched 4074 of 4087 chunks Delta from 2021-05-29 to 2021-07-03 Would download 198463 of 3960442 bytes Matched 4032 of 4087 chunks Delta from 2020-01-02 to 2021-07-03 Would download 1438581 of 3960442 bytes Matched 2938 of 4087 chunks So the chunked file is 400k or about 12% larger than the simply compressed setup.ini (that's the additional header and some loss in compression ratio due to the shorter blocks the compressor sees). On the other hand, each update request would typically only pull between 140k=E2=80=A6200k (or about 5% of the compressed setup.ini) if the local co= py is reasonably recent. Even an update of a system that's over a year old still only needs about half of the data for getting up-to-date. Zchunk would fold the download of the missing chunks (the update to the current version) into a single HTTP request (via ranges) most of the time after requesting and analysing just the header at the beginning of the file, so in terms of request rate the change would be negligible (about one or two additional roundtrips). Regards, Achim. --=20 +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada