From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 125290 invoked by alias); 23 Aug 2018 20:40:46 -0000 Mailing-List: contact cygwin-apps-help@cygwin.com; run by ezmlm Precedence: bulk Sender: cygwin-apps-owner@cygwin.com List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Mail-Followup-To: cygwin-apps@cygwin.com Received: (qmail 125265 invoked by uid 89); 23 Aug 2018 20:40:45 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=archives, act, Hx-spam-relays-external:ESMTPA X-HELO: vsmx009.vodafonemail.xion.oxcs.net Received: from Unknown (HELO vsmx009.vodafonemail.xion.oxcs.net) (153.92.174.87) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 23 Aug 2018 20:40:43 +0000 Received: from vsmx001.vodafonemail.xion.oxcs.net (unknown [192.168.75.191]) by mta-5-out.mta.xion.oxcs.net (Postfix) with ESMTP id E82BDD6F39 for ; Thu, 23 Aug 2018 20:40:34 +0000 (UTC) Received: from Gertrud (unknown [87.185.215.3]) by mta-5-out.mta.xion.oxcs.net (Postfix) with ESMTPA id C0B433005F7 for ; Thu, 23 Aug 2018 20:40:32 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: Re: Zstandard support for setup References: <874lg0d6l8.fsf@Rainer.invalid> Date: Thu, 23 Aug 2018 20:40:00 -0000 In-Reply-To: <874lg0d6l8.fsf@Rainer.invalid> (Achim Gratz's message of "Sat, 11 Aug 2018 21:52:03 +0200") Message-ID: <878t4wolz7.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-SW-Source: 2018-08/txt/msg00041.txt.bz2 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-length: 1798 [=E2=80=A6] Here's a bit of support scaffolding for anybody who wants to play with this (if the looks of the scripts scare you, stay away from it :-), obviously you'll also need to compile yourself a setup.exe with the patches to use with. It assumes the Cygwin local repo files / cache in a directory named install and will produce a re-compressed copy in install-zstd. I have these in /mnt/upload in my examples (they're also not exact copies of the Cygwin mirror repos since I maintain these with another script in order to mix in local packages as needed, so I hope it just works on a direct copy as well, but have not tested that). The new hierarchy should be initially empty and the scripts assume that there's nothing else in there that you want to keep. Again, anything already in install-zstd can get deleted when running the scripts, so don't keep your bitcoins there. Then these invocations will re-compress all archives from the original and rewrite the setup.ini files so you can later point setup to install from install-zstd: /mnt/upload/maint-zstd.pl | parallel /mnt/upload/recompress.pl ls install/x86*/setup*xz | parallel /mnt/upload/rewrite-setup.pl That is GNU parallel above, which uses all available CPU without any parameters, you can do the same with moreutils parallel or xargs with the appropriate invocation. There's output to STDERR going around the pipe that keeps you informed about what's going on. After each update of the Cygwin repo you'll have to re-run the scripts, which will then only act upon new or updated files (and remove those files and directories that were deleted, if any). It will always re-calculate all the SHA512 checksums at the moment, though (I might implement some sort of cache for that later, but it doesn't bother me too much). --=-=-= Content-Type: application/x-perl Content-Disposition: attachment; filename=maint-zstd.pl Content-Transfer-Encoding: base64 Content-length: 3323 IyEvdXNyL2Jpbi9wZXJsIC13CnVzZSBjb21tb246OnNlbnNlOwoKdXNlIERh dGVUaW1lOwp1c2UgRmlsZTo6QmFzZW5hbWUgcXcoIGZpbGVwYXJzZSApOwp1 c2UgRmlsZTo6RmluZCBxdyggZmluZCApOwp1c2UgRmlsZTo6U3BlYzo6RnVu Y3Rpb25zIHF3KCBjYXRmaWxlIGNhdGRpciBzcGxpdGRpciApOwp1c2UgRmls ZTo6UGF0aCBxdyhtYWtlX3BhdGgpOwoKbXkgKCAkZGlycywgJGZpbGVzLCAk cm1kaXJzLCAkcm1maWxlcywgJHpzdGRmaWxlcyApID0gKCB7fSwge30sIHt9 LCB7fSwge30gKTsKCmZpbmQoCiAgICBzdWIgewoJaWYgKG0vXEFzZXR1cF9j dXJyZW50XFovKSB7CgkgICAgJEZpbGU6OkZpbmQ6OnBydW5lID0gMTsKCSAg ICByZXR1cm47Cgl9CglteSAoICRmZCwgJGZkbiwgJGZuICkgPSAoICRGaWxl OjpGaW5kOjpkaXIsICRGaWxlOjpGaW5kOjpuYW1lLCAkXyApOwoJbXkgQGZz ID0gbHN0YXQgJF87CglpZiAoLWYgXykgewoJICAgIG15ICggJGJuLCB1bmRl ZiwgJGJzICApID0gZmlsZXBhcnNlKCAkZm4sIHF3KCAuYnoyIC54eiAuenN0 ICkgKTsKCSAgICByZXR1cm4gdW5sZXNzICRiczsKCSAgICBteSBAcGQgPSBz cGxpdGRpciAkZmQ7CgkgICAgbXkgJHBkbiA9IGNhdGZpbGUgQHBkWzEuLiQj cGRdLCAkYm47CiMJICAgIHNheSAiJGZkID0+ICRwZFswXSAoJGJzKTogJHBk biA9ICRmc1s3XSIgaWYgKC9ndWlsZS8pOwoJICAgICRmaWxlcy0+eyRwZG59 eyRwZFswXX0gPSBbICRicywgQGZzWzcsOV0gXTsKCX0gZWxzaWYgKC1kIF8p IHsKCSAgICBteSBAcGQgPSBzcGxpdGRpciAkZmRuOwoJICAgIG15ICRwZG4g PSBjYXRkaXIgQHBkWzEuLiQjcGRdOwojCSAgICBzYXkgIiRmZG4gPT4gJHBk WzBdOiAkcGRuIiBpZiAoL2d1aWxlLyk7CgkgICAgJGRpcnMtPnskcGRufXsk cGRbMF19Kys7Cgl9CiAgICB9LCBxdyggaW5zdGFsbCBpbnN0YWxsLXpzdGQg KSk7CmZvcmVhY2ggbXkgJGZuICggcmV2ZXJzZSBzb3J0IGtleXMgJSRmaWxl cyApIHsKICAgIG15ICVmID0gJXskZmlsZXMtPnskZm59fTsKICAgIG15ICgg JGksICR6ICkgPSBAZntxdyggaW5zdGFsbCBpbnN0YWxsLXpzdGQgKX07CiAg ICBpZiAoZGVmaW5lZCAkaSBhbmQgbm90IGRlZmluZWQgJHopIHsKIwlzYXkg Ik9ubHkgaW4gaW5zdGFsbDogICAgICAkZm4iOwoJIyBtYWtlIHBhdGggJiBy ZWNvbXByZXNzIGZpbGUKCSR6c3RkZmlsZXMtPnskZm59ID0gJGk7CiAgICB9 IGVsc2lmIChkZWZpbmVkICR6IGFuZCBub3QgZGVmaW5lZCAkaSkgewojCXNh eSAiT25seSBpbiBpbnN0YWxsLXpzdGQ6ICRmbiI7CgkjIHJlbW92ZSBmaWxl Cgkkcm1maWxlcy0+eyRmbn0rKzsKICAgIH0gZWxzZSB7CgkjIG1ha2UgcGF0 aCAmIHJlY29tcHJlc3MgaWYgenN0ZCBmaWxlIGlzIG9sZGVyCgkkenN0ZGZp bGVzLT57JGZufSA9ICRpIGlmICgkaS0+WzJdIGd0ICR6LT5bMl0pOwogICAg fQp9CmZvcmVhY2ggbXkgJGRuICggcmV2ZXJzZSBzb3J0IGtleXMgJSRkaXJz ICkgewogICAgbXkgJWQgPSAleyRkaXJzLT57JGRufX07CiAgICBteSAoICRp LCAkeiApID0gQGR7cXcoIGluc3RhbGwgaW5zdGFsbC16c3RkICl9OwojICAg IHNheSAiJGRuIiBpZiAoJGRuID1+IG0vZ3VpbGUvKTsKICAgIGlmIChkZWZp bmVkICRpIGFuZCBub3QgZGVmaW5lZCAkeikgewojCXNheSAiT25seSBpbiBp bnN0YWxsOiAgICAgICRkbiI7CgkjIG1ha2UgcGF0aAoJc2F5IFNUREVSUiAi bWtkaXIgaW5zdGFsbC16c3RkLyRkbi56c3QiOwoJbWFrZV9wYXRoICJpbnN0 YWxsLXpzdGQvIiAuICRkbjsKICAgIH0gZWxzaWYgKGRlZmluZWQgJHogYW5k IG5vdCBkZWZpbmVkICRpKSB7CiMJc2F5ICJPbmx5IGluIGluc3RhbGwtenN0 ZDogJGRuIiA7CgkjIHJlbW92ZSBkaXJlY3RvcnkKCSRybWRpcnMtPnskZG59 Kys7CQogICAgfQp9CmZvcmVhY2ggbXkgJGZuICggcmV2ZXJzZSBzb3J0IGtl eXMgJSRybWZpbGVzICkgewogICAgc2F5IFNUREVSUiAicm0gICAgaW5zdGFs bC16c3RkLyRmbiI7CiAgICB1bmxpbmsgImluc3RhbGwtenN0ZC8iIC4gJGZu IC4gIi56c3QiOwp9CmZvcmVhY2ggbXkgJGRuICggcmV2ZXJzZSBzb3J0IGtl eXMgJSRybWRpcnMgKSB7CiAgICBzYXkgU1RERVJSICJybWRpciBpbnN0YWxs LXpzdGQvJGRuIjsKICAgIHJtZGlyICJpbnN0YWxsLXpzdGQvIiAuICRkbjsK fQpmb3JlYWNoIG15ICR6biAoIHNvcnQgeyAkenN0ZGZpbGVzLT57JGJ9WzFd IDw9PiAkenN0ZGZpbGVzLT57JGF9WzFdIH0ga2V5cyAlJHpzdGRmaWxlcyAp IHsKICAgIHNheSBTVERFUlIgc3ByaW50ZiggIiUxMmkiLCAkenN0ZGZpbGVz LT57JHpufVsxXSApLCAiICMgIiwKCURhdGVUaW1lLT5mcm9tX2Vwb2NoKGVw b2NoPT4kenN0ZGZpbGVzLT57JHpufVsyXSktPmlzbzg2MDEoKSwgIlx0IiwK CSIkem4kenN0ZGZpbGVzLT57JHpufVswXSI7CiAgICBzYXkgIiR6biR6c3Rk ZmlsZXMtPnskem59WzBdIjsKfQo= --=-=-= Content-Type: application/x-perl Content-Disposition: attachment; filename=recompress.pl Content-Transfer-Encoding: base64 Content-length: 627 IyEvdXNyL2Jpbi9wZXJsIC13CnVzZSBjb21tb246OnNlbnNlOwoKdXNlIElP OjpVbmNvbXByZXNzOjpBbnlVbmNvbXByZXNzIHF3KCBhbnl1bmNvbXByZXNz ICRBbnlVbmNvbXByZXNzRXJyb3IgKTsKCm15ICR0YiA9IHNoaWZ0OyAkdGIg PX4gczpeXC4vOjo7Cm15ICRvZiA9ICJpbnN0YWxsLyIuJHRiOwpteSAkemYg PSAiaW5zdGFsbC16c3RkLyIuJHRiOwokemYgPX4gcy9cLih4enxiejIpLy56 c3QvOwojZXhpdCAwIGlmIC1zICR6ZjsKI3NheSAiJHRiXG5cdCRvZlxuICA9 PT5cdCR6ZiI7CiNleGl0IDA7Cm15ICRmaF9vID0gSU86OlVuY29tcHJlc3M6 OkFueVVuY29tcHJlc3MtPm5ldyggJG9mICk7Cm9wZW4gbXkgJGZoX3osICJ8 LSIsIHF3KCAvdXNyL2Jpbi96c3RkIC0xOSAtZm8gKSwgJHpmOwpiaW5tb2Rl ICRmaF96OwpzZWxlY3QgJGZoX3o7CndoaWxlICg8JGZoX28+KSB7CiBwcmlu dDsKfQogICAgCg== --=-=-= Content-Type: application/x-perl Content-Disposition: attachment; filename=rewrite-setup.pl Content-Transfer-Encoding: base64 Content-length: 1635 IyFwZXJsIC13CnVzZSBjb21tb246OnNlbnNlOwoKdXNlIElPOjpVbmNvbXBy ZXNzOjpBbnlVbmNvbXByZXNzIHF3KGFueXVuY29tcHJlc3MgJEFueVVuY29t cHJlc3NFcnJvcik7CiMgY2hlY2tzdW1zCnVzZSBEaWdlc3Q6OlNIQSBxdygg c2hhNTEyICk7CnVzZSBNSU1FOjpCYXNlNjQgcXcoIGVuY29kZV9iYXNlNjR1 cmwgKTsKCm15ICggJGRzLCAkb3MsICR6cyApID0gKCAwLCAwLCAwICk7Cm15 ICRvZm4gPSBzaGlmdDsKJG9mbiA9ICJpbnN0YWxsLyIgLiAkb2ZuIHVubGVz cyAoJG9mbiA9fiBtOl5pbnN0YWxsLzopOwpteSAkT0ZIID0gbmV3IElPOjpV bmNvbXByZXNzOjpBbnlVbmNvbXByZXNzICRvZm4KICAgIG9yIGRpZSAiQ2Fu bm90IG9wZW4gJG9mbiBmb3IgcmVhZGluZy5cblx0JCEiOwpteSAkemZuID0g JG9mbjsKJHpmbiA9fiBzOmluc3RhbGwvOmluc3RhbGwtenN0ZC86OwokemZu ID1+IHM6XC54eiQ6LnpzdDo7CiNzYXkgU1RET1VUICIkb2ZuID0+ICR6Zm4i OwpvcGVuIG15ICRaRkgsICJ8LSIsIHF3KCAvdXNyL2Jpbi96c3RkIC0tdWx0 cmEgLTE5IC1mbyApLCAkemZuOwpiaW5tb2RlICRaRkg7CnNlbGVjdCAkWkZI Owp3aGlsZSAoPCRPRkg+KSB7CiAgICBzL15yZWxlYXNlOiBzZXR1cF9jdXJy ZW50L3JlbGVhc2U6IHNldHVwLXpzdGQvOwogICAgaWYgKG0vXmluc3RhbGw6 LykgewoJbXkgKHVuZGVmLCAkZm4sICRmcywgJHNmbikgPSBzcGxpdDsKCSRm biA9fiBzOlwuKHh6fGJ6MikkOi56c3Q6OwoJJHNmbiA9ICRmbjsKCSRzZm4g PX4gczpeXC5cLjppbnN0YWxsLXpzdGQ6OwoJJHNmbiA9fiBzIV4oPz0oeDg2 fHg4Nl82NHxub2FyY2gpLykhaW5zdGFsbC16c3RkLyE7CiMJc2F5IFNURE9V VCAiJGZuID0+ICRzZm4iOwoJbXkgQGZzID0gc3RhdCAkc2ZuOwoJJG9zICs9 ICRmczsKCSRkcyAtPSAkZnMgLSAkZnNbN107CgkkZnMgID0gJGZzWzddOwoJ JHpzICs9ICRmczsKCW15ICRzaGE1MTIgPSBlbmNvZGVfYmFzZTY0dXJsKCBE aWdlc3Q6OlNIQS0+bmV3KCAiU0hBLTUxMiIgKS0+YWRkZmlsZSggJHNmbiAp LT5kaWdlc3QgKTsKCXNheSAiaW5zdGFsbDogJGZuICRmcyAkc2hhNTEyIjsK ICAgIH0gZWxzZSB7CglwcmludDsKICAgIH0KfQpzZWxlY3QgU1RET1VUOwpz YXkgIlVzZWQgJGRzIGFkZGl0aW9uYWwgYnl0ZXM6ICRvcyA9PiAkenMgKCIs ICgoMTAwLiokenMpLyRvcyksICIlKSBpbiAkemZuLiI7Cg== --=-=-= Content-Type: text/plain Content-length: 191 Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables --=-=-=--