Zstandard support for setup

Zstandard support for setup

[Achim Gratz]

I've finally implemented ZStandard support for setup and tested it today
with a fresh install on a Celeron 1037U dual-core machine from and to
local SSD.  The archive extraction part of setup went from taking 23
minutes to about 14 minutes.  A (much) faster machine at work did the
same install yesterday in under 12 minutes, limited mainly by the
latency of requesting those files from a remote filer (found a defect
trunk NIC the admin didn't know about while doing that) and writing it
out to iSCSI block storage.  For large writes to disk I've seen it peek
in the 200MiB/s range today, which is close to the performance limit for
my SSD through the file system.  However, for unpacking lots of smaller
files the latency of the SSD limits the average write speed to around
35MiB/s (up from 20MiB/s).  It would take a major reorganization of
setup to allow for overlapped / multithreaded extraction in order to
improve significantly on that, minor improvements might still be had
with more aggressive buffering in memory to reduce read/write turnaround
and perhaps by using zero-copy data streaming.

The current code is available here:

http://repo.or.cz/w/cygwin-setup/local.git

The debug output statements are still in the code (although commented),
I'll let that sit a while and see if I find something else I want to
clean up before I submit it for the upstream repo.

Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Zstandard support for setup

[Achim Gratz]

Achim Gratz writes:
> I've finally implemented ZStandard support for setup and tested it today
> with a fresh install on a Celeron 1037U dual-core machine from and to
> local SSD.  The archive extraction part of setup went from taking 23
> minutes to about 14 minutes.

This morning I've installed on the Win10 machine (same specs, different
box) over gigabit network, served from the Win8.1 machine from
yesterday.  The SHA512 check was taking 2:29 instead of 1:00, but the
archive extraction took just 13:08 (slightly faster than locally).

Maybe larger buffers and zero copy could have more leverage than I
thought they would…  can we do a lazy mmap on the input files?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Zstandard support for setup

[Achim Gratz]

While it seems like I'm talking to myself, in case anybody is listening:

With the "normal" maximum compression level of 19 the package archive
for my installation (including sources and debuginfo) got about 10%
larger, taking about the same time as Xz for compression.  Using the
"ultra" compression level 22 gets that difference to about 3% at the
expense of twice the CPU time for the compression.  My repo is about
8GiB compressed and it took about 8 (resp. 16 hours) of single-core CPU
time to re-compress (uncompress from BZip2 or Xz and recompress
on-the-fly to ZStdandard).  The higher compression level needs just a
bit over 1.5GiB on average and around 3GiB peak.

I've had 16 cores to run that on and those stayed at 2.8GHz all the
time, so it didn't take too long.  Here at home I've tasked two slower
4-core machines with it, so they were running a few hours of wall-time.
I might try long-range matching to get even better compression at work
since I've got 128GB memory now to see if there's an improvement.  At
home I've run out of 8GiB memory more than once even with a single
compression for the largest debuginfo packages.  Incidentally, these
packages always take the longest time to compress, so if you want to do
a parallel mass conversion it's advisable to start these first (keeping
an eye on available memory) and have the faster compression of the
smaller package archives fill the remaining CPU/time.

I've installed and updated over a dozen of machines by now with this, so
I'm reasonably certain that the implementation in setup is OK.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Zstandard support for setup

[Marco Atzeri]

Am 17.08.2018 um 20:49 schrieb Achim Gratz:
>
> While it seems like I'm talking to myself, in case anybody is listening:
>
>
> I've had 16 cores to run that on and those stayed at 2.8GHz all the
> time, so it didn't take too long.  Here at home I've tasked two slower
> 4-core machines with it, so they were running a few hours of wall-time.
> I might try long-range matching to get even better compression at work
> since I've got 128GB memory now to see if there's an improvement.  At
> home I've run out of 8GiB memory more than once even with a single
> compression for the largest debuginfo packages.  Incidentally, these
> packages always take the longest time to compress, so if you want to do
> a parallel mass conversion it's advisable to start these first (keeping
> an eye on available memory) and have the faster compression of the
> smaller package archives fill the remaining CPU/time.

memory and time requirements seem not so appealing to me
as my machine has only 8GiB and already the debuginfo on 32bit is
a time consuming effort


> I've installed and updated over a dozen of machines by now with this, so
> I'm reasonably certain that the implementation in setup is OK.
>
>
> Regards,
> Achim.
>

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: Zstandard support for setup

[Achim Gratz]

Marco Atzeri writes:
> memory and time requirements seem not so appealing to me
> as my machine has only 8GiB and already the debuginfo on 32bit is
> a time consuming effort

That happens only with ultra compression and long-range matching
enabled.  I've not run out of 6GiB memory across two parallel
compressions with the highest non-ultra settings and long-range matching
disabled.  That costs only about 7% vs. Xz compression, so it's still OK
I'd say.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Zstandard support for setup

[Achim Gratz]

Achim Gratz writes:
> I've had 16 cores to run that on and those stayed at 2.8GHz all the
> time, so it didn't take too long.  Here at home I've tasked two slower
> 4-core machines with it, so they were running a few hours of wall-time.
> I might try long-range matching to get even better compression at work
> since I've got 128GB memory now to see if there's an improvement.  At
> home I've run out of 8GiB memory more than once even with a single
> compression for the largest debuginfo packages.

I used some spare cycles today to look into that.  It turns out that if
you enable long-range matching, each zstd process reserves about 12GB of
memory, which Windows insists to "commit", i.e. your page file must be
large enough to hold all committed memory.  If it's dynamically sized it
will never grow larger than some fixed percentage of the free space on
that drive, so I've ran into that limit promptly.  It never actually
uses that memory best I can tell, the actual VM used stays below 4GB per
zstd process.  So I just blew up the pagefile.sys manually to 384GiB
(224GiB would have sufficed, but anyway) and started 20 parallel
compressions.  I used up 48GiB peak of the 128GiB in the machine and was
done in less than 50 minutes (the four extra threads not backed by a CPU
helped to keep the machine loaded while it was spinning up new
compressions when it came to the smaller files which are dominated by
I/O).  The compression ration only improved about 1% vs. the
non-long-range option to 103% vs. the original data, so using that
option isn't really necessary or recommended.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Zstandard support for setup

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

[…]

Here's a bit of support scaffolding for anybody who wants to play with
this (if the looks of the scripts scare you, stay away from it :-),
obviously you'll also need to compile yourself a setup.exe with the
patches to use with.  It assumes the Cygwin local repo files / cache in
a directory named install and will produce a re-compressed copy in
install-zstd.  I have these in /mnt/upload in my examples (they're also
not exact copies of the Cygwin mirror repos since I maintain these with
another script in order to mix in local packages as needed, so I hope it
just works on a direct copy as well, but have not tested that).  The new
hierarchy should be initially empty and the scripts assume that there's
nothing else in there that you want to keep.  Again, anything already in
install-zstd can get deleted when running the scripts, so don't keep
your bitcoins there.  Then these invocations will re-compress all
archives from the original and rewrite the setup.ini files so you can
later point setup to install from install-zstd:

/mnt/upload/maint-zstd.pl | parallel /mnt/upload/recompress.pl
ls install/x86*/setup*xz | parallel /mnt/upload/rewrite-setup.pl

That is GNU parallel above, which uses all available CPU without any
parameters, you can do the same with moreutils parallel or xargs with
the appropriate invocation.  There's output to STDERR going around the
pipe that keeps you informed about what's going on.

After each update of the Cygwin repo you'll have to re-run the scripts,
which will then only act upon new or updated files (and remove those
files and directories that were deleted, if any).  It will always
re-calculate all the SHA512 checksums at the moment, though (I might
implement some sort of cache for that later, but it doesn't bother me
too much).


[-- Attachment #2: maint-zstd.pl --]
[-- Type: application/x-perl, Size: 2450 bytes --]

[-- Attachment #3: recompress.pl --]
[-- Type: application/x-perl, Size: 460 bytes --]

[-- Attachment #4: rewrite-setup.pl --]
[-- Type: application/x-perl, Size: 1204 bytes --]

[-- Attachment #5: Type: text/plain, Size: 191 bytes --]



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Zstandard support for setup

[Jon Turney]

On 11/08/2018 20:52, Achim Gratz wrote:
[...]
> 
> The current code is available here:
> 
> http://repo.or.cz/w/cygwin-setup/local.git

Some comments:

This should check for ztd in configure.ac using PKG_CHECK_MODULES, 
rather than just assuming -lzstd is going to work.

The setup executables are cross-built on Fedora, so a 
mingw{32,64}-zstd-static package will need to become available.

compress_zstd.c:

+ using namespace std;

Please don't

compress_zstd.h:

+/* this is the parent class for all compress IO operations.
+ */

Comment is incorrect

I'd suggest removing all the 'virtual' since this class is final, but 
since all the compress subclasses have it, don't bother.

> The debug output statements are still in the code (although commented),
> I'll let that sit a while and see if I find something else I want to
> clean up before I submit it for the upstream repo.

I'd suggest keeping potentially useful ones under #ifdef DEBUG/#endif 
rather than just removing them all.

Re: Zstandard support for setup

[Achim Gratz]

Jon Turney writes:
> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
> rather than just assuming -lzstd is going to work.

Sure.  Waiting for your patch to actually use pkg_config to rebase it
onto.

> The setup executables are cross-built on Fedora, so a
> mingw{32,64}-zstd-static package will need to become available.

I don't have any contacts into the Fedora community, so I'd appreciate
if maybe Yaakov could help?  I have not yet found a distro that already
has packages for MinGW.

> compress_zstd.c:
>
> + using namespace std;
>
> Please don't

That's a copy from the compress_xz files and it's used in many files
elsewhere.  Care to explain why you'd want it removed?  If there's a
good reason I'd tend to move this to a clean-up commit that touches all
those files.  We should do that before hauling Zstdandard in.

> compress_zstd.h:
>
> +/* this is the parent class for all compress IO operations.
> + */
>
> Comment is incorrect

Another copy-paste, so that cleanup also needs to be applied to more
files (gz and xz).

> I'd suggest removing all the 'virtual' since this class is final, but
> since all the compress subclasses have it, don't bother.

Same as above, so we should clean up there first as well.

>> The debug output statements are still in the code (although commented),
>> I'll let that sit a while and see if I find something else I want to
>> clean up before I submit it for the upstream repo.
>
> I'd suggest keeping potentially useful ones under #ifdef DEBUG/#endif
> rather than just removing them all.

At the moment I'd rather remove them all as they were just meant for me
to check if things were working as I had assumed.  These produce a lot
more output than the usual debugging statements and you don't really
benefit if I keep just a subset enabled.  If we had a debugging facility
that knew about subsystems it would be a different matter (it's long
been on my TODO list, near the bottom unfortunately).

I'm attending a conference most of next week, so I need to postpone any
real work until the next weekend at the minimum.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Zstandard support for setup

[Achim Gratz]

Achim Gratz writes:
>> + using namespace std;
>>
>> Please don't
>
> That's a copy from the compress_xz files and it's used in many files
> elsewhere.  Care to explain why you'd want it removed?  If there's a
> good reason I'd tend to move this to a clean-up commit that touches all
> those files.  We should do that before hauling Zstdandard in.

What an unholy mess… Here's that patch, only lightly tested:
http://repo.or.cz/cygwin-setup/local.git/patch/e9802f50304f66984f6aad5544def465e9f8033b


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Zstandard support for setup

[Jon Turney]

On 02/09/2018 18:02, Achim Gratz wrote:
> Achim Gratz writes:
>>> + using namespace std;
>>>
>>> Please don't
>>
>> That's a copy from the compress_xz files and it's used in many files
>> elsewhere.  Care to explain why you'd want it removed?  If there's a
>> good reason I'd tend to move this to a clean-up commit that touches all
>> those files.  We should do that before hauling Zstdandard in.
> 
> What an unholy mess… Here's that patch, only lightly tested:
> http://repo.or.cz/cygwin-setup/local.git/patch/e9802f50304f66984f6aad5544def465e9f8033b

Yeah. Thanks, and sorry for the delay.

Applied.

Re: Zstandard support for setup

[Achim Gratz]

Jon Turney writes:
> On 02/09/2018 18:02, Achim Gratz wrote:
>> What an unholy mess… Here's that patch, only lightly tested:
>> http://repo.or.cz/cygwin-setup/local.git/patch/e9802f50304f66984f6aad5544def465e9f8033b
>
> Yeah. Thanks, and sorry for the delay.

No sweat and thanks.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Zstandard support for setup

[Jon Turney]

On 02/09/2018 09:29, Achim Gratz wrote:
> Jon Turney writes:
>> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
>> rather than just assuming -lzstd is going to work.
> 
> Sure.  Waiting for your patch to actually use pkg_config to rebase it
> onto.

This is applied now.

>> The setup executables are cross-built on Fedora, so a
>> mingw{32,64}-zstd-static package will need to become available.
> 
> I don't have any contacts into the Fedora community, so I'd appreciate
> if maybe Yaakov could help?  I have not yet found a distro that already
> has packages for MinGW.

https://github.com/jon-turney/mingw-zstd-rpm
https://copr.fedorainfracloud.org/coprs/jturney/mingw-zstd/

Re: Zstandard support for setup

[Achim Gratz]

Jon Turney writes:
> On 02/09/2018 09:29, Achim Gratz wrote:
>> Jon Turney writes:
>>> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
>>> rather than just assuming -lzstd is going to work.
>>
>> Sure.  Waiting for your patch to actually use pkg_config to rebase it
>> onto.
>
> This is applied now.

Thanks.  I'll see to get a round tuit soon-ish.

>>> The setup executables are cross-built on Fedora, so a
>>> mingw{32,64}-zstd-static package will need to become available.
>>
>> I don't have any contacts into the Fedora community, so I'd appreciate
>> if maybe Yaakov could help?  I have not yet found a distro that already
>> has packages for MinGW.
>
> https://github.com/jon-turney/mingw-zstd-rpm
> https://copr.fedorainfracloud.org/coprs/jturney/mingw-zstd/

\o/


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Zstandard support for setup

[Achim Gratz]

Jon Turney writes:
> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
> rather than just assuming -lzstd is going to work.

Done.

> + using namespace std;
>
> Please don't

Done.

https://repo.or.cz/cygwin-setup/local.git/patch/fff4b1448072efe5ebb6549cfe04bc15344a3973

> compress_zstd.h:
>
> +/* this is the parent class for all compress IO operations.
> + */
>
> Comment is incorrect

Done in the above, additionally remove the same copy&paste comment from
two existing files:

https://repo.or.cz/cygwin-setup/local.git/patch/abf5afbfce47d1942f6d4c2f62fc8b1566920e2e


> I'd suggest keeping potentially useful ones under #ifdef DEBUG/#endif
> rather than just removing them all.

Later...


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Zstandard support for setup

[Jon Turney]

On 10/10/2018 19:36, Achim Gratz wrote:
> Jon Turney writes:
>> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
>> rather than just assuming -lzstd is going to work.
> 
> Done.
> 
>> + using namespace std;
>>
>> Please don't
> 
> Done.
> 
> https://repo.or.cz/cygwin-setup/local.git/patch/fff4b1448072efe5ebb6549cfe04bc15344a3973

Applied.

I removed the commented out logging, updated build instructions and 
build prerequisites installed in CI

Thanks.

>> compress_zstd.h:
>>
>> +/* this is the parent class for all compress IO operations.
>> + */
>>
>> Comment is incorrect
> 
> Done in the above, additionally remove the same copy&paste comment from
> two existing files:
> 
> https://repo.or.cz/cygwin-setup/local.git/patch/abf5afbfce47d1942f6d4c2f62fc8b1566920e2e

Applied.

Re: Zstandard support for setup

[Achim Gratz]

Jon Turney writes:
> Applied.
>
> I removed the commented out logging, updated build instructions and
> build prerequisites installed in CI

Great, thank you very much.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada