* Re: Zstandard support for setup
2018-08-11 19:52 Zstandard support for setup Achim Gratz
@ 2018-08-12 13:28 ` Achim Gratz
2018-08-17 18:49 ` Achim Gratz
` (2 subsequent siblings)
3 siblings, 0 replies; 17+ messages in thread
From: Achim Gratz @ 2018-08-12 13:28 UTC (permalink / raw)
To: cygwin-apps
Achim Gratz writes:
> I've finally implemented ZStandard support for setup and tested it today
> with a fresh install on a Celeron 1037U dual-core machine from and to
> local SSD. The archive extraction part of setup went from taking 23
> minutes to about 14 minutes.
This morning I've installed on the Win10 machine (same specs, different
box) over gigabit network, served from the Win8.1 machine from
yesterday. The SHA512 check was taking 2:29 instead of 1:00, but the
archive extraction took just 13:08 (slightly faster than locally).
Maybe larger buffers and zero copy could have more leverage than I
thought they would… can we do a lazy mmap on the input files?
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-11 19:52 Zstandard support for setup Achim Gratz
2018-08-12 13:28 ` Achim Gratz
@ 2018-08-17 18:49 ` Achim Gratz
2018-08-20 18:11 ` Marco Atzeri
2018-08-22 19:12 ` Achim Gratz
2018-08-23 20:40 ` Achim Gratz
2018-08-29 16:42 ` Jon Turney
3 siblings, 2 replies; 17+ messages in thread
From: Achim Gratz @ 2018-08-17 18:49 UTC (permalink / raw)
To: cygwin-apps
While it seems like I'm talking to myself, in case anybody is listening:
With the "normal" maximum compression level of 19 the package archive
for my installation (including sources and debuginfo) got about 10%
larger, taking about the same time as Xz for compression. Using the
"ultra" compression level 22 gets that difference to about 3% at the
expense of twice the CPU time for the compression. My repo is about
8GiB compressed and it took about 8 (resp. 16 hours) of single-core CPU
time to re-compress (uncompress from BZip2 or Xz and recompress
on-the-fly to ZStdandard). The higher compression level needs just a
bit over 1.5GiB on average and around 3GiB peak.
I've had 16 cores to run that on and those stayed at 2.8GHz all the
time, so it didn't take too long. Here at home I've tasked two slower
4-core machines with it, so they were running a few hours of wall-time.
I might try long-range matching to get even better compression at work
since I've got 128GB memory now to see if there's an improvement. At
home I've run out of 8GiB memory more than once even with a single
compression for the largest debuginfo packages. Incidentally, these
packages always take the longest time to compress, so if you want to do
a parallel mass conversion it's advisable to start these first (keeping
an eye on available memory) and have the faster compression of the
smaller package archives fill the remaining CPU/time.
I've installed and updated over a dozen of machines by now with this, so
I'm reasonably certain that the implementation in setup is OK.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-17 18:49 ` Achim Gratz
@ 2018-08-20 18:11 ` Marco Atzeri
2018-08-20 19:54 ` Achim Gratz
2018-08-22 19:12 ` Achim Gratz
1 sibling, 1 reply; 17+ messages in thread
From: Marco Atzeri @ 2018-08-20 18:11 UTC (permalink / raw)
To: cygwin-apps
Am 17.08.2018 um 20:49 schrieb Achim Gratz:
>
> While it seems like I'm talking to myself, in case anybody is listening:
>
>
> I've had 16 cores to run that on and those stayed at 2.8GHz all the
> time, so it didn't take too long. Here at home I've tasked two slower
> 4-core machines with it, so they were running a few hours of wall-time.
> I might try long-range matching to get even better compression at work
> since I've got 128GB memory now to see if there's an improvement. At
> home I've run out of 8GiB memory more than once even with a single
> compression for the largest debuginfo packages. Incidentally, these
> packages always take the longest time to compress, so if you want to do
> a parallel mass conversion it's advisable to start these first (keeping
> an eye on available memory) and have the faster compression of the
> smaller package archives fill the remaining CPU/time.
memory and time requirements seem not so appealing to me
as my machine has only 8GiB and already the debuginfo on 32bit is
a time consuming effort
> I've installed and updated over a dozen of machines by now with this, so
> I'm reasonably certain that the implementation in setup is OK.
>
>
> Regards,
> Achim.
>
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-20 18:11 ` Marco Atzeri
@ 2018-08-20 19:54 ` Achim Gratz
0 siblings, 0 replies; 17+ messages in thread
From: Achim Gratz @ 2018-08-20 19:54 UTC (permalink / raw)
To: cygwin-apps
Marco Atzeri writes:
> memory and time requirements seem not so appealing to me
> as my machine has only 8GiB and already the debuginfo on 32bit is
> a time consuming effort
That happens only with ultra compression and long-range matching
enabled. I've not run out of 6GiB memory across two parallel
compressions with the highest non-ultra settings and long-range matching
disabled. That costs only about 7% vs. Xz compression, so it's still OK
I'd say.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-17 18:49 ` Achim Gratz
2018-08-20 18:11 ` Marco Atzeri
@ 2018-08-22 19:12 ` Achim Gratz
1 sibling, 0 replies; 17+ messages in thread
From: Achim Gratz @ 2018-08-22 19:12 UTC (permalink / raw)
To: cygwin-apps
Achim Gratz writes:
> I've had 16 cores to run that on and those stayed at 2.8GHz all the
> time, so it didn't take too long. Here at home I've tasked two slower
> 4-core machines with it, so they were running a few hours of wall-time.
> I might try long-range matching to get even better compression at work
> since I've got 128GB memory now to see if there's an improvement. At
> home I've run out of 8GiB memory more than once even with a single
> compression for the largest debuginfo packages.
I used some spare cycles today to look into that. It turns out that if
you enable long-range matching, each zstd process reserves about 12GB of
memory, which Windows insists to "commit", i.e. your page file must be
large enough to hold all committed memory. If it's dynamically sized it
will never grow larger than some fixed percentage of the free space on
that drive, so I've ran into that limit promptly. It never actually
uses that memory best I can tell, the actual VM used stays below 4GB per
zstd process. So I just blew up the pagefile.sys manually to 384GiB
(224GiB would have sufficed, but anyway) and started 20 parallel
compressions. I used up 48GiB peak of the 128GiB in the machine and was
done in less than 50 minutes (the four extra threads not backed by a CPU
helped to keep the machine loaded while it was spinning up new
compressions when it came to the smaller files which are dominated by
I/O). The compression ration only improved about 1% vs. the
non-long-range option to 103% vs. the original data, so using that
option isn't really necessary or recommended.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-11 19:52 Zstandard support for setup Achim Gratz
2018-08-12 13:28 ` Achim Gratz
2018-08-17 18:49 ` Achim Gratz
@ 2018-08-23 20:40 ` Achim Gratz
2018-08-29 16:42 ` Jon Turney
3 siblings, 0 replies; 17+ messages in thread
From: Achim Gratz @ 2018-08-23 20:40 UTC (permalink / raw)
To: cygwin-apps
[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]
[…]
Here's a bit of support scaffolding for anybody who wants to play with
this (if the looks of the scripts scare you, stay away from it :-),
obviously you'll also need to compile yourself a setup.exe with the
patches to use with. It assumes the Cygwin local repo files / cache in
a directory named install and will produce a re-compressed copy in
install-zstd. I have these in /mnt/upload in my examples (they're also
not exact copies of the Cygwin mirror repos since I maintain these with
another script in order to mix in local packages as needed, so I hope it
just works on a direct copy as well, but have not tested that). The new
hierarchy should be initially empty and the scripts assume that there's
nothing else in there that you want to keep. Again, anything already in
install-zstd can get deleted when running the scripts, so don't keep
your bitcoins there. Then these invocations will re-compress all
archives from the original and rewrite the setup.ini files so you can
later point setup to install from install-zstd:
/mnt/upload/maint-zstd.pl | parallel /mnt/upload/recompress.pl
ls install/x86*/setup*xz | parallel /mnt/upload/rewrite-setup.pl
That is GNU parallel above, which uses all available CPU without any
parameters, you can do the same with moreutils parallel or xargs with
the appropriate invocation. There's output to STDERR going around the
pipe that keeps you informed about what's going on.
After each update of the Cygwin repo you'll have to re-run the scripts,
which will then only act upon new or updated files (and remove those
files and directories that were deleted, if any). It will always
re-calculate all the SHA512 checksums at the moment, though (I might
implement some sort of cache for that later, but it doesn't bother me
too much).
[-- Attachment #2: maint-zstd.pl --]
[-- Type: application/x-perl, Size: 2450 bytes --]
[-- Attachment #3: recompress.pl --]
[-- Type: application/x-perl, Size: 460 bytes --]
[-- Attachment #4: rewrite-setup.pl --]
[-- Type: application/x-perl, Size: 1204 bytes --]
[-- Attachment #5: Type: text/plain, Size: 191 bytes --]
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-11 19:52 Zstandard support for setup Achim Gratz
` (2 preceding siblings ...)
2018-08-23 20:40 ` Achim Gratz
@ 2018-08-29 16:42 ` Jon Turney
2018-09-02 8:31 ` Achim Gratz
2018-10-10 18:37 ` Achim Gratz
3 siblings, 2 replies; 17+ messages in thread
From: Jon Turney @ 2018-08-29 16:42 UTC (permalink / raw)
To: cygwin-apps
On 11/08/2018 20:52, Achim Gratz wrote:
[...]
>
> The current code is available here:
>
> http://repo.or.cz/w/cygwin-setup/local.git
Some comments:
This should check for ztd in configure.ac using PKG_CHECK_MODULES,
rather than just assuming -lzstd is going to work.
The setup executables are cross-built on Fedora, so a
mingw{32,64}-zstd-static package will need to become available.
compress_zstd.c:
+ using namespace std;
Please don't
compress_zstd.h:
+/* this is the parent class for all compress IO operations.
+ */
Comment is incorrect
I'd suggest removing all the 'virtual' since this class is final, but
since all the compress subclasses have it, don't bother.
> The debug output statements are still in the code (although commented),
> I'll let that sit a while and see if I find something else I want to
> clean up before I submit it for the upstream repo.
I'd suggest keeping potentially useful ones under #ifdef DEBUG/#endif
rather than just removing them all.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-29 16:42 ` Jon Turney
@ 2018-09-02 8:31 ` Achim Gratz
2018-09-02 17:03 ` Achim Gratz
2018-10-07 22:09 ` Jon Turney
2018-10-10 18:37 ` Achim Gratz
1 sibling, 2 replies; 17+ messages in thread
From: Achim Gratz @ 2018-09-02 8:31 UTC (permalink / raw)
To: cygwin-apps
Jon Turney writes:
> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
> rather than just assuming -lzstd is going to work.
Sure. Waiting for your patch to actually use pkg_config to rebase it
onto.
> The setup executables are cross-built on Fedora, so a
> mingw{32,64}-zstd-static package will need to become available.
I don't have any contacts into the Fedora community, so I'd appreciate
if maybe Yaakov could help? I have not yet found a distro that already
has packages for MinGW.
> compress_zstd.c:
>
> + using namespace std;
>
> Please don't
That's a copy from the compress_xz files and it's used in many files
elsewhere. Care to explain why you'd want it removed? If there's a
good reason I'd tend to move this to a clean-up commit that touches all
those files. We should do that before hauling Zstdandard in.
> compress_zstd.h:
>
> +/* this is the parent class for all compress IO operations.
> + */
>
> Comment is incorrect
Another copy-paste, so that cleanup also needs to be applied to more
files (gz and xz).
> I'd suggest removing all the 'virtual' since this class is final, but
> since all the compress subclasses have it, don't bother.
Same as above, so we should clean up there first as well.
>> The debug output statements are still in the code (although commented),
>> I'll let that sit a while and see if I find something else I want to
>> clean up before I submit it for the upstream repo.
>
> I'd suggest keeping potentially useful ones under #ifdef DEBUG/#endif
> rather than just removing them all.
At the moment I'd rather remove them all as they were just meant for me
to check if things were working as I had assumed. These produce a lot
more output than the usual debugging statements and you don't really
benefit if I keep just a subset enabled. If we had a debugging facility
that knew about subsystems it would be a different matter (it's long
been on my TODO list, near the bottom unfortunately).
I'm attending a conference most of next week, so I need to postpone any
real work until the next weekend at the minimum.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Zstandard support for setup
2018-08-29 16:42 ` Jon Turney
2018-09-02 8:31 ` Achim Gratz
@ 2018-10-10 18:37 ` Achim Gratz
2018-10-13 17:19 ` Jon Turney
1 sibling, 1 reply; 17+ messages in thread
From: Achim Gratz @ 2018-10-10 18:37 UTC (permalink / raw)
To: cygwin-apps
Jon Turney writes:
> This should check for ztd in configure.ac using PKG_CHECK_MODULES,
> rather than just assuming -lzstd is going to work.
Done.
> + using namespace std;
>
> Please don't
Done.
https://repo.or.cz/cygwin-setup/local.git/patch/fff4b1448072efe5ebb6549cfe04bc15344a3973
> compress_zstd.h:
>
> +/* this is the parent class for all compress IO operations.
> + */
>
> Comment is incorrect
Done in the above, additionally remove the same copy&paste comment from
two existing files:
https://repo.or.cz/cygwin-setup/local.git/patch/abf5afbfce47d1942f6d4c2f62fc8b1566920e2e
> I'd suggest keeping potentially useful ones under #ifdef DEBUG/#endif
> rather than just removing them all.
Later...
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
^ permalink raw reply [flat|nested] 17+ messages in thread