Re: scallywag / cygport not pulling lzip

[Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>]

On 2021-01-08 10:13, Achim Gratz wrote:
> Brian Inglis writes:
>>> Yeah, perhaps lzip should be a dependency of cygport.
> 
> Since it promises to enable it's use, yes.  I wouldn't mind to add a
> dependency to tar also.

I would support that for technical reasons alone, although hoped for uptake of 
lzip across GNU or by other projects has been limited to the package itself and 
others by the same developer, as its goals of providing archive redundancy and 
recoverability have been obviated by reliable network links and larger storage 
sizes.

>> ...and zstd as Achim is working on that.
> 
> That dependency has already been added to tar, so there is no need to do
> anything special.
> 
> [as you could hav seen by looking at any of the recent CI logs]
> 
>> I do not think we should encourage use of lower compression ratio
>> packages in cygport and calm, and their libraries in setup.
>> It appears zstd gives up compression to gain faster decompression with
>> less memory which may be advantageous for loading a kernel, but less
>> optimal for MBs of base packages.
> 
> That difference is in the order of 3% over the full distribution at the
> highest compression setting and roughly 5% is you use a compression
> setting that is about as slow as xz.  The installation time on the other
> hand is dramatically shorter with zstd.

Do we know what the frequency weighted difference is on bandwidth of packages 
actually downloaded?
I am more concerned with mirror providers (and also the lack of them) especially 
those with limited resources, and those in marginal locations and circumstances, 
for whom download time and charges may override other considerations, and 
perhaps prevent them (or many) from accessing or taking full advantage of 
available software.
I doubt the unarchiving time difference is more than a blip in the total time 
required to *download* *AND* install any package, greatly outweighed by the 
download time difference, unless you are on a big pipe to a nearby mirror.
[I take into consideration that it used to be cheaper to dialup NIST ACTS 
Colorado to set the time, than anywhere in Canada, and downloads did not become 
reasonably affordable until I could get network access from a local dialup.] ;^>

>> Perhaps consideration should also be given to setting appropriate
>> compression defaults in cygport for all supported package compressors
>> e.g. GZIP=-9, XZ_OPT=-7 which does not increase the memory required
>> from the default -6 but improves compression, ZSTD_CLEVEL=19 and
>> ZSTD_NBTHREADS=`nproc` perhaps?
> 
> It is relatively useless to have zstd run with multiple threads in
> compression mode since a lot of what takes time there is inherently
> single threaded.  It already does use an I/O thread to keep the data
> pipes fed unless you specifically forbid that.

That is valid consideration indeed as suggested.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]