From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out-so.shaw.ca (smtp-out-so.shaw.ca [64.59.136.138]) by sourceware.org (Postfix) with ESMTPS id 0A32538438A3 for ; Fri, 8 Jan 2021 18:09:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 0A32538438A3 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=SystematicSw.ab.ca Authentication-Results: sourceware.org; spf=none smtp.mailfrom=brian.inglis@systematicsw.ab.ca Received: from [192.168.1.104] ([24.64.172.44]) by shaw.ca with ESMTP id xwCaki1dBbYg3xwCbkpjxM; Fri, 08 Jan 2021 11:09:34 -0700 X-Authority-Analysis: v=2.4 cv=Q4RsX66a c=1 sm=1 tr=0 ts=5ff89fde a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17 a=IkcTkHD0fZMA:10 a=TImcKGuyeGIbufSLrCcA:9 a=QEXdDO2ut3YA:10 Reply-To: cygwin-apps@cygwin.com To: cygwin-apps@cygwin.com References: <3cd5f2f8-b292-0ac1-de18-753a4513f6ba@gmail.com> <87wnwnk1er.fsf@Otto.invalid> <87a6tj9wst.fsf@Rainer.invalid> From: Brian Inglis Organization: Systematic Software Subject: Re: scallywag / cygport not pulling lzip Message-ID: Date: Fri, 8 Jan 2021 11:09:32 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <87a6tj9wst.fsf@Rainer.invalid> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-CA Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4xfJs4MmFMUajWcZFrCCBO+ma1eeXfoQHAdS7fhOx5/xDpiVYdWGU5lm4Of/dQVi6EH4ueipjFNN5Z+FqQKQmL3LVCxp13rt1pvwhf/bUdOyLJJKkQCHtJ 6t14nERfUP8PZMKDBR1X/pwPuJ5Vi5qSuv+aYV9MXBzDEVAdPpD8ZUELY925lC2RVKldMOnnKAi7IToU/mlf6RfP444EwxKKgrI= X-Spam-Status: No, score=-5.9 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-apps@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin package maintainer discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jan 2021 18:09:36 -0000 On 2021-01-08 10:13, Achim Gratz wrote: > Brian Inglis writes: >>> Yeah, perhaps lzip should be a dependency of cygport. > > Since it promises to enable it's use, yes. I wouldn't mind to add a > dependency to tar also. I would support that for technical reasons alone, although hoped for uptake of lzip across GNU or by other projects has been limited to the package itself and others by the same developer, as its goals of providing archive redundancy and recoverability have been obviated by reliable network links and larger storage sizes. >> ...and zstd as Achim is working on that. > > That dependency has already been added to tar, so there is no need to do > anything special. > > [as you could hav seen by looking at any of the recent CI logs] > >> I do not think we should encourage use of lower compression ratio >> packages in cygport and calm, and their libraries in setup. >> It appears zstd gives up compression to gain faster decompression with >> less memory which may be advantageous for loading a kernel, but less >> optimal for MBs of base packages. > > That difference is in the order of 3% over the full distribution at the > highest compression setting and roughly 5% is you use a compression > setting that is about as slow as xz. The installation time on the other > hand is dramatically shorter with zstd. Do we know what the frequency weighted difference is on bandwidth of packages actually downloaded? I am more concerned with mirror providers (and also the lack of them) especially those with limited resources, and those in marginal locations and circumstances, for whom download time and charges may override other considerations, and perhaps prevent them (or many) from accessing or taking full advantage of available software. I doubt the unarchiving time difference is more than a blip in the total time required to *download* *AND* install any package, greatly outweighed by the download time difference, unless you are on a big pipe to a nearby mirror. [I take into consideration that it used to be cheaper to dialup NIST ACTS Colorado to set the time, than anywhere in Canada, and downloads did not become reasonably affordable until I could get network access from a local dialup.] ;^> >> Perhaps consideration should also be given to setting appropriate >> compression defaults in cygport for all supported package compressors >> e.g. GZIP=-9, XZ_OPT=-7 which does not increase the memory required >> from the default -6 but improves compression, ZSTD_CLEVEL=19 and >> ZSTD_NBTHREADS=`nproc` perhaps? > > It is relatively useless to have zstd run with multiple threads in > compression mode since a lot of what takes time there is inherently > single threaded. It already does use an I/O thread to keep the data > pipes fed unless you specifically forbid that. That is valid consideration indeed as suggested. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.]