From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out-so.shaw.ca (smtp-out-so.shaw.ca [64.59.136.138]) by sourceware.org (Postfix) with ESMTPS id 331383857C65 for ; Mon, 11 Jan 2021 22:05:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 331383857C65 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=SystematicSw.ab.ca Authentication-Results: sourceware.org; spf=none smtp.mailfrom=brian.inglis@systematicsw.ab.ca Received: from [192.168.1.104] ([24.64.172.44]) by shaw.ca with ESMTP id z5JBkkEqQktFkz5JCkAJVj; Mon, 11 Jan 2021 15:05:06 -0700 X-Authority-Analysis: v=2.4 cv=NYRYa0P4 c=1 sm=1 tr=0 ts=5ffccb92 a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17 a=IkcTkHD0fZMA:10 a=TImcKGuyeGIbufSLrCcA:9 a=QEXdDO2ut3YA:10 Reply-To: cygwin-apps@cygwin.com To: cygwin-apps@cygwin.com References: <3cd5f2f8-b292-0ac1-de18-753a4513f6ba@gmail.com> <87wnwnk1er.fsf@Otto.invalid> <87a6tj9wst.fsf@Rainer.invalid> <871rev9rbx.fsf@Rainer.invalid> From: Brian Inglis Organization: Systematic Software Subject: Re: scallywag / cygport not pulling lzip Message-ID: <6bbcf804-593b-16be-dfd1-e1430c1c4bf8@SystematicSw.ab.ca> Date: Mon, 11 Jan 2021 15:05:05 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <871rev9rbx.fsf@Rainer.invalid> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-CA Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4xfFwRLT96Glnw043w8UpkXvM96ZC1V4CwwBPF/W+jO5cWlSzIOk3RSJs5fGOUxNFmUXPbsnbtW1w6DwMpXJWrpvKjNdtVwGagZZApXVrVbzbUfvzFg4he ErIKSenv4oq7oGhDWFLz2cQ3bo1NrTGVqu0QOmrttznU+5Vk0oMv9sGx40irAdaA/vTgNvnL/zIcuYuVfnbvN3A6tofzL+DOTyM= X-Spam-Status: No, score=-5.7 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-apps@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin package maintainer discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2021 22:05:09 -0000 On 2021-01-08 12:11, Achim Gratz wrote: > Brian Inglis writes: >> Do we know what the frequency weighted difference is on bandwidth of >> packages actually downloaded? > > Not that I know of, as everything goes through mirrors. But I happen to > have a complete Cygwin mirror on disk at the moment plus another one > that only has the packages for my install and that's a fairly large > installation, but without a desktop environment: > > 30G /mnt/mirror/cygwin > 149G /mnt/fullmirror/cygwin > > So you can probably assume that only about 20% of the files are > frequently accessed (likely significantly less since most folks would > not install the debuginfo or source packages that are included in the > above figure). > >> I am more concerned with mirror providers (and also the lack of them) >> especially those with limited resources, and those in marginal >> locations and circumstances, for whom download time and charges may >> override other considerations, and perhaps prevent them (or many) from >> accessing or taking full advantage of available software. > > We could save way more space than that by de-duplicating the noarch > parts into their own archives as I have already demonstrated before. > The last time I did that I was cutting out around 30GiB IIRC. > >> I doubt the unarchiving time difference is more than a blip in the >> total time required to *download* *AND* install any package, greatly >> outweighed by the download time difference, unless you are on a big >> pipe to a nearby mirror. > > It is not, with a typical VDSL connection I'd be able to download faster > than I can install on a more typical machine, I need only about 5MiB/s > to saturate the filesystem for small files and around 20…40MiB/s for > large ones (to an NVMe drive, a spinning disk or some of the slower SSD > can't sustain that). But that point is somewhat moot since setup will > always mirror to disk first and that's not easy to change since we read > the file twice: once for the SHA512 check (which can use up to around > 300MiB/s input bandwidth somewhat higher in peaks and then the actual > installation). Some setup phase stats from my own most recent upgrade of about 130MB of downloads, and stats since 2013 (nearly 8 years) since I last cleared setup.log: $ cyg-setup-phase-times.awk /var/log/setup.log.full sv 00:04:26 dl 00:01:28 pr 00:00:02 ui 00:00:36 ex 00:03:35 pi 00:07:12 $ cyg-setup-phase-times.awk /var/log/setup.log sv 04:32:46 dl 02:31:27 pr 00:16:08 ui 00:51:10 ex 06:06:49 pi 00:00:41 phases are: sv - solve formerly Adding required packages - high times are interaction delays dl - download pr - preremove ui - uninstall ex - extract pi - postinstall so your comments about extracts are validated, taking about 3 times as long as downloads even on a currently 2MByte/s medium speed cable modem link to a nearby (7.5km direct, 11km drive, 15 hop 10ms round trip) university campus mirror in recent years. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.]