From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtpout2.vodafonemail.de (smtpout2.vodafonemail.de [145.253.239.133]) by sourceware.org (Postfix) with ESMTPS id 04B0038438A3 for ; Fri, 8 Jan 2021 19:11:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 04B0038438A3 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=nexgo.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=Stromeko@nexgo.de Received: from smtp.vodafone.de (unknown [10.2.0.36]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id DCE4F123470 for ; Fri, 8 Jan 2021 20:11:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nexgo.de; s=vfde-smtpout-mb-15sep; t=1610133094; bh=hAqQgeJ8Na2+2H5jxqIjRKln6N2/KMylxouDmiTU9i0=; h=From:To:Subject:References:Date:In-Reply-To; b=Scje4Db7TjNqOeqbhFmPm9mzOff3yLyapWJQbwIUWwO6RkBOnhE2SzCicxpZyXwvc GV8/oc/nFJNynFpNH3sL3N3aVFdXvvQfqPizT4Ubu8f4Ab6F30R0aLQDAVUT7PDbWl tMeQJl7d727G4VxAukMlGZOWkCba+XCSTQWV3HJk= Received: from Gertrud (p54a0ca05.dip0.t-ipconnect.de [84.160.202.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp.vodafone.de (Postfix) with ESMTPSA id 76B4A1420B0 for ; Fri, 8 Jan 2021 19:11:34 +0000 (UTC) From: Achim Gratz To: cygwin-apps@cygwin.com Subject: Re: scallywag / cygport not pulling lzip References: <3cd5f2f8-b292-0ac1-de18-753a4513f6ba@gmail.com> <87wnwnk1er.fsf@Otto.invalid> <87a6tj9wst.fsf@Rainer.invalid> Date: Fri, 08 Jan 2021 20:11:30 +0100 In-Reply-To: (Brian Inglis's message of "Fri, 8 Jan 2021 11:09:32 -0700") Message-ID: <871rev9rbx.fsf@Rainer.invalid> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate: clean X-purgate-size: 2751 X-purgate-ID: 155817::1610133094-000048A3-C76773FD/0/0 X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin-apps@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin package maintainer discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jan 2021 19:11:39 -0000 Brian Inglis writes: > Do we know what the frequency weighted difference is on bandwidth of > packages actually downloaded? Not that I know of, as everything goes through mirrors. But I happen to have a complete Cygwin mirror on disk at the moment plus another one that only has the packages for my install and that's a fairly large installation, but without a desktop environment: 30G /mnt/mirror/cygwin 149G /mnt/fullmirror/cygwin So you can probably assume that only about 20% of the files are frequently accessed (likely significantly less since most folks would not install the debuginfo or source packages that are included in the above figure). > I am more concerned with mirror providers (and also the lack of them) > especially those with limited resources, and those in marginal > locations and circumstances, for whom download time and charges may > override other considerations, and perhaps prevent them (or many) from > accessing or taking full advantage of available software. We could save way more space than that by de-duplicating the noarch parts into their own archives as I have already demonstrated before. The last time I did that I was cutting out around 30GiB IIRC. > I doubt the unarchiving time difference is more than a blip in the > total time required to *download* *AND* install any package, greatly > outweighed by the download time difference, unless you are on a big > pipe to a nearby mirror. It is not, with a typical VDSL connection I'd be able to download faster than I can install on a more typical machine, I need only about 5MiB/s to saturate the filesystem for small files and around 20=E2=80=A640MiB/s for large ones (to an NVMe drive, a spinning disk or some of the slower SSD can't sustain that). But that point is somewhat moot since setup will always mirror to disk first and that's not easy to change since we read the file twice: once for the SHA512 check (which can use up to around 300MiB/s input bandwidth somewhat higher in peaks and then the actual installation). I have a large base of internal installations that I feed from a (single) local repo and some of those machines are behind rather slow links (not quite modem speed, but still slow by todays standards) and using zstd still makes quite the difference there. The more typical install time was reduced by a bit under 50% for both slow and fast connections, so I no longer recommend that folks reserve time specifically for the Cygwin installation as I had done before. Regards, Achim. --=20 +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf Blofeld V1.15B11: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada