From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.gentoo.org (smtp.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) by sourceware.org (Postfix) with ESMTP id 3598F3858C2D for ; Tue, 8 Nov 2022 07:40:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3598F3858C2D Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gentoo.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gentoo.org Content-Type: multipart/signed; boundary="Apple-Mail=_67EC2E69-7F93-4E87-90B1-DDEFBC56C423"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.200.110.1.12\)) Subject: Re: [PATCH] maintainer-scripts/gcc_release: compress xz in parallel From: Sam James In-Reply-To: <6e8d39e1e3e9d6ce5f0be5781023127187bcb995.camel@xry111.site> Date: Tue, 8 Nov 2022 07:40:02 +0000 Cc: gcc-patches@gcc.gnu.org, Gerald Pfeifer , Jakub Jelinek , Jeff Law Message-Id: <9E6C1D21-0F77-4740-A9D3-9EE415D0AABD@gentoo.org> References: <20221108071438.2523863-1-sam@gentoo.org> <6e8d39e1e3e9d6ce5f0be5781023127187bcb995.camel@xry111.site> To: Xi Ruoyao X-Mailer: Apple Mail (2.3731.200.110.1.12) X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,JMQ_SPF_NEUTRAL,KAM_DMARC_STATUS,SPF_HELO_PASS,SPF_PASS,TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --Apple-Mail=_67EC2E69-7F93-4E87-90B1-DDEFBC56C423 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii > On 8 Nov 2022, at 07:33, Xi Ruoyao wrote: > > On Tue, 2022-11-08 at 07:14 +0000, Sam James via Gcc-patches wrote: >> 1. This should speed up decompression for folks, as parallel xz >> creates a different archive which can be decompressed in parallel. >> >> Note that this different method is enabled by default in a new >> xz release coming shortly anyway (>= 5.3.3_alpha1). >> >> I build GCC regularly from the weekly snapshots >> and so the decompression time adds up. >> >> 2. It should speed up compression on the webserver a bit. >> >> Note that -T0 won't be the default in the new xz release, >> only the parallel compression mode (which enables parallel >> decompression). >> >> -T0 detects the number of cores available. >> >> So, if a different number of threads is preferred, it's fine >> to set e.g. -T2, etc. > > I'm wondering if running xz -T0 on different machines (with different > core numbers) may produce different compressed data. The difference can > cause trouble distributing checksums. > Your question is a good one - xz -T0 produces different results to xz -T1 but: 1. The tarballs for GCC are only created on one machine and aren't created repeatedly then compared with each other wrt mirroring; 2. Decompression still gives the same result; 3. xz is going to switch to this threaded decompressor output mode shortly anyway. i.e. there's a slight change in output, but it's what future versions are going to use anyway. It's deterministic wrt -T1 and -Tn > 1. i.e. it's about the compressor method (it produces chunks) rather than anything else. Plenty of other projects like LLVM (which also has a large distribution tarball) use it without any problems. Best, sam --Apple-Mail=_67EC2E69-7F93-4E87-90B1-DDEFBC56C423 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iNUEARYKAH0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCY2oH018UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MAAKCRBzhAn1IN+R kJtVAQDYJqbbCj4V6by159SCSr/EmMpQB24CrHbAPOtpFOyLUQD9HXbgR3lkFo8c +DgXDK3NWBFR6YlJtXKkTAvR+zwwxQM= =hzDB -----END PGP SIGNATURE----- --Apple-Mail=_67EC2E69-7F93-4E87-90B1-DDEFBC56C423--