Re: [PATCH] maintainer-scripts/gcc_release: compress xz in parallel

[Sam James <sam@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]



> On 8 Nov 2022, at 07:33, Xi Ruoyao <xry111@xry111.site> wrote:
> 
> On Tue, 2022-11-08 at 07:14 +0000, Sam James via Gcc-patches wrote:
>> 1. This should speed up decompression for folks, as parallel xz
>>    creates a different archive which can be decompressed in parallel.
>> 
>>    Note that this different method is enabled by default in a new
>>    xz release coming shortly anyway (>= 5.3.3_alpha1).
>> 
>>    I build GCC regularly from the weekly snapshots
>>    and so the decompression time adds up.
>> 
>> 2. It should speed up compression on the webserver a bit.
>> 
>>    Note that -T0 won't be the default in the new xz release,
>>    only the parallel compression mode (which enables parallel
>>    decompression).
>> 
>>    -T0 detects the number of cores available.
>> 
>>    So, if a different number of threads is preferred, it's fine
>>    to set e.g. -T2, etc.
> 
> I'm wondering if running xz -T0 on different machines (with different
> core numbers) may produce different compressed data.  The difference can
> cause trouble distributing checksums.
> 

Your question is a good one - xz -T0 produces different results to xz -T1
but:
1. The tarballs for GCC are only created on one machine and aren't
created repeatedly then compared with each other wrt mirroring;

2. Decompression still gives the same result;

3. xz is going to switch to this threaded decompressor output mode
shortly anyway. i.e. there's a slight change in output, but it's
what future versions are going to use anyway. It's deterministic
wrt -T1 and -Tn > 1.

i.e. it's about the compressor method (it produces chunks) rather
than anything else.

Plenty of other projects like LLVM (which also has a large distribution
tarball) use it without any problems.

Best,
sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]