[PATCH] src-release.sh: Use -T0 for xz compression

[PATCH] src-release.sh: Use -T0 for xz compression

[Mark Wielaard]

Use parallel compression to create the xz archive.

On my machine (using 4 cores) this reduces the time to create
binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.

xz has supported -T0 since version 5.2.0 (released 2014-12-21).
---
 src-release.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src-release.sh b/src-release.sh
index e454893298da..f29362aac207 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -239,7 +239,7 @@ do_xz()
     ver=$2
     echo "==> Xzipping $package-$ver.tar.xz"
     rm -f $package-$ver.tar.xz
-    $XZPROG -k -v -9 $package-$ver.tar
+    $XZPROG -k -v -9 -T0 $package-$ver.tar
 }
 
 # Compress the output with zstd
-- 
2.45.1

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

Mark Wielaard <mark@klomp.org> writes:

> Use parallel compression to create the xz archive.
>
> On my machine (using 4 cores) this reduces the time to create
> binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
>
> xz has supported -T0 since version 5.2.0 (released 2014-12-21).

LGTM and it's the default from >=5.6.0 anyway. For >=5.4.0, you can use
it for parallel *de*compression too.

> ---
>  src-release.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src-release.sh b/src-release.sh
> index e454893298da..f29362aac207 100755
> --- a/src-release.sh
> +++ b/src-release.sh
> @@ -239,7 +239,7 @@ do_xz()
>      ver=$2
>      echo "==> Xzipping $package-$ver.tar.xz"
>      rm -f $package-$ver.tar.xz
> -    $XZPROG -k -v -9 $package-$ver.tar
> +    $XZPROG -k -v -9 -T0 $package-$ver.tar
>  }
>  
>  # Compress the output with zstd

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

Sam James <sam@gentoo.org> writes:

> Mark Wielaard <mark@klomp.org> writes:
>
>> Use parallel compression to create the xz archive.
>>
>> On my machine (using 4 cores) this reduces the time to create
>> binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
>>
>> xz has supported -T0 since version 5.2.0 (released 2014-12-21).
>
> LGTM and it's the default from >=5.6.0 anyway. For >=5.4.0, you can use
> it for parallel *de*compression too.
>

Note that we made the same change for GCC a while ago in r13-4261-g2650b6f9147c0d.

>> ---
>>  src-release.sh | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/src-release.sh b/src-release.sh
>> index e454893298da..f29362aac207 100755
>> --- a/src-release.sh
>> +++ b/src-release.sh
>> @@ -239,7 +239,7 @@ do_xz()
>>      ver=$2
>>      echo "==> Xzipping $package-$ver.tar.xz"
>>      rm -f $package-$ver.tar.xz
>> -    $XZPROG -k -v -9 $package-$ver.tar
>> +    $XZPROG -k -v -9 -T0 $package-$ver.tar
>>  }
>>  
>>  # Compress the output with zstd

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Simon Richter]

Hi,

On 5/31/24 06:43, Mark Wielaard wrote:

> On my machine (using 4 cores) this reduces the time to create
> binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.

Is the output still deterministic from the input?

    Simon

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Eli Zaretskii]

> From: Sam James <sam@gentoo.org>
> Cc: binutils@sourceware.org,  gdb-patches@sourceware.org
> Date: Thu, 30 May 2024 22:56:50 +0100
> 
> > On my machine (using 4 cores) this reduces the time to create
> > binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
> >
> > xz has supported -T0 since version 5.2.0 (released 2014-12-21).
> 
> LGTM and it's the default from >=5.6.0 anyway.

The same 5.6.0 which introduced the backdoor?  How are its defaults
relevant, when no one in their right mind will ever use those
versions??

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Mark Wielaard]

Hi Simon,

On Fri, May 31, 2024 at 11:04:32AM +0900, Simon Richter wrote:
> On 5/31/24 06:43, Mark Wielaard wrote:
> 
> >On my machine (using 4 cores) this reduces the time to create
> >binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
> 
> Is the output still deterministic from the input?

Yes, as long as one uses mult-threaded mode -T0, it doesn't matter how
many cores are used. It does differ from the output of serial mode
though, when no -T is used.

Cheers,

Mark

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Sam James <sam@gentoo.org>
>> Cc: binutils@sourceware.org,  gdb-patches@sourceware.org
>> Date: Thu, 30 May 2024 22:56:50 +0100
>> 
>> > On my machine (using 4 cores) this reduces the time to create
>> > binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
>> >
>> > xz has supported -T0 since version 5.2.0 (released 2014-12-21).
>> 
>> LGTM and it's the default from >=5.6.0 anyway.
>
> The same 5.6.0 which introduced the backdoor?  How are its defaults
> relevant, when no one in their right mind will ever use those
> versions??

xz continues as a project after this incident and fixed releases were
issued 2 days ago. I'm not sure what your point is here.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Richard Earnshaw (lists)]

On 31/05/2024 10:47, Mark Wielaard wrote:
> Hi Simon,
> 
> On Fri, May 31, 2024 at 11:04:32AM +0900, Simon Richter wrote:
>> On 5/31/24 06:43, Mark Wielaard wrote:
>>
>>> On my machine (using 4 cores) this reduces the time to create
>>> binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
>>
>> Is the output still deterministic from the input?
> 
> Yes, as long as one uses mult-threaded mode -T0, it doesn't matter how
> many cores are used. It does differ from the output of serial mode
> though, when no -T is used.
> 
> Cheers,
> 
> Mark

Perhaps more importantly, for reproducibility, does it produce different output if run on a machine with a different number of available CPUs?  That might matter if you're validating the checksums against a version generated by somebody else from supposedly the same sources.

R.

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

"Richard Earnshaw (lists)" <Richard.Earnshaw@arm.com> writes:

> On 31/05/2024 10:47, Mark Wielaard wrote:
>> Hi Simon,
>> 
>> On Fri, May 31, 2024 at 11:04:32AM +0900, Simon Richter wrote:
>>> On 5/31/24 06:43, Mark Wielaard wrote:
>>>
>>>> On my machine (using 4 cores) this reduces the time to create
>>>> binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
>>>
>>> Is the output still deterministic from the input?
>> 
>> Yes, as long as one uses mult-threaded mode -T0, it doesn't matter how
>> many cores are used. It does differ from the output of serial mode
>> though, when no -T is used.
>> 
>> Cheers,
>> 
>> Mark
>
> Perhaps more importantly, for reproducibility, does it produce
> different output if run on a machine with a different number of
> available CPUs?  That might matter if you're validating the checksums
> against a version generated by somebody else from supposedly the same
> sources.

No, fortunately, the threaded compressor output is deterministic
irrespective of the hardware it is run on.

That is, xz has two output modes here:
1) the non-threaded compressor, which doesn't chunk into blocks;
2) the threaded compressor, which chunks into blocks.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Mark Wielaard]

Hi Richard,

On Fri, 2024-05-31 at 14:18 +0100, Richard Earnshaw (lists) wrote:
> On 31/05/2024 10:47, Mark Wielaard wrote:
> > On Fri, May 31, 2024 at 11:04:32AM +0900, Simon Richter wrote:
> > > On 5/31/24 06:43, Mark Wielaard wrote:
> > > 
> > > > On my machine (using 4 cores) this reduces the time to create
> > > > binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
> > > 
> > > Is the output still deterministic from the input?
> > 
> > Yes, as long as one uses mult-threaded mode -T0, it doesn't matter how
> > many cores are used. It does differ from the output of serial mode
> > though, when no -T is used.
> 
> Perhaps more importantly, for reproducibility, does it produce different
> output if run on a machine with a different number of available CPUs?

No, output is reproducible independent of number of cores used with -T.

Cheers,

Mark

Re: [PATCH] src-release.sh: Use -T0 for xz compression

[Mark Wielaard]

Hi,

On Thu, May 30, 2024 at 10:56:50PM +0100, Sam James wrote:
> Mark Wielaard <mark@klomp.org> writes:
> > Use parallel compression to create the xz archive.
> >
> > On my machine (using 4 cores) this reduces the time to create
> > binutils-2.42.50.tar.xz from 1 minute 40 seconds to 56 seconds.
> >
> > xz has supported -T0 since version 5.2.0 (released 2014-12-21).
> 
> LGTM and it's the default from >=5.6.0 anyway.

Thanks. I have pushed this now because I believe all questions people
had have been answered. And it helps the snapshot builder become a bit
faster.

Cheers,

Mark