public inbox for binutils@sourceware.org
 help / color / mirror / Atom feed
* RFC: Sort tarballs created by the src-release.sh script
@ 2022-09-28 12:59 Nick Clifton
  2022-09-28 13:05 ` Andreas Schwab
  0 siblings, 1 reply; 16+ messages in thread
From: Nick Clifton @ 2022-09-28 12:59 UTC (permalink / raw)
  To: Binutils, gdb-patches; +Cc: Tzvetelin Katchov

Hi Guys,

   A contributor recently pointed out that the binutils release tarballs are
   not sorted by name, making it hard to reproducibly recreate them.  At first
   we thought that using tar's --sort=name option would solve this, but it
   turns out that the src-release.sh script creates its own list of input file
   names, so instead I am proposing this patch:

diff --git a/src-release.sh b/src-release.sh
index 079b545ae7c..caae5148034 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -185,7 +185,7 @@ do_tar()
      echo "==> Making $package-$ver.tar"
      rm -f $package-$ver.tar
      find $package-$ver -follow \( $CVS_NAMES \) -prune \
-	-o -type f -print \
+	-o -type f -print | sort \
  	| tar cTfh - $package-$ver.tar
  }

   Any comments or objections ?

Cheers
   Nick


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-09-28 12:59 RFC: Sort tarballs created by the src-release.sh script Nick Clifton
@ 2022-09-28 13:05 ` Andreas Schwab
  2022-09-28 13:34   ` Nick Clifton
  0 siblings, 1 reply; 16+ messages in thread
From: Andreas Schwab @ 2022-09-28 13:05 UTC (permalink / raw)
  To: Nick Clifton via Gdb-patches; +Cc: Binutils, Nick Clifton, Tzvetelin Katchov

On Sep 28 2022, Nick Clifton via Gdb-patches wrote:

> diff --git a/src-release.sh b/src-release.sh
> index 079b545ae7c..caae5148034 100755
> --- a/src-release.sh
> +++ b/src-release.sh
> @@ -185,7 +185,7 @@ do_tar()
>      echo "==> Making $package-$ver.tar"
>      rm -f $package-$ver.tar
>      find $package-$ver -follow \( $CVS_NAMES \) -prune \
> -	-o -type f -print \
> +	-o -type f -print | sort \

Better set LC_ALL=C to be independent from the locale sorting.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-09-28 13:05 ` Andreas Schwab
@ 2022-09-28 13:34   ` Nick Clifton
  2022-09-29 12:24     ` Nick Clifton
  0 siblings, 1 reply; 16+ messages in thread
From: Nick Clifton @ 2022-09-28 13:34 UTC (permalink / raw)
  To: Andreas Schwab, Nick Clifton via Gdb-patches; +Cc: Binutils, Tzvetelin Katchov

Hi Andreas,

> Better set LC_ALL=C to be independent from the locale sorting.
Like this ?

diff --git a/src-release.sh b/src-release.sh
index 079b545ae7c..8c2a8d897fd 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -185,7 +185,7 @@ do_tar()
      echo "==> Making $package-$ver.tar"
      rm -f $package-$ver.tar
      find $package-$ver -follow \( $CVS_NAMES \) -prune \
-       -o -type f -print \
+       -o -type f -print | LC_ALL=C sort \
         | tar cTfh - $package-$ver.tar
  }


Cheers
   Nick

PS.  Would sort's --dictionary-order option have a similar effect ?


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-09-28 13:34   ` Nick Clifton
@ 2022-09-29 12:24     ` Nick Clifton
  2022-09-29 12:36       ` Jan Beulich
  0 siblings, 1 reply; 16+ messages in thread
From: Nick Clifton @ 2022-09-29 12:24 UTC (permalink / raw)
  To: Andreas Schwab, Nick Clifton via Gdb-patches; +Cc: Binutils, Tzvetelin Katchov

Hi Guys,

Thinking about this a little more last night, it occurred to me that if
we want reproducible tarballs then we should not be storing user names,
group names or modification times either. So what do you think about
this extended version of the patch:

diff --git a/src-release.sh b/src-release.sh
index 079b545ae7c..908492c28f7 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -185,8 +185,8 @@ do_tar()
      echo "==> Making $package-$ver.tar"
      rm -f $package-$ver.tar
      find $package-$ver -follow \( $CVS_NAMES \) -prune \
-       -o -type f -print \
-       | tar cTfh - $package-$ver.tar
+       -o -type f -print | LC_ALL=C sort \
+       | tar cTfh - $package-$ver.tar --mtime=0 --group=0 --owner=0
  }

  # Compress the output with bzip2


Cheers
   Nick


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-09-29 12:24     ` Nick Clifton
@ 2022-09-29 12:36       ` Jan Beulich
  2022-09-30 11:38         ` Nick Clifton
  0 siblings, 1 reply; 16+ messages in thread
From: Jan Beulich @ 2022-09-29 12:36 UTC (permalink / raw)
  To: Nick Clifton
  Cc: Binutils, Tzvetelin Katchov, Andreas Schwab,
	Nick Clifton via Gdb-patches

On 29.09.2022 14:24, Nick Clifton via Binutils wrote:
> Thinking about this a little more last night, it occurred to me that if
> we want reproducible tarballs then we should not be storing user names,
> group names or modification times either. So what do you think about
> this extended version of the patch:
> 
> diff --git a/src-release.sh b/src-release.sh
> index 079b545ae7c..908492c28f7 100755
> --- a/src-release.sh
> +++ b/src-release.sh
> @@ -185,8 +185,8 @@ do_tar()
>       echo "==> Making $package-$ver.tar"
>       rm -f $package-$ver.tar
>       find $package-$ver -follow \( $CVS_NAMES \) -prune \
> -       -o -type f -print \
> -       | tar cTfh - $package-$ver.tar
> +       -o -type f -print | LC_ALL=C sort \
> +       | tar cTfh - $package-$ver.tar --mtime=0 --group=0 --owner=0
>   }

I wanted to indicate that an mtime of zero isn't the neatest, but the
two tar versions I've tried this with said anyway "Treating date '0' as
2022-09-29 00:00:00". A non-zero date with a time of zero is fine with
me, but won't make much of a difference in terms of reproducibility.

Jan

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-09-29 12:36       ` Jan Beulich
@ 2022-09-30 11:38         ` Nick Clifton
  2022-10-02  7:54           ` Sam James
  0 siblings, 1 reply; 16+ messages in thread
From: Nick Clifton @ 2022-09-30 11:38 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Binutils, Tzvetelin Katchov, Andreas Schwab,
	Nick Clifton via Gdb-patches

Hi Guys,

   Right, here is the latest and greatest - and hopefully last - version
   of the patch.  I added a parseable string to the --mtime option and a
   comment explaining why these options are being used.

   Any more comments/suggestions ?

Cheers
   Nick

diff --git a/src-release.sh b/src-release.sh
index 079b545ae7c..8a2ac125030 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -184,9 +184,11 @@ do_tar()
      ver=$2
      echo "==> Making $package-$ver.tar"
      rm -f $package-$ver.tar
+    # The sort command and --mtime, --group and --owner options are
+    # used in order to create consistent, reproducible tarballs.
      find $package-$ver -follow \( $CVS_NAMES \) -prune \
-       -o -type f -print \
-       | tar cTfh - $package-$ver.tar
+       -o -type f -print | LC_ALL=C sort \
+       | tar cTfh - $package-$ver.tar --mtime="1970-01-01 00:00:00" --group=0 --owner=0
  }

  # Compress the output with bzip2


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-09-30 11:38         ` Nick Clifton
@ 2022-10-02  7:54           ` Sam James
  2022-10-03  6:55             ` Jan Beulich
  2022-10-03  7:47             ` Andreas Schwab
  0 siblings, 2 replies; 16+ messages in thread
From: Sam James @ 2022-10-02  7:54 UTC (permalink / raw)
  To: Nick Clifton
  Cc: Jan Beulich, Andreas Schwab, Binutils,
	Nick Clifton via Gdb-patches, Tzvetelin Katchov

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]



> On 30 Sep 2022, at 12:38, Nick Clifton via Binutils <binutils@sourceware.org> wrote:
> 
> Hi Guys,
> 
>  Right, here is the latest and greatest - and hopefully last - version
>  of the patch.  I added a parseable string to the --mtime option and a
>  comment explaining why these options are being used.
> 
>  Any more comments/suggestions ?
> 
> Cheers
>  Nick
> 
> diff --git a/src-release.sh b/src-release.sh
> index 079b545ae7c..8a2ac125030 100755
> --- a/src-release.sh
> +++ b/src-release.sh
> @@ -184,9 +184,11 @@ do_tar()
>     ver=$2
>     echo "==> Making $package-$ver.tar"
>     rm -f $package-$ver.tar
> +    # The sort command and --mtime, --group and --owner options are
> +    # used in order to create consistent, reproducible tarballs.
>     find $package-$ver -follow \( $CVS_NAMES \) -prune \
> -       -o -type f -print \
> -       | tar cTfh - $package-$ver.tar
> +       -o -type f -print | LC_ALL=C sort \
> +       | tar cTfh - $package-$ver.tar --mtime="1970-01-01 00:00:00" --group=0 --owner=0
> }
> 
> # Compress the output with bzip2
> 

I think this might hit a problem I faced when trying to do this with Go tarballs: https://www.gnu.org/software/tar/manual/tar.html#warnings.

With that date, I got "implausibly old time stamp" warnings from tar. I haven't tested this patchthough (writing from mobile, apologies).

Maybe default to the creation date of Binutils and allow overriding via https://reproducible-builds.org/docs/source-date-epoch/?

best,
sam

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-02  7:54           ` Sam James
@ 2022-10-03  6:55             ` Jan Beulich
  2022-10-03  6:59               ` Sam James
  2022-10-03 14:40               ` Nick Clifton
  2022-10-03  7:47             ` Andreas Schwab
  1 sibling, 2 replies; 16+ messages in thread
From: Jan Beulich @ 2022-10-03  6:55 UTC (permalink / raw)
  To: Sam James, Nick Clifton
  Cc: Andreas Schwab, Binutils, Nick Clifton via Gdb-patches,
	Tzvetelin Katchov

On 02.10.2022 09:54, Sam James wrote:
> 
> 
>> On 30 Sep 2022, at 12:38, Nick Clifton via Binutils <binutils@sourceware.org> wrote:
>>
>> Hi Guys,
>>
>>  Right, here is the latest and greatest - and hopefully last - version
>>  of the patch.  I added a parseable string to the --mtime option and a
>>  comment explaining why these options are being used.
>>
>>  Any more comments/suggestions ?
>>
>> Cheers
>>  Nick
>>
>> diff --git a/src-release.sh b/src-release.sh
>> index 079b545ae7c..8a2ac125030 100755
>> --- a/src-release.sh
>> +++ b/src-release.sh
>> @@ -184,9 +184,11 @@ do_tar()
>>     ver=$2
>>     echo "==> Making $package-$ver.tar"
>>     rm -f $package-$ver.tar
>> +    # The sort command and --mtime, --group and --owner options are
>> +    # used in order to create consistent, reproducible tarballs.
>>     find $package-$ver -follow \( $CVS_NAMES \) -prune \
>> -       -o -type f -print \
>> -       | tar cTfh - $package-$ver.tar
>> +       -o -type f -print | LC_ALL=C sort \
>> +       | tar cTfh - $package-$ver.tar --mtime="1970-01-01 00:00:00" --group=0 --owner=0
>> }
>>
>> # Compress the output with bzip2
>>
> 
> I think this might hit a problem I faced when trying to do this with Go tarballs: https://www.gnu.org/software/tar/manual/tar.html#warnings.
> 
> With that date, I got "implausibly old time stamp" warnings from tar. I haven't tested this patchthough (writing from mobile, apologies).
> 
> Maybe default to the creation date of Binutils and allow overriding via https://reproducible-builds.org/docs/source-date-epoch/?

Not sure what "creation date" might mean here. Assuming the script is
(typically) run from a git tree, perhaps the commit date of the top
level commit on the branch would be best to use?

Jan

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-03  6:55             ` Jan Beulich
@ 2022-10-03  6:59               ` Sam James
  2022-10-03  7:41                 ` Andreas Schwab
  2022-10-03 14:40               ` Nick Clifton
  1 sibling, 1 reply; 16+ messages in thread
From: Sam James @ 2022-10-03  6:59 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Nick Clifton, Andreas Schwab, Binutils,
	Nick Clifton via Gdb-patches, Tzvetelin Katchov



> On 3 Oct 2022, at 07:56, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 02.10.2022 09:54, Sam James wrote:
>> 
>> 
>>>> On 30 Sep 2022, at 12:38, Nick Clifton via Binutils <binutils@sourceware.org> wrote:
>>> 
>>> Hi Guys,
>>> 
>>> Right, here is the latest and greatest - and hopefully last - version
>>> of the patch.  I added a parseable string to the --mtime option and a
>>> comment explaining why these options are being used.
>>> 
>>> Any more comments/suggestions ?
>>> 
>>> Cheers
>>> Nick
>>> 
>>> diff --git a/src-release.sh b/src-release.sh
>>> index 079b545ae7c..8a2ac125030 100755
>>> --- a/src-release.sh
>>> +++ b/src-release.sh
>>> @@ -184,9 +184,11 @@ do_tar()
>>>    ver=$2
>>>    echo "==> Making $package-$ver.tar"
>>>    rm -f $package-$ver.tar
>>> +    # The sort command and --mtime, --group and --owner options are
>>> +    # used in order to create consistent, reproducible tarballs.
>>>    find $package-$ver -follow \( $CVS_NAMES \) -prune \
>>> -       -o -type f -print \
>>> -       | tar cTfh - $package-$ver.tar
>>> +       -o -type f -print | LC_ALL=C sort \
>>> +       | tar cTfh - $package-$ver.tar --mtime="1970-01-01 00:00:00" --group=0 --owner=0
>>> }
>>> 
>>> # Compress the output with bzip2
>>> 
>> 
>> I think this might hit a problem I faced when trying to do this with Go tarballs: https://www.gnu.org/software/tar/manual/tar.html#warnings.
>> 
>> With that date, I got "implausibly old time stamp" warnings from tar. I haven't tested this patchthough (writing from mobile, apologies).
>> 
>> Maybe default to the creation date of Binutils and allow overriding via https://reproducible-builds.org/docs/source-date-epoch/?
> 
> Not sure what "creation date" might mean here.

I meant whatever date folks consider to have been the creation of the Binutils project. First commit, announcement, first release, whatever.

Not that it really matters, it just can't be the unix epoch.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-03  6:59               ` Sam James
@ 2022-10-03  7:41                 ` Andreas Schwab
  0 siblings, 0 replies; 16+ messages in thread
From: Andreas Schwab @ 2022-10-03  7:41 UTC (permalink / raw)
  To: Sam James via Gdb-patches
  Cc: Jan Beulich, Sam James, Tzvetelin Katchov, Nick Clifton, Binutils

On Okt 03 2022, Sam James via Gdb-patches wrote:

> I meant whatever date folks consider to have been the creation of the Binutils project. First commit, announcement, first release, whatever.

I think using the date of the commit from which the tarball is being
created makes the most sense (this is what git archive does).

Another thing to consider is the recorded permission of the files.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-02  7:54           ` Sam James
  2022-10-03  6:55             ` Jan Beulich
@ 2022-10-03  7:47             ` Andreas Schwab
  1 sibling, 0 replies; 16+ messages in thread
From: Andreas Schwab @ 2022-10-03  7:47 UTC (permalink / raw)
  To: Sam James via Gdb-patches
  Cc: Nick Clifton, Sam James, Binutils, Jan Beulich, Tzvetelin Katchov

On Okt 02 2022, Sam James via Gdb-patches wrote:

> With that date, I got "implausibly old time stamp" warnings from tar.

That happens because --mtime uses the local timezone.  To get the true
Epoch you can use --mtime=@0.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-03  6:55             ` Jan Beulich
  2022-10-03  6:59               ` Sam James
@ 2022-10-03 14:40               ` Nick Clifton
  2022-10-03 19:56                 ` Andreas Schwab
  2022-10-04  7:10                 ` Jan Beulich
  1 sibling, 2 replies; 16+ messages in thread
From: Nick Clifton @ 2022-10-03 14:40 UTC (permalink / raw)
  To: Jan Beulich, Sam James
  Cc: Andreas Schwab, Binutils, Nick Clifton via Gdb-patches,
	Tzvetelin Katchov

Hi Guys,

   [This appears to be getting slightly out of hand...]

> Not sure what "creation date" might mean here. Assuming the script is > (typically) run from a git tree, perhaps the commit date of the top> level commit on the branch would be best to use?
Except that a commit to the branch that does not affect something that
would go into the tarball would then result in a changed date.

We could use the src-release.sh file itself, like this:

diff --git a/src-release.sh b/src-release.sh
index 079b545ae7c..de1f98a70bb 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -184,9 +184,15 @@ do_tar()
      ver=$2
      echo "==> Making $package-$ver.tar"
      rm -f $package-$ver.tar
+    # The sort command and --mtime, --group and --owner options are
+    # used in order to create consistent, reproducible tarballs.
+    # BUILD_DATE is set to SOURCE_DATE_EPOCH if defined, or the
+    # modification date of this file otherwise.  cf:
+    # https://reproducible-builds.org/docs/source-date-epoch/
+    BUILD_DATE="$(date --utc --date="@${SOURCE_DATE_EPOCH:-$(date -r src-release.sh +%s)}" +%Y-%m-%d)"
      find $package-$ver -follow \( $CVS_NAMES \) -prune \
-       -o -type f -print \
-       | tar cTfh - $package-$ver.tar
+       -o -type f -print | LC_ALL=C sort \
+       | tar cTfh - $package-$ver.tar --mtime=$BUILD_DATE --group=0 --owner=0
  }

  # Compress the output with bzip2


Would that work ?

Cheers
   Nick


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-03 14:40               ` Nick Clifton
@ 2022-10-03 19:56                 ` Andreas Schwab
  2022-10-04  7:10                 ` Jan Beulich
  1 sibling, 0 replies; 16+ messages in thread
From: Andreas Schwab @ 2022-10-03 19:56 UTC (permalink / raw)
  To: Nick Clifton
  Cc: Jan Beulich, Sam James, Binutils, Nick Clifton via Gdb-patches,
	Tzvetelin Katchov

On Okt 03 2022, Nick Clifton wrote:

> We could use the src-release.sh file itself, like this:

The timestamp of checked out files has no meaning, and is generally not
reproducible.

> diff --git a/src-release.sh b/src-release.sh
> index 079b545ae7c..de1f98a70bb 100755
> --- a/src-release.sh
> +++ b/src-release.sh
> @@ -184,9 +184,15 @@ do_tar()
>      ver=$2
>      echo "==> Making $package-$ver.tar"
>      rm -f $package-$ver.tar
> +    # The sort command and --mtime, --group and --owner options are
> +    # used in order to create consistent, reproducible tarballs.
> +    # BUILD_DATE is set to SOURCE_DATE_EPOCH if defined, or the
> +    # modification date of this file otherwise.  cf:
> +    # https://reproducible-builds.org/docs/source-date-epoch/
> +    BUILD_DATE="$(date --utc --date="@${SOURCE_DATE_EPOCH:-$(date -r src-release.sh +%s)}" +%Y-%m-%d)"
>      find $package-$ver -follow \( $CVS_NAMES \) -prune \
> -       -o -type f -print \
> -       | tar cTfh - $package-$ver.tar
> +       -o -type f -print | LC_ALL=C sort \
> +       | tar cTfh - $package-$ver.tar --mtime=$BUILD_DATE --group=0 --owner=0

That won't work, as --mtime=$BUILD_DATE is interpreted in the local time zone.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-03 14:40               ` Nick Clifton
  2022-10-03 19:56                 ` Andreas Schwab
@ 2022-10-04  7:10                 ` Jan Beulich
  2022-10-05 12:23                   ` Nick Clifton
  1 sibling, 1 reply; 16+ messages in thread
From: Jan Beulich @ 2022-10-04  7:10 UTC (permalink / raw)
  To: Nick Clifton
  Cc: Andreas Schwab, Binutils, Nick Clifton via Gdb-patches,
	Tzvetelin Katchov, Sam James

On 03.10.2022 16:40, Nick Clifton wrote:
> Hi Guys,
> 
>    [This appears to be getting slightly out of hand...]

It might seem like that, yes, but I guess that's not entirely unexpected
for a topic of that kind - different people have different expectations
and habits.

>> Not sure what "creation date" might mean here. Assuming the script is > (typically) run from a git tree, perhaps the commit date of the top> level commit on the branch would be best to use?
> Except that a commit to the branch that does not affect something that
> would go into the tarball would then result in a changed date.

Every commit should be considered to affect the tarball, imo, as such
effects could also be indirect. If you really wanted to go that route,
then perhaps an alternative would be to use the commit date of the
most recent commit touching bfd/version.m4.

Jan

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-04  7:10                 ` Jan Beulich
@ 2022-10-05 12:23                   ` Nick Clifton
  2022-10-05 13:00                     ` Jan Beulich
  0 siblings, 1 reply; 16+ messages in thread
From: Nick Clifton @ 2022-10-05 12:23 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Andreas Schwab, Binutils, Nick Clifton via Gdb-patches,
	Tzvetelin Katchov, Sam James

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

Hi Guys,

On 10/4/22 08:10, Jan Beulich wrote:
> Every commit should be considered to affect the tarball, imo, as such
> effects could also be indirect. If you really wanted to go that route,
> then perhaps an alternative would be to use the commit date of the
> most recent commit touching bfd/version.m4.

Hmm, except that would probably only be appropriate for binutils tarballs,
not others.

So how about the attached patch ?  This one adds a new command line option to
src-release.sh.  If it is not used then the behaviour is not changed in any
way.  If the new option is used, it provides a date that is passed to tar's
--mtime option, along with triggering the use of sort and the other tar
options necessary to make a reproducible tarball.  So:

   src-release.sh -x -r `git log -1 --format=%cd --date=format:%F bfd/version.m4` binutils

should create a pretty consistent tarball.

Cheers
   Nick



[-- Attachment #2: src-release.sh.patch --]
[-- Type: text/x-patch, Size: 1656 bytes --]

diff --git a/src-release.sh b/src-release.sh
index 079b545ae7c..6e16cb00992 100755
--- a/src-release.sh
+++ b/src-release.sh
@@ -30,6 +30,7 @@ SHA256PROG=sha256sum
 MAKE=make
 CC=gcc
 CXX=g++
+release_date=
 
 # Default to avoid splitting info files by setting the threshold high.
 MAKEINFOFLAGS=--split-size=5000000
@@ -184,9 +185,17 @@ do_tar()
     ver=$2
     echo "==> Making $package-$ver.tar"
     rm -f $package-$ver.tar
-    find $package-$ver -follow \( $CVS_NAMES \) -prune \
-	-o -type f -print \
-	| tar cTfh - $package-$ver.tar
+    if test x$release_date == "x" ; then
+       find $package-$ver -follow \( $CVS_NAMES \) -prune -o -type f -print \
+	   | tar cTfh - $package-$ver.tar
+    else
+	# Attempt to create a consistent, reproducible tarball using the
+	# specified date.
+	find $package-$ver -follow \( $CVS_NAMES \) -prune -o -type f -print \
+	    | LC_ALL=C sort \
+	    | tar cTfh - $package-$ver.tar \
+		  --mtime=$release_date --group=0 --owner=0
+    fi
 }
 
 # Compress the output with bzip2
@@ -340,6 +349,7 @@ usage()
     echo "  -g: Compress with gzip"
     echo "  -l: Compress with lzip"
     echo "  -x: Compress with xz"
+    echo "  -r <date>: Create a reproducible tarall using <date> as the mtime"
     exit 1
 }
 
@@ -363,7 +373,7 @@ build_release()
 
 compressors=""
 
-while getopts ":bglx" opt; do
+while getopts ":bglr:x" opt; do
     case $opt in
 	b)
 	    compressors="$compressors bz2";;
@@ -371,6 +381,8 @@ while getopts ":bglx" opt; do
 	    compressors="$compressors gz";;
 	l)
 	    compressors="$compressors lz";;
+	r)
+	    release_date=$OPTARG;;
 	x)
 	    compressors="$compressors xz";;
 	\?)

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: RFC: Sort tarballs created by the src-release.sh script
  2022-10-05 12:23                   ` Nick Clifton
@ 2022-10-05 13:00                     ` Jan Beulich
  0 siblings, 0 replies; 16+ messages in thread
From: Jan Beulich @ 2022-10-05 13:00 UTC (permalink / raw)
  To: Nick Clifton
  Cc: Andreas Schwab, Binutils, Nick Clifton via Gdb-patches,
	Tzvetelin Katchov, Sam James

On 05.10.2022 14:23, Nick Clifton wrote:
> Hi Guys,
> 
> On 10/4/22 08:10, Jan Beulich wrote:
>> Every commit should be considered to affect the tarball, imo, as such
>> effects could also be indirect. If you really wanted to go that route,
>> then perhaps an alternative would be to use the commit date of the
>> most recent commit touching bfd/version.m4.
> 
> Hmm, except that would probably only be appropriate for binutils tarballs,
> not others.
> 
> So how about the attached patch ?  This one adds a new command line option to
> src-release.sh.  If it is not used then the behaviour is not changed in any
> way.  If the new option is used, it provides a date that is passed to tar's
> --mtime option, along with triggering the use of sort and the other tar
> options necessary to make a reproducible tarball.  So:
> 
>    src-release.sh -x -r `git log -1 --format=%cd --date=format:%F bfd/version.m4` binutils
> 
> should create a pretty consistent tarball.

Lgtm, fwiw. Just one nit: You may want to add the missing 'b' for
"tarball" in the new help text line.

Jan

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2022-10-05 13:00 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-28 12:59 RFC: Sort tarballs created by the src-release.sh script Nick Clifton
2022-09-28 13:05 ` Andreas Schwab
2022-09-28 13:34   ` Nick Clifton
2022-09-29 12:24     ` Nick Clifton
2022-09-29 12:36       ` Jan Beulich
2022-09-30 11:38         ` Nick Clifton
2022-10-02  7:54           ` Sam James
2022-10-03  6:55             ` Jan Beulich
2022-10-03  6:59               ` Sam James
2022-10-03  7:41                 ` Andreas Schwab
2022-10-03 14:40               ` Nick Clifton
2022-10-03 19:56                 ` Andreas Schwab
2022-10-04  7:10                 ` Jan Beulich
2022-10-05 12:23                   ` Nick Clifton
2022-10-05 13:00                     ` Jan Beulich
2022-10-03  7:47             ` Andreas Schwab

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).