[PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Christian Franke]

[-- Attachment #1: Type: text/plain, Size: 330 bytes --]

A small step towards reproducible packaging...

Currently only tested with upcoming smartmontools package - contains 
only exe, man, doc files (no dll, lib, ...). Multiple cygport runs 
produce binary identical distribution tarballs if SOURCE_DATE_EPOCH 
(from the past) is specified in the cygport file.

-- 
Regards,
Christian


[-- Attachment #2: 0001-Add-initial-support-for-SOURCE_DATE_EPOCH.patch --]
[-- Type: text/plain, Size: 4000 bytes --]

From 146b1df83a20ccd71e57d6123c7ee24b8390ca3a Mon Sep 17 00:00:00 2001
From: Christian Franke <christian.franke@t-online.de>
Date: Wed, 23 Aug 2023 19:27:02 +0200
Subject: [PATCH] Add initial support for SOURCE_DATE_EPOCH

If specified, ensure that the header timestamps of executables
and of compressed man pages are set to this value.
Instruct tar to avoid more recent modification times and
sort all entries by name.
---
 bin/cygport.in           | 14 +++++++++++++-
 lib/pkg_pkg.cygpart      | 11 +++++++++--
 lib/src_postinst.cygpart | 21 +++++++++++++++++++++
 3 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/bin/cygport.in b/bin/cygport.in
index 3f89ac67..f7c476b0 100755
--- a/bin/cygport.in
+++ b/bin/cygport.in
@@ -232,7 +232,7 @@ source ${_privlibdir}/check_funcs.cygpart
 
 # check now for all mandatory programs
 for _myprog in bzip2 cat chmod cp diff diffstat dos2unix file find gawk grep gzip \
-               install ln mkdir mv patch rm rsync sed sort tar xargs which xz
+               install ln mkdir mv patch rm rsync sed sort tar touch xargs which xz
 do
 	if ! check_prog ${_myprog}
 	then
@@ -490,6 +490,18 @@ do
 done
 unset restrict
 
+if [ "${SOURCE_DATE_EPOCH+y}" = "y" ]
+then
+	if [ -n "$(echo "${SOURCE_DATE_EPOCH}" | sed -e 's/^$/X/' -e 's/[0-9]//g')" ]
+	then
+		error "Malformed SOURCE_DATE_EPOCH: '${SOURCE_DATE_EPOCH}'"
+	fi
+	case $(peflags --version 2>/dev/null | sed -n '1s/^.* //p') in
+		4.6.[6-9]|4.[7-9]*|[5-9]*) ;;
+		*) error "SOURCE_DATE_EPOCH requires peflags 4.6.6 or later"
+	esac
+fi
+
 
 ################################################################################
 #
diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 2a2bb663..3869bdb7 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -42,7 +42,7 @@ TAR_COMPRESSION_EXT="${TAR_COMPRESSION_EXT:-xz}"
 #****
 
 __tar() {
-	local TAR_COMPRESSION_OPT;
+	local TAR_COMPRESSION_OPT TAR_SOURCE_DATE_OPTS;
 
 	# We could use --auto-compress, but this also constrains the extension
 	# to the currently valid set. We could probe if tar supports the
@@ -65,7 +65,14 @@ __tar() {
 			error "tar option for TAR_COMPRESSION_EXT='${TAR_COMPRESSION_EXT}' unknown"
 			;;
 	esac
-	tar ${TAR_COMPRESSION_OPT} --owner=Guest:501 --group=None:513 -cvf "$@"
+
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		# Ensure reproducible sort order and last modification times <= SOURCE_DATE_EPOCH
+		TAR_SOURCE_DATE_OPTS="--sort=name --mtime=@${SOURCE_DATE_EPOCH} --clamp-mtime"
+	fi
+
+	tar ${TAR_COMPRESSION_OPT} ${TAR_SOURCE_DATE_OPTS} --owner=Guest:501 --group=None:513 -cvf "$@"
 }
 
 __pkg_binpkg() {
diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart
index dd947311..f0ab0f8b 100644
--- a/lib/src_postinst.cygpart
+++ b/lib/src_postinst.cygpart
@@ -775,6 +775,11 @@ __prepman() {
 		while read -d $'\0' manpage
 		do
 			echo "        ${manpage##*/}";
+			if [ -n "${SOURCE_DATE_EPOCH}" ]
+			then
+				# Ensure that the timestamp in gzip header is reproducible
+				touch -d @${SOURCE_DATE_EPOCH} "${manpage}"
+			fi
 			gzip -q "${manpage}";
 		done
 	fi
@@ -989,6 +994,12 @@ __prepstrip_one() {
 
 	objdump=${objcopy/copy/dump}
 
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		# Let objcopy preserve the timestamps
+		objcopy+=" --enable-deterministic-archives --preserve-dates"
+	fi
+
 	# Static libraries should not be fully stripped, but we can
 	# still provide split debuginfo if desired
 	case "${exe}" in
@@ -1074,6 +1085,16 @@ __prepstrip_one() {
 	# keep sticky bit if present
 	chmod u+w,a+x "${exe}";
 
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		case "${exe}" in
+		*.exe|*.dll|*.so|*.so.*|*.oct|*.mex|*.cmxs)
+			# Ensure PE header timestamp is reproducible and checksum is correct
+			# objcopy later inherits the timestamp to debug info and stripped file
+			peflags --checksum=1 --timestamp=${SOURCE_DATE_EPOCH} ${exe} >/dev/null ;;
+		esac
+	fi
+
 	if defined _CYGPORT_RESTRICT_debuginfo_
 	then
 		${objcopy} --strip-all "${exe}";
-- 
2.39.0

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Brian Inglis]

On 2023-08-23 11:39, Christian Franke via Cygwin-apps wrote:
> A small step towards reproducible packaging...
> 
> Currently only tested with upcoming smartmontools package - contains only exe, 
> man, doc files (no dll, lib, ...). Multiple cygport runs produce binary 
> identical distribution tarballs if SOURCE_DATE_EPOCH (from the past) is 
> specified in the cygport file.

What format is this variable?
Would be best to use some ISO format e.g.

	%Y-%m-%dT%H:%M:%SZ	or
	%Y-%m-%dT%H:%M:%S	[local]

that is human readable and can be used by all POSIX command options
[e.g. see touch(1p) -d] that accept date-time stamps.

Could do with docs and NEWS entries for new variables, otherwise packagers do 
not know it exists, and how to use it e.g.

#****f* Information/BUILD_REQUIRES
#  SYNOPSIS
#  BUILD_REQUIRES="ATOM [ATOM] ..."
#  DESCRIPTION
#  A single-line strings containing a list of packages on which this source
#  package depends at build-time. This will be added to the build-depends:
#  field of the auto-generated -src.hint file.
#  NOTES
#  * Unlike REQUIRES, there is no auto-detection of build-time requirements.
#  * Any newlines in this variable must be escaped.
#  * Unlike the deprecated DEPEND variable, only actual package names, or
#    atoms that are listed PROVIDES of other packages, may be used in this
#    variable.
#****

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Christian Franke]

Brian Inglis via Cygwin-apps wrote:
> On 2023-08-23 11:39, Christian Franke via Cygwin-apps wrote:
>> A small step towards reproducible packaging...
>>
>> Currently only tested with upcoming smartmontools package - contains 
>> only exe, man, doc files (no dll, lib, ...). Multiple cygport runs 
>> produce binary identical distribution tarballs if SOURCE_DATE_EPOCH 
>> (from the past) is specified in the cygport file.
>
> What format is this variable?

Unix epoch, see: https://reproducible-builds.org/docs/source-date-epoch/

Some tools like gcc and gropdf already support this:

$ echo '__DATE__ __TIME__' | SOURCE_DATE_EPOCH=1672531200 gcc -E -xc -
# 0 "<stdin>"
# 0 "<built-in>"
# 0 "<command-line>"
# 1 "<stdin>"
"Jan  1 2023" "00:00:00"

$ TZ=UTC SOURCE_DATE_EPOCH=1672531200 man -Tpdf ls | TZ=UTC pdfinfo -
Creator:        groff version 1.23.0
Producer:       gropdf version 1.23.0
CreationDate:   Sun Jan  1 00:00:00 2023 UTC
ModDate:        Sun Jan  1 00:00:00 2023 UTC
...

If binutils, gzip and tar also would support this, the patch would be 
empty :-)


> Would be best to use some ISO format e.g.
>
>     %Y-%m-%dT%H:%M:%SZ    or
>     %Y-%m-%dT%H:%M:%S    [local]
>
> that is human readable and can be used by all POSIX command options
> [e.g. see touch(1p) -d] that accept date-time stamps.


I use this in smartmontools.cygport:

SOURCE_DATE=2023-08-23
SOURCE_DATE_EPOCH=$(date -d "$SOURCE_DATE UTC" +%s)

This relies on GNU date, BSD date is very different, a POSIX-only date 
would be useless in this context.


> Could do with docs and NEWS entries for new variables, otherwise 
> packagers do not know it exists, and how to use it e.g.
>
> #****f* Information/BUILD_REQUIRES
> #  SYNOPSIS
> ...

Of course this should be documented, but I left this for a later patch.

-- 
Regards,
Christian

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[ASSI]

Christian Franke via Cygwin-apps writes:
> If binutils, gzip and tar also would support this, the patch would be
> empty :-)

GZip has -n though and GNU tar --mtime and --clamp-mtime, so why not use
that?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Christian Franke]

ASSI via Cygwin-apps wrote:
> Christian Franke via Cygwin-apps writes:
>> If binutils, gzip and tar also would support this, the patch would be
>> empty :-)
> GZip has -n though ...

For gzip I decided to keep the timestamp and use (the GNU version of) 
touch instead.


> and GNU tar --mtime and --clamp-mtime, so why not use
> that?

Already done:

+    if [ -n "${SOURCE_DATE_EPOCH}" ]
+    then
+        # Ensure reproducible sort order and last modification times <= 
SOURCE_DATE_EPOCH
+        TAR_SOURCE_DATE_OPTS="--sort=name --mtime=@${SOURCE_DATE_EPOCH} 
--clamp-mtime"
+    fi


:-)

-- 
Regards,
Christian

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Christian Franke]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

Christian Franke wrote:
> A small step towards reproducible packaging...
>
> Currently only tested with upcoming smartmontools package - contains 
> only exe, man, doc files (no dll, lib, ...). Multiple cygport runs 
> produce binary identical distribution tarballs if SOURCE_DATE_EPOCH 
> (from the past) is specified in the cygport file.
>

A slightly enhanced patch is attached. For .gz files, 'gzip -n' is now 
used as suggested.


Result from a test with the cygwin-3.4.8-1 source package:

These two lines were added to cygwin.cygport:

  VERSION="3.4.8"
  RELEASE="1"
+export SOURCE_DATE_EPOCH=$(date -d '2023-08-18 UTC' +%s)
+export TZ=UTC

(TZ=UTC is a workaround for a minor bug in SOURCE_DATE_EPOCH handling of 
gropdf)

These distribution files are always identical:
- cygwin-3.4.8-1-src.tar.xz
- cygwin-doc-3.4.8-1.tar.xz

These individual files differ if the build-path of multiple test-builds 
is always the same:

- usr/bin/cygwin1.dll:
-- Contains a build timestamp not based on '__DATE__ __TIME__'. Could be 
easily addressed in scripts/mkvers.sh.
-- The PE Export Tables header contains a timestamp independent from the 
PE header. There is possibly no easy way to fix this without an extra 
tool (or further enhancing peflags).

- usr/lib/lib*.a:
-- Object file time timestamps differ. Could be addressed by adding 'ar' 
option 'D' (deterministic) in Makefiles and scripts/speclib.

- usr/lib/debug/usr/bin/cygwin1.dll.dbg: As expected, because 
cygwin1.dll differ.

If the build-path changes, more files differ (cygcheck.exe, ldd.exe, 
cygserver.exe) because __FILE__ is used and expands to an absolute path 
name.

-- 
Regards,
Christian


[-- Attachment #2: 0001-Add-initial-support-for-SOURCE_DATE_EPOCH.patch --]
[-- Type: text/plain, Size: 5370 bytes --]

From 73dde4d2dabb74b7b9ee40655204f84e1d4086d6 Mon Sep 17 00:00:00 2001
From: Christian Franke <christian.franke@t-online.de>
Date: Mon, 28 Aug 2023 16:24:36 +0200
Subject: [PATCH] Add initial support for SOURCE_DATE_EPOCH

If specified, set the header timestamps of executables and
patch files to SOURCE_DATE_EPOCH.
Suppress the header timestamp of .gz files.
Instruct tar to avoid more recent modification times and
to sort all entries by name.
---
 bin/cygport.in           | 17 +++++++++++++++--
 lib/pkg_pkg.cygpart      | 20 ++++++++++++++++++--
 lib/src_postinst.cygpart | 22 +++++++++++++++++++---
 3 files changed, 52 insertions(+), 7 deletions(-)

diff --git a/bin/cygport.in b/bin/cygport.in
index 3f89ac67..e2fe1785 100755
--- a/bin/cygport.in
+++ b/bin/cygport.in
@@ -231,8 +231,9 @@ source ${_privlibdir}/check_funcs.cygpart
 ###
 
 # check now for all mandatory programs
-for _myprog in bzip2 cat chmod cp diff diffstat dos2unix file find gawk grep gzip \
-               install ln mkdir mv patch rm rsync sed sort tar xargs which xz
+for _myprog in bzip2 cat chmod cp date diff diffstat dos2unix file find gawk grep \
+               gzip install ln mkdir mv patch rm rsync sed sort tar touch which \
+               xargs xz
 do
 	if ! check_prog ${_myprog}
 	then
@@ -490,6 +491,18 @@ do
 done
 unset restrict
 
+if [ "${SOURCE_DATE_EPOCH+y}" = "y" ]
+then
+	if [ -n "$(echo "${SOURCE_DATE_EPOCH}" | sed -e 's/^$/X/' -e 's/[0-9]//g')" ]
+	then
+		error "Malformed SOURCE_DATE_EPOCH: '${SOURCE_DATE_EPOCH}'"
+	fi
+	case $(peflags --version 2>/dev/null | sed -n '1s/^.* //p') in
+		4.6.[6-9]|4.[7-9]*|[5-9]*) ;;
+		*) error "SOURCE_DATE_EPOCH requires peflags 4.6.6 or later"
+	esac
+fi
+
 
 ################################################################################
 #
diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 2a2bb663..4e6a7cd2 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -42,7 +42,7 @@ TAR_COMPRESSION_EXT="${TAR_COMPRESSION_EXT:-xz}"
 #****
 
 __tar() {
-	local TAR_COMPRESSION_OPT;
+	local TAR_COMPRESSION_OPT TAR_SOURCE_DATE_OPTS;
 
 	# We could use --auto-compress, but this also constrains the extension
 	# to the currently valid set. We could probe if tar supports the
@@ -65,7 +65,14 @@ __tar() {
 			error "tar option for TAR_COMPRESSION_EXT='${TAR_COMPRESSION_EXT}' unknown"
 			;;
 	esac
-	tar ${TAR_COMPRESSION_OPT} --owner=Guest:501 --group=None:513 -cvf "$@"
+
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		# Ensure reproducible sort order and last modification times <= SOURCE_DATE_EPOCH
+		TAR_SOURCE_DATE_OPTS="--sort=name --mtime=@${SOURCE_DATE_EPOCH} --clamp-mtime"
+	fi
+
+	tar ${TAR_COMPRESSION_OPT} ${TAR_SOURCE_DATE_OPTS} --owner=Guest:501 --group=None:513 -cvf "$@"
 }
 
 __pkg_binpkg() {
@@ -319,6 +326,7 @@ __pkg_diff() {
 	local difflevel;
 	local exclude;
 	local optional_patchfiles;
+	local source_date;
 
 	default_excludes="CYGWIN-PATCHES aclocal.m4~ aclocal.m4t autom4te.cache
 		config.cache config.guess config.log config.status config.sub
@@ -498,6 +506,14 @@ __pkg_diff() {
 
 	sed -b -e '/^diff -u/d' -i ${optional_patchfiles} ${patchdir}/${src_patchfile};
 
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		# Ensure that the timestamp comment of the new file is reproducible
+		source_date=$(date -d @"${SOURCE_DATE_EPOCH}" -u +'%Y-%m-%d %H:%M:%S.000000000 +0000')
+		sed -b -e  "s|^\(+++ [^\t]*\t\).*\$|\1${source_date}|" \
+			-i ${optional_patchfiles} ${patchdir}/${src_patchfile}
+	fi
+
 	diffstat -p${difflevel} ${optional_patchfiles} ${patchdir}/${src_patchfile};
 }
 
diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart
index dd947311..53eaa71a 100644
--- a/lib/src_postinst.cygpart
+++ b/lib/src_postinst.cygpart
@@ -41,7 +41,7 @@ __prep_fonts_dir() {
 	for catalogue in ${D}${cataloguedir}/*
 	do
 		fontdir=$(readlink -f ${catalogue})
-		find ${D}${fontdir} -name '*.pcf' -exec gzip -q '{}' +
+		find ${D}${fontdir} -name '*.pcf' -exec gzip -q ${SOURCE_DATE_EPOCH:+-n} '{}' +
 		mkfontscale ${D}${fontdir}
 		mkfontdir ${D}${fontdir}
 	done
@@ -775,7 +775,7 @@ __prepman() {
 		while read -d $'\0' manpage
 		do
 			echo "        ${manpage##*/}";
-			gzip -q "${manpage}";
+			gzip -q ${SOURCE_DATE_EPOCH:+-n} "${manpage}";
 		done
 	fi
 }
@@ -819,7 +819,7 @@ __prepinfo() {
 		while read -d $'\0' infopage
 		do
 			echo "        ${infopage##*/}";
-			gzip -q "${infopage}";
+			gzip -q ${SOURCE_DATE_EPOCH:+-n} "${infopage}";
 		done
 	fi
 }
@@ -989,6 +989,12 @@ __prepstrip_one() {
 
 	objdump=${objcopy/copy/dump}
 
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		# Let objcopy preserve the timestamps
+		objcopy+=" --enable-deterministic-archives --preserve-dates"
+	fi
+
 	# Static libraries should not be fully stripped, but we can
 	# still provide split debuginfo if desired
 	case "${exe}" in
@@ -1074,6 +1080,16 @@ __prepstrip_one() {
 	# keep sticky bit if present
 	chmod u+w,a+x "${exe}";
 
+	if [ -n "${SOURCE_DATE_EPOCH}" ]
+	then
+		case "${exe}" in
+		*.exe|*.dll|*.so|*.so.*|*.oct|*.mex|*.cmxs)
+			# Ensure PE header timestamp is reproducible and checksum is correct
+			# objcopy later inherits the timestamp to debug info and stripped file
+			peflags --checksum=1 --timestamp=${SOURCE_DATE_EPOCH} ${exe} >/dev/null ;;
+		esac
+	fi
+
 	if defined _CYGPORT_RESTRICT_debuginfo_
 	then
 		${objcopy} --strip-all "${exe}";
-- 
2.39.0

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[ASSI]

Christian Franke via Cygwin-apps writes:
> If the build-path changes, more files differ (cygcheck.exe, ldd.exe,
> cygserver.exe) because __FILE__ is used and expands to an absolute
> path name.

This could be fixed by adding -fmacro-prefix and/or -ffile-prefix arguments.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Christian Franke]

ASSI via Cygwin-apps wrote:
> Christian Franke via Cygwin-apps writes:
>> If the build-path changes, more files differ (cygcheck.exe, ldd.exe,
>> cygserver.exe) because __FILE__ is used and expands to an absolute
>> path name.
> This could be fixed by adding -fmacro-prefix and/or -ffile-prefix arguments.

Yes, -fmacro-prefix-map should do the trick.

Meantime I found a workaround for the additional timestamp in the PE 
Export Tables header:
The ld option --no-insert-timestamp also sets this timestamp to 0. This 
is apparently preserved by objcopy, regardless of --preserve-dates option.

-- 
Regards,
Christian

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Jon Turney]

On 28/08/2023 16:12, Christian Franke via Cygwin-apps wrote:
> Christian Franke wrote:
>> A small step towards reproducible packaging...
>>

Thanks very much for this. Sorry for taking so long to look at it.

A few questions and suggestions interspersed

[...]
> 
> -- 
> Regards,
> Christian
> 
> 
> 0001-Add-initial-support-for-SOURCE_DATE_EPOCH.patch
> 
>  From 73dde4d2dabb74b7b9ee40655204f84e1d4086d6 Mon Sep 17 00:00:00 2001
> From: Christian Franke<christian.franke@t-online.de>
> Date: Mon, 28 Aug 2023 16:24:36 +0200
> Subject: [PATCH] Add initial support for SOURCE_DATE_EPOCH
> 
> If specified, set the header timestamps of executables and
> patch files to SOURCE_DATE_EPOCH.
> Suppress the header timestamp of .gz files.
> Instruct tar to avoid more recent modification times and
> to sort all entries by name.
> ---
>   bin/cygport.in           | 17 +++++++++++++++--
>   lib/pkg_pkg.cygpart      | 20 ++++++++++++++++++--
>   lib/src_postinst.cygpart | 22 +++++++++++++++++++---
>   3 files changed, 52 insertions(+), 7 deletions(-)
> 
> diff --git a/bin/cygport.in b/bin/cygport.in
> index 3f89ac67..e2fe1785 100755
> --- a/bin/cygport.in
> +++ b/bin/cygport.in
> @@ -231,8 +231,9 @@ source ${_privlibdir}/check_funcs.cygpart
>   ###
>   
>   # check now for all mandatory programs
> -for _myprog in bzip2 cat chmod cp diff diffstat dos2unix file find gawk grep gzip \
> -               install ln mkdir mv patch rm rsync sed sort tar xargs which xz
> +for _myprog in bzip2 cat chmod cp date diff diffstat dos2unix file find gawk grep \
> +               gzip install ln mkdir mv patch rm rsync sed sort tar touch which \
> +               xargs xz
>   do
>   	if ! check_prog ${_myprog}
>   	then
> @@ -490,6 +491,18 @@ do
>   done
>   unset restrict
>   
> +if [ "${SOURCE_DATE_EPOCH+y}" = "y" ]
> +then
> +	if [ -n "$(echo "${SOURCE_DATE_EPOCH}" | sed -e 's/^$/X/' -e 's/[0-9]//g')" ]
> +	then
> +		error "Malformed SOURCE_DATE_EPOCH: '${SOURCE_DATE_EPOCH}'"

This error message should perhaps say what a well-formed 
SOURCE_DATE_EPOCH looks like: an integer number of seconds since the 
unix epoch?

> +	fi
> +	case $(peflags --version 2>/dev/null | sed -n '1s/^.* //p') in
> +		4.6.[6-9]|4.[7-9]*|[5-9]*) ;;
> +		*) error "SOURCE_DATE_EPOCH requires peflags 4.6.6 or later"
> +	esac
> +fi
> +
>   
>   ################################################################################
>   #
> diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
> index 2a2bb663..4e6a7cd2 100644
> --- a/lib/pkg_pkg.cygpart
> +++ b/lib/pkg_pkg.cygpart
> @@ -42,7 +42,7 @@ TAR_COMPRESSION_EXT="${TAR_COMPRESSION_EXT:-xz}"
>   #****
>   
>   __tar() {
> -	local TAR_COMPRESSION_OPT;
> +	local TAR_COMPRESSION_OPT TAR_SOURCE_DATE_OPTS;
>   
>   	# We could use --auto-compress, but this also constrains the extension
>   	# to the currently valid set. We could probe if tar supports the
> @@ -65,7 +65,14 @@ __tar() {
>   			error "tar option for TAR_COMPRESSION_EXT='${TAR_COMPRESSION_EXT}' unknown"
>   			;;
>   	esac
> -	tar ${TAR_COMPRESSION_OPT} --owner=Guest:501 --group=None:513 -cvf "$@"
> +
> +	if [ -n "${SOURCE_DATE_EPOCH}" ]
> +	then
> +		# Ensure reproducible sort order and last modification times <= SOURCE_DATE_EPOCH
> +		TAR_SOURCE_DATE_OPTS="--sort=name --mtime=@${SOURCE_DATE_EPOCH} --clamp-mtime"

I'm slightly concerned that maybe this is masking problems elsewhere, at 
least when making the source archive: In 2eb7c0eb I started making an 
effort so that if the "source inputs" have fixed, upstream timestamps, 
we'll preserve those in the output the source package.

(Obviously that's not always going to be the case, e.g. where cygport 
patches are from a local git checkout, so maybe that's not a real 
problem...)

> +	fi
> +
> +	tar ${TAR_COMPRESSION_OPT} ${TAR_SOURCE_DATE_OPTS} --owner=Guest:501 --group=None:513 -cvf "$@"
>   }
>   
>   __pkg_binpkg() {
> @@ -319,6 +326,7 @@ __pkg_diff() {
>   	local difflevel;
>   	local exclude;
>   	local optional_patchfiles;
> +	local source_date;
>   
>   	default_excludes="CYGWIN-PATCHES aclocal.m4~ aclocal.m4t autom4te.cache
>   		config.cache config.guess config.log config.status config.sub
> @@ -498,6 +506,14 @@ __pkg_diff() {
>   
>   	sed -b -e '/^diff -u/d' -i ${optional_patchfiles} ${patchdir}/${src_patchfile};
>   
> +	if [ -n "${SOURCE_DATE_EPOCH}" ]
> +	then
> +		# Ensure that the timestamp comment of the new file is reproducible

"timestamp comments in the generated patch files" or suchlike?

> +		source_date=$(date -d @"${SOURCE_DATE_EPOCH}" -u +'%Y-%m-%d %H:%M:%S.000000000 +0000')
> +		sed -b -e  "s|^\(+++ [^\t]*\t\).*\$|\1${source_date}|" \
> +			-i ${optional_patchfiles} ${patchdir}/${src_patchfile}
> +	fi
> +
>   	diffstat -p${difflevel} ${optional_patchfiles} ${patchdir}/${src_patchfile};
>   }
>   
> diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart
> index dd947311..53eaa71a 100644
> --- a/lib/src_postinst.cygpart
> +++ b/lib/src_postinst.cygpart
> @@ -41,7 +41,7 @@ __prep_fonts_dir() {
>   	for catalogue in ${D}${cataloguedir}/*
>   	do
>   		fontdir=$(readlink -f ${catalogue})
> -		find ${D}${fontdir} -name '*.pcf' -exec gzip -q '{}' +
> +		find ${D}${fontdir} -name '*.pcf' -exec gzip -q ${SOURCE_DATE_EPOCH:+-n} '{}' +
>   		mkfontscale ${D}${fontdir}
>   		mkfontdir ${D}${fontdir}
>   	done
> @@ -775,7 +775,7 @@ __prepman() {
>   		while read -d $'\0' manpage
>   		do
>   			echo "        ${manpage##*/}";
> -			gzip -q "${manpage}";
> +			gzip -q ${SOURCE_DATE_EPOCH:+-n} "${manpage}";
>   		done
>   	fi
>   }
> @@ -819,7 +819,7 @@ __prepinfo() {
>   		while read -d $'\0' infopage
>   		do
>   			echo "        ${infopage##*/}";
> -			gzip -q "${infopage}";
> +			gzip -q ${SOURCE_DATE_EPOCH:+-n} "${infopage}";
>   		done
>   	fi
>   }
> @@ -989,6 +989,12 @@ __prepstrip_one() {
>   
>   	objdump=${objcopy/copy/dump}
>   
> +	if [ -n "${SOURCE_DATE_EPOCH}" ]
> +	then
> +		# Let objcopy preserve the timestamps

I found this comment confusing.

"Do not embed timestamps in archives"

Why to we need "--preserve-dates"? The input file presumably has a 
timestamp when the build actually took place, which we're going to clamp 
to SOURCE_DATE_EPOCH when we make the archive?

It seems like maybe "--enable-deterministic-archives" should be in 
LDFLAGS, if the build produces static or unstripped archives?

> +		objcopy+=" --enable-deterministic-archives --preserve-dates"
> +	fi
> +
>   	# Static libraries should not be fully stripped, but we can
>   	# still provide split debuginfo if desired
>   	case "${exe}" in
> @@ -1074,6 +1080,16 @@ __prepstrip_one() {
>   	# keep sticky bit if present
>   	chmod u+w,a+x "${exe}";
>   
> +	if [ -n "${SOURCE_DATE_EPOCH}" ]
> +	then
> +		case "${exe}" in
> +		*.exe|*.dll|*.so|*.so.*|*.oct|*.mex|*.cmxs)
> +			# Ensure PE header timestamp is reproducible and checksum is correct
> +			# objcopy later inherits the timestamp to debug info and stripped file
> +			peflags --checksum=1 --timestamp=${SOURCE_DATE_EPOCH} ${exe} >/dev/null ;;
> +		esac
> +	fi
> +
>   	if defined_CYGPORT_RESTRICT_debuginfo_
>   	then
>   		${objcopy} --strip-all "${exe}";

Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH

[Christian Franke]

Jon Turney wrote:
> On 28/08/2023 16:12, Christian Franke via Cygwin-apps wrote:
>> Christian Franke wrote:
>>> A small step towards reproducible packaging...
>>>
>
> Thanks very much for this. Sorry for taking so long to look at it.
>

No problem.


> A few questions and suggestions interspersed
>
> [...]
>>
>> -- 
>> Regards,
>> Christian
>>
>>
>> 0001-Add-initial-support-for-SOURCE_DATE_EPOCH.patch
>>
>>  From 73dde4d2dabb74b7b9ee40655204f84e1d4086d6 Mon Sep 17 00:00:00 2001
>> From: Christian Franke<christian.franke@t-online.de>
>> Date: Mon, 28 Aug 2023 16:24:36 +0200
>> Subject: [PATCH] Add initial support for SOURCE_DATE_EPOCH
>>
>> If specified, set the header timestamps of executables and
>> patch files to SOURCE_DATE_EPOCH.
>> Suppress the header timestamp of .gz files.
>> Instruct tar to avoid more recent modification times and
>> to sort all entries by name.
>> ---
>>   bin/cygport.in           | 17 +++++++++++++++--
>>   lib/pkg_pkg.cygpart      | 20 ++++++++++++++++++--
>>   lib/src_postinst.cygpart | 22 +++++++++++++++++++---
>>   3 files changed, 52 insertions(+), 7 deletions(-)
>>
>> diff --git a/bin/cygport.in b/bin/cygport.in
>> index 3f89ac67..e2fe1785 100755
>> --- a/bin/cygport.in
>> +++ b/bin/cygport.in
>> @@ -231,8 +231,9 @@ source ${_privlibdir}/check_funcs.cygpart
>>   ###
>>     # check now for all mandatory programs
>> -for _myprog in bzip2 cat chmod cp diff diffstat dos2unix file find 
>> gawk grep gzip \
>> -               install ln mkdir mv patch rm rsync sed sort tar xargs 
>> which xz
>> +for _myprog in bzip2 cat chmod cp date diff diffstat dos2unix file 
>> find gawk grep \
>> +               gzip install ln mkdir mv patch rm rsync sed sort tar 
>> touch which \
>> +               xargs xz
>>   do
>>       if ! check_prog ${_myprog}
>>       then
>> @@ -490,6 +491,18 @@ do
>>   done
>>   unset restrict
>>   +if [ "${SOURCE_DATE_EPOCH+y}" = "y" ]
>> +then
>> +    if [ -n "$(echo "${SOURCE_DATE_EPOCH}" | sed -e 's/^$/X/' -e 
>> 's/[0-9]//g')" ]
>> +    then
>> +        error "Malformed SOURCE_DATE_EPOCH: '${SOURCE_DATE_EPOCH}'"
>
> This error message should perhaps say what a well-formed 
> SOURCE_DATE_EPOCH looks like: an integer number of seconds since the 
> unix epoch?
>

error "SOURCE_DATE_EPOCH must be an integer number (seconds since the 
unix epoch)" ?


>> +    fi
>> +    case $(peflags --version 2>/dev/null | sed -n '1s/^.* //p') in
>> +        4.6.[6-9]|4.[7-9]*|[5-9]*) ;;
>> +        *) error "SOURCE_DATE_EPOCH requires peflags 4.6.6 or later"
>> +    esac
>> +fi
>> +
>> ################################################################################
>>   #
>> diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
>> index 2a2bb663..4e6a7cd2 100644
>> --- a/lib/pkg_pkg.cygpart
>> +++ b/lib/pkg_pkg.cygpart
>> @@ -42,7 +42,7 @@ TAR_COMPRESSION_EXT="${TAR_COMPRESSION_EXT:-xz}"
>>   #****
>>     __tar() {
>> -    local TAR_COMPRESSION_OPT;
>> +    local TAR_COMPRESSION_OPT TAR_SOURCE_DATE_OPTS;
>>         # We could use --auto-compress, but this also constrains the 
>> extension
>>       # to the currently valid set. We could probe if tar supports the
>> @@ -65,7 +65,14 @@ __tar() {
>>               error "tar option for 
>> TAR_COMPRESSION_EXT='${TAR_COMPRESSION_EXT}' unknown"
>>               ;;
>>       esac
>> -    tar ${TAR_COMPRESSION_OPT} --owner=Guest:501 --group=None:513 
>> -cvf "$@"
>> +
>> +    if [ -n "${SOURCE_DATE_EPOCH}" ]
>> +    then
>> +        # Ensure reproducible sort order and last modification times 
>> <= SOURCE_DATE_EPOCH
>> +        TAR_SOURCE_DATE_OPTS="--sort=name 
>> --mtime=@${SOURCE_DATE_EPOCH} --clamp-mtime"
>
> I'm slightly concerned that maybe this is masking problems elsewhere, 
> at least when making the source archive: In 2eb7c0eb I started making 
> an effort so that if the "source inputs" have fixed, upstream 
> timestamps, we'll preserve those in the output the source package.
>
> (Obviously that's not always going to be the case, e.g. where cygport 
> patches are from a local git checkout, so maybe that's not a real 
> problem...)

If the following condition holds, the timestamps of "source inputs" are 
not affected:
newest_source_timestamp < SOURCE_DATE_EPOCH < build_time


>
>> +    fi
>> +
>> +    tar ${TAR_COMPRESSION_OPT} ${TAR_SOURCE_DATE_OPTS} 
>> --owner=Guest:501 --group=None:513 -cvf "$@"
>>   }
>>     __pkg_binpkg() {
>> @@ -319,6 +326,7 @@ __pkg_diff() {
>>       local difflevel;
>>       local exclude;
>>       local optional_patchfiles;
>> +    local source_date;
>>         default_excludes="CYGWIN-PATCHES aclocal.m4~ aclocal.m4t 
>> autom4te.cache
>>           config.cache config.guess config.log config.status config.sub
>> @@ -498,6 +506,14 @@ __pkg_diff() {
>>         sed -b -e '/^diff -u/d' -i ${optional_patchfiles} 
>> ${patchdir}/${src_patchfile};
>>   +    if [ -n "${SOURCE_DATE_EPOCH}" ]
>> +    then
>> +        # Ensure that the timestamp comment of the new file is 
>> reproducible
>
> "timestamp comments in the generated patch files" or suchlike?

OK.


>
>
>> +        source_date=$(date -d @"${SOURCE_DATE_EPOCH}" -u +'%Y-%m-%d 
>> %H:%M:%S.000000000 +0000')
>> +        sed -b -e  "s|^\(+++ [^\t]*\t\).*\$|\1${source_date}|" \
>> +            -i ${optional_patchfiles} ${patchdir}/${src_patchfile}
>> +    fi
>> +
>>       diffstat -p${difflevel} ${optional_patchfiles} 
>> ${patchdir}/${src_patchfile};
>>   }
>>   diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart
>> index dd947311..53eaa71a 100644
>> --- a/lib/src_postinst.cygpart
>> +++ b/lib/src_postinst.cygpart
>> @@ -41,7 +41,7 @@ __prep_fonts_dir() {
>>       for catalogue in ${D}${cataloguedir}/*
>>       do
>>           fontdir=$(readlink -f ${catalogue})
>> -        find ${D}${fontdir} -name '*.pcf' -exec gzip -q '{}' +
>> +        find ${D}${fontdir} -name '*.pcf' -exec gzip -q 
>> ${SOURCE_DATE_EPOCH:+-n} '{}' +
>>           mkfontscale ${D}${fontdir}
>>           mkfontdir ${D}${fontdir}
>>       done
>> @@ -775,7 +775,7 @@ __prepman() {
>>           while read -d $'\0' manpage
>>           do
>>               echo "        ${manpage##*/}";
>> -            gzip -q "${manpage}";
>> +            gzip -q ${SOURCE_DATE_EPOCH:+-n} "${manpage}";
>>           done
>>       fi
>>   }
>> @@ -819,7 +819,7 @@ __prepinfo() {
>>           while read -d $'\0' infopage
>>           do
>>               echo "        ${infopage##*/}";
>> -            gzip -q "${infopage}";
>> +            gzip -q ${SOURCE_DATE_EPOCH:+-n} "${infopage}";
>>           done
>>       fi
>>   }
>> @@ -989,6 +989,12 @@ __prepstrip_one() {
>>         objdump=${objcopy/copy/dump}
>>   +    if [ -n "${SOURCE_DATE_EPOCH}" ]
>> +    then
>> +        # Let objcopy preserve the timestamps
>
> I found this comment confusing.
>
> "Do not embed timestamps in archives"

OK.


>
> Why to we need "--preserve-dates"? The input file presumably has a 
> timestamp when the build actually took place, which we're going to 
> clamp to SOURCE_DATE_EPOCH when we make the archive?

Option --preserve-dates also preserves the PE header timestamps. This is 
missing in the objcopy documentation, The timestamp of an original exe 
is set to SOURCE_DATE_EPOCH by peflags and then preserved by objcopy.


>
>
> It seems like maybe "--enable-deterministic-archives" should be in 
> LDFLAGS, if the build produces static or unstripped archives?

$ ld --enable-deterministic-archives foo.o
ld: unrecognized option '--enable-deterministic-archives'

Ld supports --no-insert-timestamp which is a different approach which 
would work without peflags.


>
>> +        objcopy+=" --enable-deterministic-archives --preserve-dates"
>> +    fi
>> +
>>       # Static libraries should not be fully stripped, but we can
>>       # still provide split debuginfo if desired
>>       case "${exe}" in
>> @@ -1074,6 +1080,16 @@ __prepstrip_one() {
>>       # keep sticky bit if present
>>       chmod u+w,a+x "${exe}";
>>   +    if [ -n "${SOURCE_DATE_EPOCH}" ]
>> +    then
>> +        case "${exe}" in
>> +        *.exe|*.dll|*.so|*.so.*|*.oct|*.mex|*.cmxs)
>> +            # Ensure PE header timestamp is reproducible and 
>> checksum is correct
>> +            # objcopy later inherits the timestamp to debug info and 
>> stripped file
>> +            peflags --checksum=1 --timestamp=${SOURCE_DATE_EPOCH} 
>> ${exe} >/dev/null ;;
>> +        esac
>> +    fi
>> +
>>       if defined_CYGPORT_RESTRICT_debuginfo_
>>       then
>>           ${objcopy} --strip-all "${exe}";
>
>