* [PATCH cygport] Add repro-check command
@ 2024-02-18 15:54 Christian Franke
2024-03-01 19:16 ` Christian Franke
0 siblings, 1 reply; 3+ messages in thread
From: Christian Franke @ 2024-02-18 15:54 UTC (permalink / raw)
To: cygwin-apps
[-- Attachment #1: Type: text/plain, Size: 545 bytes --]
This could be used to check whether a package is possibly reproducible.
Then it could make sense to add a reasonable SOURCE_DATE_EPOCH value to
the cygport file.
Example:
$ export SOURCE_DATE_EPOCH=$(date +%s)
$ cygport project.cygport all repro-check
...
*** Info: Build reproducibility test succeeded
$ TZ=UTC cygport project.cygport repro-check
...
*** Info: Build reproducibility test succeeded
$ unset SOURCE_DATE_EPOCH
$ cygport project.cygport repro-check
...
*** ERROR: Build reproducibility test failed
--
Regards,
Christian
[-- Attachment #2: 0001-Add-repro-check-command.patch --]
[-- Type: text/plain, Size: 3859 bytes --]
From 97f518478dac722647b8a423068f2a5461c82f19 Mon Sep 17 00:00:00 2001
From: Christian Franke <christian.franke@t-online.de>
Date: Sun, 18 Feb 2024 16:33:07 +0100
Subject: [PATCH] Add repro-check command
This command checks for reproducibility of distribution packages.
The source package from the dist directory is unpacked to the
temp directory. A nested rebuild of the packages is run there.
If successful, original and rebuild packages are compared and the
result is reported.
---
README | 1 +
bin/cygport.in | 8 ++++++++
lib/help.cygpart | 1 +
lib/pkg_pkg.cygpart | 42 +++++++++++++++++++++++++++++++++++++++++-
4 files changed, 51 insertions(+), 1 deletion(-)
diff --git a/README b/README
index fd16df6b..fec46b13 100644
--- a/README
+++ b/README
@@ -163,6 +163,7 @@ Other COMMANDs are meant primarily for maintainers:
diff - write a patch file capturing changes to source in the working directory
stage - as upload, but don't request processing of uploaded packages
announce - compose and send a package announcement
+ repro-check - check whether a rebuild produces binary identical packages
The standard arguments --help or --version may also be passed to cygport.
diff --git a/bin/cygport.in b/bin/cygport.in
index 5fc89eaf..6acbc85b 100755
--- a/bin/cygport.in
+++ b/bin/cygport.in
@@ -29,6 +29,10 @@ set -e;
#
################################################################################
+# Preserve original environment for repro-check command
+declare -r _cygport_orig_env=$(export)
+declare -r _cygport_orig_pwd=$(pwd)
+
# for regexes, sort, etc.
export LC_COLLATE=C
@@ -784,6 +788,10 @@ do
test ${PIPESTATUS[0]} -eq 0
_status=$?;
;;
+ repro-check)
+ __pkg_repro_check
+ _status=$?
+ ;;
help)
__show_help;
exit 0;
diff --git a/lib/help.cygpart b/lib/help.cygpart
index a7f30f7a..d851762e 100644
--- a/lib/help.cygpart
+++ b/lib/help.cygpart
@@ -56,6 +56,7 @@ __show_help() {
finish delete the working directory
all run prep, compile, install and package
all-test run prep, compile, install and package-test
+ repro-check check whether a rebuild produces binary identical packages
See the included README file for further documentation.
diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 756a687c..719ffcd1 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -992,6 +992,46 @@ _EOF
fi
}
+__pkg_repro_check() {
+ local rc srcpkg t_cygport t_spkgdir
+
+ srcpkg=${distdir}/${PN}/${PF}-src.tar.${TAR_COMPRESSION_EXT}
+ t_spkgdir=${T}/${spkgdir##*/}
+
+ echo
+ __stage "Checking reproducibility of"
+
+ echo
+ __step "Unpacking ${srcpkg}"
+ [ -f "${srcpkg}" ] || error "Packages not built yet"
+ tar xf ${srcpkg} -C ${T} || error "tar xf ${srcpkg} -C ${T} failed"
+
+ echo
+ __step "Rebuilding in ${t_spkgdir}"
+ t_cygport="cygport ${cygportfile} finish all"
+ echo "${_cygport_orig_env}" > ${T}/.cygport_orig_env
+ __step "=== Start: ${t_cygport} ================================="
+
+ # Start nested cygport with original environment in temp directory
+ rc=0
+ env --chdir=${_cygport_orig_pwd} --ignore-environment /bin/bash -c \
+ "source ${T}/.cygport_orig_env && cd ${t_spkgdir} && ${t_cygport}" \
+ || rc=$?
+
+ __step "=== Done: ${t_cygport} (exit $rc) ========================="
+ echo
+ [ $rc = 0 ] || error "Rebuild in ${t_spkgdir} failed"
+
+ __step "Comparing original and rebuilt packages"
+ if ! diff -qr ${distdir} ${t_spkgdir}/${PF}.${ARCH}/dist
+ then
+ echo
+ error "Build reproducibility test failed"
+ fi
+ echo
+ inform "Build reproducibility test succeeded"
+}
+
# protect functions
readonly -f __pkg_binpkg __pkg_diff __gpg_sign __pkg_srcpkg __pkg_dist \
- __squeeze_whitespace __tar
+ __pkg_repro_check __squeeze_whitespace __tar
--
2.43.0
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH cygport] Add repro-check command
2024-02-18 15:54 [PATCH cygport] Add repro-check command Christian Franke
@ 2024-03-01 19:16 ` Christian Franke
2024-03-10 17:20 ` Jon Turney
0 siblings, 1 reply; 3+ messages in thread
From: Christian Franke @ 2024-03-01 19:16 UTC (permalink / raw)
To: cygwin-apps
[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]
Christian Franke wrote:
> This could be used to check whether a package is possibly
> reproducible. Then it could make sense to add a reasonable
> SOURCE_DATE_EPOCH value to the cygport file.
>
> Example:
>
> $ export SOURCE_DATE_EPOCH=$(date +%s)
>
> $ cygport project.cygport all repro-check
> ...
> *** Info: Build reproducibility test succeeded
>
> $ TZ=UTC cygport project.cygport repro-check
> ...
> *** Info: Build reproducibility test succeeded
>
> $ unset SOURCE_DATE_EPOCH
>
> $ cygport project.cygport repro-check
> ...
> *** ERROR: Build reproducibility test failed
>
An enhanced version of the patch is attached. The build and diff could
now be run also individually and the diff report includes individual
files from the packages.
As a side effect, this enables another use case: Check whether changes
to cygport only change the expected files.
$ cygport project.cygport all repro-check
...
*** Info: Rebuild produced identical packages
$ editor project.cygport
... Change some comments ...
$ cygport project.cygport all repro-diff
...
Differing files found:
! dist/project/project-1.2-3-src.tar.xz
! spkg/project-1.2-3.src/project.cygport
*** ERROR: Rebuild differs from original
--
Regards,
Christian
[-- Attachment #2: 0001-Add-repro-build-repro-diff-und-repro-check-commands.patch --]
[-- Type: text/plain, Size: 5097 bytes --]
From 152a21dfad4c786cff1712e6aa1c33f2db0b6a75 Mon Sep 17 00:00:00 2001
From: Christian Franke <christian.franke@t-online.de>
Date: Fri, 1 Mar 2024 19:46:50 +0100
Subject: [PATCH] Add repro-build, repro-diff und repro-check commands
These commands check for reproducibility of distribution packages.
The repro-build command unpacks the source package from the dist
directory to the temp directory and performs a nested rebuild of
the packages there.
The repro-diff command compares original and rebuild packages.
If different, a report about individual differing files in dist,
inst and spkg directories is printed.
The repro-check command combines both commands.
---
README | 3 +++
bin/cygport.in | 17 +++++++++++++
lib/help.cygpart | 3 +++
lib/pkg_pkg.cygpart | 58 ++++++++++++++++++++++++++++++++++++++++++++-
4 files changed, 80 insertions(+), 1 deletion(-)
diff --git a/README b/README
index fd16df6b..3c9e4d4a 100644
--- a/README
+++ b/README
@@ -163,6 +163,9 @@ Other COMMANDs are meant primarily for maintainers:
diff - write a patch file capturing changes to source in the working directory
stage - as upload, but don't request processing of uploaded packages
announce - compose and send a package announcement
+ repro-build - rebuild from created source package to temp directory
+ repro-diff - check whether packages from original and rebuild differ
+ repro-check - run repro-build and repro-diff
The standard arguments --help or --version may also be passed to cygport.
diff --git a/bin/cygport.in b/bin/cygport.in
index 5fc89eaf..a2c2b5a3 100755
--- a/bin/cygport.in
+++ b/bin/cygport.in
@@ -29,6 +29,10 @@ set -e;
#
################################################################################
+# Preserve original environment for repro-check command
+declare -r _cygport_orig_env=$(export)
+declare -r _cygport_orig_pwd=$(pwd)
+
# for regexes, sort, etc.
export LC_COLLATE=C
@@ -784,6 +788,19 @@ do
test ${PIPESTATUS[0]} -eq 0
_status=$?;
;;
+ repro-build)
+ __pkg_repro_build
+ _status=$?
+ ;;
+ repro-diff)
+ __pkg_repro_diff
+ _status=$?
+ ;;
+ repro-check)
+ __pkg_repro_build && \
+ __pkg_repro_diff
+ _status=$?
+ ;;
help)
__show_help;
exit 0;
diff --git a/lib/help.cygpart b/lib/help.cygpart
index a7f30f7a..d28fd7bb 100644
--- a/lib/help.cygpart
+++ b/lib/help.cygpart
@@ -56,6 +56,9 @@ __show_help() {
finish delete the working directory
all run prep, compile, install and package
all-test run prep, compile, install and package-test
+ repro-build rebuild from created source package to temp directory
+ repro-diff check whether packages from original and rebuild differ
+ repro-check run repro-build and repro-diff
See the included README file for further documentation.
diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 756a687c..3c531c0e 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -992,6 +992,62 @@ _EOF
fi
}
+__pkg_repro_build() {
+ local srcpkg=${distdir}/${PN}/${PF}-src.tar.${TAR_COMPRESSION_EXT}
+ local t_spkgdir=${T}/${spkgdir##*/}
+ local t_workdir=${t_spkgdir}/${PF}.${ARCH}
+ local t_cygport="cygport ${cygportfile} finish all"
+ local rc
+
+ __stage "Rebuilding"
+ __step "Unpacking ${srcpkg}"
+ [ -f ${srcpkg} ] || error "Packages not built yet"
+ tar xf ${srcpkg} -C ${T} || error "tar xf ${srcpkg} -C ${T} failed"
+
+ __step "Rebuilding in ${t_spkgdir}"
+ echo "${_cygport_orig_env}" > ${T}/.cygport_orig_env
+ echo
+ __step "=== Start: ${t_cygport} ================================="
+
+ # Start nested cygport with original environment in temp directory
+ rc=0
+ env --chdir=${_cygport_orig_pwd} --ignore-environment /bin/bash -c \
+ "source ${T}/.cygport_orig_env && cd ${t_spkgdir} && ${t_cygport}" \
+ || rc=$?
+
+ __step "=== Done: ${t_cygport} (exit $rc) ========================="
+ echo
+ [ $rc = 0 ] || error "Rebuild failed"
+}
+
+__pkg_repro_diff() {
+ local t_spkgdir=${T}/${spkgdir##*/}
+ local t_workdir=${t_spkgdir}/${PF}.${ARCH}
+ local t_srcpkg=${t_workdir}/dist/${PN}/${PF}-src.tar.${TAR_COMPRESSION_EXT}
+ local d
+
+ __stage "Comparing original and rebuild of"
+ inform "Rebuild dir: ${t_spkgdir}"
+ [ -f ${t_srcpkg} ] || error "Packages not rebuilt yet"
+
+ if ! diff -qr --no-dereference ${distdir} ${t_workdir}/dist >/dev/null
+ then
+ echo "Differing files found:"
+ for d in dist inst spkg
+ do
+ LC_MESSAGES=C \
+ diff -qr --no-dereference ${workdir}/${d} ${t_workdir}/${d} \
+ | sed -n -e "s|^Files ${workdir}/\([^ ][^ ]*\) and .* differ\$|! \\1|p" \
+ -e "s|^Only in ${workdir}/\(${d}/[^:]*\): |- \\1/|p" \
+ -e "s|^Only in ${t_workdir}/\(${d}/[^:]*\): |+ \\1/|p"
+ done
+ echo
+ error "Rebuild differs from original"
+ fi
+ echo
+ inform "Rebuild produced identical packages"
+}
+
# protect functions
readonly -f __pkg_binpkg __pkg_diff __gpg_sign __pkg_srcpkg __pkg_dist \
- __squeeze_whitespace __tar
+ __pkg_repro_build __pkg_repro_diff __squeeze_whitespace __tar
--
2.43.0
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH cygport] Add repro-check command
2024-03-01 19:16 ` Christian Franke
@ 2024-03-10 17:20 ` Jon Turney
0 siblings, 0 replies; 3+ messages in thread
From: Jon Turney @ 2024-03-10 17:20 UTC (permalink / raw)
To: Christian Franke; +Cc: cygwin-apps
On 01/03/2024 19:16, Christian Franke via Cygwin-apps wrote:
> Christian Franke wrote:
>> This could be used to check whether a package is possibly
>> reproducible. Then it could make sense to add a reasonable
>> SOURCE_DATE_EPOCH value to the cygport file.
>>
[...]
>
> An enhanced version of the patch is attached. The build and diff could
> now be run also individually and the diff report includes individual
> files from the packages.
>
> As a side effect, this enables another use case: Check whether changes
> to cygport only change the expected files.
>
> $ cygport project.cygport all repro-check
> ...
> *** Info: Rebuild produced identical packages
Applied. Thanks!
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-03-10 17:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-18 15:54 [PATCH cygport] Add repro-check command Christian Franke
2024-03-01 19:16 ` Christian Franke
2024-03-10 17:20 ` Jon Turney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).