From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout11.t-online.de (mailout11.t-online.de [194.25.134.85]) by sourceware.org (Postfix) with ESMTPS id 175183858CDA for ; Mon, 30 Oct 2023 17:43:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 175183858CDA Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=t-online.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=t-online.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 175183858CDA Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=194.25.134.85 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698687820; cv=none; b=nrl+Q7Prr5s80bbod7L+sz+PjOgG6fQbmYEv/078o056lNc+fry0bZxNgQr9bZuVWymiyQwCVTw4pA1lMy+lukB98GECsqMvl3NwQAu4YBCl9riSvRVGQNogeTpHc87LrAWSIzWyM17bqq2yfe5BCGDJJuohqxiL3mbUPt1SnLI= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698687820; c=relaxed/simple; bh=BIubfXiivlENSKC6rD/A6A76nrcMEAz3Byvw1QY4NPU=; h=Subject:To:From:Message-ID:Date:MIME-Version; b=Lez0xf17ObP5V2cunwIGBld6r3Cexl9T8oaM/jJE0WnLMZYXcySHajmU9QW5kXR+vQAOQAPr0WOtX1NEv6m7MWLBXd9j4Du9SbDfDB00jJa9zu3R0tJ6KvH7Hhp0zMeL9r4/Z3OSI5i3DpkI44egXNm89JNCj1bJdUdYRERDUts= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from fwd82.aul.t-online.de (fwd82.aul.t-online.de [10.223.144.108]) by mailout11.t-online.de (Postfix) with SMTP id B0A891D47A for ; Mon, 30 Oct 2023 18:43:36 +0100 (CET) Received: from [192.168.2.101] ([79.230.168.59]) by fwd82.t-online.de with (TLSv1.3:TLS_AES_256_GCM_SHA384 encrypted) esmtp id 1qxWIZ-3OIHwW0; Mon, 30 Oct 2023 18:43:35 +0100 Subject: Re: [PATCH cygport] Add initial support for SOURCE_DATE_EPOCH To: cygwin-apps@cygwin.com References: <000a4c2b-e6eb-4a16-e3c4-c5cedab1867b@t-online.de> <92055123-6de0-44dd-888a-904b496d9037@dronecode.org.uk> From: Christian Franke Message-ID: <82d5dad3-d0e6-5c74-4a92-211143a785bc@t-online.de> Date: Mon, 30 Oct 2023 18:43:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 SeaMonkey/2.53.16 MIME-Version: 1.0 In-Reply-To: <92055123-6de0-44dd-888a-904b496d9037@dronecode.org.uk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TOI-EXPURGATEID: 150726::1698687815-A0FF6ED1-B59E7007/0/0 CLEAN NORMAL X-TOI-MSGID: 6d47d074-7712-435c-8973-2018378c9351 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00,BODY_8BITS,FREEMAIL_FROM,GIT_PATCH_0,KAM_DMARC_STATUS,NICE_REPLY_A,RCVD_IN_BARRACUDACENTRAL,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Jon Turney wrote: > On 28/08/2023 16:12, Christian Franke via Cygwin-apps wrote: >> Christian Franke wrote: >>> A small step towards reproducible packaging... >>> > > Thanks very much for this. Sorry for taking so long to look at it. > No problem. > A few questions and suggestions interspersed > > [...] >> >> -- >> Regards, >> Christian >> >> >> 0001-Add-initial-support-for-SOURCE_DATE_EPOCH.patch >> >>  From 73dde4d2dabb74b7b9ee40655204f84e1d4086d6 Mon Sep 17 00:00:00 2001 >> From: Christian Franke >> Date: Mon, 28 Aug 2023 16:24:36 +0200 >> Subject: [PATCH] Add initial support for SOURCE_DATE_EPOCH >> >> If specified, set the header timestamps of executables and >> patch files to SOURCE_DATE_EPOCH. >> Suppress the header timestamp of .gz files. >> Instruct tar to avoid more recent modification times and >> to sort all entries by name. >> --- >>   bin/cygport.in           | 17 +++++++++++++++-- >>   lib/pkg_pkg.cygpart      | 20 ++++++++++++++++++-- >>   lib/src_postinst.cygpart | 22 +++++++++++++++++++--- >>   3 files changed, 52 insertions(+), 7 deletions(-) >> >> diff --git a/bin/cygport.in b/bin/cygport.in >> index 3f89ac67..e2fe1785 100755 >> --- a/bin/cygport.in >> +++ b/bin/cygport.in >> @@ -231,8 +231,9 @@ source ${_privlibdir}/check_funcs.cygpart >>   ### >>     # check now for all mandatory programs >> -for _myprog in bzip2 cat chmod cp diff diffstat dos2unix file find >> gawk grep gzip \ >> -               install ln mkdir mv patch rm rsync sed sort tar xargs >> which xz >> +for _myprog in bzip2 cat chmod cp date diff diffstat dos2unix file >> find gawk grep \ >> +               gzip install ln mkdir mv patch rm rsync sed sort tar >> touch which \ >> +               xargs xz >>   do >>       if ! check_prog ${_myprog} >>       then >> @@ -490,6 +491,18 @@ do >>   done >>   unset restrict >>   +if [ "${SOURCE_DATE_EPOCH+y}" = "y" ] >> +then >> +    if [ -n "$(echo "${SOURCE_DATE_EPOCH}" | sed -e 's/^$/X/' -e >> 's/[0-9]//g')" ] >> +    then >> +        error "Malformed SOURCE_DATE_EPOCH: '${SOURCE_DATE_EPOCH}'" > > This error message should perhaps say what a well-formed > SOURCE_DATE_EPOCH looks like: an integer number of seconds since the > unix epoch? > error "SOURCE_DATE_EPOCH must be an integer number (seconds since the unix epoch)" ? >> +    fi >> +    case $(peflags --version 2>/dev/null | sed -n '1s/^.* //p') in >> +        4.6.[6-9]|4.[7-9]*|[5-9]*) ;; >> +        *) error "SOURCE_DATE_EPOCH requires peflags 4.6.6 or later" >> +    esac >> +fi >> + >> ################################################################################ >>   # >> diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart >> index 2a2bb663..4e6a7cd2 100644 >> --- a/lib/pkg_pkg.cygpart >> +++ b/lib/pkg_pkg.cygpart >> @@ -42,7 +42,7 @@ TAR_COMPRESSION_EXT="${TAR_COMPRESSION_EXT:-xz}" >>   #**** >>     __tar() { >> -    local TAR_COMPRESSION_OPT; >> +    local TAR_COMPRESSION_OPT TAR_SOURCE_DATE_OPTS; >>         # We could use --auto-compress, but this also constrains the >> extension >>       # to the currently valid set. We could probe if tar supports the >> @@ -65,7 +65,14 @@ __tar() { >>               error "tar option for >> TAR_COMPRESSION_EXT='${TAR_COMPRESSION_EXT}' unknown" >>               ;; >>       esac >> -    tar ${TAR_COMPRESSION_OPT} --owner=Guest:501 --group=None:513 >> -cvf "$@" >> + >> +    if [ -n "${SOURCE_DATE_EPOCH}" ] >> +    then >> +        # Ensure reproducible sort order and last modification times >> <= SOURCE_DATE_EPOCH >> +        TAR_SOURCE_DATE_OPTS="--sort=name >> --mtime=@${SOURCE_DATE_EPOCH} --clamp-mtime" > > I'm slightly concerned that maybe this is masking problems elsewhere, > at least when making the source archive: In 2eb7c0eb I started making > an effort so that if the "source inputs" have fixed, upstream > timestamps, we'll preserve those in the output the source package. > > (Obviously that's not always going to be the case, e.g. where cygport > patches are from a local git checkout, so maybe that's not a real > problem...) If the following condition holds, the timestamps of "source inputs" are not affected: newest_source_timestamp < SOURCE_DATE_EPOCH < build_time > >> +    fi >> + >> +    tar ${TAR_COMPRESSION_OPT} ${TAR_SOURCE_DATE_OPTS} >> --owner=Guest:501 --group=None:513 -cvf "$@" >>   } >>     __pkg_binpkg() { >> @@ -319,6 +326,7 @@ __pkg_diff() { >>       local difflevel; >>       local exclude; >>       local optional_patchfiles; >> +    local source_date; >>         default_excludes="CYGWIN-PATCHES aclocal.m4~ aclocal.m4t >> autom4te.cache >>           config.cache config.guess config.log config.status config.sub >> @@ -498,6 +506,14 @@ __pkg_diff() { >>         sed -b -e '/^diff -u/d' -i ${optional_patchfiles} >> ${patchdir}/${src_patchfile}; >>   +    if [ -n "${SOURCE_DATE_EPOCH}" ] >> +    then >> +        # Ensure that the timestamp comment of the new file is >> reproducible > > "timestamp comments in the generated patch files" or suchlike? OK. > > >> +        source_date=$(date -d @"${SOURCE_DATE_EPOCH}" -u +'%Y-%m-%d >> %H:%M:%S.000000000 +0000') >> +        sed -b -e  "s|^\(+++ [^\t]*\t\).*\$|\1${source_date}|" \ >> +            -i ${optional_patchfiles} ${patchdir}/${src_patchfile} >> +    fi >> + >>       diffstat -p${difflevel} ${optional_patchfiles} >> ${patchdir}/${src_patchfile}; >>   } >>   diff --git a/lib/src_postinst.cygpart b/lib/src_postinst.cygpart >> index dd947311..53eaa71a 100644 >> --- a/lib/src_postinst.cygpart >> +++ b/lib/src_postinst.cygpart >> @@ -41,7 +41,7 @@ __prep_fonts_dir() { >>       for catalogue in ${D}${cataloguedir}/* >>       do >>           fontdir=$(readlink -f ${catalogue}) >> -        find ${D}${fontdir} -name '*.pcf' -exec gzip -q '{}' + >> +        find ${D}${fontdir} -name '*.pcf' -exec gzip -q >> ${SOURCE_DATE_EPOCH:+-n} '{}' + >>           mkfontscale ${D}${fontdir} >>           mkfontdir ${D}${fontdir} >>       done >> @@ -775,7 +775,7 @@ __prepman() { >>           while read -d $'\0' manpage >>           do >>               echo "        ${manpage##*/}"; >> -            gzip -q "${manpage}"; >> +            gzip -q ${SOURCE_DATE_EPOCH:+-n} "${manpage}"; >>           done >>       fi >>   } >> @@ -819,7 +819,7 @@ __prepinfo() { >>           while read -d $'\0' infopage >>           do >>               echo "        ${infopage##*/}"; >> -            gzip -q "${infopage}"; >> +            gzip -q ${SOURCE_DATE_EPOCH:+-n} "${infopage}"; >>           done >>       fi >>   } >> @@ -989,6 +989,12 @@ __prepstrip_one() { >>         objdump=${objcopy/copy/dump} >>   +    if [ -n "${SOURCE_DATE_EPOCH}" ] >> +    then >> +        # Let objcopy preserve the timestamps > > I found this comment confusing. > > "Do not embed timestamps in archives" OK. > > Why to we need "--preserve-dates"? The input file presumably has a > timestamp when the build actually took place, which we're going to > clamp to SOURCE_DATE_EPOCH when we make the archive? Option --preserve-dates also preserves the PE header timestamps. This is missing in the objcopy documentation, The timestamp of an original exe is set to SOURCE_DATE_EPOCH by peflags and then preserved by objcopy. > > > It seems like maybe "--enable-deterministic-archives" should be in > LDFLAGS, if the build produces static or unstripped archives? $ ld --enable-deterministic-archives foo.o ld: unrecognized option '--enable-deterministic-archives' Ld supports --no-insert-timestamp which is a different approach which would work without peflags. > >> +        objcopy+=" --enable-deterministic-archives --preserve-dates" >> +    fi >> + >>       # Static libraries should not be fully stripped, but we can >>       # still provide split debuginfo if desired >>       case "${exe}" in >> @@ -1074,6 +1080,16 @@ __prepstrip_one() { >>       # keep sticky bit if present >>       chmod u+w,a+x "${exe}"; >>   +    if [ -n "${SOURCE_DATE_EPOCH}" ] >> +    then >> +        case "${exe}" in >> +        *.exe|*.dll|*.so|*.so.*|*.oct|*.mex|*.cmxs) >> +            # Ensure PE header timestamp is reproducible and >> checksum is correct >> +            # objcopy later inherits the timestamp to debug info and >> stripped file >> +            peflags --checksum=1 --timestamp=${SOURCE_DATE_EPOCH} >> ${exe} >/dev/null ;; >> +        esac >> +    fi >> + >>       if defined_CYGPORT_RESTRICT_debuginfo_ >>       then >>           ${objcopy} --strip-all "${exe}"; > >