binutils snapshot builds

binutils snapshot builds

[Mark Wielaard]

Hi,

The following two patches implement binutils snapshots for
https://snapshots.sourceware.org/

[PATCH 1/2] Add binutils-snapshots builder
[PATCH 2/2] binutils snapshot steps should run in the git step

Sorry about the second patch, I pushed the first before realizing I
got the directory wrong.

This really is just two commands:

sed -i "s/m4_define(\[BFD_VERSION\], \[\([0-9\\.]\+\)\])/m4_define(\[BFD_VERSION\], [\1-$(git rev-parse --short @)])/" bfd/version.m4

To make the version unique.

./src-release.sh -g binutils

To create a binutils tar.gz release tarball.

This is then put up on:

https://snapshots.sourceware.org/binutils/trunk/

It gets regenerated every 15 minutes, if there have been any changes
since the last build. The latest can always be found here:

https://snapshots.sourceware.org/binutils/trunk/latest/src/

Cheers,

Mark

[PATCH 1/2] Add binutils-snapshots builder

[Mark Wielaard]

---
 builder/master.cfg | 56 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/builder/master.cfg b/builder/master.cfg
index e462f8426ca0..e5c1796c6996 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -954,6 +954,19 @@ binutils_try_scheduler = schedulers.AnyBranchScheduler(
                       "binutils-try-ubuntu-riscv"])
 c['schedulers'].append(binutils_try_scheduler)
 
+binutils_snapshot_scheduler = schedulers.Periodic(
+        name="binutils-snapshots",
+        change_filter=util.ChangeFilter(project="binutils-gdb",
+                                        branch="master"),
+        branch="master",
+        periodicBuildTimer=15*60, # 15 minutes in seconds
+        onlyIfChanged=True,
+        fileIsImportant=binutilsImportant,
+        onlyImportant=True,
+        reason="binutils periodic project master branch snapshot",
+        builderNames=["binutils-snapshots"])
+c['schedulers'].append(binutils_snapshot_scheduler)
+
 # Only trigger scheduler for changes to gdb (or deps)
 gdb_files = ["bfd/",
              "config/",
@@ -2865,6 +2878,20 @@ binutils_make_clean_step = steps.ShellCommand(
         command=["make", "clean"],
         name="make clean")
 
+binutils_step_setversion = steps.ShellCommand(
+        name="set bfd version",
+        command='sed -i "s/m4_define(\\[BFD_VERSION\\], \\[\\([0-9\\\\.]\\+\\)\\])/m4_define(\\[BFD_VERSION\\], [\\1-$(git rev-parse --short @)])/" bfd/version.m4')
+binutils_step_src_release = steps.ShellCommand(
+        name="src-release",
+        command="./src-release.sh -g binutils")
+binutils_create_output_step = steps.ShellCommand(
+        name="create output",
+        command="mkdir -p /home/builder/shared/output/src &&"
+        + "mv ./binutils-*tar.gz /home/builder/shared/output/src")
+binutils_create_publish_file_step = steps.ShellCommand(
+        name="create publish file",
+        command="echo binutils/trunk > /home/builder/shared/publish")
+
 binutils_factory = util.BuildFactory()
 binutils_factory.addStep(binutils_step_git)
 binutils_factory.addStep(binutils_step_rm_build_dir)
@@ -2899,6 +2926,14 @@ binutils_gdb_autoregen_factory.addStep(binutils_gdb_step_git)
 binutils_gdb_autoregen_factory.addStep(autoregen_step)
 binutils_gdb_autoregen_factory.addStep(git_diff_step)
 
+binutils_factory_snapshot = util.BuildFactory()
+binutils_factory_snapshot.addStep(binutils_step_git)
+binutils_factory_snapshot.addStep(binutils_step_setversion)
+binutils_factory_snapshot.addStep(binutils_step_src_release)
+binutils_factory_snapshot.addStep(wait_snapshots_output_ready_step)
+binutils_factory_snapshot.addStep(binutils_create_output_step)
+binutils_factory_snapshot.addStep(binutils_create_publish_file_step)
+
 binutils_debian_amd64_builder = util.BuilderConfig(
 	name="binutils-debian-amd64",
         properties={'container-file':
@@ -3093,6 +3128,16 @@ binutils_gdb_autoregen_builder = util.BuilderConfig(
         factory=binutils_gdb_autoregen_factory)
 c['builders'].append(binutils_gdb_autoregen_builder)
 
+binutils_snapshots_builder = util.BuilderConfig(
+        name="binutils-snapshots",
+        collapseRequests=True,
+        properties={'container-file':
+                    readContainerFile('debian-stable')},
+        workernames="snapshots",
+        tags=["binutils-snapshots"],
+        factory=binutils_factory_snapshot)
+c['builders'].append(binutils_snapshots_builder)
+
 # gdb build steps, factory and builders
 
 gdb_git_step = steps.Git(
@@ -5403,6 +5448,17 @@ mn_binutils_try = reporters.MailNotifier(
         generators=[generator_binutils_try])
 c['services'].append(mn_binutils_try)
 
+# Snapshot builder reports
+generator_binutils_snapshot_problem = reporters.BuildStatusGenerator(
+        tags=["binutils-snapshots"],
+        mode=('problem',))
+mn_binutils_snapshot_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=True,
+        extraRecipients=['binutils@sourceware.org'],
+        generators=[generator_binutils_snapshot_problem])
+c['services'].append(mn_binutils_snapshot_change)
+
 # Problem report for the whole gdb tagged builder set
 generator_gdb = reporters.BuildSetStatusGenerator(
         mode=('problem',), tags=['gdb'])
-- 
2.45.1

[PATCH 2/2] binutils snapshot steps should run in the git step workdir

[Mark Wielaard]

---
 builder/master.cfg | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/builder/master.cfg b/builder/master.cfg
index e5c1796c6996..09f359ca736f 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -2879,12 +2879,15 @@ binutils_make_clean_step = steps.ShellCommand(
         name="make clean")
 
 binutils_step_setversion = steps.ShellCommand(
+        workdir='binutils-gdb',
         name="set bfd version",
         command='sed -i "s/m4_define(\\[BFD_VERSION\\], \\[\\([0-9\\\\.]\\+\\)\\])/m4_define(\\[BFD_VERSION\\], [\\1-$(git rev-parse --short @)])/" bfd/version.m4')
 binutils_step_src_release = steps.ShellCommand(
+        workdir='binutils-gdb',
         name="src-release",
         command="./src-release.sh -g binutils")
 binutils_create_output_step = steps.ShellCommand(
+        workdir='binutils-gdb',
         name="create output",
         command="mkdir -p /home/builder/shared/output/src &&"
         + "mv ./binutils-*tar.gz /home/builder/shared/output/src")
-- 
2.45.1

Re: binutils snapshot builds

[Frank Scheiner]

Hi Mark,

On 23.05.24 00:08, Mark Wielaard wrote:
> Hi,
>
> The following two patches implement binutils snapshots for
> https://snapshots.sourceware.org/
> [...]
> To create a binutils tar.gz release tarball.
>
> This is then put up on:
>
> https://snapshots.sourceware.org/binutils/trunk/
>
> It gets regenerated every 15 minutes, if there have been any changes
> since the last build. The latest can always be found here:
>
> https://snapshots.sourceware.org/binutils/trunk/latest/src/

Much obliged, this will be very useful for what I have in mind for ia64.

I looked into it and also the snapshots for other projects like glibc
and I didn't find any hash sums that accompany the tarballs available
for download. I wonder, is there maybe a way to serve the tarballs
**and** their hash sums for consumption by external parties?

I mean, yeah, it's already served via HTTPS, but I still would like to
know if what was downloaded was also downloaded correctly.

Cheers,
Frank

Re: binutils snapshot builds

[Nick Clifton]

Hi Mark,

> The following two patches implement binutils snapshots for
> https://snapshots.sourceware.org/

Thanks for implementing this.

How long do the snapshots stay on the site ?

Cheers
   Nick

Re: binutils snapshot builds

[Andreas Schwab]

On Mai 23 2024, Frank Scheiner wrote:

> I looked into it and also the snapshots for other projects like glibc
> and I didn't find any hash sums that accompany the tarballs available
> for download. I wonder, is there maybe a way to serve the tarballs
> **and** their hash sums for consumption by external parties?

$ xz -cd glibc-2.39.9000-260-g6d3b523eb5.tar.xz | git get-tar-commit-id 
6d3b523eb54198d15af6e042054912cf9f5210de

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: binutils snapshot builds

[Frank Scheiner]

On 23.05.24 13:36, Andreas Schwab wrote:
> On Mai 23 2024, Frank Scheiner wrote:
>
>> I looked into it and also the snapshots for other projects like glibc
>> and I didn't find any hash sums that accompany the tarballs available
>> for download. I wonder, is there maybe a way to serve the tarballs
>> **and** their hash sums for consumption by external parties?
>
> $ xz -cd glibc-2.39.9000-260-g6d3b523eb5.tar.xz | git get-tar-commit-id
> 6d3b523eb54198d15af6e042054912cf9f5210de

Thanks. I actually meant the "checksum" of the tarball itself, not of
the commit. But that is still useful on its own.

Cheers,
Frank

Re: binutils snapshot builds

[Jose E. Marchesi]

Hi Mark.
Thanks for doing this!

I see that the name of the snapshot tarball for a version

  m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])

is

  binutils-2.42.50.tar.gz

What about having:

  binutils-2.42.50-fc12b44ec0e.tar.gz

This would allow for example to easily cache already downloaded
snapshots on the client side.

Also, would it be possible to have a file README (or LATEST or MANIFEST
or whatever) in the directory with a single line containing the name of
the tarball?  This would ease automation.

WDYT?

> The following two patches implement binutils snapshots for
> https://snapshots.sourceware.org/
>
> [PATCH 1/2] Add binutils-snapshots builder
> [PATCH 2/2] binutils snapshot steps should run in the git step
>
> Sorry about the second patch, I pushed the first before realizing I
> got the directory wrong.
>
> This really is just two commands:
>
> sed -i "s/m4_define(\[BFD_VERSION\],
> \[\([0-9\\.]\+\)\])/m4_define(\[BFD_VERSION\], [\1-$(git rev-parse
> --short @)])/" bfd/version.m4
>
> To make the version unique.
>
> ./src-release.sh -g binutils
>
> To create a binutils tar.gz release tarball.
>
> This is then put up on:
>
> https://snapshots.sourceware.org/binutils/trunk/
>
> It gets regenerated every 15 minutes, if there have been any changes
> since the last build. The latest can always be found here:
>
> https://snapshots.sourceware.org/binutils/trunk/latest/src/
>
> Cheers,
>
> Mark

Re: binutils snapshot builds

[Frank Ch. Eigler]

Hi -

>   binutils-2.42.50.tar.gz
> What about having:
>   binutils-2.42.50-fc12b44ec0e.tar.gz

(Or else: git-describe.)

> This would allow for example to easily cache already downloaded
> snapshots on the client side.

By the way .... could someone spell out in what way these source
tarball snapshots are useful, when the git repo is already easily
available?


- FChE

Re: binutils snapshot builds

[Jose E. Marchesi]

> Hi -
>
>>   binutils-2.42.50.tar.gz
>> What about having:
>>   binutils-2.42.50-fc12b44ec0e.tar.gz
>
> (Or else: git-describe.)
>
>> This would allow for example to easily cache already downloaded
>> snapshots on the client side.
>
> By the way .... could someone spell out in what way these source
> tarball snapshots are useful, when the git repo is already easily
> available?

We are in the process of getting a CI set up to test the binutils/GCC
support for the kernel BPF selftests [1].  We are told they need
tarballs.

Having this CI is very important to us due to the fragile nature of
compiled BPF.  Without it any update to the kernel BPF selftests can
potentially break the build with GCC.

[1] https://github.com/kernel-patches/bpf/actions/workflows/test.yml

>
> - FChE

Re: binutils snapshot builds

[Sam James]

"Frank Ch. Eigler" <fche@elastic.org> writes:

> Hi -
>
>>   binutils-2.42.50.tar.gz
>> What about having:
>>   binutils-2.42.50-fc12b44ec0e.tar.gz
>
> (Or else: git-describe.)
>
>> This would allow for example to easily cache already downloaded
>> snapshots on the client side.
>
> By the way .... could someone spell out in what way these source
> tarball snapshots are useful, when the git repo is already easily
> available?

We can't easily/cheaply mirror git repos for the purposes of consuming
within e.g. packaging. It also requires plumbing for outsourcing
verification to an external tool as opposed to fetch-somehow-and-compare-checksum.

>
>
> - FChE

Re: binutils snapshot builds

[Mark Wielaard]

Hi Frank,

On Thu, May 23, 2024 at 10:57:41AM +0200, Frank Scheiner wrote:
> On 23.05.24 00:08, Mark Wielaard wrote:
> >It gets regenerated every 15 minutes, if there have been any changes
> >since the last build. The latest can always be found here:
> >
> >https://snapshots.sourceware.org/binutils/trunk/latest/src/
> 
> Much obliged, this will be very useful for what I have in mind for ia64.
> 
> I looked into it and also the snapshots for other projects like glibc
> and I didn't find any hash sums that accompany the tarballs available
> for download. I wonder, is there maybe a way to serve the tarballs
> **and** their hash sums for consumption by external parties?
> 
> I mean, yeah, it's already served via HTTPS, but I still would like to
> know if what was downloaded was also downloaded correctly.

Do note that these snapshots are generated totally automatically, it
could right after a bad/accidental commit. Nobody double checks any of
the snapshots (except that they could be generated). So please don't
trust them even if some checksum (which would also be generated
automatically) matches.

That said, we could generate sha512.sum files in each directory.
Like we do on https://sourceware.org/pub/binutils/releases/

Would that be useful?

I think it is just noise though. It might make the snapshots look more
trustworthy than they really are.

Cheers,

Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Nick,

On Thu, May 23, 2024 at 11:35:47AM +0100, Nick Clifton wrote:
> >The following two patches implement binutils snapshots for
> >https://snapshots.sourceware.org/
> 
> Thanks for implementing this.
> 
> How long do the snapshots stay on the site ?

Currently indefinitely. The server (thanks OSUOSL!) can hold 600G of
snapshots. Currently (after a year) it generated/collected 60G of
snapshots. So we should be good for another 9 years...

Of course we all know these things grow more rapidly than you
expect. So pessimistically we probably have to make some retention
decission in about a year :)

What do people think is a good policy?

- Just keep one snapshot a day after a year.
- Only keep snapshots for 1, 2 or 3 years.
- Just throw money at it, disks are cheap.
  Just buy another 600G when it fills up?

Currently we don't actually pay anything thanks to OSUOSL. But there
is a Sourceware hardware replacement fund that allows us to pay for
extra disks if necessary. https://sourceware.org/donate.html

Cheers,

Mark

Re: binutils snapshot builds

[Sam James]

Mark Wielaard <mark@klomp.org> writes:

> Hi Frank,
>
> On Thu, May 23, 2024 at 10:57:41AM +0200, Frank Scheiner wrote:
>> On 23.05.24 00:08, Mark Wielaard wrote:
>> >It gets regenerated every 15 minutes, if there have been any changes
>> >since the last build. The latest can always be found here:
>> >
>> >https://snapshots.sourceware.org/binutils/trunk/latest/src/
>> 
>> Much obliged, this will be very useful for what I have in mind for ia64.
>> 
>> I looked into it and also the snapshots for other projects like glibc
>> and I didn't find any hash sums that accompany the tarballs available
>> for download. I wonder, is there maybe a way to serve the tarballs
>> **and** their hash sums for consumption by external parties?
>> 
>> I mean, yeah, it's already served via HTTPS, but I still would like to
>> know if what was downloaded was also downloaded correctly.
>
> Do note that these snapshots are generated totally automatically, it
> could right after a bad/accidental commit. Nobody double checks any of
> the snapshots (except that they could be generated). So please don't
> trust them even if some checksum (which would also be generated
> automatically) matches.

This is fine as I already check the diff before using it anywhere
non-automated.

But I would appreciate if you could consider signing them with an
automated key, at least for the GCC snapshots (which are stored
elsewhere), because we currently grab them from mirrors to avoid
overloading sourceware. But we have no way of verifying mirrors didn't
tamper...

>
> That said, we could generate sha512.sum files in each directory.
> Like we do on https://sourceware.org/pub/binutils/releases/
>
> Would that be useful?
>
> I think it is just noise though. It might make the snapshots look more
> trustworthy than they really are.
>
> Cheers,
>
> Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Jose,

On Thu, May 23, 2024 at 05:11:40PM +0200, Jose E. Marchesi wrote:
> I see that the name of the snapshot tarball for a version
> 
>   m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])
> 
> is
> 
>   binutils-2.42.50.tar.gz
> 
> What about having:
> 
>   binutils-2.42.50-fc12b44ec0e.tar.gz
> 
> This would allow for example to easily cache already downloaded
> snapshots on the client side.

Yeah, that is what we do for glibc. I assume it should then also
unpack in a binutils-2.42.50-fc12b44ec0e directory?

> Also, would it be possible to have a file README (or LATEST or MANIFEST
> or whatever) in the directory with a single line containing the name of
> the tarball?  This would ease automation.

Maybe that is a valid usecase for the md512.sum file?
Or does it have to be just the filename or a symlink to it.

Do you already have some automation for something similar?

Thanks,

Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Frank,

On Thu, May 23, 2024 at 11:28:38AM -0400, Frank Ch. Eigler wrote:
> >   binutils-2.42.50.tar.gz
> > What about having:
> >   binutils-2.42.50-fc12b44ec0e.tar.gz
> 
> (Or else: git-describe.)

git describe produces some funky results because binutils-gdb is a
shared repo for multiple projects creating their own tags. So the
above currently produces gdb-14-branchpoint-2499-gfc12b44ec0e0

> > This would allow for example to easily cache already downloaded
> > snapshots on the client side.
> 
> By the way .... could someone spell out in what way these source
> tarball snapshots are useful, when the git repo is already easily
> available?

The snapshots mimic actual releases, which are non-trivial to
produce. Again some of this is historical, since it is a shared repo
between projects. You can of course run src-release.sh from git
yourself, but then you do need some extra tools installed.

Cheers,

Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Sam,

On Fri, May 24, 2024 at 12:37:00AM +0100, Sam James wrote:
> > Do note that these snapshots are generated totally automatically, it
> > could right after a bad/accidental commit. Nobody double checks any of
> > the snapshots (except that they could be generated). So please don't
> > trust them even if some checksum (which would also be generated
> > automatically) matches.
> 
> This is fine as I already check the diff before using it anywhere
> non-automated.
> 
> But I would appreciate if you could consider signing them with an
> automated key, at least for the GCC snapshots (which are stored
> elsewhere), because we currently grab them from mirrors to avoid
> overloading sourceware. But we have no way of verifying mirrors didn't
> tamper...

hmmmm, but that would mean keeping a private signing key somewhere
that the automated process can use (but nobody else can access...)

If we provide checksums, can you just fetch those from the primary
server and check them against the file you fetched from a mirror?

I think you should treat these snapshot as if someone could have
tempered with them (maybe not deliberately, but they really are
snapshots and not formal releases for a reason).

Cheers,

Mark

Re: binutils snapshot builds

[Sam James]

Mark Wielaard <mark@klomp.org> writes:

> Hi Sam,
>
> On Fri, May 24, 2024 at 12:37:00AM +0100, Sam James wrote:
>> > Do note that these snapshots are generated totally automatically, it
>> > could right after a bad/accidental commit. Nobody double checks any of
>> > the snapshots (except that they could be generated). So please don't
>> > trust them even if some checksum (which would also be generated
>> > automatically) matches.
>> 
>> This is fine as I already check the diff before using it anywhere
>> non-automated.
>> 
>> But I would appreciate if you could consider signing them with an
>> automated key, at least for the GCC snapshots (which are stored
>> elsewhere), because we currently grab them from mirrors to avoid
>> overloading sourceware. But we have no way of verifying mirrors didn't
>> tamper...
>
> hmmmm, but that would mean keeping a private signing key somewhere
> that the automated process can use (but nobody else can access...)
>
> If we provide checksums, can you just fetch those from the primary
> server and check them against the file you fetched from a mirror?

Yeah, we can do this, I think - it'll be a bit icky as we don't have
machinery to easily check just checksums like this from diff. places
but I can try figure something out.

But to be clear, the threat model I have in mind here is purely "evil
mirror" rather than anything else.

>
> I think you should treat these snapshot as if someone could have
> tempered with them (maybe not deliberately, but they really are
> snapshots and not formal releases for a reason).

That's part of why I'm asking ;)

I do check the diffs to make sure they look reasonable but I'd like to
avoid having to roll my own every week. It'd deter me from doing testing
to the same extent I do now if I had to ship a bunch of tarballs myself.

>
> Cheers,
>
> Mark

thanks,
sam

Re: binutils snapshot builds

[Jose E. Marchesi]

> Hi Jose,
>
> On Thu, May 23, 2024 at 05:11:40PM +0200, Jose E. Marchesi wrote:
>> I see that the name of the snapshot tarball for a version
>> 
>>   m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])
>> 
>> is
>> 
>>   binutils-2.42.50.tar.gz
>> 
>> What about having:
>> 
>>   binutils-2.42.50-fc12b44ec0e.tar.gz
>> 
>> This would allow for example to easily cache already downloaded
>> snapshots on the client side.
>
> Yeah, that is what we do for glibc. I assume it should then also
> unpack in a binutils-2.42.50-fc12b44ec0e directory?

Yep.

>> Also, would it be possible to have a file README (or LATEST or MANIFEST
>> or whatever) in the directory with a single line containing the name of
>> the tarball?  This would ease automation.
>
> Maybe that is a valid usecase for the md512.sum file?
> Or does it have to be just the filename or a symlink to it.
>
> Do you already have some automation for something similar?

Any file from which it is easy to extract the name of the latest
snapshot would do.

For GCC snapshots I have this:

GCC_TARBALL=`wget https://gcc.gnu.org/pub/gcc/snapshots/LATEST-15 -O - -o /dev/null | grep -E 'gcc-15-[0-9]+.tar.xz' | sed -e 's/.*\(gcc-15-[^<]*\).*/\1/'`

Re: binutils snapshot builds

[Nick Clifton]

Hi Mark,

> What do people think is a good policy?
> 
> - Just keep one snapshot a day after a year.
> - Only keep snapshots for 1, 2 or 3 years.
> - Just throw money at it, disks are cheap.
>    Just buy another 600G when it fills up?

Given that any of these snapshots can be reproduced from the git repository
I do not see any great benefit in maintaining them long term.  My feeling
is that is they will loose their usefulness once an official release is
made, or possibly after two releases.  So how about:

   * Snapshots are retained for at least 1 year after their creation.

   * After that they can be culled at the discretion of the overseers
     (and without needing to consult with the binutils project) as and
     when they wish.

Cheers
   Nick

Re: binutils snapshot builds

[Jose E. Marchesi]

> Hi Mark,
>
>> What do people think is a good policy?
>> - Just keep one snapshot a day after a year.
>> - Only keep snapshots for 1, 2 or 3 years.
>> - Just throw money at it, disks are cheap.
>>    Just buy another 600G when it fills up?
>
> Given that any of these snapshots can be reproduced from the git repository
> I do not see any great benefit in maintaining them long term.  My feeling
> is that is they will loose their usefulness once an official release is
> made, or possibly after two releases.  So how about:
>
>   * Snapshots are retained for at least 1 year after their creation.
>
>   * After that they can be culled at the discretion of the overseers
>     (and without needing to consult with the binutils project) as and
>     when they wish.

I would add, probably doing the snapshots in a weekly basis would be
enough, like GCC does?

Re: binutils snapshot builds

[Frank Scheiner]

Hi Mark,

On 24.05.24 01:15, Mark Wielaard wrote:
> [...]
> Do note that these snapshots are generated totally automatically, it
> could right after a bad/accidental commit. Nobody double checks any of
> the snapshots (except that they could be generated). So please don't
> trust them even if some checksum (which would also be generated
> automatically) matches.
>
> That said, we could generate sha512.sum files in each directory.
> Like we do on https://sourceware.org/pub/binutils/releases/
>
> Would that be useful?

Sure. I'd like to be able to verify in some way that what I get is what
you publish, especially if I get it from a mirror.

> I think it is just noise though. It might make the snapshots look more
> trustworthy than they really are.

Just for clarification: With "trustworthy" you mean that people could
expect that the snapshots build and work correctly, right?

Well, I'd say that is for me or the build system to find out. The goal
for these builds will be to find out early on when things break for ia64.

Cheers,
Frank

Re: binutils snapshot builds

[Mark Wielaard]

Hi Jose,

On Fri, May 24, 2024 at 11:58:56AM +0200, Jose E. Marchesi wrote:
> > On Thu, May 23, 2024 at 05:11:40PM +0200, Jose E. Marchesi wrote:
> >> I see that the name of the snapshot tarball for a version
> >> 
> >>   m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])
> >> 
> >> is
> >> 
> >>   binutils-2.42.50.tar.gz
> >> 
> >> What about having:
> >> 
> >>   binutils-2.42.50-fc12b44ec0e.tar.gz
> >> 
> >> This would allow for example to easily cache already downloaded
> >> snapshots on the client side.
> >
> > Yeah, that is what we do for glibc. I assume it should then also
> > unpack in a binutils-2.42.50-fc12b44ec0e directory?
> 
> Yep.

OK, that was a bit of work and needed a little hack.  ./src-release
uses the output of bfd/configure --version to create the directory/tar
name.
https://sourceware.org/cgit/builder/commit/?id=9a85f672549a68c7114cef580c13cbfd7caf65f9

> >> Also, would it be possible to have a file README (or LATEST or MANIFEST
> >> or whatever) in the directory with a single line containing the name of
> >> the tarball?  This would ease automation.
> >
> > Maybe that is a valid usecase for the md512.sum file?
> > Or does it have to be just the filename or a symlink to it.
> >
> > Do you already have some automation for something similar?
> 
> Any file from which it is easy to extract the name of the latest
> snapshot would do.
> 
> For GCC snapshots I have this:
> 
> GCC_TARBALL=`wget https://gcc.gnu.org/pub/gcc/snapshots/LATEST-15 -O - -o /dev/null | grep -E 'gcc-15-[0-9]+.tar.xz' | sed -e 's/.*\(gcc-15-[^<]*\).*/\1/'`

OK, all new dirs have a sha512.sum file now. You should be able to
grep/sed the file name out of that one:

 $ BINUTILS_TARBALL=`wget https://snapshots.sourceware.org/binutils/trunk/latest/src/sha512.sum -O - -o /dev/null | grep -E 'binutils-[0-9a-f.-]+.tar.xz' | sed -e 's/.*\(binutils-[^<]*\).*/\1/'`
 $ echo $BINUTILS_TARBALL
binutils-2.42.50-0daa17bf187.tar.xz

Note that I changed the binutils, gdb and glibc snapshots to produce
tar.xz files. Those are almost half of the tar.gz sized.

xz does use a bit more process/memory. It might be nice to use zstd
instead. If zstd is common enough of course. What do people think?

Cheers,

Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Frank,

On Sat, May 25, 2024 at 07:40:18PM +0200, Frank Scheiner wrote:
> On 24.05.24 01:15, Mark Wielaard wrote:
> >[...]
> >Do note that these snapshots are generated totally automatically, it
> >could right after a bad/accidental commit. Nobody double checks any of
> >the snapshots (except that they could be generated). So please don't
> >trust them even if some checksum (which would also be generated
> >automatically) matches.
> >
> >That said, we could generate sha512.sum files in each directory.
> >Like we do on https://sourceware.org/pub/binutils/releases/
> >
> >Would that be useful?
> 
> Sure. I'd like to be able to verify in some way that what I get is what
> you publish, especially if I get it from a mirror.

OK. There are sha512.sum files in each dir now (for new snapshots).

> >I think it is just noise though. It might make the snapshots look more
> >trustworthy than they really are.
> 
> Just for clarification: With "trustworthy" you mean that people could
> expect that the snapshots build and work correctly, right?
> 
> Well, I'd say that is for me or the build system to find out. The goal
> for these builds will be to find out early on when things break for ia64.

Right. I just want to make clear that the snapshot hasn't seen any
checking. Someone could have pushed a really bad commit right before
the snapshot was made. It could be boobytrapped.

Cheers,

Mark

Re: binutils snapshot builds

[Frank Scheiner]

Hi Mark,

On 26.05.24 02:31, Mark Wielaard wrote:
> On Sat, May 25, 2024 at 07:40:18PM +0200, Frank Scheiner wrote:
>> On 24.05.24 01:15, Mark Wielaard wrote:
>>> [...]
>>> Do note that these snapshots are generated totally automatically, it
>>> could right after a bad/accidental commit. Nobody double checks any of
>>> the snapshots (except that they could be generated). So please don't
>>> trust them even if some checksum (which would also be generated
>>> automatically) matches.
>>>
>>> That said, we could generate sha512.sum files in each directory.
>>> Like we do on https://sourceware.org/pub/binutils/releases/
>>>
>>> Would that be useful?
>>
>> Sure. I'd like to be able to verify in some way that what I get is what
>> you publish, especially if I get it from a mirror.
>
> OK. There are sha512.sum files in each dir now (for new snapshots).

Thanks a lot for that.

>>> I think it is just noise though. It might make the snapshots look more
>>> trustworthy than they really are.
>>
>> Just for clarification: With "trustworthy" you mean that people could
>> expect that the snapshots build and work correctly, right?
>>
>> Well, I'd say that is for me or the build system to find out. The goal
>> for these builds will be to find out early on when things break for ia64.
>
> Right. I just want to make clear that the snapshot hasn't seen any
> checking. Someone could have pushed a really bad commit right before
> the snapshot was made. It could be boobytrapped.

It'll turn out ok. (-;

Cheers,
Frank

Re: binutils snapshot builds

[Nick Clifton]

Hi Mark,

> Note that I changed the binutils, gdb and glibc snapshots to produce
> tar.xz files. Those are almost half of the tar.gz sized.
> 
> xz does use a bit more process/memory. It might be nice to use zstd
> instead. If zstd is common enough of course. What do people think?

Does tar automatically understand it ?  ie can I run:

   tar xvf <foo>.zst

and have it work ?  If so, then I would definitely think that
adding zstd support would be a good idea.

Cheers
   Nick

Re: binutils snapshot builds

[Mark Wielaard]

Hi Nick,

On Tue, 2024-05-28 at 15:35 +0100, Nick Clifton wrote:
> > xz does use a bit more process/memory. It might be nice to use zstd
> > instead. If zstd is common enough of course. What do people think?
> 
> Does tar automatically understand it ?  ie can I run:
> 
>    tar xvf <foo>.zst
> 
> and have it work ?

Yes, it does, since GNU tar 1.31 (released 2019-01-02 - 5 years ago).

Cheers,

Mark