binutils snapshot builds

binutils snapshot builds

[Mark Wielaard]

Hi,

The following two patches implement binutils snapshots for
https://snapshots.sourceware.org/

[PATCH 1/2] Add binutils-snapshots builder
[PATCH 2/2] binutils snapshot steps should run in the git step

Sorry about the second patch, I pushed the first before realizing I
got the directory wrong.

This really is just two commands:

sed -i "s/m4_define(\[BFD_VERSION\], \[\([0-9\\.]\+\)\])/m4_define(\[BFD_VERSION\], [\1-$(git rev-parse --short @)])/" bfd/version.m4

To make the version unique.

./src-release.sh -g binutils

To create a binutils tar.gz release tarball.

This is then put up on:

https://snapshots.sourceware.org/binutils/trunk/

It gets regenerated every 15 minutes, if there have been any changes
since the last build. The latest can always be found here:

https://snapshots.sourceware.org/binutils/trunk/latest/src/

Cheers,

Mark

[PATCH 1/2] Add binutils-snapshots builder

[Mark Wielaard]

---
 builder/master.cfg | 56 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/builder/master.cfg b/builder/master.cfg
index e462f8426ca0..e5c1796c6996 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -954,6 +954,19 @@ binutils_try_scheduler = schedulers.AnyBranchScheduler(
                       "binutils-try-ubuntu-riscv"])
 c['schedulers'].append(binutils_try_scheduler)
 
+binutils_snapshot_scheduler = schedulers.Periodic(
+        name="binutils-snapshots",
+        change_filter=util.ChangeFilter(project="binutils-gdb",
+                                        branch="master"),
+        branch="master",
+        periodicBuildTimer=15*60, # 15 minutes in seconds
+        onlyIfChanged=True,
+        fileIsImportant=binutilsImportant,
+        onlyImportant=True,
+        reason="binutils periodic project master branch snapshot",
+        builderNames=["binutils-snapshots"])
+c['schedulers'].append(binutils_snapshot_scheduler)
+
 # Only trigger scheduler for changes to gdb (or deps)
 gdb_files = ["bfd/",
              "config/",
@@ -2865,6 +2878,20 @@ binutils_make_clean_step = steps.ShellCommand(
         command=["make", "clean"],
         name="make clean")
 
+binutils_step_setversion = steps.ShellCommand(
+        name="set bfd version",
+        command='sed -i "s/m4_define(\\[BFD_VERSION\\], \\[\\([0-9\\\\.]\\+\\)\\])/m4_define(\\[BFD_VERSION\\], [\\1-$(git rev-parse --short @)])/" bfd/version.m4')
+binutils_step_src_release = steps.ShellCommand(
+        name="src-release",
+        command="./src-release.sh -g binutils")
+binutils_create_output_step = steps.ShellCommand(
+        name="create output",
+        command="mkdir -p /home/builder/shared/output/src &&"
+        + "mv ./binutils-*tar.gz /home/builder/shared/output/src")
+binutils_create_publish_file_step = steps.ShellCommand(
+        name="create publish file",
+        command="echo binutils/trunk > /home/builder/shared/publish")
+
 binutils_factory = util.BuildFactory()
 binutils_factory.addStep(binutils_step_git)
 binutils_factory.addStep(binutils_step_rm_build_dir)
@@ -2899,6 +2926,14 @@ binutils_gdb_autoregen_factory.addStep(binutils_gdb_step_git)
 binutils_gdb_autoregen_factory.addStep(autoregen_step)
 binutils_gdb_autoregen_factory.addStep(git_diff_step)
 
+binutils_factory_snapshot = util.BuildFactory()
+binutils_factory_snapshot.addStep(binutils_step_git)
+binutils_factory_snapshot.addStep(binutils_step_setversion)
+binutils_factory_snapshot.addStep(binutils_step_src_release)
+binutils_factory_snapshot.addStep(wait_snapshots_output_ready_step)
+binutils_factory_snapshot.addStep(binutils_create_output_step)
+binutils_factory_snapshot.addStep(binutils_create_publish_file_step)
+
 binutils_debian_amd64_builder = util.BuilderConfig(
 	name="binutils-debian-amd64",
         properties={'container-file':
@@ -3093,6 +3128,16 @@ binutils_gdb_autoregen_builder = util.BuilderConfig(
         factory=binutils_gdb_autoregen_factory)
 c['builders'].append(binutils_gdb_autoregen_builder)
 
+binutils_snapshots_builder = util.BuilderConfig(
+        name="binutils-snapshots",
+        collapseRequests=True,
+        properties={'container-file':
+                    readContainerFile('debian-stable')},
+        workernames="snapshots",
+        tags=["binutils-snapshots"],
+        factory=binutils_factory_snapshot)
+c['builders'].append(binutils_snapshots_builder)
+
 # gdb build steps, factory and builders
 
 gdb_git_step = steps.Git(
@@ -5403,6 +5448,17 @@ mn_binutils_try = reporters.MailNotifier(
         generators=[generator_binutils_try])
 c['services'].append(mn_binutils_try)
 
+# Snapshot builder reports
+generator_binutils_snapshot_problem = reporters.BuildStatusGenerator(
+        tags=["binutils-snapshots"],
+        mode=('problem',))
+mn_binutils_snapshot_change = reporters.MailNotifier(
+        fromaddr="builder@sourceware.org",
+        sendToInterestedUsers=True,
+        extraRecipients=['binutils@sourceware.org'],
+        generators=[generator_binutils_snapshot_problem])
+c['services'].append(mn_binutils_snapshot_change)
+
 # Problem report for the whole gdb tagged builder set
 generator_gdb = reporters.BuildSetStatusGenerator(
         mode=('problem',), tags=['gdb'])
-- 
2.45.1

[PATCH 2/2] binutils snapshot steps should run in the git step workdir

[Mark Wielaard]

---
 builder/master.cfg | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/builder/master.cfg b/builder/master.cfg
index e5c1796c6996..09f359ca736f 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -2879,12 +2879,15 @@ binutils_make_clean_step = steps.ShellCommand(
         name="make clean")
 
 binutils_step_setversion = steps.ShellCommand(
+        workdir='binutils-gdb',
         name="set bfd version",
         command='sed -i "s/m4_define(\\[BFD_VERSION\\], \\[\\([0-9\\\\.]\\+\\)\\])/m4_define(\\[BFD_VERSION\\], [\\1-$(git rev-parse --short @)])/" bfd/version.m4')
 binutils_step_src_release = steps.ShellCommand(
+        workdir='binutils-gdb',
         name="src-release",
         command="./src-release.sh -g binutils")
 binutils_create_output_step = steps.ShellCommand(
+        workdir='binutils-gdb',
         name="create output",
         command="mkdir -p /home/builder/shared/output/src &&"
         + "mv ./binutils-*tar.gz /home/builder/shared/output/src")
-- 
2.45.1

Re: binutils snapshot builds

[Nick Clifton]

Hi Mark,

> The following two patches implement binutils snapshots for
> https://snapshots.sourceware.org/

Thanks for implementing this.

How long do the snapshots stay on the site ?

Cheers
   Nick

Re: binutils snapshot builds

[Jose E. Marchesi]

Hi Mark.
Thanks for doing this!

I see that the name of the snapshot tarball for a version

  m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])

is

  binutils-2.42.50.tar.gz

What about having:

  binutils-2.42.50-fc12b44ec0e.tar.gz

This would allow for example to easily cache already downloaded
snapshots on the client side.

Also, would it be possible to have a file README (or LATEST or MANIFEST
or whatever) in the directory with a single line containing the name of
the tarball?  This would ease automation.

WDYT?

> The following two patches implement binutils snapshots for
> https://snapshots.sourceware.org/
>
> [PATCH 1/2] Add binutils-snapshots builder
> [PATCH 2/2] binutils snapshot steps should run in the git step
>
> Sorry about the second patch, I pushed the first before realizing I
> got the directory wrong.
>
> This really is just two commands:
>
> sed -i "s/m4_define(\[BFD_VERSION\],
> \[\([0-9\\.]\+\)\])/m4_define(\[BFD_VERSION\], [\1-$(git rev-parse
> --short @)])/" bfd/version.m4
>
> To make the version unique.
>
> ./src-release.sh -g binutils
>
> To create a binutils tar.gz release tarball.
>
> This is then put up on:
>
> https://snapshots.sourceware.org/binutils/trunk/
>
> It gets regenerated every 15 minutes, if there have been any changes
> since the last build. The latest can always be found here:
>
> https://snapshots.sourceware.org/binutils/trunk/latest/src/
>
> Cheers,
>
> Mark

Re: binutils snapshot builds

[Frank Ch. Eigler]

Hi -

>   binutils-2.42.50.tar.gz
> What about having:
>   binutils-2.42.50-fc12b44ec0e.tar.gz

(Or else: git-describe.)

> This would allow for example to easily cache already downloaded
> snapshots on the client side.

By the way .... could someone spell out in what way these source
tarball snapshots are useful, when the git repo is already easily
available?


- FChE

Re: binutils snapshot builds

[Jose E. Marchesi]

> Hi -
>
>>   binutils-2.42.50.tar.gz
>> What about having:
>>   binutils-2.42.50-fc12b44ec0e.tar.gz
>
> (Or else: git-describe.)
>
>> This would allow for example to easily cache already downloaded
>> snapshots on the client side.
>
> By the way .... could someone spell out in what way these source
> tarball snapshots are useful, when the git repo is already easily
> available?

We are in the process of getting a CI set up to test the binutils/GCC
support for the kernel BPF selftests [1].  We are told they need
tarballs.

Having this CI is very important to us due to the fragile nature of
compiled BPF.  Without it any update to the kernel BPF selftests can
potentially break the build with GCC.

[1] https://github.com/kernel-patches/bpf/actions/workflows/test.yml

>
> - FChE

Re: binutils snapshot builds

[Sam James]

"Frank Ch. Eigler" <fche@elastic.org> writes:

> Hi -
>
>>   binutils-2.42.50.tar.gz
>> What about having:
>>   binutils-2.42.50-fc12b44ec0e.tar.gz
>
> (Or else: git-describe.)
>
>> This would allow for example to easily cache already downloaded
>> snapshots on the client side.
>
> By the way .... could someone spell out in what way these source
> tarball snapshots are useful, when the git repo is already easily
> available?

We can't easily/cheaply mirror git repos for the purposes of consuming
within e.g. packaging. It also requires plumbing for outsourcing
verification to an external tool as opposed to fetch-somehow-and-compare-checksum.

>
>
> - FChE

Re: binutils snapshot builds

[Mark Wielaard]

Hi Nick,

On Thu, May 23, 2024 at 11:35:47AM +0100, Nick Clifton wrote:
> >The following two patches implement binutils snapshots for
> >https://snapshots.sourceware.org/
> 
> Thanks for implementing this.
> 
> How long do the snapshots stay on the site ?

Currently indefinitely. The server (thanks OSUOSL!) can hold 600G of
snapshots. Currently (after a year) it generated/collected 60G of
snapshots. So we should be good for another 9 years...

Of course we all know these things grow more rapidly than you
expect. So pessimistically we probably have to make some retention
decission in about a year :)

What do people think is a good policy?

- Just keep one snapshot a day after a year.
- Only keep snapshots for 1, 2 or 3 years.
- Just throw money at it, disks are cheap.
  Just buy another 600G when it fills up?

Currently we don't actually pay anything thanks to OSUOSL. But there
is a Sourceware hardware replacement fund that allows us to pay for
extra disks if necessary. https://sourceware.org/donate.html

Cheers,

Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Jose,

On Thu, May 23, 2024 at 05:11:40PM +0200, Jose E. Marchesi wrote:
> I see that the name of the snapshot tarball for a version
> 
>   m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])
> 
> is
> 
>   binutils-2.42.50.tar.gz
> 
> What about having:
> 
>   binutils-2.42.50-fc12b44ec0e.tar.gz
> 
> This would allow for example to easily cache already downloaded
> snapshots on the client side.

Yeah, that is what we do for glibc. I assume it should then also
unpack in a binutils-2.42.50-fc12b44ec0e directory?

> Also, would it be possible to have a file README (or LATEST or MANIFEST
> or whatever) in the directory with a single line containing the name of
> the tarball?  This would ease automation.

Maybe that is a valid usecase for the md512.sum file?
Or does it have to be just the filename or a symlink to it.

Do you already have some automation for something similar?

Thanks,

Mark

Re: binutils snapshot builds

[Mark Wielaard]

Hi Frank,

On Thu, May 23, 2024 at 11:28:38AM -0400, Frank Ch. Eigler wrote:
> >   binutils-2.42.50.tar.gz
> > What about having:
> >   binutils-2.42.50-fc12b44ec0e.tar.gz
> 
> (Or else: git-describe.)

git describe produces some funky results because binutils-gdb is a
shared repo for multiple projects creating their own tags. So the
above currently produces gdb-14-branchpoint-2499-gfc12b44ec0e0

> > This would allow for example to easily cache already downloaded
> > snapshots on the client side.
> 
> By the way .... could someone spell out in what way these source
> tarball snapshots are useful, when the git repo is already easily
> available?

The snapshots mimic actual releases, which are non-trivial to
produce. Again some of this is historical, since it is a shared repo
between projects. You can of course run src-release.sh from git
yourself, but then you do need some extra tools installed.

Cheers,

Mark

Re: binutils snapshot builds

[Jose E. Marchesi]

> Hi Jose,
>
> On Thu, May 23, 2024 at 05:11:40PM +0200, Jose E. Marchesi wrote:
>> I see that the name of the snapshot tarball for a version
>> 
>>   m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])
>> 
>> is
>> 
>>   binutils-2.42.50.tar.gz
>> 
>> What about having:
>> 
>>   binutils-2.42.50-fc12b44ec0e.tar.gz
>> 
>> This would allow for example to easily cache already downloaded
>> snapshots on the client side.
>
> Yeah, that is what we do for glibc. I assume it should then also
> unpack in a binutils-2.42.50-fc12b44ec0e directory?

Yep.

>> Also, would it be possible to have a file README (or LATEST or MANIFEST
>> or whatever) in the directory with a single line containing the name of
>> the tarball?  This would ease automation.
>
> Maybe that is a valid usecase for the md512.sum file?
> Or does it have to be just the filename or a symlink to it.
>
> Do you already have some automation for something similar?

Any file from which it is easy to extract the name of the latest
snapshot would do.

For GCC snapshots I have this:

GCC_TARBALL=`wget https://gcc.gnu.org/pub/gcc/snapshots/LATEST-15 -O - -o /dev/null | grep -E 'gcc-15-[0-9]+.tar.xz' | sed -e 's/.*\(gcc-15-[^<]*\).*/\1/'`

Re: binutils snapshot builds

[Nick Clifton]

Hi Mark,

> What do people think is a good policy?
> 
> - Just keep one snapshot a day after a year.
> - Only keep snapshots for 1, 2 or 3 years.
> - Just throw money at it, disks are cheap.
>    Just buy another 600G when it fills up?

Given that any of these snapshots can be reproduced from the git repository
I do not see any great benefit in maintaining them long term.  My feeling
is that is they will loose their usefulness once an official release is
made, or possibly after two releases.  So how about:

   * Snapshots are retained for at least 1 year after their creation.

   * After that they can be culled at the discretion of the overseers
     (and without needing to consult with the binutils project) as and
     when they wish.

Cheers
   Nick

Re: binutils snapshot builds

[Jose E. Marchesi]

> Hi Mark,
>
>> What do people think is a good policy?
>> - Just keep one snapshot a day after a year.
>> - Only keep snapshots for 1, 2 or 3 years.
>> - Just throw money at it, disks are cheap.
>>    Just buy another 600G when it fills up?
>
> Given that any of these snapshots can be reproduced from the git repository
> I do not see any great benefit in maintaining them long term.  My feeling
> is that is they will loose their usefulness once an official release is
> made, or possibly after two releases.  So how about:
>
>   * Snapshots are retained for at least 1 year after their creation.
>
>   * After that they can be culled at the discretion of the overseers
>     (and without needing to consult with the binutils project) as and
>     when they wish.

I would add, probably doing the snapshots in a weekly basis would be
enough, like GCC does?

Re: binutils snapshot builds

[Mark Wielaard]

Hi Jose,

On Fri, May 24, 2024 at 11:58:56AM +0200, Jose E. Marchesi wrote:
> > On Thu, May 23, 2024 at 05:11:40PM +0200, Jose E. Marchesi wrote:
> >> I see that the name of the snapshot tarball for a version
> >> 
> >>   m4_define([BFD_VERSION], [2.42.50-fc12b44ec0e])
> >> 
> >> is
> >> 
> >>   binutils-2.42.50.tar.gz
> >> 
> >> What about having:
> >> 
> >>   binutils-2.42.50-fc12b44ec0e.tar.gz
> >> 
> >> This would allow for example to easily cache already downloaded
> >> snapshots on the client side.
> >
> > Yeah, that is what we do for glibc. I assume it should then also
> > unpack in a binutils-2.42.50-fc12b44ec0e directory?
> 
> Yep.

OK, that was a bit of work and needed a little hack.  ./src-release
uses the output of bfd/configure --version to create the directory/tar
name.
https://sourceware.org/cgit/builder/commit/?id=9a85f672549a68c7114cef580c13cbfd7caf65f9

> >> Also, would it be possible to have a file README (or LATEST or MANIFEST
> >> or whatever) in the directory with a single line containing the name of
> >> the tarball?  This would ease automation.
> >
> > Maybe that is a valid usecase for the md512.sum file?
> > Or does it have to be just the filename or a symlink to it.
> >
> > Do you already have some automation for something similar?
> 
> Any file from which it is easy to extract the name of the latest
> snapshot would do.
> 
> For GCC snapshots I have this:
> 
> GCC_TARBALL=`wget https://gcc.gnu.org/pub/gcc/snapshots/LATEST-15 -O - -o /dev/null | grep -E 'gcc-15-[0-9]+.tar.xz' | sed -e 's/.*\(gcc-15-[^<]*\).*/\1/'`

OK, all new dirs have a sha512.sum file now. You should be able to
grep/sed the file name out of that one:

 $ BINUTILS_TARBALL=`wget https://snapshots.sourceware.org/binutils/trunk/latest/src/sha512.sum -O - -o /dev/null | grep -E 'binutils-[0-9a-f.-]+.tar.xz' | sed -e 's/.*\(binutils-[^<]*\).*/\1/'`
 $ echo $BINUTILS_TARBALL
binutils-2.42.50-0daa17bf187.tar.xz

Note that I changed the binutils, gdb and glibc snapshots to produce
tar.xz files. Those are almost half of the tar.gz sized.

xz does use a bit more process/memory. It might be nice to use zstd
instead. If zstd is common enough of course. What do people think?

Cheers,

Mark

Re: binutils snapshot builds

[Nick Clifton]

Hi Mark,

> Note that I changed the binutils, gdb and glibc snapshots to produce
> tar.xz files. Those are almost half of the tar.gz sized.
> 
> xz does use a bit more process/memory. It might be nice to use zstd
> instead. If zstd is common enough of course. What do people think?

Does tar automatically understand it ?  ie can I run:

   tar xvf <foo>.zst

and have it work ?  If so, then I would definitely think that
adding zstd support would be a good idea.

Cheers
   Nick

Re: binutils snapshot builds

[Mark Wielaard]

Hi Nick,

On Tue, 2024-05-28 at 15:35 +0100, Nick Clifton wrote:
> > xz does use a bit more process/memory. It might be nice to use zstd
> > instead. If zstd is common enough of course. What do people think?
> 
> Does tar automatically understand it ?  ie can I run:
> 
>    tar xvf <foo>.zst
> 
> and have it work ?

Yes, it does, since GNU tar 1.31 (released 2019-01-02 - 5 years ago).

Cheers,

Mark