value of gdbadmin effort to create gdb daily/weekly source tarballs

value of gdbadmin effort to create gdb daily/weekly source tarballs

[Frank Ch. Eigler]

Hi -

Observing regular system activity on sourceware, the gdbadmin cron
jobs to create the https://sourceware.org/pub/gdb/snapshots/ struck me
as something that perhaps is no longer that useful.  With git-archive
being able to immediately generate tarballs of any version, with git
diffs producing the deltas, is there any utility at all in still
producing these tarballs?

Would you like me to check logs as to how many accesses to this have
taken place?  (A very rough brief search indicated 99%+ were bots like
search engines and ai ingesters.)

- FChE

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Joel Brobecker]

Hi Frank,

> Observing regular system activity on sourceware, the gdbadmin cron
> jobs to create the https://sourceware.org/pub/gdb/snapshots/ struck me
> as something that perhaps is no longer that useful.  With git-archive
> being able to immediately generate tarballs of any version, with git
> diffs producing the deltas, is there any utility at all in still
> producing these tarballs?
> 
> Would you like me to check logs as to how many accesses to this have
> taken place?  (A very rough brief search indicated 99%+ were bots like
> search engines and ai ingesters.)

If this isn't too much effort, that would give some factual data
that would hopefully turn the decision into a no-brainer.

Personally, I'm having a hard time seeing this as being sufficiently
useful to be worth the effort (speaking as the only person who gets
the alerts when the process breaks, and thus goes to fix it afterwards).

A small remark perhaps: The source packages I believe are created
using the src-release.sh script, which does more than just tar-ing
the files from the repository. We do the same where we create
official releases, so the current process produces snapshot which
are closer to the releases than a simple tar-ing operation would do.
With that said, I personally have experience re-building both binutils
and GDB from repository rather than sources, and it's no problem.
So for me this is not a significant reason to justify preserving
this capability.

On the other hand, it does act a periodic test of the script being
src-release.sh script staying operational. But if that were the only
reason we had, we can preserve this as a pure test, where we do
a periodic source packaging, but instead of saving it for people
to download, we simple discard it.

-- 
Joel

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Mark Wielaard]

Hi,

On Mon, Apr 15, 2024 at 10:48:07AM -0700, Joel Brobecker via Gdb wrote:
> On the other hand, it does act a periodic test of the script being
> src-release.sh script staying operational. But if that were the only
> reason we had, we can preserve this as a pure test, where we do
> a periodic source packaging, but instead of saving it for people
> to download, we simple discard it.

Other projects like glibc, valgrind, libabigail, etc. have moved their
snapshots builds to http://snapshots.sourceware.org/ which creates a
snapshot build and creates documentation from current git. This makes
sure the release processes/scripts keep working. And it has caught
release issues early (which often don't occur during a normal
developer build). Since this is part of the normal
builder.sourceware.org infrastructure it also provides better
visibility.

Cheers,

Mark

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Joel Brobecker]

> Other projects like glibc, valgrind, libabigail, etc. have moved their
> snapshots builds to http://snapshots.sourceware.org/ which creates a

https?

> snapshot build and creates documentation from current git. This makes
> sure the release processes/scripts keep working. And it has caught
> release issues early (which often don't occur during a normal
> developer build). Since this is part of the normal
> builder.sourceware.org infrastructure it also provides better
> visibility.

FWIW, that would definitely work for me. I do not have the time
to work on that, though :-(.

-- 
Joel

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Frank Ch. Eigler]

Hi, Joel -

> > Would you like me to check logs as to how many accesses to this have
> > taken place?  (A very rough brief search indicated 99%+ were bots like
> > search engines and ai ingesters.)
> 
> If this isn't too much effort, that would give some factual data
> that would hopefully turn the decision into a no-brainer.

Doing a more thorough search:

% zgrep pub/gdb/snapsho sourceware-combined_log* | fgrep .xz | egrep -iv 'bingbot|googlebot|amazonbot|gptbot|yandex|semrush|googleother|3BSZgF|crawler|8.219|8.222'| wc -l
12969
% zgrep pub/gdb/snapsho sourceware-combined_log* | fgrep .xz | egrep -i 'bingbot|googlebot|amazonbot|gptbot|yandex|semrush|googleother|3BSZgF|crawler|8.219|8.222' | wc -l
35001

i.e., 35001 tarball/diff downloads by known-to-be-bots and 12969 by
not-definitely-bots, over three months.  But I'm pretty sure most of
those 12969's are also bots, just not identifying themselves with
user-agent or other obvious-definite-botness signs.


> Personally, I'm having a hard time seeing this as being sufficiently
> useful to be worth the effort (speaking as the only person who gets
> the alerts when the process breaks, and thus goes to fix it afterwards).

Yeah.

> [...]
> On the other hand, it does act a periodic test of the script being
> src-release.sh script staying operational. But if that were the only
> reason we had, we can preserve this as a pure test, where we do
> a periodic source packaging, but instead of saving it for people
> to download, we simple discard it.

Or: try running it monthly rather than daily, dropping the diffs.


- FChE

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]

Hi Joel,

On Mon, Apr 15, 2024 at 11:26:21AM -0700, Joel Brobecker wrote:
> > Other projects like glibc, valgrind, libabigail, etc. have moved their
> > snapshots builds to http://snapshots.sourceware.org/ which creates a
> 
> https?

Yes, sorry. http will always redirect to the https site.

> > snapshot build and creates documentation from current git. This makes
> > sure the release processes/scripts keep working. And it has caught
> > release issues early (which often don't occur during a normal
> > developer build). Since this is part of the normal
> > builder.sourceware.org infrastructure it also provides better
> > visibility.
> 
> FWIW, that would definitely work for me. I do not have the time
> to work on that, though :-(.

I added snapshots for trunk based on the binutils snapshots using
src-release.sh. Which is what ss/make-snapshot ultimately calls.

The snapshots can be found at:

https://snapshots.sourceware.org/gdb/trunk/

It gets regenerated every 15 minutes, if there have been any changes
since the last build. The latest can always be found here:

https://snapshots.sourceware.org/gdb/trunk/latest/src/

It doesn't generate snapshots for the branches yet. I think we don't
really need the diffs. We have to see if we can also do the ari and
docs updates on snapshots.sourceware.org.

Cheers,

Mark

[-- Attachment #2: 0001-Add-gdb-snapshots-builder.patch --]
[-- Type: text/plain, Size: 3535 bytes --]

From 5520d4377b37890d8866910503ded940b54da054 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Sat, 25 May 2024 00:35:59 +0200
Subject: [PATCH] Add gdb-snapshots builder

---
 builder/master.cfg | 47 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/builder/master.cfg b/builder/master.cfg
index db82c8b4122f..cbf9e30fd055 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -1038,6 +1038,19 @@ gdb_try_scheduler = schedulers.AnyBranchScheduler(
                       "gdb-try-fedora-ppc64le"])
 c['schedulers'].append(gdb_try_scheduler)
 
+gdb_snapshot_scheduler = schedulers.Periodic(
+        name="gdb-snapshots",
+        change_filter=util.ChangeFilter(project="binutils-gdb",
+                                        branch="master"),
+        branch="master",
+        periodicBuildTimer=15*60, # 15 minutes in seconds
+        onlyIfChanged=True,
+        fileIsImportant=gdbImportant,
+        onlyImportant=True,
+        reason="gdb periodic project master branch snapshot",
+        builderNames=["gdb-snapshots"])
+c['schedulers'].append(gdb_snapshot_scheduler)
+
 # A scheduler for everything binutils-gdb & gcc without filters
 binutils_gdb_scheduler = schedulers.SingleBranchScheduler(
         name="binutils-gdb",
@@ -3210,6 +3223,23 @@ def make_gdb_check_step(runtestflags = None):
                 haltOnFailure=False, flunkOnFailure=True)
         return step
 
+gdb_step_setversion = steps.ShellCommand(
+        workdir='binutils-gdb',
+        name="set gdb version",
+        command='sed -i -e "s/DATE/$(date -u +%Y%m%d)/;s/git/$(git rev-parse --short @)/" gdb/version.in')
+gdb_step_src_release = steps.ShellCommand(
+        workdir='binutils-gdb',
+        name="src-release",
+        command="./src-release.sh -g gdb")
+gdb_create_output_step = steps.ShellCommand(
+        workdir='binutils-gdb',
+        name="create output",
+        command="mkdir -p /home/builder/shared/output/src &&"
+        + "mv ./gdb-*tar.gz /home/builder/shared/output/src")
+gdb_create_publish_file_step = steps.ShellCommand(
+        name="create publish file",
+        command="echo gdb/trunk > /home/builder/shared/publish")
+
 gdb_factory = util.BuildFactory()
 gdb_factory.addStep(gdb_git_step)
 gdb_factory.addStep(gdb_rm_step)
@@ -3327,6 +3357,13 @@ gdb_factory_sanitize.addSteps(bunsen_logfile_upload_cpio_steps(
         tagsuffix='/extended-gdbserver'))
 gdb_factory_sanitize.addStep(gdb_rm_step)
 
+gdb_factory_snapshot = util.BuildFactory()
+gdb_factory_snapshot.addStep(gdb_git_step)
+gdb_factory_snapshot.addStep(gdb_step_setversion)
+gdb_factory_snapshot.addStep(gdb_step_src_release)
+gdb_factory_snapshot.addStep(wait_snapshots_output_ready_step)
+gdb_factory_snapshot.addStep(gdb_create_output_step)
+gdb_factory_snapshot.addStep(gdb_create_publish_file_step)
 
 gdb_alma_x86_64_builder = util.BuilderConfig(
 	name="gdb-alma-x86_64",
@@ -3534,6 +3571,16 @@ gdb_ubuntu_riscv_builder = util.BuilderConfig(
         factory=gdb_factory)
 c['builders'].append(gdb_ubuntu_riscv_builder)
 
+gdb_snapshots_builder = util.BuilderConfig(
+        name="gdb-snapshots",
+        collapseRequests=True,
+        properties={'container-file':
+                    readContainerFile('debian-stable')},
+        workernames="snapshots",
+        tags=["gdb-snapshots"],
+        factory=gdb_factory_snapshot)
+c['builders'].append(gdb_snapshots_builder)
+
 # binutils-gdb build steps, factory and builders
 # just a native build
 
-- 
2.45.1