value of gdbadmin effort to create gdb daily/weekly source tarballs

value of gdbadmin effort to create gdb daily/weekly source tarballs

[Frank Ch. Eigler]

Hi -

Observing regular system activity on sourceware, the gdbadmin cron
jobs to create the https://sourceware.org/pub/gdb/snapshots/ struck me
as something that perhaps is no longer that useful.  With git-archive
being able to immediately generate tarballs of any version, with git
diffs producing the deltas, is there any utility at all in still
producing these tarballs?

Would you like me to check logs as to how many accesses to this have
taken place?  (A very rough brief search indicated 99%+ were bots like
search engines and ai ingesters.)

- FChE

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Joel Brobecker]

Hi Frank,

> Observing regular system activity on sourceware, the gdbadmin cron
> jobs to create the https://sourceware.org/pub/gdb/snapshots/ struck me
> as something that perhaps is no longer that useful.  With git-archive
> being able to immediately generate tarballs of any version, with git
> diffs producing the deltas, is there any utility at all in still
> producing these tarballs?
> 
> Would you like me to check logs as to how many accesses to this have
> taken place?  (A very rough brief search indicated 99%+ were bots like
> search engines and ai ingesters.)

If this isn't too much effort, that would give some factual data
that would hopefully turn the decision into a no-brainer.

Personally, I'm having a hard time seeing this as being sufficiently
useful to be worth the effort (speaking as the only person who gets
the alerts when the process breaks, and thus goes to fix it afterwards).

A small remark perhaps: The source packages I believe are created
using the src-release.sh script, which does more than just tar-ing
the files from the repository. We do the same where we create
official releases, so the current process produces snapshot which
are closer to the releases than a simple tar-ing operation would do.
With that said, I personally have experience re-building both binutils
and GDB from repository rather than sources, and it's no problem.
So for me this is not a significant reason to justify preserving
this capability.

On the other hand, it does act a periodic test of the script being
src-release.sh script staying operational. But if that were the only
reason we had, we can preserve this as a pure test, where we do
a periodic source packaging, but instead of saving it for people
to download, we simple discard it.

-- 
Joel

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Mark Wielaard]

Hi,

On Mon, Apr 15, 2024 at 10:48:07AM -0700, Joel Brobecker via Gdb wrote:
> On the other hand, it does act a periodic test of the script being
> src-release.sh script staying operational. But if that were the only
> reason we had, we can preserve this as a pure test, where we do
> a periodic source packaging, but instead of saving it for people
> to download, we simple discard it.

Other projects like glibc, valgrind, libabigail, etc. have moved their
snapshots builds to http://snapshots.sourceware.org/ which creates a
snapshot build and creates documentation from current git. This makes
sure the release processes/scripts keep working. And it has caught
release issues early (which often don't occur during a normal
developer build). Since this is part of the normal
builder.sourceware.org infrastructure it also provides better
visibility.

Cheers,

Mark

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Joel Brobecker]

> Other projects like glibc, valgrind, libabigail, etc. have moved their
> snapshots builds to http://snapshots.sourceware.org/ which creates a

https?

> snapshot build and creates documentation from current git. This makes
> sure the release processes/scripts keep working. And it has caught
> release issues early (which often don't occur during a normal
> developer build). Since this is part of the normal
> builder.sourceware.org infrastructure it also provides better
> visibility.

FWIW, that would definitely work for me. I do not have the time
to work on that, though :-(.

-- 
Joel

Re: value of gdbadmin effort to create gdb daily/weekly source tarballs

[Frank Ch. Eigler]

Hi, Joel -

> > Would you like me to check logs as to how many accesses to this have
> > taken place?  (A very rough brief search indicated 99%+ were bots like
> > search engines and ai ingesters.)
> 
> If this isn't too much effort, that would give some factual data
> that would hopefully turn the decision into a no-brainer.

Doing a more thorough search:

% zgrep pub/gdb/snapsho sourceware-combined_log* | fgrep .xz | egrep -iv 'bingbot|googlebot|amazonbot|gptbot|yandex|semrush|googleother|3BSZgF|crawler|8.219|8.222'| wc -l
12969
% zgrep pub/gdb/snapsho sourceware-combined_log* | fgrep .xz | egrep -i 'bingbot|googlebot|amazonbot|gptbot|yandex|semrush|googleother|3BSZgF|crawler|8.219|8.222' | wc -l
35001

i.e., 35001 tarball/diff downloads by known-to-be-bots and 12969 by
not-definitely-bots, over three months.  But I'm pretty sure most of
those 12969's are also bots, just not identifying themselves with
user-agent or other obvious-definite-botness signs.


> Personally, I'm having a hard time seeing this as being sufficiently
> useful to be worth the effort (speaking as the only person who gets
> the alerts when the process breaks, and thus goes to fix it afterwards).

Yeah.

> [...]
> On the other hand, it does act a periodic test of the script being
> src-release.sh script staying operational. But if that were the only
> reason we had, we can preserve this as a pure test, where we do
> a periodic source packaging, but instead of saving it for people
> to download, we simple discard it.

Or: try running it monthly rather than daily, dropping the diffs.


- FChE