Using builder for (documentation) snapshots

Using builder for (documentation) snapshots

[Mark Wielaard]

Hi,

Thanks to OSUOSL we now have an extra buildbot worker with extra
storage so we can do container builds and publish (documentation)
snapshots. https://snapshots.sourceware.org/

It acts like a normal buildbot (x86_64) container worker, except that
whatever is left in the output directory will be available through the
snapshots server.

Some examples of where this is useful:

- elfutils test coverage reports. Currently they are only produced (by
  hand) once an official release is made. Ideally we have coverage
  reports of current git.

- snapshots of binutils or gnupoke which are either currently done by
  hand (and so need a script to be written to automate) or which have
  a non-trivial boostrap script.

- generating the gcc documentation using the lowest and highest
  supported version of texinfo to make sure it works and looks as
  intended.

- Generating the documentation/website of libabigail which is
  currently done by hand and needs a specific setup to generate from
  the source code with sphynx.

- valgrind the html and pdf manuals are currently only generated by
  hand during a release.

- dwarfstd.org is automatically generated through a git hook from the
  main branch, but people have wanted to show how things look from a
  specific branch.

There is an example of how to do the elfutils coverage report in
builbot.git:
https://sourceware.org/cgit/builder/commit/?id=365a64a19b9a4b3cd677ec377dd1574e4dcdf022
This also contains the setup instructions of the snapshots server plus
scripts used. The output can be seen here:
https://snapshots.sourceware.org/elfutils/coverage/

I'll help setup the other examples.

Retention
---------

The server has 600GB free storage, so even if we really create a new
snapshot every 15 minutes we have some way to go. We could install a
script that scrubs snapshots after a year, or that only keeps one
snapshot a day after a month. A snapshot can be recreated from some
git checkout and aren't intended to be official releases so we can
erase them whenever we want. We just need to write a script to do it.

Write back
----------

Although the original idea was that the "latest" snapshot could be
written back to the sourceware.org webserver dir and/or htdocs git
repo I didn't add such functionality. It is probably better to see how
useful these documentation/web snapshots are before replacing existing
update machanisms.

Cheers,

Mark

Re: Using builder for (documentation) snapshots

[Mark Wielaard]

Hi,

On Sun, Apr 09, 2023 at 01:30:50PM +0200, Mark Wielaard via Overseers wrote:
> Thanks to OSUOSL we now have an extra buildbot worker with extra
> storage so we can do container builds and publish (documentation)
> snapshots. https://snapshots.sourceware.org/
> 
> It acts like a normal buildbot (x86_64) container worker, except that
> whatever is left in the output directory will be available through the
> snapshots server.
> 
> Some examples of where this is useful:
> 
> - elfutils test coverage reports. Currently they are only produced (by
>   hand) once an official release is made. Ideally we have coverage
>   reports of current git.
> 
> - snapshots of binutils or gnupoke which are either currently done by
>   hand (and so need a script to be written to automate) or which have
>   a non-trivial boostrap script.
> 
> - generating the gcc documentation using the lowest and highest
>   supported version of texinfo to make sure it works and looks as
>   intended.
> 
> - Generating the documentation/website of libabigail which is
>   currently done by hand and needs a specific setup to generate from
>   the source code with sphynx.
> 
> - valgrind the html and pdf manuals are currently only generated by
>   hand during a release.
> 
> - dwarfstd.org is automatically generated through a git hook from the
>   main branch, but people have wanted to show how things look from a
>   specific branch.
> 
> There is an example of how to do the elfutils coverage report in
> builbot.git:
> https://sourceware.org/cgit/builder/commit/?id=365a64a19b9a4b3cd677ec377dd1574e4dcdf022
> This also contains the setup instructions of the snapshots server plus
> scripts used. The output can be seen here:
> https://snapshots.sourceware.org/elfutils/coverage/
> 
> I'll help setup the other examples.

I added two more:

libabigail manual, api docs and website:
https://snapshots.sourceware.org/libabigail/html-doc/
https://sourceware.org/cgit/builder/commit/?id=39641bd063c4bbeab4ce4f38625f1140a84f5b4d

valgrind trunk snapshot, manual and doc (pdf) downloads:
https://snapshots.sourceware.org/valgrind/trunk/
https://sourceware.org/cgit/builder/commit/?id=f338a430db0382d9bd26b447cfe762566c9e7935

The idea is the same:
- Pick or create a new container file (add any build dependencies)
- Add a Periodic for the project (trigger every 15 minutes, checking
  for any changes on a specific branch)
- Create a BuildFactory with build steps (ideally existing make
  targets or a build script in the repo) which put any artifacts under
  the output dir and create a publish file containing the intended
  (branch/topic) name.
- Add a BuildConfig using the container file and build factory.

Cheers,

Mark