Roadmap update

[Mark Wielaard <mark@klomp.org>]

Given the recent additions of a bunsenwidget, the patchwork upgrade,
the git users try branches and getting more worker compute resources I
updated the roadmap a bit adding a bit more context. More ideas,
comments, welcome!

The same, in HTML, with a few more URLs added, can be found here:
https://gnu.wildebeest.org/blog/mjw/2022/06/22/sourceware-gnu-toolchain-infrastructure-roadmap/

Sourceware – GNU Toolchain Infrastructure roadmap
=================================================

Making email/git based workflow more fun, secure and productive by
automating contribution tracking and testing across different distros
and architectures.

What is Sourceware?
-------------------

Sourceware, https://sourceware.org/, is community run infrastructure,
mailinglists, git, bug trackers, wikis, etc. hosted in the Red Hat
Open Source Community Infrastructure Community Cage together with
servers from e.g. Ceph, CentOS, Fedora and Gnome.

Sourceware is mainly known for hosting the GNU Toolchain projects,
like gcc at https://gcc.gnu.org/, glibc, binutils and gdb. But also
hosts projects like annobin, bunsen, bzip2, cgen, cygwin at
https://cygwin.org/, debugedit, dwz, elfutils at http://elfutils.org,
gccrs, gnu-abi, insight, kawa, libffi, libabigail, mauve, newlib,
systemtap and valgrind at https://valgrind.org/.

A longer list of Sourceware projects, those without their own domain
name, including several dormant projects, can be found here:
https://sourceware.org/mailman/listinfo.

Most of these projects use a email/git based workflow using
mailinglists for discussing patches in preference to web based
"forges".

Zero maintenance automation
---------------------------

Although email based git workflows are great for real patch
discussions, they do not always make tracking the state of patches
easy.

Just like our other services, such as bugzilla, mailinglists and git
repos we like to provide zero maintenance infrastructure for tracking
and automation of patches and testing.

So we are trying to consolidate around a shared buildbot for (test)
automation and patchwork for tracking the state of contributions. By
sharing experiences between the Sourceware projects and coordination
and fully automating the infrastructure services.

A shared buildbot
-----------------

We have a shared buildbot for Sourceware projects at
https://builder.sourceware.org/. This includes compute resources
(buildbot-workers) for various architectures thanks to some generous
sponsors. We have native/VM workers for x86_64, ppc64le, s390x, ppc64,
i386, arm64 and armhf for debian, fedora and centos (although not all
combinations yet) and x86_64 container builders for fedora, debian and
opensuse.

There are currently 95 builders on 15 workers, doing ~300 builds a day
(more on week days, less on weekends). There are a couple of full
testsuite builders (for gcc and binutils-gdb), but most builders are
"quick" CI builders, which will sent email whenever a regression is
detected. It seems to catch and report a couple of issues a week
across all projects.

Builder is its own project on Sourceware which comes with its own git
repo, mailinglist and amazing community, that can help you integrate
new builders, add workers, containers and get you access to systems to
replicate any failures where the buildbot logs don't give enough
information.

And buildbot itself is automated, so whenever a change is made to add
a new builder, or define a new container, the buildbot automatically
reconfigures itself and the workers will start using the new container
images starting with the next build.

The same mechanism can also be used to run tasks on specific commits
or periodically. Tasks which are now often done by a cron job or git
hook.  For example to update documentation, websites, generate release
tars or update bugzilla. The advantage over cron jobs is that it can
be done more immediately and/or only when really needed based on
specific commit files. The advantage over git hooks is that they run
in the builder context, not in the context of the specific user that
pushed a commit.

Picking your (CI) tests
-----------------------

Although the buildbot itself is zero maintenance, getting and acting
on the results of course is not. We already divide the tests into
quick CI tests and full test runs. And most tests upload all results
to bunsendb. bunsen can help pick good CI tests by indicating which
tests are flaky or compare results across different setups.

A prototype testsuite log comparison bunsenweb widget is running at
https://builder.sourceware.org/testruns/

Lots of things will be coming here, including taking advantage of
testrun cluster analysis that's already being done, a per-testrun
testcase search/browse engine, other search operators, testsuite
summary (vs detail) grids, who knows, ideas welcome!

What about pre-commit checks?
-----------------------------

The builder CI checks what has been committed on the main branch of
the projects. This makes sure that what is checked out is in a good
state and that any pushed regressions are found early and often.

There is also support for git user try branches. When a user pushes to
their try branch the same builder CI checks are ran, so a project
developer knows their proposed patch(es) won't break the build or
introduce regressions.

The binutils and gdb communities are currently trying this out.  Once
new builder resources from OSUOSL are installed we'll roll this out to
other Sourceware projects.

What about non-committers?
--------------------------

The above only helps developers that have commit access on sourceware,
but not others who sent in patches. For that we have
https://patchwork.sourceware.org/ plus the CICD trybot that DJ wrote
https://sourceware.org/glibc/wiki/CICDDesign. The glibc community is
already using this. We would like to connect patchwork, buildbot and
the trybot for other Sourceware projects

The current trybot doesn't do authentication, this might not be OK for
all builders. So we want to either require checking for known GPG keys
on the patch emails or let a trusted developer set a flag in patchwork
before the trybot triggers. Once we have public-inbox setup we could
also use b4 for DKIM attestation for known/trusted hackers.

Some projects have already experimented with public-inbox. But we
don't have an instance running on sourceware itself yet. This would
resolve complaints of not very usable mailman archives.

But I really like to have a webforge!
-------------------------------------

You are in luck. We already have a sourcehut mirror at
https://sr.ht/~sourceware/ This allows anybody to fork any sourceware
project on sourcehut, prepare their patches and submit a merge request
through email (without having to locally setup git send-email or smtp,
the patch emails are generated server side).

Sourcehut is designed around email based workflows, fully Free
Software, doesn't use javascript and is much faster and resource
constrained compared to (proprietary) alternatives.

The sourcehut mirror is currently read-only (but syncs automatically
with any git updates made on sourceware). When sourcehut supports
project groups (one of the beta goals) we will test a self-hosted
instance to see whether this is a good way to attract more
contributors without loosing the advantages of the email based
workflow. The various sr.ht components are very modular so we can only
use those parts we need.