Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard <mark@klomp.org>]

A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.

This email isn't about the xz-backdoor itself. Do see Sam James FAQ
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
(Sorry for the github link, but this one does seem viewable without
proprietary javascript)

We should discuss what we have been doing and should do more to
mitigate and prevent the next xz-backdoor. There are a couple of
Sourceware services that can help with that.

TLDR;
- Replicatable isolated container/VMs are nice, we want more.
- autoregen buildbots, it should be transparent (and automated) how to
  regenerate build/source files.
- Automate (snapshot) releases tarballs.
- Reproducible releases (from git).

The impact on Sourceware was minimal because we were able to just zap
the affected systems. That is because we started putting everything
into containers and isolated VMs that are easy to reinstall.

In this case we were prepared, but it was also luck that the systems
that were potentially affected were already setup this way. We would
like to be able to do this for any system we deploy. That is why the
Sourceware Project Leadership Committee together with the Software
Freedom Conservancy staff put together "Sourceware 2024 - The Plan"
with as main theme isolation.

https://inbox.sourceware.org/20240325095827.GI5673@gnu.wildebeest.org/

We should be proactive and make sure we are prepared. This will not
just help security of the systems. But by moving services into their
own container or VM will help scaling by making it easy to move
between separate physical machines. And it makes us provide services
that anybody can replicate and setup themselves.

A large part of the xz-backdoor was possible because of social
engineering, which is difficult to solve with technical measures. But
another part was because generation of some of the build/source files
and the release tarball was not transparent and not reproducible.

Having source files or release tarballs being hard to (re)generate
lowers our own trust and the trust that our users have that the
sources in git matches what the released software is build from. It
also makes reviews and automated (pre-commit) CI harder. We have been
working on these issues in two ways, creating automated release
snapshots and the autoregen buildbots.

For projects like gcc, gdb and binutils we have introduced autoregen
buildbots which warn when some generated files are not properly
regenerated from source.
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
This has caught various issues, but isn't complete yet. Christophe
Lyon and Simon Marchi have been working on more generic make
regenerate targets.
https://inbox.sourceware.org/20240313080237.1143034-1-christophe.lyon@linaro.org

See also Eric Gallager's discussion on automake make distcheck:
https://lists.gnu.org/archive/html/automake/2024-03/msg00007.html

With the help of OSUOSL we have also introduced automated release
snapshots. Projects, like glibc, valgrind, libabigail have fully
automated release tarball (and documentation) generation against
current git. https://snapshots.sourceware.org/

This has caught some issues early which would otherwise only have been
detected at release time and has made sure the generation of other
release artifacts e.g. documentation keeps working. All this is done
in known containers, so also checks that releases can be made with
stable tools.

There are some possible future steps that we can do to make it easier
to check that the release tarballs are created from the actual git
sources. We can ask release managers to check release tarballs they
create against the automated snapshots releases. And/or ask them to
reuse the containers from snapshots or autoregen builders to create
releases. https://sourceware.org/cgit/builder/tree/README_containers

And we should try to work with the reproducible builds project to make
sure that release artifacts can be reproducibly created from the git
sources. https://reproducible-builds.org/ Reproducible builds is also
a Software Freedom Conservancy member project.