public inbox for overseers@sourceware.org
 help / color / mirror / Atom feed
From: Mark Wielaard <mark@klomp.org>
To: overseers@sourceware.org
Cc: gcc@gcc.gnu.org, binutils@sourceware.org, gdb@sourceware.org,
	libc-alpha@sourceware.org
Subject: Sourceware mitigating and preventing the next xz-backdoor
Date: Mon, 1 Apr 2024 17:06:17 +0200	[thread overview]
Message-ID: <20240401150617.GF19478@gnu.wildebeest.org> (raw)
In-Reply-To: <20240329203909.GS9427@gnu.wildebeest.org>

A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.

This email isn't about the xz-backdoor itself. Do see Sam James FAQ
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
(Sorry for the github link, but this one does seem viewable without
proprietary javascript)

We should discuss what we have been doing and should do more to
mitigate and prevent the next xz-backdoor. There are a couple of
Sourceware services that can help with that.

TLDR;
- Replicatable isolated container/VMs are nice, we want more.
- autoregen buildbots, it should be transparent (and automated) how to
  regenerate build/source files.
- Automate (snapshot) releases tarballs.
- Reproducible releases (from git).

The impact on Sourceware was minimal because we were able to just zap
the affected systems. That is because we started putting everything
into containers and isolated VMs that are easy to reinstall.

In this case we were prepared, but it was also luck that the systems
that were potentially affected were already setup this way. We would
like to be able to do this for any system we deploy. That is why the
Sourceware Project Leadership Committee together with the Software
Freedom Conservancy staff put together "Sourceware 2024 - The Plan"
with as main theme isolation.

https://inbox.sourceware.org/20240325095827.GI5673@gnu.wildebeest.org/

We should be proactive and make sure we are prepared. This will not
just help security of the systems. But by moving services into their
own container or VM will help scaling by making it easy to move
between separate physical machines. And it makes us provide services
that anybody can replicate and setup themselves.

A large part of the xz-backdoor was possible because of social
engineering, which is difficult to solve with technical measures. But
another part was because generation of some of the build/source files
and the release tarball was not transparent and not reproducible.

Having source files or release tarballs being hard to (re)generate
lowers our own trust and the trust that our users have that the
sources in git matches what the released software is build from. It
also makes reviews and automated (pre-commit) CI harder. We have been
working on these issues in two ways, creating automated release
snapshots and the autoregen buildbots.

For projects like gcc, gdb and binutils we have introduced autoregen
buildbots which warn when some generated files are not properly
regenerated from source.
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
This has caught various issues, but isn't complete yet. Christophe
Lyon and Simon Marchi have been working on more generic make
regenerate targets.
https://inbox.sourceware.org/20240313080237.1143034-1-christophe.lyon@linaro.org

See also Eric Gallager's discussion on automake make distcheck:
https://lists.gnu.org/archive/html/automake/2024-03/msg00007.html

With the help of OSUOSL we have also introduced automated release
snapshots. Projects, like glibc, valgrind, libabigail have fully
automated release tarball (and documentation) generation against
current git. https://snapshots.sourceware.org/

This has caught some issues early which would otherwise only have been
detected at release time and has made sure the generation of other
release artifacts e.g. documentation keeps working. All this is done
in known containers, so also checks that releases can be made with
stable tools.

There are some possible future steps that we can do to make it easier
to check that the release tarballs are created from the actual git
sources. We can ask release managers to check release tarballs they
create against the automated snapshots releases. And/or ask them to
reuse the containers from snapshots or autoregen builders to create
releases. https://sourceware.org/cgit/builder/tree/README_containers

And we should try to work with the reproducible builds project to make
sure that release artifacts can be reproducibly created from the git
sources. https://reproducible-builds.org/ Reproducible builds is also
a Software Freedom Conservancy member project.

  reply	other threads:[~2024-04-01 15:06 UTC|newest]

Thread overview: 58+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-03-29 20:39 Security warning about xz library compromise Mark Wielaard
2024-04-01 15:06 ` Mark Wielaard [this message]
2024-04-02 19:54   ` Sourceware mitigating and preventing the next xz-backdoor Sandra Loosemore
2024-04-02 20:03     ` Paul Eggert
2024-04-02 20:20       ` Paul Koning
2024-04-02 20:28         ` Ian Lance Taylor
2024-04-03  6:26           ` Martin Uecker
2024-04-03 14:00             ` Michael Matz
2024-04-03 14:14               ` Paul Koning
2024-04-03 14:32               ` Martin Uecker
2024-04-03 14:46                 ` Jeffrey Walton
2024-04-03 16:02                 ` Michael Matz
2024-04-03 16:26                   ` Joel Sherrill
2024-04-03 16:32                   ` Martin Uecker
2024-04-03 16:51                 ` Andreas Schwab
2024-04-03 16:56                 ` Jonathan Wakely
2024-04-03 18:46               ` Jonathon Anderson
2024-04-03 19:01                 ` Martin Uecker
2024-04-05 21:15                   ` Andrew Sutton
2024-04-06 13:00                     ` Richard Biener
2024-04-06 15:59                       ` Martin Uecker
2024-04-04 13:59                 ` Michael Matz
2024-04-09 16:44                   ` anderson.jonathonm
2024-04-09 17:57                     ` Andreas Schwab
2024-04-09 19:59                       ` Jonathon Anderson
2024-04-09 20:11                         ` Paul Koning
2024-04-09 21:40                           ` Jeffrey Walton
2024-04-09 21:50                             ` Paul Eggert
2024-04-09 21:58                               ` Sam James
2024-04-09 22:15                                 ` Paul Eggert
2024-04-09 22:22                                   ` Sam James
2024-04-09 22:53                                     ` Paul Eggert
2024-04-09 22:03                               ` Jonathon Anderson
2024-04-09 22:10                                 ` Sam James
2024-04-09 21:54                           ` Jonathon Anderson
2024-04-09 22:00                             ` Sam James
2024-04-10 14:09                             ` Frank Ch. Eigler
2024-04-10 18:47                               ` Jonathon Anderson
2024-04-10 19:00                                 ` Frank Ch. Eigler
2024-04-10 10:26                       ` Claudio Bantaloukas
2024-04-02 22:08     ` Guinevere Larsen
2024-04-02 22:21       ` Guinevere Larsen
2024-04-02 22:50       ` Jeffrey Walton
2024-04-02 23:20       ` Mark Wielaard
2024-04-02 23:34       ` Paul Koning
2024-04-03  0:37         ` Jeffrey Walton
2024-04-03  8:08       ` Florian Weimer
2024-04-03 13:53         ` Joel Sherrill
2024-04-04 10:25           ` Mark Wielaard
2024-04-10 16:30           ` Alejandro Colomar
2024-04-21 15:30             ` Mark Wielaard
2024-04-21 20:40               ` Alejandro Colomar
2024-04-21 20:52                 ` Alejandro Colomar
2024-04-30 11:28                 ` Alejandro Colomar
2024-04-03 14:04         ` Tom Tromey
2024-04-03 14:42           ` Jeff Law
2024-04-04 10:48             ` Mark Wielaard
2024-07-13 13:56 Trenton Davison

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240401150617.GF19478@gnu.wildebeest.org \
    --to=mark@klomp.org \
    --cc=binutils@sourceware.org \
    --cc=gcc@gcc.gnu.org \
    --cc=gdb@sourceware.org \
    --cc=libc-alpha@sourceware.org \
    --cc=overseers@sourceware.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).