Security warning about xz library compromise

Security warning about xz library compromise

[Mark Wielaard]

Sourceware hosts are not affected by the latest xz backdoor.

But we have reset the https://builder.sourceware.org containers of
debian-testing, fedora-rawhide and opensuse-tumbleweed. These
containers however didn't have ssh installed, were running on isolated
VMs on separate machines from our main hosts, snapshots and backup
servers.

If you are running one of these distros on your development machines
then please consult your distro security announcements:

https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
https://lists.debian.org/debian-security-announce/2024/msg00057.html
https://archlinux.org/news/the-xz-package-has-been-backdoored/
https://news.opensuse.org/2024/03/29/xz-backdoor/

Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

A big thanks to everybody working this long Easter weekend who helped
analyze the xz-backdoor and making sure the impact on Sourceware and
the hosted projects was minimal.

This email isn't about the xz-backdoor itself. Do see Sam James FAQ
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
(Sorry for the github link, but this one does seem viewable without
proprietary javascript)

We should discuss what we have been doing and should do more to
mitigate and prevent the next xz-backdoor. There are a couple of
Sourceware services that can help with that.

TLDR;
- Replicatable isolated container/VMs are nice, we want more.
- autoregen buildbots, it should be transparent (and automated) how to
  regenerate build/source files.
- Automate (snapshot) releases tarballs.
- Reproducible releases (from git).

The impact on Sourceware was minimal because we were able to just zap
the affected systems. That is because we started putting everything
into containers and isolated VMs that are easy to reinstall.

In this case we were prepared, but it was also luck that the systems
that were potentially affected were already setup this way. We would
like to be able to do this for any system we deploy. That is why the
Sourceware Project Leadership Committee together with the Software
Freedom Conservancy staff put together "Sourceware 2024 - The Plan"
with as main theme isolation.

https://inbox.sourceware.org/20240325095827.GI5673@gnu.wildebeest.org/

We should be proactive and make sure we are prepared. This will not
just help security of the systems. But by moving services into their
own container or VM will help scaling by making it easy to move
between separate physical machines. And it makes us provide services
that anybody can replicate and setup themselves.

A large part of the xz-backdoor was possible because of social
engineering, which is difficult to solve with technical measures. But
another part was because generation of some of the build/source files
and the release tarball was not transparent and not reproducible.

Having source files or release tarballs being hard to (re)generate
lowers our own trust and the trust that our users have that the
sources in git matches what the released software is build from. It
also makes reviews and automated (pre-commit) CI harder. We have been
working on these issues in two ways, creating automated release
snapshots and the autoregen buildbots.

For projects like gcc, gdb and binutils we have introduced autoregen
buildbots which warn when some generated files are not properly
regenerated from source.
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
This has caught various issues, but isn't complete yet. Christophe
Lyon and Simon Marchi have been working on more generic make
regenerate targets.
https://inbox.sourceware.org/20240313080237.1143034-1-christophe.lyon@linaro.org

See also Eric Gallager's discussion on automake make distcheck:
https://lists.gnu.org/archive/html/automake/2024-03/msg00007.html

With the help of OSUOSL we have also introduced automated release
snapshots. Projects, like glibc, valgrind, libabigail have fully
automated release tarball (and documentation) generation against
current git. https://snapshots.sourceware.org/

This has caught some issues early which would otherwise only have been
detected at release time and has made sure the generation of other
release artifacts e.g. documentation keeps working. All this is done
in known containers, so also checks that releases can be made with
stable tools.

There are some possible future steps that we can do to make it easier
to check that the release tarballs are created from the actual git
sources. We can ask release managers to check release tarballs they
create against the automated snapshots releases. And/or ask them to
reuse the containers from snapshots or autoregen builders to create
releases. https://sourceware.org/cgit/builder/tree/README_containers

And we should try to work with the reproducible builds project to make
sure that release artifacts can be reproducibly created from the git
sources. https://reproducible-builds.org/ Reproducible builds is also
a Software Freedom Conservancy member project.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sandra Loosemore]

On 4/1/24 09:06, Mark Wielaard wrote:
> A big thanks to everybody working this long Easter weekend who helped
> analyze the xz-backdoor and making sure the impact on Sourceware and
> the hosted projects was minimal.
> 
> This email isn't about the xz-backdoor itself. Do see Sam James FAQ
> https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
> (Sorry for the github link, but this one does seem viewable without
> proprietary javascript)
> 
> We should discuss what we have been doing and should do more to
> mitigate and prevent the next xz-backdoor. There are a couple of
> Sourceware services that can help with that.
> 
> TLDR;
> - Replicatable isolated container/VMs are nice, we want more.
> - autoregen buildbots, it should be transparent (and automated) how to
>    regenerate build/source files.
> - Automate (snapshot) releases tarballs.
> - Reproducible releases (from git).
> 
> [snip]

While I appreciate the effort to harden the Sourceware infrastructure 
against malicious attacks and want to join in on thanking everyone who 
helped analyze this issue, to me it seems like the much bigger problem 
is that XZ had a maintainer who appears to have acted in bad faith.  Are 
the development processes used by the GNU toolchain components robust 
enough to cope with deliberate sabotage of the code base?  Do we have 
enough eyes available to ensure that every commit, even those by 
designated maintainers, is vetted by someone else?  Do we to harden our 
process, too, to require all patches to be signed off by someone else 
before committing?

-Sandra

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/2/24 12:54, Sandra Loosemore wrote:
> Do we to harden our process, too, to require all patches to be signed 
> off by someone else before committing?

It's easy for an attacker to arrange to have "someone else" in cahoots.

Although signoffs can indeed help catch inadvertent mistakes, they're 
relatively useless against determined attacks of this form, and we must 
assume that nation-state attackers will be determined.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning]

> On Apr 2, 2024, at 4:03 PM, Paul Eggert <eggert@CS.UCLA.EDU> wrote:
> 
> On 4/2/24 12:54, Sandra Loosemore wrote:
>> Do we to harden our process, too, to require all patches to be signed off by someone else before committing?
> 
> It's easy for an attacker to arrange to have "someone else" in cahoots.
> 
> Although signoffs can indeed help catch inadvertent mistakes, they're relatively useless against determined attacks of this form, and we must assume that nation-state attackers will be determined.

Another consideration is the size of the project.  "Many eyeballs" helps if there are plenty of people watching.  For smaller tools that have only a small body of contributors, it's easier for one or two malicious ones to subvert things.

Would it help to require (rather than just recommend) "don't use root except for the actual 'install' step" ?

	paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Ian Lance Taylor]

On Tue, Apr 2, 2024 at 1:21 PM Paul Koning via Gcc <gcc@gcc.gnu.org> wrote:
>
> Would it help to require (rather than just recommend) "don't use root except for the actual 'install' step" ?

Seems reasonable, but note that it wouldn't make any difference to
this attack.  The liblzma library was modified to corrupt the sshd
binary, when sshd was linked against liblzma.  The actual attack
occurred via a connection to a corrupt sshd.  If sshd was running as
root, as is normal, the attacker had root access to the machine.  None
of the attacking steps had anything to do with having root access
while building or installing the program.

Ian

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker]

Am Dienstag, dem 02.04.2024 um 13:28 -0700 schrieb Ian Lance Taylor via Gcc:
> > On Tue, Apr 2, 2024 at 1:21 PM Paul Koning via Gcc <gcc@gcc.gnu.org> wrote:
> > > > 
> > > > Would it help to require (rather than just recommend) "don't use root except for the actual 'install' step" ?
> > 
> > Seems reasonable, but note that it wouldn't make any difference to
> > this attack.  The liblzma library was modified to corrupt the sshd
> > binary, when sshd was linked against liblzma.  The actual attack
> > occurred via a connection to a corrupt sshd.  If sshd was running as
> > root, as is normal, the attacker had root access to the machine.  None
> > of the attacking steps had anything to do with having root access
> > while building or installing the program.

There does not seem a single good solution against something like this.

My take a way is that software needs to become less complex. Do 
we really still need complex build systems such as autoconf?
Are there still so many different configurations with subtle differences 
that every single feature needs to be tested individually by
running code at build time?

Martin

Re: Sourceware mitigating and preventing the next xz-backdoor

[Michael Matz]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

Hello,

On Wed, 3 Apr 2024, Martin Uecker via Gcc wrote:

> > > Seems reasonable, but note that it wouldn't make any difference to
> > > this attack.  The liblzma library was modified to corrupt the sshd
> > > binary, when sshd was linked against liblzma.  The actual attack
> > > occurred via a connection to a corrupt sshd.  If sshd was running as
> > > root, as is normal, the attacker had root access to the machine.  None
> > > of the attacking steps had anything to do with having root access
> > > while building or installing the program.
> 
> There does not seem a single good solution against something like this.
> 
> My take a way is that software needs to become less complex. Do 
> we really still need complex build systems such as autoconf?

Do we really need complex languages like C++ to write our software in?  
SCNR :)  Complexity lies in the eye of the beholder, but to be honest in 
the software that we're dealing with here, the build system or autoconf 
does _not_ come to mind first when thinking about complexity.

(And, FWIW, testing for features isn't "complex".  And have you looked at 
other build systems?  I have, and none of them are less complex, just 
opaque in different ways from make+autotools).


Ciao,
Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning]

> On Apr 3, 2024, at 10:00 AM, Michael Matz <matz@suse.de> wrote:
> 
> Hello,
> 
> On Wed, 3 Apr 2024, Martin Uecker via Gcc wrote:
> 
>>>> Seems reasonable, but note that it wouldn't make any difference to
>>>> this attack.  The liblzma library was modified to corrupt the sshd
>>>> binary, when sshd was linked against liblzma.  The actual attack
>>>> occurred via a connection to a corrupt sshd.  If sshd was running as
>>>> root, as is normal, the attacker had root access to the machine.  None
>>>> of the attacking steps had anything to do with having root access
>>>> while building or installing the program.
>> 
>> There does not seem a single good solution against something like this.
>> 
>> My take a way is that software needs to become less complex. Do 
>> we really still need complex build systems such as autoconf?
> 
> Do we really need complex languages like C++ to write our software in?  
> SCNR :)  Complexity lies in the eye of the beholder, but to be honest in 
> the software that we're dealing with here, the build system or autoconf 
> does _not_ come to mind first when thinking about complexity.
> 
> (And, FWIW, testing for features isn't "complex".  And have you looked at 
> other build systems?  I have, and none of them are less complex, just 
> opaque in different ways from make+autotools).
> 
> Ciao,
> Michael.

I would tend to agree with that even given my limited exposure to alternatives.

One aspect of the present attack that needs to be cured is that -- as I understand it -- the open source repository was fine but the kit as distributed had been subverted.  In other words, the standard assumption that the repository actually corresponds to the released code was not valid.  And furthermore, that it wasn't unusual for the kit to contain different or additional elements, just that it wasn't supposed to differ in malicious ways.

One possible answer is for all elements of kits to be made explicitly visible, though generated files probably don't want to be held in a normal source control system.  Another possible answer is for consumers of kits to treat kits as suspect, and have them unpacked and examined -- including any elements not source controlled -- before acceptance.  I think the first option is better because it exposes these additional elements to ongoing scrutiny from the entire community, rather than only one-time inspection by release managers who are probably quite pressed for time.

Either way, the reasons for these extra files to exist and the manner in which they are supposed to be generated would need to be both well documented and readily reproducible by outside parties.

	paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker]

Am Mittwoch, dem 03.04.2024 um 16:00 +0200 schrieb Michael Matz:
> Hello,
> 
> On Wed, 3 Apr 2024, Martin Uecker via Gcc wrote:
> 
> > > > Seems reasonable, but note that it wouldn't make any difference to
> > > > this attack.  The liblzma library was modified to corrupt the sshd
> > > > binary, when sshd was linked against liblzma.  The actual attack
> > > > occurred via a connection to a corrupt sshd.  If sshd was running as
> > > > root, as is normal, the attacker had root access to the machine.  None
> > > > of the attacking steps had anything to do with having root access
> > > > while building or installing the program.
> > 
> > There does not seem a single good solution against something like this.
> > 
> > My take a way is that software needs to become less complex. Do 
> > we really still need complex build systems such as autoconf?
> 
> Do we really need complex languages like C++ to write our software in?  
> SCNR :)  

Probably not.

> Complexity lies in the eye of the beholder, but to be honest in 
> the software that we're dealing with here, the build system or autoconf 
> does _not_ come to mind first when thinking about complexity.

The backdoor was hidden in a complicated autoconf script...

> 
> (And, FWIW, testing for features isn't "complex".  And have you looked at 
> other build systems?  I have, and none of them are less complex, just 
> opaque in different ways from make+autotools).

I ask a very specific question: To what extend is testing 
for features instead of semantic versions and/or supported
standards still necessary?  This seems like a problematic approach
that  may have been necessary decades ago, but it seems it may be
time to move on.


Martin

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton]

On Wed, Apr 3, 2024 at 10:33 AM Martin Uecker via Gdb
<gdb@sourceware.org> wrote:
>
> Am Mittwoch, dem 03.04.2024 um 16:00 +0200 schrieb Michael Matz:
> > [...]
> > (And, FWIW, testing for features isn't "complex".  And have you looked at
> > other build systems?  I have, and none of them are less complex, just
> > opaque in different ways from make+autotools).
>
> I ask a very specific question: To what extend is testing
> for features instead of semantic versions and/or supported
> standards still necessary?  This seems like a problematic approach
> that  may have been necessary decades ago, but it seems it may be
> time to move on.

I think it is still needed. As a first example, Musl does not define
preprocessor macros to identify itself. The project feels features
should be probed at build time with a tool like Autoconf. As a second
example, activating code paths, like AVX2 and AVX512, requires
ensuring the compiler actually supports the ISA. Building a program on
an older distribution with an older compiler could run afoul.

Jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Michael Matz]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

Hello,

On Wed, 3 Apr 2024, Martin Uecker wrote:

> The backdoor was hidden in a complicated autoconf script...

Which itself had multiple layers and could just as well have been a 
complicated cmake function.

> > (And, FWIW, testing for features isn't "complex".  And have you looked at 
> > other build systems?  I have, and none of them are less complex, just 
> > opaque in different ways from make+autotools).
> 
> I ask a very specific question: To what extend is testing 
> for features instead of semantic versions and/or supported
> standards still necessary?

I can't answer this with absolute certainty, but points to consider: the 
semantic versions need to be maintained just as well, in some magic way.  
Because ultimately software depend on features of dependencies not on 
arbitrary numbers given to them.  The numbers encode these features, in 
the best case, when there are no errors.  So, no, version numbers are not 
a replacement for feature tests, they are a proxy.  One that is manually 
maintained, and hence prone to errors.

Now, supported standards: which one? ;-)  Or more in earnest: while on 
this mailing list here we could chose a certain set, POSIX, some 
languages, Windows, MacOS (versions so-and-so).  What about other 
software relying on other 3rdparty feature providers (libraries or system 
services)?  Without standards?

So, without absolute certainty, but with a little bit of it: yes, feature 
tests are required in general.  That doesn't mean that we could not 
do away with quite some of them for (e.g.) GCC, those that hold true on 
any platform we support.  But we can't get rid of the infrastructure for 
that, and can't get rid of certain classes of tests.

> This seems like a problematic approach that may have been necessary 
> decades ago, but it seems it may be time to move on.

I don't see that.  Many aspects of systems remain non-standardized.


Ciao,
Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Joel Sherrill]

[-- Attachment #1: Type: text/plain, Size: 3327 bytes --]

On Wed, Apr 3, 2024 at 11:03 AM Michael Matz via Gdb <gdb@sourceware.org>
wrote:

> Hello,
>
> On Wed, 3 Apr 2024, Martin Uecker wrote:
>
> > The backdoor was hidden in a complicated autoconf script...
>
> Which itself had multiple layers and could just as well have been a
> complicated cmake function.
>
> > > (And, FWIW, testing for features isn't "complex".  And have you looked
> at
> > > other build systems?  I have, and none of them are less complex, just
> > > opaque in different ways from make+autotools).
> >
> > I ask a very specific question: To what extend is testing
> > for features instead of semantic versions and/or supported
> > standards still necessary?
>
> I can't answer this with absolute certainty, but points to consider: the
> semantic versions need to be maintained just as well, in some magic way.
> Because ultimately software depend on features of dependencies not on
> arbitrary numbers given to them.  The numbers encode these features, in
> the best case, when there are no errors.  So, no, version numbers are not
> a replacement for feature tests, they are a proxy.  One that is manually
> maintained, and hence prone to errors.
>
> Now, supported standards: which one? ;-)  Or more in earnest: while on
> this mailing list here we could chose a certain set, POSIX, some
> languages, Windows, MacOS (versions so-and-so).  What about other
> software relying on other 3rdparty feature providers (libraries or system
> services)?  Without standards?
>
> So, without absolute certainty, but with a little bit of it: yes, feature
> tests are required in general.  That doesn't mean that we could not
> do away with quite some of them for (e.g.) GCC, those that hold true on
> any platform we support.  But we can't get rid of the infrastructure for
> that, and can't get rid of certain classes of tests.
>

Even assuming a set of standards still leaves issues like supporting a
wide range of OS versions which may evolve to support different versions
of the same standard. And then there is the case of standards having
deliberately implementation defined behavior. Last week, I had to double
check what malloc(0) and free(NULL) are supposed to do per POSIX.
Guess what -- implementation defined behavior options.

And worse, POSIX doesn't define the default attributes for pthreads and
the associated synchronization objects. These vary widely across OSes.

Then there are the broken things autoconf checks for. Somehow the
"broken sed from Solaris" check sticks in my head. Or needing to use
GNU sed instead of FreeBSD sed to build a mips embedded target.

And this ignores issues like word size issues. 32 vs 64 bits for one,
Or weird CPUs where nothing is byte addressable (TI C3x/4x is one
example).


> > This seems like a problematic approach that may have been necessary
> > decades ago, but it seems it may be time to move on.
>
> I don't see that.  Many aspects of systems remain non-standardized.
>

Again from pthreads, manipulating affinity on multi-core systems and
having names for pthreads are non-standard but commonly available
with varying APIs.

And the standards have plenty of wiggle room in them. Undefined areas
or deliberately implementation defined.

--joel

>
>
> Ciao,
> Michael.
>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker]

Am Mittwoch, dem 03.04.2024 um 18:02 +0200 schrieb Michael Matz:
> Hello,
> 
> On Wed, 3 Apr 2024, Martin Uecker wrote:
> 
> > The backdoor was hidden in a complicated autoconf script...
> 
> Which itself had multiple layers and could just as well have been a 
> complicated cmake function.

Don't me wrong, cmake is no way better. Another problem was 
actually hidden in a cmake test in upstream git in plain
sight:

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00;hp=af071ef7702debef4f1d324616a0137a5001c14c

> 
> > > (And, FWIW, testing for features isn't "complex".  And have you looked at 
> > > other build systems?  I have, and none of them are less complex, just 
> > > opaque in different ways from make+autotools).
> > 
> > I ask a very specific question: To what extend is testing 
> > for features instead of semantic versions and/or supported
> > standards still necessary?
> 
> I can't answer this with absolute certainty, but points to consider: the 
> semantic versions need to be maintained just as well, in some magic way.  

It would certainly need to be maintained just like now autoconf
configuration needs to be maintained.

> Because ultimately software depend on features of dependencies not on 
> arbitrary numbers given to them.  The numbers encode these features, in 
> the best case, when there are no errors.  So, no, version numbers are not 
> a replacement for feature tests, they are a proxy.  One that is manually 
> maintained, and hence prone to errors.

Tests are also prone to errors and - as the example above shows -
also very fragile and susceptible to manipulation.

> 
> Now, supported standards: which one? ;-)  Or more in earnest: while on 
> this mailing list here we could chose a certain set, POSIX, some 
> languages, Windows, MacOS (versions so-and-so).  What about other 
> software relying on other 3rdparty feature providers (libraries or system 
> services)?  Without standards?
> 
> So, without absolute certainty, but with a little bit of it: yes, feature 
> tests are required in general.  That doesn't mean that we could not 
> do away with quite some of them for (e.g.) GCC, those that hold true on 
> any platform we support.  But we can't get rid of the infrastructure for 
> that, and can't get rid of certain classes of tests.
> 
> > This seems like a problematic approach that may have been necessary 
> > decades ago, but it seems it may be time to move on.
> 
> I don't see that.  Many aspects of systems remain non-standardized.

This is just part of the problem.

Martin

> 
> 
> Ciao,
> Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andreas Schwab]

On Apr 03 2024, Martin Uecker wrote:

> I ask a very specific question: To what extend is testing 
> for features instead of semantic versions and/or supported
> standards still necessary?  This seems like a problematic approach
> that  may have been necessary decades ago, but it seems it may be
> time to move on.

CMake is doing mostly the same feature tests under the hood.  Because
they are proven convenient and effective.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathan Wakely]

On Wed, 3 Apr 2024 at 15:34, Martin Uecker via Gcc <gcc@gcc.gnu.org> wrote:
> I ask a very specific question: To what extend is testing
> for features instead of semantic versions and/or supported
> standards still necessary?  This seems like a problematic approach
> that  may have been necessary decades ago, but it seems it may be
> time to move on.


What standard or version should I check for to detect nl_langinfo_l
support? It's in POSIX 2017 but macOS supports it in <xlocale.h>
despite not being POSIX 2017 compliant.

What about _get_osfhandle? Or fwrite_unlocked?

What about whether the platform ABI allows aligning global objects to
the cacheline size?

Those are just a few of the most recent autoconf checks I've written
for libstdc++ in the past few months, not decades ago.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson]

[-- Attachment #1: Type: text/plain, Size: 3204 bytes --]

Hello all,

On Wed, 2024-04-03 at 16:00 +0200, Michael Matz wrote:
> > My take a way is that software needs to become less complex. Do 
> > we really still need complex build systems such as autoconf?
>
> (And, FWIW, testing for features isn't "complex".  And have you looked at 
> other build systems?  I have, and none of them are less complex, just 
> opaque in different ways from make+autotools).

Some brief opinions from a humble end-user:

I think the key difference here is that Autotools allows arbitrarily generated code to be executed at any time. More modern build systems require the use of specific commands/files to run arbitrary code, e.g. CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
IMHO there are similarities here to the memory "safety" of Rust: Rust code can have memory errors, but it can only come from Rust code declared as `unsafe` (or bugs in the compiler itself). The scope is limited and those scopes can be audited with more powerful microscopes... and removed if/when the build system gains first-class support upstream.

There are other features in the newest build systems listed here (Meson and Cargo) that make this particular attack vector harder. These build systems don't have release tarballs with auxiliary files (e.g. [Meson's is very close to `git archive`][5]), nor do their DSLs allow writing files to the source tree. One could imagine a build/CI worker where the source tree is a read-only bind-mount of a `git archive` extract, that might help defend against attacks of this specific design.

It's also worth noting that Meson and Cargo use non-Turing-complete declarative DSLs for their build configuration. This implies there is an upper bound on the [cyclomatic complexity][6]-per-line of the build script DSL itself. That doesn't mean you can't write complex Meson code (I have), but it ends up being suspiciously long and thus clear something complex and out of the ordinary is being done.

Of course, this doesn't make the build system any less complex, but projects using newer build systems seem easier to secure and audit than those using overly flexible build systems like Autotools and maybe even CMake. IMHO using a late-model build system is a relatively low technical hurdle to overcome for the benefits noted above, switching should be considered and in a positive light.

(For context: my team recently switched our main C/C++ project from Autotools to Meson. The one-time refactor itself was an effort, but after that we got back up to speed quickly and we haven't looked back. Other projects may have an easier time using an unofficial port in the [Meson WrapDB][7] as a starting point.)

-Jonathon

[1]: https://mesonbuild.com/External-commands.html
[2]: https://cmake.org/cmake/help/latest/command/execute_process.html#execute-process
[3]: https://cmake.org/cmake/help/latest/module/ExternalProject.html
[4]: https://doc.rust-lang.org/cargo/reference/build-scripts.html
[5]: https://mesonbuild.com/Creating-releases.html
[6]: https://en.wikipedia.org/wiki/Cyclomatic_complexity
[7]: https://mesonbuild.com/Wrapdb-projects.html

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker]

Am Mittwoch, dem 03.04.2024 um 13:46 -0500 schrieb Jonathon Anderson via Gcc:
> Hello all,
> 
> On Wed, 2024-04-03 at 16:00 +0200, Michael Matz wrote:
> > > My take a way is that software needs to become less complex. Do 
> > > we really still need complex build systems such as autoconf?
> > 
> > (And, FWIW, testing for features isn't "complex".  And have you looked at 
> > other build systems?  I have, and none of them are less complex, just 
> > opaque in different ways from make+autotools).
> 
> Some brief opinions from a humble end-user:
> 
> I think the key difference here is that Autotools allows arbitrarily generated code to be executed at any time. More modern build systems require the use of specific commands/files to run arbitrary code, e.g. CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson ([`run_command()`][1]), Cargo ([`build.rs`][4]).\

To me it seems that Cargo is the absolute worst case with respect to
supply chain attacks.

It pulls in dependencies recursively from a relatively uncurated
list of projects, puts the source of all those dependencies into a 
hidden directory in home, and runs Build.rs automatically with
user permissions.

Martin





> IMHO there are similarities here to the memory "safety" of Rust: Rust code can have memory errors, but it can only come from Rust code declared as `unsafe` (or bugs in the compiler itself). The scope is limited and those scopes can be audited with more powerful microscopes... and removed if/when the build system gains first-class support upstream.
> 
> There are other features in the newest build systems listed here (Meson and Cargo) that make this particular attack vector harder. These build systems don't have release tarballs with auxiliary files (e.g. [Meson's is very close to `git archive`][5]), nor do their DSLs allow writing files to the source tree. One could imagine a build/CI worker where the source tree is a read-only bind-mount of a `git archive` extract, that might help defend against attacks of this specific design.
> 
> It's also worth noting that Meson and Cargo use non-Turing-complete declarative DSLs for their build configuration. This implies there is an upper bound on the [cyclomatic complexity][6]-per-line of the build script DSL itself. That doesn't mean you can't write complex Meson code (I have), but it ends up being suspiciously long and thus clear something complex and out of the ordinary is being done.
> 
> Of course, this doesn't make the build system any less complex, but projects using newer build systems seem easier to secure and audit than those using overly flexible build systems like Autotools and maybe even CMake. IMHO using a late-model build system is a relatively low technical hurdle to overcome for the benefits noted above, switching should be considered and in a positive light.
> 
> (For context: my team recently switched our main C/C++ project from Autotools to Meson. The one-time refactor itself was an effort, but after that we got back up to speed quickly and we haven't looked back. Other projects may have an easier time using an unofficial port in the [Meson WrapDB][7] as a starting point.)
> 
> -Jonathon
> 
> [1]: https://mesonbuild.com/External-commands.html
> [2]: https://cmake.org/cmake/help/latest/command/execute_process.html#execute-process
> [3]: https://cmake.org/cmake/help/latest/module/ExternalProject.html
> [4]: https://doc.rust-lang.org/cargo/reference/build-scripts.html
> [5]: https://mesonbuild.com/Creating-releases.html
> [6]: https://en.wikipedia.org/wiki/Cyclomatic_complexity
> [7]: https://mesonbuild.com/Wrapdb-projects.html

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andrew Sutton]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

>
>
>
> > I think the key difference here is that Autotools allows arbitrarily
> generated code to be executed at any time. More modern build systems
> require the use of specific commands/files to run arbitrary code, e.g.
> CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson
> ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
>
> To me it seems that Cargo is the absolute worst case with respect to
> supply chain attacks.
>
> It pulls in dependencies recursively from a relatively uncurated
> list of projects, puts the source of all those dependencies into a
> hidden directory in home, and runs Build.rs automatically with
> user permissions.
>

100% this. Wait until you learn how proc macros work.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Richard Biener]

On Fri, Apr 5, 2024 at 11:18 PM Andrew Sutton via Gcc <gcc@gcc.gnu.org> wrote:
>
> >
> >
> >
> > > I think the key difference here is that Autotools allows arbitrarily
> > generated code to be executed at any time. More modern build systems
> > require the use of specific commands/files to run arbitrary code, e.g.
> > CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson
> > ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
> >
> > To me it seems that Cargo is the absolute worst case with respect to
> > supply chain attacks.
> >
> > It pulls in dependencies recursively from a relatively uncurated
> > list of projects, puts the source of all those dependencies into a
> > hidden directory in home, and runs Build.rs automatically with
> > user permissions.
> >
>
> 100% this. Wait until you learn how proc macros work.

proc macro execution should be heavily sandboxed, otherwise it seems
compiling something is enough to get arbitrary code executed with the
permission of the compiling user.  I mean it's not rocket science - browsers
do this for javascript.  Hmm, we need a webassembly target ;)

Richard.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Martin Uecker]

Am Samstag, dem 06.04.2024 um 15:00 +0200 schrieb Richard Biener:
> On Fri, Apr 5, 2024 at 11:18 PM Andrew Sutton via Gcc <gcc@gcc.gnu.org> wrote:
> > 
> > > 
> > > 
> > > 
> > > > I think the key difference here is that Autotools allows arbitrarily
> > > generated code to be executed at any time. More modern build systems
> > > require the use of specific commands/files to run arbitrary code, e.g.
> > > CMake (IIRC [`execute_process()`][2] and [`ExternalProject`][3]), Meson
> > > ([`run_command()`][1]), Cargo ([`build.rs`][4]).\
> > > 
> > > To me it seems that Cargo is the absolute worst case with respect to
> > > supply chain attacks.
> > > 
> > > It pulls in dependencies recursively from a relatively uncurated
> > > list of projects, puts the source of all those dependencies into a
> > > hidden directory in home, and runs Build.rs automatically with
> > > user permissions.
> > > 
> > 
> > 100% this. Wait until you learn how proc macros work.
> 
> proc macro execution should be heavily sandboxed, otherwise it seems
> compiling something is enough to get arbitrary code executed with the
> permission of the compiling user.  I mean it's not rocket science - browsers
> do this for javascript.  Hmm, we need a webassembly target ;)

This would be useful anyhow. 

And locking down the compiler using landlock to only access specified
files / directories would also be nice in general.

Martin

Re: Sourceware mitigating and preventing the next xz-backdoor

[Michael Matz]

Hello,

On Wed, 3 Apr 2024, Jonathon Anderson wrote:

> Of course, this doesn't make the build system any less complex, but 
> projects using newer build systems seem easier to secure and audit than 
> those using overly flexible build systems like Autotools and maybe even 
> CMake. IMHO using a late-model build system is a relatively low 
> technical hurdle to overcome for the benefits noted above, switching 
> should be considered and in a positive light.

Note that we're talking not (only) about the build system itself, i.e. how 
to declare dependencies within the sources, and how to declare how to 
build them.  make it just fine for that (as are many others).  (In a way 
I think we meanwhile wouldn't really need automake and autogen, but 
rewriting all that in pure GNUmake is a major undertaking).

But Martin also specifically asked about alternatives for feature tests, 
i.e. autoconfs purpose.  I simply don't see how any alternative to it 
could be majorly "easier" or "less complex" at its core.  Going with the 
examples given upthread there is usually only one major solution: to check 
if a given system supports FOOBAR you need to bite the bullet and compile 
(and potentially run!) a small program using FOOBAR.  A configuration 
system that can do that (and I don't see any real alternative to that), no 
matter in which language it's written and how traditional or modern it is, 
also gives you enough rope to hang yourself, if you so choose.

If you get away without many configuration tests in your project then this 
is because what (e.g.) the compiler gives you, in the form of libstdc++ 
for example, abstracts away many of the peculiarities of a system.  But 
in order to be able to do that something (namely the config system of 
libstdc++) needs to determine what is or isn't supported by the system in 
order to correctly implement these abstractions.  I.e. things you depend 
on did the major lifting of hiding system divergence.

(Well, that, or you are very limited in the number of systems you support, 
which can be the right thing as well!)


Ciao,
Michael.

Re: Sourceware mitigating and preventing the next xz-backdoor

[anderson.jonathonm]

[-- Attachment #1: Type: text/plain, Size: 7168 bytes --]

Hello,

On Thu, Apr 4, 2024, 09:00 Michael Matz <matz@suse.de> wrote:

> Hello,
>
> On Wed, 3 Apr 2024, Jonathon Anderson wrote:
>
> > Of course, this doesn't make the build system any less complex, but 
> > projects using newer build systems seem easier to secure and audit than 
> > those using overly flexible build systems like Autotools and maybe even 
> > CMake. IMHO using a late-model build system is a relatively low 
> > technical hurdle to overcome for the benefits noted above, switching 
> > should be considered and in a positive light.
>
> Note that we're talking not (only) about the build system itself, i.e. how 
> to declare dependencies within the sources, and how to declare how to 
> build them. [...]
>
> But Martin also specifically asked about alternatives for feature tests, 
> i.e. autoconfs purpose.  I simply don't see how any alternative to it 
> could be majorly "easier" or "less complex" at its core.  

My point was not that newer build systems are any less complex taken wholistically. The upthread has already discussed why configuration tests are necessary and aren't going away anytime soon. My horror stories would not add much to that conversation.

My point, or rather my opinion as a humble end-user, is that newer build systems (e.g. Meson) make identifying the xz backdoor and injections following the same pattern much easier than older build systems (e.g. autoconf+automake+libtool). Paraphrasing my original message to highlight the backdoor protections:

- This xz backdoor injection unpacked attacker-controlled files and ran them during `configure`. Newer build systems implement a build abstraction (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only code run during `meson setup` is from `meson.build` files and CMake). Generally speaking the only way to disobey those rules is via an "escape" command (e.g. `run_command()`) of which there are few. This reduces the task of auditing the build scripts for sandbox-breaking malicious intent significantly, only the "escapes" need investigation and they which should(tm) be rare for well-behaved projects.

- This xz backdoor injected worked due to altered versions of the build system implementation being shipped with the source. Newer build systems do not ship any part of the build system's implementation with the project's source, it somes as a separate package and the most the project does is specify a compatible version range. This removes the possibility of "hidden" code being mixed to the shipped source tarball, well-behaved projects will have source tarballs that can be byte-for-byte reproduced from the VCS (e.g. Git).

- This xz backdoor injected the payload code by altering the files in the source directory. Newer build systems do not allow altering the source directory (for good reason) in their build abstraction. For build workers/CI, this restriction can be enforced using low-level sandboxing (e.g. conainers or landlock) and well-behaved projects should(tm) be unaffected. This reduces the possibilities on how code can appear in the project, only code generation sequences (e.g. `custom_target()` or `generator()`) can produce arbitrary code.

Just by transitioning to a newer build system (and being "well-behaved") it seems like you get certain protections almost for free. And IMHO compared to other options to improve project security (e.g. automated fuzzing), transitioning build systems is a medium-effort, low-maintainence option with a comparatively high yield.

> make is just fine for that (as are many others).  (In a way 
> I think we meanwhile wouldn't really need automake and autogen, but 
> rewriting all that in pure GNUmake is a major undertaking).

Makefile (the language) is extremely complex, it is nigh impossible to ensure something won't expand to some malicious code injection when the right file arrangement is present. (IMHO Make also is not a great build system, it has none of the protections described above... and it's [just plain slow](https://david.rothlis.net/ninja-benchmark/).)

Newer build systems seem to be moving in a direction where the bulk of the build configuration/scripts are non-Turing-complete. This is a highly effective method for clarifying the high-level complexity of the DSL: simple things are short and sweet, complex things (and discouraged solutions) are long and complex. Less complex high-level build logic makes it easier to verify and audit for security (and other) purposes.

> Going with the 
> examples given upthread there is usually only one major solution: to check 
> if a given system supports FOOBAR you need to bite the bullet and compile 
> (and potentially run!) a small program using FOOBAR.  A configuration 
> system that can do that (and I don't see any real alternative to that), no 
> matter in which language it's written and how traditional or modern it is, 
> also gives you enough rope to hang yourself, if you so choose.

Both very true, as discussed upthread.

It may be a helpful policy to use templates for configure tests where possible. For example the [now-famous XZ landlock configure test (that contained a rouge syntax error)](https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7) could be written via Meson template configure tests as:

    cc = meson.get_compiler('c')
    if (
        cc.has_function('prctl', prefix: '#include <sys/prctl.h>')
        and cc.has_header_symbol('sys/syscall.h', 'SYS_landlock_create_ruleset', prefix: '#include <linux/landlock.h>')
        and cc.has_header_symbol('sys/syscall.h', 'SYS_landlock_restrict', prefix: '#include <linux/landlock.h>')
        and cc.has_header_symbol('linux/landlock.h', 'LANDLOCK_CREATE_RULESET_VERSION')
    )
    # ...
    endif

The templates are well-tested as part of the build system, so if the `prefix:` has no syntax error (and the spelling is right) I'm confident these checks are sane. And IMHO this expresses what is being checked much better than an actual snippet of C/C++ code.

>
> If you get away without many configuration tests in your project then this 
> is because what (e.g.) the compiler gives you, in the form of libstdc++ 
> for example, abstracts away many of the peculiarities of a system.  But 
> in order to be able to do that something (namely the config system of 
> libstdc++) needs to determine what is or isn't supported by the system in 
> order to correctly implement these abstractions.  I.e. things you depend 
> on did the major lifting of hiding system divergence.
>
> (Well, that, or you are very limited in the number of systems you support, 
> which can be the right thing as well!)

We do get away with less than a dozen configuration tests, in part because we are fairly limited in what we support (GNU/Linux) and we use the C, C++ and POSIX standards quite heavily. But we're also an end-user application with a heavy preference towards bleeding-edge systems, not a low-level library that can run on 1980-something technology, so our experience might be a bit... unique. ;)

Thanks,\
-Jonathon

Re: Sourceware mitigating and preventing the next xz-backdoor

[Andreas Schwab]

On Apr 09 2024, anderson.jonathonm@gmail.com wrote:

> - This xz backdoor injection unpacked attacker-controlled files and ran them during `configure`. Newer build systems implement a build abstraction (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only code run during `meson setup` is from `meson.build` files and CMake). Generally speaking the only way to disobey those rules is via an "escape" command (e.g. `run_command()`) of which there are few. This reduces the task of auditing the build scripts for sandbox-breaking malicious intent significantly, only the "escapes" need investigation and they which should(tm) be rare for well-behaved projects.

Just like you can put your backdoor in *.m4 files, you can put them in
*.cmake files.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

On Tue, Apr 9, 2024, 10:57 Andreas Schwab <schwab@linux-m68k.org> wrote:

> On Apr 09 2024, anderson.jonathonm@gmail.com wrote:
>
> > - This xz backdoor injection unpacked attacker-controlled files and ran
> them during `configure`. Newer build systems implement a build abstraction
> (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only
> code run during `meson setup` is from `meson.build` files and CMake).
> Generally speaking the only way to disobey those rules is via an "escape"
> command (e.g. `run_command()`) of which there are few. This reduces the
> task of auditing the build scripts for sandbox-breaking malicious intent
> significantly, only the "escapes" need investigation and they which
> should(tm) be rare for well-behaved projects.
>
> Just like you can put your backdoor in *.m4 files, you can put them in
> *.cmake files.


CMake has its own sandbox and rules and escapes (granted, much more of
them). But regardless, the injection code would be committed to the
repository (point 2) and would not hold up to a source directory mounted
read-only (point 3).

If your build system is Meson, you can easily consider CMake code to be an
escape and give it a little more auditing attention. Or just avoid shipping
CMake scripts entirely, they are are rarely necessary.

-Jonathon

>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning]

> On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc <gcc@gcc.gnu.org> wrote:
> 
> On Tue, Apr 9, 2024, 10:57 Andreas Schwab <schwab@linux-m68k.org> wrote:
> 
>> On Apr 09 2024, anderson.jonathonm@gmail.com wrote:
>> 
>>> - This xz backdoor injection unpacked attacker-controlled files and ran
>> them during `configure`. Newer build systems implement a build abstraction
>> (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only
>> code run during `meson setup` is from `meson.build` files and CMake).
>> Generally speaking the only way to disobey those rules is via an "escape"
>> command (e.g. `run_command()`) of which there are few. This reduces the
>> task of auditing the build scripts for sandbox-breaking malicious intent
>> significantly, only the "escapes" need investigation and they which
>> should(tm) be rare for well-behaved projects.
>> 
>> Just like you can put your backdoor in *.m4 files, you can put them in
>> *.cmake files.
> 
> 
> CMake has its own sandbox and rules and escapes (granted, much more of
> them). But regardless, the injection code would be committed to the
> repository (point 2) and would not hold up to a source directory mounted
> read-only (point 3).

Why would the injection code necessarily be committed to the repository?  It wasn't in the xz attack -- one hole in the procedures is that the kits didn't match the repository and no checks caught this.  I don't see how a different build system would cure that issue.  Instead, there needs to be some sort of audit that verifies there aren't rogue or modified elements in the kit.

	paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton]

On Tue, Apr 9, 2024 at 4:11 PM Paul Koning via Gdb <gdb@sourceware.org> wrote:
>
> > On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > On Tue, Apr 9, 2024, 10:57 Andreas Schwab <schwab@linux-m68k.org> wrote:
> >
> >> On Apr 09 2024, anderson.jonathonm@gmail.com wrote:
> >>
> >>> - This xz backdoor injection unpacked attacker-controlled files and ran
> >> them during `configure`. Newer build systems implement a build abstraction
> >> (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only
> >> code run during `meson setup` is from `meson.build` files and CMake).
> >> Generally speaking the only way to disobey those rules is via an "escape"
> >> command (e.g. `run_command()`) of which there are few. This reduces the
> >> task of auditing the build scripts for sandbox-breaking malicious intent
> >> significantly, only the "escapes" need investigation and they which
> >> should(tm) be rare for well-behaved projects.
> >>
> >> Just like you can put your backdoor in *.m4 files, you can put them in
> >> *.cmake files.
> >
> >
> > CMake has its own sandbox and rules and escapes (granted, much more of
> > them). But regardless, the injection code would be committed to the
> > repository (point 2) and would not hold up to a source directory mounted
> > read-only (point 3).
>
> Why would the injection code necessarily be committed to the repository?  It wasn't in the xz attack -- one hole in the procedures is that the kits didn't match the repository and no checks caught this.

In the Xz case, the victims should not have been able to execute a
well known but modified *.m4 file. The *.m4 files should have been on
the build machine already; or, the latest *.m4 files installed on the
build machine via an updated Autoconf; or the latest serial-numbered
*.m4 macro should have been fetched from a trusted source like GNU.
They provide the provenance needed to allow their execution.

In the browser world, content security policy (CSP) allows third party
code to execute, assuming a script-src and an integrity tag. The
provenance and integrity is enforced.

> I don't see how a different build system would cure that issue.  Instead, there needs to be some sort of audit that verifies there aren't rogue or modified elements in the kit.

Code provenance and code integrity was not enforced. Part of the
problem is the Autotools design. It is from a bygone era. No one
should be able to override a named, GNU supplied m4 macro. That's a
bug, not a feature.

Jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/9/24 14:40, Jeffrey Walton wrote:
> Code provenance and code integrity was not enforced. Part of the
> problem is the Autotools design. It is from a bygone era.

No, Andreas is right. This isn't an Autotools-vs-Meson thing.

Most of the Autotools-based projects I help maintain would have been 
immune to this particular exploit, partly because they don't maintain 
their own of Gnulib .m4 files. Conversely, any Meson-based project that 
had the same sort of out-of-repository sloppiness and lack of review 
that xz had, would be vulnerable to similar attacks.

> No one should be able to override a named, GNU supplied m4 macro.

That ship sailed long ago, for Autoconf and for Meson and for every 
other widely-available build tool I know of. Everyone can write and run 
their own code, whether it comes from GNU or not. That's a feature that 
developers want and need. Although this feature can be misused, it's not 
a bug per se.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/9/24 14:40, Jeffrey Walton wrote:
>> Code provenance and code integrity was not enforced. Part of the
>> problem is the Autotools design. It is from a bygone era.
>
> No, Andreas is right. This isn't an Autotools-vs-Meson thing.
>
> Most of the Autotools-based projects I help maintain would have been
> immune to this particular exploit, partly because they don't maintain
> their own of Gnulib .m4 files. Conversely, any Meson-based project
> that had the same sort of out-of-repository sloppiness and lack of
> review that xz had, would be vulnerable to similar attacks.
>

While it could indeed happen via a variety of methods, it's still worth
talking about how to reduce places for it to hide in, and make review
easier, I think.

>> No one should be able to override a named, GNU supplied m4 macro.
>
> That ship sailed long ago, for Autoconf and for Meson and for every
> other widely-available build tool I know of. Everyone can write and
> run their own code, whether it comes from GNU or not. That's a feature
> that developers want and need. Although this feature can be misused,
> it's not a bug per se.

Meson doesn't allow user-defined functions and macros to avoid
metaprogramming hell:
https://mesonbuild.com/FAQ.html#why-doesnt-meson-have-user-defined-functionsmacros.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/9/24 14:58, Sam James wrote:
> Meson doesn't allow user-defined functions

Meson has ways to execute arbitrary user-defined code, so it's not 
immune to this sort of exploit.

It's of course better (all other things being equal) to use a build 
system with a smaller attack surface. However, any surface of nonzero 
size is attackable, so I'm not convinced that Meson is significantly 
safer against a determined insider. Although the xz exploit was tricky 
and is now famous (hey! the front page of the New York Times!) 
fundamentally it was sloppy and amateurish and it succeeded only because 
xz's project management was even sloppier.

Yes, we need to defend against amateurish attacks. But we shouldn't 
waste valuable developer time on defenses that won't work against 
obvious future attacks and that will likely cost more than they'll 
benefit. That's just security theater.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/9/24 14:58, Sam James wrote:
>> Meson doesn't allow user-defined functions
>
> Meson has ways to execute arbitrary user-defined code, so it's not
> immune to this sort of exploit.

To be clear - not saying it's immune. Just that it scopes the
user-defined code part to clearly defined sections.

I think it makes sense to optimise for ease of review.

>
> It's of course better (all other things being equal) to use a build
> system with a smaller attack surface. However, any surface of nonzero
> size is attackable, so I'm not convinced that Meson is significantly
> safer against a determined insider. Although the xz exploit was tricky
> and is now famous (hey! the front page of the New York Times!)
> fundamentally it was sloppy and amateurish and it succeeded only
> because xz's project management was even sloppier.
>
> Yes, we need to defend against amateurish attacks. But we shouldn't
> waste valuable developer time on defenses that won't work against
> obvious future attacks and that will likely cost more than they'll
> benefit. That's just security theater.

Right, I'm not advocating that. It's just easy to go too far the other
way too and not change anything because it won't hold up against a state
actor.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Eggert]

On 4/9/24 15:22, Sam James wrote:
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
>> On 4/9/24 14:58, Sam James wrote:
>>> Meson doesn't allow user-defined functions
>> Meson has ways to execute arbitrary user-defined code, so it's not
>> immune to this sort of exploit.
> To be clear - not saying it's immune.

Sure, but someone who's not expert in Meson could easily misread "Meson 
doesn't allow user-defined functions" and think that this means Meson is 
immune to an xz-style attack, which it's not.

> Just that it scopes the
> user-defined code part to clearly defined sections.

As does Autoconf. To a determined attacker I daresay there's not much 
difference.

> I think it makes sense to optimise for ease of review.

Ease of review definitely a good thing, all other things being equal.

> It's just easy to go too far the other
> way too and not change anything

I'm certainly not advocating that! All I'm saying is that we should use 
our limited development resources wisely.

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

On Tue, 2024-04-09 at 14:50 -0700, Paul Eggert wrote:
> On 4/9/24 14:40, Jeffrey Walton wrote:
> 
> > Code provenance and code integrity was not enforced. Part of the
> > problem is the Autotools design. It is from a bygone era.
> 
> 
> No, Andreas is right. This isn't an Autotools-vs-Meson thing.
> 
> Most of the Autotools-based projects I help maintain would have been  
> immune to this particular exploit, partly because they don't maintain  
> their own of Gnulib .m4 files. Conversely, any Meson-based project that  
> had the same sort of out-of-repository sloppiness and lack of review  
> that xz had, would be vulnerable to similar attacks.

Xz doesn't either, the exploit was unique to the distributed `make dist` tarballs. Which is an Autotools quirk present in all Autotools projects.

I won't deny that a project could use Meson and be sloppy, a project could use SSL/TLS/whatever and be completely insecure. But Autotools encourages and semi-requires this sloppy behavior, and CMake and Meson strongly discourage this behavior.

-Jonathon

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

Jonathon Anderson <anderson.jonathonm@gmail.com> writes:

> On Tue, 2024-04-09 at 14:50 -0700, Paul Eggert wrote:
>
>  On 4/9/24 14:40, Jeffrey Walton wrote:
>
>  Code provenance and code integrity was not enforced. Part of the problem is the Autotools design. It is from a
>  bygone era.
>
>  No, Andreas is right. This isn't an Autotools-vs-Meson thing.
>
>  Most of the Autotools-based projects I help maintain would have been
>  immune to this particular exploit, partly because they don't maintain
>  their own of Gnulib .m4 files. Conversely, any Meson-based project that
>  had the same sort of out-of-repository sloppiness and lack of review
>  that xz had, would be vulnerable to similar attacks.
>
> Xz doesn't either, the exploit was unique to the distributed make dist tarballs. Which is an Autotools quirk present in
> all Autotools projects.
>
> I won't deny that a project could use Meson and be sloppy, a project could use SSL/TLS/whatever and be completely
> insecure. But Autotools encourages and semi-requires this sloppy behavior, and CMake and Meson strongly discourage this
> behavior.

Indeed. Talking about things in the context of "how can we make it
easier to spot" is a good thing. Obviously if we're trying to resist a
state actor, things are very hard. It doesn't mean don't bother.

>
> -Jonathon

thanks,
sam

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]

On Tue, 2024-04-09 at 16:11 -0400, Paul Koning wrote:
>
> On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc <[gcc@gcc.gnu.org](mailto:gcc@gcc.gnu.org)> wrote:
>
> > CMake has its own sandbox and rules and escapes (granted, much more of
> > them). But regardless, the injection code would be committed to the
> > repository (point 2) and would not hold up to a source directory mounted
> > read-only (point 3).
> 
> Why would the injection code necessarily be committed to the repository?  It wasn't in the xz attack -- one hole in the procedures is that the kits didn't match the repository and no checks caught this.  I don't see how a different build system would cure that issue.  Instead, there needs to be some sort of audit that verifies there aren't rogue or modified elements in the kit.

In Autotools, `make dist` produces a tarball that contains many files not present in the source respoitory, it includes build system core files and this fact was used for the xz attack. In contrast, for newer build systems the "release tarball" is purely a snapshot of the source repository: there is no `cmake dist`, and `meson dist` is essentially `git archive` ([docs](https://mesonbuild.com/Creating-releases.html)). Thus for the injection code to be present in the release tarball, it needs to have first been checked into the repository.

In fact, packagers don't *need* to use the tarballs, they can (and should) use the Git history from the source repository itself. In Debian this is one workflow implemented by the popular git-buildpackage ([docs](https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.upstream-git.html)). The third-party package manager [Spack](https://spack.readthedocs.io/en/latest/packaging_guide.html#git) clones directly from the source repository. Others may have support for this as well, this isn't a novel idea.

-Jonathon

Re: Sourceware mitigating and preventing the next xz-backdoor

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 2201 bytes --]

Jonathon Anderson via Gdb <gdb@sourceware.org> writes:

> On Tue, 2024-04-09 at 16:11 -0400, Paul Koning wrote:
>>
>> On Apr 9, 2024, at 3:59 PM, Jonathon Anderson via Gcc <[gcc@gcc.gnu.org](mailto:gcc@gcc.gnu.org)> wrote:
>>
>> > CMake has its own sandbox and rules and escapes (granted, much more of
>> > them). But regardless, the injection code would be committed to the
>> > repository (point 2) and would not hold up to a source directory mounted
>> > read-only (point 3).
>> 
>> Why would the injection code necessarily be committed to the
>> repository?  It wasn't in the xz attack -- one hole in the
>> procedures is that the kits didn't match the repository and no
>> checks caught this.  I don't see how a different build system would
>> cure that issue.  Instead, there needs to be some sort of audit that
>> verifies there aren't rogue or modified elements in the kit.
>
> In Autotools, `make dist` produces a tarball that contains many files
> not present in the source respoitory, it includes build system core
> files and this fact was used for the xz attack. In contrast, for newer
> build systems the "release tarball" is purely a snapshot of the source
> repository: there is no `cmake dist`, and `meson dist` is essentially
> `git archive`
> ([docs](https://mesonbuild.com/Creating-releases.html)). Thus for the
> injection code to be present in the release tarball, it needs to have
> first been checked into the repository.

(Of course, one could modify it after, but the point here is that it's by
design reproducible so any differences are suspicious, just to be
clear.)

>
> In fact, packagers don't *need* to use the tarballs, they can (and
> should) use the Git history from the source repository itself. In
> Debian this is one workflow implemented by the popular
> git-buildpackage
> ([docs](https://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.upstream-git.html)). The
> third-party package manager
> [Spack](https://spack.readthedocs.io/en/latest/packaging_guide.html#git)
> clones directly from the source repository. Others may have support
> for this as well, this isn't a novel idea.
>
> -Jonathon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Frank Ch. Eigler]

Hi -

> In Autotools, `make dist` produces a tarball that contains many
> files not present in the source respoitory, it includes build system
> core files and this fact was used for the xz attack. In contrast,
> for newer build systems the "release tarball" is purely a snapshot
> of the source repository: there is no `cmake dist`, and `meson dist`
> is essentially `git archive` [...]

For what it's worth, not every auto* using project uses "make dist" to
build their release tarballs.  If they can get over the matter of
including auto*-generated scripts being located in the source repo,
then indeed a "git archive" is sufficient.  Several of the projects I
work on do just this.  (As a bonus, that makes the git repos immediately
buildable by developers, without need to re-auto* anything.)

- FChE

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jonathon Anderson]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

On Wed, Apr 10, 2024, 07:09 Frank Ch. Eigler <fche@elastic.org> wrote:

> Hi -
>
> > In Autotools, `make dist` produces a tarball that contains many
> > files not present in the source respoitory, it includes build system
> > core files and this fact was used for the xz attack. In contrast,
> > for newer build systems the "release tarball" is purely a snapshot
> > of the source repository: there is no `cmake dist`, and `meson dist`
> > is essentially `git archive` [...]
>
> For what it's worth, not every auto* using project uses "make dist" to
> build their release tarballs.  If they can get over the matter of
> including auto*-generated scripts being located in the source repo,
> then indeed a "git archive" is sufficient.


This is very true, however a few words of caution: IME this is a
maintainability nightmare. Fixing patches that forgot to regenerate,
regenerating on rebase, confirming everything is up-to-date before merge,
etc etc. It can be handled, I have, but it was painful and
time-consuming.The hardest part was ensuring everyone was actually running
the "right" version of Auto*. (
Did you know Debian ships a different version of the *.m4? That caused more
than a few hours lost to confusion:
https://sources.debian.org/src/autoconf/2.72-2/debian/patches/add-runstatedir.patch
)

To make matters worse, this behavior adds a lot of near-duplicate code and
large unreadable changes to patches. For my team that meant we didn't often
read the generated parts of patches with build system changes, and
definitely not close enough to detect any malicious injections. Which
should make everyone here squeamish given the recent xz attack.

Thanks,
-Jonathon

>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Frank Ch. Eigler]

Hi -

>    This is very true, however a few words of caution: IME this is a
>    maintainability nightmare. Fixing patches that forgot to regenerate,
>    regenerating on rebase, confirming everything is up-to-date before
>    merge, etc etc. It can be handled, I have, but it was painful and
>    time-consuming.The hardest part was ensuring everyone was actually
>    running the "right" version of Auto* [...]

One way to make the nightmare into a light hassle is to let developers
commit auto* hand-written inputs with or without Complete Properly
refreshed generated bits, and let a maintainer or bot (but I repeat
myself) periodically regenerate the derived auto* content.

- FChE

Re: Sourceware mitigating and preventing the next xz-backdoor

[Claudio Bantaloukas]

On 09/04/2024 18:57, Andreas Schwab wrote:
> On Apr 09 2024, anderson.jonathonm@gmail.com wrote:
> 
>> - This xz backdoor injection unpacked attacker-controlled files and ran them during `configure`. Newer build systems implement a build abstraction (aka DSL) that acts similar to a sandbox and enforces rules (e.g. the only code run during `meson setup` is from `meson.build` files and CMake). Generally speaking the only way to disobey those rules is via an "escape" command (e.g. `run_command()`) of which there are few. This reduces the task of auditing the build scripts for sandbox-breaking malicious intent significantly, only the "escapes" need investigation and they which should(tm) be rare for well-behaved projects.
> 
> Just like you can put your backdoor in *.m4 files, you can put them in
> *.cmake files.
>

Hi Andreas,
Indeed you're right and seeing the hijacked CMakeLists.txt in the commit 
was eye opening.

There is a not so subtle difference though. The amount of nasty that the 
attacker thought could get away with was different between the 
build-to-host.m4 and the CMakeLists.txt attack vectors.

For the CMakeLists.txt file, the wanted change was very small, adding a 
dot to a piece of c code so that the test that runs it goes into one of 
the perfectly acceptable states (cannot compile the c file), thus 
disabling a security feature.
This change was "hidden" in a patch containing a bunch of pointless 
renames and veiled in plausible deniability (oops, that dot went in the 
by mistake, silly me, here's a patch to fix the file). The attacker was 
lucky because noone really checked.

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7;hp=eb8ad59e9bab32a8d655796afd39597ea6dcc64d

Compare that to what the m4 file did. Russ Cox has an interesting 
analysis https://research.swtch.com/xz-script

 From which I'll pick a choice quote: "makes a handful of inscrutable 
changes that don’t look terribly out of place".

I figured out the problem with the CMakeFile.txt quickly. I'm not 100% 
sure if the configure.ac is ok (after looking at it for 10 minutes, it 
looks ok, but I'm not sure!). I would not be able to recognise good code 
from bad in the m4 file.

Admittedly, I'm biased in favour of cmake's DSL, have more experience 
with it despite using ./configure since the mid 90s and have a 
preference. But it would be hard to argue against the fact that benign 
m4, as practiced in the wild today is hard to separate from malicious m4 
by a majority of developers, including experienced ones like Mr. Cox above.

Cheers,
Claudio Bantaloukas

Re: Sourceware mitigating and preventing the next xz-backdoor

[Guinevere Larsen]

On 4/2/24 16:54, Sandra Loosemore wrote:
> On 4/1/24 09:06, Mark Wielaard wrote:
>> A big thanks to everybody working this long Easter weekend who helped
>> analyze the xz-backdoor and making sure the impact on Sourceware and
>> the hosted projects was minimal.
>>
>> This email isn't about the xz-backdoor itself. Do see Sam James FAQ
>> https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
>> (Sorry for the github link, but this one does seem viewable without
>> proprietary javascript)
>>
>> We should discuss what we have been doing and should do more to
>> mitigate and prevent the next xz-backdoor. There are a couple of
>> Sourceware services that can help with that.
>>
>> TLDR;
>> - Replicatable isolated container/VMs are nice, we want more.
>> - autoregen buildbots, it should be transparent (and automated) how to
>>    regenerate build/source files.
>> - Automate (snapshot) releases tarballs.
>> - Reproducible releases (from git).
>>
>> [snip]
>
> While I appreciate the effort to harden the Sourceware infrastructure 
> against malicious attacks and want to join in on thanking everyone who 
> helped analyze this issue, to me it seems like the much bigger problem 
> is that XZ had a maintainer who appears to have acted in bad faith.  
> Are the development processes used by the GNU toolchain components 
> robust enough to cope with deliberate sabotage of the code base?  Do 
> we have enough eyes available to ensure that every commit, even those 
> by designated maintainers, is vetted by someone else?  Do we to harden 
> our process, too, to require all patches to be signed off by someone 
> else before committing?
>
> -Sandra
>
>
What likely happened for the maintainer who acted in bad faith was that 
they entered the project with bad faith intent from the start - seeing 
as they were only involved with the project for 2 years, and there was 
much social pressure from fake email accounts for the single maintainer 
of XZ to accept help.

While we would obviously like to have more area maintainers and possibly 
global maintainers to help spread the load, I don't think any of the 
projects listed here are all that susceptible to the same type of social 
engineering. For one, getting the same type of blanket approval would be 
a much more involved process because we already have a reasonable amount 
of people with those privileges, no one is dealing with burnout and 
sassy customers saying we aren't doing enough.

Beyond that, we (GDB) are already experimenting with approved-by, and I 
think glibc was doing the same. That guarantees at least a second set of 
eyes that analyzed and agreed with the patch, I don't think signed-off 
would add more than that tag (even if security was not the reason why we 
implemented them).

-- 
Cheers,
Guinevere Larsen
She/Her/Hers

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton]

On Tue, Apr 2, 2024 at 6:09 PM Guinevere Larsen via Gdb
<gdb@sourceware.org> wrote:
> [...]
> What likely happened for the maintainer who acted in bad faith was that
> they entered the project with bad faith intent from the start - seeing
> as they were only involved with the project for 2 years, and there was
> much social pressure from fake email accounts for the single maintainer
> of XZ to accept help.

The infiltration appears to have started offline, earlier than June
2022. See <https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html>.

> While we would obviously like to have more area maintainers and possibly
> global maintainers to help spread the load, I don't think any of the
> projects listed here are all that susceptible to the same type of social
> engineering. For one, getting the same type of blanket approval would be
> a much more involved process because we already have a reasonable amount
> of people with those privileges, no one is dealing with burnout and
> sassy customers saying we aren't doing enough.
>
> Beyond that, we (GDB) are already experimenting with approved-by, and I
> think glibc was doing the same. That guarantees at least a second set of
> eyes that analyzed and agreed with the patch, I don't think signed-off
> would add more than that tag (even if security was not the reason why we
> implemented them).

Jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi,

On Tue, Apr 02, 2024 at 07:08:59PM -0300, Guinevere Larsen wrote:
> On 4/2/24 16:54, Sandra Loosemore wrote:
> >On 4/1/24 09:06, Mark Wielaard wrote:
> >>
> >>We should discuss what we have been doing and should do more to
> >>mitigate and prevent the next xz-backdoor. There are a couple of
> >>Sourceware services that can help with that.
> >>
> >>TLDR;
> >>- Replicatable isolated container/VMs are nice, we want more.
> >>- autoregen buildbots, it should be transparent (and automated) how to
> >>   regenerate build/source files.
> >>- Automate (snapshot) releases tarballs.
> >>- Reproducible releases (from git).
> >>
> >>[snip]
> >
> >While I appreciate the effort to harden the Sourceware
> >infrastructure against malicious attacks and want to join in on
> >thanking everyone who helped analyze this issue, to me it seems
> >like the much bigger problem is that XZ had a maintainer who
> >appears to have acted in bad faith.  Are the development processes
> >used by the GNU toolchain components robust enough to cope with
> >deliberate sabotage of the code base?  Do we have enough eyes
> >available to ensure that every commit, even those by designated
> >maintainers, is vetted by someone else?  Do we to harden our
> >process, too, to require all patches to be signed off by someone
> >else before committing?
>
> [...]
> 
> Beyond that, we (GDB) are already experimenting with approved-by,
> and I think glibc was doing the same. That guarantees at least a
> second set of eyes that analyzed and agreed with the patch, I don't
> think signed-off would add more than that tag (even if security was
> not the reason why we implemented them).

Yes, I agree having a policy of requiring all patches to be reviewed
and showing that by adding tags like Approved-By is a good thing. And
I think the gdb approach is simple and sane:
https://sourceware.org/gdb/wiki/ContributionChecklist#Receiving_positive_reviews

Like all the other suggestions it wouldn't really prevent all
attacks. But you shouldn't do this just for "security" anyway. I would
even say that if you do something only to get a security "badge" then
you are doing it wrong.

The "hardening" projects above were also not just for "security" but
also because they are good and sane things to do anyway. We try to put
everything in separate containters or isolated VMs also because that
just makes maintenance easier, helps migrating workloads, shows users
how setup the services locally. Having automated reproducible
snapshots and releases also helps having clear procedures for
regenerating files, makes (pre-commit) CI more reliable, helps
bootstrappability.

The reason I only listed the above four "hardening" steps was because
they have clear Sourceware infrastructure services that could help and
that we are working on already anyway. Having a Reviewed-By policy is
really "just" a project policy that doesn't require any infrastructure
help. Although we could have a git push hook that makes it mandatory.

You could however go further and use signed commits or pushes, Both of
which Sourceware infrastructue supports:
https://inbox.sourceware.org/ZJ3Tihvu6GbOb8%2FR@elastic.org/

Or if the project wants to we could migrate them to gitolite and/or
install strict hooks for who may push to (release) branches.

Just let us know. The Sourceware Infrastructure is ready, but most of
these issues are really project policy.

Cheers,

Mark

Re: Sourceware mitigating and preventing the next xz-backdoor

[Paul Koning]

> On Apr 2, 2024, at 6:08 PM, Guinevere Larsen <blarsen@redhat.com> wrote:
> 
> On 4/2/24 16:54, Sandra Loosemore wrote:
>> On 4/1/24 09:06, Mark Wielaard wrote:
>>> A big thanks to everybody working this long Easter weekend who helped
>>> analyze the xz-backdoor and making sure the impact on Sourceware and
>>> the hosted projects was minimal.
>>> 
>>> This email isn't about the xz-backdoor itself. Do see Sam James FAQ
>>> https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
>>> (Sorry for the github link, but this one does seem viewable without
>>> proprietary javascript)
>>> 
>>> We should discuss what we have been doing and should do more to
>>> mitigate and prevent the next xz-backdoor. There are a couple of
>>> Sourceware services that can help with that.
>>> 
>>> TLDR;
>>> - Replicatable isolated container/VMs are nice, we want more.
>>> - autoregen buildbots, it should be transparent (and automated) how to
>>>    regenerate build/source files.
>>> - Automate (snapshot) releases tarballs.
>>> - Reproducible releases (from git).
>>> 
>>> [snip]
>> 
>> While I appreciate the effort to harden the Sourceware infrastructure against malicious attacks and want to join in on thanking everyone who helped analyze this issue, to me it seems like the much bigger problem is that XZ had a maintainer who appears to have acted in bad faith.  Are the development processes used by the GNU toolchain components robust enough to cope with deliberate sabotage of the code base?  Do we have enough eyes available to ensure that every commit, even those by designated maintainers, is vetted by someone else?  Do we to harden our process, too, to require all patches to be signed off by someone else before committing?
>> 
>> -Sandra
>> 
>> 
> What likely happened for the maintainer who acted in bad faith was that they entered the project with bad faith intent from the start - seeing as they were only involved with the project for 2 years, and there was much social pressure from fake email accounts for the single maintainer of XZ to accept help.
> 
> While we would obviously like to have more area maintainers and possibly global maintainers to help spread the load, I don't think any of the projects listed here are all that susceptible to the same type of social engineering. For one, getting the same type of blanket approval would be a much more involved process because we already have a reasonable amount of people with those privileges, no one is dealing with burnout and sassy customers saying we aren't doing enough.
> 
> Beyond that, we (GDB) are already experimenting with approved-by, and I think glibc was doing the same. That guarantees at least a second set of eyes that analyzed and agreed with the patch, I don't think signed-off would add more than that tag (even if security was not the reason why we implemented them).
> 
> -- 
> Cheers,
> Guinevere Larsen
> She/Her/Hers

I agree that GDB, and for that matter other projects with significant numbers of contributors, are not nearly as likely to be vulnerable to this sort of attack.  But I worry that xz may not be the only project that's small enough to be vulnerable, and be security-relevant in not so obvious ways.

One question that comes to mind is whether there has been an effort across the open source community to identify possible other targets of such attacks.  Contributions elsewhere by the suspect in this case are an obvious concern, but similar scenarios with different names could also be.  That probably should be an ongoing activity: whenever some external component is used, it would be worth knowing how it is maintained, and how many eyeballs are involved.  Even if this isn't done by everyone, it seems like a proper precaution for security sensitive projects.

Another question that comes to mind: I would guess that relevant law enforcement agencies are already looking into this, but it would seem appropriate for those closest to the attacked software to reach out explicitly and assist in any criminal investigations.

	paul

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeffrey Walton]

On Tue, Apr 2, 2024 at 7:35 PM Paul Koning via Gdb <gdb@sourceware.org> wrote:
> [...]
>
> I agree that GDB, and for that matter other projects with significant numbers of contributors, are not nearly as likely to be vulnerable to this sort of attack.  But I worry that xz may not be the only project that's small enough to be vulnerable, and be security-relevant in not so obvious ways.

This cuts a lot deeper than folks think. Here are two other examples
off the top of my head...

Other vulnerable projects include ncurses and libnettle. Ncurses is
run by Thomas Dickey (https://invisible-island.net/). libnettle is run
by Niels Möller (https://www.lysator.liu.se/~nisse/nettle/). Both are
one-man shows with no continuity plans. Dickey does not even run a
public version control system. You have to download his release
tarballs, and there's no history to review or make pull requests
against. If DIckey or Möller got hit by a bus crossing the street,
there would be problems for years.

Jeff



> One question that comes to mind is whether there has been an effort across the open source community to identify possible other targets of such attacks.  Contributions elsewhere by the suspect in this case are an obvious concern, but similar scenarios with different names could also be.  That probably should be an ongoing activity: whenever some external component is used, it would be worth knowing how it is maintained, and how many eyeballs are involved.  Even if this isn't done by everyone, it seems like a proper precaution for security sensitive projects.
>
> Another question that comes to mind: I would guess that relevant law enforcement agencies are already looking into this, but it would seem appropriate for those closest to the attacked software to reach out explicitly and assist in any criminal investigations.
>
>         paul
>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Florian Weimer]

* Guinevere Larsen via Overseers:

> Beyond that, we (GDB) are already experimenting with approved-by, and
> I think glibc was doing the same.

The glibc project uses Reviewed-by:, but it's completely unrelated to
this.  Everyone still pushes their own patches, and there are no
technical countermeasures in place to ensure that the pushed version is
the reviewed version.

Thanks,
Florian

Re: Sourceware mitigating and preventing the next xz-backdoor

[Joel Sherrill]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

On Wed, Apr 3, 2024, 3:09 AM Florian Weimer via Gdb <gdb@sourceware.org>
wrote:

> * Guinevere Larsen via Overseers:
>
> > Beyond that, we (GDB) are already experimenting with approved-by, and
> > I think glibc was doing the same.
>
> The glibc project uses Reviewed-by:, but it's completely unrelated to
> this.  Everyone still pushes their own patches, and there are no
> technical countermeasures in place to ensure that the pushed version is
> the reviewed version.
>

Or that there isn't "collusion" between a malicious author and reviewer.
Just tagging it approved or reviewed by just gives you two people to blame.
It is not a perfect solution either.

But double checking and checklists are good practices. They are not
foolproof if some bad actor is determined enough.

--joel



> Thanks,
> Florian
>
>

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi,

On Wed, 2024-04-03 at 08:53 -0500, Joel Sherrill wrote:
> On Wed, Apr 3, 2024, 3:09 AM Florian Weimer via Gdb <gdb@sourceware.org> wrote:
> > * Guinevere Larsen via Overseers:
> > 
> > > Beyond that, we (GDB) are already experimenting with approved-by, and
> > > I think glibc was doing the same.
> > 
> > The glibc project uses Reviewed-by:, but it's completely unrelated to
> > this.  Everyone still pushes their own patches, and there are no
> > technical countermeasures in place to ensure that the pushed version is
> > the reviewed version.
> 
> Or that there isn't "collusion" between a malicious author and reviewer.
> Just tagging it approved or reviewed by just gives you two people to blame.
> It is not a perfect solution either.
> 
> But double checking and checklists are good practices.
> They are not foolproof if some bad actor is determined enough.

Agreed. If you just focus on completely fool proof technically
checkable measures then you end up doing nothing. But making things
like always getting a Reviewed-by or Tested-by tag in your commit
message does strengthen the social norms. And once they are common
practice you could even add some technical checks.

I am sure a really determined bad actor can always find some social or
technical engineering trick to "defeat" our project policies. But that
doesn't mean we shouldn't do things which are good practices anyway.

Cheers,

Mark

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2980 bytes --]

Hi Joel,

On Wed, Apr 03, 2024 at 08:53:21AM -0500, Joel Sherrill wrote:
> On Wed, Apr 3, 2024, 3:09 AM Florian Weimer via Gdb <gdb@sourceware.org>
> wrote:
> 
> > * Guinevere Larsen via Overseers:
> >
> > > Beyond that, we (GDB) are already experimenting with approved-by, and
> > > I think glibc was doing the same.
> >
> > The glibc project uses Reviewed-by:, but it's completely unrelated to
> > this.  Everyone still pushes their own patches, and there are no
> > technical countermeasures in place to ensure that the pushed version is
> > the reviewed version.
> >
> 
> Or that there isn't "collusion" between a malicious author and reviewer.
> Just tagging it approved or reviewed by just gives you two people to blame.
> It is not a perfect solution either.

If those tags are given in a mailing list _and_ mails to the mailing
list are PGP-signed, then you can verify that the tags were valid, and
not just invented.

And with signed commits you have a guarantee that one doesn't overwrite
history attributing commits to other committers (it could happen with
authors, and indeed it seems to have happened in this case, but if
patches are sent via signed mail, then it's also verifyiable).

In the email side, there are a few things to improve:

For sending signed emails, there's patatt(5) (used by b4(5)), but it
might not be comfortable to use for everyone.  For those preferring
normal MUAs, neomutt(1) is an alternative:

<https://neomutt.org/feature/cli-crypto>

And I have a few patches for improving the security of protected
messages:

<https://github.com/neomutt/neomutt/pull/4221>
<https://github.com/neomutt/neomutt/pull/4227>

And the corresponding security-vulnerability reports:

<https://github.com/neomutt/neomutt/issues/4223>
<https://github.com/neomutt/neomutt/issues/4226>
<https://github.com/neomutt/neomutt/issues/4234>
<https://github.com/neomutt/neomutt/issues/4236>
<https://github.com/neomutt/neomutt/issues/4237>

(I find it funny that I didn't know about this xz issue until yesterday,
 so not when I reported those issues, but they are interestingly
 related.)

It would also be interesting to require showing range-diffs between
patch revisions.  They make it much more difficult to introduce a
vulnerability after a reviewer has turned its mins into approving the
patch.  Of course, the patch could go in if the submitter lies in the
range-diff and the vuln is undetected, but then it can be verified a
posteriory to prove that there was a lie.

I recently started applying all of these (signing all email, including
patches, sign all commits and tags, and provide range-diffs), and it's
not too uncomfrotable; I'd say it's even more comfortable than not doing
it, as it allows me to easily roll back a patch to an older revision if
I find I introduced a mistake, and find where I introduced it.


Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi Alejandro,

On Wed, Apr 10, 2024 at 06:30:42PM +0200, Alejandro Colomar wrote:
> It would also be interesting to require showing range-diffs between
> patch revisions.  They make it much more difficult to introduce a
> vulnerability after a reviewer has turned its mins into approving the
> patch.  Of course, the patch could go in if the submitter lies in the
> range-diff and the vuln is undetected, but then it can be verified a
> posteriory to prove that there was a lie.

Could you give an example of using git range-diff? How do you go from
v1 of a patch (series) to a v2? Normally when asked for changes to a
patch (series) I do an git rebase -i (on the local branch I used to
develop the feature/bug fix) and split/commit all requested changes
and then sent the new patches with git send-email again. But I guess
to use/combine that with git range-diffs I should start creating new
local branches for each patch (series) in development?

Thanks,

Mark

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 6047 bytes --]

On Sun, Apr 21, 2024 at 05:30:52PM +0200, Mark Wielaard wrote:
> Hi Alejandro,

Hi Mark,

I've reordered your message, to organize my response.

> On Wed, Apr 10, 2024 at 06:30:42PM +0200, Alejandro Colomar wrote:
> > It would also be interesting to require showing range-diffs between
> > patch revisions.  They make it much more difficult to introduce a
> > vulnerability after a reviewer has turned its mins into approving the
> > patch.  Of course, the patch could go in if the submitter lies in the
> > range-diff and the vuln is undetected, but then it can be verified a
> > posteriory to prove that there was a lie.
> 
> Could you give an example of using git range-diff?

Sure!

> Normally when asked for changes to a
> patch (series) I do an git rebase -i (on the local branch I used to
> develop the feature/bug fix) and split/commit all requested changes
> and then sent the new patches with git send-email again.

I do the same thing.

> But I guess
> to use/combine that with git range-diffs I should start creating new
> local branches for each patch (series) in development?

No; it's compatible with that workflow.

> How do you go from
> v1 of a patch (series) to a v2?

I'll start first with how you go from nothing to v1, which will help
explain how you go from v1 to v2.

Let's say I have a branch 'unifont' for adding the Unifont font to the
Linux man-pages' PDF book.  I have it in a branch that applies on top of
'master'.

	$ git log --oneline --graph master..unifont;
	* 892a12470 (HEAD -> unifont, alx/contrib, contrib) share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
	* d80376b08 share/mk/: build-pdf-book: Use Unifont
	* 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf

I want to send that as a patch set (v1, of course, since it's the first
send).  I would already use range-diff when generating the patches:

	$ git format-patch -o ./patches master..HEAD \
		--range-diff=master -v1 --cover-letter;
	./patches/v1-0000-cover-letter.patch
	./patches/v1-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
	./patches/v1-0002-share-mk-build-pdf-book-Use-Unifont.patch
	./patches/v1-0003-share-mk-build-fonts-unifont-Specify-spacewidth-i.patch

Why would you do that in v1?  It notes the commit IDs of the patches, so
you can later check them when preparing v2; we'll come back to that.
For now, see what this adds to the patch set:

	$ tail ./patches/v1-0000-cover-letter.patch ;
	 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
	 create mode 100644 share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk

	Range-diff against v0:
	-:  --------- > 1:  7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
	-:  --------- > 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
	-:  --------- > 3:  892a12470 share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
	-- 
	2.43.0

You can see the IDs of the commits that form this patch set, which match
the git-log(1) that I showed above.  Now someone suggests a change.  For
this example, I wasn't happy with a commit message, and added a space to
the third commit message subject line.  The git-log(1) is now:

	$ git log --oneline --graph master..unifont
	* bc7fa7d92 (HEAD -> unifont) share/mk/: build-fonts-unifont: Specify space width in afmtodit(1)
	* d80376b08 share/mk/: build-pdf-book: Use Unifont
	* 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf

Let's generate a v2 patch set, showing the range-diff against v1.  We
need to check the commit IDs of the first set, which can be found in the
mailing list archives, thanks to the trick we used.  The v1 range was
7ec952012^..892a12470.  So we just pass that range:

	$ git format-patch -o ./patches/ master..HEAD \
		--range-diff=7ec952012^..892a12470 -v2 --cover-letter;
	./patches/v2-0000-cover-letter.patch
	./patches/v2-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
	./patches/v2-0002-share-mk-build-pdf-book-Use-Unifont.patch
	./patches/v2-0003-share-mk-build-fonts-unifont-Specify-space-width-.patch

The v2 cover letter shows the changes introduced since v1:

	$ tail -n20 ./patches/v2-0000-cover-letter.patch 
	 create mode 100644 share/mk/build/fonts/unifont/dit.mk
	 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
	 create mode 100644 share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk

	Range-diff against v1:
	1:  7ec952012 = 1:  7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
	2:  d80376b08 = 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
	3:  892a12470 ! 3:  bc7fa7d92 share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
	    @@ Metadata
	     Author: Alejandro Colomar <alx@kernel.org>
	     
	      ## Commit message ##
	    -    share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
	    +    share/mk/: build-fonts-unifont: Specify space width in afmtodit(1)
	     
		 Link: <https://lore.kernel.org/linux-man/ZiQ_mTQHPq3ig723@debian/T/#t>
		 Suggested-by: "G. Branden Robinson" <branden@debian.org>
	-- 
	2.43.0

If too much time passes between the last time the commit was rebased and
when you create the range diff, you might need to use tags, such as
'unifont-v1', to avoid git(1) collecting garbagge and removing those
old commits.  But if you send shortly after rebasing, you won't need
that.

Lately, I haven't sent many patches to mailing lists, except for trivial
oneliners, so I don't remember an example where I used this, but I can
point to several github PRs where I've used this to document all changes
that have happened to my patch sets, even when the maintainer has edited
them, to be fully transparent.  For example:
<https://github.com/neomutt/neomutt/pull/4221#issuecomment-2061341972>
<https://github.com/neomutt/neomutt/pull/4221#issuecomment-2052533059>

Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 6759 bytes --]

On Sun, Apr 21, 2024 at 10:40:14PM +0200, Alejandro Colomar wrote:
> On Sun, Apr 21, 2024 at 05:30:52PM +0200, Mark Wielaard wrote:
> > Hi Alejandro,
> 
> Hi Mark,
> 
> I've reordered your message, to organize my response.
> 
> > On Wed, Apr 10, 2024 at 06:30:42PM +0200, Alejandro Colomar wrote:
> > > It would also be interesting to require showing range-diffs between
> > > patch revisions.  They make it much more difficult to introduce a
> > > vulnerability after a reviewer has turned its mins into approving the
> > > patch.  Of course, the patch could go in if the submitter lies in the
> > > range-diff and the vuln is undetected, but then it can be verified a
> > > posteriory to prove that there was a lie.
> > 
> > Could you give an example of using git range-diff?
> 
> Sure!
> 
> > Normally when asked for changes to a
> > patch (series) I do an git rebase -i (on the local branch I used to
> > develop the feature/bug fix) and split/commit all requested changes
> > and then sent the new patches with git send-email again.
> 
> I do the same thing.
> 
> > But I guess
> > to use/combine that with git range-diffs I should start creating new
> > local branches for each patch (series) in development?
> 
> No; it's compatible with that workflow.
> 
> > How do you go from
> > v1 of a patch (series) to a v2?
> 
> I'll start first with how you go from nothing to v1, which will help
> explain how you go from v1 to v2.
> 
> Let's say I have a branch 'unifont' for adding the Unifont font to the
> Linux man-pages' PDF book.  I have it in a branch that applies on top of
> 'master'.
> 
> 	$ git log --oneline --graph master..unifont;
> 	* 892a12470 (HEAD -> unifont, alx/contrib, contrib) share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
> 	* d80376b08 share/mk/: build-pdf-book: Use Unifont
> 	* 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
> 
> I want to send that as a patch set (v1, of course, since it's the first
> send).  I would already use range-diff when generating the patches:
> 
> 	$ git format-patch -o ./patches master..HEAD \
> 		--range-diff=master -v1 --cover-letter;
> 	./patches/v1-0000-cover-letter.patch
> 	./patches/v1-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
> 	./patches/v1-0002-share-mk-build-pdf-book-Use-Unifont.patch
> 	./patches/v1-0003-share-mk-build-fonts-unifont-Specify-spacewidth-i.patch
> 
> Why would you do that in v1?  It notes the commit IDs of the patches, so
> you can later check them when preparing v2; we'll come back to that.
> For now, see what this adds to the patch set:
> 
> 	$ tail ./patches/v1-0000-cover-letter.patch ;
> 	 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
> 	 create mode 100644 share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk
> 
> 	Range-diff against v0:
> 	-:  --------- > 1:  7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
> 	-:  --------- > 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
> 	-:  --------- > 3:  892a12470 share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
> 	-- 
> 	2.43.0
> 
> You can see the IDs of the commits that form this patch set, which match
> the git-log(1) that I showed above.  Now someone suggests a change.  For
> this example, I wasn't happy with a commit message, and added a space to
> the third commit message subject line.  The git-log(1) is now:
> 
> 	$ git log --oneline --graph master..unifont
> 	* bc7fa7d92 (HEAD -> unifont) share/mk/: build-fonts-unifont: Specify space width in afmtodit(1)
> 	* d80376b08 share/mk/: build-pdf-book: Use Unifont
> 	* 7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
> 
> Let's generate a v2 patch set, showing the range-diff against v1.  We
> need to check the commit IDs of the first set, which can be found in the
> mailing list archives, thanks to the trick we used.  The v1 range was
> 7ec952012^..892a12470.  So we just pass that range:
> 
> 	$ git format-patch -o ./patches/ master..HEAD \
> 		--range-diff=7ec952012^..892a12470 -v2 --cover-letter;
> 	./patches/v2-0000-cover-letter.patch
> 	./patches/v2-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
> 	./patches/v2-0002-share-mk-build-pdf-book-Use-Unifont.patch
> 	./patches/v2-0003-share-mk-build-fonts-unifont-Specify-space-width-.patch
> 
> The v2 cover letter shows the changes introduced since v1:
> 
> 	$ tail -n20 ./patches/v2-0000-cover-letter.patch 
> 	 create mode 100644 share/mk/build/fonts/unifont/dit.mk
> 	 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
> 	 create mode 100644 share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk
> 
> 	Range-diff against v1:
> 	1:  7ec952012 = 1:  7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
> 	2:  d80376b08 = 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
> 	3:  892a12470 ! 3:  bc7fa7d92 share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
> 	    @@ Metadata
> 	     Author: Alejandro Colomar <alx@kernel.org>
> 	     
> 	      ## Commit message ##
> 	    -    share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
> 	    +    share/mk/: build-fonts-unifont: Specify space width in afmtodit(1)
> 	     
> 		 Link: <https://lore.kernel.org/linux-man/ZiQ_mTQHPq3ig723@debian/T/#t>
> 		 Suggested-by: "G. Branden Robinson" <branden@debian.org>
> 	-- 
> 	2.43.0
> 
> If too much time passes between the last time the commit was rebased and
> when you create the range diff, you might need to use tags, such as
> 'unifont-v1', to avoid git(1) collecting garbagge and removing those
> old commits.  But if you send shortly after rebasing, you won't need
> that.
> 
> Lately, I haven't sent many patches to mailing lists, except for trivial
> oneliners, so I don't remember an example where I used this, but I can
> point to several github PRs where I've used this to document all changes
> that have happened to my patch sets, even when the maintainer has edited
> them, to be fully transparent.  For example:
> <https://github.com/neomutt/neomutt/pull/4221#issuecomment-2061341972>
> <https://github.com/neomutt/neomutt/pull/4221#issuecomment-2052533059>

And here's an example where the maintainer pushed a version different
from my last revision, because he rebased after applying other changes
to his branch, and I showed a last range diff to prove that he didn't
do anything weird with my patches:

<https://github.com/neomutt/neomutt/pull/4247#issuecomment-2061145881>

> 
> Have a lovely day!
> Alex
> 
> -- 
> <https://www.alejandro-colomar.es/>

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2883 bytes --]

Hi Mark,

On Sun, Apr 21, 2024 at 10:40:14PM +0200, Alejandro Colomar wrote:

[...]

> Let's generate a v2 patch set, showing the range-diff against v1.  We
> need to check the commit IDs of the first set, which can be found in the
> mailing list archives, thanks to the trick we used.  The v1 range was
> 7ec952012^..892a12470.  So we just pass that range:
> 
> 	$ git format-patch -o ./patches/ master..HEAD \
> 		--range-diff=7ec952012^..892a12470 -v2 --cover-letter;
> 	./patches/v2-0000-cover-letter.patch
> 	./patches/v2-0001-share-mk-build-fonts-unifont-Build-UnifontR-from-.patch
> 	./patches/v2-0002-share-mk-build-pdf-book-Use-Unifont.patch
> 	./patches/v2-0003-share-mk-build-fonts-unifont-Specify-space-width-.patch
> 
> The v2 cover letter shows the changes introduced since v1:
> 
> 	$ tail -n20 ./patches/v2-0000-cover-letter.patch 
> 	 create mode 100644 share/mk/build/fonts/unifont/dit.mk
> 	 create mode 100644 share/mk/build/fonts/unifont/pfa.mk
> 	 create mode 100644 share/mk/configure/build-depends/fonts-unifont/unifont.otf.mk
> 
> 	Range-diff against v1:
> 	1:  7ec952012 = 1:  7ec952012 share/mk/: build-fonts-unifont: Build UnifontR from unifont.otf
> 	2:  d80376b08 = 2:  d80376b08 share/mk/: build-pdf-book: Use Unifont
> 	3:  892a12470 ! 3:  bc7fa7d92 share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
> 	    @@ Metadata
> 	     Author: Alejandro Colomar <alx@kernel.org>
> 	     
> 	      ## Commit message ##
> 	    -    share/mk/: build-fonts-unifont: Specify spacewidth in afmtodit(1)
> 	    +    share/mk/: build-fonts-unifont: Specify space width in afmtodit(1)
> 	     
> 		 Link: <https://lore.kernel.org/linux-man/ZiQ_mTQHPq3ig723@debian/T/#t>
> 		 Suggested-by: "G. Branden Robinson" <branden@debian.org>
> 	-- 
> 	2.43.0

I've added a recommendation in the Linux man-pages contributing
documentation that patches be sent with a range diff, and also that
patches be sent in PGP-signed mail (if the user has a PGP key).  It has
specific instructions like the above (but simplified).

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/CONTRIBUTING.d/patches?id=12aca537ce78a41bbcdaf485209691e10f8002d7#n91>

Feel free to copy any of that documentation.

I also recommended specific mutt(1) settings:

            set crypt_autosign = yes
            set crypt_protected_headers_write = yes

And git-send-email(1) configuration for using with neomutt(1):

           [sendemail]
               sendmailcmd = neomutt -C -H - && true

For all the documentation for mail and patches, see these two files:
<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/CONTRIBUTING.d/patches>
<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/CONTRIBUTING.d/mail>

Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Sourceware mitigating and preventing the next xz-backdoor

[Tom Tromey]

>>>>> "Florian" == Florian Weimer <fweimer@redhat.com> writes:

Florian> Everyone still pushes their own patches, and there are no
Florian> technical countermeasures in place to ensure that the pushed version is
Florian> the reviewed version.

This is a problem for gdb as well.

Probably we should switch to some kind of pull-request model, where
patches can only be landed via the UI, after sufficient review; and
where all generated files are regenerated by the robot before checkin.
(Or alternatively some CI runs and rejects patches where they don't
match.)

Tom

Re: Sourceware mitigating and preventing the next xz-backdoor

[Jeff Law]

On 4/3/24 8:04 AM, Tom Tromey wrote:
>>>>>> "Florian" == Florian Weimer <fweimer@redhat.com> writes:
> 
> Florian> Everyone still pushes their own patches, and there are no
> Florian> technical countermeasures in place to ensure that the pushed version is
> Florian> the reviewed version.
> 
> This is a problem for gdb as well.
> 
> Probably we should switch to some kind of pull-request model, where
> patches can only be landed via the UI, after sufficient review; and
> where all generated files are regenerated by the robot before checkin.
> (Or alternatively some CI runs and rejects patches where they don't
> match.)
I've very much prefer to move to a pull-request model.

jeff

Re: Sourceware mitigating and preventing the next xz-backdoor

[Mark Wielaard]

Hi,

On Wed, 2024-04-03 at 08:42 -0600, Jeff Law wrote:
> On 4/3/24 8:04 AM, Tom Tromey wrote:
> > > > > > > "Florian" == Florian Weimer <fweimer@redhat.com> writes:
> > 
> > Florian> Everyone still pushes their own patches, and there are no
> > Florian> technical countermeasures in place to ensure that the pushed version is
> > Florian> the reviewed version.
> > 
> > This is a problem for gdb as well.
> > 
> > Probably we should switch to some kind of pull-request model, where
> > patches can only be landed via the UI, after sufficient review; and
> > where all generated files are regenerated by the robot before checkin.
> > (Or alternatively some CI runs and rejects patches where they don't
> > match.)
> I've very much prefer to move to a pull-request model.

Do you need any infrastructure updates to help (experiment) with that?
Now would be a great time to request some updates to patchwork or get
us to resurrect the gerrit server if that would be helpful.

We just published the Sourceware 2024 infrastructure plan:
https://inbox.sourceware.org/20240325095827.GI5673@gnu.wildebeest.org/
Setting priorities for the infrastructure for 2024 (and beyond). We are
just now scheduling and budgeting that work. So please get your
requests in.

Cheers,

Mark