Hi all, 

> The outcome I want is for there to be fewer defects in the development branch,
> and by proxy fewer defects in the release branch.
> 
> (1) Do we have evidence of an increased rate of defects?

I can't contribute more than anecdotal evidence myself since I do not do 
regular builds (see below). 

I myself hit (from memory) a bug in the Intel optimizations once, which was
rather quickly fixed with two more backported commits.
(The symptom was failing tests on my own machine, so found before installation
and no real harm done.)

> (2) What is your distro policy for updating from the rolling release branch?
> 
> While upstream gives a policy, what is your own policy?

For Gentoo, nothing automated. 

When I become aware of an important fix, someone pokes me or I otherwise
feel like it I cherry-pick new commits from the release branch onto our
Gentoo branch [1] and spin a new tarball (a tag here [1]). Usually this 
catches up with all new additions from release/2.xx/master.

This then becomes an "unkeyworded" ebuild in the Gentoo repository, meaning
noone gets it unless explicitly requested. After some testing by me and
other Gentoo developers, it enters Gentoo "testing/unstable".
In case of new releases (i.e. first 2.37 introduction) also a tinderbox
run will typically be done (mass test build with new glibc, which in itself
becomes some sort of runtime testing too).

Gentoo stable is most of the times one release back [2]. Bugzilla is
monitored closely before "stabilization", which typically jumps from 
an "advanced" 2.xx revision to an "advanced" 2.xy revision, say from
2.35-r11 to 2.36-r5. This somewhat assumes that during the lifetime of
the release branch its state continuously improves (e.g., bug fixes only).

[1] https://gitweb.gentoo.org/fork/glibc.git/
[2] http://packages.gentoo.org/packages/sys-libs/glibc

> (3) How does delaying backports impact our outcome?

Much of the quality control for Gentoo stable is effectively the feedback
from Gentoo unstable/testing.
Given that we unleash only released versions on our userbase, code that has been
in a release naturally gets a lot more testing. (Both build testing with all
combinations of flags and architectures and runtime testing.)

Since installation of the package fails when the testsuite fails, not many
of our users enable the test phase (and we also do not recommend that in general).

Installing a git master version of glibc to your system is trivial in Gentoo.
However, if you do it to your main system you'll be told that you're insane. :)
So this is mostly for test containers.
It's not easily possible right now but a similar mechanism could be implemented
for release branches.

> One way is that we use this extra time to do additional testing that discovers
> defects, and then we work with the machine maintainer, IHVs, etc, and correct
> the defects.
> 
> This means that the action we want to take is not delaying, but some kind of
> increase in testing. In fact delaying may solve nothing if additional
> validation and verification is not carried out in that delay period.

The testsuite of glibc catches a lot of potential problems.
It'll never beat real-world usage though.

> How could Canonical, Gentoo or Linaro support additional upstream testing?

The Gentoo-specific part is actually very small, right now 9 patches [3]: 
* 5 of which are Adhemerval's readdir patchset
* 1 is the ia64-specific fix from 
    https://sourceware.org/pipermail/libc-alpha/2020-May/114028.html
    (which turned out to be not ideal, but a better solution never materialized)
* 2 are small Gentoo-specific adaptions
* 1 hardenes the conformance test suite against CC="gcc -O2" by adding -O0 there
    https://bugs.gentoo.org/659030#c6

[3] https://gitweb.gentoo.org/proj/toolchain/glibc-patches.git/tree/9999

> Can we work together to turn on more distro-specific pre-commit CI testing?
> We have patchwork, it has a REST API, and we can submit test results via that
> API, like we do today for i686 test results (Fedora Rawhide-based).

I certainly have no objections to additional pre- or post-commit CI testing.
That said, we don't really have the means or infrastructure to do that frequent
mass rebuilds and runtime testing.

And I tend to be more worried about random unexpected things popping up that
are not caught by the testsuite. Not because they happen frequently, they
clearly don't, and on the whole I am really very happy about the code quality
and backport discipline. Because of the wide potential impact.

Best,
Andreas

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)