Re: release branch policy and distributions

[Michael Hudson-Doyle <michael.hudson@canonical.com>]

[-- Attachment #1: Type: text/plain, Size: 8325 bytes --]

On Fri, 3 Mar 2023 at 07:04, Carlos O'Donell <carlos@redhat.com> wrote:

> On 2/16/23 17:57, Michael Hudson-Doyle wrote:
> > Would it be unreasonable to suggest a policy where performance
> improvements
> > are not backported to release branches until say a month after they have
> > been included in a glibc release? I realize this would add some overhead
> to
> > keep track of these 'pending' backports but I personally would be happier
> > consuming the release branches directly if there was this sort of policy.
>
> Michael, Andreas, Adhemerval,
>
> Thank you all for raising this.
>

Thank you for the thought provoking reply.


> I want to talk about outcomes.
>
> The outcome I want is for there to be fewer defects in the development
> branch,
> and by proxy fewer defects in the release branch.
>

Obviously this is not an outcome I would oppose so it's interesting to
think about why I don't feel it quite gets to the point of my concerns.
More on this later :-)

(1) Do we have evidence of an increased rate of defects?
>
> I know we have some anecdotal evidence that we recently had defects in the
> rolling release branches. Have we collected that evidence to determine what
> kind of action is required?


I don't have data, no. When I think of this sort of thing, I think of two
recentish bugs:

1) https://sourceware.org/bugzilla/show_bug.cgi?id=29611 which is the one
where code assumed that if AVX2 was available, BMI2 was too
2) https://sourceware.org/bugzilla/show_bug.cgi?id=30065 which was
confusion around the semantics of strncat (this didn't get backported to a
release branch, although it came pretty close to ending up in a release)


> Do we have a gap in our hardware or testing that
> needs to be improved?
>

Well, clearly, yes there are gaps. On the hardware front, especially in x86
land, is it realistic to cover all possibilities? I'm very far from an
expert on this stuff but I know the kernel defines more than
300 X86_FEATURE_* macros and while lots are presumably always true on all
hardware glibc still supports and it's not like they are all independently
available, it's still an intimidating landscape. Maybe I am being overly
pessimistic.

On the semantic front, I kind of feel the same way. It's clearly _possible_
to have tests that cover all aspects of the semantics of the string
functions and glibc surely has tests that cover _most_ semantics already,
but absent something like autogeneration of test cases from some kind of
formal description of these semantics -- which are then executed on a wide
range of hardware! -- I can't see how we can be confident no gaps remain.

A gap here could be that we need to setup x86_64 pre-commit CI with an
> AVX512
> system to test all the IFUNCs (which may catch nothing if the tests are
> missing).
>

That's certainly one example.


> Another gap here could be that we need to setup pre-commit CI to rebuild
> certain
> packages under the modified glibc (similar to Fedora Rawhide CI).
>

Again this might help catch some issues, but I doubt it would have caught
the above issues.


> (2) What is your distro policy for updating from the rolling release
> branch?
>

Poorly defined, which is something I would like to change. In practice

1) we follow the release branch for a short while after release, with the
final update a bit before Ubuntu itself releases
2) for non-LTS releases we generally don't update glibc at all
3) for LTS releases we do occasional updates on request, basically

Security updates trump all of this of course (but are not handled by my
team).

Do you know what RHEL's policy is for glibc updates?

While upstream gives a policy, what is your own policy?
>

Well by default, it doesn't change. A strict reading of
https://wiki.ubuntu.com/StableReleaseUpdates would suggest that a glibc
update would have to be accompanied by an explicit test case for each
change that has been included in the release branch since the previous
update. I think glibc should be covered by the "micro release exception"
though:
https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases

Example: Fedora Rawhide CI rebuilds a number of packages using the new glibc
> we sync weekly, and we review the rebuild failures and their testsuite
> results
> before putting the new glibc into Rawhide (or stable Fedora releases).
> For example, rebuilding lua and running their testsuite, particularly the
> string
> testsuite is good at detecting further string-related optimization defects.
>

We don't do anything like this as regularly. We do a rebuild of every
package in the archive with a snap shot glibc at some point in the
development cycle but usually only once. Each new upload of glibc, to
development or a stable release, triggers the testing of almost every other
package as well. The issue we have of course -- and I assume Fedora is the
same here -- is that these tests all run on essentially the same hardware.
We found BZ# 30065 in our rebuild testing but this was partly just luck.


> (3) How does delaying backports impact our outcome?
>
> One way is that we use this extra time to do additional testing that
> discovers
> defects, and then we work with the machine maintainer, IHVs, etc, and
> correct
> the defects.
>

Well, if the delay is past a glibc or distribution release, that might make
a difference.

I do think there is a real difference in how bad things are for a defect to
be in different places. To be parochial and concentrate on Ubuntu:

1) a bug in the development branch only of glibc is currently of very
little impact to Ubuntu. We don't upload pre-releases to the primary
archive.
2) a bug in a glibc release will get uploaded to the "proposed pocket" of
the development series of Ubuntu, where a lot of testing happens (on
homogenous hardware though). A bug here still doesn't impact users but can
interfere with distribution development
3) if a bug gets past the automated testing it migrates to the "release
pocket" of the development series of Ubuntu, which can affect users of the
development series of Ubuntu, but these people are expected to know what
they are letting themselves in for
4) if a bug makes it into the Ubuntu release, it can affect more regular
users, which is starting to get into bad news territory but at least it
will only affect new installs or newly updated installs.
5) if a bug is included in a stable release update, that's... really really
bad. It leads to bad press and a culture of people not applying updates.

Obviously we don't just unleash stable release updates on people without
any testing but as I've said a few times, the automated testing hardware is
quite homogenous.


> This means that the action we want to take is not delaying, but some kind
> of
> increase in testing. In fact delaying may solve nothing if additional
> validation and verification is not carried out in that delay period.
>

Well yes. Maybe the release being deployed to distributions isn't "testing"
explicitly but it's certainly "use".

My opinion is that delaying alone is not an outcome changing activity,


I humbly disagree on this point, as above. Delaying in and of itself is not
an outcome changing activity, but delaying past glibc and distributions
releases can be.


> and as
> a steward for the project I do not want to delay code from reach our users
> unless we can show that delay allowed the users to capture some value e.g.
> higher stability.
>

I also want to get updated code to users more quickly, that's why I started
this thread!


> How could Canonical, Gentoo or Linaro support additional upstream testing?
>

I think running the glibc testsuite on a wider range of hardware would be
the most significant thing we could do here. We do have quite a range of
hardware for testing but I wouldn't know where to start about using it for
glibc pre-commit CI, and I also doubt it's comprehensive in a way that
would be useful in this context. I wonder if the silicon vendors have
anything like this...


> Can we work together to turn on more distro-specific pre-commit CI testing?
>

I don't think "distro-specific" is quite the point here.

Cheers,
mwh

We have patchwork, it has a REST API, and we can submit test results via
> that
> API, like we do today for i686 test results (Fedora Rawhide-based).
>
> --
> Cheers,
> Carlos.
>
>