Retrospective on glibc 2.34 and glibc 2.35 release.

Retrospective on glibc 2.34 and glibc 2.35 release.

[Carlos O'Donell]

Community,

I was the release manager for 2.34 and 2.35, and in retrospect I have
played the hard ABI deadline too loose for my own liking, and I think
this causes additional stress and work at the last minute that doesn't
help the project.

If I think about improving the release process it is to come back to
a harder ABI freeze deadline within the first week of the freeze.
That is to say that it looks like this:

1st week
- Freeze ABI.
- Review outstanding patches that change ABI that we want in the release.
- Be critical about which can be resolved within the 1st week.
- Move all others that need more time to the next release.

As RM I ran ABI changes very late into 2.34 and 2.35, and while I'm happy
with the results of the release, I'm not happy with the process. I consider
it my own failure as RM for not hardening that release sooner. The time boxed
release for glibc supports downstreams that have aligned their release windows
against our ABI changes, and last minute ABI changes are not good for anyone.

In order to support these ABI changes I wonder if we don't need a specific
window within the release like "Month 1: ABI" where we focus on this theme
of getting new ABIs into the first month of development? There is a similarity
here with respect to gcc's staged development.

Thoughts?

-- 
Cheers,
Carlos.

Re: Retrospective on glibc 2.34 and glibc 2.35 release.

[Joseph Myers]

On Wed, 9 Feb 2022, Carlos O'Donell via Libc-alpha wrote:

> In order to support these ABI changes I wonder if we don't need a specific
> window within the release like "Month 1: ABI" where we focus on this theme
> of getting new ABIs into the first month of development? There is a similarity
> here with respect to gcc's staged development.

I don't think we need such a short period for getting new ABIs in.  Five 
months for general development and one month for stablization and testing 
should be fine if we return to having a proper release freeze with no 
major architecture-independent changes, whether or not they affect ABIs, 
in at least the last two weeks of the freeze (unless significant problems 
with a new ABI are discovered late, in which case the normal expectation 
should be to remove it from the release and reconsider in fixed form for 
the next release).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Retrospective on glibc 2.34 and glibc 2.35 release.

[Stafford Horne]

On Wed, Feb 09, 2022 at 09:57:42AM -0500, Carlos O'Donell via Libc-alpha wrote:
> Community,
> 
> I was the release manager for 2.34 and 2.35, and in retrospect I have
> played the hard ABI deadline too loose for my own liking, and I think
> this causes additional stress and work at the last minute that doesn't
> help the project.
> 
> If I think about improving the release process it is to come back to
> a harder ABI freeze deadline within the first week of the freeze.
> That is to say that it looks like this:
> 
> 1st week
> - Freeze ABI.
> - Review outstanding patches that change ABI that we want in the release.
> - Be critical about which can be resolved within the 1st week.
> - Move all others that need more time to the next release.
> 
> As RM I ran ABI changes very late into 2.34 and 2.35, and while I'm happy
> with the results of the release, I'm not happy with the process. I consider
> it my own failure as RM for not hardening that release sooner. The time boxed
> release for glibc supports downstreams that have aligned their release windows
> against our ABI changes, and last minute ABI changes are not good for anyone.
> 
> In order to support these ABI changes I wonder if we don't need a specific
> window within the release like "Month 1: ABI" where we focus on this theme
> of getting new ABIs into the first month of development? There is a similarity
> here with respect to gcc's staged development.
> 
> Thoughts?

Re, ABI freeze I think it sounds OK to me and it gives port maintainers time to
test.

Regarding port testing, I usually test with GCC mainline.  During the test phase
it seemed to coincide with some big changes in GCC which broke things for a lot
of architectures.  This made it hard to test, and actually I am just getting the
OpenRISC port to run the test suite now.

 PR: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104153

Do we see any possiblity to sync our 1-month phase with a stable GCC period?

-Stafford

Re: Retrospective on glibc 2.34 and glibc 2.35 release.

[Siddhesh Poyarekar]

On 09/02/2022 20:27, Carlos O'Donell via Libc-alpha wrote:
> Community,
> 
> I was the release manager for 2.34 and 2.35, and in retrospect I have
> played the hard ABI deadline too loose for my own liking, and I think
> this causes additional stress and work at the last minute that doesn't
> help the project.
> 
> If I think about improving the release process it is to come back to
> a harder ABI freeze deadline within the first week of the freeze.
> That is to say that it looks like this:
> 
> 1st week
> - Freeze ABI.
> - Review outstanding patches that change ABI that we want in the release.
> - Be critical about which can be resolved within the 1st week.
> - Move all others that need more time to the next release.
> 
> As RM I ran ABI changes very late into 2.34 and 2.35, and while I'm happy
> with the results of the release, I'm not happy with the process. I consider
> it my own failure as RM for not hardening that release sooner. The time boxed
> release for glibc supports downstreams that have aligned their release windows
> against our ABI changes, and last minute ABI changes are not good for anyone.
> 
> In order to support these ABI changes I wonder if we don't need a specific
> window within the release like "Month 1: ABI" where we focus on this theme
> of getting new ABIs into the first month of development? There is a similarity
> here with respect to gcc's staged development.

I concur with Joseph; the process as we defined it is not broken, we 
were only lax in implementing it.  You're not the sole person to blame 
either IMO; we share blame because all of us let it slide for various 
reasons.  I would normally have called you out for letting the freeze 
slip but I'm a hypocrite ;)  I was glad that it slid for 2.34 because it 
allowed me to drop malloc hooks and similarly we got some needed ABI 
changes in at the last moment in 2.35 and so far it doesn't look like we 
had any fallout from it.

We just need to make sure that we put a hard freeze on approving ABI 
changes beyond the freeze deadline and then as Joseph suggested, 
consider backing out ABI changes that continue to be broken during the 
last couple of weeks of the freeze.

The one concern I have about the freeze period though is that once a 
year it descends on us during the year-end holidays.  Getting reviews 
during that period has always been hard, which means that the freeze is 
implicitly a couple of weeks earlier for anyone who doesn't have enough 
leverage within the community to get reviews for their patches.  I 
wonder if there's scope for a 2 week shift in our schedule so that we 
freeze on 15th Jan and 15th Jul and release on 15th Feb and 15th Aug.

Thanks,
Siddhesh

Re: Retrospective on glibc 2.34 and glibc 2.35 release.

[Siddhesh Poyarekar]

On 10/02/2022 06:39, Stafford Horne via Libc-alpha wrote:
> 
> Do we see any possiblity to sync our 1-month phase with a stable GCC period?
> 

Stable is a bit far out (around April) but a mid-Jan freeze as I 
suggested in the earlier email will end up syncing with stage 4, which I 
suppose is not that far worse than stable.  That would be a good second 
reason to shift our release timelines by 15 days from the current 
timelines, giving us a whole month to test with a stage 4 gcc.

Thanks,
Siddhesh

Re: Retrospective on glibc 2.34 and glibc 2.35 release.

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

> Community,
> 
> I was the release manager for 2.34 and 2.35, and in retrospect I have
> played the hard ABI deadline too loose for my own liking, and I think
> this causes additional stress and work at the last minute that doesn't
> help the project.
> 

How about a much simpler change (which, I grant it, causes extra work):

* What is now the release date (with all related rules) moves one week
earlier, and results in a release candidate *tarball* (with a tag) from
the release branch.
* One week later, the actual release is done (and contains absolutely
only bugfixes compared to the rc)....

Cheers,
Andreas

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: Retrospective on glibc 2.34 and glibc 2.35 release.

[Joseph Myers]

On Thu, 10 Feb 2022, Andreas K. Huettel via Libc-alpha wrote:

> * What is now the release date (with all related rules) moves one week
> earlier, and results in a release candidate *tarball* (with a tag) from
> the release branch.

The use of such a tarball is marginal - there's already supposed to be one 
at the start of the freeze for submission to the Translation Project.

The main thing is having a (long enough) proper freeze after which there 
are no major architecture-independent changes (ABI or otherwise) other 
than reverting the addition of any new features that have proved 
problematic at a late stage.

-- 
Joseph S. Myers
joseph@codesourcery.com