2.25 freeze - a little more than the holiday fortnight to go

2.25 freeze - a little more than the holiday fortnight to go

[Siddhesh Poyarekar]

Hi,

A quick reminder to everyone that we're approximately one holiday
fortnight away from the slushy freeze for 2.25.  I'm posting this for
two reasons:

0. Most of us will likely disappear towards the end of the year and
there will likely be little movement on pending patches, so this is a
reminder to get things off your plate early to avoid leaving things
until it is too late

1. I volunteered to manage the 2.25 release and nobody objected, so I
assume calling this is my responsibility :)

I had proposed tunables as a blocker for 2.25 during Cauldron and I want
to stick with that position unless someone has a good reason to delay it
for yet another release.  If there is anything else that needs attention
but is not getting it then please ping early and ping often so that it
gets the desired attention.

Siddhesh

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Nix]

On 13 Dec 2016, Siddhesh Poyarekar stated:

> A quick reminder to everyone that we're approximately one holiday
> fortnight away from the slushy freeze for 2.25.  I'm posting this for
> two reasons:

Florian: given this, is a re-review of the revised stack-protector
patches I posted a couple of weeks ago likely, or will we have to
postpone the thing again to 2.26? :(

(On the availability front, I'll be away from the 20th to the 4th, but
might be able to trip off some new test builds for any changes suggested
after that on ARM and x86 anyway, but not SPARC.)

-- 
NULL && (void)

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Torvald Riegel]

On Tue, 2016-12-13 at 21:13 +0530, Siddhesh Poyarekar wrote:
> I had proposed tunables as a blocker for 2.25 during Cauldron and I want
> to stick with that position unless someone has a good reason to delay it
> for yet another release.  If there is anything else that needs attention
> but is not getting it then please ping early and ping often so that it
> gets the desired attention.

I've added the new condvar and rwlock as blockers.  We've been testing
them for several weeks in Rawhide and haven't heard any complaints so
far.  Carlos has reviewed the patches, basically.

I have still to add support for the new pretty printers, but I haven't
had time to revise the patches yet.  Expect this very early next year,
or perhaps even end of this week if the robust mutex issues take less
time than expected.

Depending on how things go, getting the robust mutex fixes included in
the 2.25 release would be good too, I believe.

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Joseph Myers]

I'm dubious about keeping piling on "blockers" without evidence that there 
are actually people to review the patches no later than 31 December 
(earlier than that if revisions and re-review are needed).  I think 
listing something as a blocker should require the listing to name a 
reviewer who has committed to reviewing the patch by a specified date 
(where that date must be before the freeze, i.e. no later than 31 
December, for any significant architecture-independent patch; patches to 
architecture-specific code might have slightly later dates if the people 
testing on that architecture are happy with the reduced stabilization 
time).  If the review is not forthcoming on time or any required revisions 
are not ready, reviewed and committed no later than 31 December, the 
expectation should be that such a patch is postponed.

We should expect 1 January onwards to be available for architecture 
testing and stabilization / bug fixes, so avoid any significant changes 
after then that would require redoing such testing.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Siddhesh Poyarekar]

On Thursday 22 December 2016 02:48 AM, Joseph Myers wrote:
> I'm dubious about keeping piling on "blockers" without evidence that there 
> are actually people to review the patches no later than 31 December 
> (earlier than that if revisions and re-review are needed).  I think 

That is a fair point.  I've added Florian as reviewer for tunables since
he had agreed to review them next week.

Siddhesh

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Florian Weimer]

On 12/21/2016 10:18 PM, Joseph Myers wrote:
> I'm dubious about keeping piling on "blockers" without evidence that there
> are actually people to review the patches no later than 31 December
> (earlier than that if revisions and re-review are needed).

My understanding of the meaning of “release blocker” and “freeze” is 
quite different.  An accepted release blocker can be fixed *in* the 
freeze period.  As a result, I have been focusing on things that are 
clearly *not* release blockers, and my attention will shift after 
January 1st.

On the other hand, features and non-regression bug fixes should 
generally not be release blockers because I do not think we want to 
potentially deviate from our time-based releases for their sake.

Thanks,
Florian

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Florian Weimer]

On 12/21/2016 09:19 PM, Torvald Riegel wrote:
> On Tue, 2016-12-13 at 21:13 +0530, Siddhesh Poyarekar wrote:
>> I had proposed tunables as a blocker for 2.25 during Cauldron and I want
>> to stick with that position unless someone has a good reason to delay it
>> for yet another release.  If there is anything else that needs attention
>> but is not getting it then please ping early and ping often so that it
>> gets the desired attention.
>
> I've added the new condvar and rwlock as blockers.  We've been testing
> them for several weeks in Rawhide and haven't heard any complaints so
> far.  Carlos has reviewed the patches, basically.

The condvar layout change breaks on-disk Berkeley DB database 
environments.  The breakage is persistent across reboots.  Depending on 
the application which uses Berkeley DB, there might not be an easy way 
to recover affected database environments because the usual.

   <https://bugzilla.redhat.com/show_bug.cgi?id=1394862>

This is arguably a bug in Berkeley DB (after a reboot, the first thing 
applications have to do is to reinitialize synchronization objects 
because there definitely are no processes which map the objects left), 
but something we need to solve for Fedora 26 because it affects system 
upgrades.

It does not help that for licensing reasons, we use a quite old version 
of Berkeley DB in Fedora, but other users will be in a similar position 
because the Berkeley DB programming interface changes between releases 
in ways which are not completely source-compatible, so many applications 
cannot upgrade easily, either.

Note that I'm *not* saying we need to fix this in glibc through a compat 
symbol or with the help of flags.  We just need to offer *any* solution 
whatsoever.

Florian

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Joseph Myers]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

On Thu, 22 Dec 2016, Florian Weimer wrote:

> On 12/21/2016 10:18 PM, Joseph Myers wrote:
> > I'm dubious about keeping piling on "blockers" without evidence that there
> > are actually people to review the patches no later than 31 December
> > (earlier than that if revisions and re-review are needed).
> 
> My understanding of the meaning of “release blocker” and “freeze” is quite
> different.  An accepted release blocker can be fixed *in* the freeze period.
> As a result, I have been focusing on things that are clearly *not* release
> blockers, and my attention will shift after January 1st.

In my view, the freeze period is for architecture testing and 
stabilization, and people doing such testing need to be able to plan to 
start it on 1 January / 1 July, not on some unknown later date after 
"blocker" features have gone in.  Thus it's inappropriate to add tunables, 
new condvar and rwlock, or any other such substantial 
architecture-independent changes that would invalidate architecture 
testing, after 31 December.

The other things listed as blockers are architecture-specific so 
implementing them during the freeze should be OK if people testing on 
those architectures are OK with it.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Joseph Myers]

On Thu, 22 Dec 2016, Joseph Myers wrote:

> The other things listed as blockers are architecture-specific so 
> implementing them during the freeze should be OK if people testing on 
> those architectures are OK with it.

Also, the architecture-specific things look like bug fixes, which are OK 
during freeze in general (subject to the risk involved in a particular 
fix).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Torvald Riegel]

On Thu, 2016-12-22 at 14:12 +0100, Florian Weimer wrote:
> On 12/21/2016 09:19 PM, Torvald Riegel wrote:
> > On Tue, 2016-12-13 at 21:13 +0530, Siddhesh Poyarekar wrote:
> >> I had proposed tunables as a blocker for 2.25 during Cauldron and I want
> >> to stick with that position unless someone has a good reason to delay it
> >> for yet another release.  If there is anything else that needs attention
> >> but is not getting it then please ping early and ping often so that it
> >> gets the desired attention.
> >
> > I've added the new condvar and rwlock as blockers.  We've been testing
> > them for several weeks in Rawhide and haven't heard any complaints so
> > far.  Carlos has reviewed the patches, basically.
> 
> The condvar layout change breaks on-disk Berkeley DB database 
> environments.  The breakage is persistent across reboots.  Depending on 
> the application which uses Berkeley DB, there might not be an easy way 
> to recover affected database environments because the usual.
> 
>    <https://bugzilla.redhat.com/show_bug.cgi?id=1394862>
> 
> This is arguably a bug in Berkeley DB

My statement regarding "complaints" did not include buggy user programs.
Sorry if that wasn't clear.

Carlos' RFC about this, in which he said that he thinks this is a user
bug, only got one reply so far, in which Mike Frysinger agreed with
Carlos.  In the absence of further opinions to the contrary, I think we
have consensus that this is a bug in Berkeley DB.

> (after a reboot, the first thing 
> applications have to do is to reinitialize synchronization objects 
> because there definitely are no processes which map the objects left)

That's something a correct DB would do when recovering a database that
contains third-party synchronization primitives (eg, glibc condvars),
whose internals are supposed to be internals and not a public interface.

> Note that I'm *not* saying we need to fix this in glibc through a compat 
> symbol or with the help of flags.  We just need to offer *any* solution 
> whatsoever.

So what do you think this means for accepting the new condvar in 2.25?
Is having "something" a blocker for the condvar?  I would like to not
set the precedent that single buggy programs can affect glibc's
development; the emacs case was unfortunate enough.

We should also remember that not including the new condvar in 2.25 will
cause all users to have to use the old condvar which is not fully
conforming to POSIX.  Fixing them bug for them in a timely fashion seems
more important than giving Berkeley DB more time to fix its bug.

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Siddhesh Poyarekar]

On Thursday 22 December 2016 06:52 PM, Joseph Myers wrote:
> In my view, the freeze period is for architecture testing and 
> stabilization, and people doing such testing need to be able to plan to 
> start it on 1 January / 1 July, not on some unknown later date after 
> "blocker" features have gone in.  Thus it's inappropriate to add tunables, 
> new condvar and rwlock, or any other such substantial 
> architecture-independent changes that would invalidate architecture 
> testing, after 31 December.

While I understand the idea of only having bug fixes and regressions as
blockers, I am hopeful that making them blockers adds incentive for
other maintainers (and the release manager) to step in to review these
patches.  We don't currently have a mechanism for this  - there is the
"Desirable this Release" section, but it hardly seems to suggest the
kind of importance we might want to associate with key features that
have already missed multiple releases.  If the intent of "desirable..."
was indeed that, then we could just move condvar and tunables there and
hope that they don't miss yet another release.

Siddhesh

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Siddhesh Poyarekar]

On Thursday 22 December 2016 07:13 PM, Siddhesh Poyarekar wrote:
> While I understand the idea of only having bug fixes and regressions as
> blockers, I am hopeful that making them blockers adds incentive for

The 'them' refers to feature patchsets - sorry I wasn't clear.

Siddhesh

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Florian Weimer]

On 12/22/2016 02:30 PM, Torvald Riegel wrote:

>> Note that I'm *not* saying we need to fix this in glibc through a compat
>> symbol or with the help of flags.  We just need to offer *any* solution
>> whatsoever.
>
> So what do you think this means for accepting the new condvar in 2.25?
> Is having "something" a blocker for the condvar?  I would like to not
> set the precedent that single buggy programs can affect glibc's
> development; the emacs case was unfortunate enough.

It's definitely a bug Fedora (as a distribution) must solve before 
releasing a new version which is based on a glibc which has the new 
condvar.  It may affect other distributions using RPM.  The same issue 
affects self-compiled applications using Berkeley DB.

If someone could write an external tool which massages a dormant 
Berkeley DB environment (i.e., one that hasn't been opened by an 
application) so that it is compatible with the new condvar, and 
integrate that seamlessly into the Fedora update process to patch the 
RPM database, that would be a perfectly fine solution from my point of 
view.  (Bonus points if the tool can be run against arbitrary Berkeley 
DB environments, not just the RPM database.)

I'm pretty confident that such a tool could be written, but until we 
have demonstrated that it can be integrated into the Fedora update 
process, there is a risk that the new condvar implementation, as it 
exists today, is technically *impossible* to release as an in-place 
Fedora update.  (Some of the manual recovery steps can lead to 
unrecoverable data loss, unfortunately, so Fedora as a distribution 
really does not have much choice here and must find a technical, 
automated solution.  A release note will not be good enough.)

Of course, this is not your problem, and not the problem of glibc 
upstream project.  But it could well affected important glibc users.  It 
is definitely a problem for Fedora, and something Fedora and Red Hat 
have to solve.  And based on my experience, assistance from the 
toolchain team in this matter will be expected.  If we merge the new 
condvar as it exists today, and the tool cannot be written after all, 
then we'll have to maintain a suitably patched condvar fork in Fedora 
(and we'd likely need your help with that).  Other distributions which 
have encountered a similar problem will have to maintain a similar fork.

With Emacs, we delayed changing malloc until we were sure we found an 
approach which kept existing Emacs binaries working.  Similarly, we can 
make the condvar change in Fedora as long as it does not break in-place 
upgrades.

Thanks,
Florian

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Torvald Riegel]

On Thu, 2016-12-22 at 14:59 +0100, Florian Weimer wrote:
> there is a risk that the new condvar implementation, as it 
> exists today, is technically *impossible* to release as an in-place 
> Fedora update.

This would be bad for the practice of deploying glibc updates, I agree
with that.  However, not having a complete solution for that other
problem should not delay our testing of the new condvar.  We can always
revert the condvar closer to the 2.25 release.

> If we merge the new 
> condvar as it exists today, and the tool cannot be written after all, 
> then we'll have to maintain a suitably patched condvar fork in Fedora 
> (and we'd likely need your help with that).  Other distributions which 
> have encountered a similar problem will have to maintain a similar fork.

Or they would have to use an older or forked glibc for the affected
buggy program only.

Also, if such a tool cannot be written ever, then we're in a very bad
situation too because all users of condvars will have a slightly broken
condvar (because of the bug in the old condvar algorithm).  This might
even affect bdb too!

> With Emacs, we delayed changing malloc until we were sure we found an 
> approach which kept existing Emacs binaries working.  Similarly, we can 
> make the condvar change in Fedora as long as it does not break in-place 
> upgrades.

The difference is that the glibc change affecting Emacs was an
enhancement, not a bug fix.  The current condvar is not conforming to
POSIX.  Delaying bug fixes for everyone is bad.

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Florian Weimer]

On 12/22/2016 03:25 PM, Torvald Riegel wrote:

>> With Emacs, we delayed changing malloc until we were sure we found an
>> approach which kept existing Emacs binaries working.  Similarly, we can
>> make the condvar change in Fedora as long as it does not break in-place
>> upgrades.
>
> The difference is that the glibc change affecting Emacs was an
> enhancement, not a bug fix.  The current condvar is not conforming to
> POSIX.  Delaying bug fixes for everyone is bad.

The malloc change enabled us to fix powerpc and MIPS ABI.  Previously, 
the alignment returned by malloc was not correct and less than what was 
required by the ABI:

   <https://sourceware.org/bugzilla/show_bug.cgi?id=6527>

Florian

Re: 2.25 freeze - a little more than the holiday fortnight to go

[Torvald Riegel]

On Thu, 2016-12-22 at 15:44 +0100, Florian Weimer wrote:
> On 12/22/2016 03:25 PM, Torvald Riegel wrote:
> 
> >> With Emacs, we delayed changing malloc until we were sure we found an
> >> approach which kept existing Emacs binaries working.  Similarly, we can
> >> make the condvar change in Fedora as long as it does not break in-place
> >> upgrades.
> >
> > The difference is that the glibc change affecting Emacs was an
> > enhancement, not a bug fix.  The current condvar is not conforming to
> > POSIX.  Delaying bug fixes for everyone is bad.
> 
> The malloc change enabled us to fix powerpc and MIPS ABI.  Previously, 
> the alignment returned by malloc was not correct and less than what was 
> required by the ABI:

Well, in that case, I'd certainly have valued the bug fix higher than
keeping an inherently broken emacs hack working.