Release Quality Control (was: DOC PATCH: GCC 3.2.3 release schedule)

Release Quality Control (was: DOC PATCH: GCC 3.2.3 release schedule)

[Gerald Pfeifer]

On Thu, 27 Mar 2003, Gabriel Dos Reis wrote:
>| [...] considering the large amount of changes on the branch, I'd like
>| to suggest we try to get test-results from all primary and secondary
>| platforms, at least (but this is something where we depend on many
>| volunteers).
> I think your suggestion is an excellent one.
>
> I can manage to setup testing on solaris-2.[67]/SPARC and GNU/Linux
> here before leaving.

That's good to have!

> I, unfortunately, do not have access to the other primary and
> secondary plateforms so I would welcome any help.

I thought about it during a long run today, and I believe we really need
to change the way we are performing quality control for releases, even if
that may mean delaying a release by another week.

Until now, we basically said ``Nobody has complained that the release
branch is broken on his platform, so let's go'' while we should wait
to receive positive confirmation for every single primary and secondary
platform (with a timeout for secondary platforms, but none for primary
platforms).


But perhaps you and Mark have been handling it that way all the time
anyways (just ignore this message then), but I noticed this was not
documented at <http://gcc.gnu.org/releasing.html>.

If you, Mark and Gaby, agree, I'll add this to that page; if not, let's
discuss it. ;-)

Gerald
-- 
Gerald "Jerry"   pfeifer@dbai.tuwien.ac.at   http://www.pfeifer.com/gerald/

Re: Release Quality Control (was: DOC PATCH: GCC 3.2.3 release schedule)

[Joseph S. Myers]

On Sun, 6 Apr 2003, Gerald Pfeifer wrote:

> Until now, we basically said ``Nobody has complained that the release
> branch is broken on his platform, so let's go'' while we should wait
> to receive positive confirmation for every single primary and secondary
> platform (with a timeout for secondary platforms, but none for primary
> platforms).

Positive confirmation with testresults for the current testsuite on both
the to-be-released version and the previous versions mentioned in the
release criteria relative to which there are meant to be no regressions?
This used to be done and we should at least try to avoid regressions that
the testsuite shows up.

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: Release Quality Control (was: DOC PATCH: GCC 3.2.3 release schedule)

[Gabriel Dos Reis]

Gerald Pfeifer <pfeifer@dbai.tuwien.ac.at> writes:

[...]

| If you, Mark and Gaby, agree, I'll add this to that page; if not, let's
| discuss it. ;-)

Like many people here, I would like to see quality improvement
increase at least proportionally (by a large factor) as releases go.
Your suggestion of waiting for positive confirmations on primary and
secondary plateforms is certainly a good one.  I would like to hear
other people's opinions. 

-- Gaby

Re: Release Quality Control (was: DOC PATCH: GCC 3.2.3 release schedule)

[Joe Buck]

On Sat, Apr 12, 2003 at 04:01:00PM +0200, Gabriel Dos Reis wrote:
> Like many people here, I would like to see quality improvement
> increase at least proportionally (by a large factor) as releases go.
> Your suggestion of waiting for positive confirmations on primary and
> secondary plateforms is certainly a good one.  I would like to hear
> other people's opinions. 

The distinction between primary and secondary platforms that the SC
agreed apon is this: for a primary platform, any critical issues will hold up
a release (at least until we decide to ship anyway), but for a secondary
platform, all we say is that we have a dedicated volunteer who will
run the tests and do the best he/she can to draw attention to problems
and try to get them fixed (or recruit others to fix them).  We've been
a bit sloppy about this in the past, but what this should mean for you
as RM is to pay attention to the primary platforms; the secondary
platforms are Not Your Problem (though you can leave a window of
time for people to get fixes for those platforms into the release).

To be precise, this is my recollection of what we agreed to, it may
not be precise.

I would suggest waiting for a successful build and test from the tarball
on each primary platform.  For secondaries, if there's a bootstrap
failure and someone produces a clearly safe patch that fixes it, by
all means include it if it arrives early enough.

Re: Release Quality Control (was: DOC PATCH: GCC 3.2.3 release schedule)

[Mark Mitchell]

On Mon, 2003-04-14 at 09:30, Joe Buck wrote:
> On Sat, Apr 12, 2003 at 04:01:00PM +0200, Gabriel Dos Reis wrote:
> > Like many people here, I would like to see quality improvement
> > increase at least proportionally (by a large factor) as releases go.
> > Your suggestion of waiting for positive confirmations on primary and
> > secondary plateforms is certainly a good one.  I would like to hear
> > other people's opinions. 
> 
> The distinction between primary and secondary platforms that the SC
> agreed apon is this: for a primary platform, any critical issues will hold up
> a release (at least until we decide to ship anyway), but for a secondary
> platform, all we say is that we have a dedicated volunteer who will
> run the tests and do the best he/she can to draw attention to problems
> and try to get them fixed (or recruit others to fix them).  We've been
> a bit sloppy about this in the past, but what this should mean for you
> as RM is to pay attention to the primary platforms; the secondary
> platforms are Not Your Problem (though you can leave a window of
> time for people to get fixes for those platforms into the release).
> 
> To be precise, this is my recollection of what we agreed to, it may
> not be precise.

That's my recollection as well.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com

Re: Release Quality Control

[Gerald Pfeifer]

On Mon, 14 Apr 2003, Mark Mitchell wrote:
>> To be precise, this is my recollection of what we agreed to, it may
>> not be precise.
> That's my recollection as well.

Mine, too.  But the point I was trying to make was the following:

 o As far as I am aware of(!), in the past we didn't actually ensure our
 quality criteria for primary platforms, for example, in that we did not
 solicit   current test results etc. for primary platforms where these
 were missing and explicitly verify that everything was fine.

 o Even if secondary platform do not absolutely need to work, we could try
 to proactively solicit test results (more) proactively, even for tertiary
 platforms.

That's all I had in mind, and especially given the problems with GCC 3.2.2
on a couple of platforms I still think it's a good idea to put a bit more
emphasis on that.

(One simple way would be to announce some "last" pre-release snapshot
on the gcc-announce list and ask for testers, waiting, say, one week
before the actual release.)

Gerald
-- 
Gerald "Jerry"   pfeifer@dbai.tuwien.ac.at   http://www.pfeifer.com/gerald/

Re: Release Quality Control

[Joe Buck]

On Mon, Apr 14, 2003 at 10:04:29PM +0200, Gerald Pfeifer wrote:
>  o As far as I am aware of(!), in the past we didn't actually ensure our
>  quality criteria for primary platforms, for example, in that we did not
>  solicit   current test results etc. for primary platforms where these
>  were missing and explicitly verify that everything was fine.

Let's try to avoid creating new bureaucracy.

In my opinion, the RM is in the best position to keep track of this, or
to ask for assistance if the bookkeeping is too much.  He is the one
with the best feeling for what the deadlines are going to be.

>  o Even if secondary platform do not absolutely need to work, we could try
>  to proactively solicit test results (more) proactively, even for tertiary
>  platforms.

When the RM puts out a prerelease tarball, this should be interpreted as a
request that the keepers of the secondary platforms do tests, ASAP.
This could be explicitly mentioned when Gaby or Mark puts out an
announcement, but it should be understood in any case.  If we go through
three prerelease tarballs and it isn't noticed that, say, Solaris X86
doesn't bootstrap, well, it wasn't a primary platform and no one who cared
about it bothered to do tests, so sorry.  The lesson to be learned is that
people who do care will have an incentive to get involved next time.

> (One simple way would be to announce some "last" pre-release snapshot
> on the gcc-announce list and ask for testers, waiting, say, one week
> before the actual release.)

I don't think that this is workable.  Every pre-release tarball should be
treated as potentially "last": if it's OK, it ships, if not, there's
another one, until the criteria are met.

The 3.2.3-pre tarball Gaby put out has problems on Solaris, so we'll need
at least one more.  But if there's a secondary platform you (general you,
not Gerald) care about and there isn't an already posted test result, I
strongly suggest:

1) try to build and test Gaby's prerelease
2) if that fails, try to build and test from the CVS 3.2 branch
3) let the gcc list know about the result!

If you run a minority platform and no user of that platform does this, it
is likely that 3.2.3 won't work on your platform, and it will not be
Gaby's or Mark's fault.

Re: Release Quality Control

[Gerald Pfeifer]

On Mon, 14 Apr 2003, Joe Buck wrote:
> When the RM puts out a prerelease tarball, this should be interpreted as a
> request that the keepers of the secondary platforms do tests, ASAP.

Point is, we do _not_ have a sensible way of allowing people to follow
GCC (and be it just for doing this kind of test) without having to keep
up with the high volume mailing lists.

This is fine for those of us who work on GCC for a living or who spend
much (if not most) of their spare time working on GCC, but it is becoming
increasingly hard to be a tester for a secondary platform or just track
GCC decently.

> If we go through three prerelease tarballs and it isn't noticed that,
> say, Solaris X86 doesn't bootstrap, well, it wasn't a primary platform
> and no one who cared about it bothered to do tests, so sorry.  The
> lesson to be learned is that people who do care will have an incentive
> to get involved next time.

...but have no way of getting the relevant information ("Oops, I should
test now") easily.

> If you run a minority platform and no user of that platform does this,
> it is likely that 3.2.3 won't work on your platform, and it will not be
> Gaby's or Mark's fault.

Sure.  Though still we should try to make the entry level of contributing
and testing not too high.

Both are serious problems these days, not the least due to the sheer
amount of develoment going on (which is, mostly, a good thing per se).

Gerald
-- 
Gerald "Jerry"   pfeifer@dbai.tuwien.ac.at   http://www.pfeifer.com/gerald/

Re: Release Quality Control

[Joel Sherrill]

Gerald Pfeifer wrote:
> 
> On Mon, 14 Apr 2003, Joe Buck wrote:
> > When the RM puts out a prerelease tarball, this should be interpreted as a
> > request that the keepers of the secondary platforms do tests, ASAP.
> 
> Point is, we do _not_ have a sensible way of allowing people to follow
> GCC (and be it just for doing this kind of test) without having to keep
> up with the high volume mailing lists.
> 
> This is fine for those of us who work on GCC for a living or who spend
> much (if not most) of their spare time working on GCC, but it is becoming
> increasingly hard to be a tester for a secondary platform or just track
> GCC decently.

I try to keep up with this list and it is very difficult.  There is a
lot of
volume and a few days of any real life distraction are deadly.

> > If we go through three prerelease tarballs and it isn't noticed that,
> > say, Solaris X86 doesn't bootstrap, well, it wasn't a primary platform
> > and no one who cared about it bothered to do tests, so sorry.  The
> > lesson to be learned is that people who do care will have an incentive
> > to get involved next time.
> 
> ...but have no way of getting the relevant information ("Oops, I should
> test now") easily.

I don't know if gcc-announce is the appropriate place to make these
announcements
but if it isn't, then a dedicated list for "please test announcements"
might
be useful.  Limit posts to the RMs.
 
> > If you run a minority platform and no user of that platform does this,
> > it is likely that 3.2.3 won't work on your platform, and it will not be
> > Gaby's or Mark's fault.
> 
> Sure.  Though still we should try to make the entry level of contributing
> and testing not too high.
> 
> Both are serious problems these days, not the least due to the sheer
> amount of develoment going on (which is, mostly, a good thing per se).

I certainly like this part but the volume of email to keep up with is
pretty high just to make sure I test at the right times.  It is really
easy to miss one.

> Gerald
> --
> Gerald "Jerry"   pfeifer@dbai.tuwien.ac.at   http://www.pfeifer.com/gerald/

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel@OARcorp.com                 On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

Re: Release Quality Control

[Michael S. Zick]

On Thursday 17 April 2003 12:28 pm, Joel Sherrill wrote:
> Gerald Pfeifer wrote:
> > On Mon, 14 Apr 2003, Joe Buck wrote:
> > > When the RM puts out a prerelease tarball, this should be interpreted
> > > as a request that the keepers of the secondary platforms do tests,
> > > ASAP.
- - - -
>
> I don't know if gcc-announce is the appropriate place to make these
> announcements
> but if it isn't, then a dedicated list for "please test announcements"
> might
> be useful.  Limit posts to the RMs.
>
Perhaps two ways could be used -

1) An (auto?) notice sent to the mailing list(s) similar to the "snapshot"
notices.  (auto? - well, triggered by to RM, just auto to the right places.)
That should handle those with a casual interest in testing (list followers).

2) An (auto?) notice sent to (private e-mail of) of those that have asked
the RM to include them on such a (RM only access) list.
That should handle those with a need for "test it now" notices and 
might be otherwise too busy to follow a ML.

Mike
PS: The "CD-ROM of the CVS, rsync image" project to help kickstart
new regression testers isn't dead here;
I am just painfully slow in getting it done.

Re: Release Quality Control

[Joe Buck]

On Thu, Apr 17, 2003 at 07:21:23PM +0200, Gerald Pfeifer wrote:
> On Mon, 14 Apr 2003, Joe Buck wrote:
> > When the RM puts out a prerelease tarball, this should be interpreted as a
> > request that the keepers of the secondary platforms do tests, ASAP.
> 
> Point is, we do _not_ have a sensible way of allowing people to follow
> GCC (and be it just for doing this kind of test) without having to keep
> up with the high volume mailing lists.

We could set up a special mailing list for the purpose (gcc-prereleases
or some such).  However, if the secondary platform owners don't follow
the gcc list, and don't test anything on the CVS branch until the first
pre-release tarball comes out, I'm afraid that what's going to happen is
that the code just isn't going to work.  If people don't get involved
earlier than that, bitrot sets in and the code breaks.

Re: Release Quality Control

[Gabriel Dos Reis]

Joe Buck <jbuck@synopsys.com> writes:

| On Thu, Apr 17, 2003 at 07:21:23PM +0200, Gerald Pfeifer wrote:
| > On Mon, 14 Apr 2003, Joe Buck wrote:
| > > When the RM puts out a prerelease tarball, this should be interpreted as a
| > > request that the keepers of the secondary platforms do tests, ASAP.
| > 
| > Point is, we do _not_ have a sensible way of allowing people to follow
| > GCC (and be it just for doing this kind of test) without having to keep
| > up with the high volume mailing lists.
| 
| We could set up a special mailing list for the purpose (gcc-prereleases
| or some such).


Is there a reason why gcc-announce will not be appropriate?  

| However, if the secondary platform owners don't follow
| the gcc list, and don't test anything on the CVS branch until the first
| pre-release tarball comes out, I'm afraid that what's going to happen is
| that the code just isn't going to work.

I can't argue with that :-)

-- Gaby

Re: Release Quality Control

[Gerald Pfeifer]

On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
>| We could set up a special mailing list for the purpose (gcc-prereleases
>| or some such).
> Is there a reason why gcc-announce will not be appropriate?

Well, actually... Let's try it, for the next snapshot?

Gerald
-- 
Gerald "Jerry"   pfeifer@dbai.tuwien.ac.at   http://www.pfeifer.com/gerald/

Re: Release Quality Control

[Joe Buck]

On Fri, Apr 18, 2003 at 06:47:31PM +0200, Gerald Pfeifer wrote:
> On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
> >| We could set up a special mailing list for the purpose (gcc-prereleases
> >| or some such).
> > Is there a reason why gcc-announce will not be appropriate?
> 
> Well, actually... Let's try it, for the next snapshot?

Maybe we could start it for the 3.3 release process.  It would need to
be widely announced, and we'd need to come up with an appropriate blurb
to set people's expectations correctly: that the tarballs being announced
are prereleases for testing, don't use them for production; that there is
no commitment to delaying the actual release if problems are found on
non-primary platforms, etc.

Re: Release Quality Control

[David Taylor]

> Date: Fri, 18 Apr 2003 10:24:42 -0700
> From: Joe Buck <jbuck@synopsys.com>
> 
> On Fri, Apr 18, 2003 at 06:47:31PM +0200, Gerald Pfeifer wrote:
> > On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
> > >| We could set up a special mailing list for the purpose (gcc-prereleases
> > >| or some such).
> > > Is there a reason why gcc-announce will not be appropriate?
> > 
> > Well, actually... Let's try it, for the next snapshot?
> 
> Maybe we could start it for the 3.3 release process.  It would need to
> be widely announced, and we'd need to come up with an appropriate blurb
> to set people's expectations correctly: that the tarballs being announced
> are prereleases for testing, don't use them for production; that there is
> no commitment to delaying the actual release if problems are found on
> non-primary platforms, etc.

If you're going to wait, may I suggest what gdb does?  Create a low
volume list for that explicit purpose?

. GDB has a low volume list gdb-testers, announcements of pre-releases
  are sent there as well as being sent other places.

. anyone submitting either multiple GDB bug reports or engaging in
  discussion of a bug report that they had submitted was explictly
  invited to join the list.  (I don't know if this still happens.)

  [Everyone was premitted to join the list, but since many people
  submitting bug reports didn't know about the list, and such people
  had already shown an interest (by submitting and discussing bug
  reports) in helping to improve the quality of the program, it was
  thought that they might be interested in being pre-release testers...
  so, it wouldn't hurt to explicitly invite them to join.]

David
--
David Taylor
dtaylor@emc.com, taylor@candd.org

Re: Release Quality Control

[Gabriel Dos Reis]

Gerald Pfeifer <pfeifer@dbai.tuwien.ac.at> writes:

| On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
| >| We could set up a special mailing list for the purpose (gcc-prereleases
| >| or some such).
| > Is there a reason why gcc-announce will not be appropriate?
| 
| Well, actually... Let's try it, for the next snapshot?

I would suggest we start it with 3.3.0 prerealeses.  Mark?

-- Gaby

Re: Release Quality Control

[Gabriel Dos Reis]

David Taylor <dtaylor@emc.com> writes:

| > Date: Fri, 18 Apr 2003 10:24:42 -0700
| > From: Joe Buck <jbuck@synopsys.com>
| > 
| > On Fri, Apr 18, 2003 at 06:47:31PM +0200, Gerald Pfeifer wrote:
| > > On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
| > > >| We could set up a special mailing list for the purpose (gcc-prereleases
| > > >| or some such).
| > > > Is there a reason why gcc-announce will not be appropriate?
| > > 
| > > Well, actually... Let's try it, for the next snapshot?
| > 
| > Maybe we could start it for the 3.3 release process.  It would need to
| > be widely announced, and we'd need to come up with an appropriate blurb
| > to set people's expectations correctly: that the tarballs being announced
| > are prereleases for testing, don't use them for production; that there is
| > no commitment to delaying the actual release if problems are found on
| > non-primary platforms, etc.
| 
| If you're going to wait, may I suggest what gdb does?  Create a low
| volume list for that explicit purpose?
| 
| . GDB has a low volume list gdb-testers, announcements of pre-releases
|   are sent there as well as being sent other places.

This pretty much matches what Joe was initially proposing before I
mentioned re-using gcc-announce.  It seems like there is a precendent,
so I'm going to withdraw my gcc-announce proposal.

-- Gaby

Re: Release Quality Control

[Mark Mitchell]

On Mon, 2003-04-21 at 06:46, Gabriel Dos Reis wrote:
> Gerald Pfeifer <pfeifer@dbai.tuwien.ac.at> writes:
> 
> | On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
> | >| We could set up a special mailing list for the purpose (gcc-prereleases
> | >| or some such).
> | > Is there a reason why gcc-announce will not be appropriate?
> | 
> | Well, actually... Let's try it, for the next snapshot?
> 
> I would suggest we start it with 3.3.0 prerealeses.  Mark?

That's fine by me.  I'm happy with as much testing as possible!

But we still have some bugs that really need fixing.

I'm about to check in another compile-time improvement patch, but there
are a lot of target-specific details we need to look at carefully.  

I plan on triaging some of these down to medium-priority this week, as
we did with the 3.2 release.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com

Re: Release Quality Control

[Gabriel Dos Reis]

Mark Mitchell <mark@codesourcery.com> writes:

| On Mon, 2003-04-21 at 06:46, Gabriel Dos Reis wrote:
| > Gerald Pfeifer <pfeifer@dbai.tuwien.ac.at> writes:
| > 
| > | On Thu, 17 Apr 2003, Gabriel Dos Reis wrote:
| > | >| We could set up a special mailing list for the purpose (gcc-prereleases
| > | >| or some such).
| > | > Is there a reason why gcc-announce will not be appropriate?
| > | 
| > | Well, actually... Let's try it, for the next snapshot?
| > 
| > I would suggest we start it with 3.3.0 prerealeses.  Mark?
| 
| That's fine by me.  I'm happy with as much testing as possible!

Great!.

| But we still have some bugs that really need fixing.

Agreed.  Once 3.2.3 is out, I will request the SC to move the
gcc-3_2-branch into a "closed state" and I will ask everybody who
helped with 3.2.3 release to move resources to 3.3.0 -- I myself will
certainly do so.

| I'm about to check in another compile-time improvement patch, but there

I'm looking forward to seeing it -- as I too have some patches along
those lines to submit.

| are a lot of target-specific details we need to look at carefully.  
| 
| I plan on triaging some of these down to medium-priority this week, as
| we did with the 3.2 release.

I'm waiting to see what will get down before commenting :-).

Thanks,

-- Gaby

Re: Release Quality Control

[Joe Buck]

On Mon, Apr 21, 2003 at 06:58:54PM +0200, Gabriel Dos Reis wrote:
> Agreed.  Once 3.2.3 is out, I will request the SC to move the
> gcc-3_2-branch into a "closed state" and I will ask everybody who
> helped with 3.2.3 release to move resources to 3.3.0 -- I myself will
> certainly do so.

Gaby, we gave you control of the 3.2 branch; as far as I'm concerned,
you can declare it closed on your own authority.

I agree that 3.2.3 should be the end of the line, unless we have
a "paper bag release" incident, to use Linus Torvalds' phrase.