regression hunt status, question about reports to mailing list

regression hunt status, question about reports to mailing list

[Janis Johnson]

Should I continue to send mail to gcc@gcc.gnu.org when I identify the
patch that caused a regression or that fixed a mainline regression that
still exists on a release branch?  This seemed like a good idea when
there was one every couple of days, but I'm continuing to discover
short-cuts that make it faster and easier, so the trickle of mail might
be getting large enough to be annoying (I've got three more almost ready
to report this morning).

Besides sending these to the gcc list I've also been sending mail via
GNATS to include in the PR, which also goes to gcc-bugs.  If I stop
sending mail to the gcc list I'll be sure to add the patch submitter to
the list of recipients for the GNATS mail.

Wolfgang has been a great help in all of this.  He continues to send me
information about two-week windows when regressions came or went, plus
other information that's useful in deciding which ones to do next.  The
cut-down test cases that Wolfgang, Volker, and others have added to
GNATS are invaluable.

There are dozens more regressions to track down, including several for
platforms that I can't test, particularly sparc.  I'd be happy to answer
questions for other people who want to join in the hunt.

Janis

Re: regression hunt status, question about reports to mailing list

[Wolfgang Bangerth]

> Should I continue to send mail to gcc@gcc.gnu.org when I identify the
> patch that caused a regression or that fixed a mainline regression that
> still exists on a release branch?  This seemed like a good idea when
> there was one every couple of days, but I'm continuing to discover
> short-cuts that make it faster and easier, so the trickle of mail might
> be getting large enough to be annoying

I think you should. I guess, some nagging is ok, after all we haven't seen 
a rush to fix the regressions for which the offending patch has been 
identified. Yesterday, we had 90 (!) listed regressions, and little 
movement to get them fixed (except for Eric's noticable work on the 
optimization bugs). A little public pressure seems appropriate.

The only thing that worried me once was that there was a discussion on 
gcc@gcc.gnu.org after one of your mails, which is not recorded in the 
database however. But one might add a pointer to this discussion to the 
database.

Apart from that, after my search through the regression list, I am not 
concinced that you will be going on for long with your regression search. 
There don't seem to be so many other reports any more for which this kind 
of search seems appropriate. We need more people with knowledge of the 
internals of the compiler to do something.

To keep things in people's minds, and if the number of regressions does
not decrease noticably, you might want to send a summary by mid-January,
or so.

W.

-------------------------------------------------------------------------
Wolfgang Bangerth              email:           bangerth@ticam.utexas.edu
                               www: http://www.ticam.utexas.edu/~bangerth

Re: regression hunt status, question about reports to mailing list

[Janis Johnson]

On Fri, Dec 20, 2002 at 11:27:07AM -0600, Wolfgang Bangerth wrote:
> 
> (Janis wrote:)
> > Should I continue to send mail to gcc@gcc.gnu.org when I identify the
> > patch that caused a regression or that fixed a mainline regression that
> > still exists on a release branch?  This seemed like a good idea when
> > there was one every couple of days, but I'm continuing to discover
> > short-cuts that make it faster and easier, so the trickle of mail might
> > be getting large enough to be annoying
> 
> I think you should. I guess, some nagging is ok, after all we haven't seen 
> a rush to fix the regressions for which the offending patch has been 
> identified. Yesterday, we had 90 (!) listed regressions, and little 
> movement to get them fixed (except for Eric's noticable work on the 
> optimization bugs). A little public pressure seems appropriate.

The people I've notified are all very active contributors to GCC, and it
doesn't surprise me that they haven't fixed the regressions immediately.
I don't expect them to need nagging, but we should certainly remind them
about remaining regressions when we get close to deadlines.  It's also
quite possible that the people who fix the bugs will not be the ones
whose patches caused them to show up.

Janis

Re: regression hunt status, question about reports to mailing list

[Wolfgang Bangerth]

> > optimization bugs). A little public pressure seems appropriate.
> 
> The people I've notified are all very active contributors to GCC, and it
> doesn't surprise me that they haven't fixed the regressions immediately.

Please don't get me wrong, I very much appreciate the work everyone is 
doing on gcc, and I'm far from condemning anyone for doing one thing and 
not the other.

It's just that I see that there are more than 1800 open bug reports (of 
which alone 590 are for C++), and that reports are filed at a much higher 
rate than they are fixed. Sometimes I just wished we had more focus on 
fixing existing bugs than introducing yet another middle-end optimization. 
But that's just my opinion.

With public pressure I meant "bringing the existence of regressions and 
the number of open problems" to the attention it deserves. Not to anyone 
in particular, but to the community of developers.

Regards
  Wolfgang

-------------------------------------------------------------------------
Wolfgang Bangerth              email:           bangerth@ticam.utexas.edu
                               www: http://www.ticam.utexas.edu/~bangerth

Re: regression hunt status, question about reports to mailing list

[Joe Buck]

> It's just that I see that there are more than 1800 open bug reports (of 
> which alone 590 are for C++), and that reports are filed at a much higher 
> rate than they are fixed. Sometimes I just wished we had more focus on 
> fixing existing bugs than introducing yet another middle-end optimization. 
> But that's just my opinion.

To make progress against such a huge flow of bugs, we need to prioritize
better.  A switch to Bugzilla will make this easier.  The current
definition of any regression as priority "high", even if is an ICE on
illegal code that is almost line noise (see, for example, PR 8799),
makes it too easy to generate an unlimited supply of work.  We need
to prioritize better: which bugs are likely to harm users?

+: it's a duplicate, with two or more distinct testcases
++: wrong code is generated without any warning to the user
++: test case is a correctly coded, widely used free software program
-: it's not a valid program
-: there's a simple workaround
-: an obscure corner of the language is being tested

etc.

It's a useful service to narrow down the point where a regression appeared
to a single patch, but it's not a guarantee that this technique will find
a bug.  There are two possibilities: one is that the patch itself is
flawed; in such cases, if we're lucky the test case will quickly expose
what is wrong with the patch, but the issue might me more subtle.  The
other is that what we have is a "latent bug", as Kenner has often pointed
out: the patch merely perturbs something so that a bug in a completely
different part of the compiler is now triggered.  Reverting the patch
might not be a good choice, because we'd just be trading one bug for
another (the one that was fixed by the "offending" patch).  For this
reason we can't assume, just because we've isolated the apparently
offending patch, that the program can be immediately resolved.  We've
had this discussion before, and it does seem appropriate to put the
onus on a patch proposer to try to be sure that the patch is correct.
But if we find out later that there's an issue, deciding what to do is
trickier.

Re: regression hunt status, question about reports to mailing list

[Wolfgang Bangerth]

> > It's just that I see that there are more than 1800 open bug reports (of 
> > which alone 590 are for C++), and that reports are filed at a much higher 
> > rate than they are fixed. Sometimes I just wished we had more focus on 
> > fixing existing bugs than introducing yet another middle-end optimization. 
> > But that's just my opinion.
> 
> To make progress against such a huge flow of bugs, we need to prioritize
> better.  A switch to Bugzilla will make this easier.

The situation is not improved by the continuing delays in introducing 
Bugzilla. I am really looking forward to have it.

Two observations though:
- The fact that we have so many reports shouldn't be so surpring. It's not
  as if a tidal wave is suddenly crashing onto us -- these reports have 
  been accumulating for a long time.
- It seems true that we have an army of people hacking on various new
  optimization steps on several branches, while we have only a handful
  that try to fix the existing bugs. Whether this situation changes
  noticably when we have more options for prioritizing remains to be
  seen, but I'm not too optimistic in this respect. (Besides: who's going
  to assign priorities to 1800+ reports?)


> It's a useful service to narrow down the point where a regression appeared
> to a single patch, but it's not a guarantee that this technique will find
> a bug.

I don't think we expect that the bug will be fixed by the same person the
next day. At least I understand finding the causing patch as one way to
generate additional data points, to make searching for the real cause
simpler for those who know about the compiler. Just like stripping down a
testcase from 30,000 lines to 20.

I don't disagree with you on the relevance of knowing this patch.  
Unfortunately, searching for the patches that cause a regression, and
helping to strip down testcases to a managable size is all I can do, not
knowing anything about gcc's internals.

Regards
  Wolfgang

-------------------------------------------------------------------------
Wolfgang Bangerth              email:           bangerth@ticam.utexas.edu
                               www: http://www.ticam.utexas.edu/~bangerth

Re: regression hunt status, question about reports to mailing list

[Mark Mitchell]

--On Friday, December 20, 2002 09:14:45 AM -0800 Janis Johnson 
<janis187@us.ibm.com> wrote:

> Should I continue to send mail to gcc@gcc.gnu.org when I identify the
> patch that caused a regression or that fixed a mainline regression that
> still exists on a release branch?  This seemed like a good idea when
> there was one every couple of days, but I'm continuing to discover
> short-cuts that make it faster and easier, so the trickle of mail might
> be getting large enough to be annoying (I've got three more almost ready
> to report this morning).

I think what you're doing is tremendously useful.  I'd just like you to
keep doing it. :-)

-- 
Mark Mitchell                mark@codesourcery.com
CodeSourcery, LLC            http://www.codesourcery.com

Re: regression hunt status, question about reports to mailing list

[Tom Lord]

   Janis Johnsen:

   > Should I continue to send mail to gcc@gcc.gnu.org when I identify the
   > patch that caused a regression or that fixed a mainline regression that
   > still exists on a release branch?  This seemed like a good idea when
   > there was one every couple of days, but I'm continuing to discover
   > short-cuts that make it faster and easier, so the trickle of mail might
   > be getting large enough to be annoying (I've got three more almost ready
   > to report this morning).


   Mark Mitchell:

   I think what you're doing is tremendously useful.  I'd just like you to
   keep doing it. :-)


With changeset-oriented revision control and a bug tracking system
that produces machine processable results (I still think QMtest is
needlessly complicated but it does have that property), the work Janis
is doing could be fully automated, completely precise, and
continuously performed.  And more besides (how about identifying a
patch on anyone's branch that fixes a mainline regression).


"This Way Up",
-t

Re: regression hunt status, question about reports to mailing list

[Kaveh R. Ghazi]

 > From: Janis Johnson <janis187 at us dot ibm dot com> 
 > 
 > There are dozens more regressions to track down, including several for
 > platforms that I can't test, particularly sparc.
 > Janis

Hi Janis,

I was wondering if you could utilize your hunting method with a cross
compiler.  If a given regression was e.g. an ICE and reproducable when
targetting the appropriate platform, if you were supplied with the .i
file and appropriate flags, then I'm thinking it should be possible.

If so, would you be able to track down the critical patch point for
some of the sparc regressions I've filed?

I.e. PRs 7227, 7794, 7796, perhaps more filed elsewhere by others.  I
can supply the specific info and .i file if the PRs aren't enough.

I think this would be a tremendously valuable addition to your already
helpful hunting.

		Thanks,
		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: regression hunt status, question about reports to mailing list

[Janis Johnson]

On Thu, Dec 26, 2002 at 04:48:31PM -0500, Kaveh R. Ghazi wrote:
>  > From: Janis Johnson <janis187 at us dot ibm dot com> 
>  > 
>  > There are dozens more regressions to track down, including several for
>  > platforms that I can't test, particularly sparc.
>  > Janis
> 
> Hi Janis,
> 
> I was wondering if you could utilize your hunting method with a cross
> compiler.  If a given regression was e.g. an ICE and reproducable when
> targetting the appropriate platform, if you were supplied with the .i
> file and appropriate flags, then I'm thinking it should be possible.
> 
> If so, would you be able to track down the critical patch point for
> some of the sparc regressions I've filed?
> 
> I.e. PRs 7227, 7794, 7796, perhaps more filed elsewhere by others.  I
> can supply the specific info and .i file if the PRs aren't enough.
> 
> I think this would be a tremendously valuable addition to your already
> helpful hunting.

I was rather hoping that by providing information about how to do the
hunts, as well as the framework script, other people would join in.
I'm planning to go through my scripts that update cvs, do partial
builds, and run tests to provide some representative examples of those,
although they're all pretty straightforward.

If no one else jumps in, though, I could try some searches with cross
compilers; there's no reason that shouldn't work.  I'm on vacation
until January 2, occasionally running simple hunts while I'm doing
vacation-type things at home.

I'm running into several regressions whose searches turn up nonsensical
patches.  These are apparently bugs that show up randomly, rather than
genuine regressions.  Some of them get different results depending on
whether I run them in the foreground or from a script.  I'll look into
those and report their behavior in GNATS.  It's possible that some of
the patches I've already reported also fall into that category.

Janis