Print KFAIL's in dejagnu summary?

Print KFAIL's in dejagnu summary?

[Andrew Cagney]

Hello,

At present KFAILs are supressed from the summary output (the stuff on 
the terminal from "make check").  I'd like to change this so that 
KFAILs, just like FAILs, are included in the summary.  A KFAIL, just 
like a FAIL, indicates a bug in the system under test, and hence should 
be included in the summary.

Having seen this feature in action for a year now, I think it's 
reasonable to conclude that people are ignoring KFAILed tests just like 
they ignored GDB's bogus XFAIL tests that went before.

This would mean that any summary output - XPASS, KFAIL, KPASS, and ERROR 
would need action.

thoughts?
Andrew

PS: Some [non] history.  Neither Fernando nor I can figure out how/why 
the current behavior came to be.  Our best guess is that there was some 
flip-flopping (the lists suggest this) and the current behavior was 
chosen because it happened to be that way in the last version of the 
patch.  Enjoy!

Re: Print KFAIL's in dejagnu summary?

[David Carlton]

On Thu, 25 Sep 2003 17:44:27 -0400, Andrew Cagney <ac131313@redhat.com> said:

> At present KFAILs are supressed from the summary output (the stuff
> on the terminal from "make check").  I'd like to change this so that
> KFAILs, just like FAILs, are included in the summary.  A KFAIL, just
> like a FAIL, indicates a bug in the system under test, and hence
> should be included in the summary.

I have a mild preference for the current behavior.  Mostly I use the
summary output to get a feel for whether or not a change of mine has
obviously gone wrong; the noisier the summary output is, the harder it
is to use it this way.  Of course, I always search the entire gdb.sum
for regressions, just to make sure, so it won't make a big practical
difference to me one way or another.

I also think that, right now, a bigger issue is to get the testsuite
and GNATS to agree about what the bugs are in GDB.  So I'm more
interesting in removing the undiagnosed XFAILs and in making some
effort to diagnose existing FAILs than I am in making the KFAILs more
prominent.  (I'd also like to see some effort towards making sure that
open bugs in GNATS are reflected via KFAILs in the test suite.)  It
would also be nice if we had a feel for what bugs are low-hanging
fruit.

The part of the test suite that I pay the most attention to is, of
course, gdb.cp; I haven't noticed that the addition of KFAILs there
has had much of an effect on the rate of bug fixing.  (There are a lot
of old bugs that are still present, but they'd been present for a
while.)  Of course, a lot of the KFAIL additions are from tests that
used to be XFAILs, which didn't change the visibility level of the
bugs.

Some more random musings about the possible roles of the testsuite:

* Its main role is to catch regressions.  Here, I think that making
  FAILs more prominent is good: a FAIL is more likely to be a
  regression than a KFAIL (at least if people delete the appropriate
  setup_kfails from the test suite when they fix bugs).

* It's also useful for people trying to fix bugs: if you've decided to
  fix bug X and if the testsuite contains a KFAIL associated to X,
  then that makes it a lot easier to check if you've fixed the bug
  (and to try to figure out what's causing the bug, for that matter).
  Here, too, there's no reason for KFAILs to be prominent.

* Another role is to remind people of the existence of bugs, to exhort
  people to fix them; that's important to you, but a little less
  important to me than the other two roles.

David Carlton
carlton@kealia.com

Re: Print KFAIL's in dejagnu summary?

[Jim Blandy]

Andrew Cagney <ac131313@redhat.com> writes:
> At present KFAILs are supressed from the summary output (the stuff on
> the terminal from "make check").  I'd like to change this so that
> KFAILs, just like FAILs, are included in the summary.  A KFAIL, just
> like a FAIL, indicates a bug in the system under test, and hence
> should be included in the summary.
> 
> Having seen this feature in action for a year now, I think it's
> reasonable to conclude that people are ignoring KFAILed tests just
> like they ignored GDB's bogus XFAIL tests that went before.

I think this might be a good thing.  They certainly are failures.

I'm usually working in one of two modes:

- I'm evaluating a particular change.  Here I do before-and-after
  comparisons of the .sum files, to look for regressions; what appears
  in the summary output makes no difference to me at all, so the
  change you suggest wouldn't affect me.

- I'm stabilizing a target for a release.  Here, every failure is of
  potential interest to me, and KFAILs are no exception.  But I
  usually work from the gdb.sum files in this situation too, so again,
  it doesn't make much difference.

When you say "I think it's reasonable to conclude that people are
ignoring KFAILed tests", I get this image of someone running a 'make
check', being shocked to see a blip in the output there, and getting
hot and bothered about fixing it.  But it's hard for me to imagine
someone actually working in such an impulsive way; bugs take (me) too
long to fix to just dive in when I hadn't planned on it.  I'm always
more directed about what I'm going to work on.  I do other things for
fun.

If other folks are like me, then I don't think the change you suggest
will have much effect on the rate at which bugs are fixed in GDB.
(Not that that was the only point in its favor.)

Re: Print KFAIL's in dejagnu summary?

[Andrew Cagney]

> On Thu, 25 Sep 2003 17:44:27 -0400, Andrew Cagney <ac131313@redhat.com> said:
> 
> 
>> At present KFAILs are supressed from the summary output (the stuff
>> on the terminal from "make check").  I'd like to change this so that
>> KFAILs, just like FAILs, are included in the summary.  A KFAIL, just
>> like a FAIL, indicates a bug in the system under test, and hence
>> should be included in the summary.
> 
> 
> I have a mild preference for the current behavior.  Mostly I use the
> summary output to get a feel for whether or not a change of mine has
> obviously gone wrong; the noisier the summary output is, the harder it
> is to use it this way.  Of course, I always search the entire gdb.sum
> for regressions, just to make sure, so it won't make a big practical
> difference to me one way or another.

The numbers at the bottom should tell you that:

- no errors
- no unexpected passes
- no unknown failures.

unfortunatly, the only truely robust way is to compare the .sum files.

Andrew

Re: Print KFAIL's in dejagnu summary?

[David Carlton]

On Sat, 27 Sep 2003 10:24:16 -0400, Andrew Cagney <ac131313@redhat.com> said:
>> On Thu, 25 Sep 2003 17:44:27 -0400, Andrew Cagney <ac131313@redhat.com> said:

>>> At present KFAILs are supressed from the summary output (the stuff
>>> on the terminal from "make check").  I'd like to change this so
>>> that KFAILs, just like FAILs, are included in the summary.  A
>>> KFAIL, just like a FAIL, indicates a bug in the system under test,
>>> and hence should be included in the summary.

>> I have a mild preference for the current behavior.  Mostly I use
>> the summary output to get a feel for whether or not a change of
>> mine has obviously gone wrong; the noisier the summary output is,
>> the harder it is to use it this way.  Of course, I always search
>> the entire gdb.sum for regressions, just to make sure, so it won't
>> make a big practical difference to me one way or another.

> The numbers at the bottom should tell you that:

> - no errors
> - no unexpected passes
> - no unknown failures.

Right, I'm just saying that I sometimes like to look at the output
while the test run is in progress.  That's pretty much the only time
that I do look at the output: I never pay attention to the numbers at
the bottom, because once I get to that stage, I have the entire
gdb.sum file available for use.

> unfortunatly, the only truely robust way is to compare the .sum
> files.

Yes; that's one reason that my preference for the current behavior is
only mild.  What would make more of a difference for me (and,
presumably, for other people) would be to make it as easy as possible
to compare sum files: regularize the thread output a bit more, get rid
of gratuitous numbers (especially the value of $fp in pc-fp.exp).

David Carlton
carlton@kealia.com

Re: Print KFAIL's in dejagnu summary?

[Rob Savoye]

On Thu, Sep 25, 2003 at 05:44:27PM -0400, Andrew Cagney wrote:

> PS: Some [non] history.  Neither Fernando nor I can figure out how/why 
> the current behavior came to be.  Our best guess is that there was some 
> flip-flopping (the lists suggest this) and the current behavior was 
> chosen because it happened to be that way in the last version of the 
> patch.  Enjoy!

  I think you're right. It's the way it is, cause that's the way it is...
I think KFAILs should go in the summary. I've never really liked the idea of
either XFAIL or KFAIL, but understand others find these useful. I worry that
once setup as a expected failure, they'll get lost, and never fixed. So KFAIL
should be in the summary to beat developers over the head with them so they
eventually get fixed and removed.

  btw - I'm still waiting for KFAIL documentation... :-)

	- rob -

Re: Print KFAIL's in dejagnu summary?

[Ben Elliston]

Rob Savoye <rob@welcomehome.org> writes:

> Right now the only documentation for KFAIL is in the mostly
> unmaintained dejagnu.texi file. It needs to be added to the SGML
> format doc as well. Yes, I know the GNU project uses texinfo, but
> most GNU/Linux projects these days use DocBook, which is also
> supported by all the fancy help tools under GNOME.

I found tools to generate Texinfo output from Docbook so I am happy to
see the documentation retained in Docbook, but we definitely need a
single source for documentation and we need Texinfo files, just like
any GNU package.

Ben

Re: Print KFAIL's in dejagnu summary?

[Rob Savoye]

On Fri, Oct 03, 2003 at 11:57:00AM -0400, Michael Elizabeth Chastain wrote:
> > btw - I'm still waiting for KFAIL documentation... :-)
> 
> Oops.  What kind of documentation are you waiting for?
> I can write doco.

  Right now the only documentation for KFAIL is in the mostly unmaintained 
dejagnu.texi file. It needs to be added to the SGML format doc as well. Yes, I
know the GNU project uses texinfo, but most GNU/Linux projects these days use
DocBook, which is also supported by all the fancy help tools under GNOME. 

	- rob -

Re: Print KFAIL's in dejagnu summary?

[Michael Elizabeth Chastain]

> btw - I'm still waiting for KFAIL documentation... :-)

Oops.  What kind of documentation are you waiting for?
I can write doco.

Michael C

Re: Print KFAIL's in dejagnu summary?

[Michael Elizabeth Chastain]

dc> What would make more of a difference for me (and,
dc> presumably, for other people) would be to make it as easy as possible
dc> to compare sum files: regularize the thread output a bit more, get rid
dc> of gratuitous numbers (especially the value of $fp in pc-fp.exp).

I'm all for that.

Michael C

Re: Print KFAIL's in dejagnu summary?

[Michael Elizabeth Chastain]

ac> At present KFAILs are supressed from the summary output (the stuff on 
ac> the terminal from "make check").  I'd like to change this so that 
ac> KFAILs, just like FAILs, are included in the summary.  A KFAIL, just 
ac> like a FAIL, indicates a bug in the system under test, and hence should 
ac> be included in the summary.

I would really like this.

My scripts treat every non-PASS result as an item of interest,
so I don't miss anything.

For some actual counts:

  http://www.shout.net/~mec/sunday/2003-09-22/Counts.html

Michael C