A testcase library

A testcase library

[Volker Reichelt]

[-- Attachment #1: Type: TEXT/plain, Size: 697 bytes --]

Since GNATS has its limitations with respect to searching for relevant
information about testcases, I started collecting *small* testcases
for easy grepping. Until now, I've collected (with the help of
Wolfgang and of course the submitters) more than 200 of such testcases.

Maybe the testcase library is useful for other bug-hunters, so here it
is. I also wrote a little script that helps me keeping track of the
information. Maybe that is of interest, too. The library and the script
come with a small Readme, which you might want to read first.

Comments and testcases for inclusion in the library are welcome.
If this is of interest I could post the updated stuff regularly.

Regards,
Volker


[-- Attachment #2: TestCases.zip --]
[-- Type: APPLICATION/octet-stream, Size: 73424 bytes --]

Re: A testcase library

[Jakub Jelinek]

On Fri, Dec 20, 2002 at 01:40:49PM +0100, Volker Reichelt wrote:
> Since GNATS has its limitations with respect to searching for relevant
> information about testcases, I started collecting *small* testcases
> for easy grepping. Until now, I've collected (with the help of
> Wolfgang and of course the submitters) more than 200 of such testcases.
> 
> Maybe the testcase library is useful for other bug-hunters, so here it
> is. I also wrote a little script that helps me keeping track of the
> information. Maybe that is of interest, too. The library and the script
> come with a small Readme, which you might want to read first.
> 
> Comments and testcases for inclusion in the library are welcome.
> If this is of interest I could post the updated stuff regularly.

Good work, thanks.
Unless they are already in gcc testsuite, shouldn't they all be checked
in after converting them to dg (or c-torture) format, ideally on all of 3.2/3.3 
branches and trunk?

	Jakub

Re: A testcase library

[Volker Reichelt]

On 20 Dec, Jakub Jelinek wrote:
> On Fri, Dec 20, 2002 at 01:40:49PM +0100, Volker Reichelt wrote:
>> Since GNATS has its limitations with respect to searching for relevant
>> information about testcases, I started collecting *small* testcases
>> for easy grepping. Until now, I've collected (with the help of
>> Wolfgang and of course the submitters) more than 200 of such testcases.
>> 
>> Maybe the testcase library is useful for other bug-hunters, so here it
>> is. I also wrote a little script that helps me keeping track of the
>> information. Maybe that is of interest, too. The library and the script
>> come with a small Readme, which you might want to read first.
>> 
>> Comments and testcases for inclusion in the library are welcome.
>> If this is of interest I could post the updated stuff regularly.
> 
> Good work, thanks.
> Unless they are already in gcc testsuite, shouldn't they all be checked
> in after converting them to dg (or c-torture) format, ideally on all of 3.2/3.3 
> branches and trunk?

This is not current policy. Christian has asked a similar question
before, and got the following mail snippet from Craig:

=================================================================================
On Fri, Nov 29, 2002 at 09:47:24AM +0000, Nathan Sidwell wrote:
> Craig Rodrigues wrote:
> > A few questions:
> > (1) Can I commit my testcase to the testsuite, since it is definitely
> >     a regression from gcc 3.1 and gcc 3.2?
> No, we only commit test cases when the bug's fixed. The rationale being you
> don't want to add a new failing test case -- that'd confuse people. Marking
> it XFAIL isn't really the right solution, as then when you fix the bug, you
> don't know whether fixing the XFAIL was accidental, and end up adding a new
> test case anyway. Also, in fixing the bug, you (well I at least), can
> construct a better test case having undersytood the failure mode.
=================================================================================

But that doesn't keep us from doing something similar in a different
location. I had some discussion over the topic in private mail with
Wolfgang and we decided to wait for Bugzilla before setting up something
new. But maybe there are better suggestions?

Regards,
Volker

Re: A testcase library

[Eric Botcazou]

> But that doesn't keep us from doing something similar in a different
> location. I had some discussion over the topic in private mail with
> Wolfgang and we decided to wait for Bugzilla before setting up something
> new. But maybe there are better suggestions?

Why not a special directory in the testsuite that is not tested by the 
toplevel 'make check' but by a magic-like command invoked only by 
those-who-know ? It would not be included in the official tarballs.

-- 
Eric Botcazou

Re: A testcase library

[Wolfgang Bangerth]

On Fri, 20 Dec 2002, Eric Botcazou wrote:

> > But that doesn't keep us from doing something similar in a different
> > location. I had some discussion over the topic in private mail with
> > Wolfgang and we decided to wait for Bugzilla before setting up something
> > new. But maybe there are better suggestions?
> 
> Why not a special directory in the testsuite that is not tested by the 
> toplevel 'make check' but by a magic-like command invoked only by 
> those-who-know ? It would not be included in the official tarballs.

That was my original thought. It would allow us to get the presently 
smallest testcase to check for something, and if we can further reduce it, 
we could check in the new testcase as well. It would also allow potentail 
bug fixers simpler access to the testcases.

On the other hand, the list of people with CVS write privileges would have 
to be expanded.

W.

-------------------------------------------------------------------------
Wolfgang Bangerth              email:           bangerth@ticam.utexas.edu
                               www: http://www.ticam.utexas.edu/~bangerth

Re: A testcase library

[Volker Reichelt]

On 20 Dec, Wolfgang Bangerth wrote:
> On Fri, 20 Dec 2002, Eric Botcazou wrote:
> 
>> > But that doesn't keep us from doing something similar in a different
>> > location. I had some discussion over the topic in private mail with
>> > Wolfgang and we decided to wait for Bugzilla before setting up something
>> > new. But maybe there are better suggestions?
>> 
>> Why not a special directory in the testsuite that is not tested by the 
>> toplevel 'make check' but by a magic-like command invoked only by 
>> those-who-know ? It would not be included in the official tarballs.
> 
> That was my original thought. It would allow us to get the presently 
> smallest testcase to check for something, and if we can further reduce it, 
> we could check in the new testcase as well. It would also allow potentail 
> bug fixers simpler access to the testcases.
> 
> On the other hand, the list of people with CVS write privileges would have 
> to be expanded.

We'd probably also want a looser check-in policy for the problem
reports.

Regards,
Volker

Re: A testcase library

[Jason R Thorpe]

On Fri, Dec 20, 2002 at 02:13:23PM +0100, Volker Reichelt wrote:

 > =================================================================================
 > On Fri, Nov 29, 2002 at 09:47:24AM +0000, Nathan Sidwell wrote:
 > > Craig Rodrigues wrote:
 > > > A few questions:
 > > > (1) Can I commit my testcase to the testsuite, since it is definitely
 > > >     a regression from gcc 3.1 and gcc 3.2?
 > > No, we only commit test cases when the bug's fixed. The rationale being you
 > > don't want to add a new failing test case -- that'd confuse people. Marking
 > > it XFAIL isn't really the right solution, as then when you fix the bug, you
 > > don't know whether fixing the XFAIL was accidental, and end up adding a new
 > > test case anyway. Also, in fixing the bug, you (well I at least), can
 > > construct a better test case having undersytood the failure mode.
 > =================================================================================

I have to say that I disagree with this policy, esp. if the testcase shows
a regression from a previous GCC release.  Gaps in the testsuite mean it
will take longer for bugs to be fixed, and give people a false sense of
confidence in the quality of the compiler when they see results that show
few, if any, failures.

One example is a PR I filed recently, with test case that causes the compiler
to loop forever.  The rationale I got for not adding the testcase was "it
would make everyone's bootstrap/test cycle 5 minutes longer".  My initial
thought was "...and that will motivate someone with the knowledge to do so
to fix it." (The test case in question shows a regression from a previous
release of GCC.)

...the principle is not too different than a suggestion someone made a
while ago about making GCC maintainers do bootstraps on 100MHz Pentiums
(maybe GCC wouldn't slow down so much over time if they did :-)

-- 
        -- Jason R. Thorpe <thorpej@wasabisystems.com>

Re: A testcase library

[Ben Elliston]

>>>>> "Volker" == Volker Reichelt <reichelt@igpm.rwth-aachen.de> writes:

  >> Unless they are already in gcc testsuite, shouldn't they all be checked
  >> in after converting them to dg (or c-torture) format, ideally on all of 3.2/3.3 
  >> branches and trunk?

  Volker> This is not current policy. Christian has asked a similar question
  Volker> before, and got the following mail snippet from Craig:

  >> No, we only commit test cases when the bug's fixed. The rationale
  >> being you don't want to add a new failing test case -- that'd
  >> confuse people. Marking it XFAIL isn't really the right solution,
  >> as then when you fix the bug, you don't know whether fixing the
  >> XFAIL was accidental, and end up adding a new test case
  >> anyway. Also, in fixing the bug, you (well I at least), can
  >> construct a better test case having undersytood the failure mode.

How about committing the test cases and marking them as known failures
using DejaGnu's spiffy new KFAIL result?

Ben

Re: A testcase library

[Mike Stump]

On Sunday, December 22, 2002, at 01:47 PM, Ben Elliston wrote:
> How about committing the test cases and marking them as known failures 
> using DejaGnu's spiffy new KFAIL result?

:-(  I think people have a hard enough time with expected and 
unexpected, now we have to add known.  I think it is a bad idea.  Do we 
also have unexpected known failure, and expected unknown pass?

I think we should just avoid it for 3 years while we think about it.

Re: A testcase library

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 11:24:33AM -0800, Mike Stump wrote:
> On Sunday, December 22, 2002, at 01:47 PM, Ben Elliston wrote:
> >How about committing the test cases and marking them as known failures 
> >using DejaGnu's spiffy new KFAIL result?
> 
> :-(  I think people have a hard enough time with expected and 
> unexpected, now we have to add known.  I think it is a bad idea.  Do we 
> also have unexpected known failure, and expected unknown pass?

I have no idea if you meant that seriously or not... think about those,
they're meaningless.  Either it's expected to fail, known to fail, or
unexpected.  Either it's expected to pass, or unexpected (or unexpected
known pass, which doesn't make much sense to me, but it's easy to
figure out what it means.)

> I think we should just avoid it for 3 years while we think about it.

I smell progress, yes I do.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: A testcase library

[Mike Stump]

On Monday, December 23, 2002, at 11:37 AM, Daniel Jacobowitz wrote:
> I have no idea if you meant that seriously or not...

I'm serious.

> think about those, they're meaningless.

Great, now we have all these meaningless states.

We already had the ability to add tracking numbers, we already had the 
ability to sort reports and put failures with tracking numbers up 
higher in priority, if we wanted.  We already had a way of knowing if 
the report was known.

I just don't see any advantages yet.  Maybe the dejagnu document could 
be expanded to include the advantages of these extra states.

Re: A testcase library

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 01:50:24PM -0800, Mike Stump wrote:
> On Monday, December 23, 2002, at 11:37 AM, Daniel Jacobowitz wrote:
> >I have no idea if you meant that seriously or not...
> 
> I'm serious.
> 
> >think about those, they're meaningless.
> 
> Great, now we have all these meaningless states.

No, we don't have them; DejaGNU won't express them.  The states are:
pass, fail, xpass, xfail, kpass, kfail
  (and warning, perror, note, untested, unresolved, unsupported)

> We already had the ability to add tracking numbers, we already had the 
> ability to sort reports and put failures with tracking numbers up 
> higher in priority, if we wanted.  We already had a way of knowing if 
> the report was known.
> 
> I just don't see any advantages yet.  Maybe the dejagnu document could 
> be expanded to include the advantages of these extra states.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: A testcase library

[Ben Elliston]

>>>>> "Mike" == Mike Stump <mstump@apple.com> writes:

  >> How about committing the test cases and marking them as known
  >> failures using DejaGnu's spiffy new KFAIL result?

  Mike> I think people have a hard enough time with expected and
  Mike> unexpected, now we have to add known.  I think it is a bad
  Mike> idea.  Do we also have unexpected known failure, and expected
  Mike> unknown pass?

I think that KFAIL clarifies the situation (although I, too, have my
reservations about having too many test states).  It seems few people
properly understood the purpose of XFAIL, but most instances of XFAIL
are used to express known failures (take the G++ testsuite for
example).  The number of times XFAIL should be used should be
comparatively few.

FWIW, this approach has been adopted by The GDB Community.

Ben

Re: A testcase library

[Zack Weinberg]

Ben Elliston <bje@redhat.com> writes:

> I think that KFAIL clarifies the situation (although I, too, have my
> reservations about having too many test states).  It seems few people
> properly understood the purpose of XFAIL, but most instances of XFAIL
> are used to express known failures (take the G++ testsuite for
> example).  The number of times XFAIL should be used should be
> comparatively few.

Can you expand on the difference between KFAIL and XFAIL, please?  I
don't have this new dejagnu version and from over here it sounds like
a distinction without a difference.

zw

Re: A testcase library

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 01:25:37PM -0800, Zack Weinberg wrote:
> Ben Elliston <bje@redhat.com> writes:
> 
> > I think that KFAIL clarifies the situation (although I, too, have my
> > reservations about having too many test states).  It seems few people
> > properly understood the purpose of XFAIL, but most instances of XFAIL
> > are used to express known failures (take the G++ testsuite for
> > example).  The number of times XFAIL should be used should be
> > comparatively few.
> 
> Can you expand on the difference between KFAIL and XFAIL, please?  I
> don't have this new dejagnu version and from over here it sounds like
> a distinction without a difference.

Implementation-wise they are essentially the same.  They recognize two
different categories of problems however.  The best way I can explain
it is with an example.

Suppose that GDB is debugging a program, and the information it needs
is simply missing from the debug info.  Zilch, nada, absolutely no way
to recover it.  That's an XFAIL.

Suppose that a testcase is for a new bug that no one's figured out how
to fix yet.  That's a KFAIL.  All KFAILs are required to have an open PR
associated with them, at least in GDB-land.

The difference, in theory, is that KFAILs represent real problems that
have not been fixed; and XFAILs represent "expected" failures, problems
in the system or tools that can not be fixed in the program-under-test. 
They are primarily a device to have no unexpected failures without
abusing XFAIL; this is useful because it makes new regressions very
obvious.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: A testcase library

[Joseph S. Myers]

On Fri, 20 Dec 2002, Jakub Jelinek wrote:

> Good work, thanks.
> Unless they are already in gcc testsuite, shouldn't they all be checked
> in after converting them to dg (or c-torture) format, ideally on all of 3.2/3.3 
> branches and trunk?

Yes.  The policy is in <http://gcc.gnu.org/bugs.html#manage>; it doesn't
specifically say what happens where the bug is already in GNATS and the
testcase is to be added, but it should be clear from the instructions that
you can add the testcase to the testsuite, include a comment in the
testcase referencing the PR and add a note referencing the testcase to the
PR, and mark the testcase XFAIL iff it's not a regression.

Release engineering has in the past involved people running the branch
testsuite against both branch and previous releases (instructions for
running with an installed GCC being provided) as an additional check for
regressions (which may show up those that are only regressions on some
platforms).

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: A testcase library

[Richard Kenner]

    :-( I think people have a hard enough time with expected and
    unexpected, now we have to add known.  I think it is a bad idea.  Do
    we also have unexpected known failure, and expected unknown pass?

I agree.

Indeed can somebody explain what the diffence is between an "expected"
failure and a "known" failure?  Intuitively, they sound the same to me.

Re: A testcase library

[David Carlton]

On Mon, 23 Dec 02 20:39:06 EST, kenner@vlsi1.ultra.nyu.edu (Richard Kenner) said:

> Indeed can somebody explain what the diffence is between an "expected"
> failure and a "known" failure?  Intuitively, they sound the same to me.

Yeah, the naming is a little weird.  These days I think of the X in
XFAIL as standing for "external" rather than "expected".

David Carlton
carlton@math.stanford.edu

Re: A testcase library

[Richard Kenner]

    Suppose that GDB is debugging a program, and the information it needs
    is simply missing from the debug info.  Zilch, nada, absolutely no way
    to recover it.  That's an XFAIL.

    Suppose that a testcase is for a new bug that no one's figured out how
    to fix yet.  That's a KFAIL.  All KFAILs are required to have an open PR
    associated with them, at least in GDB-land.

    The difference, in theory, is that KFAILs represent real problems that
    have not been fixed; and XFAILs represent "expected" failures, problems
    in the system or tools that can not be fixed in the program-under-test. 

I still don't get it.  Your first example sounds like something that's not
a bug at all, in which case why would the "test case" still be in the suite?

Re: A testcase library

[Daniel Jacobowitz]

On Mon, Dec 23, 2002 at 08:45:36PM -0500, Richard Kenner wrote:
>     Suppose that GDB is debugging a program, and the information it needs
>     is simply missing from the debug info.  Zilch, nada, absolutely no way
>     to recover it.  That's an XFAIL.
> 
>     Suppose that a testcase is for a new bug that no one's figured out how
>     to fix yet.  That's a KFAIL.  All KFAILs are required to have an open PR
>     associated with them, at least in GDB-land.
> 
>     The difference, in theory, is that KFAILs represent real problems that
>     have not been fixed; and XFAILs represent "expected" failures, problems
>     in the system or tools that can not be fixed in the program-under-test. 
> 
> I still don't get it.  Your first example sounds like something that's not
> a bug at all, in which case why would the "test case" still be in the suite?

It's a test.  It can have multiple results:
 - A correct result
 - An incorrect, unavoidable result as a consequence of a bug in the
system
 - An incorrect result as a consequence of a bug in the program being
tested.

This is not hypothetical.  The GDB testsuite is filled with examples of
this.  Often it's discovered the other way round: someone finds a bug
and fixes it, but the test case doesn't pass on some other system and
there's nothing we can do.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: A testcase library

[David Carlton]

On Mon, 23 Dec 02 20:45:36 EST, kenner@vlsi1.ultra.nyu.edu (Richard Kenner) said:

>> Suppose that GDB is debugging a program, and the information it
>> needs is simply missing from the debug info.  Zilch, nada,
>> absolutely no way to recover it.  That's an XFAIL.

>> Suppose that a testcase is for a new bug that no one's figured out
>> how to fix yet.  That's a KFAIL.  All KFAILs are required to have
>> an open PR associated with them, at least in GDB-land.

>> The difference, in theory, is that KFAILs represent real problems
>> that have not been fixed; and XFAILs represent "expected" failures,
>> problems in the system or tools that can not be fixed in the
>> program-under-test.

> I still don't get it.  Your first example sounds like something
> that's not a bug at all, in which case why would the "test case"
> still be in the suite?

If something XFAILs on all systems, then it would be a little strange
to have it in the test suite.  But a more typical situation is that
there's something that GDB tries to get right, and can on certain
systems, but can't on other systems; so we XFAIL it where we can't get
it right.

For example, say that I'm doing a bunch of tests for GDB's namespace
support.  Then there might be some areas of namespace support that GDB
doesn't try to handle at all; for those, I submit PR's, and add
KFAILed tests.

Then there might be some tests that, say, GDB gets right if we have
DWARF-2 debug info but not with stabs, because stabs doesn't have good
enough nesting constructs.  I'll add tests for those to the test
suite, but I'll XFAIL them if the current debug format is stabs.

We'll see how it turns out: we added the first KFAIL to GDB's test
suite all of a week ago, so it's not yet clear exactly how the extra
information will help.  Probably its main benefit won't be in that
extra information, but will rather be that it gives people an excuse
to spend time auditing current FAILs and XFAILs (and XPASSes, for that
matter), with the result that the bug database will more closely
reflect GDB's actual state.  (Though that only helps if it in turn
encourages people to fix bugs...)

David Carlton
carlton@math.stanford.edu