Testcases in Bugzilla

Testcases in Bugzilla

[Volker Reichelt]

Hi again,

we still haven't reached a conclusion about the testcases in Bugzilla.

Bugzilla offers us the possibility to mark an attachment as testcase
and a possibility to download them as pointed out in

http://gcc.gnu.org/ml/gcc/2003-05/msg01212.html

But how can we actually test the testcases?
The dejagnu testing harness doesn't allow you to specify a branch
or something like that where the test is supposed to pass/fail AFAIK.
And without such a feature the whole testing becomes a nightmare.

There are some resolutions, but they all have their drawbacks:

a) Create a testcase for each branch. (Way too much work IMHO.)

b) Mark all tests as XPASS and supply lists for each branch with the failures.
   (Too complicated in practice IMHO. And what about other architectures?)

c) Add a couple of lines at the top of each testcase that specify the
   compiler versions where we should XFAIL.
   Run them through a little preprocessor script that removes these lines
   and adds an XFAIL where appropriate. Hand the result over to dejagnu.

d) We can rely on my testing script which is just a small shell script
   which doesn't use dejagnu and therefore has only limited capabilities,
   but works for a lot of testcases and can deal with different compiler
   versions. (See http://gcc.gnu.org/ml/gcc/2002-12/msg01250.html )
   In that case I'd post a revamped version.

e) We could modify the script in d) to accomplish c).

f) Extend dejagnu with versioning.

g) ...?


I think we should reach a conclusion *before* we start adding the testcases
to bugzilla so that we don't have to modify the testcases over and over again.

Any suggestions?

Regards,
Volker

PS: To avoid name clashes between testcases, we should agree on a naming
convention (e.g. PR12345A.c).

Re: Testcases in Bugzilla

[Janis Johnson]

On Tue, May 13, 2003 at 01:48:26PM +0200, Volker Reichelt wrote:
> Hi again,
> 
> we still haven't reached a conclusion about the testcases in Bugzilla.

No, but we're making progress.

> Bugzilla offers us the possibility to mark an attachment as testcase
> and a possibility to download them as pointed out in
> 
> http://gcc.gnu.org/ml/gcc/2003-05/msg01212.html
> 
> But how can we actually test the testcases?
> The dejagnu testing harness doesn't allow you to specify a branch
> or something like that where the test is supposed to pass/fail AFAIK.
> And without such a feature the whole testing becomes a nightmare.

The testing harness is actually quite flexible and lets you add whatever
new functionality you need in the .exp files used by the tests.

> There are some resolutions, but they all have their drawbacks:
> 
> a) Create a testcase for each branch. (Way too much work IMHO.)

No, although in some cases it might be necessary to have different
test cases.

> b) Mark all tests as XPASS and supply lists for each branch with the failures.
>    (Too complicated in practice IMHO. And what about other architectures?)

We will want to be able to handle tests that are only expected to fail
on some targets, as well as on some branches.

> c) Add a couple of lines at the top of each testcase that specify the
>    compiler versions where we should XFAIL.

Yes.

>    Run them through a little preprocessor script that removes these lines
>    and adds an XFAIL where appropriate. Hand the result over to dejagnu.

The code in the .exp file could handle this directly.

> d) We can rely on my testing script which is just a small shell script
>    which doesn't use dejagnu and therefore has only limited capabilities,
>    but works for a lot of testcases and can deal with different compiler
>    versions. (See http://gcc.gnu.org/ml/gcc/2002-12/msg01250.html )
>    In that case I'd post a revamped version.

I'd prefer to use the existing testing tools.

> e) We could modify the script in d) to accomplish c).
> 
> f) Extend dejagnu with versioning.

Use DejaGNU functionality to support versioning.

> g) ...?
> 
> 
> I think we should reach a conclusion *before* we start adding the testcases
> to bugzilla so that we don't have to modify the testcases over and over again.

Yes, that's a good point.  I can spend some time on this now.

> Any suggestions?
> 
> Regards,
> Volker
> 
> PS: To avoid name clashes between testcases, we should agree on a naming
> convention (e.g. PR12345A.c).

Or PR12345-1.c, in line with the existing convention for multiple tests
of the same bug/fix.

We'll need some kind of restriction on execute tests to assure people
running these tests that they aren't going to do anything antisocial.

Janis

Re: Testcases in Bugzilla

[Wolfgang Bangerth]

> > I can't imagine a method to _automatically_ extract these testcases and do
> > something reasonable. If there is one, it would increase the entry level
> > for people working in GNATS.
> 
> Err, we can easily extract attachments marked "testcase" from bugzilla in
> about 9 lines of perl.
> The rest is up to other scripts and whatnot.

That was the point. It's the "do something reasonable" with it that's the 
problem, I guess.

W.

-------------------------------------------------------------------------
Wolfgang Bangerth              email:            bangerth@ices.utexas.edu
                               www: http://www.ices.utexas.edu/~bangerth/

Re: Testcases in Bugzilla

[Daniel Berlin]

On Tue, 13 May 2003, Wolfgang Bangerth wrote:

>
> > we still haven't reached a conclusion about the testcases in Bugzilla.
>
> To be quite honest, I think what you suggest won't work. What we put into
> the bug database is just too diverse:
> - we have bug reports where just the text of an error message became
>   worse than before
> - we have bug reports where we get the compiler into an endless loop
> - we have reports where compilation takes quadratic instead of linear
>   time in the input
> - reports with two or more testcases that should do the same but don't
> - some other really weird stuff happening.
>
> I can't imagine a method to _automatically_ extract these testcases and do
> something reasonable. If there is one, it would increase the entry level
> for people working in GNATS.

Err, we can easily extract attachments marked "testcase" from bugzilla in
about 9 lines of perl.
The rest is up to other scripts and whatnot.
>
>
>

Re: Testcases in Bugzilla

[Wolfgang Bangerth]

> we still haven't reached a conclusion about the testcases in Bugzilla.

To be quite honest, I think what you suggest won't work. What we put into 
the bug database is just too diverse:
- we have bug reports where just the text of an error message became
  worse than before
- we have bug reports where we get the compiler into an endless loop
- we have reports where compilation takes quadratic instead of linear
  time in the input
- reports with two or more testcases that should do the same but don't
- some other really weird stuff happening.

I can't imagine a method to _automatically_ extract these testcases and do 
something reasonable. If there is one, it would increase the entry level 
for people working in GNATS. We're happy that we recently got quite a 
number of people working on GNATS; that might or might not be due to the 
relatively low barrier (you don't have to know anything about gcc's 
internals) and I wouldn't be happy to increase that by adding more 
procedures one has to follow.

I should say that this shouldn't stop you from coming up with a simple
solution. I think you do a very good job with your hand-extracted
testsuire. But whatever we do here, it should be simple. If it's not, it
might not be worth the trouble: if a bug is fixed and remains so for a
couple of months -- is that a problem?

W.

-------------------------------------------------------------------------
Wolfgang Bangerth              email:            bangerth@ices.utexas.edu
                               www: http://www.ices.utexas.edu/~bangerth/