Re: Mauve wishlist

[Anthony Balkissoon <abalkiss@redhat.com>]

On Fri, 2006-03-17 at 11:32 -0500, Thomas Fitzsimmons wrote:
> Hi,
> 
> Anthony Balkissoon has expressed interest in improving Mauve so we'd
> like to know what would be the best things to work on.
> 

Another suggestion that Tom Fitzsimmons had was to change the way we
count the number of tests.  Counting each invocation of the test()
method rather than each call to harness.check() has two benefits:

1) constant number of tests, regardless of exceptions being thrown or
which if-else branch is taken

2) more realistic number of tests, to accurately reflect the extent of
our testing


For point 1) this will help us see if we are making progress.  Right now
a Mauve run might say we have 113 fails out of 13200 tests and then a
later run could say 200 fails out of 34000 tests.  Is this an
improvement?  Hard to say.  But if we count each call to test() as a
test, and also detect hanging tests, then we should have a constant
number of tests in each run and will be able to say if changes made have
a positive impact on Mauve test results.  Of course, if in one
particular test file there are 1000 calls to harness.check() and only
one of them fails, it's not helpful to just report that the entire test
failed.  So the output will have to pinpoint which call to harness.check
failed (and preferably a line number).  The negative side here is that
the results will be overly pessimistic because any failing harness.check
trumps all the passing harness.check calls and the test is reported as a
failure.

What do people have to say about this idea?

--Tony