Re: Mauve wishlist

[Bryce McKinlay <mckinlay@redhat.com>]

David Gilbert wrote:
> 1) constant number of tests, regardless of exceptions being thrown or
> Mauve does have a design flaw where it can be tricky to automatically 
> assign a unique identifier to each check(), and this makes it hard to 
> compare two Mauve runs (say a test of the latest Classpath CVS vs the 
> last release, or the Classpath vs JDK 1.5 - both of which would be 
> interesting).

Right. We all understand the problem - its just the solution is what we 
need to agree on :)
> I think the absolute number is meaningless however you count the 
> tests, so I don't see this as an advantage. 

Yes, numbers alone are meaningless, but with the current design, all the 
results are meaningless without a lot of context. The real issue is 
having a simple way to uniquely identify each test case for the purpose 
of identifying regressions. This becomes fundamentally much easier when 
1 test() method corresponds to one test case.

It is not reasonable to expect test case developers ensure that all 
tests "run linearly". Exceptions can potentially be thrown at any time, 
so to ensure linearity, every check() call would need wrapped with a 
try/catch.

> You'll lose the ability to distinguish between an existing failure 
> where (say) 1 out of 72 checks fail, and after some clever patch 43 
> out of 72 checks fail, but the new system reports both as 1 test failure.

This is a valid concern. However, we will still track exactly which 
check() calls fail so that in the event of a test failure, a full 
diagnostic can be provided. In addition, we can still count the total 
number of check() calls executed, for statistical purposes.

If the reduced test-case granularity does prove to be problematic in 
some cases - say some test() where a small number of checks fail for 
cases that are "hard to fix" problems and thus should be "xfailed" (to 
use gcc/dejagnu lingo), then that test case should probably be split. 
Alternatively, to avoid splitting things across multiple test classes, 
we could add JUnit-style support for multiple test() methods in a single 
test class.

Bryce