classpath+mauve

classpath+mauve

[Norman Hendrich]

Hello David and Audrius,

first of all, thanks for your answers to my mauve-setup question(s).
I tried the MakeTestClassList program, which seems to work fine.
However, after spending another few hours trying to get the whole 
testsuite running, my worst fears are confirmed... To summarize:

* writing new testcases is easy

* running individual testcases is easy (from the shell or eclipse)

* running parts of the testsuite via an index file works.

* BUT there is no easy way to run the whole testsuite. A developer
  first has to create a custom list of exclusion testcases, before
  running the remaining fraction of the testsuite... 
 

Despite David's nice program, command line parameters, and exclusion 
testcase lists, I did not manage to run the whole testsuite reliably 
against today's classpath CVS. After passing some tests,
jamvm encounters a test that throws some unexpected exception,
runs into a stack-overflow, or complains about an IllegalStateException
in loadClass(). In all cases, gnu.testlet.runner.Mauve does not run to 
the end, and the user doesn't even get the summary of test results.

This basically leads to the situation that a developer needs to run the
testsuite, try to understand the last error, remove the problematic test 
question from the list of testcases, and re-run. I tried this from scratch
without David's exclusion list, but stopped after the first 20 iterations
of this (with no end in sight). I then proceeded to remove larger chunks
of the tests, and finally got a test report - but this in turn means
that many tests were never run at all.

---

The current all-testcases-at-once model used by gnu.testlet.runner.Mauve
is not robust enough to survive all tests, while the opposite strategy
of running each test in a new VM via (Simple)TestHarness is extremely
wasteful due to the VM initialization overhead.

Perhaps we could switch to a two-level model, where an outer process
watches the progress of the inner runner process, and restarts the 
inner process whenever this crashes or deadlocks?

Also, a test that doesn't compile should be marked as FAIL instead
of stopping the testsuite to run at all...


- Norman

PS: a matrix of today's mauve experiments:

* mauve+jdk1.5.0 with 'make check': doesn't even compile
* mauve+jdk1.4.2 with 'make check': works, but subset of tests only
* mauve+jdk1.4.2 with 'ant': doesn't even compile
* mauve+jdk1.5.0 with 'ant': doesn't even compile
* mauve+classpath+jamvm+jikes:  compiles, but crashes on lots of testcases

Re: classpath+mauve

[Michael Koch]

On Fri, Oct 14, 2005 at 03:56:28PM +0200, Norman Hendrich wrote:
> Hello David and Audrius,
> 
> first of all, thanks for your answers to my mauve-setup question(s).
> I tried the MakeTestClassList program, which seems to work fine.
> However, after spending another few hours trying to get the whole 
> testsuite running, my worst fears are confirmed... To summarize:
> 
> * writing new testcases is easy
> 
> * running individual testcases is easy (from the shell or eclipse)
> 
> * running parts of the testsuite via an index file works.
> 
> * BUT there is no easy way to run the whole testsuite. A developer
>   first has to create a custom list of exclusion testcases, before
>   running the remaining fraction of the testsuite... 

Not true. I just use the batch_run utility to run them all. I know that
Mark uses this too.


Cheers,
Michael
-- 
Escape the Java Trap with GNU Classpath!
http://www.gnu.org/philosophy/java-trap.html

Join the community at http://planet.classpath.org/

Re: classpath+mauve

[Brian Jones]

Michael Koch wrote:

>On Fri, Oct 14, 2005 at 03:56:28PM +0200, Norman Hendrich wrote:
>  
>
>>Hello David and Audrius,
>>
>>first of all, thanks for your answers to my mauve-setup question(s).
>>I tried the MakeTestClassList program, which seems to work fine.
>>However, after spending another few hours trying to get the whole 
>>testsuite running, my worst fears are confirmed... To summarize:
>>
>>* writing new testcases is easy
>>
>>* running individual testcases is easy (from the shell or eclipse)
>>
>>* running parts of the testsuite via an index file works.
>>
>>* BUT there is no easy way to run the whole testsuite. A developer
>>  first has to create a custom list of exclusion testcases, before
>>  running the remaining fraction of the testsuite... 
>>    
>>
>
>Not true. I just use the batch_run utility to run them all. I know that
>Mark uses this too.
>
>
>Cheers,
>Michael
>  
>
Yes, due to the nature of how a JVM can fail... especially with respect 
to hanging and doing nothing, you have to wrap execution of each test 
case in a fashion that will kill the JVM if it does not exit in a 
reasonable period of time.  I think this batch_run utility does that.  I 
used to have a nice little set of scripts that did something similar and 
created pretty graphs.