From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 16295 invoked by alias); 14 Oct 2005 13:56:41 -0000 Mailing-List: contact mauve-discuss-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: mauve-discuss-owner@sources.redhat.com Received: (qmail 16283 invoked by uid 22791); 14 Oct 2005 13:56:38 -0000 Received: from rzdspc1.informatik.uni-hamburg.de (HELO rzdspc1.informatik.uni-hamburg.de) (134.100.9.61) by sourceware.org (qpsmtpd/0.30-dev) with ESMTP; Fri, 14 Oct 2005 13:56:38 +0000 Received: from rzdspc10.informatik.uni-hamburg.de (hendrich@rzdspc10.informatik.uni-hamburg.de [134.100.9.63]) by rzdspc1.informatik.uni-hamburg.de (8.13.0/8.13.0) with SMTP id j9EDuS4O015670; Fri, 14 Oct 2005 15:56:28 +0200 (CEST) Message-Id: <200510141356.j9EDuS4O015670@rzdspc1.informatik.uni-hamburg.de> Date: Fri, 14 Oct 2005 13:56:00 -0000 From: Norman Hendrich Reply-To: Norman Hendrich Subject: classpath+mauve To: classpath@gnu.org Cc: mauve-discuss@sourceware.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: Ocm1kz9U02RuNBh8mImPyQ== X-SW-Source: 2005-q4/txt/msg00006.txt.bz2 Hello David and Audrius, first of all, thanks for your answers to my mauve-setup question(s). I tried the MakeTestClassList program, which seems to work fine. However, after spending another few hours trying to get the whole testsuite running, my worst fears are confirmed... To summarize: * writing new testcases is easy * running individual testcases is easy (from the shell or eclipse) * running parts of the testsuite via an index file works. * BUT there is no easy way to run the whole testsuite. A developer first has to create a custom list of exclusion testcases, before running the remaining fraction of the testsuite... Despite David's nice program, command line parameters, and exclusion testcase lists, I did not manage to run the whole testsuite reliably against today's classpath CVS. After passing some tests, jamvm encounters a test that throws some unexpected exception, runs into a stack-overflow, or complains about an IllegalStateException in loadClass(). In all cases, gnu.testlet.runner.Mauve does not run to the end, and the user doesn't even get the summary of test results. This basically leads to the situation that a developer needs to run the testsuite, try to understand the last error, remove the problematic test question from the list of testcases, and re-run. I tried this from scratch without David's exclusion list, but stopped after the first 20 iterations of this (with no end in sight). I then proceeded to remove larger chunks of the tests, and finally got a test report - but this in turn means that many tests were never run at all. --- The current all-testcases-at-once model used by gnu.testlet.runner.Mauve is not robust enough to survive all tests, while the opposite strategy of running each test in a new VM via (Simple)TestHarness is extremely wasteful due to the VM initialization overhead. Perhaps we could switch to a two-level model, where an outer process watches the progress of the inner runner process, and restarts the inner process whenever this crashes or deadlocks? Also, a test that doesn't compile should be marked as FAIL instead of stopping the testsuite to run at all... - Norman PS: a matrix of today's mauve experiments: * mauve+jdk1.5.0 with 'make check': doesn't even compile * mauve+jdk1.4.2 with 'make check': works, but subset of tests only * mauve+jdk1.4.2 with 'ant': doesn't even compile * mauve+jdk1.5.0 with 'ant': doesn't even compile * mauve+classpath+jamvm+jikes: compiles, but crashes on lots of testcases