From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 21523 invoked by alias); 18 Nov 2005 19:08:10 -0000 Received: (qmail 21515 invoked by uid 22791); 18 Nov 2005 19:08:07 -0000 Received: from mail.gmx.de (HELO mail.gmx.net) (213.165.64.20) by sourceware.org (qpsmtpd/0.30-dev) with SMTP; Fri, 18 Nov 2005 19:08:07 +0000 Received: (qmail invoked by alias); 18 Nov 2005 19:08:04 -0000 Received: from 85-124-81-148.dynamic.xdsl-line.inode.at (EHLO yukon) [85.124.81.148] by mail.gmx.net (mp001) with SMTP; 18 Nov 2005 20:08:04 +0100 X-Authenticated: #9981821 Received: by yukon (Postfix, from userid 1000) id 5E36E55449D; Fri, 18 Nov 2005 20:08:04 +0100 (CET) Date: Fri, 18 Nov 2005 19:08:00 -0000 From: Edwin Steiner To: mauve-discuss@sources.redhat.com Subject: inconsistent test names Message-ID: <20051118190804.GA10989@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.9i X-Y-GMX-Trusted: 0 X-IsSubscribed: yes Mailing-List: contact mauve-discuss-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: mauve-discuss-owner@sourceware.org X-SW-Source: 2005-q4/txt/msg00034.txt.bz2 Hello! I'm one of the developers of the cacao jvm[1]. We use mauve intensively to test cacao and it is a very good test suite. Currently I'm developing an automated testing framework for cacao[2]. You can see preliminary results here (large page!): http://www.complang.tuwien.ac.at/cacaojvm//tgolem/2005-11-18T03-22/mauve.html One problem I've run accross is that some mauve tests use inconsistent names to report PASSes and FAILs. The most troublesome cases are those, where a *named* check/fail is followed by an *unnamed* one. For example: gnu/testlet/java/net/MulticastSocket/MulticastSocketTest.java:207 there is a harness.fail("could not create FIRST multicast socket on shared port " + sharedMcastPort); at this line. The next check on line 216 and unnamed. Now if the first check fails, and the second succeeds, you get a bogus PASS line like: PASS: gnu.testlet.java.net.MulticastSocket.MulticastSocketTest: could not create SECOND multicast socket on shared port 1234 (number 2) from the harness. If you also think these cases should be fixed (by naming the checks), I would be happy to provide patches. Makes more sense than teaching my tgolem to work around these cases. -Edwin [1] http://www.cacaojvm.org/ [2] http://www.cacaojvm.org/tgolem Most of the tgolem code is not cacao specific and I plan to release it as a GPL project within the next months.