From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 16663 invoked by alias); 21 Mar 2006 23:08:01 -0000 Received: (qmail 16651 invoked by uid 22791); 21 Mar 2006 23:08:01 -0000 X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.31) with ESMTP; Tue, 21 Mar 2006 23:07:59 +0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.12.11.20060308/8.12.11) with ESMTP id k2LN7vSH003566; Tue, 21 Mar 2006 18:07:57 -0500 Received: from pobox.toronto.redhat.com (pobox.toronto.redhat.com [172.16.14.4]) by int-mx1.corp.redhat.com (8.12.11.20060308/8.11.6) with ESMTP id k2LN7vDG023254; Tue, 21 Mar 2006 18:07:57 -0500 Received: from [172.16.14.67] (towel.toronto.redhat.com [172.16.14.67]) by pobox.toronto.redhat.com (8.12.8/8.12.8) with ESMTP id k2LN7uxX022481; Tue, 21 Mar 2006 18:07:56 -0500 Message-ID: <4420874C.2090800@redhat.com> Date: Tue, 21 Mar 2006 23:08:00 -0000 From: Bryce McKinlay User-Agent: Thunderbird 1.5 (X11/20051025) MIME-Version: 1.0 To: David Gilbert CC: Anthony Balkissoon , classpath@gnu.org, mauve-discuss@sources.redhat.com Subject: Re: Mauve wishlist References: <1142613140.3805.20.camel@rh-ibm-t41> <1142873502.3112.16.camel@tony.toronto.redhat.com> <4420336F.4070602@object-refinery.com> In-Reply-To: <4420336F.4070602@object-refinery.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Mailing-List: contact mauve-discuss-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: mauve-discuss-owner@sourceware.org X-SW-Source: 2006-q1/txt/msg00070.txt.bz2 David Gilbert wrote: > 1) constant number of tests, regardless of exceptions being thrown or > Mauve does have a design flaw where it can be tricky to automatically > assign a unique identifier to each check(), and this makes it hard to > compare two Mauve runs (say a test of the latest Classpath CVS vs the > last release, or the Classpath vs JDK 1.5 - both of which would be > interesting). Right. We all understand the problem - its just the solution is what we need to agree on :) > I think the absolute number is meaningless however you count the > tests, so I don't see this as an advantage. Yes, numbers alone are meaningless, but with the current design, all the results are meaningless without a lot of context. The real issue is having a simple way to uniquely identify each test case for the purpose of identifying regressions. This becomes fundamentally much easier when 1 test() method corresponds to one test case. It is not reasonable to expect test case developers ensure that all tests "run linearly". Exceptions can potentially be thrown at any time, so to ensure linearity, every check() call would need wrapped with a try/catch. > You'll lose the ability to distinguish between an existing failure > where (say) 1 out of 72 checks fail, and after some clever patch 43 > out of 72 checks fail, but the new system reports both as 1 test failure. This is a valid concern. However, we will still track exactly which check() calls fail so that in the event of a test failure, a full diagnostic can be provided. In addition, we can still count the total number of check() calls executed, for statistical purposes. If the reduced test-case granularity does prove to be problematic in some cases - say some test() where a small number of checks fail for cases that are "hard to fix" problems and thus should be "xfailed" (to use gcc/dejagnu lingo), then that test case should probably be split. Alternatively, to avoid splitting things across multiple test classes, we could add JUnit-style support for multiple test() methods in a single test class. Bryce