From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 27987 invoked by alias); 20 Mar 2006 16:51:50 -0000 Received: (qmail 27979 invoked by uid 22791); 20 Mar 2006 16:51:50 -0000 X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.31) with ESMTP; Mon, 20 Mar 2006 16:51:49 +0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.12.11.20060308/8.12.11) with ESMTP id k2KGplxV030096 for ; Mon, 20 Mar 2006 11:51:47 -0500 Received: from pobox.toronto.redhat.com (pobox.toronto.redhat.com [172.16.14.4]) by int-mx1.corp.redhat.com (8.11.6/8.11.6) with ESMTP id k2KGpi108863; Mon, 20 Mar 2006 11:51:45 -0500 Received: from tony.toronto.redhat.com (tony.toronto.redhat.com [172.16.14.158]) by pobox.toronto.redhat.com (8.12.8/8.12.8) with ESMTP id k2KGpgxX012901; Mon, 20 Mar 2006 11:51:42 -0500 Subject: Re: Mauve wishlist From: Anthony Balkissoon To: classpath@gnu.org, mauve-discuss@sources.redhat.com In-Reply-To: <1142613140.3805.20.camel@rh-ibm-t41> References: <1142613140.3805.20.camel@rh-ibm-t41> Content-Type: text/plain Date: Mon, 20 Mar 2006 16:51:00 -0000 Message-Id: <1142873502.3112.16.camel@tony.toronto.redhat.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact mauve-discuss-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: mauve-discuss-owner@sourceware.org X-SW-Source: 2006-q1/txt/msg00067.txt.bz2 On Fri, 2006-03-17 at 11:32 -0500, Thomas Fitzsimmons wrote: > Hi, > > Anthony Balkissoon has expressed interest in improving Mauve so we'd > like to know what would be the best things to work on. > Another suggestion that Tom Fitzsimmons had was to change the way we count the number of tests. Counting each invocation of the test() method rather than each call to harness.check() has two benefits: 1) constant number of tests, regardless of exceptions being thrown or which if-else branch is taken 2) more realistic number of tests, to accurately reflect the extent of our testing For point 1) this will help us see if we are making progress. Right now a Mauve run might say we have 113 fails out of 13200 tests and then a later run could say 200 fails out of 34000 tests. Is this an improvement? Hard to say. But if we count each call to test() as a test, and also detect hanging tests, then we should have a constant number of tests in each run and will be able to say if changes made have a positive impact on Mauve test results. Of course, if in one particular test file there are 1000 calls to harness.check() and only one of them fails, it's not helpful to just report that the entire test failed. So the output will have to pinpoint which call to harness.check failed (and preferably a line number). The negative side here is that the results will be overly pessimistic because any failing harness.check trumps all the passing harness.check calls and the test is reported as a failure. What do people have to say about this idea? --Tony