From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 24390 invoked by alias); 21 Mar 2006 22:24:47 -0000 Received: (qmail 24379 invoked by uid 22791); 21 Mar 2006 22:24:46 -0000 X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.31) with ESMTP; Tue, 21 Mar 2006 22:24:43 +0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.12.11.20060308/8.12.11) with ESMTP id k2LMOgh9022334 for ; Tue, 21 Mar 2006 17:24:42 -0500 Received: from opsy.redhat.com (vpn50-157.rdu.redhat.com [172.16.50.157]) by int-mx1.corp.redhat.com (8.12.11.20060308/8.11.6) with ESMTP id k2LMOehF010045; Tue, 21 Mar 2006 17:24:41 -0500 Received: by opsy.redhat.com (Postfix, from userid 500) id D355D2DC18D; Tue, 21 Mar 2006 15:18:41 -0700 (MST) To: David Gilbert Cc: Anthony Balkissoon , classpath@gnu.org, mauve-discuss@sources.redhat.com Subject: Re: Mauve wishlist References: <1142613140.3805.20.camel@rh-ibm-t41> <1142873502.3112.16.camel@tony.toronto.redhat.com> <4420336F.4070602@object-refinery.com> From: Tom Tromey Reply-To: tromey@redhat.com X-Attribution: Tom Date: Tue, 21 Mar 2006 22:24:00 -0000 In-Reply-To: <4420336F.4070602@object-refinery.com> Message-ID: User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3.50 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-IsSubscribed: yes Mailing-List: contact mauve-discuss-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: mauve-discuss-owner@sourceware.org X-SW-Source: 2006-q1/txt/msg00069.txt.bz2 >>>>> "David" == David Gilbert writes: >> Another suggestion that Tom Fitzsimmons had was to change the way we >> count the number of tests. Counting each invocation of the test() >> method rather than each call to harness.check() has two benefits: David> We can work around that by ensuring that all the tests run linearly David> (no if-else branches - I've written a large number of tests this way David> and not found it to be a limitation, but I don't know what lurks in David> the depths of the older Mauve tests). There is still the problem that David> an exception being thrown during a test means some checks don't get David> run, but a new Mauve comparison report (not yet developed, although David> I've done a little experimenting with it) could highlight those. I've always tried to write tests the way you suggest, but the exception problem turns out to be a real one, preventing test stability in some cases. One thing I like about this current proposal is that it automates test stability -- the only failure modes possible are if a test hangs or if the VM crashes. As far as having more granular information -- we can still print a message when a check() fails. A command line option to the test harness could control this, for instance. I think we don't want to just print a plain 'FAIL', we want some explanation; the detailed info could go there. Tom