From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 17515 invoked by alias); 7 Sep 2011 18:43:53 -0000 Received: (qmail 17504 invoked by uid 22791); 7 Sep 2011 18:43:52 -0000 X-SWARE-Spam-Status: No, hits=-1.1 required=5.0 tests=AWL,BAYES_00,FRT_OFFER2 X-Spam-Check-By: sourceware.org Received: from cantor2.suse.de (HELO mx2.suse.de) (195.135.220.15) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Wed, 07 Sep 2011 18:43:37 +0000 Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.221.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx2.suse.de (Postfix) with ESMTP id E5EDD8A908 for ; Wed, 7 Sep 2011 20:43:35 +0200 (CEST) From: Andreas Jaeger To: gcc@gcc.gnu.org Subject: Re: RFC: Improving support for known testsuite failures Date: Wed, 07 Sep 2011 18:43:00 -0000 User-Agent: KMail/1.13.7 (Linux/3.0.3-0.0.0.0.e4182fa-desktop; KDE/4.6.5; x86_64; ; ) References: <20110907152813.GA28540@google.com> In-Reply-To: <20110907152813.GA28540@google.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201109072043.33199.aj@suse.com> Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org X-SW-Source: 2011-09/txt/msg00050.txt.bz2 On Wednesday, September 07, 2011 05:28:15 PM Diego Novillo wrote: > One of the most vexing aspects of GCC development is dealing with > failures in the various testsuites. In general, we are unable to > keep failures down to zero. We tolerate some failures and tell > people to "compare your build against a clean build". >=20 > This forces developers to either double their testing time by > building the compiler twice or search in gcc-testresults and hope > to find a relatively similar build to compare against. >=20 > Additionally, the marking mechanisms in DejaGNU are generally > cumbersome and hard to add. Even worse, depending on the > controlling script, there may not be an XFAIL marker at all. >=20 > So, while we would ideally keep NO failures in the testsuite, the > reality is that we are content with having KNOWN failures. For a > given set of failures out of 'make check', I would like to have a > simple filtering mechanism that prunes the known failures out. >=20 > Desired features: >=20 > - List of known failures lives in SVN. > - Each target can have its own list. > - Supports ignoring FAIL, UNRESOLVED and XPASS results. > - Supports pattern matching to glob sets of failures. > - Co-exists with the existing XFAIL support in DejaGNU. > - Supports flaky tests. > - Supports timestamps to avoid having tests in a knonw-to-fail > state forever. >=20 > In terms of implementation, this filter could be part of 'make > check'. We'd pipe make check's output to it and it would decide > whether to emit FAIL/UNRESOLVED/XPASS lines based on the black > list. >=20 > I could also make this a post-check filter that runs on all the > generated .sum files. The filter could live in > /contrib and be used on demand. >=20 > I am not thrilled about the prospect of implementing this in > DejaGNU directly. >=20 > Thoughts? good idea. I have my own homegrown script that builds gcc and outputs something like: trunk: successfull (New passes: 32, new fails: 0, time: 1:55 h) 4.6: successfull (New passes: 28, new fails: 12, time: 1:43 h) I'll send it to you offline, Andreas --=20 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 N=FCrnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imend=F6rffer, HRB 16746 (AG N=FCrn= berg) GPG fingerprint =3D 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126