From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 25627 invoked by alias); 7 Sep 2011 15:28:47 -0000 Received: (qmail 25614 invoked by uid 22791); 7 Sep 2011 15:28:44 -0000 X-SWARE-Spam-Status: No, hits=-2.4 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RP_MATCHES_RCVD,SPF_HELO_PASS X-Spam-Check-By: sourceware.org Received: from smtp-out.google.com (HELO smtp-out.google.com) (216.239.44.51) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Wed, 07 Sep 2011 15:28:18 +0000 Received: from hpaq6.eem.corp.google.com (hpaq6.eem.corp.google.com [172.25.149.6]) by smtp-out.google.com with ESMTP id p87FSHVT028302 for ; Wed, 7 Sep 2011 08:28:17 -0700 Received: from tobiano.tor.corp.google.com (tobiano.tor.corp.google.com [172.29.41.6]) by hpaq6.eem.corp.google.com with ESMTP id p87FSFfl025847 for ; Wed, 7 Sep 2011 08:28:16 -0700 Received: by tobiano.tor.corp.google.com (Postfix, from userid 54752) id 501A6AE1E5; Wed, 7 Sep 2011 11:28:15 -0400 (EDT) Date: Wed, 07 Sep 2011 15:28:00 -0000 From: Diego Novillo To: gcc@gcc.gnu.org Subject: RFC: Improving support for known testsuite failures Message-ID: <20110907152813.GA28540@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) X-System-Of-Record: true X-IsSubscribed: yes Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org X-SW-Source: 2011-09/txt/msg00044.txt.bz2 One of the most vexing aspects of GCC development is dealing with failures in the various testsuites. In general, we are unable to keep failures down to zero. We tolerate some failures and tell people to "compare your build against a clean build". This forces developers to either double their testing time by building the compiler twice or search in gcc-testresults and hope to find a relatively similar build to compare against. Additionally, the marking mechanisms in DejaGNU are generally cumbersome and hard to add. Even worse, depending on the controlling script, there may not be an XFAIL marker at all. So, while we would ideally keep NO failures in the testsuite, the reality is that we are content with having KNOWN failures. For a given set of failures out of 'make check', I would like to have a simple filtering mechanism that prunes the known failures out. Desired features: - List of known failures lives in SVN. - Each target can have its own list. - Supports ignoring FAIL, UNRESOLVED and XPASS results. - Supports pattern matching to glob sets of failures. - Co-exists with the existing XFAIL support in DejaGNU. - Supports flaky tests. - Supports timestamps to avoid having tests in a knonw-to-fail state forever. In terms of implementation, this filter could be part of 'make check'. We'd pipe make check's output to it and it would decide whether to emit FAIL/UNRESOLVED/XPASS lines based on the black list. I could also make this a post-check filter that runs on all the generated .sum files. The filter could live in /contrib and be used on demand. I am not thrilled about the prospect of implementing this in DejaGNU directly. Thoughts? Thanks. Diego.