Re: RFC: Improving support for known testsuite failures

[Andreas Jaeger <aj@suse.com>]

On Wednesday, September 07, 2011 05:28:15 PM Diego Novillo wrote:
> One of the most vexing aspects of GCC development is dealing with
> failures in the various testsuites.  In general, we are unable to
> keep failures down to zero.  We tolerate some failures and tell
> people to "compare your build against a clean build".
> 
> This forces developers to either double their testing time by
> building the compiler twice or search in gcc-testresults and hope
> to find a relatively similar build to compare against.
> 
> Additionally, the marking mechanisms in DejaGNU are generally
> cumbersome and hard to add.  Even worse, depending on the
> controlling script, there may not be an XFAIL marker at all.
> 
> So, while we would ideally keep NO failures in the testsuite, the
> reality is that we are content with having KNOWN failures.  For a
> given set of failures out of 'make check', I would like to have a
> simple filtering mechanism that prunes the known failures out.
> 
> Desired features:
> 
> - List of known failures lives in SVN.
> - Each target can have its own list.
> - Supports ignoring FAIL, UNRESOLVED and XPASS results.
> - Supports pattern matching to glob sets of failures.
> - Co-exists with the existing XFAIL support in DejaGNU.
> - Supports flaky tests.
> - Supports timestamps to avoid having tests in a knonw-to-fail
>   state forever.
> 
> In terms of implementation, this filter could be part of 'make
> check'.  We'd pipe make check's output to it and it would decide
> whether to emit FAIL/UNRESOLVED/XPASS lines based on the black
> list.
> 
> I could also make this a post-check filter that runs on all the
> generated <tool>.sum files.  The filter could live in
> <src>/contrib and be used on demand.
> 
> I am not thrilled about the prospect of implementing this in
> DejaGNU directly.
> 
> Thoughts?

good idea.

I have my own homegrown script that builds gcc and outputs something like:

trunk: successfull (New passes: 32, new fails: 0, time: 1:55 h)
4.6: successfull (New passes: 28, new fails: 12, time: 1:43 h)

I'll send it to you offline,

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
    GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126