RFC: Improving support for known testsuite failures

[Diego Novillo <dnovillo@google.com>]

One of the most vexing aspects of GCC development is dealing with
failures in the various testsuites.  In general, we are unable to
keep failures down to zero.  We tolerate some failures and tell
people to "compare your build against a clean build".

This forces developers to either double their testing time by
building the compiler twice or search in gcc-testresults and hope
to find a relatively similar build to compare against.

Additionally, the marking mechanisms in DejaGNU are generally
cumbersome and hard to add.  Even worse, depending on the
controlling script, there may not be an XFAIL marker at all.

So, while we would ideally keep NO failures in the testsuite, the
reality is that we are content with having KNOWN failures.  For a
given set of failures out of 'make check', I would like to have a
simple filtering mechanism that prunes the known failures out.

Desired features:

- List of known failures lives in SVN.
- Each target can have its own list.
- Supports ignoring FAIL, UNRESOLVED and XPASS results.
- Supports pattern matching to glob sets of failures.
- Co-exists with the existing XFAIL support in DejaGNU.
- Supports flaky tests.
- Supports timestamps to avoid having tests in a knonw-to-fail
  state forever.

In terms of implementation, this filter could be part of 'make
check'.  We'd pipe make check's output to it and it would decide
whether to emit FAIL/UNRESOLVED/XPASS lines based on the black
list.

I could also make this a post-check filter that runs on all the
generated <tool>.sum files.  The filter could live in
<src>/contrib and be used on demand.

I am not thrilled about the prospect of implementing this in
DejaGNU directly.

Thoughts?


Thanks.  Diego.