Re: RFC: Improving support for known testsuite failures

[Richard Guenther <richard.guenther@gmail.com>]

On Wed, Sep 7, 2011 at 5:28 PM, Diego Novillo <dnovillo@google.com> wrote:
> One of the most vexing aspects of GCC development is dealing with
> failures in the various testsuites.  In general, we are unable to
> keep failures down to zero.  We tolerate some failures and tell
> people to "compare your build against a clean build".
>
> This forces developers to either double their testing time by
> building the compiler twice or search in gcc-testresults and hope
> to find a relatively similar build to compare against.
>
> Additionally, the marking mechanisms in DejaGNU are generally
> cumbersome and hard to add.  Even worse, depending on the
> controlling script, there may not be an XFAIL marker at all.
>
> So, while we would ideally keep NO failures in the testsuite, the
> reality is that we are content with having KNOWN failures.  For a
> given set of failures out of 'make check', I would like to have a
> simple filtering mechanism that prunes the known failures out.
>
> Desired features:
>
> - List of known failures lives in SVN.
> - Each target can have its own list.
> - Supports ignoring FAIL, UNRESOLVED and XPASS results.
> - Supports pattern matching to glob sets of failures.
> - Co-exists with the existing XFAIL support in DejaGNU.
> - Supports flaky tests.
> - Supports timestamps to avoid having tests in a knonw-to-fail
>  state forever.
>
> In terms of implementation, this filter could be part of 'make
> check'.  We'd pipe make check's output to it and it would decide
> whether to emit FAIL/UNRESOLVED/XPASS lines based on the black
> list.
>
> I could also make this a post-check filter that runs on all the
> generated <tool>.sum files.  The filter could live in
> <src>/contrib and be used on demand.
>
> I am not thrilled about the prospect of implementing this in
> DejaGNU directly.
>
> Thoughts?

I think it would be more useful to have a script parse gcc-testresults@
postings from the various autotesters and produce a nice webpage
with revisions and known FAIL/XPASSes for the target triplets that
are tested.

That's been a long time on my TODO list, but my web/script FU is
weak enough that I've been pushing that back.

Maybe you have some web-stuff-capable folks at Google even? ;)

Richard.

>
> Thanks.  Diego.
>