Re: RFC: Improving support for known testsuite failures

[Diego Novillo <dnovillo@google.com>]

On Thu, Sep 8, 2011 at 07:49, Richard Guenther
<richard.guenther@gmail.com> wrote:

> Well, I'd rather _fix_ dejagnu then.  Any specific example you can't
> eventually xfail by dg-skipping the testcase?

Several I mentioned upthread:
- Some .exp files do no support xfail markers.
- Different directories will have their own syntactic sugar (though
most are using dg-xfail now).
- When using dg-xfail-if, you also need to xfail other dg- markers to
avoid UNRESOLVED.
- Similarly, a dg-xfail-if will cause UNRESOLVED if the test was dg-do
run or dg-do link.
- If you are forced to use dg-skip-if, you will miss state changes in the test.
- Some tests are simply flaky (e.g. the ones that depend on
environment like those using gdb).

I am purposely avoiding tangling with DejaGNU, though if I could find
a way of adding a global known-to-fail marker, that would solve my
problem too.  Janis, do you think that's feasible?

>> Perhaps if there was a global marker that one could add, that would
>> solve my problem too.  I think I'll start with a post-check filter in
>> contrib/
>
> You seem to have a very specific problem ;)

Apparently :)  I am thinking about release and devel branches, in
particular.  That's why I think I'll just add a script in contrib/

> I suppose some
> patch autotester?  Our patch autotester simply bootstraps twice
> and compares the result.

We have those, but doubling the amount of work done by the testers is
not acceptable.  What we need is something that will quickly decide
whether the testsuite run is OK or not.


Diego.