[Bug testsuite/25352] New: xfail within dg-do command has no effect

[Bug testsuite/25352] New: xfail within dg-do command has no effect

[ghazi at gcc dot gnu dot org]

If you use an xfail command within a dg-do, it has no effect.  E.g this generic
testcase will FAIL rather than XFAIL:

/* { dg-do preprocess { xfail *-*-* } } */

#error intentional failure!

int main(void)
{
  return 0;
}

Right now, none of the ~100 testcases in the testsuite using this idiom are
honoring it.  So they're either now passing or unnecessarily cluttering up
testsuite results.


-- 
           Summary: xfail within dg-do command has no effect
           Product: gcc
           Version: 4.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: testsuite
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: ghazi at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[ghazi at gcc dot gnu dot org]

------- Comment #1 from ghazi at gcc dot gnu dot org  2005-12-11 15:14 -------
We probably should have some sort of dg conformance test directory with simple
tests ensuring all the directives work as advertised.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #2 from janis at gcc dot gnu dot org  2005-12-12 17:57 -------
This is odd, if the test case has a syntax error the xfail works as expected,
but with #error it doesn't.  I'll look into it.

Take a look at gcc/testsuite/gcc.test-framework, which tests lots of
combinations
of directives as well as specific cases.  I run it as part of my nightly build
of
trunk.  It's meant to be run separately from the rest of the testsuite since
the
outcome of a test is not the results that the test itself reports, but whether
the result is the expected one.


-- 

janis at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at gcc dot gnu   |janis at gcc dot gnu dot org
                   |dot org                     |
             Status|UNCONFIRMED                 |ASSIGNED
     Ever Confirmed|0                           |1
   Last reconfirmed|0000-00-00 00:00:00         |2005-12-12 17:57:54
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #3 from janis at gcc dot gnu dot org  2005-12-12 22:14 -------
How embarrassing, I tried these things by adding new tests to
gcc.test-framework and the awk script that checks the results was ignoring some
of the results because of magic needed for the generated tests.  Now I'm able
to properly reproduce failures.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #4 from janis at gcc dot gnu dot org  2005-12-13 19:08 -------
xfail within a dg-do command other than run has no effect, at least not with
DejaGnu 1.4.4 used with GCC at least as far back as 3.0.  The DejaGnu
documentation doesn't mention the dg-* directives.  The comments in
dejagnu/dg.exp file implies that xfail should be recognized for all dg-do
options.

There are 50+ tests in gcc/testsuite that use xfail with dg-do assemble,
compile, and link, so perhaps it worked at one time.

The generated tests in gcc/testsuite/gcc.test-framework only test xfail with
"dg-do run", and there are no permanent tests for xfail with dg-do and other
actions.

There's undoubtedly a way for the GCC dg scripts to capture the information
that the test's dg-do had an xfail, although it will probably be ugly.  I'll
keep investigating to find out just how ugly it will need to be.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #5 from janis at gcc dot gnu dot org  2005-12-14 18:17 -------
I've got an ugly fix that reaches up a couple of levels to get the variable
from the DejaGnu proc that records that the test should be xfailed.  Ben
Elliston, who is a DejaGnu maintainer, says that 1.4.5 will be released in a
month or so and it could be fixed there as well.  Having GCC depend on a fix in
DejaGnu would not be popular unless there were other compelling reasons to
require people to upgrade.  Thoughts?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[ghazi at gcc dot gnu dot org]

------- Comment #6 from ghazi at gcc dot gnu dot org  2005-12-14 18:36 -------
(In reply to comment #5)
> Having GCC depend on a fix in
> DejaGnu would not be popular unless there were other compelling reasons to
> require people to upgrade.  Thoughts?

If the breakage is in dejagnu, certainly a fix should go there.  However I
don't feel strongly about whether to fix it in GCC additionally and separately.
 If forced to pick, I'd lean towards dejagnu only since I'll upgrade once the
new version is out.

If we don't fix it in GCC, then we should remove the documentation that says it
works.  Perhaps we could recommend dg-xfail-if instead?

Either way we should probably continue to remove the existing dg-do xfails
since we'll start getting a bunch of spurious XPASSes once people start
upgrading to the new dejagnu.  I'll try and handle the ones that trigger on my
platforms but some I can't test to know whether simply to remove the xfail or
change it to dg-xfail-if.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[ghazi at gcc dot gnu dot org]

------- Comment #7 from ghazi at gcc dot gnu dot org  2007-05-10 17:44 -------
Ben, Janis,

Would you please offer an update on the status of this PR?  I'm running into
another situation where I need to xfail a dg-do link, and due to the issues
discussed in this PR it doesn't work.  Has dejagnu cvs sources been fixed to
handle this case?  Has 1.4.5 been released?  (I don't see it on ftp.gnu.org).

Thanks,
--Kaveh


-- 

ghazi at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bje at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #8 from janis at gcc dot gnu dot org  2007-05-11 00:51 -------
I had forgotten all about this.  Today I checked to see how older versions of
DejaGnu and GCC tests handled xfail for dg-do keywords other than "run".  I
only went back as far as GCC 3.0.4 and DejaGnu 1.4, but they also ignored xfail
in dg-do for preprocess/compile/assemble/link.

There are currently 27 tests that use this construct: three in libstdc++-v3 and
24 in gcc.

My recommendations are to document, in the description of test directives in
the GCC Internals manual, that xfail for dg-do is only honored for dg-do run
and ignored for other keywords, recommending the use of dg-xfail-if instead;
and to remove the xfail section from the 27 tests that use it when it's
ignored.

I'll do these if it sounds reasonable.  I don't think this is worth working
around in the GCC testsuite support or fixing in DejaGnu.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[ghazi at gcc dot gnu dot org]

------- Comment #9 from ghazi at gcc dot gnu dot org  2007-05-15 17:47 -------
(In reply to comment #8)
> My recommendations are to document, in the description of test directives in
> the GCC Internals manual, that xfail for dg-do is only honored for dg-do run
> and ignored for other keywords, recommending the use of dg-xfail-if instead;
> and to remove the xfail section from the 27 tests that use it when it's
> ignored.
> I'll do these if it sounds reasonable.  I don't think this is worth working
> around in the GCC testsuite support or fixing in DejaGnu.

Sounds fine.  I just want to clear up the confusion.  There may be some
similarly vague stuff in the Wiki (on the topic of testcases) that should be
clarified as well IMHO.

Thanks!


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #10 from janis at gcc dot gnu dot org  2007-11-01 16:22 -------
Subject: Bug 25352

Author: janis
Date: Thu Nov  1 16:22:36 2007
New Revision: 129823

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=129823
Log:
gcc/
        PR testsuite/25352
        * doc/sourcebuild.texi (Test Directives): Document that xfail
        has no effect for dg-do except when dg-do-what is "run".

libstdc++-v3/
        PR testsuite/25352
        * testsuite/26_numerics/headers/cmath/c99_classification_macros_c.cc:
        Remove xfail from dg-do and add dg-skip-if.

gcc/testsuite/
        PR testsuite/25352
        * gcc.c-torture/compile/20000804-1.c: Remove xfail from dg-do and
        add dg-skip-if.
        * gcc.c-torture/compile/20001205-1.c: Ditto.
        * gcc.c-torture/compile/20001226-1.c: Ditto.
        * gcc.c-torture/compile/20020312-1.c: Ditto.
        * gcc.c-torture/compile/20020604-1.c: Ditto.
        * gcc.c-torture/compile/920501-12.c: Ditto.
        * gcc.c-torture/compile/920501-4.c: Ditto.
        * gcc.c-torture/compile/920520-1.c: Ditto.
        * gcc.c-torture/compile/980506-1.c: Ditto.
        * gcc.c-torture/compile/990617-1.c: Ditto.
        * gcc.dg/compare6.c: Ditto.
        * g++.old-deja/g++.brendan/crash43.C: Ditto.
        * g++.old-deja/g++.brendan/synth1.C: Ditto.
        * g++.old-deja/g++.oliva/dwarf2.C: Ditto.
        * g++.old-deja/g++.oliva/dwarf3.C: Ditto.
        * g++.old-deja/g++.pt/static6.C: Ditto.
        * g++.old-deja/g++.pt/crash16.C: Ditto.
        * g++.old-deja/g++.mike/p10416.C: Ditto.
        * g++.old-deja/g++.mike/ns15.C: Ditto.
        * g++.old-deja/g++.mike/eh46.C: Ditto.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/doc/sourcebuild.texi
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/g++.old-deja/g++.brendan/crash43.C
    trunk/gcc/testsuite/g++.old-deja/g++.brendan/synth1.C
    trunk/gcc/testsuite/g++.old-deja/g++.mike/eh46.C
    trunk/gcc/testsuite/g++.old-deja/g++.mike/ns15.C
    trunk/gcc/testsuite/g++.old-deja/g++.mike/p10416.C
    trunk/gcc/testsuite/g++.old-deja/g++.oliva/dwarf2.C
    trunk/gcc/testsuite/g++.old-deja/g++.oliva/dwarf3.C
    trunk/gcc/testsuite/g++.old-deja/g++.pt/crash16.C
    trunk/gcc/testsuite/g++.old-deja/g++.pt/static6.C
    trunk/gcc/testsuite/gcc.c-torture/compile/20000804-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/20001205-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/20001226-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/20020312-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/20020604-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/920501-12.c
    trunk/gcc/testsuite/gcc.c-torture/compile/920501-4.c
    trunk/gcc/testsuite/gcc.c-torture/compile/920520-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/980506-1.c
    trunk/gcc/testsuite/gcc.c-torture/compile/990617-1.c
    trunk/gcc/testsuite/gcc.dg/compare6.c
    trunk/libstdc++-v3/ChangeLog
   
trunk/libstdc++-v3/testsuite/26_numerics/headers/cmath/c99_classification_macros_c.cc


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352

[Bug testsuite/25352] xfail within dg-do command has no effect

[janis at gcc dot gnu dot org]

------- Comment #11 from janis at gcc dot gnu dot org  2008-05-13 22:19 -------
Fixed, via documentation and workarounds.


-- 

janis at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=25352