Re: Pushing XFAILed test cases

[Tobias Burnus <tobias@codesourcery.com>]

Hi all, hi Thomas (2x), hi Sandra,

On 16.07.21 09:52, Thomas Koenig via Fortran wrote:
>> The part of the patch to add tests for this goes on top of my base
>> TS29113 testsuite patch, which hasn't been reviewed or committed yet.
>
> It is my understanding that it is not gcc policy to add xfailed test
> cases for things that do not yet work. Rather, xfail is for tests that
> later turn out not to work, especially on certain architectures.

...

On 17.07.21 09:25, Thomas Koenig via Fortran wrote:
> Is it or is it not gcc policy to push a large number of test cases
> that currently do not work and XFAIL them?

In my opinion, it is bad to add testcases which _only_ consist of
xfails for 'target *-*-*'; however, for an extensive set of test
cases, I think it is better to xfail missing parts than to comment
them out - or not having them at all. That permits a better
test coverage once the features have been implemented.

For the TS29113 patch, which Sandra has posted on July 7, I count:

* 77 'dg-do run' tests - of which 27 are xfailed (35%)
* 28 compile-time tests
* 291 dg-error - of which 59 are xfailed (20%)
* 29 dg-bogus - of which are 25 are xfailed (86%)
(And of course, those lines which are valid do not have
a dg-error - and usually also no dg-bogus.)

And in total:
* 1 '.exp' file
* 105 '.f90' files (with 8232 lines in total including comment lines)
* 53 '.c'files (5281 lines)
* 1 '.h' file (12 lines)

Hence, for me this sounds a rather reasonable amount of xfail.
Especially, given that several pending patches do/will reduce
the amount of xfails by fixing issues exposed by the testsuite
(which has been posted but so far not reviewed).

Of course, in an ideal world, xfail would not exist :-)

On 07.07.21 05:40, Sandra Loosemore wrote:
> There was a question in one of the issues about why this testsuite
> references TS29113 instead of the 2018 standard.  Well, that is what
> our customer is interested in: finding out what parts of the TS29113
> functionality remain unimplemented or broken, and fixing them, so that
> gfortran can say that it implements that specification.

I believe the only real difference between TS29113 and
Fortran 2018's interoperability support is that
'select rank' was added in Fortran 2018.

The testsuite also tests 'select rank'; in that sense,
it is also for Fortran 2018. Thus, ts29113 + ts29113.exp
or 'f2018-c-interop' + 'f2018-c-interop.exp' are both
fine to me. — 'ts29113' is shorter while the other is
clearer to those who did not follow the Fortran standards
and missed that there was a technical specification (TS)
between F2008 and F2018, incorporated (with tiny modifications)
in F2018.

Tobias

-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955