Re: Pushing XFAILed test cases

[Sandra Loosemore <sandra@codesourcery.com>]

On 7/16/21 9:32 AM, Thomas Schwinge wrote:
> 
> [much snipped]
> 
> Of course, we shall assume a certain level of quality in the XFAILed test
> cases: I'm certainly not suggesting we put any random junk into the
> testsuite, coarsely XFAILed.  (I have not reviewed Sandra's test cases to
> that effect, but knowing here, I'd be surprised if that were the problem
> here.)

FWIW, Tobias already did an extensive review of an early version of the 
testsuite patches in question and pointed out several cases where 
failures were due to my misunderstanding of the language standard or 
general confusion about what the expected behavior was supposed to be 
when gfortran wasn't implementing it or was tripping over other bugs. 
:-S  I hope I incorporated all his suggestions and rewrote the 
previously-bogus tests to be more useful for the version I posted for 
review on the Fortran list, but shouldn't the normal patch review 
process be adequate to take care of any additional concerns about quality?

My previous understanding of the development process and testsuite 
conventions is that adding tests that FAIL is bad, but XFAILing them 
with reference to a PR is OK, and certainly much better than simply not 
having test coverage of those things at all.  Especially in the case of 
something like the TS29113 testsuite where the explicit goal is to track 
standards compliance and/or the completeness of the existing 
implementation.  :-S  So it seems to me rather surprising to take the 
position that we should not be committing any new test cases that need 
to be XFAILed.  :-S

-Sandra