Invalid "dg-do run" in the testsuite (with 2 blanks)

Invalid "dg-do run" in the testsuite (with 2 blanks)

[Harald Anlauf]

Dear all,

I noticed by chance that we have quite a lot of improperly specified do-do
directives in the testsuite.

% grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
83

Note that with two blanks instead of just one a testcase does not get executed.

Does anybody want to earn the honors to change the directives and
check for "fallout" in the testsuite?

Cheers,
Harald

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Jerry D]

On 3/24/24 1:19 PM, Harald Anlauf wrote:
> Dear all,
> 
> I noticed by chance that we have quite a lot of improperly specified do-do
> directives in the testsuite.
> 
> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
> 83
> 
> Note that with two blanks instead of just one a testcase does not get executed.
> 
> Does anybody want to earn the honors to change the directives and
> check for "fallout" in the testsuite?
> 
> Cheers,
> Harald
> 

I will do this for the betterment of the order of all good people. I 
need the therapy after beating my head against a wall last week eating 
separators. :)

Cheers,

Jerry -

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Jerry D]

On 3/24/24 1:19 PM, Harald Anlauf wrote:
> Dear all,
> 
> I noticed by chance that we have quite a lot of improperly specified do-do
> directives in the testsuite.
> 
> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
> 83
> 
> Note that with two blanks instead of just one a testcase does not get executed.
> 
> Does anybody want to earn the honors to change the directives and
> check for "fallout" in the testsuite?
> 
> Cheers,
> Harald
> 

One failure after fixing all the spaces ( sed is our friend ).

FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times 
optimized "_gfortran_matmul" 0

Jerry -

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Manfred Schwarb]

Am 25.03.24 um 05:53 schrieb Jerry D:
> On 3/24/24 1:19 PM, Harald Anlauf wrote:
>> Dear all,
>>
>> I noticed by chance that we have quite a lot of improperly specified do-do
>> directives in the testsuite.
>>
>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
>> 83
>>

I think this is on purpose.
The idea to use a "feature" in dejagnu to only iterate once and not
over all possible options. So execution time can be lowered a bit.

But I don't know if this hack still works, it definitely did work some years ago.

Cheers,
Manfred


>> Note that with two blanks instead of just one a testcase does not get executed.
>>
>> Does anybody want to earn the honors to change the directives and
>> check for "fallout" in the testsuite?
>>
>> Cheers,
>> Harald
>>
>
> One failure after fixing all the spaces ( sed is our friend ).
>
> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
>
> Jerry -

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Jerry D]

On 3/25/24 1:27 AM, Manfred Schwarb wrote:
> Am 25.03.24 um 05:53 schrieb Jerry D:
>> On 3/24/24 1:19 PM, Harald Anlauf wrote:
>>> Dear all,
>>>
>>> I noticed by chance that we have quite a lot of improperly specified do-do
>>> directives in the testsuite.
>>>
>>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
>>> 83
>>>
> 
> I think this is on purpose.
> The idea to use a "feature" in dejagnu to only iterate once and not
> over all possible options. So execution time can be lowered a bit.
> 
> But I don't know if this hack still works, it definitely did work some years ago.
> 
> Cheers,
> Manfred
> 
> 
>>> Note that with two blanks instead of just one a testcase does not get executed.
>>>
>>> Does anybody want to earn the honors to change the directives and
>>> check for "fallout" in the testsuite?
>>>
>>> Cheers,
>>> Harald
>>>
>>
>> One failure after fixing all the spaces ( sed is our friend ).
>>
>> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
>>
>> Jerry -
> 

Yes it still works, I noticed the tests will do two passes each rather 
than 12 per file.

I will hold off on doing anything with these.

Regards,

Jerry -

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Harald Anlauf]

> >>> I noticed by chance that we have quite a lot of improperly specified do-do
> >>> directives in the testsuite.
> >>>
> >>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
> >>> 83
> >>>
> > 
> > I think this is on purpose.
> > The idea to use a "feature" in dejagnu to only iterate once and not
> > over all possible options. So execution time can be lowered a bit.
> > 
> > But I don't know if this hack still works, it definitely did work some years ago.
> > 
> > Cheers,
> > Manfred

Is this "feature" documented somewhere?  I don't see it on

https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gccint/Directives.html

Given that the dg-directives are important and possibly fragile,
and since we had issues in the past, can we check that a test
that was added works the way intended?

> >>> Note that with two blanks instead of just one a testcase does not get executed.
> >>>
> >>> Does anybody want to earn the honors to change the directives and
> >>> check for "fallout" in the testsuite?
> >>>
> >>> Cheers,
> >>> Harald
> >>>
> >>
> >> One failure after fixing all the spaces ( sed is our friend ).
> >>
> >> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0

This does actually point to an issue with the testcase:
it only works properly with optimization enabled.

Manual inspection of this test and the expected dump suggests
that e.g. -O1 could have been added to the dg-options directive.

Shouldn't we fix at least the dg-options of that testcase?

Cheers,
Harald

> >> Jerry -
> > 
> 
> Yes it still works, I noticed the tests will do two passes each rather 
> than 12 per file.
> 
> I will hold off on doing anything with these.
> 
> Regards,
> 
> Jerry -
>

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Jerry D]

On 3/25/24 12:53 PM, Harald Anlauf wrote:
>>>>> I noticed by chance that we have quite a lot of improperly specified do-do
>>>>> directives in the testsuite.
>>>>>
>>>>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
>>>>> 83
>>>>>
>>>
>>> I think this is on purpose.
>>> The idea to use a "feature" in dejagnu to only iterate once and not
>>> over all possible options. So execution time can be lowered a bit.
>>>
>>> But I don't know if this hack still works, it definitely did work some years ago.
>>>
>>> Cheers,
>>> Manfred
> 
> Is this "feature" documented somewhere?  I don't see it on
> 
> https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gccint/Directives.html
> 
> Given that the dg-directives are important and possibly fragile,
> and since we had issues in the past, can we check that a test
> that was added works the way intended?
> 
>>>>> Note that with two blanks instead of just one a testcase does not get executed.
>>>>>
>>>>> Does anybody want to earn the honors to change the directives and
>>>>> check for "fallout" in the testsuite?
>>>>>
>>>>> Cheers,
>>>>> Harald
>>>>>
>>>>
>>>> One failure after fixing all the spaces ( sed is our friend ).
>>>>
>>>> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
> 
> This does actually point to an issue with the testcase:
> it only works properly with optimization enabled.
> 
> Manual inspection of this test and the expected dump suggests
> that e.g. -O1 could have been added to the dg-options directive.
> 
> Shouldn't we fix at least the dg-options of that testcase?
> 
> Cheers,
> Harald
> 

I restored the one test that appeared to fail so that it had the two 
spaces 'trick'.  When run in the test suite, it is compiled with -O 
which does invoke the optimization.  I manually checked the tree dump 
with this option and it indeed has all the _gfortran_matmul calls removed.

I am inclined to leave these all untouched with the two spaces in place.

 From the test log:

PASS: gfortran.dg/inline_matmul_1.f90   -O  execution test
PASS: gfortran.dg/inline_matmul_1.f90   -O   scan-tree-dump-times 
optimized "_gfortran_matmul" 0

Regards,

Jerry

Aw: Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Harald Anlauf]

> On 3/25/24 12:53 PM, Harald Anlauf wrote:
> >>>>> I noticed by chance that we have quite a lot of improperly specified do-do
> >>>>> directives in the testsuite.
> >>>>>
> >>>>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
> >>>>> 83
> >>>>>
> >>>
> >>> I think this is on purpose.
> >>> The idea to use a "feature" in dejagnu to only iterate once and not
> >>> over all possible options. So execution time can be lowered a bit.
> >>>
> >>> But I don't know if this hack still works, it definitely did work some years ago.
> >>>
> >>> Cheers,
> >>> Manfred
> > 
> > Is this "feature" documented somewhere?  I don't see it on
> > 
> > https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gccint/Directives.html
> > 
> > Given that the dg-directives are important and possibly fragile,
> > and since we had issues in the past, can we check that a test
> > that was added works the way intended?
> > 
> >>>>> Note that with two blanks instead of just one a testcase does not get executed.
> >>>>>
> >>>>> Does anybody want to earn the honors to change the directives and
> >>>>> check for "fallout" in the testsuite?
> >>>>>
> >>>>> Cheers,
> >>>>> Harald
> >>>>>
> >>>>
> >>>> One failure after fixing all the spaces ( sed is our friend ).
> >>>>
> >>>> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
> > 
> > This does actually point to an issue with the testcase:
> > it only works properly with optimization enabled.
> > 
> > Manual inspection of this test and the expected dump suggests
> > that e.g. -O1 could have been added to the dg-options directive.
> > 
> > Shouldn't we fix at least the dg-options of that testcase?
> > 
> > Cheers,
> > Harald
> > 
> 
> I restored the one test that appeared to fail so that it had the two 
> spaces 'trick'.  When run in the test suite, it is compiled with -O 
> which does invoke the optimization.  I manually checked the tree dump 
> with this option and it indeed has all the _gfortran_matmul calls removed.
> 
> I am inclined to leave these all untouched with the two spaces in place.
> 
>  From the test log:
> 
> PASS: gfortran.dg/inline_matmul_1.f90   -O  execution test
> PASS: gfortran.dg/inline_matmul_1.f90   -O   scan-tree-dump-times 
> optimized "_gfortran_matmul" 0

Alright, then leave it that way.

I find it somewhat unsatisfactory though, to have a behavior of the
testsuite harness that is so intranparent.

If it is a simple oversight that the behavior of double space was
never documented, it could simply be fixed, for the sake of everybody.

The take-home message for me is - whenever I write a testcase that
relies on this behavior - to add a comment in the header that this
is intended behavior, and set all compiler flags appropriately...

Cheers,
Harald


> Regards,
> 
> Jerry
> 
>

Re: Aw: Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Jerry D]

On 3/25/24 3:30 PM, Harald Anlauf wrote:
>> On 3/25/24 12:53 PM, Harald Anlauf wrote:
>>>>>>> I noticed by chance that we have quite a lot of improperly specified do-do
>>>>>>> directives in the testsuite.
>>>>>>>
>>>>>>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
>>>>>>> 83
>>>>>>>
>>>>>
>>>>> I think this is on purpose.
>>>>> The idea to use a "feature" in dejagnu to only iterate once and not
>>>>> over all possible options. So execution time can be lowered a bit.
>>>>>
>>>>> But I don't know if this hack still works, it definitely did work some years ago.
>>>>>
>>>>> Cheers,
>>>>> Manfred
>>>
>>> Is this "feature" documented somewhere?  I don't see it on
>>>
>>> https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gccint/Directives.html
>>>
>>> Given that the dg-directives are important and possibly fragile,
>>> and since we had issues in the past, can we check that a test
>>> that was added works the way intended?
>>>
>>>>>>> Note that with two blanks instead of just one a testcase does not get executed.
>>>>>>>
>>>>>>> Does anybody want to earn the honors to change the directives and
>>>>>>> check for "fallout" in the testsuite?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Harald
>>>>>>>
>>>>>>
>>>>>> One failure after fixing all the spaces ( sed is our friend ).
>>>>>>
>>>>>> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
>>>
>>> This does actually point to an issue with the testcase:
>>> it only works properly with optimization enabled.
>>>
>>> Manual inspection of this test and the expected dump suggests
>>> that e.g. -O1 could have been added to the dg-options directive.
>>>
>>> Shouldn't we fix at least the dg-options of that testcase?
>>>
>>> Cheers,
>>> Harald
>>>
>>
>> I restored the one test that appeared to fail so that it had the two
>> spaces 'trick'.  When run in the test suite, it is compiled with -O
>> which does invoke the optimization.  I manually checked the tree dump
>> with this option and it indeed has all the _gfortran_matmul calls removed.
>>
>> I am inclined to leave these all untouched with the two spaces in place.
>>
>>   From the test log:
>>
>> PASS: gfortran.dg/inline_matmul_1.f90   -O  execution test
>> PASS: gfortran.dg/inline_matmul_1.f90   -O   scan-tree-dump-times
>> optimized "_gfortran_matmul" 0
> 
> Alright, then leave it that way.
> 
> I find it somewhat unsatisfactory though, to have a behavior of the
> testsuite harness that is so intranparent.
> 
> If it is a simple oversight that the behavior of double space was
> never documented, it could simply be fixed, for the sake of everybody.
> 
> The take-home message for me is - whenever I write a testcase that
> relies on this behavior - to add a comment in the header that this
> is intended behavior, and set all compiler flags appropriately...
> 
> Cheers,
> Harald
> 
> 

With two spaces there are 9 tests run at -O, with one space there are 54 
variations executed. Specifying the optimization option in the 
dg-options also runs 54 variations.  It appears that the "feature" was 
done on purpose, but it is frustrating that this is not documented anywhere.

I do remember discussions way back about some tests taking a long time 
to run and needing a way to speed up the overall testing.  I will look 
over the documentation and see if we can get this addressed.

For my own test cases, I have always specified the options I want and 
for front-end work, optimizations don't usually matter except in this 
case.  I don't have time to go through all 83 tests.

 From my system here:

Testing of trunk complete..... one space

real	4m56.294s
user	35m51.082s
sys	12m51.283s


Testing of trunk complete..... two spaces

real	4m44.421s
user	34m3.215s
sys	12m38.778s

Not a huge difference.

Jerry

Re: Aw: Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Richard Biener]

On Tue, Mar 26, 2024 at 2:24 AM Jerry D <jvdelisle2@gmail.com> wrote:
>
> On 3/25/24 3:30 PM, Harald Anlauf wrote:
> >> On 3/25/24 12:53 PM, Harald Anlauf wrote:
> >>>>>>> I noticed by chance that we have quite a lot of improperly specified do-do
> >>>>>>> directives in the testsuite.
> >>>>>>>
> >>>>>>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
> >>>>>>> 83
> >>>>>>>
> >>>>>
> >>>>> I think this is on purpose.
> >>>>> The idea to use a "feature" in dejagnu to only iterate once and not
> >>>>> over all possible options. So execution time can be lowered a bit.
> >>>>>
> >>>>> But I don't know if this hack still works, it definitely did work some years ago.
> >>>>>
> >>>>> Cheers,
> >>>>> Manfred
> >>>
> >>> Is this "feature" documented somewhere?  I don't see it on
> >>>
> >>> https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gccint/Directives.html
> >>>
> >>> Given that the dg-directives are important and possibly fragile,
> >>> and since we had issues in the past, can we check that a test
> >>> that was added works the way intended?
> >>>
> >>>>>>> Note that with two blanks instead of just one a testcase does not get executed.
> >>>>>>>
> >>>>>>> Does anybody want to earn the honors to change the directives and
> >>>>>>> check for "fallout" in the testsuite?
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Harald
> >>>>>>>
> >>>>>>
> >>>>>> One failure after fixing all the spaces ( sed is our friend ).
> >>>>>>
> >>>>>> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
> >>>
> >>> This does actually point to an issue with the testcase:
> >>> it only works properly with optimization enabled.
> >>>
> >>> Manual inspection of this test and the expected dump suggests
> >>> that e.g. -O1 could have been added to the dg-options directive.
> >>>
> >>> Shouldn't we fix at least the dg-options of that testcase?
> >>>
> >>> Cheers,
> >>> Harald
> >>>
> >>
> >> I restored the one test that appeared to fail so that it had the two
> >> spaces 'trick'.  When run in the test suite, it is compiled with -O
> >> which does invoke the optimization.  I manually checked the tree dump
> >> with this option and it indeed has all the _gfortran_matmul calls removed.
> >>
> >> I am inclined to leave these all untouched with the two spaces in place.
> >>
> >>   From the test log:
> >>
> >> PASS: gfortran.dg/inline_matmul_1.f90   -O  execution test
> >> PASS: gfortran.dg/inline_matmul_1.f90   -O   scan-tree-dump-times
> >> optimized "_gfortran_matmul" 0
> >
> > Alright, then leave it that way.
> >
> > I find it somewhat unsatisfactory though, to have a behavior of the
> > testsuite harness that is so intranparent.
> >
> > If it is a simple oversight that the behavior of double space was
> > never documented, it could simply be fixed, for the sake of everybody.
> >
> > The take-home message for me is - whenever I write a testcase that
> > relies on this behavior - to add a comment in the header that this
> > is intended behavior, and set all compiler flags appropriately...
> >
> > Cheers,
> > Harald
> >
> >
>
> With two spaces there are 9 tests run at -O, with one space there are 54
> variations executed. Specifying the optimization option in the
> dg-options also runs 54 variations.  It appears that the "feature" was
> done on purpose, but it is frustrating that this is not documented anywhere.

It would be nice to document it in sourcebuild.texi and to put a comment before
the dg-do with two spaces indicating that two spaces are on purpose.

I also don't remember this being implemented ...

Richard.

> I do remember discussions way back about some tests taking a long time
> to run and needing a way to speed up the overall testing.  I will look
> over the documentation and see if we can get this addressed.
>
> For my own test cases, I have always specified the options I want and
> for front-end work, optimizations don't usually matter except in this
> case.  I don't have time to go through all 83 tests.
>
>  From my system here:
>
> Testing of trunk complete..... one space
>
> real    4m56.294s
> user    35m51.082s
> sys     12m51.283s
>
>
> Testing of trunk complete..... two spaces
>
> real    4m44.421s
> user    34m3.215s
> sys     12m38.778s
>
> Not a huge difference.
>
> Jerry
>
>

Re: Aw: Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Mikael Morin]

Le 25/03/2024 à 23:30, Harald Anlauf a écrit :
>> On 3/25/24 12:53 PM, Harald Anlauf wrote:
>>>>>>>
(...)
>>>>>>
>>>>>> One failure after fixing all the spaces ( sed is our friend ).
>>>>>>
>>>>>> FAIL: gfortran.dg/inline_matmul_1.f90   -O0   scan-tree-dump-times optimized "_gfortran_matmul" 0
>>>
>>> This does actually point to an issue with the testcase:
>>> it only works properly with optimization enabled.
>>>
>>> Manual inspection of this test and the expected dump suggests
>>> that e.g. -O1 could have been added to the dg-options directive.
>>>
>>> Shouldn't we fix at least the dg-options of that testcase?
>>>
For what it's worth, the scan-tree-dump-times directive can be 
conditionalized on target __OPTIMIZE__, so that it's active with 
optimization and inactive with -O0.  This would make the test pass 
regardless of compiler flags.
Of course if the goal is lowering the test execution time, it wouldn't help.

(...)
> 
> I find it somewhat unsatisfactory though, to have a behavior of the
> testsuite harness that is so intranparent.
> 
Agreed.

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Andrew Pinski]

On Sun, Mar 24, 2024 at 1:19 PM Harald Anlauf <anlauf@gmx.de> wrote:
>
> Dear all,
>
> I noticed by chance that we have quite a lot of improperly specified do-do
> directives in the testsuite.
>
> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
> 83
>
> Note that with two blanks instead of just one a testcase does not get executed.
>
> Does anybody want to earn the honors to change the directives and
> check for "fallout" in the testsuite?

Note this seems related to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=28032 (or maybe this is
the "workaround" for not doing torture options on the testcases).

If this gets documented to avoid doing the torture options, then the
testsuite should be auditted for dg-options with `-ON` and add the
extra space for dg-do to get the non torture way.

Thanks,
Andrew

>
> Cheers,
> Harald
>

Re: Invalid "dg-do run" in the testsuite (with 2 blanks)

[Jerry D]

On 4/1/24 11:46 PM, Andrew Pinski wrote:
> On Sun, Mar 24, 2024 at 1:19 PM Harald Anlauf <anlauf@gmx.de> wrote:
>>
>> Dear all,
>>
>> I noticed by chance that we have quite a lot of improperly specified do-do
>> directives in the testsuite.
>>
>> % grep "dg-do  run" gcc/testsuite/gfortran.dg/ -rl|wc -l
>> 83
>>
>> Note that with two blanks instead of just one a testcase does not get executed.
>>
>> Does anybody want to earn the honors to change the directives and
>> check for "fallout" in the testsuite?
> 
> Note this seems related to
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=28032 (or maybe this is
> the "workaround" for not doing torture options on the testcases).
> 
> If this gets documented to avoid doing the torture options, then the
> testsuite should be auditted for dg-options with `-ON` and add the
> extra space for dg-do to get the non torture way.
> 
> Thanks,
> Andrew
> 
>>
>> Cheers,
>> Harald
>>

Thanks for this additional info. I have not settled on exactly what to say in the documentation.

I noticed that when I specified a specific -ON it still did them all regardless.  I did not know if this was expected or a feature.

Jerry -