Re: parallel check output changes?

[Andrew MacLeod <amacleod@redhat.com>]

On 09/24/2014 01:58 PM, Andrew MacLeod wrote:
> On 09/24/2014 12:29 PM, Andrew MacLeod wrote:
>>
>
> AH. interesting.
>
> The third run has a gcc.sum that is exactly the same as the first run. 
> so only the second run differs, and it seems to be from an 
> alphabetical sort.  So run 3 and 1 match.
> the gfortran.sum from the third run is identical to the *second* run, 
> but it is different from the *first* run.  so run 2 and 3 match.
>
> the two runs that match (2nd and 3rd run) look like:
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single  -O2 (test 
> for excess errors)
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single  -O2 
> execution test
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib  -O2 
> -lcaf_single (test for excess errors)
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib  -O2 
> -lcaf_single execution test
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single  -O2 (test 
> for excess errors)
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single  -O2 
> execution test
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib  -O2 
> -lcaf_single (test for excess errors)
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib  -O2 
> -lcaf_single execution test
>
> and the odd one out (firstrun:)
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib  -O2 
> -lcaf_single (test for excess errors)
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=lib  -O2 
> -lcaf_single execution test
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single  -O2 (test 
> for excess errors)
> PASS: gfortran.dg/coarray/this_image_1.f90 -fcoarray=single  -O2 
> execution test
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib  -O2 
> -lcaf_single (test for excess errors)
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=lib  -O2 
> -lcaf_single execution test
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single  -O2 (test 
> for excess errors)
> PASS: gfortran.dg/coarray/this_image_2.f90 -fcoarray=single  -O2 
> execution test
>
> looks like the first run was sorted, and the other 2 weren't.
>
> There must be some condition under which we don't sort the results? or 
> another place which needs to be tweaked to do the sort as well...?
>
> Andrew
>
So to be fair, I could use test_summary, but I think the concern is 
warranted because if this inconsistent ordering can happen to PASS, I 
would expect the same non-deterministic behaviour if those tests happen 
to FAIL.  we just have far less FAILS so we aren't seeing it with 
test_summary at the moment...

Aggregating all my .sum files,  I see a sampling of about 257,000 PASSs, 
whereas I see a total of 141 FAILs.  FAILs only account for < 0.06% of 
the output. ( I'm getting an average of about 510 mis-ordered PASSs, so 
it only affects a small portion of them as well.)

I would think the output of .sum needs to be consistent from one run to 
the next in order for test_summary to consistently report its results as 
well.

Andrew