parallel testsuite results differ from a single task approach

parallel testsuite results differ from a single task approach

[Dennis Clarke]

I gad received some good advice to try using gmake -j X -k check for some X > 1 when running a testsuite in order to speed the process up. So I decided to test that theory with an eye on the results. I was surprised to see that the results when running "gmake -j 8 -k check" differ from a simple "gmake -k check" in the area of go and libmudflap : 

    1 ) parallel testsuite : 

        http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01903.html

    2 ) single process testsuite : 

        http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01935.html


In parallel we see this for "go" : 

                === go tests ===


Running target unix
FAIL: go.test/test/stack.go execution,  -O2 -g 

                === go Summary ===

# of expected passes            3293
# of unexpected failures        1
# of expected failures          4
# of untested testcases         4
/home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo  version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00) 




While the single process apporach reports : 


                === go Summary ===

# of expected passes            3294
# of expected failures          4
# of untested testcases         4
/home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo  version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00) 


Perhaps the go test "go.test/test/stack.go" is somehow sensitive to running on a busy machine? I am only guessing. 

The "libmudflap tests" results are wildly different. 

So at first glance it appears that the testsuite will in fact product different results if one chooses to use "gmake -j X" for X>1 but I am not sure why.  Felt is was worth pointing out as maybe no one else has done the comparison.

Dennis 

Re: parallel testsuite results differ from a single task approach

[Tim Prince]

On 12/20/2012 11:06 PM, Dennis Clarke wrote:
> I gad received some good advice to try using gmake -j X -k check for some X > 1 when running a testsuite in order to speed the process up. So I decided to test that theory with an eye on the results. I was surprised to see that the results when running "gmake -j 8 -k check" differ from a simple "gmake -k check" in the area of go and libmudflap :
>
>      1 ) parallel testsuite :
>
>          http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01903.html
>
>      2 ) single process testsuite :
>
>          http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01935.html
>
>
> In parallel we see this for "go" :
>
>                  === go tests ===
>
>
> Running target unix
> FAIL: go.test/test/stack.go execution,  -O2 -g
>
>                  === go Summary ===
>
> # of expected passes            3293
> # of unexpected failures        1
> # of expected failures          4
> # of untested testcases         4
> /home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo  version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00)
>
>
>
>
> While the single process apporach reports :
>
>
>                  === go Summary ===
>
> # of expected passes            3294
> # of expected failures          4
> # of untested testcases         4
> /home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo  version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00)
>
>
> Perhaps the go test "go.test/test/stack.go" is somehow sensitive to running on a busy machine? I am only guessing.
>
> The "libmudflap tests" results are wildly different.
>
> So at first glance it appears that the testsuite will in fact product different results if one chooses to use "gmake -j X" for X>1 but I am not sure why.  Felt is was worth pointing out as maybe no one else has done the comparison.
>
> Dennis
>
>
Did you read any reports from recent years about problems with this?  If 
a test dupiicates itself while accessing the same file system, it may 
create a kind of race condition.
If the additional failures are time-outs, it's to be expected on a "busy 
machine."  In particular, you may have multiple tests contending for a 
single rotating disk.  Number of cores, OS, and other factors will make 
a large difference.  How are we to know what you aren't telling?

-- 
Tim Prince

Re: parallel testsuite results differ from a single task approach

[Dennis Clarke]

>  How are we to know what you aren't telling?

I am generally very detailed in that regard. 

I think that I will refrain from running gmake -j X for X > ( number of cores available -2 ) 
and then the problem may go away. 

Otherwise a single "gmake" is safest. Slow. But safe. 

dc