legitimate parallel make check?

legitimate parallel make check?

[IainS]

for the sake of keeping information segregated in log files (and  
balancing out test times)
  I tried the following (from within a regression testing script):

'make -k check-gcc-c RUNTESTFLAGS=$runTEST >${STATE}/$svnRevision- 
check-c-log.txt 2>&1' &
(sleep 2) 'make -k check-gcc-c++ RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-c++-log.txt 2>&1' &
(sleep 2) 'make -k check-gcc-fortran RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-fortran-log.txt 2>&1' &
(sleep 2) ' make -k check-gcc-objc RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-objc-log.txt 2>&1' &
(sleep 2) 'make -k check-gcc-obj-c++ RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-obc++-log.txt 2>&1' &
(sleep 2) 'make -k check-target-libstdc++-v3 RUNTESTFLAGS=$runTEST >$ 
{STATE}/$svnRevision-check-target-libstdc++-log.txt 2>&1'&
(sleep 2) 'make -k check-target-libgomp RUNTESTFLAGS=$runTEST >$ 
{STATE}/$svnRevision-check-target-libgomp-log.txt 2>&1' &

firstly; this actually breaks the make process permanently (I suspect  
a race to make site.exp) unless the parallel jobs are launched with a  
time delay.
by permanently I mean that after trying this - "make check-gcc"  fails  
from the command line until config.status is re-run in the gcc dir.

====

secondly;  if I use  "-k -j3 check-gcc-c "
    together with "-k -j2 check-gcc-fortran " (for example)

I get sporadic bus errors :
/bin/sh: line 1: 67618 Bus error               `if [ -f ${srcdir}/../ 
dejagnu/runtest ] ; then echo ${srcdir}/../dejagnu/runtest ; else echo  
runtest; fi` --tool gfortran --target_board=unix\{-m32,-m64\}  
$runtestflags
make[2]: [check-parallel-gfortran] Error 138 (ignored)

large chunks of tests go silently missing at this point...

===

Am I trying something that is unsupported - or is this is a bug?

===

"make -k -j8 check " is not particularly helpful (a) because it tests  
gmp/mpfr/mpc every time (b) any redirected output is hard to scan for  
problems.

Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

> Am I trying something that is unsupported - or is this is a bug?
>
> ===
>
> "make -k -j8 check " is not particularly helpful (a) because it tests
> gmp/mpfr/mpc every time (b) any redirected output is hard to scan for
> problems.

What you're trying is far too complicated: just use -k -j<N> check from
the toplevel and forget about the make check output, which, as you say,
is interleaved and useless, just like the make -j<N> bootstrap output.

Instead, run make mail-report.log afterwards and check that.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 12:10, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>
>> Am I trying something that is unsupported - or is this is a bug?
>>
>> ===
>>
>> "make -k -j8 check " is not particularly helpful (a) because it tests
>> gmp/mpfr/mpc every time (b) any redirected output is hard to scan for
>> problems.
>
> What you're trying is far too complicated: just use -k -j<N> check  
> from
> the toplevel and forget about the make check output, which, as you  
> say,
> is interleaved and useless, just like the make -j<N> bootstrap output.

OK.

> Instead, run make mail-report.log afterwards and check that.

Although, I note that contrib/test_summary only reports what gets into  
the *.sum files -- i.e. tests that complete or timeout.
It doesn't log problems in the actual make process itself - to find  
those one still has to trawl through the interleaved stuff.

It would be nice to allow the apparently independent targets [e.g. gcc- 
c,fortran,c++ etc.] to be (explicitly) make-checked in parallel.

thanks,
Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

>> Instead, run make mail-report.log afterwards and check that.
>
> Although, I note that contrib/test_summary only reports what gets into the
> *.sum files -- i.e. tests that complete or timeout.
> It doesn't log problems in the actual make process itself - to find those
> one still has to trawl through the interleaved stuff.

No, you haven't.  You need to look at the corresponding *.log files
instead; most of this isn't in the regular make check output anyway.

> It would be nice to allow the apparently independent targets [e.g. gcc-
> c,fortran,c++ etc.] to be (explicitly) make-checked in parallel.

This is exactly what happens when you invoke make -j<N> -k check,
provided that the parallelism specified is big enough (and the machine
can handle the load).  Otherwise, you may end up running (say) the c
testsuite in parallel, which isn't bad either.

As I wrote before, I'm going to use this on an (effectively) 64-core
machine and hope to achieve to use all cores in parallel :-)

        Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 12:46, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>
>>> Instead, run make mail-report.log afterwards and check that.

... <snip>

>> suite in parallel, which isn't bad either.
>
> As I wrote before, I'm going to use this on an (effectively) 64-core
> machine and hope to achieve to use all cores in parallel :-)

on my mere 8 cores :-)   I'm still getting sporadic Bus errors on:

"make -k -j8 check RUNTESTFLAGS ... "
[from the command line on a bootstrapped clean trunk @157307]

so there's something else wrong somewhere...
... I guess a target-related issue ({i686,powerpc}-apple-darwin9)

(note that this error does *not* get into the .log files, AFAICS)

Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

> on my mere 8 cores :-)   I'm still getting sporadic Bus errors on:
>
> "make -k -j8 check RUNTESTFLAGS ... "
> [from the command line on a bootstrapped clean trunk @157307]
>
> so there's something else wrong somewhere...

Probably make gets SIGBUS, which obviously won't be in the runtest output.

> ... I guess a target-related issue ({i686,powerpc}-apple-darwin9)

Or simply a hardware issue?

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 19:13, Janis Johnson wrote:

> To run all of the compiler tests in parallel you can do "make -jn -k  
> check-gcc" from the top level and let the existing build machinery  
> take care of running chunks of tests in parallel and putting the  
> results back together.

that's fine (I couldn't see why not from looking at Makefile.in)
  (maybe Rainer is right and I have a memory fault on my 8-core  
box .. :-(  ) -- just backing up prior to a single user check of  
that ...

..  I don't seem to get the bus errors on a 4CPU g5 or a Core 2 duo ..  
but .. the 8-core machine is faster .. so ... race conditions are more  
likely to manifest there.

> You'd still have to handle the library tests you want to run.

I assume that " make -jn -k check-target RUNTESTFLAGS.... " is equally  
reasonable?

> Perhaps what you want is a way to skip the host library testing,  
> although if you're building them with your new compiler then testing  
> them is a good part of testing that compiler.

I do build gmp/mpfr/mpc in-tree...
given that I almost always run the whole testsuite - I'm torn between  
thinking it's a good idea..
... and that I'm essentially proving little when the versions of gmp/ 
mpfr/and mpc are static.

--

FWIW: I'm trying to update contrib/btest.sh script to handle m32 & m64  
plus a few other things ...

Iain

Re: legitimate parallel make check?

[NightStrike]

On Tue, Mar 9, 2010 at 2:27 PM, IainS <developer@sandoe-acoustics.co.uk> wrote:
> I do build gmp/mpfr/mpc in-tree...

How?  Last I tried, it didn't work, as mpc used the system gmp/mpfr,
not the just-built in-tree versions.  Therefore, it's not really an
"in-tree" build, and you can't build on a system that doesn't already
have gmp/mpfr installed.  At least, that's what it was... does that
work now?

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 19:31, NightStrike wrote:

> On Tue, Mar 9, 2010 at 2:27 PM, IainS <developer@sandoe-acoustics.co.uk 
> > wrote:
>> I do build gmp/mpfr/mpc in-tree...
>
> How?  Last I tried, it didn't work, as mpc used the system gmp/mpfr,
> not the just-built in-tree versions.  Therefore, it's not really an
> "in-tree" build, and you can't build on a system that doesn't already
> have gmp/mpfr installed.  At least, that's what it was... does that
> work now?

it's been  fixed for a while (resolution of PR42424)
I keep one copy of gmp mpfr and mpc in a directory at the root of my  
source disk and ln -s to that from within the different gcc source  
trees.

Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

> ..  I don't seem to get the bus errors on a 4CPU g5 or a Core 2 duo .. but
> .. the 8-core machine is faster .. so ... race conditions are more  likely
> to manifest there.

But race conditions don't manifest themselves in make SEGVs ;-(  I'm
regularly running make -k -j128 on a T5220, or -j32 on a Sun Fire X4450
with 4 4-core CPUs, without any problems.

>> You'd still have to handle the library tests you want to run.
>
> I assume that " make -jn -k check-target RUNTESTFLAGS.... " is equally
> reasonable?

Indeed, as well as using a site.exp and pointing the DEJAGNU environment
variable at it, as I've learned from looking at the regression tester
sources in contrib/regression.  Here's what I use for that:

global target_list

case "$target_triplet" in {
    { "i?86-*-solaris2.1[0-9]" } {
	# FIXME: Disable multilib testing if the host cannot execute AMD64
	# binaries.  Should be exceedingly rare now (cf. erebus).
	set target_list { "unix{,-m64}" }
    }
    { "mips-sgi-irix6*" } {
	# FIXME: Disable multilib testing if the host cannot execute N64
	# binaries.  We cannot selectively disable only one multilib during
	# the build.
	set target_list { "unix{,-mabi=32,-mabi=64}" }
    }
    { "sparc*-*-solaris2*" } {
	set target_list { "unix{,-m64}" }
    }
    default {
	# Works for alpha*-*-osf*, i?86-*-solaris2.[89] and mips-sgi-irix5*
	# testing. 
        set target_list { "unix" }
    }
}

> I do build gmp/mpfr/mpc in-tree...
> given that I almost always run the whole testsuite - I'm torn between
> thinking it's a good idea..
> ... and that I'm essentially proving little when the versions of gmp/
> mpfr/and mpc are static.

I'd keep gmp/mpfr/mpc at a fixed version, test them once, and be done
with it.  Testing the compiler and runtime libraries on its own takes
enough time already.

> FWIW: I'm trying to update contrib/btest.sh script to handle m32 & m64 plus
> a few other things ...

No need: works already via DEJAGNU described above.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 19:36, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>
>> ..  I don't seem to get the bus errors on a 4CPU g5 or a Core 2  
>> duo .. but
>> .. the 8-core machine is faster .. so ... race conditions are more   
>> likely
>> to manifest there.
>
> But race conditions don't manifest themselves in make SEGVs ;-(  I'm
> regularly running make -k -j128 on a T5220, or -j32 on a Sun Fire  
> X4450

I'm running memchecks on my 8-core machine ..

> Indeed, as well as using a site.exp and pointing the DEJAGNU  
> environment
> variable at it, as I've learned from looking at the regression tester
> sources in contrib/regression.  Here's what I use for that:
>
> global target_list
>
> case "$target_triplet" in {
>    { "i?86-*-solaris2.1[0-9]" } {
> 	# FIXME: Disable multilib testing if the host cannot execute AMD64
> 	# binaries.  Should be exceedingly rare now (cf. erebus).
> 	set target_list { "unix{,-m64}" }
>    }
>    { "mips-sgi-irix6*" } {
> 	# FIXME: Disable multilib testing if the host cannot execute N64
> 	# binaries.  We cannot selectively disable only one multilib during
> 	# the build.
> 	set target_list { "unix{,-mabi=32,-mabi=64}" }
>    }
>    { "sparc*-*-solaris2*" } {
> 	set target_list { "unix{,-m64}" }
>    }
>    default {
> 	# Works for alpha*-*-osf*, i?86-*-solaris2.[89] and mips-sgi-irix5*
> 	# testing.
>        set target_list { "unix" }
>    }
> }

that's really neat indeed - I should have spotted the potential  
looking at the code in contrib
...  although the site.exp is hardwired in btest.sh at present ;
  I guess one might be able to use .dejagnurc - just need to check on  
the order that the files are processed.

>> FWIW: I'm trying to update contrib/btest.sh script to handle m32 &  
>> m64 plus
>> a few other things ...
>
> No need: works already via DEJAGNU described above.

hmm.  I see that the make check would work using that neat method  
above...
... however,

1/ I want a different language list than the default...
2/ also some other different configure options...

but mainly
3/
...  As I read it -  btest.sh strips everything but the testname from  
the PASSES and FAILS files. (the awk lines that print $2 for a match  
of /FAIL: / etc.)
This means (e.g.) that if a test passes at m32 and fails at m64 I  
think it will appear in both PASSES and FAILS..
...  this seems destined to result in confusion...

I would like to  identify regressions separately for m32 and m64 -  
esp. in obj-c/c++ where there are huge differences on NeXT runtime  
systems.

if I've missed the blindingly obvious (the day job is taking up most  
of my mental resources ;-)) please point it out :-)

Iain

Re: legitimate parallel make check?

[IainS]

FWIW;
the bus errors were consistently coming from expect in a strcpy [about  
100 tcl levels down]....
the actual fault was a corrupted dejagnu installation - it's not clear  
how that happened - nothing present in syslogs to indicate disk  
problems.
a clean install of dejagnu appears to have cleared the problem.

On 10 Mar 2010, at 09:47, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>>> I've got some local patches for btest-gcc.sh which allow just that
>>> (e.g. support for Ada testing), which I'll submit once I'm  
>>> reasonably
>>> confident they are general enough.
>> OK - (but I'm happy that my local mods for this are also working).
>
> Without doubt, but why should everyone trying to use this have to
> reinvent the wheel?

indeed, I have every intention of sharing my changes too (at least  
those that are of any merit).

thanks for the help, and apologies that it was noise in the end...
Iain

Re: legitimate parallel make check?

[Tim Prince]

On 3/9/2010 4:28 AM, IainS wrote:
>
>
> It would be nice to allow the apparently independent targets [e.g. 
> gcc-c,fortran,c++ etc.] to be (explicitly) make-checked in parallel.
>
>
On certain targets, it has been necessary to do this explicitly for a 
long time, submitting make check-gcc, make check-fortran, make check-g++ 
separately.  Perhaps a script could be made which would detect when the 
build is complete, then submit the separate make check serial jobs together.


-- 
Tim Prince