legitimate parallel make check?

legitimate parallel make check?

[IainS]

for the sake of keeping information segregated in log files (and  
balancing out test times)
  I tried the following (from within a regression testing script):

'make -k check-gcc-c RUNTESTFLAGS=$runTEST >${STATE}/$svnRevision- 
check-c-log.txt 2>&1' &
(sleep 2) 'make -k check-gcc-c++ RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-c++-log.txt 2>&1' &
(sleep 2) 'make -k check-gcc-fortran RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-fortran-log.txt 2>&1' &
(sleep 2) ' make -k check-gcc-objc RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-objc-log.txt 2>&1' &
(sleep 2) 'make -k check-gcc-obj-c++ RUNTESTFLAGS=$runTEST >${STATE}/ 
$svnRevision-check-obc++-log.txt 2>&1' &
(sleep 2) 'make -k check-target-libstdc++-v3 RUNTESTFLAGS=$runTEST >$ 
{STATE}/$svnRevision-check-target-libstdc++-log.txt 2>&1'&
(sleep 2) 'make -k check-target-libgomp RUNTESTFLAGS=$runTEST >$ 
{STATE}/$svnRevision-check-target-libgomp-log.txt 2>&1' &

firstly; this actually breaks the make process permanently (I suspect  
a race to make site.exp) unless the parallel jobs are launched with a  
time delay.
by permanently I mean that after trying this - "make check-gcc"  fails  
from the command line until config.status is re-run in the gcc dir.

====

secondly;  if I use  "-k -j3 check-gcc-c "
    together with "-k -j2 check-gcc-fortran " (for example)

I get sporadic bus errors :
/bin/sh: line 1: 67618 Bus error               `if [ -f ${srcdir}/../ 
dejagnu/runtest ] ; then echo ${srcdir}/../dejagnu/runtest ; else echo  
runtest; fi` --tool gfortran --target_board=unix\{-m32,-m64\}  
$runtestflags
make[2]: [check-parallel-gfortran] Error 138 (ignored)

large chunks of tests go silently missing at this point...

===

Am I trying something that is unsupported - or is this is a bug?

===

"make -k -j8 check " is not particularly helpful (a) because it tests  
gmp/mpfr/mpc every time (b) any redirected output is hard to scan for  
problems.

Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

> Am I trying something that is unsupported - or is this is a bug?
>
> ===
>
> "make -k -j8 check " is not particularly helpful (a) because it tests
> gmp/mpfr/mpc every time (b) any redirected output is hard to scan for
> problems.

What you're trying is far too complicated: just use -k -j<N> check from
the toplevel and forget about the make check output, which, as you say,
is interleaved and useless, just like the make -j<N> bootstrap output.

Instead, run make mail-report.log afterwards and check that.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 12:10, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>
>> Am I trying something that is unsupported - or is this is a bug?
>>
>> ===
>>
>> "make -k -j8 check " is not particularly helpful (a) because it tests
>> gmp/mpfr/mpc every time (b) any redirected output is hard to scan for
>> problems.
>
> What you're trying is far too complicated: just use -k -j<N> check  
> from
> the toplevel and forget about the make check output, which, as you  
> say,
> is interleaved and useless, just like the make -j<N> bootstrap output.

OK.

> Instead, run make mail-report.log afterwards and check that.

Although, I note that contrib/test_summary only reports what gets into  
the *.sum files -- i.e. tests that complete or timeout.
It doesn't log problems in the actual make process itself - to find  
those one still has to trawl through the interleaved stuff.

It would be nice to allow the apparently independent targets [e.g. gcc- 
c,fortran,c++ etc.] to be (explicitly) make-checked in parallel.

thanks,
Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

>> Instead, run make mail-report.log afterwards and check that.
>
> Although, I note that contrib/test_summary only reports what gets into the
> *.sum files -- i.e. tests that complete or timeout.
> It doesn't log problems in the actual make process itself - to find those
> one still has to trawl through the interleaved stuff.

No, you haven't.  You need to look at the corresponding *.log files
instead; most of this isn't in the regular make check output anyway.

> It would be nice to allow the apparently independent targets [e.g. gcc-
> c,fortran,c++ etc.] to be (explicitly) make-checked in parallel.

This is exactly what happens when you invoke make -j<N> -k check,
provided that the parallelism specified is big enough (and the machine
can handle the load).  Otherwise, you may end up running (say) the c
testsuite in parallel, which isn't bad either.

As I wrote before, I'm going to use this on an (effectively) 64-core
machine and hope to achieve to use all cores in parallel :-)

        Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[Tim Prince]

On 3/9/2010 4:28 AM, IainS wrote:
>
>
> It would be nice to allow the apparently independent targets [e.g. 
> gcc-c,fortran,c++ etc.] to be (explicitly) make-checked in parallel.
>
>
On certain targets, it has been necessary to do this explicitly for a 
long time, submitting make check-gcc, make check-fortran, make check-g++ 
separately.  Perhaps a script could be made which would detect when the 
build is complete, then submit the separate make check serial jobs together.


-- 
Tim Prince

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 12:46, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>
>>> Instead, run make mail-report.log afterwards and check that.

... <snip>

>> suite in parallel, which isn't bad either.
>
> As I wrote before, I'm going to use this on an (effectively) 64-core
> machine and hope to achieve to use all cores in parallel :-)

on my mere 8 cores :-)   I'm still getting sporadic Bus errors on:

"make -k -j8 check RUNTESTFLAGS ... "
[from the command line on a bootstrapped clean trunk @157307]

so there's something else wrong somewhere...
... I guess a target-related issue ({i686,powerpc}-apple-darwin9)

(note that this error does *not* get into the .log files, AFAICS)

Iain

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

> on my mere 8 cores :-)   I'm still getting sporadic Bus errors on:
>
> "make -k -j8 check RUNTESTFLAGS ... "
> [from the command line on a bootstrapped clean trunk @157307]
>
> so there's something else wrong somewhere...

Probably make gets SIGBUS, which obviously won't be in the runtest output.

> ... I guess a target-related issue ({i686,powerpc}-apple-darwin9)

Or simply a hardware issue?

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 19:13, Janis Johnson wrote:

> To run all of the compiler tests in parallel you can do "make -jn -k  
> check-gcc" from the top level and let the existing build machinery  
> take care of running chunks of tests in parallel and putting the  
> results back together.

that's fine (I couldn't see why not from looking at Makefile.in)
  (maybe Rainer is right and I have a memory fault on my 8-core  
box .. :-(  ) -- just backing up prior to a single user check of  
that ...

..  I don't seem to get the bus errors on a 4CPU g5 or a Core 2 duo ..  
but .. the 8-core machine is faster .. so ... race conditions are more  
likely to manifest there.

> You'd still have to handle the library tests you want to run.

I assume that " make -jn -k check-target RUNTESTFLAGS.... " is equally  
reasonable?

> Perhaps what you want is a way to skip the host library testing,  
> although if you're building them with your new compiler then testing  
> them is a good part of testing that compiler.

I do build gmp/mpfr/mpc in-tree...
given that I almost always run the whole testsuite - I'm torn between  
thinking it's a good idea..
... and that I'm essentially proving little when the versions of gmp/ 
mpfr/and mpc are static.

--

FWIW: I'm trying to update contrib/btest.sh script to handle m32 & m64  
plus a few other things ...

Iain

Re: legitimate parallel make check?

[NightStrike]

On Tue, Mar 9, 2010 at 2:27 PM, IainS <developer@sandoe-acoustics.co.uk> wrote:
> I do build gmp/mpfr/mpc in-tree...

How?  Last I tried, it didn't work, as mpc used the system gmp/mpfr,
not the just-built in-tree versions.  Therefore, it's not really an
"in-tree" build, and you can't build on a system that doesn't already
have gmp/mpfr installed.  At least, that's what it was... does that
work now?

Re: legitimate parallel make check?

[Rainer Orth]

IainS <developer@sandoe-acoustics.co.uk> writes:

> ..  I don't seem to get the bus errors on a 4CPU g5 or a Core 2 duo .. but
> .. the 8-core machine is faster .. so ... race conditions are more  likely
> to manifest there.

But race conditions don't manifest themselves in make SEGVs ;-(  I'm
regularly running make -k -j128 on a T5220, or -j32 on a Sun Fire X4450
with 4 4-core CPUs, without any problems.

>> You'd still have to handle the library tests you want to run.
>
> I assume that " make -jn -k check-target RUNTESTFLAGS.... " is equally
> reasonable?

Indeed, as well as using a site.exp and pointing the DEJAGNU environment
variable at it, as I've learned from looking at the regression tester
sources in contrib/regression.  Here's what I use for that:

global target_list

case "$target_triplet" in {
    { "i?86-*-solaris2.1[0-9]" } {
	# FIXME: Disable multilib testing if the host cannot execute AMD64
	# binaries.  Should be exceedingly rare now (cf. erebus).
	set target_list { "unix{,-m64}" }
    }
    { "mips-sgi-irix6*" } {
	# FIXME: Disable multilib testing if the host cannot execute N64
	# binaries.  We cannot selectively disable only one multilib during
	# the build.
	set target_list { "unix{,-mabi=32,-mabi=64}" }
    }
    { "sparc*-*-solaris2*" } {
	set target_list { "unix{,-m64}" }
    }
    default {
	# Works for alpha*-*-osf*, i?86-*-solaris2.[89] and mips-sgi-irix5*
	# testing. 
        set target_list { "unix" }
    }
}

> I do build gmp/mpfr/mpc in-tree...
> given that I almost always run the whole testsuite - I'm torn between
> thinking it's a good idea..
> ... and that I'm essentially proving little when the versions of gmp/
> mpfr/and mpc are static.

I'd keep gmp/mpfr/mpc at a fixed version, test them once, and be done
with it.  Testing the compiler and runtime libraries on its own takes
enough time already.

> FWIW: I'm trying to update contrib/btest.sh script to handle m32 & m64 plus
> a few other things ...

No need: works already via DEJAGNU described above.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 19:31, NightStrike wrote:

> On Tue, Mar 9, 2010 at 2:27 PM, IainS <developer@sandoe-acoustics.co.uk 
> > wrote:
>> I do build gmp/mpfr/mpc in-tree...
>
> How?  Last I tried, it didn't work, as mpc used the system gmp/mpfr,
> not the just-built in-tree versions.  Therefore, it's not really an
> "in-tree" build, and you can't build on a system that doesn't already
> have gmp/mpfr installed.  At least, that's what it was... does that
> work now?

it's been  fixed for a while (resolution of PR42424)
I keep one copy of gmp mpfr and mpc in a directory at the root of my  
source disk and ln -s to that from within the different gcc source  
trees.

Iain

Re: legitimate parallel make check?

[IainS]

On 9 Mar 2010, at 19:36, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>
>> ..  I don't seem to get the bus errors on a 4CPU g5 or a Core 2  
>> duo .. but
>> .. the 8-core machine is faster .. so ... race conditions are more   
>> likely
>> to manifest there.
>
> But race conditions don't manifest themselves in make SEGVs ;-(  I'm
> regularly running make -k -j128 on a T5220, or -j32 on a Sun Fire  
> X4450

I'm running memchecks on my 8-core machine ..

> Indeed, as well as using a site.exp and pointing the DEJAGNU  
> environment
> variable at it, as I've learned from looking at the regression tester
> sources in contrib/regression.  Here's what I use for that:
>
> global target_list
>
> case "$target_triplet" in {
>    { "i?86-*-solaris2.1[0-9]" } {
> 	# FIXME: Disable multilib testing if the host cannot execute AMD64
> 	# binaries.  Should be exceedingly rare now (cf. erebus).
> 	set target_list { "unix{,-m64}" }
>    }
>    { "mips-sgi-irix6*" } {
> 	# FIXME: Disable multilib testing if the host cannot execute N64
> 	# binaries.  We cannot selectively disable only one multilib during
> 	# the build.
> 	set target_list { "unix{,-mabi=32,-mabi=64}" }
>    }
>    { "sparc*-*-solaris2*" } {
> 	set target_list { "unix{,-m64}" }
>    }
>    default {
> 	# Works for alpha*-*-osf*, i?86-*-solaris2.[89] and mips-sgi-irix5*
> 	# testing.
>        set target_list { "unix" }
>    }
> }

that's really neat indeed - I should have spotted the potential  
looking at the code in contrib
...  although the site.exp is hardwired in btest.sh at present ;
  I guess one might be able to use .dejagnurc - just need to check on  
the order that the files are processed.

>> FWIW: I'm trying to update contrib/btest.sh script to handle m32 &  
>> m64 plus
>> a few other things ...
>
> No need: works already via DEJAGNU described above.

hmm.  I see that the make check would work using that neat method  
above...
... however,

1/ I want a different language list than the default...
2/ also some other different configure options...

but mainly
3/
...  As I read it -  btest.sh strips everything but the testname from  
the PASSES and FAILS files. (the awk lines that print $2 for a match  
of /FAIL: / etc.)
This means (e.g.) that if a test passes at m32 and fails at m64 I  
think it will appear in both PASSES and FAILS..
...  this seems destined to result in confusion...

I would like to  identify regressions separately for m32 and m64 -  
esp. in obj-c/c++ where there are huge differences on NeXT runtime  
systems.

if I've missed the blindingly obvious (the day job is taking up most  
of my mental resources ;-)) please point it out :-)

Iain

Re: legitimate parallel make check?

[IainS]

FWIW;
the bus errors were consistently coming from expect in a strcpy [about  
100 tcl levels down]....
the actual fault was a corrupted dejagnu installation - it's not clear  
how that happened - nothing present in syslogs to indicate disk  
problems.
a clean install of dejagnu appears to have cleared the problem.

On 10 Mar 2010, at 09:47, Rainer Orth wrote:

> IainS <developer@sandoe-acoustics.co.uk> writes:
>>> I've got some local patches for btest-gcc.sh which allow just that
>>> (e.g. support for Ada testing), which I'll submit once I'm  
>>> reasonably
>>> confident they are general enough.
>> OK - (but I'm happy that my local mods for this are also working).
>
> Without doubt, but why should everyone trying to use this have to
> reinvent the wheel?

indeed, I have every intention of sharing my changes too (at least  
those that are of any merit).

thanks for the help, and apologies that it was noise in the end...
Iain