RE: [PATCH] RE: gcc parallel make check

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

[-- Attachment #1: Type: text/plain, Size: 330 bytes --]

>> > Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges.
>> > Thus \[0-9A-Zhjqvx-z\]*

OK, works fine with the attached patch, and looks cleaner in Make-lang.in.

Now, with the proper email address for gcc-patches... I wonder how many time I'll be punished for typos.

unmodified CL.

Joost




[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch-speedup-checkfortran-v03.diff --]
[-- Type: text/x-patch; name="patch-speedup-checkfortran-v03.diff", Size: 4755 bytes --]

Index: contrib/generate_tcl_patterns.sh
===================================================================
--- contrib/generate_tcl_patterns.sh	(revision 0)
+++ contrib/generate_tcl_patterns.sh	(revision 0)
@@ -0,0 +1,108 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+#   cd gcc/gcc/testsuite/gfortran.dg
+#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele <Joost.VandeVondele@mat.ethz.ch>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+  # list of allowed starting chars for a file name in a dir to split
+  achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+  ranget="11111111112222222222222222222222222233333333333333333333333333"
+}
+{
+  nfiles++ ; files[nfiles]=$1
+}
+END{
+  for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+  for(i=1; i<=nfiles; i++) {
+     if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+  };
+  asort(count,ordered)
+  countsingle=0
+  groups=0
+  label=""
+  for(i=length(achars);i>=1;i--) {
+    countsingle=countsingle+ordered[i] 
+    for(j=1;j<=length(achars);j++) {
+       if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+    }
+    count[found]=-1
+    label=label found
+    if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+    if(countsingle+val>maxcount) {
+      subset[label]=countsingle
+      print "Adding label: ", label, "matching files:" countsingle
+      groups++
+      countsingle=0
+      label=""
+    }
+  }
+  print "patterns:"
+  asort(subset,ordered)
+  for(i=groups;i>=1;i--) {
+    for(j in subset){
+      if(subset[j]==ordered[i]) found=j
+    }
+    subset[found]=-1
+    if (length(found)==1) {
+       printf("%s%s* \\\n",prefix,found)
+    } else {
+       sortandcompress()
+       printf("%s\\[%s\\]* \\\n",prefix,found)
+    }
+  }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+  n=length(found)
+  for(i=1; i<=n; i++) tmp[i]=substr(found,i,1) 
+  asort(tmp)
+  for(i=1;i<=n;i++){
+    ipos=index(achars,tmp[i])
+    for(j=i;j<=n;j++){
+      jpos=index(achars,tmp[j])
+      if (jpos-ipos==j-i && substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+    }
+    if (bestj-i>3) {
+      tmp[i+1]="-" 
+      for(j=i+2;j<bestj;j++) tmp[j]=" "
+    }
+    i=bestj 
+  }
+  found=""
+  for(i=1; i<=n; i++) found=found tmp[i]
+  gsub(/ /,"",found)
+}
+'
Index: gcc/fortran/Make-lang.in
===================================================================
--- gcc/fortran/Make-lang.in	(revision 214949)
+++ gcc/fortran/Make-lang.in	(working copy)
@@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = execute.exp \
+			dg.exp=gfortran.dg/p* \
+			dg.exp=gfortran.dg/c* \
+			dg.exp=gfortran.dg/a* \
+			dg.exp=gfortran.dg/i* \
+			dg.exp=gfortran.dg/\[glow\]* \
+			dg.exp=gfortran.dg/\[mu\]* \
+			dg.exp=gfortran.dg/d* \
+			dg.exp=gfortran.dg/s* \
+			dg.exp=gfortran.dg/b* \
+			dg.exp=gfortran.dg/t* \
+			dg.exp=gfortran.dg/f* \
+			dg.exp=gfortran.dg/e* \
+			dg.exp=gfortran.dg/r* \
+			dg.exp=gfortran.dg/n* \
+			dg.exp=gfortran.dg/\[0-9A-Zhjkqvxyz\]*
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

[-- Attachment #1: Type: text/plain, Size: 1797 bytes --]

Attached is an extended version of the patch, it brings a 100% improvement in make -j32 -k check-gcc (down from 20min to <10min) by modification of check_gcc_parallelize.

It includes one non-trivial part, namely a split of the target exps. They are now all split using a common choice (based on i386), which I believe is reasonable as it is the target with most tests, and the patterns will be somewhat similar for other targets (e.g. split of p(rxxx)). The implementation of this in the makefile uses an odd looking technique to substitute spaces with commas in a variable, if this can be done more elegantly, I'm happy to make the change.

Bootstrap and testing revealed one issue, i386.exp hard-codes a loop for the testcase 'vect-args.c' in order to test 10 different combinations of options. With the current split (i.e. target x4) this test will thus be executed 4 times. There are two easy options

1) keep the current setup, overhead is small
2) keep the .exp file simple and just replicate this test 10x 

I've selected 1), but I can update a patch with 2). Ideally dg-options in the testcase file itself could be repeated, but I haven't found an example of this. 

The script now includes sorting and compression of the ranges, and an additional sanity check on the input, i.e. that file names start with [0-9A-Za-z]. Some (few) files seem to start with _ or # (in ./gcc.dg/cpp/).

I'll follow up with a separate patch to improve check_g++_parallelize.

Full 'make -j k32 check' is now dominated by libstdc++ testing, which contains single goals that run ~1100s (e.g. regex related tests). These uses a slightly different syntax (see gcc/libstdc++-v3/testsuite/Makefile.am) and I'm not yet sure how to deal with the .am files.

current patch OK for trunk ?

Joost


[-- Attachment #2: patch-speedup-checkfortran-v05.CL --]
[-- Type: application/octet-stream, Size: 606 bytes --]

contrib/ChangeLog

2014-09-08  Joost VandeVondele  <vondele@gcc.gnu.org>

	* generate_tcl_patterns.sh: New file.

gcc/fortran/ChangeLog

2014-09-08  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Make-lang.in (check_gfortran_parallelize): Improved parallelism.

gcc/Changelog

2014-09-08  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Makefile.in (check_gcc_parallelize): Improved parallelism.
	(check_p_numbers): Increase maximum value.
	(dg_target_exps): Mention targets as separate words only.
	(null,space,comma,dg_target_exps_p1,dg_target_exps_p2,
	dg_target_exps_p3,dg_target_exps_p4): New variables.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: patch-speedup-checkfortran-v05.diff --]
[-- Type: text/x-patch; name="patch-speedup-checkfortran-v05.diff", Size: 9479 bytes --]

Index: contrib/generate_tcl_patterns.sh
===================================================================
--- contrib/generate_tcl_patterns.sh	(revision 0)
+++ contrib/generate_tcl_patterns.sh	(revision 0)
@@ -0,0 +1,114 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input, starting with [0-9A-Za-z],
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+#   cd gcc/gcc/testsuite/gfortran.dg
+#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele <Joost.VandeVondele@mat.ethz.ch>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+  # list of allowed starting chars for a file name in a dir to split
+  achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+  ranget="11111111112222222222222222222222222233333333333333333333333333"
+}
+{
+  if (index(achars,substr($1,1,1))==0){
+     print "file : " $1 " does not start with an allowed character."
+     _assert_exit = 1
+     exit 1
+  }
+  nfiles++ ; files[nfiles]=$1
+}
+END{
+  if (_assert_exit) exit 1
+  for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+  for(i=1; i<=nfiles; i++) {
+     if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+  };
+  asort(count,ordered)
+  countsingle=0
+  groups=0
+  label=""
+  for(i=length(achars);i>=1;i--) {
+    countsingle=countsingle+ordered[i] 
+    for(j=1;j<=length(achars);j++) {
+       if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+    }
+    count[found]=-1
+    label=label found
+    if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+    if(countsingle+val>maxcount) {
+      subset[label]=countsingle
+      print "Adding label: ", label, "matching files:" countsingle
+      groups++
+      countsingle=0
+      label=""
+    }
+  }
+  print "patterns:"
+  asort(subset,ordered)
+  for(i=groups;i>=1;i--) {
+    for(j in subset){
+      if(subset[j]==ordered[i]) found=j
+    }
+    subset[found]=-1
+    if (length(found)==1) {
+       printf("%s%s* \\\n",prefix,found)
+    } else {
+       sortandcompress()
+       printf("%s\\[%s\\]* \\\n",prefix,found)
+    }
+  }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+  n=length(found)
+  for(i=1; i<=n; i++) tmp[i]=substr(found,i,1) 
+  asort(tmp)
+  for(i=1;i<=n;i++){
+    ipos=index(achars,tmp[i])
+    for(j=i;j<=n;j++){
+      jpos=index(achars,tmp[j])
+      if (jpos-ipos==j-i && substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+    }
+    if (bestj-i>3) {
+      tmp[i+1]="-" 
+      for(j=i+2;j<bestj;j++) tmp[j]=" "
+    }
+    i=bestj 
+  }
+  found=""
+  for(i=1; i<=n; i++) found=found tmp[i]
+  gsub(/ /,"",found)
+}
+'

Property changes on: contrib/generate_tcl_patterns.sh
___________________________________________________________________
Added: svn:executable
   + *

Index: gcc/fortran/Make-lang.in
===================================================================
--- gcc/fortran/Make-lang.in	(revision 215008)
+++ gcc/fortran/Make-lang.in	(working copy)
@@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = execute.exp \
+			dg.exp=gfortran.dg/p* \
+			dg.exp=gfortran.dg/c* \
+			dg.exp=gfortran.dg/a* \
+			dg.exp=gfortran.dg/i* \
+			dg.exp=gfortran.dg/\[glow\]* \
+			dg.exp=gfortran.dg/\[mu\]* \
+			dg.exp=gfortran.dg/d* \
+			dg.exp=gfortran.dg/s* \
+			dg.exp=gfortran.dg/b* \
+			dg.exp=gfortran.dg/t* \
+			dg.exp=gfortran.dg/f* \
+			dg.exp=gfortran.dg/e* \
+			dg.exp=gfortran.dg/r* \
+			dg.exp=gfortran.dg/n* \
+			dg.exp=gfortran.dg/\[0-9A-Zhjkqvxyz\]*
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in	(revision 215008)
+++ gcc/Makefile.in	(working copy)
@@ -513,11 +513,17 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
+dg_target_exps:=aarch64 alpha arm avr bfin cris epiphany frv i386 ia64 
+dg_target_exps:=$(dg_target_exps) m68k microblaze mips powerpc rx s390
+dg_target_exps:=$(dg_target_exps) sh sparc spu tic6x xstormy16
+# also parallelize the target exps, using one common choice (i386) of split
+null:=
+space:=$(null) $(null)
+comma:=$(null),$(null)
+dg_target_exps_p1:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/a*))
+dg_target_exps_p2:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/\[fs\]*))
+dg_target_exps_p3:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/\[0-9A-Zbcdeg-oqrt-z\]*))
+dg_target_exps_p4:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/p*))
 # This lists a couple of test files that take most time during check-gcc.
 # When doing parallelized check-gcc, these can run in parallel with the
 # remaining tests.  Each word in this variable stands for work for one
@@ -526,21 +532,42 @@ dg_target_exps:=$(dg_target_exps),tic6x.
 # should be run in the same runtest invocation (usually if they aren't
 # very long running, but still should be split of from the check-parallel-$lang
 # remaining tests runtest invocation), they should be concatenated with commas.
+# At most $(check_p_numbers) goals should be created.
 # Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
 # by tcl) and as the *.exp arguments are mached both as is and with
 # */ prefixed to it in runtest_file_p, it is usually desirable to include
 # a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+check_gcc_parallelize=$(dg_target_exps_p1) \
+			$(dg_target_exps_p2) \
+			$(dg_target_exps_p3) \
+			$(dg_target_exps_p4) \
+			execute.exp=execute/2* \
+			execute.exp=execute/p* \
+			execute.exp=execute/9* \
+			execute.exp=execute/\[013-8A-Za-oq-z\]* \
+			compile.exp=compile/p* \
+			compile.exp=compile/2* \
+			compile.exp=compile/\[Babcdfilmsvx\]* \
+			compile.exp=compile/9* \
+			compile.exp=compile/\[013-8AC-Zeghjknoqrtuwyz\]* \
+			dg-torture.exp=torture/p* \
+			dg-torture.exp=torture/\[0-9A-Za-oq-z\]* \
+			vect.exp \
+			atomic.exp \
+			builtins.exp \
+			ubsan.exp \
+			struct-layout-1.exp \
+        		stackalign.exp \
+        		dg-exp.exp \
+        		ieee.exp \
+			asan.exp,tsan.exp \
+			lto.exp \
+			cilk-plus.exp \
+			noncompile.exp,dfp.exp,graphite.exp \
+			guality.exp,unsorted.exp \
+			debug.exp,tree-ssa.exp \
+			pch.exp,compat.exp,cpp.exp \
+			tls.exp,abi-x86_64.exp
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3639,7 +3666,8 @@ check_p_vars=$(check_$(check_p_tool)_par
 check_p_subno=$(word 2,$(subst _, ,$*))
 check_p_comma=,
 check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \
+		21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
 check_p_subdir=$(subst _,,$*)
 check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
 

Re: [PATCH] RE: gcc parallel make check

[Yury Gribov]

On 09/09/2014 10:51 AM, VandeVondele Joost wrote:
 > Attached is an extended version of the patch,
 > it brings a 100% improvement in make -j32 -k check-gcc

First of all, many thanks for working on this.

+#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 
"dg.exp=gfortran.dg/"

How does this work with subdirectories? Can we replace ls with find?

-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \
+		21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

$(shell seq 1 40) ?

+  if (_assert_exit) exit 1

Haven't you already exited above?

 > A second part of the patch is a new file 
'contrib/generate_tcl_patterns.sh'
 > which generates the needed regexp

Can we provide a Makefile target to automatically update Makefile.in?

-Y

Re: [PATCH] RE: gcc parallel make check

[Jakub Jelinek]

On Tue, Sep 09, 2014 at 02:02:18PM +0400, Yury Gribov wrote:
> On 09/09/2014 10:51 AM, VandeVondele Joost wrote:
> > Attached is an extended version of the patch,
> > it brings a 100% improvement in make -j32 -k check-gcc
> 
> First of all, many thanks for working on this.
> 
> +#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300
> "dg.exp=gfortran.dg/"
> 
> How does this work with subdirectories? Can we replace ls with find?

Generally, if the argument to *.exp doesn't contain a particular
subdirectory, then the wildcard is taken against basenames of the tests.

> -check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> +check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \
> +		21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
> 
> $(shell seq 1 40) ?

Would that be sufficiently portable to weirdo hosts (M$Win, Darwin, ...)?
We require GNU make, so if it can be written in GNU make text functions,
fine, otherwise it is better to keep as is.
> 
> +  if (_assert_exit) exit 1
> 
> Haven't you already exited above?
> 
> > A second part of the patch is a new file
> 'contrib/generate_tcl_patterns.sh'
> > which generates the needed regexp
> 
> Can we provide a Makefile target to automatically update Makefile.in?

No.  As I wrote earlier, splitting on filenames and test counts only is only
very rough split, all the splits really need to be backed out by real timing
data from popular targets.  Also, I'm afraid of some tests being left out
unintentionally (e.g. the wildcards created at some point, then a new test
is added with a weird starting character that hasn't been used before and
suddenly it will not be tested with make -j?).

	Jakub

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

> No.  As I wrote earlier, splitting on filenames and test counts only is only
> very rough split, all the splits really need to be backed out by real timing
> data from popular targets.  

I'm actually doing quite some testing trying to get a reasonable balance, checking 'completed in' in all *.log.sep files. However, it is important that the procedure is semi-automatic, otherwise few people will be interested in doing so. Furthermore, for parallel performance, it is not so important that times are distributed evenly (it is anyway unlikely the number of goals is exactly divided by N of -jN), but rather that the goals are ordered (executed) from slow to fast (similar to omp schedule guided). Most of the real bottlenecks are single letter patterns (e.g. p* since prxxxx is such a common filename), and this is ultimately limiting.

In the project (CP2K) I'm working on, we also parallelize testing over directories, but we keep a list of approximate runtimes per directory, and keep that (global) list sorted. Testing follows that list. As a result, we have near perfect parallel speedup, despite (or because) timings per directory ranging from a few 100s to 1s. 

> Also, I'm afraid of some tests being left out
> unintentionally (e.g. the wildcards created at some point, then a new test
> is added with a weird starting character that hasn't been used before and
> suddenly it will not be tested with make -j?).

I agree this is an issue, partially addressed by not having to write patterns by hand anymore (i.e. a script does this), and by having the script check its input. There are something like 10 testnames that do not fall in [0-9A-Za-z], as mentioned in a previous email.

Re: [PATCH] RE: gcc parallel make check

[Jakub Jelinek]

On Tue, Sep 09, 2014 at 10:57:09AM +0000, VandeVondele  Joost wrote:
> > No.  As I wrote earlier, splitting on filenames and test counts only is only
> > very rough split, all the splits really need to be backed out by real timing
> > data from popular targets.  
> 
> Furthermore, for parallel performance, it is not
> so important that times are distributed evenly (it is anyway unlikely the
> number of goals is exactly divided by N of -jN), but rather that the goals
> are ordered (executed) from slow to fast (similar to omp schedule guided). 
> Most of the real bottlenecks are single letter patterns (e.g.  p* since
> prxxxx is such a common filename), and this is ultimately limiting.

I disagree.  If e.g. in gcc.dg/ more than a third of testcases are pr*.c,
then running dg.exp=p* in one job and dg.exp=a* in another one etc. is
simply a bad idea, the pr*.c should be split more and some other letters
just be done together.

Even that can be done semi-automatically.

If you get whitespace right, one can provide multiple different wildcards to
a single *.exp file, e.g.
make check-gcc RUNTESTFLAGS="dg.exp='p[0-9A-Za-qs-z]* pr[9A-Za-z]*'" should
cover all tests starting with p other than pr[0-8]*.c (where you could split
say pr[0-2]* into another job, pr[3-5]* into another and pr[6-8]* into
another.

The fact that some check-gcc or check-gfortran test job is early in the list
doesn't mean it will be started early, you need to consider also all other
potentially long jobs like check-g++, check-target-libgomp,
check-target-libstdc++-v3 etc.

	Jakub

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

> If you get whitespace right, one can provide multiple different wildcards to
> a single *.exp file, e.g.
> make check-gcc RUNTESTFLAGS="dg.exp='p[0-9A-Za-qs-z]* pr[9A-Za-z]*'" should
> cover all tests starting with p other than pr[0-8]*.c (where you could split
> say pr[0-2]* into another job, pr[3-5]* into another and pr[6-8]* into
> another.

I think this confirms that it becomes very delicate to try and write these more complex patterns. The above would miss p_test.c, p-1.c, etc ? 

For other classes of files the difference is even further down the filename (e.g. using dates as in 20020508-3.c going from 2000 to 2014, or avx*), making the automatic generation of the patterns more complicated.

I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup.

Re: [PATCH] RE: gcc parallel make check

[Yury Gribov]

On 09/09/2014 06:14 PM, VandeVondele Joost wrote:
> I certainly don't want to claim that the patch I have now is perfect,
>it is rather an incremental improvement on the current setup.

I'd second this. Writing patterns manually seems rather inefficient and 
error-prone
(not undoable of course but unnecessarily complicated). And with current 
(crippled)
version Joost already got 100% test time improvement.

-Y

Re: [PATCH] RE: gcc parallel make check

[Jakub Jelinek]

On Tue, Sep 09, 2014 at 06:27:10PM +0400, Yury Gribov wrote:
> On 09/09/2014 06:14 PM, VandeVondele Joost wrote:
> >I certainly don't want to claim that the patch I have now is perfect,
> >it is rather an incremental improvement on the current setup.
> 
> I'd second this. Writing patterns manually seems rather inefficient and
> error-prone
> (not undoable of course but unnecessarily complicated). And with current
> (crippled)
> version Joost already got 100% test time improvement.

But if there are jobs that just take 1s to complete, then clearly it doesn't
make sense to split them off as separate job.  I think we don't need 100%
even split, but at least roughly is highly desirable.

	Jakub

Re: [PATCH] RE: gcc parallel make check

[Yury Gribov]

On 09/09/2014 06:33 PM, Jakub Jelinek wrote:
> On Tue, Sep 09, 2014 at 06:27:10PM +0400, Yury Gribov wrote:
>> On 09/09/2014 06:14 PM, VandeVondele Joost wrote:
>>> I certainly don't want to claim that the patch I have now is perfect,
>>> it is rather an incremental improvement on the current setup.
>>
>> I'd second this. Writing patterns manually seems rather inefficient and
>> error-prone
>> (not undoable of course but unnecessarily complicated). And with current
>> (crippled)
>> version Joost already got 100% test time improvement.
>
> But if there are jobs that just take 1s to complete, then clearly it doesn't
> make sense to split them off as separate job.  I think we don't need 100%
> even split, but at least roughly is highly desirable.

You mean enhancing the script to split across arbitrarily long prefixes?
That would be great.

-Y

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

> You mean enhancing the script to split across arbitrarily long prefixes?
> That would be great.

I've now a script that does something like that:

~/test$ find /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gfortran.dg/ -maxdepth 1 -type f -printf "%f\n" | ./generate_patterns.py 500 foo
All  3947  files matched the pattern ^[0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+ without exception
Final  12  patterns and match count:
(^[j-z_#+-][p-z_#+-][0-9A-Za-i][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][0-9A-Za-o][0-9A-Za-m]([.][0-9A-Za-z_#+-]+)+)  matching  469  files
(^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n][0-9A-Za-o][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^([.][0-9A-Za-z_#+-]+)+)  matching  433  files
(^[j-z_#+-][0-9A-Za-o][n-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n][o-z_#+-]([.][0-9A-Za-z_#+-]+)+)  matching  400  files
(^[j-z_#+-][p-z_#+-][j-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i]([.][0-9A-Za-z_#+-]+)+)  matching  371  files
(^[0-9A-Za-i][o-z_#+-][s-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n]([.][0-9A-Za-z_#+-]+)+)  matching  323  files
(^[0-9A-Za-i][o-z_#+-][0-9A-Za-r][o-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][p-z_#+-]([.][0-9A-Za-z_#+-]+)+)  matching  314  files
(^[0-9A-Za-i][o-z_#+-][0-9A-Za-r][0-9A-Za-n][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][0-9A-Za-o]([.][0-9A-Za-z_#+-]+)+)  matching  314  files
(^[j-z_#+-][0-9A-Za-o][0-9A-Za-m][0-9A-Za-i][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-]([.][0-9A-Za-z_#+-]+)+)  matching  272  files
(^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n][p-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][o-z_#+-]([.][0-9A-Za-z_#+-]+)+)  matching  270  files
(^[0-9A-Za-i][0-9A-Za-n][o-z_#+-][0-9A-Za-l][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n]([.][0-9A-Za-z_#+-]+)+)  matching  265  files
(^[0-9A-Za-i][0-9A-Za-n][o-z_#+-][m-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][o-z_#+-][0-9A-Za-r]([.][0-9A-Za-z_#+-]+)+)  matching  260  files
^[j-z_#+-][0-9A-Za-o][0-9A-Za-m][j-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+  matching  256  files

It is a set of patterns that will match any file of the form '^[0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+', but such that it splits a list of input files roughly in equal chunks (e.g. between 500 and 500/2 in this example), even if files have long overlapping prefixes. However, I'm unsure if/how this can be integrated, i.e. what precisely is allowed for testsuite filenames, and if this regexp format can be employed in gcc makefiles / tcl / expect harness, suggestions/help appreciated.

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]

Now with gzipped figure.. why do these bounce ?

> But if there are jobs that just take 1s to complete, then clearly it doesn't
> make sense to split them off as separate job.  I think we don't need 100%
> even split, but at least roughly is highly desirable.

Let me add some data, attached is a graph (logscale y) showing the runtime of tests before and after my changes (including a new patch for c++). There is virtually no change for tests running shorter than 50s, only slowly running tests have been split.

Now, there are only very few slow tests remaining:

gcc_trunk/obj.new> find . -name "*.log" | xargs grep " completed in " | sort -n -k 5 | tail -n 10
./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.dg/torture/dg-torture.exp completed in 521 seconds
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 530 seconds
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 553 seconds
./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libgomp/testsuite/libgomp.fortran/fortran.exp completed in 561 seconds
./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.c-torture/compile/compile.exp completed in 625 seconds
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 683 seconds
./gcc/testsuite/g++/g++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/g++.dg/dg.exp completed in 702 seconds
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 726 seconds
./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.c-torture/execute/execute.exp completed in 752 seconds
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 904 seconds

They, of course, limit the ultimate speedup.

[-- Attachment #2: timings.png.gz --]
[-- Type: application/binary, Size: 8944 bytes --]

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

> +#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300
> "dg.exp=gfortran.dg/"
> 
> How does this work with subdirectories? Can we replace ls with find?

The input to the script is general, you can use this to your advantage. For example, I've been using:

 ls -1 g++.*/* | cut -c5- | ../../../contrib/generate_tcl_patterns.sh 700 old-deja.exp=g++.old-deja/g++.

to split at a deeper level or

find . -name "[0-9A-Za-z]*" -type f -printf "%f\n" | ../../../../contrib/generate_tcl_patterns.sh 300 dg-torture.exp=torture/

to collect statistics also from subdirs.

> +  if (_assert_exit) exit 1
>
> Haven't you already exited above?

yes, but the END{} block in awk is nevertheless executed, unless protected as above.

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Attached is a further revision of the patch, now dealing with check-c++. Roughly 50% speedup here at '-j32' (18m vs 12m). For my setup (--enable-languages=c,c++,fortran) I have now improved all targets called in 'make -j32 -k check'. The latter is now 30% faster (15m vs 20m). Note that there are +- 1m fluctuations in these numbers, easily.

I currently have no plans to work on other check targets before this patch is committed.

OK for trunk ?

Joost








[-- Attachment #2: patch-speedup-check-v06-CL.txt --]
[-- Type: text/plain, Size: 1039 bytes --]

contrib/ChangeLog

2014-09-09  Joost VandeVondele  <vondele@gcc.gnu.org>

	* generate_tcl_patterns.sh: New file.

gcc/fortran/ChangeLog

2014-09-09  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Make-lang.in (check_gfortran_parallelize): Improved parallelism.

gcc/Changelog

2014-09-09  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Makefile.in (check_gcc_parallelize): Improved parallelism.
	(check_p_numbers): Increase maximum value.
	(dg_target_exps): Mention targets as separate words only.
	(null,space,comma,dg_target_exps_p1,dg_target_exps_p2,
	dg_target_exps_p3,dg_target_exps_p4): New variables.

gcc/cp/ChangeLog

2014-09-09  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Make-lang.in (check_g++_parallelize): Improved parallelism.

libstdc++-v3/ChangeLog

2014-09-09  Joost VandeVondele  <vondele@gcc.gnu.org>

        * testsuite/Makefile.am (check_DEJAGNU_normal_targets): Add
        check-DEJAGNUnormal[11-15].
        (check-DEJAGNU): Split into 15 jobs for parallel testing.
        * testsuite/Makefile.in: Regenerated.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: patch-speedup-check-v06.diff --]
[-- Type: text/x-patch; name="patch-speedup-check-v06.diff", Size: 16052 bytes --]

Index: libstdc++-v3/testsuite/Makefile.am
===================================================================
--- libstdc++-v3/testsuite/Makefile.am	(revision 215017)
+++ libstdc++-v3/testsuite/Makefile.am	(working copy)
@@ -101,7 +101,7 @@ new-abi-baseline:
 	@test ! -f $*/site.exp || mv $*/site.exp $*/site.bak
 	@mv $*/site.exp.tmp $*/site.exp
 
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)
 $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
@@ -111,7 +111,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
+	  for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \
 	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
 	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
 	  done; \
@@ -138,25 +138,35 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	    fi; \
 	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
 	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
+	    dirs="`cd $$srcdir; echo e*/*`";; \
 	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 28_*/a*`";; \
 	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \
 	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
+	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
 	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
+	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
 	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[m-tw-z]*`";; \
 	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
+	    dirs="`cd $$srcdir; echo 26_*/*`";; \
 	  normal8) \
 	    dirs="`cd $$srcdir; echo 27_*/*`";; \
 	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
+	    dirs="`cd $$srcdir; echo 22_*/*`";; \
 	  normal10) \
 	    dirs="`cd $$srcdir; echo t*/*`";; \
+	  normal11) \
+	    dirs="`cd $$srcdir; echo 28_*/b*`";; \
+	  normal12) \
+	    dirs="`cd $$srcdir; echo 28_*/[c-z]*`";; \
+	  normal13) \
+	    dirs="`cd $$srcdir; echo de* p*/*`";; \
+	  normal14) \
+	    dirs="`cd $$srcdir; echo [ab]* 23_*/v*`";; \
+	  normal15) \
+	    dirs="`cd $$srcdir; echo 23_*/[a-k]*`";; \
 	esac; \
 	if [ -n "$*" ]; then cd "$*"; fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
Index: libstdc++-v3/testsuite/Makefile.in
===================================================================
--- libstdc++-v3/testsuite/Makefile.in	(revision 215017)
+++ libstdc++-v3/testsuite/Makefile.in	(working copy)
@@ -301,7 +301,7 @@ lists_of_files = \
 
 extract_symvers = $(glibcxx_builddir)/scripts/extract_symvers
 baseline_subdir := $(shell $(CXX) $(baseline_subdir_switch))
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)
 
 # Runs the testsuite, but in compile only mode.
 # Can be used to test sources with non-GNU FE's at various warning
@@ -562,7 +562,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
+	  for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \
 	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
 	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
 	  done; \
@@ -589,25 +589,35 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	    fi; \
 	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
 	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
+	    dirs="`cd $$srcdir; echo e*/*`";; \
 	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 28_*/a*`";; \
 	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \
 	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
+	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
 	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
+	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
 	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[m-tw-z]*`";; \
 	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
+	    dirs="`cd $$srcdir; echo 26_*/*`";; \
 	  normal8) \
 	    dirs="`cd $$srcdir; echo 27_*/*`";; \
 	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
+	    dirs="`cd $$srcdir; echo 22_*/*`";; \
 	  normal10) \
 	    dirs="`cd $$srcdir; echo t*/*`";; \
+	  normal11) \
+	    dirs="`cd $$srcdir; echo 28_*/b*`";; \
+	  normal12) \
+	    dirs="`cd $$srcdir; echo 28_*/[c-z]*`";; \
+	  normal13) \
+	    dirs="`cd $$srcdir; echo de* p*/*`";; \
+	  normal14) \
+	    dirs="`cd $$srcdir; echo [ab]* 23_*/v*`";; \
+	  normal15) \
+	    dirs="`cd $$srcdir; echo 23_*/[a-k]*`";; \
 	esac; \
 	if [ -n "$*" ]; then cd "$*"; fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
Index: contrib/generate_tcl_patterns.sh
===================================================================
--- contrib/generate_tcl_patterns.sh	(revision 0)
+++ contrib/generate_tcl_patterns.sh	(revision 0)
@@ -0,0 +1,114 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input, starting with [0-9A-Za-z],
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+#   cd gcc/gcc/testsuite/gfortran.dg
+#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele <Joost.VandeVondele@mat.ethz.ch>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+  # list of allowed starting chars for a file name in a dir to split
+  achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+  ranget="11111111112222222222222222222222222233333333333333333333333333"
+}
+{
+  if (index(achars,substr($1,1,1))==0){
+     print "file : " $1 " does not start with an allowed character."
+     _assert_exit = 1
+     exit 1
+  }
+  nfiles++ ; files[nfiles]=$1
+}
+END{
+  if (_assert_exit) exit 1
+  for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+  for(i=1; i<=nfiles; i++) {
+     if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+  };
+  asort(count,ordered)
+  countsingle=0
+  groups=0
+  label=""
+  for(i=length(achars);i>=1;i--) {
+    countsingle=countsingle+ordered[i] 
+    for(j=1;j<=length(achars);j++) {
+       if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+    }
+    count[found]=-1
+    label=label found
+    if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+    if(countsingle+val>maxcount) {
+      subset[label]=countsingle
+      print "Adding label: ", label, "matching files:" countsingle
+      groups++
+      countsingle=0
+      label=""
+    }
+  }
+  print "patterns:"
+  asort(subset,ordered)
+  for(i=groups;i>=1;i--) {
+    for(j in subset){
+      if(subset[j]==ordered[i]) found=j
+    }
+    subset[found]=-1
+    if (length(found)==1) {
+       printf("%s%s* \\\n",prefix,found)
+    } else {
+       sortandcompress()
+       printf("%s\\[%s\\]* \\\n",prefix,found)
+    }
+  }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+  n=length(found)
+  for(i=1; i<=n; i++) tmp[i]=substr(found,i,1) 
+  asort(tmp)
+  for(i=1;i<=n;i++){
+    ipos=index(achars,tmp[i])
+    for(j=i;j<=n;j++){
+      jpos=index(achars,tmp[j])
+      if (jpos-ipos==j-i && substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+    }
+    if (bestj-i>3) {
+      tmp[i+1]="-" 
+      for(j=i+2;j<bestj;j++) tmp[j]=" "
+    }
+    i=bestj 
+  }
+  found=""
+  for(i=1; i<=n; i++) found=found tmp[i]
+  gsub(/ /,"",found)
+}
+'
Index: gcc/fortran/Make-lang.in
===================================================================
--- gcc/fortran/Make-lang.in	(revision 215017)
+++ gcc/fortran/Make-lang.in	(working copy)
@@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = execute.exp \
+			dg.exp=gfortran.dg/p* \
+			dg.exp=gfortran.dg/c* \
+			dg.exp=gfortran.dg/a* \
+			dg.exp=gfortran.dg/i* \
+			dg.exp=gfortran.dg/\[glow\]* \
+			dg.exp=gfortran.dg/\[mu\]* \
+			dg.exp=gfortran.dg/d* \
+			dg.exp=gfortran.dg/s* \
+			dg.exp=gfortran.dg/b* \
+			dg.exp=gfortran.dg/t* \
+			dg.exp=gfortran.dg/f* \
+			dg.exp=gfortran.dg/e* \
+			dg.exp=gfortran.dg/r* \
+			dg.exp=gfortran.dg/n* \
+			dg.exp=gfortran.dg/\[0-9A-Zhjkqvxyz\]*
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in	(revision 215017)
+++ gcc/Makefile.in	(working copy)
@@ -513,11 +513,17 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
+dg_target_exps:=aarch64 alpha arm avr bfin cris epiphany frv i386 ia64 
+dg_target_exps:=$(dg_target_exps) m68k microblaze mips powerpc rx s390
+dg_target_exps:=$(dg_target_exps) sh sparc spu tic6x xstormy16
+# also parallelize the target exps, using one common choice (i386) of split
+null:=
+space:=$(null) $(null)
+comma:=$(null),$(null)
+dg_target_exps_p1:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/a*))
+dg_target_exps_p2:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/\[fs\]*))
+dg_target_exps_p3:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/\[0-9A-Zbcdeg-oqrt-z\]*))
+dg_target_exps_p4:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/p*))
 # This lists a couple of test files that take most time during check-gcc.
 # When doing parallelized check-gcc, these can run in parallel with the
 # remaining tests.  Each word in this variable stands for work for one
@@ -526,21 +532,42 @@ dg_target_exps:=$(dg_target_exps),tic6x.
 # should be run in the same runtest invocation (usually if they aren't
 # very long running, but still should be split of from the check-parallel-$lang
 # remaining tests runtest invocation), they should be concatenated with commas.
+# At most $(check_p_numbers) goals should be created.
 # Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
 # by tcl) and as the *.exp arguments are mached both as is and with
 # */ prefixed to it in runtest_file_p, it is usually desirable to include
 # a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+check_gcc_parallelize=$(dg_target_exps_p1) \
+			$(dg_target_exps_p2) \
+			$(dg_target_exps_p3) \
+			$(dg_target_exps_p4) \
+			execute.exp=execute/2* \
+			execute.exp=execute/p* \
+			execute.exp=execute/9* \
+			execute.exp=execute/\[013-8A-Za-oq-z\]* \
+			compile.exp=compile/p* \
+			compile.exp=compile/2* \
+			compile.exp=compile/\[Babcdfilmsvx\]* \
+			compile.exp=compile/9* \
+			compile.exp=compile/\[013-8AC-Zeghjknoqrtuwyz\]* \
+			dg-torture.exp=torture/p* \
+			dg-torture.exp=torture/\[0-9A-Za-oq-z\]* \
+			vect.exp \
+			atomic.exp \
+			builtins.exp \
+			ubsan.exp \
+			struct-layout-1.exp \
+        		stackalign.exp \
+        		dg-exp.exp \
+        		ieee.exp \
+			asan.exp,tsan.exp \
+			lto.exp \
+			cilk-plus.exp \
+			noncompile.exp,dfp.exp,graphite.exp \
+			guality.exp,unsorted.exp \
+			debug.exp,tree-ssa.exp \
+			pch.exp,compat.exp,cpp.exp \
+			tls.exp,abi-x86_64.exp
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3639,7 +3666,8 @@ check_p_vars=$(check_$(check_p_tool)_par
 check_p_subno=$(word 2,$(subst _, ,$*))
 check_p_comma=,
 check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \
+		21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
 check_p_subdir=$(subst _,,$*)
 check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
 
Index: gcc/cp/Make-lang.in
===================================================================
--- gcc/cp/Make-lang.in	(revision 215017)
+++ gcc/cp/Make-lang.in	(working copy)
@@ -156,10 +156,30 @@ check-c++-subtargets : check-g++-subtarg
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp \
-	dg.exp=g++.dg/[0-9A-Za-bd-su-z]* \
-	dg.exp=g++.dg/[ct]* \
-	dg.exp=c-c++-common/*,dg-torture.exp
+check_g++_parallelize = dg.exp=g++.dg/\[ac\]* \
+	dg.exp=g++.dg/\[eint\]* \
+	dg.exp=g++.dg/\[Wflmru\]* \
+	dg.exp=g++.dg/\[dsv\]* \
+	dg.exp=g++.dg/p* \
+	dg.exp=g++.dg/\[0-9A-VXYZbghjkoqwxyz\]* \
+	dg.exp=c-c++-common/* \
+	old-deja.exp=g++.old-deja/g++.p* \
+	old-deja.exp=g++.old-deja/g++.\[lm\]* \
+	old-deja.exp=g++.old-deja/g++.o* \
+	old-deja.exp=g++.old-deja/g++.b* \
+	old-deja.exp=g++.old-deja/g++.\[0-9A-Zac-iknq-z\]* \
+	old-deja.exp=g++.old-deja/g++.j* \
+	dg-torture.exp=torture/p* \
+	dg-torture.exp=torture/\[0-9A-Za-oq-z\]* \
+	asan.exp \
+	ubsan.exp \
+	cilk-plus.exp \
+	struct-layout-1.exp \
+	debug.exp \
+	dwarf2.exp \
+	lto.exp \
+	gomp.exp \
+	tsan.exp
 
 #\f
 # Install hooks:

Re: [PATCH] RE: gcc parallel make check

[Jakub Jelinek]

[-- Attachment #1: Type: text/plain, Size: 5854 bytes --]

On Tue, Sep 09, 2014 at 03:14:08PM +0000, VandeVondele  Joost wrote:
> Attached is a further revision of the patch, now dealing with check-c++. Roughly 50% speedup here at '-j32' (18m vs 12m). For my setup (--enable-languages=c,c++,fortran) I have now improved all targets called in 'make -j32 -k check'. The latter is now 30% faster (15m vs 20m). Note that there are +- 1m fluctuations in these numbers, easily.
> 
> I currently have no plans to work on other check targets before this patch is committed.
> 
> OK for trunk ?

So, I've tested your patch on a few years old 16way AMD,
--enable-checking=yes and with make check -j1 and make check -j2
(the former doesn't parallelize, the latter does).

First of all, the -j2 testing shows more tests tested in gcc and libstdc++:

                === gcc Summary ===
 
-# of expected passes           108100
+# of expected passes           108130
 # of unexpected failures       175
 # of unexpected successes      33
 # of expected failures         262
...
                === libstdc++ Summary ===
 
-# of expected passes           10133
+# of expected passes           10152

In gcc testsuite the difference seems to be (according to sorted PASS
lines from *.sum files):
+PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -m3dnow  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -m3dnow  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -m3dnow  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -mmmx  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -mmmx  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -mmmx  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse2  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse2  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse2  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c -msse  (test for excess errors)
+PASS: gcc.target/i386/vect-args.c  -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c  -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c  -O (test for excess errors)
+PASS: gcc.target/i386/vect-args.c   (test for excess errors)
+PASS: gcc.target/i386/vect-args.c   (test for excess errors)
+PASS: gcc.target/i386/vect-args.c   (test for excess errors)
and in libstdc++:
+PASS: 23_containers/map/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3mapIiiSt4lessIiESaISt4pairIKiiEEE5eraseB5cxx11ESt17_Rb_tree_iteratorIS4_E
+PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors)
+PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multimapIiiSt4lessIiESaISt4pairIKiiEEE5eraseB5cxx11ESt17_Rb_tree_iteratorIS4_E
+PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors)
+PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multisetIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiE
+PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multisetIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiES5_
+PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors)
+PASS: 23_containers/set/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3setIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiE
+PASS: 23_containers/set/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3setIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiES5_
+PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors)
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIdE4imagB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIdE4realB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexI[eg]E4imagB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexI[eg]E4realB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIfE4imagB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIfE4realB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIiE4imagB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIiE4realB5cxx11Ev
+PASS: 26_numerics/complex/abi_tag.cc (test for excess errors)

Not sure where the bug is, could be e.g. in i386.exp for gcc, but for
libstdc++ less likely to be there rather than in the split.

Attached are seconds result of the various *.log.sep times, in the first
attachment from my last night's stock make -j48 -k check regtest (so, note
the times might be higher because most of the CPUs are kept much more busy)
and for make -j2 -k check regtest with your patch in the second attachment.
The normal{1,2,3} libstdc++ times are still way too high (bet one of those 3
is regex, which really has to be split), while e.g. the various 0/3 or even
< 30 seconds jobs are IMHO undesirable.

	Jakub

[-- Attachment #2: 1 --]
[-- Type: text/plain, Size: 2140 bytes --]

gcc/testsuite/g++/g++.log.sep 3228
gcc/testsuite/g++1/g++.log.sep 2446
gcc/testsuite/g++2/g++.log.sep 2065
gcc/testsuite/g++3/g++.log.sep 1843
gcc/testsuite/g++4/g++.log.sep 1579
gcc/testsuite/gcc/gcc.log.sep 2956
gcc/testsuite/gcc1/gcc.log.sep 2040
gcc/testsuite/gcc10/gcc.log.sep 675
gcc/testsuite/gcc11/gcc.log.sep 1209
gcc/testsuite/gcc2/gcc.log.sep 1787
gcc/testsuite/gcc3/gcc.log.sep 2219
gcc/testsuite/gcc4/gcc.log.sep 1757
gcc/testsuite/gcc5/gcc.log.sep 2042
gcc/testsuite/gcc6/gcc.log.sep 1182
gcc/testsuite/gcc7/gcc.log.sep 1866
gcc/testsuite/gcc8/gcc.log.sep 789
gcc/testsuite/gcc9/gcc.log.sep 875
gcc/testsuite/gfortran/gfortran.log.sep 1836
gcc/testsuite/gfortran1/gfortran.log.sep 1875
gcc/testsuite/gfortran2/gfortran.log.sep 1744
gcc/testsuite/gfortran3/gfortran.log.sep 1762
gcc/testsuite/gfortran4/gfortran.log.sep 1738
gcc/testsuite/gfortran5/gfortran.log.sep 1743
gcc/testsuite/gfortran6/gfortran.log.sep 1928
gcc/testsuite/go/go.log.sep 606
gcc/testsuite/go1/go.log.sep 1304
gcc/testsuite/go2/go.log.sep 244
gcc/testsuite/go3/go.log.sep 255
gcc/testsuite/go4/go.log.sep 37
gcc/testsuite/go5/go.log.sep 42
gcc/testsuite/objc/objc.log.sep 237
gcc/testsuite/objc1/objc.log.sep 60
gcc/testsuite/objc2/objc.log.sep 391
gcc/testsuite/objc3/objc.log.sep 57
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal0/libstdc++.log.sep 1177
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal1/libstdc++.log.sep 2043
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal10/libstdc++.log.sep 820
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal2/libstdc++.log.sep 1316
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal3/libstdc++.log.sep 1347
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal4/libstdc++.log.sep 1437
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal5/libstdc++.log.sep 1754
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal6/libstdc++.log.sep 1278
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal7/libstdc++.log.sep 1471
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal8/libstdc++.log.sep 1429
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal9/libstdc++.log.sep 1951

[-- Attachment #3: 2 --]
[-- Type: text/plain, Size: 4374 bytes --]

gcc/testsuite/g++/g++.log.sep 218
gcc/testsuite/g++1/g++.log.sep 303
gcc/testsuite/g++10/g++.log.sep 170
gcc/testsuite/g++11/g++.log.sep 117
gcc/testsuite/g++12/g++.log.sep 204
gcc/testsuite/g++13/g++.log.sep 110
gcc/testsuite/g++14/g++.log.sep 332
gcc/testsuite/g++15/g++.log.sep 167
gcc/testsuite/g++16/g++.log.sep 229
gcc/testsuite/g++17/g++.log.sep 346
gcc/testsuite/g++18/g++.log.sep 226
gcc/testsuite/g++19/g++.log.sep 212
gcc/testsuite/g++2/g++.log.sep 571
gcc/testsuite/g++20/g++.log.sep 105
gcc/testsuite/g++21/g++.log.sep 17
gcc/testsuite/g++22/g++.log.sep 100
gcc/testsuite/g++23/g++.log.sep 43
gcc/testsuite/g++24/g++.log.sep 52
gcc/testsuite/g++3/g++.log.sep 46
gcc/testsuite/g++4/g++.log.sep 3
gcc/testsuite/g++5/g++.log.sep 84
gcc/testsuite/g++6/g++.log.sep 257
gcc/testsuite/g++7/g++.log.sep 59
gcc/testsuite/g++8/g++.log.sep 200
gcc/testsuite/g++9/g++.log.sep 247
gcc/testsuite/gcc/gcc.log.sep 591
gcc/testsuite/gcc1/gcc.log.sep 486
gcc/testsuite/gcc10/gcc.log.sep 399
gcc/testsuite/gcc11/gcc.log.sep 439
gcc/testsuite/gcc12/gcc.log.sep 145
gcc/testsuite/gcc13/gcc.log.sep 49
gcc/testsuite/gcc14/gcc.log.sep 415
gcc/testsuite/gcc15/gcc.log.sep 401
gcc/testsuite/gcc16/gcc.log.sep 324
gcc/testsuite/gcc17/gcc.log.sep 409
gcc/testsuite/gcc18/gcc.log.sep 301
gcc/testsuite/gcc19/gcc.log.sep 306
gcc/testsuite/gcc2/gcc.log.sep 139
gcc/testsuite/gcc20/gcc.log.sep 196
gcc/testsuite/gcc21/gcc.log.sep 166
gcc/testsuite/gcc22/gcc.log.sep 0
gcc/testsuite/gcc23/gcc.log.sep 80
gcc/testsuite/gcc24/gcc.log.sep 119
gcc/testsuite/gcc25/gcc.log.sep 101
gcc/testsuite/gcc26/gcc.log.sep 114
gcc/testsuite/gcc27/gcc.log.sep 36
gcc/testsuite/gcc28/gcc.log.sep 484
gcc/testsuite/gcc29/gcc.log.sep 168
gcc/testsuite/gcc3/gcc.log.sep 218
gcc/testsuite/gcc30/gcc.log.sep 110
gcc/testsuite/gcc31/gcc.log.sep 86
gcc/testsuite/gcc4/gcc.log.sep 46
gcc/testsuite/gcc5/gcc.log.sep 605
gcc/testsuite/gcc6/gcc.log.sep 444
gcc/testsuite/gcc7/gcc.log.sep 343
gcc/testsuite/gcc8/gcc.log.sep 345
gcc/testsuite/gcc9/gcc.log.sep 547
gcc/testsuite/gfortran/gfortran.log.sep 174
gcc/testsuite/gfortran1/gfortran.log.sep 350
gcc/testsuite/gfortran10/gfortran.log.sep 132
gcc/testsuite/gfortran11/gfortran.log.sep 126
gcc/testsuite/gfortran12/gfortran.log.sep 166
gcc/testsuite/gfortran13/gfortran.log.sep 108
gcc/testsuite/gfortran14/gfortran.log.sep 170
gcc/testsuite/gfortran15/gfortran.log.sep 182
gcc/testsuite/gfortran16/gfortran.log.sep 88
gcc/testsuite/gfortran2/gfortran.log.sep 272
gcc/testsuite/gfortran3/gfortran.log.sep 357
gcc/testsuite/gfortran4/gfortran.log.sep 398
gcc/testsuite/gfortran5/gfortran.log.sep 242
gcc/testsuite/gfortran6/gfortran.log.sep 272
gcc/testsuite/gfortran7/gfortran.log.sep 383
gcc/testsuite/gfortran8/gfortran.log.sep 170
gcc/testsuite/gfortran9/gfortran.log.sep 157
gcc/testsuite/go/go.log.sep 95
gcc/testsuite/go1/go.log.sep 255
gcc/testsuite/go2/go.log.sep 121
gcc/testsuite/go3/go.log.sep 66
gcc/testsuite/go4/go.log.sep 8
gcc/testsuite/go5/go.log.sep 19
gcc/testsuite/objc/objc.log.sep 53
gcc/testsuite/objc1/objc.log.sep 16
gcc/testsuite/objc2/objc.log.sep 80
gcc/testsuite/objc3/objc.log.sep 10
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal0/libstdc++.log.sep 597
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal1/libstdc++.log.sep 1057
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal10/libstdc++.log.sep 377
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal11/libstdc++.log.sep 604
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal12/libstdc++.log.sep 357
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal13/libstdc++.log.sep 36
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal14/libstdc++.log.sep 334
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal15/libstdc++.log.sep 403
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal2/libstdc++.log.sep 896
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal3/libstdc++.log.sep 804
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal4/libstdc++.log.sep 580
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal5/libstdc++.log.sep 617
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal6/libstdc++.log.sep 487
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal7/libstdc++.log.sep 536
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal8/libstdc++.log.sep 593
x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/normal9/libstdc++.log.sep 633

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

Thanks for testing.

The vect-args.c I explained earlier, and is indeed due to i386.exp hardcoding those.

The libstdc++ double counts didn't appear in my testing, but I'll have  a look. Note that these patterns are handwritten, so error prone.

The long tests in libstdc++ come from (in timing order, from my machine):
          normal1) \
            dirs="`cd $$srcdir; echo e*/*`";; \
          normal2) \
            dirs="`cd $$srcdir; echo 28_*/a*`";; \
          normal3) \
            dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \
          normal4) \
            dirs="`cd $$srcdir; echo 2[459]_*/*`";; \

Re: [PATCH] RE: gcc parallel make check

[Jakub Jelinek]

On Wed, Sep 10, 2014 at 01:57:01PM +0000, VandeVondele  Joost wrote:
> Thanks for testing.
> 
> The vect-args.c I explained earlier, and is indeed due to i386.exp hardcoding those.

IMHO the best fix for that is following, use the same predicate whether to
run the vect-args.c tests or not as is used for all other tests.

Ok for trunk?

2014-09-10  Jakub Jelinek  <jakub@redhat.com>

	* gcc.target/i386/i386.exp: Only run vect-args.c tests
	if runtest_file_p says they should be run.

--- gcc/testsuite/gcc.target/i386/i386.exp.jj	2014-06-10 07:58:09.000000000 +0200
+++ gcc/testsuite/gcc.target/i386/i386.exp	2014-09-10 16:09:31.572493879 +0200
@@ -316,13 +316,16 @@ if ![info exists DEFAULT_CFLAGS] then {
 dg-init
 clearcap-init
 
+global runtests
 # Special case compilation of vect-args.c so we don't have to
 # replicate it 10 times.
-foreach type { "" -mmmx -m3dnow -msse -msse2 } {
-  foreach level { "" -O } {
-    set flags "$type $level"
-    verbose -log "Testing vect-args, $flags" 1
-    dg-test $srcdir/$subdir/vect-args.c $flags ""
+if [runtest_file_p $runtests $srcdir/$subdir/vect-args.c] {
+  foreach type { "" -mmmx -m3dnow -msse -msse2 } {
+    foreach level { "" -O } {
+      set flags "$type $level"
+      verbose -log "Testing vect-args, $flags" 1
+      dg-test $srcdir/$subdir/vect-args.c $flags ""
+    }
   }
 }
 

	Jakub

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 10, 2014, at 7:16 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> IMHO the best fix for that is following, use the same predicate whether to
> run the vect-args.c tests or not as is used for all other tests.
> 
> Ok for trunk?

Looks suspiciously looks familiar.  Ok.  Thanks.

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

Jakub,

> First of all, the -j2 testing shows more tests tested in gcc and libstdc++:
>
>-# of expected passes           10133
>+# of expected passes           10152
>
>+PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors)
>[...]
>
>Not sure where the bug is, could be e.g. in i386.exp for gcc, but for
>libstdc++ less likely to be there rather than in the split.

I looked into this, and believe this problem is already in current trunk, and not due to my patch. I.e. unmodified trunk also has these tests executed several times:

libstdc++-v3/testsuite/normal4/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc
libstdc++-v3/testsuite/normal1/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc

 I believe the current trunk pattern could indeed match those twice (Makefile.in in trunk):
 	  normal1) \
	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
 	  normal4) \
	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \

could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ?


Joost

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

> could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ?

I've checked that this fixes the bug in the current trunk split. I.e. files are stil tested, but now only once. Consider this change added to the previously submitted patch.

Re: [PATCH] RE: gcc parallel make check

[Jonathan Wakely]

On 11 September 2014 07:22, VandeVondele  Joost wrote:
> Jakub,
>
>> First of all, the -j2 testing shows more tests tested in gcc and libstdc++:
>>
>>-# of expected passes           10133
>>+# of expected passes           10152
>>
>>+PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors)
>>[...]
>>
>>Not sure where the bug is, could be e.g. in i386.exp for gcc, but for
>>libstdc++ less likely to be there rather than in the split.
>
> I looked into this, and believe this problem is already in current trunk, and not due to my patch. I.e. unmodified trunk also has these tests executed several times:
>
> libstdc++-v3/testsuite/normal4/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc
> libstdc++-v3/testsuite/normal1/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc
>
>  I believe the current trunk pattern could indeed match those twice (Makefile.in in trunk):
>           normal1) \
>             dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
>           normal4) \
>             dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
>
> could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ?

Yes, we are running these tests multiple times:

PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors)
PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors)
PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors)
PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors)
PASS: 26_numerics/complex/abi_tag.cc (test for excess errors)

I'll fix that.

RE: [PATCH] RE: gcc parallel make check

[VandeVondele Joost]

>> could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ?
>
>Yes, we are running these tests multiple times:
>
>PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors)
>PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors)
>PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors)
>PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors)
>PASS: 26_numerics/complex/abi_tag.cc (test for excess errors)
>
>I'll fix that.

Actually, the proper pattern should presumably be '[ab]*/* de*/* [ep]*/*' even though it seems to make no difference in testing. I'll have this included in yet another version of the parallel make check patch (plus some further reschuffling as requested by Jakub), so I think there is no need for you to fix this now.

Re: [PATCH] RE: gcc parallel make check

[Jonathan Wakely]

On 11 September 2014 15:45, VandeVondele  Joost
<joost.vandevondele@mat.ethz.ch> wrote:
>
>>> could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ?
>>
>>Yes, we are running these tests multiple times:
>>
>>PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors)
>>PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors)
>>PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors)
>>PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors)
>>PASS: 26_numerics/complex/abi_tag.cc (test for excess errors)
>>
>>I'll fix that.
>
> Actually, the proper pattern should presumably be '[ab]*/* de*/* [ep]*/*' even though it seems to make no difference in testing.

Yes, that's what I'm testing.

> I'll have this included in yet another version of the parallel make check patch (plus some further reschuffling as requested by Jakub), so I think there is no need for you to fix this now.

This can (and should) be fixed now, without waiting for some other change.

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 9, 2014, at 8:14 AM, VandeVondele Joost <joost.vandevondele@mat.ethz.ch> wrote:
> Attached is a further revision of the patch, now dealing with check-c++.

So when last I played in this area, I wanted a command line tool that would bin-pack from the command line.  I would then grab the seconds per for each .exp, and bin pack to the fixed N, where N was the core count or related to it like, like N+1, N*1.1+1, N*2, ceil(N*1.1)).  Then, I would just have 60-100 bins, and that -j64 run would be nicer.  The only reason why I didn’t push that patch up was I didn’t know of any such program.  :-(  I mention this in case someone knows of such a tool that is open source, hopefully GNU software.  The idea being, if a user has a 64 cores or want the .exp files to be more balanced on their target, they can be bothered to download the tool, don’t have it, and you get something a little more static.

Another way is to just make the buckets 60 seconds apiece.  This way, have  nice box, 60 seconds to test, otherwise, the test time is at most 1 minute unbalanced.

Re: [PATCH] gcc parallel make check

[David Malcolm]

On Wed, 2014-09-10 at 11:19 -0700, Mike Stump wrote:
> On Sep 9, 2014, at 8:14 AM, VandeVondele Joost
> <joost.vandevondele@mat.ethz.ch> wrote:
> > Attached is a further revision of the patch, now dealing with
> check-c++.
> 
> So when last I played in this area, I wanted a command line tool that
> would bin-pack from the command line.  I would then grab the seconds
> per for each .exp, and bin pack to the fixed N, where N was the core
> count or related to it like, like N+1, N*1.1+1, N*2, ceil(N*1.1)).
> Then, I would just have 60-100 bins, and that -j64 run would be nicer.
> The only reason why I didn’t push that patch up was I didn’t know of
> any such program.  :-(  I mention this in case someone knows of such a
> tool that is open source, hopefully GNU software.  The idea being, if
> a user has a 64 cores or want the .exp files to be more balanced on
> their target, they can be bothered to download the tool, don’t have
> it, and you get something a little more static.
> 
> Another way is to just make the buckets 60 seconds apiece.  This way,
> have  nice box, 60 seconds to test, otherwise, the test time is at
> most 1 minute unbalanced.

Perhaps this is a silly question, but has anyone tried going the whole
way and not having buckets, going to an extremely fine-grained approach:
split out all of the dj work into three phases:
(A) test discovery; write out a fine-grained Makefile in which *every*
testcase is its own make target (to the extreme limit of
parallelizability e.g. on the per-input-file level)
(B) invoke the Makefile, with -jN; each make target invokes dejagnu for
an individual testcase, and gets its own .log file
(C) combine the results

That way all parallelization in (B) relies on "make" to do the right
thing in terms of total number running jobs, available cores, load
average etc, albeit with a performance hit for all of the extra
reinvocations of "expect" (and a reordering of the results, but we can
impose a stable sort in phase (C) I guess).

Has anyone tried this?

Hope this is constructive
Dave

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 10, 2014, at 1:38 PM, David Malcolm <dmalcolm@redhat.com> wrote:
> Perhaps this is a silly question, but has anyone tried going the whole
> way and not having buckets, going to an extremely fine-grained approach

No, we fear the overhead, but do not know what it is.

Re: [PATCH] gcc parallel make check

[Joseph S. Myers]

On Wed, 10 Sep 2014, David Malcolm wrote:

> (A) test discovery; write out a fine-grained Makefile in which *every*
> testcase is its own make target (to the extreme limit of
> parallelizability e.g. on the per-input-file level)

The DejaGnu design doesn't allow test discovery in general (as the set of 
tests can depend on the results of previous tests; tests are run through 
arbitrary Tcl code in .exp files which both enumerates them and runs 
them).  Hopefully the GCC tests are well enough structured not to run into 
this problem.

(Being able to enumerate tests separately from running them, and to run 
each test independently, is the QMTest model, though there are other 
issues with how that handles some things that come up in toolchain 
testing.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Wed, Sep 10, 2014 at 04:38:32PM -0400, David Malcolm wrote:
> > So when last I played in this area, I wanted a command line tool that
> > would bin-pack from the command line.  I would then grab the seconds
> > per for each .exp, and bin pack to the fixed N, where N was the core
> > count or related to it like, like N+1, N*1.1+1, N*2, ceil(N*1.1)).
> > Then, I would just have 60-100 bins, and that -j64 run would be nicer.
> > The only reason why I didn’t push that patch up was I didn’t know of
> > any such program.  :-(  I mention this in case someone knows of such a
> > tool that is open source, hopefully GNU software.  The idea being, if
> > a user has a 64 cores or want the .exp files to be more balanced on
> > their target, they can be bothered to download the tool, don’t have
> > it, and you get something a little more static.
> > 
> > Another way is to just make the buckets 60 seconds apiece.  This way,
> > have  nice box, 60 seconds to test, otherwise, the test time is at
> > most 1 minute unbalanced.
> 
> Perhaps this is a silly question, but has anyone tried going the whole
> way and not having buckets, going to an extremely fine-grained approach:
> split out all of the dj work into three phases:
> (A) test discovery; write out a fine-grained Makefile in which *every*
> testcase is its own make target (to the extreme limit of
> parallelizability e.g. on the per-input-file level)
> (B) invoke the Makefile, with -jN; each make target invokes dejagnu for
> an individual testcase, and gets its own .log file
> (C) combine the results
> 
> That way all parallelization in (B) relies on "make" to do the right
> thing in terms of total number running jobs, available cores, load
> average etc, albeit with a performance hit for all of the extra
> reinvocations of "expect" (and a reordering of the results, but we can
> impose a stable sort in phase (C) I guess).
> 
> Has anyone tried this?

I fear that is going to be too expensive, because e.g. all the caching that
dejagnu and our tcl stuff does would be gone, all the tests for lp64 etc.
would need to be repeated for each test.

Perhaps better approach might be if we have some way how to synchronize among
multiple expect processes and spawn only as many expects (of course, per
check target) as there are CPUs.  E.g. if mkdir is atomic on all
hosts/filesystems we care about, we could have some shared directory that
make would clear before spawning all the expects, and after checking
runtest_file_p we could attempt to mkdir something (e.g. testcase filename
with $(srcdir) part removed, or *.exp filename / counter what test are we
considering or something similar) in the shared directory, if that would
succeed, it would tell us that we are the process that should run the test,
if that failed, we'd know some other runtest did that.
Or perhaps not for every single test, but every 10 or 100 tests or
something.

E.g. we could just override runtest_file_p itself, so that it would first
call the original dejagnu version, and then do this check.

	Jakub

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Wed, Sep 10, 2014 at 11:08:22PM +0200, Jakub Jelinek wrote:
> Perhaps better approach might be if we have some way how to synchronize among
> multiple expect processes and spawn only as many expects (of course, per
> check target) as there are CPUs.  E.g. if mkdir is atomic on all
> hosts/filesystems we care about, we could have some shared directory that
> make would clear before spawning all the expects, and after checking
> runtest_file_p we could attempt to mkdir something (e.g. testcase filename
> with $(srcdir) part removed, or *.exp filename / counter what test are we
> considering or something similar) in the shared directory, if that would
> succeed, it would tell us that we are the process that should run the test,
> if that failed, we'd know some other runtest did that.
> Or perhaps not for every single test, but every 10 or 100 tests or
> something.
> 
> E.g. we could just override runtest_file_p itself, so that it would first
> call the original dejagnu version, and then do this check.

Seems file mkdir in tcl doesn't error on pre-existing directory, so perhaps
[open $path {WRONLY EXCL CREAT}] ?
Now, does this work properly on all hosts we care about?

	Jakub

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 10, 2014, at 2:23 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> Seems file mkdir in tcl doesn't error on pre-existing directory,

shell mkdir will.  :-)

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Wed, Sep 10, 2014 at 11:23:34PM +0200, Jakub Jelinek wrote:
> On Wed, Sep 10, 2014 at 11:08:22PM +0200, Jakub Jelinek wrote:
> > Perhaps better approach might be if we have some way how to synchronize among
> > multiple expect processes and spawn only as many expects (of course, per
> > check target) as there are CPUs.  E.g. if mkdir is atomic on all
> > hosts/filesystems we care about, we could have some shared directory that
> > make would clear before spawning all the expects, and after checking
> > runtest_file_p we could attempt to mkdir something (e.g. testcase filename
> > with $(srcdir) part removed, or *.exp filename / counter what test are we
> > considering or something similar) in the shared directory, if that would
> > succeed, it would tell us that we are the process that should run the test,
> > if that failed, we'd know some other runtest did that.
> > Or perhaps not for every single test, but every 10 or 100 tests or
> > something.
> > 
> > E.g. we could just override runtest_file_p itself, so that it would first
> > call the original dejagnu version, and then do this check.
> 
> Seems file mkdir in tcl doesn't error on pre-existing directory, so perhaps
> [open $path {WRONLY EXCL CREAT}] ?
> Now, does this work properly on all hosts we care about?

Here is a proof of concept on the tcl side.
To get a large seq of numbers in the Makefile, I guess we can use something
like
check_p_numbers0:=1 2 3 4 5 6 7 8 9  
check_p_numbers1:=0 $(check_p_numbers0)
check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers1)))
check_p_numbers3:=$(patsubst %,0%,$(check_p_numbers1)) $(check_p_numbers2)
check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers3)))
check_p_numbers5:=$(patsubst %,0%,$(check_p_numbers3)) $(check_p_numbers4)
check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers5)))
check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
(and then what
check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
uses, just with $(check_$*_parallelize) replaced with something to match the
number of desired goals.
Looking at some of the *.exp tests, it seems only some of them (though, the
majority of the time consuming ones) actually use runtest_file_p, e.g.
compat.exp or struct-layout-1.exp and several others don't.

So, IMHO what we should do in the Makefile is, right inside
        @if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \
            && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
first rm -rf $(TESTSUITEDIR)/$*-parallel; mkdir $(TESTSUITEDIR)/$*-parallel
so that we start with empty dir, compute check_p_subdirs from actual -jN
number, then in check-parallel-gcc_1 etc. goals (but not in
check-parallel-gcc) set
GCC_RUNTEST_PARALLELIZE_DIR=$(TESTSUITEDIR)/$(check_p_tool)-parallel
in the environment and use RUNTESTFLAGS with selected known to be
parallelizable *.exp files (dg.exp execute.exp compile.exp and the like),
and use all the other *.exp files for check-parallel-gcc.

Thoughts on this?

Unfortunately, not sure how would that work with the
check-subtargets stuff if people are used to parallelize testing across
multiple machines (but it is unclear to me how they are merging the log/sum
files from the multiple machines anyway).  Not sure if this works over
NFS/AFS and other networked filesystems, if it does, supposedly they could
arrange for the *-parallel directories to be shared.

I can't find how to query the -jN value passed to make check by the user
though, both $(MFLAGS) and $(MAKEFLAGS) only contain something like
--jobserver-fds=3,5 -j from which it is not possible to find out how many
goals would be the upper reasonable limit.  Running too many goals would
waste time (once scheduled, the goal would only wildcard all the test, and
for all of them find in the *-parallel directory the test has been run
already), running too few could prevent good parallelization.

--- gcc/testsuite/lib/gcc-defs.exp.jj	2014-09-01 09:43:28.000000000 +0200
+++ gcc/testsuite/lib/gcc-defs.exp	2014-09-11 08:37:43.871943270 +0200
@@ -188,6 +188,30 @@ if { [info procs runtest_file_p] == "" }
     }
 }
 
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    global gcc_runtest_parallelize_counter
+
+    set gcc_runtest_parallelize_counter 0
+    proc runtest_file_p { runtests testcase } {
+	global gcc_runtest_parallelize_counter
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+
+	set dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+	set path $dir/$gcc_runtest_parallelize_counter
+	set gcc_runtest_parallelize_counter [expr {$gcc_runtest_parallelize_counter + 1}]
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    return 1
+	}
+	return 0
+    }
+}
+
 # Like dg-options, but adds to the default options rather than replacing them.
 
 proc dg-additional-options { args } {


	Jakub

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Thu, Sep 11, 2014 at 09:51:23AM +0200, Jakub Jelinek wrote:
> I can't find how to query the -jN value passed to make check by the user
> though, both $(MFLAGS) and $(MAKEFLAGS) only contain something like
> --jobserver-fds=3,5 -j from which it is not possible to find out how many
> goals would be the upper reasonable limit.  Running too many goals would
> waste time (once scheduled, the goal would only wildcard all the test, and
> for all of them find in the *-parallel directory the test has been run
> already), running too few could prevent good parallelization.

After a little googling, it seems there is no way to do that :(, unless
one e.g. attempts to find the command line of the topmost parent make
and scan it through ps or something.

There is an option to touch say *-parallel/finished file once any of the
check-parallel-gcc-{1,2,...} goals is done (because when it finishes, it
means all the tests for the particular check-$lang that are parallelizable
have either finished, or at least touched their file) and not start runtest
at all if finished already exists, but guess it would be still undesirable to have
tens of thousands of goals by default, so perhaps we could go with say
128 subgoals by default and have some env var to override it, so on the
really highly parallel boxes you'd specify
make -j512 -k check GCC_TEST_PARALLEL_SLOTS=512
or similar.

	Jakub

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Thu, Sep 11, 2014 at 10:06:40AM +0200, Jakub Jelinek wrote:
> There is an option to touch say *-parallel/finished file once any of the
> check-parallel-gcc-{1,2,...} goals is done (because when it finishes, it
> means all the tests for the particular check-$lang that are parallelizable
> have either finished, or at least touched their file) and not start runtest
> at all if finished already exists, but guess it would be still undesirable to have
> tens of thousands of goals by default, so perhaps we could go with say
> 128 subgoals by default and have some env var to override it, so on the
> really highly parallel boxes you'd specify
> make -j512 -k check GCC_TEST_PARALLEL_SLOTS=512
> or similar.

Here is a patch I'm testing now:

--- gcc/Makefile.in.jj	2014-09-08 22:12:56.000000000 +0200
+++ gcc/Makefile.in	2014-09-11 16:06:36.641219430 +0200
@@ -513,34 +513,10 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
-# This lists a couple of test files that take most time during check-gcc.
-# When doing parallelized check-gcc, these can run in parallel with the
-# remaining tests.  Each word in this variable stands for work for one
-# make goal and one extra make goal is added to handle all the *.exp
-# files not handled explicitly already.  If multiple *.exp files
-# should be run in the same runtest invocation (usually if they aren't
-# very long running, but still should be split of from the check-parallel-$lang
-# remaining tests runtest invocation), they should be concatenated with commas.
-# Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
-# by tcl) and as the *.exp arguments are mached both as is and with
-# */ prefixed to it in runtest_file_p, it is usually desirable to include
-# a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+# Upper limit to which it is useful to parallelize this lang target.
+# It doesn't make sense to try e.g. 128 goals for small testsuites
+# like objc or go.
+check_gcc_parallelize=10000
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3631,27 +3607,32 @@ $(filter-out $(lang_checks_parallelized)
 	    export TCL_LIBRARY ; fi ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
-$(patsubst %,%-subtargets,$(filter-out $(lang_checks_parallelized),$(lang_checks))): check-%-subtargets:
+$(patsubst %,%-subtargets,$(lang_checks)): check-%-subtargets:
 	@echo check-$*
 
 check_p_tool=$(firstword $(subst _, ,$*))
-check_p_vars=$(check_$(check_p_tool)_parallelize)
+check_p_count=$(check_$(check_p_tool)_parallelize)
 check_p_subno=$(word 2,$(subst _, ,$*))
-check_p_comma=,
-check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers1)))
+check_p_numbers3:=$(patsubst %,0%,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers3)))
+check_p_numbers5:=$(patsubst %,0%,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
 check_p_subdir=$(subst _,,$*)
-check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
+check_p_subdirs=$(wordlist 1,$(check_p_count),$(wordlist 1,$(or $(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers)))
 
 # For parallelized check-% targets, this decides whether parallelization
 # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
 # but optional --target_board or --extra_opts arguments).  If desirable,
 # recursive make is run with check-parallel-$lang{,1,2,3,4,5} etc. goals,
 # which can be executed in parallel, as they are run in separate directories.
-# check-parallel-$lang{1,2,3,4,5} etc. goals invoke runtest with the longest
-# running *.exp files from the testsuite, as determined by check_$lang_parallelize
-# variable.  The check-parallel-$lang goal in that case invokes runtest with
-# all the remaining *.exp files not handled by the separate goals.
+# check-parallel-$lang{,1,2,3,4,5} etc. goals invoke runtest with
+# GCC_RUNTEST_PARALLELIZE_DIR var in the environment and runtest_file_p
+# dejaGNU procedure is overridden to additionally synchronize through
+# a $lang-parallel directory which tests will be run by which runtest instance.
 # Afterwards contrib/dg-extract-results.sh is used to merge the sum and log
 # files.  If parallelization isn't desirable, only one recursive make
 # is run with check-parallel-$lang goal and check_$lang_parallelize variable
@@ -3662,76 +3643,61 @@ check_p_subdirs=$(wordlist 1,$(words $(c
 # to lang_checks_parallelized variable and define check_$lang_parallelize
 # variable (see above check_gcc_parallelize description).
 $(lang_checks_parallelized): check-% : site.exp
+	-rm -rf $(TESTSUITEDIR)/$*-parallel
 	@if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR) || true; \
+	  test -d $(TESTSUITEDIR)/$*-parallel || mkdir $(TESTSUITEDIR)/$*-parallel || true; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/$(TESTSUITEDIR)/$(check_p_tool)-parallel ; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR ; \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check-parallel-$* \
 	    $(patsubst %,check-parallel-$*_%, $(check_p_subdirs)); \
-	  for file in $(TESTSUITEDIR)/$*/$* \
-		      $(patsubst %,$(TESTSUITEDIR)/$*%/$*,$(check_p_subdirs));\
+	  sums= ; logs= ; \
+	  for dir in $(TESTSUITEDIR)/$* \
+		     $(patsubst %,$(TESTSUITEDIR)/$*%,$(check_p_subdirs));\
 	  do \
-	    mv -f $$file.sum $$file.sum.sep; mv -f $$file.log $$file.log.sep; \
+	    if [ -d $$dir ]; then \
+	      mv -f $$dir/$*.sum $$dir/$*.sum.sep; mv -f $$dir/$*.log $$dir/$*.log.sep; \
+	      sums="$$sums $$dir/$*.sum.sep"; logs="$$logs $$dir/$*.log.sep"; \
+	    fi; \
 	  done; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh \
-	    $(TESTSUITEDIR)/$*/$*.sum.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.sum.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh $$sums \
 	    > $(TESTSUITEDIR)/$*/$*.sum; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L \
-	    $(TESTSUITEDIR)/$*/$*.log.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.log.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L $$logs \
 	    > $(TESTSUITEDIR)/$*/$*.log; \
+	  rm -rf $(TESTSUITEDIR)/$*-parallel || true; \
 	else \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check_$*_parallelize= check-parallel-$*; \
 	fi
 
-# Just print the parallelized subtargets for those that want to split
-# the testing across machines.
-$(patsubst %,%-subtargets,$(lang_checks_parallelized)): check-%-subtargets:
-	@echo check-parallel-$* \
-	  $(patsubst %,check-parallel-$*_%, $(check_p_subdirs))
-
-# In the if [ -n "$(check_p_subno)" ] case runtest should be given the name of
-# the given *.exp file(s).  See comment above check_gcc_parallelize variable
-# for details on the content of these variables.
-#
-# In the elif [ -n "$(check_p_vars)" ] case runtest should be given
-# names of all the *.exp files for this tool that aren't already handled by
-# other goals.  First it finds all the *.exp files for this tool, then
-# prunes those already specified in check_$lang_parallelize or duplicates.
-#
-# Otherwise check-$lang isn't parallelized and runtest is invoked just with
-# the $(RUNTESTFLAGS) arguments.
 check-parallel-% : site.exp
 	-test -d plugin || mkdir plugin
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
-	cd $(TESTSUITEDIR)/$(check_p_subdir); \
-	rm -f tmp-site.exp; \
-	sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
+	if [ -n "$(check_p_subno)" \
+	     -a -n "$$GCC_RUNTEST_PARALLELIZE_DIR" \
+	     -a -f $(TESTSUITEDIR)/$(check_p_tool)-parallel/finished ]; then \
+	  rm -rf $(TESTSUITEDIR)/$(check_p_subdir); \
+	else \
+	  cd $(TESTSUITEDIR)/$(check_p_subdir); \
+	  rm -f tmp-site.exp; \
+	  sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
 		< ../../site.exp > tmp-site.exp; \
-	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
-	EXPECT=${EXPECT} ; export EXPECT ; \
-	if [ -f $${rootme}/../expect/expect ] ; then  \
-	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
-	    export TCL_LIBRARY ; fi ; \
-	runtestflags= ; \
-	if [ -n "$(check_p_subno)" ] ; then \
-	  runtestflags="$(check_p_subwork)"; \
-	elif [ -n "$(check_p_vars)" ] ; then \
-	  parts="`echo ' $(strip $(subst $(check_p_comma), ,$(check_p_vars))) ' \
-		  | sed 's/=[^ ]* / /g'`"; \
-	  for part in `find $$srcdir/testsuite/$(check_p_tool)* -name \*.exp` ; do \
-	    part=`basename $$part` ; \
-	    case " $$parts $$runtestflags " in \
-	      *" $$part "*) ;; \
-	      *) runtestflags="$$runtestflags $$part" ;; \
-	    esac ; \
-	  done ; \
-	fi ; \
-	$(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS) $$runtestflags)
+	  $(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	  EXPECT=${EXPECT} ; export EXPECT ; \
+	  if [ -f $${rootme}/../expect/expect ] ; then  \
+	    TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; \
+	  fi ; \
+	  $(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS); \
+	  if [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] ; then \
+	    touch $${rootme}/$(TESTSUITEDIR)/$(check_p_tool)-parallel/finished; \
+	  fi ; \
+	fi )
 
 # QMTest targets
 
--- gcc/go/Make-lang.in.jj	2014-05-02 15:39:49.766921376 +0200
+++ gcc/go/Make-lang.in	2014-09-11 16:08:50.049547349 +0200
@@ -131,11 +131,7 @@ go.srcman: doc/gccgo.1
 
 lang_checks += check-go
 lang_checks_parallelized += check-go
-check_go_parallelize = go-test.exp=*/test/\[0-57-9a-bd-hj-qs-zA-Z\]* \
-		       go-test.exp=*/test/c* \
-		       go-test.exp=*/test/i* \
-		       go-test.exp=*/test/r* \
-		       go-test.exp=*/test/6*
+check_go_parallelize = 10
 
 # Install hooks.
 
--- gcc/fortran/Make-lang.in.jj	2014-01-03 11:40:51.548352536 +0100
+++ gcc/fortran/Make-lang.in	2014-09-11 16:08:36.788612700 +0200
@@ -168,12 +168,7 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = 10000
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
--- gcc/cp/Make-lang.in.jj	2014-09-10 21:57:10.240320174 +0200
+++ gcc/cp/Make-lang.in	2014-09-11 16:08:27.367658600 +0200
@@ -156,11 +156,7 @@ check-c++-subtargets : check-g++-subtarg
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp \
-	dg.exp=g++.dg/[0-9A-Za-bd-su-z]* \
-	dg.exp=g++.dg/[ct]* \
-	dg.exp=c-c++-common/*,dg-torture.exp
-
+check_g++_parallelize = 10000
 #\f
 # Install hooks:
 # cc1plus is installed elsewhere as part of $(COMPILERS).
--- gcc/testsuite/lib/gcc-defs.exp.jj	2014-09-11 11:02:36.045944898 +0200
+++ gcc/testsuite/lib/gcc-defs.exp	2014-09-11 14:29:54.546328102 +0200
@@ -188,6 +188,30 @@ if { [info procs runtest_file_p] == "" }
     }
 }
 
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    global gcc_runtest_parallelize_counter
+
+    set gcc_runtest_parallelize_counter 0
+    proc runtest_file_p { runtests testcase } {
+	global gcc_runtest_parallelize_counter
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+
+	set dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+	set path $dir/$gcc_runtest_parallelize_counter
+	set gcc_runtest_parallelize_counter [expr {$gcc_runtest_parallelize_counter + 1}]
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    return 1
+	}
+	return 0
+    }
+}
+
 # Like dg-options, but adds to the default options rather than replacing them.
 
 proc dg-additional-options { args } {
--- gcc/objc/Make-lang.in.jj	2014-01-03 11:40:35.471444709 +0100
+++ gcc/objc/Make-lang.in	2014-09-11 16:09:02.671483908 +0200
@@ -96,7 +96,7 @@ lang_checks += check-objc
 # The following allows you to do 'make check-objc -j2'.  The
 # execute.exp tests will be run in parallel with all the other ones.
 lang_checks_parallelized += check-objc
-check_objc_parallelize = gnu-encoding.exp execute.exp exceptions.exp
+check_objc_parallelize = 6
 
 #\f
 # Install hooks:


	Jakub

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

> Here is a patch I'm testing now:

Hi Jakub,

I also tested your patch to compare timings vs a newer patch (v8) I'll send soon

== patch v8 == make -j32 -k ==
check-fortran   4m58.178s
check-c++     ~10m
check-c       ~10m
check          15m29.873s

== patch Jakub
check-c++     ~20m
check-fortran   3m31.237s 
check-c         8m8

on the positive side, your patch provides a further speedup e.g. fortran and c testing (where it splits things nicely). The libstdc++ bottleneck is not solved, but I guess that is expected.

As you have presumably found as well, your patch introduces a number failures, because some tests seem to have additional dependencies, either explicit or implicit:

e.g. in gfortran.dg/binding_label_tests_10_main.f03
! { dg-do compile }
! This file must be compiled AFTER binding_label_tests_10.f03, which it 
! should be because dejagnu will sort the files.
module binding_label_tests_10_main

in gfortran.dg/class_45b.f03 
! { dg-do link }
! { dg-additional-sources class_45a.f03 }

This could clearly trigger as well in the current scheme of splitting, only we have been lucky that dependencies seem to be 'well behaved' in having the same initial letter in the filename.

Joost

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Thu, Sep 11, 2014 at 05:04:56PM +0000, VandeVondele  Joost wrote:
> > Here is a patch I'm testing now:
> 
> I also tested your patch to compare timings vs a newer patch (v8) I'll send soon
> 
> == patch v8 == make -j32 -k ==
> check-fortran   4m58.178s
> check-c++     ~10m
> check-c       ~10m
> check          15m29.873s
> 
> == patch Jakub
> check-c++     ~20m
> check-fortran   3m31.237s 
> check-c         8m8
> 
> on the positive side, your patch provides a further speedup e.g. fortran
> and c testing (where it splits things nicely).  The libstdc++ bottleneck
> is not solved, but I guess that is expected.

The same technique can be of course used for libstdc++, I just didn't want
to do that until the -C gcc testing is changed.

> As you have presumably found as well, your patch introduces a number failures, because some tests seem to have additional dependencies, either explicit or implicit:

I found more issues, in particular it seemed that struct-layout-1.exp,
gnu-encoding.exp, plugin.exp and some go*.exp don't call runtest_file_p
in the same amounts and same arguments in all invocations.
And these Fortran inter-test dependencies, which Tobias told me is
PR56408.

Unfortunately my remote testing box is unreachable now and I'm still waiting
for DDR4 modules to finish building better workstation, so can't test this
right now.  The patch below intends to serialize the content of the
problematic *.exp tests (the first runtest to reach one of those will simply
run all the tests from that *.exp file, others will skip it).

For go I currently have no idea why does that happen, quick hack would be
just disable parallelization of go temporarily and let Ian investigate.

For PR56408 we need some fix.

	Jakub

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Thu, Sep 11, 2014 at 07:26:37PM +0200, Jakub Jelinek wrote:
> right now.  The patch below intends to serialize the content of the
> problematic *.exp tests (the first runtest to reach one of those will simply
> run all the tests from that *.exp file, others will skip it).

Forgotten patch below.  BTW, something will probably need to be done about
acats too, either similar approach or just splitting the chapters into
little more jobs, because otherwise in make -C check -j48 acats dominated
the testing time for me.

--- gcc/Makefile.in.jj	2014-09-08 22:12:56.000000000 +0200
+++ gcc/Makefile.in	2014-09-11 16:58:01.076371437 +0200
@@ -513,34 +513,10 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
-# This lists a couple of test files that take most time during check-gcc.
-# When doing parallelized check-gcc, these can run in parallel with the
-# remaining tests.  Each word in this variable stands for work for one
-# make goal and one extra make goal is added to handle all the *.exp
-# files not handled explicitly already.  If multiple *.exp files
-# should be run in the same runtest invocation (usually if they aren't
-# very long running, but still should be split of from the check-parallel-$lang
-# remaining tests runtest invocation), they should be concatenated with commas.
-# Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
-# by tcl) and as the *.exp arguments are mached both as is and with
-# */ prefixed to it in runtest_file_p, it is usually desirable to include
-# a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+# Upper limit to which it is useful to parallelize this lang target.
+# It doesn't make sense to try e.g. 128 goals for small testsuites
+# like objc or go.
+check_gcc_parallelize=10000
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3631,27 +3607,32 @@ $(filter-out $(lang_checks_parallelized)
 	    export TCL_LIBRARY ; fi ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
-$(patsubst %,%-subtargets,$(filter-out $(lang_checks_parallelized),$(lang_checks))): check-%-subtargets:
+$(patsubst %,%-subtargets,$(lang_checks)): check-%-subtargets:
 	@echo check-$*
 
 check_p_tool=$(firstword $(subst _, ,$*))
-check_p_vars=$(check_$(check_p_tool)_parallelize)
+check_p_count=$(check_$(check_p_tool)_parallelize)
 check_p_subno=$(word 2,$(subst _, ,$*))
-check_p_comma=,
-check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers1)))
+check_p_numbers3:=$(patsubst %,0%,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers3)))
+check_p_numbers5:=$(patsubst %,0%,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
 check_p_subdir=$(subst _,,$*)
-check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
+check_p_subdirs=$(wordlist 1,$(check_p_count),$(wordlist 1,$(or $(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers)))
 
 # For parallelized check-% targets, this decides whether parallelization
 # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
 # but optional --target_board or --extra_opts arguments).  If desirable,
 # recursive make is run with check-parallel-$lang{,1,2,3,4,5} etc. goals,
 # which can be executed in parallel, as they are run in separate directories.
-# check-parallel-$lang{1,2,3,4,5} etc. goals invoke runtest with the longest
-# running *.exp files from the testsuite, as determined by check_$lang_parallelize
-# variable.  The check-parallel-$lang goal in that case invokes runtest with
-# all the remaining *.exp files not handled by the separate goals.
+# check-parallel-$lang{,1,2,3,4,5} etc. goals invoke runtest with
+# GCC_RUNTEST_PARALLELIZE_DIR var in the environment and runtest_file_p
+# dejaGNU procedure is overridden to additionally synchronize through
+# a $lang-parallel directory which tests will be run by which runtest instance.
 # Afterwards contrib/dg-extract-results.sh is used to merge the sum and log
 # files.  If parallelization isn't desirable, only one recursive make
 # is run with check-parallel-$lang goal and check_$lang_parallelize variable
@@ -3662,76 +3643,61 @@ check_p_subdirs=$(wordlist 1,$(words $(c
 # to lang_checks_parallelized variable and define check_$lang_parallelize
 # variable (see above check_gcc_parallelize description).
 $(lang_checks_parallelized): check-% : site.exp
+	-rm -rf $(TESTSUITEDIR)/$*-parallel
 	@if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR) || true; \
+	  test -d $(TESTSUITEDIR)/$*-parallel || mkdir $(TESTSUITEDIR)/$*-parallel || true; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/$(TESTSUITEDIR)/$(check_p_tool)-parallel ; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR ; \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check-parallel-$* \
 	    $(patsubst %,check-parallel-$*_%, $(check_p_subdirs)); \
-	  for file in $(TESTSUITEDIR)/$*/$* \
-		      $(patsubst %,$(TESTSUITEDIR)/$*%/$*,$(check_p_subdirs));\
+	  sums= ; logs= ; \
+	  for dir in $(TESTSUITEDIR)/$* \
+		     $(patsubst %,$(TESTSUITEDIR)/$*%,$(check_p_subdirs));\
 	  do \
-	    mv -f $$file.sum $$file.sum.sep; mv -f $$file.log $$file.log.sep; \
+	    if [ -d $$dir ]; then \
+	      mv -f $$dir/$*.sum $$dir/$*.sum.sep; mv -f $$dir/$*.log $$dir/$*.log.sep; \
+	      sums="$$sums $$dir/$*.sum.sep"; logs="$$logs $$dir/$*.log.sep"; \
+	    fi; \
 	  done; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh \
-	    $(TESTSUITEDIR)/$*/$*.sum.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.sum.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh $$sums \
 	    > $(TESTSUITEDIR)/$*/$*.sum; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L \
-	    $(TESTSUITEDIR)/$*/$*.log.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.log.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L $$logs \
 	    > $(TESTSUITEDIR)/$*/$*.log; \
+	  rm -rf $(TESTSUITEDIR)/$*-parallel || true; \
 	else \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check_$*_parallelize= check-parallel-$*; \
 	fi
 
-# Just print the parallelized subtargets for those that want to split
-# the testing across machines.
-$(patsubst %,%-subtargets,$(lang_checks_parallelized)): check-%-subtargets:
-	@echo check-parallel-$* \
-	  $(patsubst %,check-parallel-$*_%, $(check_p_subdirs))
-
-# In the if [ -n "$(check_p_subno)" ] case runtest should be given the name of
-# the given *.exp file(s).  See comment above check_gcc_parallelize variable
-# for details on the content of these variables.
-#
-# In the elif [ -n "$(check_p_vars)" ] case runtest should be given
-# names of all the *.exp files for this tool that aren't already handled by
-# other goals.  First it finds all the *.exp files for this tool, then
-# prunes those already specified in check_$lang_parallelize or duplicates.
-#
-# Otherwise check-$lang isn't parallelized and runtest is invoked just with
-# the $(RUNTESTFLAGS) arguments.
 check-parallel-% : site.exp
 	-test -d plugin || mkdir plugin
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
-	cd $(TESTSUITEDIR)/$(check_p_subdir); \
-	rm -f tmp-site.exp; \
-	sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
+	if [ -n "$(check_p_subno)" \
+	     -a -n "$$GCC_RUNTEST_PARALLELIZE_DIR" \
+	     -a -f $(TESTSUITEDIR)/$(check_p_tool)-parallel/finished ]; then \
+	  rm -rf $(TESTSUITEDIR)/$(check_p_subdir); \
+	else \
+	  cd $(TESTSUITEDIR)/$(check_p_subdir); \
+	  rm -f tmp-site.exp; \
+	  sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
 		< ../../site.exp > tmp-site.exp; \
-	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
-	EXPECT=${EXPECT} ; export EXPECT ; \
-	if [ -f $${rootme}/../expect/expect ] ; then  \
-	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
-	    export TCL_LIBRARY ; fi ; \
-	runtestflags= ; \
-	if [ -n "$(check_p_subno)" ] ; then \
-	  runtestflags="$(check_p_subwork)"; \
-	elif [ -n "$(check_p_vars)" ] ; then \
-	  parts="`echo ' $(strip $(subst $(check_p_comma), ,$(check_p_vars))) ' \
-		  | sed 's/=[^ ]* / /g'`"; \
-	  for part in `find $$srcdir/testsuite/$(check_p_tool)* -name \*.exp` ; do \
-	    part=`basename $$part` ; \
-	    case " $$parts $$runtestflags " in \
-	      *" $$part "*) ;; \
-	      *) runtestflags="$$runtestflags $$part" ;; \
-	    esac ; \
-	  done ; \
-	fi ; \
-	$(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS) $$runtestflags)
+	  $(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	  EXPECT=${EXPECT} ; export EXPECT ; \
+	  if [ -f $${rootme}/../expect/expect ] ; then  \
+	    TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; \
+	  fi ; \
+	  $(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS); \
+	  if [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] ; then \
+	    touch $${rootme}/$(TESTSUITEDIR)/$(check_p_tool)-parallel/finished; \
+	  fi ; \
+	fi )
 
 # QMTest targets
 
--- gcc/go/Make-lang.in.jj	2014-05-02 15:39:49.766921376 +0200
+++ gcc/go/Make-lang.in	2014-09-11 16:08:50.049547349 +0200
@@ -131,11 +131,7 @@ go.srcman: doc/gccgo.1
 
 lang_checks += check-go
 lang_checks_parallelized += check-go
-check_go_parallelize = go-test.exp=*/test/\[0-57-9a-bd-hj-qs-zA-Z\]* \
-		       go-test.exp=*/test/c* \
-		       go-test.exp=*/test/i* \
-		       go-test.exp=*/test/r* \
-		       go-test.exp=*/test/6*
+check_go_parallelize = 10
 
 # Install hooks.
 
--- gcc/fortran/Make-lang.in.jj	2014-01-03 11:40:51.548352536 +0100
+++ gcc/fortran/Make-lang.in	2014-09-11 16:08:36.788612700 +0200
@@ -168,12 +168,7 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = 10000
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
--- gcc/cp/Make-lang.in.jj	2014-09-10 21:57:10.240320174 +0200
+++ gcc/cp/Make-lang.in	2014-09-11 16:08:27.367658600 +0200
@@ -156,11 +156,7 @@ check-c++-subtargets : check-g++-subtarg
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp \
-	dg.exp=g++.dg/[0-9A-Za-bd-su-z]* \
-	dg.exp=g++.dg/[ct]* \
-	dg.exp=c-c++-common/*,dg-torture.exp
-
+check_g++_parallelize = 10000
 #\f
 # Install hooks:
 # cc1plus is installed elsewhere as part of $(COMPILERS).
--- gcc/testsuite/g++.dg/compat/struct-layout-1.exp.jj	2014-08-01 09:23:40.000000000 +0200
+++ gcc/testsuite/g++.dg/compat/struct-layout-1.exp	2014-09-11 18:45:09.422267240 +0200
@@ -95,6 +95,13 @@ load_lib g++-dg.exp
 
 g++_init
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if [gcc_parallel_test_run_p struct_layout_1] {
+gcc_parallel_test_enable 0
+
 # Save variables for the C++ compiler under test, which each test will
 # change a couple of times.  This must be done after calling g++-init.
 set save_gxx_under_test $GXX_UNDER_TEST
@@ -168,3 +175,6 @@ if { $status == 0 } then {
 
 # Restore the original compiler under test.
 compat-use-tst-compiler
+
+gcc_parallel_test_enable 1
+}
--- gcc/testsuite/g++.dg/plugin/plugin.exp.jj	2014-03-10 10:49:57.000000000 +0100
+++ gcc/testsuite/g++.dg/plugin/plugin.exp	2014-09-11 18:47:53.920423064 +0200
@@ -44,6 +44,13 @@ if $tracelevel then {
 # Load support procs.
 load_lib plugin-support.exp
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if [gcc_parallel_test_run_p plugin] {
+gcc_parallel_test_enable 0
+
 # Specify the plugin source file and the associated test files in a list.
 # plugin_test_list={ {plugin1 test1 test2 ...} {plugin2 test1 ...} ... }
 set plugin_test_list [list \
@@ -68,3 +75,6 @@ foreach plugin_test $plugin_test_list {
     set plugin_input_tests [lreplace $plugin_test 0 0]
     plugin-test-execute $plugin_src $plugin_input_tests
 }
+
+gcc_parallel_test_enable 1
+}
--- gcc/testsuite/gcc.dg/compat/struct-layout-1.exp.jj	2014-01-03 11:40:41.000000000 +0100
+++ gcc/testsuite/gcc.dg/compat/struct-layout-1.exp	2014-09-11 18:43:05.234916416 +0200
@@ -53,6 +53,13 @@ load_lib compat.exp
 
 gcc_init
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if [gcc_parallel_test_run_p struct_layout_1] {
+gcc_parallel_test_enable 0
+
 # Save variables for the C compiler under test, which each test will
 # change a couple of times.  This must be done after calling gcc-init.
 set compat_save_gcc_under_test $GCC_UNDER_TEST
@@ -128,3 +135,6 @@ if { $status == 0 } then {
 
 # Restore the original compiler under test.
 compat-use-tst-compiler
+
+gcc_parallel_test_enable 1
+}
--- gcc/testsuite/gcc.dg/plugin/plugin.exp.jj	2014-01-03 11:40:42.000000000 +0100
+++ gcc/testsuite/gcc.dg/plugin/plugin.exp	2014-09-11 18:47:02.220691539 +0200
@@ -44,6 +44,13 @@ if $tracelevel then {
 # Load support procs.
 load_lib plugin-support.exp
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if [gcc_parallel_test_run_p plugin] {
+gcc_parallel_test_enable 0
+
 # Specify the plugin source file and the associated test files in a list.
 # plugin_test_list={ {plugin1 test1 test2 ...} {plugin2 test1 ...} ... }
 set plugin_test_list [list \
@@ -80,3 +87,6 @@ dg-runtest [lsort [glob -nocomplain $src
 
 # All done.
 dg-finish
+
+gcc_parallel_test_enable 1
+}
--- gcc/testsuite/lib/gcc-defs.exp.jj	2014-09-11 11:02:36.045944898 +0200
+++ gcc/testsuite/lib/gcc-defs.exp	2014-09-11 18:20:01.021037249 +0200
@@ -188,6 +188,56 @@ if { [info procs runtest_file_p] == "" }
     }
 }
 
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    global gcc_runtest_parallelize_counter
+    global gcc_runtest_parallelize_enable
+
+    set gcc_runtest_parallelize_counter 0
+    set gcc_runtest_parallelize_enable 1
+
+    proc gcc_parallel_test_run_p { testcase } {
+	global gcc_runtest_parallelize_counter
+	global gcc_runtest_parallelize_enable
+
+	if { $gcc_runtest_parallelize_enable == 0 } {
+	    return 1
+	}
+	set dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+	set path $dir/$gcc_runtest_parallelize_counter
+	set gcc_runtest_parallelize_counter [expr {$gcc_runtest_parallelize_counter + 1}]
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    return 1
+	}
+	return 0
+    }
+
+    proc gcc_parallel_test_enable { val } {
+	global gcc_runtest_parallelize_enable
+	set gcc_runtest_parallelize_enable $val
+    }
+
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    proc runtest_file_p { runtests testcase } {
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+	return [gcc_parallel_test_run_p $testcase]
+    }
+
+} else {
+
+    proc gcc_parallel_test_run_p { testcase } {
+	return 1
+    }
+
+    proc gcc_parallel_test_enable { val } {
+    }
+
+}
+
 # Like dg-options, but adds to the default options rather than replacing them.
 
 proc dg-additional-options { args } {
--- gcc/testsuite/objc.dg/gnu-encoding/gnu-encoding.exp.jj	2014-01-03 11:40:43.000000000 +0100
+++ gcc/testsuite/objc.dg/gnu-encoding/gnu-encoding.exp	2014-09-11 18:12:05.575481502 +0200
@@ -28,6 +28,13 @@ if ![info exists DEFAULT_CFLAGS] then {
 # Initialize `dg'.
 dg-init
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if [gcc_parallel_test_run_p gnu_encoding] {
+gcc_parallel_test_enable 0
+
 #
 # gnu-encoding tests
 #
@@ -67,10 +74,8 @@ if { $status == 0 } then {
     warning "Could not compile objc.dg/gnu-encoding/struct-layout-encoding-1 generator"
 }
 
-
-
-
-
+gcc_parallel_test_enable 1
+}
 
 # All done.
 dg-finish
--- gcc/objc/Make-lang.in.jj	2014-01-03 11:40:35.471444709 +0100
+++ gcc/objc/Make-lang.in	2014-09-11 16:09:02.671483908 +0200
@@ -96,7 +96,7 @@ lang_checks += check-objc
 # The following allows you to do 'make check-objc -j2'.  The
 # execute.exp tests will be run in parallel with all the other ones.
 lang_checks_parallelized += check-objc
-check_objc_parallelize = gnu-encoding.exp execute.exp exceptions.exp
+check_objc_parallelize = 6
 
 #\f
 # Install hooks:

	Jakub

Re: [PATCH] gcc parallel make check

[Bernhard Reutner-Fischer]

On 11 September 2014 20:19:31 Jakub Jelinek <jakub@redhat.com> wrote:

> On Thu, Sep 11, 2014 at 07:26:37PM +0200, Jakub Jelinek wrote:
> > right now.  The patch below intends to serialize the content of the
> > problematic *.exp tests (the first runtest to reach one of those will simply
> > run all the tests from that *.exp file, others will skip it).
>
> Forgotten patch below.  BTW, something will probably need to be done about
> acats too, either similar approach or just splitting the chapters into
> little more jobs, because otherwise in make -C check -j48 acats dominated
> the testing time for me.


+	if [ -n "$(check_p_subno)" \
+	 -a -n "$$GCC_RUNTEST_PARALLELIZE_DIR" \
+	 -a -f $(TESTSUITEDIR)/$(check_p_tool)-parallel/finished ]; then \

test(1) -a and -o are obsolescent, please chain [] && [] instead.

Thanks -a cheers ;)


Sent with AquaMail for Android
http://www.aqua-mail.com

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Thu, Sep 11, 2014 at 11:24:08PM +0200, Bernhard Reutner-Fischer wrote:
> On 11 September 2014 20:19:31 Jakub Jelinek <jakub@redhat.com> wrote:
> 
> >On Thu, Sep 11, 2014 at 07:26:37PM +0200, Jakub Jelinek wrote:
> >> right now.  The patch below intends to serialize the content of the
> >> problematic *.exp tests (the first runtest to reach one of those will simply
> >> run all the tests from that *.exp file, others will skip it).
> >
> >Forgotten patch below.  BTW, something will probably need to be done about
> >acats too, either similar approach or just splitting the chapters into
> >little more jobs, because otherwise in make -C check -j48 acats dominated
> >the testing time for me.
> 
> 
> +	if [ -n "$(check_p_subno)" \
> +	 -a -n "$$GCC_RUNTEST_PARALLELIZE_DIR" \
> +	 -a -f $(TESTSUITEDIR)/$(check_p_tool)-parallel/finished ]; then \
> 
> test(1) -a and -o are obsolescent, please chain [] && [] instead.

That is news to me, but given the amount of test -a/-o uses e.g. in
gcc/configure and hundreds of places, I'd say what we care is what is more
portable to old shells.

	Jakub

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 11, 2014, at 3:15 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> That is news to me, but given the amount of test -a/-o uses e.g. in
> gcc/configure and hundreds of places, I'd say what we care is what is more
> portable to old shells.

No, we can’t care about that.  If that were true, the _ && _ in the compiler source would have been fixed.  Since it has not, then trivially it is portable enough.  One day someone will come along and fixup the -a and -o instances for us.  && should be preferred.

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

> And these Fortran inter-test dependencies, which Tobias told me is
> PR56408.
> For PR56408 we need some fix.

BTW, is there anything special about Fortran ? There are at least 180 test files that contain 'dg-additional-sources' some in a very non-local way:

	./objc.dg/foreach-2.m: /* { dg-additional-sources "../objc-obj-c++-shared/nsconstantstring-class-impl.m" } */

Joost

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Thu, Sep 11, 2014 at 06:33:27PM +0000, VandeVondele  Joost wrote:
> > And these Fortran inter-test dependencies, which Tobias told me is
> > PR56408.
> > For PR56408 we need some fix.
> 
> BTW, is there anything special about Fortran ? There are at least 180 test files that contain 'dg-additional-sources' some in a very non-local way:
> 
> 	./objc.dg/foreach-2.m: /* { dg-additional-sources "../objc-obj-c++-shared/nsconstantstring-class-impl.m" } */

gc-additional-sources is not a problem, that is the solution if you need
Fortran modules inter-TU and can do a link test, see e.g.
libgomp/testsuite/libgomp.fortran/declare-simd-{2,3}.f90 for one way how to
do that.  With gc-additional-sources one command line of the compiler driver
compiles both source files, it is a single test from dejagnu POV, so
necessarily run together.
But there are some tests that want to have other TU modules and want to be
dg-do compile only, currently this uses a keep-modules hack which creates
inter-test dependencies.  So we need something different.

	Jakub

Re: [PATCH] gcc parallel make check

[Tobias Burnus]

On 11.09.2014 20:33, VandeVondele Joost wrote:
>> >For PR56408 we need some fix.
> BTW, is there anything special about Fortran ? There are at least 180 test files that contain 'dg-additional-sources' some in a very non-local way:

Well, the question is what you want to do with the different files. If 
you just want to compile them, e.g. for linking or executing, you are 
fine. However, with Fortran module's there is a .mod file produced – 
which gives an ordering constraint: The file with the module has to be 
compiled first before the other file can be compiled. (If one puts the 
module into the same file, one has effectively one translation unit and 
some bugs do not pop up in this case.)

The current scheme comes at its limits in that case. Mainly because the 
file specified in dg-additional-sources is compiled after the one in 
which this line is written. That can be fine for linking/run-time tests, 
where one disables the by-itself compilation of the second file - and 
puts the module into the first file. However, as soon as one wants to do 
more, e.g. dg-error/dg-warning output, checking the dump/assembler etc., 
one is in trouble. See the files listed in the PR for issues.

By contrast, for C/C++, one has a header file which is included by the 
preprocessor (hence before the compiler), thus, there is no ordering 
issue as no compiler input is generated. I don't know whether one could 
run into issues with precompiled header files - but there on has at 
least the different name *.h/*.hpp and *.c/.cc. I don't know about Ada 
or C++'s upcoming ISO-version of precompiled headers ("modules"), maybe 
there one runs into similar issues?

See the PR for some attempts to fix it.

Tobias

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

>>> >For PR56408 we need some fix.
>> BTW, is there anything special about Fortran ? There are at least 180 test files that contain 'dg-additional-sources' >some in a very non-local way:
>The current scheme comes at its limits in that case. . See the files listed in the PR for issues.

So, what about a pragmatic solution, and move the tests that rely on being serialized to a subdirectory serialized/ where, like now, we rely on the implicit ordering we have now ? At least it makes this assumption somewhat explicit.

Joost

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

> a newer patch (v8) I'll send soon

attached with updated changelog. Compared to the previously posted v6, only the libstdc++-v3/testsuite/Makefile.am has been refined to split a little more the e*/* pattern, and two quickly running goal have been merged, in addition to fixing the pre-exisiting error in some of the patterns in that file.

Checked comparing testsuite results before after. 

Obviously, if Jakub's patch can be made to work around the testsuite special cases, I believe it should be superior. If not, the attached patch is working as far as I can tell, and provides a significant improvement over current trunk.

Joost

[-- Attachment #2: patch-speedup-check-v08-CL.txt --]
[-- Type: text/plain, Size: 1056 bytes --]

contrib/ChangeLog

2014-09-12  Joost VandeVondele  <vondele@gcc.gnu.org>

	* generate_tcl_patterns.sh: New file.

gcc/fortran/ChangeLog

2014-09-12  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Make-lang.in (check_gfortran_parallelize): Improved parallelism.

gcc/Changelog

2014-09-12  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Makefile.in (check_gcc_parallelize): Improved parallelism.
	(check_p_numbers): Increase maximum value.
	(dg_target_exps): Mention targets as separate words only.
	(null,space,comma,dg_target_exps_p1,dg_target_exps_p2,
	dg_target_exps_p3,dg_target_exps_p4): New variables.

gcc/cp/ChangeLog

2014-09-12  Joost VandeVondele  <vondele@gcc.gnu.org>

	* Make-lang.in (check_g++_parallelize): Improved parallelism.

libstdc++-v3/ChangeLog

2014-09-12  Joost VandeVondele  <vondele@gcc.gnu.org>

        * testsuite/Makefile.am (check_DEJAGNU_normal_targets): Add
        check-DEJAGNUnormal[11-15].
        (check-DEJAGNU): Split into 15 jobs for parallel testing, correct pattern.
        * testsuite/Makefile.in: Regenerated.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: patch-speedup-check-v08.diff --]
[-- Type: text/x-patch; name="patch-speedup-check-v08.diff", Size: 16277 bytes --]

Index: libstdc++-v3/testsuite/Makefile.in
===================================================================
--- libstdc++-v3/testsuite/Makefile.in	(revision 215147)
+++ libstdc++-v3/testsuite/Makefile.in	(working copy)
@@ -301,7 +301,7 @@ lists_of_files = \
 
 extract_symvers = $(glibcxx_builddir)/scripts/extract_symvers
 baseline_subdir := $(shell $(CXX) $(baseline_subdir_switch))
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)
 
 # Runs the testsuite, but in compile only mode.
 # Can be used to test sources with non-GNU FE's at various warning
@@ -562,7 +562,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
+	  for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \
 	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
 	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
 	  done; \
@@ -589,25 +589,35 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	    fi; \
 	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
 	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
+	    dirs="`cd $$srcdir; echo experimental/* ext/[a-m]*`";; \
 	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 28_*/a*`";; \
 	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \
 	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
+	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
 	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
+	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
 	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[m-tw-z]*`";; \
 	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
+	    dirs="`cd $$srcdir; echo 26_*/*`";; \
 	  normal8) \
 	    dirs="`cd $$srcdir; echo 27_*/*`";; \
 	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
+	    dirs="`cd $$srcdir; echo 22_*/*`";; \
 	  normal10) \
 	    dirs="`cd $$srcdir; echo t*/*`";; \
+	  normal11) \
+	    dirs="`cd $$srcdir; echo 28_*/b*`";; \
+	  normal12) \
+	    dirs="`cd $$srcdir; echo 28_*/[c-z]*`";; \
+	  normal13) \
+	    dirs="`cd $$srcdir; echo ext/[n-z]*`";; \
+	  normal14) \
+	    dirs="`cd $$srcdir; echo de*/* p*/* [ab]*/* 23_*/v*`";; \
+	  normal15) \
+	    dirs="`cd $$srcdir; echo 23_*/[a-k]*`";; \
 	esac; \
 	if [ -n "$*" ]; then cd "$*"; fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
Index: libstdc++-v3/testsuite/Makefile.am
===================================================================
--- libstdc++-v3/testsuite/Makefile.am	(revision 215147)
+++ libstdc++-v3/testsuite/Makefile.am	(working copy)
@@ -101,7 +101,7 @@ new-abi-baseline:
 	@test ! -f $*/site.exp || mv $*/site.exp $*/site.bak
 	@mv $*/site.exp.tmp $*/site.exp
 
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)
 $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
@@ -111,7 +111,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
+	  for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \
 	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
 	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
 	  done; \
@@ -138,25 +138,35 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	    fi; \
 	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
 	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
+	    dirs="`cd $$srcdir; echo experimental/* ext/[a-m]*`";; \
 	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 28_*/a*`";; \
 	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[lu]*`";; \
 	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
+	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
 	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
+	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
 	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 23_*/[m-tw-z]*`";; \
 	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
+	    dirs="`cd $$srcdir; echo 26_*/*`";; \
 	  normal8) \
 	    dirs="`cd $$srcdir; echo 27_*/*`";; \
 	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
+	    dirs="`cd $$srcdir; echo 22_*/*`";; \
 	  normal10) \
 	    dirs="`cd $$srcdir; echo t*/*`";; \
+	  normal11) \
+	    dirs="`cd $$srcdir; echo 28_*/b*`";; \
+	  normal12) \
+	    dirs="`cd $$srcdir; echo 28_*/[c-z]*`";; \
+	  normal13) \
+	    dirs="`cd $$srcdir; echo ext/[n-z]*`";; \
+	  normal14) \
+	    dirs="`cd $$srcdir; echo de*/* p*/* [ab]*/* 23_*/v*`";; \
+	  normal15) \
+	    dirs="`cd $$srcdir; echo 23_*/[a-k]*`";; \
 	esac; \
 	if [ -n "$*" ]; then cd "$*"; fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
Index: contrib/generate_tcl_patterns.sh
===================================================================
--- contrib/generate_tcl_patterns.sh	(revision 0)
+++ contrib/generate_tcl_patterns.sh	(revision 0)
@@ -0,0 +1,114 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input, starting with [0-9A-Za-z],
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+#   cd gcc/gcc/testsuite/gfortran.dg
+#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele <Joost.VandeVondele@mat.ethz.ch>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+  # list of allowed starting chars for a file name in a dir to split
+  achars="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+  ranget="11111111112222222222222222222222222233333333333333333333333333"
+}
+{
+  if (index(achars,substr($1,1,1))==0){
+     print "file : " $1 " does not start with an allowed character."
+     _assert_exit = 1
+     exit 1
+  }
+  nfiles++ ; files[nfiles]=$1
+}
+END{
+  if (_assert_exit) exit 1
+  for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+  for(i=1; i<=nfiles; i++) {
+     if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+  };
+  asort(count,ordered)
+  countsingle=0
+  groups=0
+  label=""
+  for(i=length(achars);i>=1;i--) {
+    countsingle=countsingle+ordered[i] 
+    for(j=1;j<=length(achars);j++) {
+       if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+    }
+    count[found]=-1
+    label=label found
+    if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+    if(countsingle+val>maxcount) {
+      subset[label]=countsingle
+      print "Adding label: ", label, "matching files:" countsingle
+      groups++
+      countsingle=0
+      label=""
+    }
+  }
+  print "patterns:"
+  asort(subset,ordered)
+  for(i=groups;i>=1;i--) {
+    for(j in subset){
+      if(subset[j]==ordered[i]) found=j
+    }
+    subset[found]=-1
+    if (length(found)==1) {
+       printf("%s%s* \\\n",prefix,found)
+    } else {
+       sortandcompress()
+       printf("%s\\[%s\\]* \\\n",prefix,found)
+    }
+  }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+  n=length(found)
+  for(i=1; i<=n; i++) tmp[i]=substr(found,i,1) 
+  asort(tmp)
+  for(i=1;i<=n;i++){
+    ipos=index(achars,tmp[i])
+    for(j=i;j<=n;j++){
+      jpos=index(achars,tmp[j])
+      if (jpos-ipos==j-i && substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+    }
+    if (bestj-i>3) {
+      tmp[i+1]="-" 
+      for(j=i+2;j<bestj;j++) tmp[j]=" "
+    }
+    i=bestj 
+  }
+  found=""
+  for(i=1; i<=n; i++) found=found tmp[i]
+  gsub(/ /,"",found)
+}
+'

Property changes on: contrib/generate_tcl_patterns.sh
___________________________________________________________________
Added: svn:executable
   + *

Index: gcc/cp/Make-lang.in
===================================================================
--- gcc/cp/Make-lang.in	(revision 215147)
+++ gcc/cp/Make-lang.in	(working copy)
@@ -156,10 +156,30 @@ check-c++-subtargets : check-g++-subtarg
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp \
-	dg.exp=g++.dg/[0-9A-Za-bd-su-z]* \
-	dg.exp=g++.dg/[ct]* \
-	dg.exp=c-c++-common/*,dg-torture.exp
+check_g++_parallelize = dg.exp=g++.dg/\[ac\]* \
+	dg.exp=g++.dg/\[eint\]* \
+	dg.exp=g++.dg/\[Wflmru\]* \
+	dg.exp=g++.dg/\[dsv\]* \
+	dg.exp=g++.dg/p* \
+	dg.exp=g++.dg/\[0-9A-VXYZbghjkoqwxyz\]* \
+	dg.exp=c-c++-common/* \
+	old-deja.exp=g++.old-deja/g++.p* \
+	old-deja.exp=g++.old-deja/g++.\[lm\]* \
+	old-deja.exp=g++.old-deja/g++.o* \
+	old-deja.exp=g++.old-deja/g++.b* \
+	old-deja.exp=g++.old-deja/g++.\[0-9A-Zac-iknq-z\]* \
+	old-deja.exp=g++.old-deja/g++.j* \
+	dg-torture.exp=torture/p* \
+	dg-torture.exp=torture/\[0-9A-Za-oq-z\]* \
+	asan.exp \
+	ubsan.exp \
+	cilk-plus.exp \
+	struct-layout-1.exp \
+	debug.exp \
+	dwarf2.exp \
+	lto.exp \
+	gomp.exp \
+	tsan.exp
 
 #\f
 # Install hooks:
Index: gcc/fortran/Make-lang.in
===================================================================
--- gcc/fortran/Make-lang.in	(revision 215147)
+++ gcc/fortran/Make-lang.in	(working copy)
@@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = execute.exp \
+			dg.exp=gfortran.dg/p* \
+			dg.exp=gfortran.dg/c* \
+			dg.exp=gfortran.dg/a* \
+			dg.exp=gfortran.dg/i* \
+			dg.exp=gfortran.dg/\[glow\]* \
+			dg.exp=gfortran.dg/\[mu\]* \
+			dg.exp=gfortran.dg/d* \
+			dg.exp=gfortran.dg/s* \
+			dg.exp=gfortran.dg/b* \
+			dg.exp=gfortran.dg/t* \
+			dg.exp=gfortran.dg/f* \
+			dg.exp=gfortran.dg/e* \
+			dg.exp=gfortran.dg/r* \
+			dg.exp=gfortran.dg/n* \
+			dg.exp=gfortran.dg/\[0-9A-Zhjkqvxyz\]*
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in	(revision 215147)
+++ gcc/Makefile.in	(working copy)
@@ -513,11 +513,17 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
+dg_target_exps:=aarch64 alpha arm avr bfin cris epiphany frv i386 ia64 
+dg_target_exps:=$(dg_target_exps) m68k microblaze mips powerpc rx s390
+dg_target_exps:=$(dg_target_exps) sh sparc spu tic6x xstormy16
+# also parallelize the target exps, using one common choice (i386) of split
+null:=
+space:=$(null) $(null)
+comma:=$(null),$(null)
+dg_target_exps_p1:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/a*))
+dg_target_exps_p2:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/\[fs\]*))
+dg_target_exps_p3:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/\[0-9A-Zbcdeg-oqrt-z\]*))
+dg_target_exps_p4:=$(subst $(space),$(comma),$(foreach target,$(dg_target_exps),$(target).exp=$(target)/p*))
 # This lists a couple of test files that take most time during check-gcc.
 # When doing parallelized check-gcc, these can run in parallel with the
 # remaining tests.  Each word in this variable stands for work for one
@@ -526,21 +532,42 @@ dg_target_exps:=$(dg_target_exps),tic6x.
 # should be run in the same runtest invocation (usually if they aren't
 # very long running, but still should be split of from the check-parallel-$lang
 # remaining tests runtest invocation), they should be concatenated with commas.
+# At most $(check_p_numbers) goals should be created.
 # Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
 # by tcl) and as the *.exp arguments are mached both as is and with
 # */ prefixed to it in runtest_file_p, it is usually desirable to include
 # a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+check_gcc_parallelize=$(dg_target_exps_p1) \
+			$(dg_target_exps_p2) \
+			$(dg_target_exps_p3) \
+			$(dg_target_exps_p4) \
+			execute.exp=execute/2* \
+			execute.exp=execute/p* \
+			execute.exp=execute/9* \
+			execute.exp=execute/\[013-8A-Za-oq-z\]* \
+			compile.exp=compile/p* \
+			compile.exp=compile/2* \
+			compile.exp=compile/\[Babcdfilmsvx\]* \
+			compile.exp=compile/9* \
+			compile.exp=compile/\[013-8AC-Zeghjknoqrtuwyz\]* \
+			dg-torture.exp=torture/p* \
+			dg-torture.exp=torture/\[0-9A-Za-oq-z\]* \
+			vect.exp \
+			atomic.exp \
+			builtins.exp \
+			ubsan.exp \
+			struct-layout-1.exp \
+        		stackalign.exp \
+        		dg-exp.exp \
+        		ieee.exp \
+			asan.exp,tsan.exp \
+			lto.exp \
+			cilk-plus.exp \
+			noncompile.exp,dfp.exp,graphite.exp \
+			guality.exp,unsorted.exp \
+			debug.exp,tree-ssa.exp \
+			pch.exp,compat.exp,cpp.exp \
+			tls.exp,abi-x86_64.exp
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3639,7 +3666,8 @@ check_p_vars=$(check_$(check_p_tool)_par
 check_p_subno=$(word 2,$(subst _, ,$*))
 check_p_comma=,
 check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \
+		21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
 check_p_subdir=$(subst _,,$*)
 check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
 

Re: [PATCH] gcc parallel make check

[Jonathan Wakely]

On 12/09/14 09:47 +0000, VandeVondele  Joost wrote:
>> a newer patch (v8) I'll send soon
>
>attached with updated changelog. Compared to the previously posted v6, only the libstdc++-v3/testsuite/Makefile.am has been refined to split a little more the e*/* pattern, and two quickly running goal have been merged, in addition to fixing the pre-exisiting error in some of the patterns in that file.

The libstdc++ part is OK, thanks.

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Sep 12, 2014 at 09:47:00AM +0000, VandeVondele  Joost wrote:
> Obviously, if Jakub's patch can be made to work around the testsuite
> special cases, I believe it should be superior.  If not, the attached
> patch is working as far as I can tell, and provides a significant
> improvement over current trunk.

Here is my latest version of the patch.

With this patch I get identical test_summary output on make -k check
(completely serial testing) and make -j48 -k check from toplevel directory.

Major changes since last version:
1) I've changed the granularity, now it does O_EXCL|O_CREAT attempt
   only every 10th runtest_file_p invocation rather than every iteration,
   so that it spends a little bit less time in expect processes (especially
   if used on slower networked filesystems etc.)
2) in guality.exp I've discovered that it counts as PASS/FAIL also the
   initial check whether any guality tests should be performed; that stuff
   isn't runtest_file_p guarded (of course, needs to be performed by every
   runtest instance), but made the totals dependent on how many runtest
   instances invoked guality.exp; IMHO we don't want the pass/fail counts
   to be volatile, so I've fixed it by not counting that test into the
   results
3) various other *.exp fails didn't use runtest_file_p, especially the
   gcc.misc-tests/ ones, tweaked those like struct-layout-1.exp or
   plugin.exp so that only the first runtest instance to encounter those
   runs all of the *.exp file serially
4) fixed go-test.exp so that runtest_file_p used for a test that
   has been already runtest_file_p checked is not testing for parallel
   execution - otherwise the numbers in go-parallel/ directory could be
   different between different instances
5) libstdc++-v3/testsuite has been changed too to use the same stuff;
   note that the new xmethods.exp apparently isn't invoked for
   parallel testing, but it hasn't been before either (and the tree
   I've been testing on the 16way box still didn't have it).  Either
   it should be tested serially like abi.exp or pretty-printers.exp,
   or needs to be double-checked for parallel testing (guess it
   could have similar issues as guality.exp)
6) changed -a to ] && [

Regtested on x86_64-linux, ok for trunk?

2014-09-12  Jakub Jelinek  <jakub@redhat.com>

gcc/
	* Makefile.in (dg_target_exps): Remove.
	(check_gcc_parallelize): Change to just an upper bound number.
	(check-%-subtargets): Always print the non-parallelized goals.
	(check_p_vars, check_p_comma, check_p_subwork): Remove.
	(check_p_count, check_p_numbers0, check_p_numbers1, check_p_numbers2,
	check_p_numbers3, check_p_numbers4, check_p_numbers5,
	check_p_numbers6): New variables.
	(check_p_numbers): Set to sequence from 1 to 9999.
	(check_p_subdirs): Set to sequence from 1 to minimum of
	$(check_p_count) and either GCC_TEST_PARALLEL_SLOTS env var if set,
	or 128.
	(check-%, check-parallel-%): Rewritten so that for parallelized
	testing each job runs all the *.exp files, with
	GCC_RUNTEST_PARALLELIZE_DIR set in environment.
gcc/go/
	* Make-lang.in (check_go_parallelize): Change to just an upper bound
	number.
gcc/fortran/
	* Make-lang.in (check_gfortran_parallelize): Change to just an upper
	bound number.
gcc/cp/
	* Make-lang.in (check_g++_parallelize): Change to just an upper bound
	number.
gcc/objc/
	* Make-lang.in (check_objc_parallelize): Change to just an upper
	bound number.
gcc/testsuite/
	* lib/gcc-defs.exp (gcc_parallel_test_run_p,
	gcc_parallel_test_enable): New procedures.  If
	GCC_RUNTEST_PARALLELIZE_DIR is set in environment, override
	runtest_file_p to invoke also gcc_parallel_test_run_p.
	* g++.dg/guality/guality.exp (check_guality): Save/restore
	test_counts array around the body of the procedure.
	* gcc.dg/guality/guality.exp (check_guality): Likewise.
	* g++.dg/compat/struct-layout-1.exp: Run all the tests serially
	by the first parallel runtest encountering it.
	* g++.dg/plugin/plugin.exp: Likewise.
	* gcc.dg/compat/struct-layout-1.exp: Likewise.
	* gcc.dg/plugin/plugin.exp: Likewise.
	* objc.dg/gnu-encoding/gnu-encoding.exp: Likewise.
	* gcc.misc-tests/matrix1.exp: Likewise.
	* gcc.misc-tests/dhry.exp: Likewise.
	* gcc.misc-tests/acker1.exp: Likewise.
	* gcc.misc-tests/linkage.exp: Likewise.
	* gcc.misc-tests/mg.exp: Likewise.
	* gcc.misc-tests/mg-2.exp: Likewise.
	* gcc.misc-tests/sort2.exp: Likewise.
	* gcc.misc-tests/sieve.exp: Likewise.
	* gcc.misc-tests/options.exp: Likewise.
	* gcc.misc-tests/help.exp: Likewise.
	* go.test/go-test.exp (go-gc-tests): Use
	gcc_parallel_test_enable {0, 1} around all handling of
	each test.
libstdc++-v3/
	* testsuite/Makefile.am (check_p_numbers0, check_p_numbers1,
	check_p_numbers2, check_p_numbers3, check_p_numbers4,
	check_p_numbers5, check_p_numbers6, check_p_numbers,
	check_p_subdirs): New variables.
	(check_DEJAGNU_normal_targets): Use check_p_subdirs.
	(check-DEJAGNU): Rewritten so that for parallelized
	testing each job runs all the *.exp files, with
	GCC_RUNTEST_PARALLELIZE_DIR set in environment.
	* testsuite/Makefile.in: Regenerated.
	* testsuite/lib/libstdc++.exp (gcc_parallel_test_run_p,
	gcc_parallel_test_enable): New procedures.  If
	GCC_RUNTEST_PARALLELIZE_DIR is set in environment, override
	runtest_file_p to invoke also gcc_parallel_test_run_p.

--- gcc/Makefile.in.jj	2014-09-11 20:35:28.603749840 +0200
+++ gcc/Makefile.in	2014-09-12 11:25:50.245338490 +0200
@@ -513,34 +513,10 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
-# This lists a couple of test files that take most time during check-gcc.
-# When doing parallelized check-gcc, these can run in parallel with the
-# remaining tests.  Each word in this variable stands for work for one
-# make goal and one extra make goal is added to handle all the *.exp
-# files not handled explicitly already.  If multiple *.exp files
-# should be run in the same runtest invocation (usually if they aren't
-# very long running, but still should be split of from the check-parallel-$lang
-# remaining tests runtest invocation), they should be concatenated with commas.
-# Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
-# by tcl) and as the *.exp arguments are mached both as is and with
-# */ prefixed to it in runtest_file_p, it is usually desirable to include
-# a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+# Upper limit to which it is useful to parallelize this lang target.
+# It doesn't make sense to try e.g. 128 goals for small testsuites
+# like objc or go.
+check_gcc_parallelize=10000
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3631,27 +3607,32 @@ $(filter-out $(lang_checks_parallelized)
 	    export TCL_LIBRARY ; fi ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
-$(patsubst %,%-subtargets,$(filter-out $(lang_checks_parallelized),$(lang_checks))): check-%-subtargets:
+$(patsubst %,%-subtargets,$(lang_checks)): check-%-subtargets:
 	@echo check-$*
 
 check_p_tool=$(firstword $(subst _, ,$*))
-check_p_vars=$(check_$(check_p_tool)_parallelize)
+check_p_count=$(check_$(check_p_tool)_parallelize)
 check_p_subno=$(word 2,$(subst _, ,$*))
-check_p_comma=,
-check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers1)))
+check_p_numbers3:=$(patsubst %,0%,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers3)))
+check_p_numbers5:=$(patsubst %,0%,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
 check_p_subdir=$(subst _,,$*)
-check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
+check_p_subdirs=$(wordlist 1,$(check_p_count),$(wordlist 1,$(or $(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers)))
 
 # For parallelized check-% targets, this decides whether parallelization
 # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
 # but optional --target_board or --extra_opts arguments).  If desirable,
 # recursive make is run with check-parallel-$lang{,1,2,3,4,5} etc. goals,
 # which can be executed in parallel, as they are run in separate directories.
-# check-parallel-$lang{1,2,3,4,5} etc. goals invoke runtest with the longest
-# running *.exp files from the testsuite, as determined by check_$lang_parallelize
-# variable.  The check-parallel-$lang goal in that case invokes runtest with
-# all the remaining *.exp files not handled by the separate goals.
+# check-parallel-$lang{,1,2,3,4,5} etc. goals invoke runtest with
+# GCC_RUNTEST_PARALLELIZE_DIR var in the environment and runtest_file_p
+# dejaGNU procedure is overridden to additionally synchronize through
+# a $lang-parallel directory which tests will be run by which runtest instance.
 # Afterwards contrib/dg-extract-results.sh is used to merge the sum and log
 # files.  If parallelization isn't desirable, only one recursive make
 # is run with check-parallel-$lang goal and check_$lang_parallelize variable
@@ -3662,76 +3643,61 @@ check_p_subdirs=$(wordlist 1,$(words $(c
 # to lang_checks_parallelized variable and define check_$lang_parallelize
 # variable (see above check_gcc_parallelize description).
 $(lang_checks_parallelized): check-% : site.exp
+	-rm -rf $(TESTSUITEDIR)/$*-parallel
 	@if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR) || true; \
+	  test -d $(TESTSUITEDIR)/$*-parallel || mkdir $(TESTSUITEDIR)/$*-parallel || true; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/$(TESTSUITEDIR)/$(check_p_tool)-parallel ; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR ; \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check-parallel-$* \
 	    $(patsubst %,check-parallel-$*_%, $(check_p_subdirs)); \
-	  for file in $(TESTSUITEDIR)/$*/$* \
-		      $(patsubst %,$(TESTSUITEDIR)/$*%/$*,$(check_p_subdirs));\
+	  sums= ; logs= ; \
+	  for dir in $(TESTSUITEDIR)/$* \
+		     $(patsubst %,$(TESTSUITEDIR)/$*%,$(check_p_subdirs));\
 	  do \
-	    mv -f $$file.sum $$file.sum.sep; mv -f $$file.log $$file.log.sep; \
+	    if [ -d $$dir ]; then \
+	      mv -f $$dir/$*.sum $$dir/$*.sum.sep; mv -f $$dir/$*.log $$dir/$*.log.sep; \
+	      sums="$$sums $$dir/$*.sum.sep"; logs="$$logs $$dir/$*.log.sep"; \
+	    fi; \
 	  done; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh \
-	    $(TESTSUITEDIR)/$*/$*.sum.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.sum.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh $$sums \
 	    > $(TESTSUITEDIR)/$*/$*.sum; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L \
-	    $(TESTSUITEDIR)/$*/$*.log.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.log.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L $$logs \
 	    > $(TESTSUITEDIR)/$*/$*.log; \
+	  rm -rf $(TESTSUITEDIR)/$*-parallel || true; \
 	else \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check_$*_parallelize= check-parallel-$*; \
 	fi
 
-# Just print the parallelized subtargets for those that want to split
-# the testing across machines.
-$(patsubst %,%-subtargets,$(lang_checks_parallelized)): check-%-subtargets:
-	@echo check-parallel-$* \
-	  $(patsubst %,check-parallel-$*_%, $(check_p_subdirs))
-
-# In the if [ -n "$(check_p_subno)" ] case runtest should be given the name of
-# the given *.exp file(s).  See comment above check_gcc_parallelize variable
-# for details on the content of these variables.
-#
-# In the elif [ -n "$(check_p_vars)" ] case runtest should be given
-# names of all the *.exp files for this tool that aren't already handled by
-# other goals.  First it finds all the *.exp files for this tool, then
-# prunes those already specified in check_$lang_parallelize or duplicates.
-#
-# Otherwise check-$lang isn't parallelized and runtest is invoked just with
-# the $(RUNTESTFLAGS) arguments.
 check-parallel-% : site.exp
 	-test -d plugin || mkdir plugin
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
-	cd $(TESTSUITEDIR)/$(check_p_subdir); \
-	rm -f tmp-site.exp; \
-	sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
+	if [ -n "$(check_p_subno)" ] \
+	   && [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] \
+	   && [ -f $(TESTSUITEDIR)/$(check_p_tool)-parallel/finished ]; then \
+	  rm -rf $(TESTSUITEDIR)/$(check_p_subdir); \
+	else \
+	  cd $(TESTSUITEDIR)/$(check_p_subdir); \
+	  rm -f tmp-site.exp; \
+	  sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
 		< ../../site.exp > tmp-site.exp; \
-	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
-	EXPECT=${EXPECT} ; export EXPECT ; \
-	if [ -f $${rootme}/../expect/expect ] ; then  \
-	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
-	    export TCL_LIBRARY ; fi ; \
-	runtestflags= ; \
-	if [ -n "$(check_p_subno)" ] ; then \
-	  runtestflags="$(check_p_subwork)"; \
-	elif [ -n "$(check_p_vars)" ] ; then \
-	  parts="`echo ' $(strip $(subst $(check_p_comma), ,$(check_p_vars))) ' \
-		  | sed 's/=[^ ]* / /g'`"; \
-	  for part in `find $$srcdir/testsuite/$(check_p_tool)* -name \*.exp` ; do \
-	    part=`basename $$part` ; \
-	    case " $$parts $$runtestflags " in \
-	      *" $$part "*) ;; \
-	      *) runtestflags="$$runtestflags $$part" ;; \
-	    esac ; \
-	  done ; \
-	fi ; \
-	$(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS) $$runtestflags)
+	  $(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	  EXPECT=${EXPECT} ; export EXPECT ; \
+	  if [ -f $${rootme}/../expect/expect ] ; then  \
+	    TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; \
+	  fi ; \
+	  $(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS); \
+	  if [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] ; then \
+	    touch $${rootme}/$(TESTSUITEDIR)/$(check_p_tool)-parallel/finished; \
+	  fi ; \
+	fi )
 
 # QMTest targets
 
--- gcc/go/Make-lang.in.jj	2014-09-11 20:35:28.547750131 +0200
+++ gcc/go/Make-lang.in	2014-09-12 11:25:50.271338395 +0200
@@ -131,11 +131,7 @@ go.srcman: doc/gccgo.1
 
 lang_checks += check-go
 lang_checks_parallelized += check-go
-check_go_parallelize = go-test.exp=*/test/\[0-57-9a-bd-hj-qs-zA-Z\]* \
-		       go-test.exp=*/test/c* \
-		       go-test.exp=*/test/i* \
-		       go-test.exp=*/test/r* \
-		       go-test.exp=*/test/6*
+check_go_parallelize = 10
 
 # Install hooks.
 
--- gcc/fortran/Make-lang.in.jj	2014-09-11 20:35:28.386750942 +0200
+++ gcc/fortran/Make-lang.in	2014-09-12 11:25:50.271338395 +0200
@@ -168,12 +168,7 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = 10000
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
--- gcc/cp/Make-lang.in.jj	2014-09-11 20:35:28.444750641 +0200
+++ gcc/cp/Make-lang.in	2014-09-12 11:25:50.273338382 +0200
@@ -156,11 +156,7 @@ check-c++-subtargets : check-g++-subtarg
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp \
-	dg.exp=g++.dg/[0-9A-Za-bd-su-z]* \
-	dg.exp=g++.dg/[ct]* \
-	dg.exp=c-c++-common/*,dg-torture.exp
-
+check_g++_parallelize = 10000
 #\f
 # Install hooks:
 # cc1plus is installed elsewhere as part of $(COMPILERS).
--- gcc/testsuite/g++.dg/guality/guality.exp.jj	2014-07-04 10:20:37.106214208 +0200
+++ gcc/testsuite/g++.dg/guality/guality.exp	2014-09-12 15:26:03.213193139 +0200
@@ -14,6 +14,11 @@ if { [istarget "powerpc-ibm-aix*"] } {
 }
 
 proc check_guality {args} {
+    # Don't count check_guality as PASS, or FAIL etc., that would make
+    # the total PASS count dependent on how many parallel runtest invocations
+    # ran guality.exp.  So save the counts first and restore them afterwards.
+    global test_counts
+    array set saved_test_counts [array get test_counts]
     set result [eval check_compile guality_check executable $args "-g -O0"]
     set lines [lindex $result 0]
     set output [lindex $result 1]
@@ -23,6 +28,7 @@ proc check_guality {args} {
       set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
     }
     remote_file build delete $output
+    array get test_counts [array get saved_test_counts]
     return $ret
 }
 
--- gcc/testsuite/g++.dg/compat/struct-layout-1.exp	2014-09-12 11:25:50.294338271 +0200
+++ gcc/testsuite/g++.dg/compat/struct-layout-1.exp	2014-09-12 17:10:02.945849340 +0200
@@ -95,6 +95,15 @@
 
 g++_init
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p struct_layout_1] {
+    return
+}
+gcc_parallel_test_enable 0
+
 # Save variables for the C++ compiler under test, which each test will
 # change a couple of times.  This must be done after calling g++-init.
 set save_gxx_under_test $GXX_UNDER_TEST
@@ -170,0 +180,2 @@
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/g++.dg/plugin/plugin.exp	2014-09-12 11:25:50.294338271 +0200
+++ gcc/testsuite/g++.dg/plugin/plugin.exp	2014-09-12 17:11:56.292298712 +0200
@@ -44,6 +44,15 @@
 # Load support procs.
 load_lib plugin-support.exp
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p plugin] {
+    return
+}
+gcc_parallel_test_enable 0
+
 # Specify the plugin source file and the associated test files in a list.
 # plugin_test_list={ {plugin1 test1 test2 ...} {plugin2 test1 ...} ... }
 set plugin_test_list [list \
@@ -70,0 +80,2 @@
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.dg/guality/guality.exp.jj	2014-07-04 10:20:36.000000000 +0200
+++ gcc/testsuite/gcc.dg/guality/guality.exp	2014-09-12 15:25:41.926294560 +0200
@@ -14,6 +14,11 @@ if { [istarget "powerpc-ibm-aix*"] } {
 }
 
 proc check_guality {args} {
+    # Don't count check_guality as PASS, or FAIL etc., that would make
+    # the total PASS count dependent on how many parallel runtest invocations
+    # ran guality.exp.  So save the counts first and restore them afterwards.
+    global test_counts
+    array set saved_test_counts [array get test_counts]
     set result [eval check_compile guality_check executable $args "-g -O0"]
     set lines [lindex $result 0]
     set output [lindex $result 1]
@@ -23,6 +28,7 @@ proc check_guality {args} {
       set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
     }
     remote_file build delete $output
+    array set test_counts [array get saved_test_counts]
     return $ret
 }
 
--- gcc/testsuite/gcc.dg/compat/struct-layout-1.exp	2014-09-12 11:25:50.304338220 +0200
+++ gcc/testsuite/gcc.dg/compat/struct-layout-1.exp	2014-09-12 17:08:38.185257790 +0200
@@ -53,6 +53,15 @@
 
 gcc_init
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p struct_layout_1] {
+    return
+}
+gcc_parallel_test_enable 0
+
 # Save variables for the C compiler under test, which each test will
 # change a couple of times.  This must be done after calling gcc-init.
 set compat_save_gcc_under_test $GCC_UNDER_TEST
@@ -130,0 +140,2 @@
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.dg/plugin/plugin.exp	2014-09-12 11:25:50.304338220 +0200
+++ gcc/testsuite/gcc.dg/plugin/plugin.exp	2014-09-12 17:09:19.901057981 +0200
@@ -44,6 +44,15 @@
 # Load support procs.
 load_lib plugin-support.exp
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p plugin] {
+    return
+}
+gcc_parallel_test_enable 0
+
 # Specify the plugin source file and the associated test files in a list.
 # plugin_test_list={ {plugin1 test1 test2 ...} {plugin2 test1 ...} ... }
 set plugin_test_list [list \
@@ -82,0 +92,2 @@
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/go.test/go-test.exp.jj	2014-08-01 09:23:38.000000000 +0200
+++ gcc/testsuite/go.test/go-test.exp	2014-09-12 11:39:15.712425307 +0200
@@ -489,6 +489,12 @@ proc go-gc-tests { } {
 
 	close $fd
 
+	# runtest_file_p is already run above, and the code below can run
+	# runtest_file_p again, make sure everything for this test is
+	# performed if the above runtest_file_p decided this runtest
+	# instance should execute the test
+	gcc_parallel_test_enable 0
+
 	set go_compile_args ""
 	set go_execute_args ""
 	if { [regexp "// run (\[^|&>2\].*)\$" $test_line match progargs] \
@@ -1164,6 +1170,7 @@ proc go-gc-tests { } {
 	set go_compile_args ""
 	set go_execute_args ""
         set TORTURE_OPTIONS [list { -O2 -g }]
+	gcc_parallel_test_enable 1
     }
 
     set dg-do-what-default ${saved-dg-do-what-default}
--- gcc/testsuite/lib/gcc-defs.exp.jj	2014-09-11 20:35:28.016752762 +0200
+++ gcc/testsuite/lib/gcc-defs.exp	2014-09-12 13:03:18.381369449 +0200
@@ -188,6 +188,77 @@ if { [info procs runtest_file_p] == "" }
     }
 }
 
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    global gcc_runtest_parallelize_counter
+    global gcc_runtest_parallelize_counter_minor
+    global gcc_runtest_parallelize_enable
+    global gcc_runtest_parallelize_dir
+    global gcc_runtest_parallelize_last
+
+    set gcc_runtest_parallelize_counter 0
+    set gcc_runtest_parallelize_counter_minor 0
+    set gcc_runtest_parallelize_enable 1
+    set gcc_runtest_parallelize_dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+    set gcc_runtest_parallelize_last 0
+
+    proc gcc_parallel_test_run_p { testcase } {
+	global gcc_runtest_parallelize_counter
+	global gcc_runtest_parallelize_counter_minor
+	global gcc_runtest_parallelize_enable
+	global gcc_runtest_parallelize_dir
+	global gcc_runtest_parallelize_last
+
+	if { $gcc_runtest_parallelize_enable == 0 } {
+	    return 1
+	}
+
+	# Only test the filesystem every 10th iteration
+	incr gcc_runtest_parallelize_counter_minor
+	if { $gcc_runtest_parallelize_counter_minor == 10 } {
+	    set gcc_runtest_parallelize_counter_minor 0
+	}
+	if { $gcc_runtest_parallelize_counter_minor != 1 } {
+	    return $gcc_runtest_parallelize_last
+	}
+
+	set path $gcc_runtest_parallelize_dir/$gcc_runtest_parallelize_counter
+	incr gcc_runtest_parallelize_counter
+
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    set gcc_runtest_parallelize_last 1
+	    return 1
+	}
+	set gcc_runtest_parallelize_last 0
+	return 0
+    }
+
+    proc gcc_parallel_test_enable { val } {
+	global gcc_runtest_parallelize_enable
+	set gcc_runtest_parallelize_enable $val
+    }
+
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    proc runtest_file_p { runtests testcase } {
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+	return [gcc_parallel_test_run_p $testcase]
+    }
+
+} else {
+
+    proc gcc_parallel_test_run_p { testcase } {
+	return 1
+    }
+
+    proc gcc_parallel_test_enable { val } {
+    }
+
+}
+
 # Like dg-options, but adds to the default options rather than replacing them.
 
 proc dg-additional-options { args } {
--- gcc/testsuite/objc.dg/gnu-encoding/gnu-encoding.exp.jj	2014-09-11 20:35:28.329751198 +0200
+++ gcc/testsuite/objc.dg/gnu-encoding/gnu-encoding.exp	2014-09-12 11:25:50.318338150 +0200
@@ -28,6 +28,13 @@ if ![info exists DEFAULT_CFLAGS] then {
 # Initialize `dg'.
 dg-init
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if [gcc_parallel_test_run_p gnu_encoding] {
+gcc_parallel_test_enable 0
+
 #
 # gnu-encoding tests
 #
@@ -67,10 +74,8 @@ if { $status == 0 } then {
     warning "Could not compile objc.dg/gnu-encoding/struct-layout-encoding-1 generator"
 }
 
-
-
-
-
+gcc_parallel_test_enable 1
+}
 
 # All done.
 dg-finish
--- gcc/objc/Make-lang.in.jj	2014-09-11 20:35:28.499750358 +0200
+++ gcc/objc/Make-lang.in	2014-09-12 11:25:50.318338150 +0200
@@ -96,7 +96,7 @@ lang_checks += check-objc
 # The following allows you to do 'make check-objc -j2'.  The
 # execute.exp tests will be run in parallel with all the other ones.
 lang_checks_parallelized += check-objc
-check_objc_parallelize = gnu-encoding.exp execute.exp exceptions.exp
+check_objc_parallelize = 6
 
 #\f
 # Install hooks:
--- gcc/testsuite/gcc.misc-tests/matrix1.exp.jj	2014-01-03 11:40:43.400398937 +0100
+++ gcc/testsuite/gcc.misc-tests/matrix1.exp	2014-09-12 17:15:15.918325624 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p matrix1] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase matrix1.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/dhry.exp.jj	2014-01-03 11:40:43.404398913 +0100
+++ gcc/testsuite/gcc.misc-tests/dhry.exp	2014-09-12 17:14:56.373426995 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p dhry] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase dhry.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/acker1.exp.jj	2014-01-03 11:40:43.404398913 +0100
+++ gcc/testsuite/gcc.misc-tests/acker1.exp	2014-09-12 17:14:51.335450103 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p acker1] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase acker1.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/linkage.exp.jj	2014-01-03 11:40:43.406398901 +0100
+++ gcc/testsuite/gcc.misc-tests/linkage.exp	2014-09-12 17:15:09.144367905 +0200
@@ -18,6 +18,17 @@
 # was written by Rob Savoye. (rob@cygnus.com)
 # All the other tests driven by that file have since been moved elsewhere.
 
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p linkage] {
+    return
+}
+gcc_parallel_test_enable 0
+
 if { [isnative] && ![is_remote host] } then {
     set lines [gcc_target_compile "$srcdir/$subdir/linkage-x.c" "linkage-x.o" object {additional_flags="-w"}]
     if ![string match "" $lines] then {
@@ -117,3 +128,5 @@ if { [isnative] && ![is_remote host] } t
 	file delete "linkage-x.o"
     }
 }
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/mg.exp.jj	2014-01-03 11:40:43.403398920 +0100
+++ gcc/testsuite/gcc.misc-tests/mg.exp	2014-09-12 17:15:25.406276624 +0200
@@ -17,8 +17,20 @@
 # Test the -MG flag.
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p mg] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions none-of-the-above
 set compiler_output "mg.o ?: .*mg.c \[ \\\\\n\]*nonexist.h"
 postbase mg.c "" "" "-MM -MG"
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/mg-2.exp.jj	2014-01-03 11:40:43.402398926 +0100
+++ gcc/testsuite/gcc.misc-tests/mg-2.exp	2014-09-12 17:15:20.466298272 +0200
@@ -17,8 +17,20 @@
 # Test the -MG flag with a system header file.
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p mg-2] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions none-of-the-above
 set compiler_output "mg-2.o ?: .*mg-2.c \[ \\\\\n\]*nonexist.h"
 postbase mg-2.c "" "" "-MM -MG"
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/sort2.exp.jj	2014-01-03 11:40:43.399398942 +0100
+++ gcc/testsuite/gcc.misc-tests/sort2.exp	2014-09-12 17:15:48.947116261 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p sort2] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase sort2.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/sieve.exp.jj	2014-01-03 11:40:43.407398896 +0100
+++ gcc/testsuite/gcc.misc-tests/sieve.exp	2014-09-12 17:15:43.418154804 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p sieve] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase sieve.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/options.exp.jj	2014-01-03 11:40:43.401398931 +0100
+++ gcc/testsuite/gcc.misc-tests/options.exp	2014-09-12 17:15:30.948230381 +0200
@@ -19,6 +19,17 @@
 # match the patterns COMPILER_PATTERN, AS_PATTERN and LD_PATTERN,
 # respectively.
 
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p options] {
+    return
+}
+gcc_parallel_test_enable 0
+
 proc check_for_all_options {language gcc_options compiler_pattern as_pattern ld_pattern} {
     set filename test-[pid]
     set fd [open $filename.c w]
@@ -52,3 +63,5 @@ proc check_for_all_options {language gcc
 }
 
 check_for_all_options c {--coverage} {-fprofile-arcs -ftest-coverage} {} {-lgcov}
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/help.exp.jj	2014-01-03 11:40:43.400398937 +0100
+++ gcc/testsuite/gcc.misc-tests/help.exp	2014-09-12 17:15:02.997394449 +0200
@@ -18,6 +18,16 @@
 # documented in --help, and that the various --help* options work.
 
 load_lib options.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p help] {
+    return
+}
+gcc_parallel_test_enable 0
 
 # Document --version.  Ideally, there should be no undocumented switches
 # in --help.
@@ -78,3 +88,5 @@ check_for_options c "--help=joined,^sepa
 check_for_options c "--help=joined,undocumented" "" "" ""
 # Listing only excludes gives empty results.
 check_for_options c "--help=^joined,^separate" "" "" ""
+
+gcc_parallel_test_enable 1
--- libstdc++-v3/testsuite/Makefile.am.jj	2014-01-03 11:42:07.000000000 +0100
+++ libstdc++-v3/testsuite/Makefile.am	2014-09-12 16:20:35.687366456 +0200
@@ -101,7 +101,16 @@ new-abi-baseline:
 	@test ! -f $*/site.exp || mv $*/site.exp $*/site.bak
 	@mv $*/site.exp.tmp $*/site.exp
 
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers1)))
+check_p_numbers3:=$(patsubst %,0%,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers3)))
+check_p_numbers5:=$(patsubst %,0%,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
+check_p_subdirs=$(wordlist 1,$(or $(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers))
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,$(check_p_subdirs))
 $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
@@ -110,10 +119,15 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	RANLIB="$(RANLIB)"; export RANLIB; \
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  rm -rf normal-parallel || true; \
+	  mkdir normal-parallel; \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
-	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
-	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	  rm -rf normal-parallel || true; \
+	  for idx in $(check_p_subdirs); do \
+	    if [ -d normal$$idx ]; then \
+	      mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
+	      mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	    fi; \
 	  done; \
 	  mv -f libstdc++.sum libstdc++.sum.sep; \
 	  mv -f libstdc++.log libstdc++.log.sep; \
@@ -128,47 +142,30 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	runtest=$(RUNTEST); \
 	if [ -z "$$runtest" ]; then runtest=runtest; fi; \
 	tool=libstdc++; \
-	dirs=; \
-	case "$*" in \
-	  normal0) \
-	    if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	      $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-			$(RUNTESTFLAGS) abi.exp prettyprinters.exp; \
-	    else echo "WARNING: could not find \`runtest'" 1>&2; :;\
-	    fi; \
-	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
-	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
-	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
-	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
-	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
-	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
-	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
-	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
-	  normal8) \
-	    dirs="`cd $$srcdir; echo 27_*/*`";; \
-	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
-	  normal10) \
-	    dirs="`cd $$srcdir; echo t*/*`";; \
-	esac; \
-	if [ -n "$*" ]; then cd "$*"; fi; \
-	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	  if [ -n "$$dirs" ]; then \
+	runtestargs=; \
+	if [ "$*" = normal1 ]; then \
+	  if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
 	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-		    $(RUNTESTFLAGS) \
-		    "conformance.exp=`echo $$dirs | sed 's/ /* /g;s/$$/*/'`"; \
+		      $(RUNTESTFLAGS) abi.exp prettyprinters.exp; \
 	  else \
-	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-		    $(RUNTESTFLAGS); \
+	    echo "WARNING: could not find \`runtest'" 1>&2; :; \
+	  fi; \
+	fi; \
+	if [ -n "$*" ]; then \
+	  if [ -f normal-parallel/finished ]; then rm -rf "$*"; exit 0; fi; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/normal-parallel; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR; \
+	  cd "$*"; \
+	  runtestargs=conformance.exp; \
+	fi; \
+	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
+	  $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
+		    $(RUNTESTFLAGS) $$runtestargs; \
+	  if [ -n "$*" ]; then \
+	    touch $$GCC_RUNTEST_PARALLELIZE_DIR/finished; \
 	  fi; \
-	else echo "WARNING: could not find \`runtest'" 1>&2; :;\
+	else \
+	  echo "WARNING: could not find \`runtest'" 1>&2; :;\
 	fi
 
 check-am:
--- libstdc++-v3/testsuite/Makefile.in.jj	2013-11-22 21:38:38.653239025 +0100
+++ libstdc++-v3/testsuite/Makefile.in	2014-09-12 16:21:04.201235261 +0200
@@ -301,7 +301,16 @@ lists_of_files = \
 
 extract_symvers = $(glibcxx_builddir)/scripts/extract_symvers
 baseline_subdir := $(shell $(CXX) $(baseline_subdir_switch))
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers1)))
+check_p_numbers3:=$(patsubst %,0%,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers3)))
+check_p_numbers5:=$(patsubst %,0%,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(patsubst %,$(i)%,$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
+check_p_subdirs=$(wordlist 1,$(or $(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers))
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,$(check_p_subdirs))
 
 # Runs the testsuite, but in compile only mode.
 # Can be used to test sources with non-GNU FE's at various warning
@@ -561,10 +570,15 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	RANLIB="$(RANLIB)"; export RANLIB; \
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  rm -rf normal-parallel || true; \
+	  mkdir normal-parallel; \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
-	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
-	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	  rm -rf normal-parallel || true; \
+	  for idx in $(check_p_subdirs); do \
+	    if [ -d normal$$idx ]; then \
+	      mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
+	      mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	    fi; \
 	  done; \
 	  mv -f libstdc++.sum libstdc++.sum.sep; \
 	  mv -f libstdc++.log libstdc++.log.sep; \
@@ -579,47 +593,30 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	runtest=$(RUNTEST); \
 	if [ -z "$$runtest" ]; then runtest=runtest; fi; \
 	tool=libstdc++; \
-	dirs=; \
-	case "$*" in \
-	  normal0) \
-	    if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	      $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-			$(RUNTESTFLAGS) abi.exp prettyprinters.exp; \
-	    else echo "WARNING: could not find \`runtest'" 1>&2; :;\
-	    fi; \
-	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
-	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
-	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
-	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
-	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
-	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
-	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
-	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
-	  normal8) \
-	    dirs="`cd $$srcdir; echo 27_*/*`";; \
-	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
-	  normal10) \
-	    dirs="`cd $$srcdir; echo t*/*`";; \
-	esac; \
-	if [ -n "$*" ]; then cd "$*"; fi; \
-	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	  if [ -n "$$dirs" ]; then \
+	runtestargs=; \
+	if [ "$*" = normal1 ]; then \
+	  if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
 	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-		    $(RUNTESTFLAGS) \
-		    "conformance.exp=`echo $$dirs | sed 's/ /* /g;s/$$/*/'`"; \
+		      $(RUNTESTFLAGS) abi.exp prettyprinters.exp; \
 	  else \
-	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-		    $(RUNTESTFLAGS); \
+	    echo "WARNING: could not find \`runtest'" 1>&2; :; \
+	  fi; \
+	fi; \
+	if [ -n "$*" ]; then \
+	  if [ -f normal-parallel/finished ]; then rm -rf "$*"; exit 0; fi; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/normal-parallel; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR; \
+	  cd "$*"; \
+	  runtestargs=conformance.exp; \
+	fi; \
+	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
+	  $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
+		    $(RUNTESTFLAGS) $$runtestargs; \
+	  if [ -n "$*" ]; then \
+	    touch $$GCC_RUNTEST_PARALLELIZE_DIR/finished; \
 	  fi; \
-	else echo "WARNING: could not find \`runtest'" 1>&2; :;\
+	else \
+	  echo "WARNING: could not find \`runtest'" 1>&2; :;\
 	fi
 
 check-am:
--- libstdc++-v3/testsuite/lib/libstdc++.exp.jj	2014-07-14 09:30:36.000000000 +0200
+++ libstdc++-v3/testsuite/lib/libstdc++.exp	2014-09-12 16:06:40.415394092 +0200
@@ -1823,3 +1823,74 @@ proc check_v3_target_little_endian { } {
 }
 
 set additional_prunes ""
+
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    global gcc_runtest_parallelize_counter
+    global gcc_runtest_parallelize_counter_minor
+    global gcc_runtest_parallelize_enable
+    global gcc_runtest_parallelize_dir
+    global gcc_runtest_parallelize_last
+
+    set gcc_runtest_parallelize_counter 0
+    set gcc_runtest_parallelize_counter_minor 0
+    set gcc_runtest_parallelize_enable 1
+    set gcc_runtest_parallelize_dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+    set gcc_runtest_parallelize_last 0
+
+    proc gcc_parallel_test_run_p { testcase } {
+	global gcc_runtest_parallelize_counter
+	global gcc_runtest_parallelize_counter_minor
+	global gcc_runtest_parallelize_enable
+	global gcc_runtest_parallelize_dir
+	global gcc_runtest_parallelize_last
+
+	if { $gcc_runtest_parallelize_enable == 0 } {
+	    return 1
+	}
+
+	# Only test the filesystem every 10th iteration
+	incr gcc_runtest_parallelize_counter_minor
+	if { $gcc_runtest_parallelize_counter_minor == 10 } {
+	    set gcc_runtest_parallelize_counter_minor 0
+	}
+	if { $gcc_runtest_parallelize_counter_minor != 1 } {
+	    return $gcc_runtest_parallelize_last
+	}
+
+	set path $gcc_runtest_parallelize_dir/$gcc_runtest_parallelize_counter
+	incr gcc_runtest_parallelize_counter
+
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    set gcc_runtest_parallelize_last 1
+	    return 1
+	}
+	set gcc_runtest_parallelize_last 0
+	return 0
+    }
+
+    proc gcc_parallel_test_enable { val } {
+	global gcc_runtest_parallelize_enable
+	set gcc_runtest_parallelize_enable $val
+    }
+
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    proc runtest_file_p { runtests testcase } {
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+	return [gcc_parallel_test_run_p $testcase]
+    }
+
+} else {
+
+    proc gcc_parallel_test_run_p { testcase } {
+	return 1
+    }
+
+    proc gcc_parallel_test_enable { val } {
+    }
+
+}

	Jakub

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Sep 12, 2014 at 06:32:41PM +0200, Jakub Jelinek wrote:
> Regtested on x86_64-linux, ok for trunk?

Oh, forgot to say, PR56408 isn't fixed by this patch, but given the
higher granularity (10 tests instead of 1) we don't happen to trigger it
right now.

	Jakub

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

>> Regtested on x86_64-linux, ok for trunk?
>
>Oh, forgot to say, PR56408 isn't fixed by this patch, but given the
>higher granularity (10 tests instead of 1) we don't happen to trigger it
>right now.

which means that any commit to that dir could trigger it, right ?

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Sep 12, 2014 at 04:36:05PM +0000, VandeVondele  Joost wrote:
> 
> 
> >> Regtested on x86_64-linux, ok for trunk?
> >
> >Oh, forgot to say, PR56408 isn't fixed by this patch, but given the
> >higher granularity (10 tests instead of 1) we don't happen to trigger it
> >right now.
> 
> which means that any commit to that dir could trigger it, right ?

Sure, if you are unlucky.

I mean, the bug needs to be fixed, just IMHO it doesn't need to be fixed
immediately or as precondition of this patch, the bug existed there for
a year or two, though with the patch might be more likely to be triggered.

	Jakub

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 12, 2014, at 9:32 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> Here is my latest version of the patch.
> 
> With this patch I get identical test_summary output on make -k check
> (completely serial testing) and make -j48 -k check from toplevel directory.
> 
> Major changes since last version:
> 1) I've changed the granularity, now it does O_EXCL|O_CREAT attempt
>   only every 10th runtest_file_p invocation

So, I’d love to see the numbers for 5 and 20 to double check that 10 is the right number to pick.  This sort of refinement is trivial post checkin.

> 3) various other *.exp fails didn't use runtest_file_p, especially the
>   gcc.misc-tests/ ones, tweaked those like struct-layout-1.exp or
>   plugin.exp so that only the first runtest instance to encounter those
>   runs all of the *.exp file serially

> Regtested on x86_64-linux, ok for trunk?

Ok.  Please be around after you apply it to try and sort out any major fallout.

If someone can check their target post checkin (or help out pre-checkin) and report back, that would be nice.  Times before and post checkin with core count -j setting would be nice.

I wonder if the libstdc++ problems can be sorted out merely by finding a way to sort them so the expensive ones come early (regexp -> 0regexp for example).  Or, instead of sorting them by name, sort them by some other key (md5 per line).  The idea then would be that the chance of all regexp tests being in one group is 0.

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

> So, I’d love to see the numbers for 5 and 20 to double check that 10 is the right number to pick.  This sort of refinement is trivial post checkin.

So, some timings with the patch, I think this is great. 

Doing the testing you suggest, changing the variable doesn't influence things much (at least for Fortran, and  on this system).

make -j32 -k
check-fortran
real    3m27.875s -> gcc_runtest_parallelize_counter_minor == 02 (several testsuite errors: binding_label_tests_10_main.f03, binding_label_tests_11_main.f03, class_45b.f03, class_4b.f03, class_4c.f03, coarray_29_2.f90, test_common_binding_labels_3_main.f03)
real    3m26.234s -> gcc_runtest_parallelize_counter_minor == 05 (one additional testsuite error: whole_file_31.f90)
real    3m36.405s -> gcc_runtest_parallelize_counter_minor == 10
real    3m38.736s -> gcc_runtest_parallelize_counter_minor == 20
check-c
real    8m26.935s
check-c++
real    7m4.165s
check
real   17m45.185s

Re: [PATCH] gcc parallel make check

[Bernhard Reutner-Fischer]

On 12 September 2014 19:46:33 Mike Stump <mikestump@comcast.net> wrote:

> On Sep 12, 2014, at 9:32 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> > Here is my latest version of the patch.
> >
> > With this patch I get identical test_summary output on make -k check
> > (completely serial testing) and make -j48 -k check from toplevel directory.
> >
> > Major changes since last version:
> > 1) I've changed the granularity, now it does O_EXCL|O_CREAT attempt
> >   only every 10th runtest_file_p invocation
>
> So, I’d love to see the numbers for 5 and 20 to double check that 10 is the 
> right number to pick.  This sort of refinement is trivial post checkin.
>
> > 3) various other *.exp fails didn't use runtest_file_p, especially the
> >   gcc.misc-tests/ ones, tweaked those like struct-layout-1.exp or
> >   plugin.exp so that only the first runtest instance to encounter those
> >   runs all of the *.exp file serially
>
> > Regtested on x86_64-linux, ok for trunk?
>
> Ok.  Please be around after you apply it to try and sort out any major fallout.

Usage of $(or) and $(and) will bump GNU make prerequisite version from our 
current 3.80 to at least 3.82 (IIRC).

PS: for the numbers I had used addsuffix  rather than patsubst in the hopes 
that it avoids lots of regexp calls. Very minor not though.

Cheers,
>
> If someone can check their target post checkin (or help out pre-checkin) 
> and report back, that would be nice.  Times before and post checkin with 
> core count -j setting would be nice.
>
> I wonder if the libstdc++ problems can be sorted out merely by finding a 
> way to sort them so the expensive ones come early (regexp -> 0regexp for 
> example).  Or, instead of sorting them by name, sort them by some other key 
> (md5 per line).  The idea then would be that the chance of all regexp tests 
> being in one group is 0.



Sent with AquaMail for Android
http://www.aqua-mail.com

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 12, 2014, at 9:32 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> Here is my latest version of the patch.

I did a timing test:

Before:

real	0m57.198s
user	1m24.736s
sys	0m19.816s

after:

real	0m28.224s
user	1m27.823s
sys	0m22.374s

This is a -j70 run on a 64 core power7 of check-objc, I picked an obscure test case that I had no reason to believe was other than ignored and certainly not engineered for and kinda small to ensure the overhead would penalize it…  50.66% faster.  There is still room for improvement:

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 7  0      0 99046848 8515072 16748672    0    0     0     0    1    1  0  0 100  0  0
 7  0      0 99050432 8515072 16748736    0    0     0     0 7501 9022 13  3 84  0  0
 7  0      0 99029376 8515072 16749248    0    0     0     0 7320 8777 10  2 88  0  0
 7  0      0 99070656 8515072 16749440    0    0     0  1524 7162 8156  9  2 88  1  0
 7  0      0 99034560 8515072 16749824    0    0     0     0 8096 10363  7  2 91  0  0
 7  0      0 99030080 8515072 16750720    0    0     0     0 8798 11673  8  3 90  0  0
 9  0      0 99037376 8515072 16750080    0    0     0     0 9151 12598  9  3 87  0  0
 7  0      0 99024128 8515136 16750656    0    0     0     0 9078 13168  7  3 90  0  0
10  0      0 99034496 8515136 16751488    0    0     0  1800 8633 11675  8  3 88  1  0
 8  0      0 98986304 8515136 16751296    0    0     0     0 10159 14553  7  3 90  0  0
 7  0      0 99010112 8515520 16765824    0    0     0     0 8814 12036 10  3 87  0  0
 4  0      0 99014016 8515648 16773568    0    0     0     0 8091 10445  8  3 90  0  0
 4  0      0 99064832 8515712 16773120    0    0     0     0 5416 5071  9  2 89  0  0
 3  0      0 99118976 8515712 16773184    0    0     0 12716 4743 3533  4  1 92  2  0
 3  0      0 99077504 8515840 16773248    0    0     0     0 4525 3988  3  1 96  0  0
 2  0      0 99121152 8515840 16773824    0    0     0     0 4687 3757  3  1 97  0  0
 2  0      0 99117056 8515840 16773632    0    0     0     0 4334 3156  3  1 96  0  0
 2  0      0 99105728 8515840 16774336    0    0     0     0 4355 3246  3  1 96  0  0
 3  0      0 99069120 8515904 16773632    0    0     0   648 4902 4037  2  1 97  0  0
 1  0      0 99153664 8515968 16774592    0    0     0     0 3776 2711  2  1 97  0  0
 1  0      0 99151232 8515968 16774400    0    0     0     0  877  205  4  0 96  0  0
 1  0      0 99151424 8516032 16774528    0    0     0   236  774  466  2  0 97  0  0
 2  0      0 99148032 8516032 16774656    0    0     0     0  853  350  2  0 98  0  0
 2  0      0 99146176 8516032 16774656    0    0     0  1208 1630 1363  1  0 99  0  0
 1  0      0 99156032 8516352 16777152    0    0     0     0 1919 2104  1  0 99  0  0
 0  0      0 99189376 8516416 16776512    0    0     0     0 1181  799  2  0 98  0  0
 0  0      0 99189312 8516416 16776512    0    0     0     0  118   18  0  0 100  0  0
 0  0      0 99189312 8516416 16776512    0    0     0     0   90   18  0  0 100  0  0
 0  0      0 99187968 8516416 16776512    0    0     0  5468  196   42  0  0 100  0  0
 0  0      0 99187968 8516416 16776512    0    0     0     0   92   24  0  0 100  0  0
 0  0      0 99188032 8516416 16776512    0    0     0     0  146   37  0  0 100  0  0
 0  0      0 99188160 8516416 16776512    0    0     0   128   91   36  0  0 100  0  0
 1  0      0 99188160 8516416 16776512    0    0     0     0   74   16  0  0 100  0  0
 0  0      0 99188160 8516416 16776512    0    0     0     0   72   20  0  0 100  0  0
 0  0      0 99188224 8516416 16776512    0    0     0     0   76   22  0  0 100  0  0
 0  0      0 99188224 8516416 16776512    0    0     0     0  118   29  0  0 100  0  0

which averages to 95% idle.  I changed:

check_objc_parallelize = 6

to 

check_objc_parallelize = 70

to try and get it to go faster:

real	0m21.252s
user	3m21.035s
sys	1m9.937s

:-(  7 seconds (24.6%) faster, but consumes 146% more resources to see the benefit.

with the filesystem update to 2 (instead of 10):

real	0m22.478s
user	4m38.564s
sys	1m25.293s

and filesystem update 5:

real	0m21.665s
user	3m51.615s
sys	1m16.005s

and filesystem update 20:

real	0m22.681s
user	3m2.746s
sys	1m5.576s

a -j1 filesystem update 20 for comparison:

real	1m48.127s
user	1m17.953s
sys	0m17.191s

a -j1 check_objc_parallelize 6 filesystem update 10 for comparison:

real	1m47.552s
user	1m17.410s
sys	0m16.909s

a -j70 check_objc_parallelize 10000 filesystem update 10 for comparison:

real	0m21.292s
user	3m17.368s
sys	1m10.106s

a -j70 check_objc_parallelize 10000 filesystem update 2 for comparison:

real	0m21.976s
user	4m37.600s
sys	1m26.598s

a -j70 check_objc_parallelize 10000 filesystem update 200 for comparison:

real	1m12.319s
user	2m49.975s
sys	1m4.537s

a -j70 check_objc_parallelize 12 filesystem update 10 for comparison:

real	0m23.176s
user	1m33.100s
sys	0m25.722s

=======================================================

Switching over to check-c…

-j70 before, 94.4% idle:

real	22m38.331s
user	67m11.810s
sys	13m40.974s

-j70 after (71.28% idle):

real	10m41.448s
user	160m24.871s
sys	36m5.220s

143% more resource intensive to get a 52.8% faster check.  I still see a long tail on the test suite run (30 second per line):

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 96997696 8707392 18756352    0    0     0     0    0    0  0  0 100  0  0
70  6      0 95642688 8709824 18719232    0    0     0  1231 23366 47068 54 23 18  4  0
66 10      0 95591872 8711744 18734976    0    0     0  3131 19437 37168 69 19  6  6  0
66  9      0 94251520 8716352 18780096    0    0     0  3304 18211 34222 70 18  7  6  0
60 16      0 94398400 8732288 18857152    0    0     0  2654 15808 29888 74 16  5  5  0
60 14      0 95059008 8749056 18973888    0    0     0  5678 17521 33177 72 17  6  5  0
60 12      0 94594880 8766656 18981376    0    0     0  2874 15686 28166 72 16  6  6  0
12  2      0 95515520 8773184 18997760    0    0     0  2109 14987 23655 48  9 39  4  0
 6  1      0 96211264 8774144 19010560    0    0     0  2111 5049 4993 14  1 85  0  0
 3  0      0 96441408 8774336 19016640    0    0     0   529 1870  980  7  0 93  0  0
 2  0      0 96493248 8774336 19016128    0    0     0   359  462   79  3  0 97  0  0
 2  0      0 96540992 8774400 19016000    0    0     0   417  458   89  3  0 97  0  0
 1  0      0 96564736 8774400 19012864    0    0     0   277  482  164  2  0 98  0  0
 1  0      0 96566080 8774400 19012928    0    0     0    16  194   31  2  0 98  0  0
 1  0      0 96574208 8774400 19012928    0    0     0     9  185   27  2  0 98  0  0
 1  0      0 96576192 8774400 19012672    0    0     0     9  197   32  2  0 98  0  0
 1  0      0 96584384 8774400 19012736    0    0     0     9  185   26  2  0 98  0  0
 1  0      0 96588608 8774400 19012480    0    0     0     9  187   27  2  0 98  0  0
 1  0      0 96583872 8774400 19012672    0    0     0    18  183   27  2  0 98  0  0
 1  0      0 96579072 8774528 19017472    0    0     0    32  230   55  2  0 98  0  0
 1  0      0 96603264 8774592 19016832    0    0     0    92  373  219  2  0 98  0  0
 1  0      0 96606528 8774592 19017984    0    0     0   111  357  241  2  0 98  0  0

About 3 minutes of using the machine, then 7 minutes of mostly idle.  The worse offenders are:

gcc.dg/atomic/atomic.exp completed in 522 seconds
gcc.dg/compat/struct-layout-1.exp completed in 253 seconds
gcc.c-torture/compile/compile.exp completed in 252 seconds
gcc.c-torture/compile/compile.exp completed in 252 seconds
gcc.c-torture/execute/builtins/builtins.exp completed in 193 seconds
gcc.c-torture/execute/builtins/builtins.exp completed in 177 seconds
gcc.dg/atomic/atomic.exp completed in 141 seconds
gcc.c-torture/execute/execute.exp completed in 134 seconds
gcc.c-torture/compile/compile.exp completed in 128 seconds
gcc.dg/guality/guality.exp completed in 112 seconds
gcc.dg/ubsan/ubsan.exp completed in 111 seconds
gcc.dg/torture/dg-torture.exp completed in 109 seconds
gcc.dg/guality/guality.exp completed in 108 seconds
gcc.dg/dg.exp completed in 103 seconds

(all that are over 100 seconds).

curious, when I run atomic.exp=stdatom\*.c:

  gcc.dg/atomic/atomic.exp completed in 30 seconds.

atomic.exp=c\*.c takes 522 seconds with 3, 2, 5 and 4 being the worst offenders.

I worry a little about the scaling overhead of the scheme.  The bin packing method I was thinking of would just use a larger number of bins and then bin pack them into n bins using the actual testing time taken.  Large bins, we’d just split in two.  I kinda expected a -j70 of atomic.exp to use more than 1 core.

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Sep 12, 2014 at 04:42:25PM -0700, Mike Stump wrote:
> curious, when I run atomic.exp=stdatom\*.c:
> 
>   gcc.dg/atomic/atomic.exp completed in 30 seconds.
> 
> atomic.exp=c\*.c takes 522 seconds with 3, 2, 5 and 4 being the worst offenders.

That's the
        @if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \
            && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
i.e. if you specify anything in RUNTESTFLAGS other than --target_board= or
--extra_opts, it is not parallelized.  This was done previously because
parallelization required setting the flags to something different (manually
created *.exp list).  The first [] could perhaps be removed now, if one e.g.
RUNTESTFLAGS=atomic.exp etc. with sufficiently enough tests, parallelization
will be still worth it.  I've been worried about the quick cases where
parallelization is not beneficial, like make check-gcc \
RUNTESTFLAGS=dg.exp=pr60123.c or similar, but one doesn't usually pass -jN
in that case.  So yes, the
[ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ]
can be dropped (not in libstdc++ though, there are abi.exp and
prettyprinters.exp still run serially, though even that could be handled the
struct-layout-1.exp way, of running it by the first instance to encounter
those with small changes in those *.exp files).

	Jakub

Re: [PATCH] gcc parallel make check

[Bernhard Reutner-Fischer]

On 13 September 2014 02:04:51 Jakub Jelinek <jakub@redhat.com> wrote:

> On Fri, Sep 12, 2014 at 04:42:25PM -0700, Mike Stump wrote:
> > curious, when I run atomic.exp=stdatom\*.c:
> >
> >   gcc.dg/atomic/atomic.exp completed in 30 seconds.
> >
> > atomic.exp=c\*.c takes 522 seconds with 3, 2, 5 and 4 being the worst 
> offenders.
>
> That's the
>         @if [ -z "$(filter-out --target_board=%,$(filter-out 
>         --extra_opts%,$(RUNTESTFLAGS)))" ] \
>             && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
> i.e. if you specify anything in RUNTESTFLAGS other than --target_board= or
> --extra_opts, it is not parallelized.  This was done previously because
> parallelization required setting the flags to something different (manually
> created *.exp list).  The first [] could

Yes, this is very inconvenient, especially in the light of -v in the 
runtestflags which should certainly not prohibit parallel execution.
See https://gcc.gnu.org/ml/gcc-patches/2013-11/msg00997.html
for how I would fix that.. (findstring empty instead of filter-out).

TIA,

 perhaps be removed now, if one e.g.
> RUNTESTFLAGS=atomic.exp etc. with sufficiently enough tests, parallelization
> will be still worth it.  I've been worried about the quick cases where
> parallelization is not beneficial, like make check-gcc \
> RUNTESTFLAGS=dg.exp=pr60123.c or similar, but one doesn't usually pass -jN
> in that case.  So yes, the
> [ -z "$(filter-out --target_board=%,$(filter-out 
> --extra_opts%,$(RUNTESTFLAGS)))" ]
> can be dropped (not in libstdc++ though, there are abi.exp and
> prettyprinters.exp still run serially, though even that could be handled the
> struct-layout-1.exp way, of running it by the first instance to encounter
> those with small changes in those *.exp files).
>
> 	Jakub



Sent with AquaMail for Android
http://www.aqua-mail.com

Re: [PATCH] gcc parallel make check

[Jason Merrill]

On 09/12/2014 08:04 PM, Jakub Jelinek wrote:
> I've been worried about the quick cases where
> parallelization is not beneficial, like make check-gcc \
> RUNTESTFLAGS=dg.exp=pr60123.c or similar, but one doesn't usually pass -jN
> in that case.

I have -jN in my $MAKEFLAGS, so I've been running into this with my rgt 
shell function:

rgt ()
{
     ( cd ~/m/$CANON/gcc/gcc;
     make check-c++ ${1:+RUNTESTFLAGS="$*"} )
}

If I say 'rgt dg.exp=var-templ1.C' the actual test results are lost in 
the explosion of shell verbosity.  Could we add some '@'s to more of the 
rules, perhaps?

Jason

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Mon, Sep 22, 2014 at 11:21:14AM -0400, Jason Merrill wrote:
> On 09/12/2014 08:04 PM, Jakub Jelinek wrote:
> >I've been worried about the quick cases where
> >parallelization is not beneficial, like make check-gcc \
> >RUNTESTFLAGS=dg.exp=pr60123.c or similar, but one doesn't usually pass -jN
> >in that case.
> 
> I have -jN in my $MAKEFLAGS, so I've been running into this with my rgt
> shell function:
> 
> rgt ()
> {
>     ( cd ~/m/$CANON/gcc/gcc;
>     make check-c++ ${1:+RUNTESTFLAGS="$*"} )
> }
> 
> If I say 'rgt dg.exp=var-templ1.C' the actual test results are lost in the
> explosion of shell verbosity.  Could we add some '@'s to more of the rules,
> perhaps?

I've been considering that too, but not sure what info people find valuable
and what they don't.

	Jakub

Re: [PATCH] gcc parallel make check

[Jason Merrill]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

On 09/22/2014 11:26 AM, Jakub Jelinek wrote:
> On Mon, Sep 22, 2014 at 11:21:14AM -0400, Jason Merrill wrote:
>> If I say 'rgt dg.exp=var-templ1.C' the actual test results are lost in the
>> explosion of shell verbosity.  Could we add some '@'s to more of the rules,
>> perhaps?
>
> I've been considering that too, but not sure what info people find valuable
> and what they don't.

I don't see much information in the ~128 repetitions of the 
check-parallel rules with different numbers; the actual runtest command 
is the same in all of them.  Adding @ to all of the commands of the 
check-parallel-% rule makes things much better for me:


[-- Attachment #2: check-parallel.patch --]
[-- Type: text/x-patch, Size: 813 bytes --]

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 6f251a5..be4c840 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -3674,10 +3674,10 @@ $(lang_checks_parallelized): check-% : site.exp
 	fi
 
 check-parallel-% : site.exp
-	-test -d plugin || mkdir plugin
-	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
-	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
-	-(rootme=`${PWD_COMMAND}`; export rootme; \
+	-@test -d plugin || mkdir plugin
+	-@test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
+	@test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
+	-@(rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
 	if [ -n "$(check_p_subno)" ] \
 	   && [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] \

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Mon, Sep 22, 2014 at 11:43:35AM -0400, Jason Merrill wrote:
> On 09/22/2014 11:26 AM, Jakub Jelinek wrote:
> >On Mon, Sep 22, 2014 at 11:21:14AM -0400, Jason Merrill wrote:
> >>If I say 'rgt dg.exp=var-templ1.C' the actual test results are lost in the
> >>explosion of shell verbosity.  Could we add some '@'s to more of the rules,
> >>perhaps?
> >
> >I've been considering that too, but not sure what info people find valuable
> >and what they don't.
> 
> I don't see much information in the ~128 repetitions of the check-parallel
> rules with different numbers; the actual runtest command is the same in all
> of them.  Adding @ to all of the commands of the check-parallel-% rule makes
> things much better for me:

LGTM (though, supposedly we want similar change in
libstdc++-v3/testsuite/Makefile.am).
Or, if people would really like to see the commands, we could print them
just once, using e.g.
	-$(if $(check_p_subno),@)(rootme= ...
(then e.g. check-parallel-gcc goal would print the command, but
check-parallel-gcc-1 or check-parallel-gcc-112 would not).

> --- a/gcc/Makefile.in
> +++ b/gcc/Makefile.in
> @@ -3674,10 +3674,10 @@ $(lang_checks_parallelized): check-% : site.exp
>  	fi
>  
>  check-parallel-% : site.exp
> -	-test -d plugin || mkdir plugin
> -	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
> -	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
> -	-(rootme=`${PWD_COMMAND}`; export rootme; \
> +	-@test -d plugin || mkdir plugin
> +	-@test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
> +	@test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
> +	-@(rootme=`${PWD_COMMAND}`; export rootme; \
>  	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
>  	if [ -n "$(check_p_subno)" ] \
>  	   && [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] \


	Jakub

Re: [PATCH] gcc parallel make check

[Jason Merrill]

[-- Attachment #1: Type: text/plain, Size: 423 bytes --]

On 09/22/2014 11:58 AM, Jakub Jelinek wrote:
> LGTM (though, supposedly we want similar change in
> libstdc++-v3/testsuite/Makefile.am).
> Or, if people would really like to see the commands, we could print them
> just once, using e.g.
> 	-$(if $(check_p_subno),@)(rootme= ...
> (then e.g. check-parallel-gcc goal would print the command, but
> check-parallel-gcc-1 or check-parallel-gcc-112 would not).

So, like this?




[-- Attachment #2: check-parallel.patch --]
[-- Type: text/x-patch, Size: 3219 bytes --]

commit c750897381a3f936e27cabd825cfa85ce936a6a9
Author: Jason Merrill <jason@redhat.com>
Date:   Mon Sep 22 11:44:00 2014 -0400

    gcc/
    	* Makefile.in (check-parallel-%): Add @.
    libstdc++-v3/
    	* testsuite/Makefile.am (%/site.exp): Add @.
    	(check-DEJAGNU): Likewise.
    	* testsuite/Makefile.in: Regenerate.

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 6f251a5..97b439a 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -3674,10 +3674,10 @@ $(lang_checks_parallelized): check-% : site.exp
 	fi
 
 check-parallel-% : site.exp
-	-test -d plugin || mkdir plugin
-	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
-	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
-	-(rootme=`${PWD_COMMAND}`; export rootme; \
+	-@test -d plugin || mkdir plugin
+	-@test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
+	@test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
+	-$(if $(check_p_subno),@)(rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
 	if [ -n "$(check_p_subno)" ] \
 	   && [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] \
diff --git a/libstdc++-v3/testsuite/Makefile.am b/libstdc++-v3/testsuite/Makefile.am
index e206aba..b4c9e85 100644
--- a/libstdc++-v3/testsuite/Makefile.am
+++ b/libstdc++-v3/testsuite/Makefile.am
@@ -91,9 +91,9 @@ new-abi-baseline:
 	  ${extract_symvers} ../src/.libs/libstdc++.so $${output})
 
 %/site.exp: site.exp
-	-test -d $* || mkdir $*
+	-@test -d $* || mkdir $*
 	@srcdir=`cd $(srcdir); ${PWD_COMMAND}`;
-	objdir=`${PWD_COMMAND}`/$*; \
+	@objdir=`${PWD_COMMAND}`/$*; \
 	sed -e "s|^set srcdir .*$$|set srcdir $$srcdir|" \
 	    -e "s|^set objdir .*$$|set objdir $$objdir|" \
 	    site.exp > $*/site.exp.tmp
@@ -115,7 +115,7 @@ $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
 check-DEJAGNU $(check_DEJAGNU_normal_targets): check-DEJAGNU%: site.exp
-	AR="$(AR)"; export AR; \
+	$(if $*,@)AR="$(AR)"; export AR; \
 	RANLIB="$(RANLIB)"; export RANLIB; \
 	if [ -z "$*" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  rm -rf normal-parallel || true; \
diff --git a/libstdc++-v3/testsuite/Makefile.in b/libstdc++-v3/testsuite/Makefile.in
index 59060b8..0fc26f4 100644
--- a/libstdc++-v3/testsuite/Makefile.in
+++ b/libstdc++-v3/testsuite/Makefile.in
@@ -553,9 +553,9 @@ new-abi-baseline:
 	  ${extract_symvers} ../src/.libs/libstdc++.so $${output})
 
 %/site.exp: site.exp
-	-test -d $* || mkdir $*
+	-@test -d $* || mkdir $*
 	@srcdir=`cd $(srcdir); ${PWD_COMMAND}`;
-	objdir=`${PWD_COMMAND}`/$*; \
+	@objdir=`${PWD_COMMAND}`/$*; \
 	sed -e "s|^set srcdir .*$$|set srcdir $$srcdir|" \
 	    -e "s|^set objdir .*$$|set objdir $$objdir|" \
 	    site.exp > $*/site.exp.tmp
@@ -566,7 +566,7 @@ $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
 check-DEJAGNU $(check_DEJAGNU_normal_targets): check-DEJAGNU%: site.exp
-	AR="$(AR)"; export AR; \
+	$(if $*,@)AR="$(AR)"; export AR; \
 	RANLIB="$(RANLIB)"; export RANLIB; \
 	if [ -z "$*" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  rm -rf normal-parallel || true; \

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Mon, Sep 22, 2014 at 12:21:08PM -0400, Jason Merrill wrote:
> On 09/22/2014 11:58 AM, Jakub Jelinek wrote:
> >LGTM (though, supposedly we want similar change in
> >libstdc++-v3/testsuite/Makefile.am).
> >Or, if people would really like to see the commands, we could print them
> >just once, using e.g.
> >	-$(if $(check_p_subno),@)(rootme= ...
> >(then e.g. check-parallel-gcc goal would print the command, but
> >check-parallel-gcc-1 or check-parallel-gcc-112 would not).
> 
> So, like this?

Ok, thanks.

> commit c750897381a3f936e27cabd825cfa85ce936a6a9
> Author: Jason Merrill <jason@redhat.com>
> Date:   Mon Sep 22 11:44:00 2014 -0400
> 
>     gcc/
>     	* Makefile.in (check-parallel-%): Add @.
>     libstdc++-v3/
>     	* testsuite/Makefile.am (%/site.exp): Add @.
>     	(check-DEJAGNU): Likewise.
>     	* testsuite/Makefile.in: Regenerate.
> 
> diff --git a/gcc/Makefile.in b/gcc/Makefile.in
> index 6f251a5..97b439a 100644
> --- a/gcc/Makefile.in
> +++ b/gcc/Makefile.in
> @@ -3674,10 +3674,10 @@ $(lang_checks_parallelized): check-% : site.exp
>  	fi
>  
>  check-parallel-% : site.exp
> -	-test -d plugin || mkdir plugin
> -	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
> -	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
> -	-(rootme=`${PWD_COMMAND}`; export rootme; \
> +	-@test -d plugin || mkdir plugin
> +	-@test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
> +	@test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
> +	-$(if $(check_p_subno),@)(rootme=`${PWD_COMMAND}`; export rootme; \
>  	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
>  	if [ -n "$(check_p_subno)" ] \
>  	   && [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] \
> diff --git a/libstdc++-v3/testsuite/Makefile.am b/libstdc++-v3/testsuite/Makefile.am
> index e206aba..b4c9e85 100644
> --- a/libstdc++-v3/testsuite/Makefile.am
> +++ b/libstdc++-v3/testsuite/Makefile.am
> @@ -91,9 +91,9 @@ new-abi-baseline:
>  	  ${extract_symvers} ../src/.libs/libstdc++.so $${output})
>  
>  %/site.exp: site.exp
> -	-test -d $* || mkdir $*
> +	-@test -d $* || mkdir $*
>  	@srcdir=`cd $(srcdir); ${PWD_COMMAND}`;
> -	objdir=`${PWD_COMMAND}`/$*; \
> +	@objdir=`${PWD_COMMAND}`/$*; \
>  	sed -e "s|^set srcdir .*$$|set srcdir $$srcdir|" \
>  	    -e "s|^set objdir .*$$|set objdir $$objdir|" \
>  	    site.exp > $*/site.exp.tmp
> @@ -115,7 +115,7 @@ $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
>  
>  # Run the testsuite in normal mode.
>  check-DEJAGNU $(check_DEJAGNU_normal_targets): check-DEJAGNU%: site.exp
> -	AR="$(AR)"; export AR; \
> +	$(if $*,@)AR="$(AR)"; export AR; \
>  	RANLIB="$(RANLIB)"; export RANLIB; \
>  	if [ -z "$*" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
>  	  rm -rf normal-parallel || true; \
> diff --git a/libstdc++-v3/testsuite/Makefile.in b/libstdc++-v3/testsuite/Makefile.in
> index 59060b8..0fc26f4 100644
> --- a/libstdc++-v3/testsuite/Makefile.in
> +++ b/libstdc++-v3/testsuite/Makefile.in
> @@ -553,9 +553,9 @@ new-abi-baseline:
>  	  ${extract_symvers} ../src/.libs/libstdc++.so $${output})
>  
>  %/site.exp: site.exp
> -	-test -d $* || mkdir $*
> +	-@test -d $* || mkdir $*
>  	@srcdir=`cd $(srcdir); ${PWD_COMMAND}`;
> -	objdir=`${PWD_COMMAND}`/$*; \
> +	@objdir=`${PWD_COMMAND}`/$*; \
>  	sed -e "s|^set srcdir .*$$|set srcdir $$srcdir|" \
>  	    -e "s|^set objdir .*$$|set objdir $$objdir|" \
>  	    site.exp > $*/site.exp.tmp
> @@ -566,7 +566,7 @@ $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
>  
>  # Run the testsuite in normal mode.
>  check-DEJAGNU $(check_DEJAGNU_normal_targets): check-DEJAGNU%: site.exp
> -	AR="$(AR)"; export AR; \
> +	$(if $*,@)AR="$(AR)"; export AR; \
>  	RANLIB="$(RANLIB)"; export RANLIB; \
>  	if [ -z "$*" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
>  	  rm -rf normal-parallel || true; \


	Jakub

Re: [PATCH] gcc parallel make check

[Segher Boessenkool]

On Mon, Sep 22, 2014 at 05:26:04PM +0200, Jakub Jelinek wrote:
> I've been considering that too, but not sure what info people find valuable
> and what they don't.

The ten million "Running blablablalba.exp ..." messages on a very parallel
run aren't helpful in my opinion.  There might be more but that drowns out
everything else :-)


Segher

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Mon, Sep 22, 2014 at 10:44:06AM -0500, Segher Boessenkool wrote:
> On Mon, Sep 22, 2014 at 05:26:04PM +0200, Jakub Jelinek wrote:
> > I've been considering that too, but not sure what info people find valuable
> > and what they don't.
> 
> The ten million "Running blablablalba.exp ..." messages on a very parallel
> run aren't helpful in my opinion.  There might be more but that drowns out
> everything else :-)

It has some value, it shows the actual progress.  Sure, you can just watch
the *.log files as they are populated and get better picture.  I think the
Running *.exp messages go from dejagnu, not from gcc testsuite changes.

	Jakub

Re: [PATCH] gcc parallel make check

[Segher Boessenkool]

On Mon, Sep 22, 2014 at 05:49:12PM +0200, Jakub Jelinek wrote:
> On Mon, Sep 22, 2014 at 10:44:06AM -0500, Segher Boessenkool wrote:
> > On Mon, Sep 22, 2014 at 05:26:04PM +0200, Jakub Jelinek wrote:
> > > I've been considering that too, but not sure what info people find valuable
> > > and what they don't.
> > 
> > The ten million "Running blablablalba.exp ..." messages on a very parallel
> > run aren't helpful in my opinion.  There might be more but that drowns out
> > everything else :-)
> 
> It has some value, it shows the actual progress.  Sure, you can just watch
> the *.log files as they are populated and get better picture.  I think the
> Running *.exp messages go from dejagnu, not from gcc testsuite changes.

Hrm.  Looking at the log files it seems there are not more of those messages
at all since the changes.  Maybe it just all got too fast! :-)


Segher

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Sep 12, 2014 at 04:42:25PM -0700, Mike Stump wrote:
> On Sep 12, 2014, at 9:32 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> > Here is my latest version of the patch.
> 
> I did a timing test:

Here is an updated version.
Changes since last version:
1) acats parallelized the same way (just, because it is in shell,
   using mkdir instead of open with O_EXCL|O_CREAT);
   also, as acats has pretty significant initial setup time
   (up to a minute or so on not really fast box), I'm now
   creating the /support stuff before spawning the parallel
   jobs and let the parallel jobs use the same shared support
   directory
2) I'm now using addprefix instead of patsubst where appropriate
3) I'm using $(if ...) instead of $(or ...) to make it usable
   with make 3.80 (3.81 already supports or)
4) parallelization is performed for any kinds of RUNTESTFLAGS arguments now
5) struct-layout-1.exp apparently doesn't have to be performed serially,
   and for gnu-encoding.exp I've used similar change as for go-test.exp
6) in libstdc++, abi.exp, pretty-printers.exp and xmethods.exp are performed
   together with conformance.exp, so again parallelization for any
   RUNTESTFLAGS flags; abi.exp and xmethods.exp are serially tested
   by the first runtest instance to encounter them

Regtested on x86_64-linux, without the patch toplevel make -k check
took 8hrs3minutes (don't have time data for that run), without the
patch toplevel make -j48 -k check took:
real    40m21.984s
user    341m51.675s
sys     112m46.993s
and with the patch make -j48 -k check took:
real    32m22.066s
user    355m1.788s
sys     117m5.809s
I saw over 45 jobs running pretty much as the point where all the
testing was done, and test_summary run from the non-parallel testing
is the same as test_summary from the -j48 testing with the patch.
Is this version ok for trunk?

2014-09-14  Jakub Jelinek  <jakub@redhat.com>

gcc/
	* Makefile.in (dg_target_exps): Remove.
	(check_gcc_parallelize): Change to just an upper bound number.
	(check-%-subtargets): Always print the non-parallelized goals.
	(check_p_vars, check_p_comma, check_p_subwork): Remove.
	(check_p_count, check_p_numbers0, check_p_numbers1, check_p_numbers2,
	check_p_numbers3, check_p_numbers4, check_p_numbers5,
	check_p_numbers6): New variables.
	(check_p_numbers): Set to sequence from 1 to 9999.
	(check_p_subdirs): Set to sequence from 1 to minimum of
	$(check_p_count) and either GCC_TEST_PARALLEL_SLOTS env var if set,
	or 128.
	(check-%, check-parallel-%): Rewritten so that for parallelized
	testing each job runs all the *.exp files, with
	GCC_RUNTEST_PARALLELIZE_DIR set in environment.
gcc/go/
	* Make-lang.in (check_go_parallelize): Change to just an upper bound
	number.
gcc/fortran/
	* Make-lang.in (check_gfortran_parallelize): Change to just an upper
	bound number.
gcc/cp/
	* Make-lang.in (check_g++_parallelize): Change to just an upper bound
	number.
gcc/objc/
	* Make-lang.in (check_objc_parallelize): Change to just an upper
	bound number.
gcc/ada/
	* gcc-interface/Make-lang.in (check_acats_numbers0,
	check_acats_numbers1, check_acats_numbers2, check_acats_numbers3,
	check_acats_numbers4, check_acats_numbers5, check_acats_numbers6,
	check_acats_numbers, check_acats_subdirs): New variables.
	(check_acats_targets): Use $(check_acats_subdirs).
	(check-acats, check-acats%): Rewritten so that for parallelized
	testing each job runs all the chapters files, with
	GCC_RUNTEST_PARALLELIZE_DIR set in environment.  Prepare the support
	directory sequentially and share it.
	(check-acats-subtargets): Always print just check-acats.
gcc/testsuite/
	* lib/gcc-defs.exp (gcc_parallel_test_run_p,
	gcc_parallel_test_enable): New procedures.  If
	GCC_RUNTEST_PARALLELIZE_DIR is set in environment, override
	runtest_file_p to invoke also gcc_parallel_test_run_p.
	* g++.dg/guality/guality.exp (check_guality): Save/restore
	test_counts array around the body of the procedure.
	* gcc.dg/guality/guality.exp (check_guality): Likewise.
	* g++.dg/plugin/plugin.exp: Run all the tests serially
	by the first parallel runtest encountering it.
	* gcc.dg/plugin/plugin.exp: Likewise.
	* gcc.misc-tests/matrix1.exp: Likewise.
	* gcc.misc-tests/dhry.exp: Likewise.
	* gcc.misc-tests/acker1.exp: Likewise.
	* gcc.misc-tests/linkage.exp: Likewise.
	* gcc.misc-tests/mg.exp: Likewise.
	* gcc.misc-tests/mg-2.exp: Likewise.
	* gcc.misc-tests/sort2.exp: Likewise.
	* gcc.misc-tests/sieve.exp: Likewise.
	* gcc.misc-tests/options.exp: Likewise.
	* gcc.misc-tests/help.exp: Likewise.
	* go.test/go-test.exp (go-gc-tests): Use
	gcc_parallel_test_enable {0, 1} around all handling of
	each test.
	* objc.dg/gnu-encoding/gnu-encoding.exp: Likewise.
	* ada/acats/run_all.sh: Parallelize using mkdir inside of
	a shared directory.  Avoid forking and expr uses if
	shell supports $(()).
libstdc++-v3/
	* testsuite/Makefile.am (check_p_numbers0, check_p_numbers1,
	check_p_numbers2, check_p_numbers3, check_p_numbers4,
	check_p_numbers5, check_p_numbers6, check_p_numbers,
	check_p_subdirs): New variables.
	(check_DEJAGNU_normal_targets): Use check_p_subdirs.
	(check-DEJAGNU): Rewritten so that for parallelized
	testing each job runs all the *.exp files, with
	GCC_RUNTEST_PARALLELIZE_DIR set in environment.
	* testsuite/Makefile.in: Regenerated.
	* testsuite/lib/libstdc++.exp (gcc_parallel_test_run_p,
	gcc_parallel_test_enable): New procedures.  If
	GCC_RUNTEST_PARALLELIZE_DIR is set in environment, override
	runtest_file_p to invoke also gcc_parallel_test_run_p.
	* testsuite/libstdc++-abi/abi.exp: Run all the tests serially
	by the first parallel runtest encountering it.  Fix up path
	of the extract_symvers script.
	* testsuite/libstdc++-xmethods/xmethods.exp: Run all the tests
	serially by the first parallel runtest encountering it.  Run
	dg-finish even in case of error.

--- gcc/Makefile.in.jj	2014-09-14 21:09:04.612498827 +0200
+++ gcc/Makefile.in	2014-09-14 21:30:27.509141045 +0200
@@ -513,34 +513,10 @@ xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=
 lang_checks_parallelized=
-dg_target_exps:=aarch64.exp,alpha.exp,arm.exp,avr.exp,bfin.exp,cris.exp
-dg_target_exps:=$(dg_target_exps),epiphany.exp,frv.exp,i386.exp,ia64.exp
-dg_target_exps:=$(dg_target_exps),m68k.exp,microblaze.exp,mips.exp,powerpc.exp
-dg_target_exps:=$(dg_target_exps),rx.exp,s390.exp,sh.exp,sparc.exp,spu.exp
-dg_target_exps:=$(dg_target_exps),tic6x.exp,xstormy16.exp
-# This lists a couple of test files that take most time during check-gcc.
-# When doing parallelized check-gcc, these can run in parallel with the
-# remaining tests.  Each word in this variable stands for work for one
-# make goal and one extra make goal is added to handle all the *.exp
-# files not handled explicitly already.  If multiple *.exp files
-# should be run in the same runtest invocation (usually if they aren't
-# very long running, but still should be split of from the check-parallel-$lang
-# remaining tests runtest invocation), they should be concatenated with commas.
-# Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
-# by tcl) and as the *.exp arguments are mached both as is and with
-# */ prefixed to it in runtest_file_p, it is usually desirable to include
-# a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-fA-F\]* \
-		      execute.exp=execute/\[pP\]*,dg.exp \
-		      execute.exp=execute/\[g-oq-zG-OQ-Z\]*,compile.exp=compile/2* \
-		      compile.exp=compile/\[9pP\]*,builtins.exp \
-		      compile.exp=compile/\[013-8a-oq-zA-OQ-Z\]* \
-		      dg-torture.exp,ieee.exp \
-		      vect.exp,unsorted.exp \
-		      guality.exp \
-		      struct-layout-1.exp,stackalign.exp \
-		      $(dg_target_exps)
+# Upper limit to which it is useful to parallelize this lang target.
+# It doesn't make sense to try e.g. 128 goals for small testsuites
+# like objc or go.
+check_gcc_parallelize=10000
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -3631,27 +3607,34 @@ $(filter-out $(lang_checks_parallelized)
 	    export TCL_LIBRARY ; fi ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
-$(patsubst %,%-subtargets,$(filter-out $(lang_checks_parallelized),$(lang_checks))): check-%-subtargets:
+$(patsubst %,%-subtargets,$(lang_checks)): check-%-subtargets:
 	@echo check-$*
 
 check_p_tool=$(firstword $(subst _, ,$*))
-check_p_vars=$(check_$(check_p_tool)_parallelize)
+check_p_count=$(check_$(check_p_tool)_parallelize)
 check_p_subno=$(word 2,$(subst _, ,$*))
-check_p_comma=,
-check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers1)))
+check_p_numbers3:=$(addprefix 0,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers3)))
+check_p_numbers5:=$(addprefix 0,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
 check_p_subdir=$(subst _,,$*)
-check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
+check_p_subdirs=$(wordlist 1,$(check_p_count),$(wordlist 1, \
+		$(if $(GCC_TEST_PARALLEL_SLOTS),$(GCC_TEST_PARALLEL_SLOTS),128), \
+		$(check_p_numbers)))
 
 # For parallelized check-% targets, this decides whether parallelization
 # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
 # but optional --target_board or --extra_opts arguments).  If desirable,
 # recursive make is run with check-parallel-$lang{,1,2,3,4,5} etc. goals,
 # which can be executed in parallel, as they are run in separate directories.
-# check-parallel-$lang{1,2,3,4,5} etc. goals invoke runtest with the longest
-# running *.exp files from the testsuite, as determined by check_$lang_parallelize
-# variable.  The check-parallel-$lang goal in that case invokes runtest with
-# all the remaining *.exp files not handled by the separate goals.
+# check-parallel-$lang{,1,2,3,4,5} etc. goals invoke runtest with
+# GCC_RUNTEST_PARALLELIZE_DIR var in the environment and runtest_file_p
+# dejaGNU procedure is overridden to additionally synchronize through
+# a $lang-parallel directory which tests will be run by which runtest instance.
 # Afterwards contrib/dg-extract-results.sh is used to merge the sum and log
 # files.  If parallelization isn't desirable, only one recursive make
 # is run with check-parallel-$lang goal and check_$lang_parallelize variable
@@ -3662,76 +3645,60 @@ check_p_subdirs=$(wordlist 1,$(words $(c
 # to lang_checks_parallelized variable and define check_$lang_parallelize
 # variable (see above check_gcc_parallelize description).
 $(lang_checks_parallelized): check-% : site.exp
-	@if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \
-	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	-rm -rf $(TESTSUITEDIR)/$*-parallel
+	@if [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR) || true; \
+	  test -d $(TESTSUITEDIR)/$*-parallel || mkdir $(TESTSUITEDIR)/$*-parallel || true; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/$(TESTSUITEDIR)/$(check_p_tool)-parallel ; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR ; \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check-parallel-$* \
 	    $(patsubst %,check-parallel-$*_%, $(check_p_subdirs)); \
-	  for file in $(TESTSUITEDIR)/$*/$* \
-		      $(patsubst %,$(TESTSUITEDIR)/$*%/$*,$(check_p_subdirs));\
+	  sums= ; logs= ; \
+	  for dir in $(TESTSUITEDIR)/$* \
+		     $(patsubst %,$(TESTSUITEDIR)/$*%,$(check_p_subdirs));\
 	  do \
-	    mv -f $$file.sum $$file.sum.sep; mv -f $$file.log $$file.log.sep; \
+	    if [ -d $$dir ]; then \
+	      mv -f $$dir/$*.sum $$dir/$*.sum.sep; mv -f $$dir/$*.log $$dir/$*.log.sep; \
+	      sums="$$sums $$dir/$*.sum.sep"; logs="$$logs $$dir/$*.log.sep"; \
+	    fi; \
 	  done; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh \
-	    $(TESTSUITEDIR)/$*/$*.sum.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.sum.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh $$sums \
 	    > $(TESTSUITEDIR)/$*/$*.sum; \
-	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L \
-	    $(TESTSUITEDIR)/$*/$*.log.sep \
-	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.log.sep,$(check_p_subdirs)) \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L $$logs \
 	    > $(TESTSUITEDIR)/$*/$*.log; \
+	  rm -rf $(TESTSUITEDIR)/$*-parallel || true; \
 	else \
 	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
 	    check_$*_parallelize= check-parallel-$*; \
 	fi
 
-# Just print the parallelized subtargets for those that want to split
-# the testing across machines.
-$(patsubst %,%-subtargets,$(lang_checks_parallelized)): check-%-subtargets:
-	@echo check-parallel-$* \
-	  $(patsubst %,check-parallel-$*_%, $(check_p_subdirs))
-
-# In the if [ -n "$(check_p_subno)" ] case runtest should be given the name of
-# the given *.exp file(s).  See comment above check_gcc_parallelize variable
-# for details on the content of these variables.
-#
-# In the elif [ -n "$(check_p_vars)" ] case runtest should be given
-# names of all the *.exp files for this tool that aren't already handled by
-# other goals.  First it finds all the *.exp files for this tool, then
-# prunes those already specified in check_$lang_parallelize or duplicates.
-#
-# Otherwise check-$lang isn't parallelized and runtest is invoked just with
-# the $(RUNTESTFLAGS) arguments.
 check-parallel-% : site.exp
 	-test -d plugin || mkdir plugin
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
-	cd $(TESTSUITEDIR)/$(check_p_subdir); \
-	rm -f tmp-site.exp; \
-	sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
+	if [ -n "$(check_p_subno)" ] \
+	   && [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] \
+	   && [ -f $(TESTSUITEDIR)/$(check_p_tool)-parallel/finished ]; then \
+	  rm -rf $(TESTSUITEDIR)/$(check_p_subdir); \
+	else \
+	  cd $(TESTSUITEDIR)/$(check_p_subdir); \
+	  rm -f tmp-site.exp; \
+	  sed '/set tmpdir/ s|testsuite$$|$(TESTSUITEDIR)/$(check_p_subdir)|' \
 		< ../../site.exp > tmp-site.exp; \
-	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
-	EXPECT=${EXPECT} ; export EXPECT ; \
-	if [ -f $${rootme}/../expect/expect ] ; then  \
-	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
-	    export TCL_LIBRARY ; fi ; \
-	runtestflags= ; \
-	if [ -n "$(check_p_subno)" ] ; then \
-	  runtestflags="$(check_p_subwork)"; \
-	elif [ -n "$(check_p_vars)" ] ; then \
-	  parts="`echo ' $(strip $(subst $(check_p_comma), ,$(check_p_vars))) ' \
-		  | sed 's/=[^ ]* / /g'`"; \
-	  for part in `find $$srcdir/testsuite/$(check_p_tool)* -name \*.exp` ; do \
-	    part=`basename $$part` ; \
-	    case " $$parts $$runtestflags " in \
-	      *" $$part "*) ;; \
-	      *) runtestflags="$$runtestflags $$part" ;; \
-	    esac ; \
-	  done ; \
-	fi ; \
-	$(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS) $$runtestflags)
+	  $(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	  EXPECT=${EXPECT} ; export EXPECT ; \
+	  if [ -f $${rootme}/../expect/expect ] ; then  \
+	    TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; \
+	  fi ; \
+	  $(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS); \
+	  if [ -n "$$GCC_RUNTEST_PARALLELIZE_DIR" ] ; then \
+	    touch $${rootme}/$(TESTSUITEDIR)/$(check_p_tool)-parallel/finished; \
+	  fi ; \
+	fi )
 
 # QMTest targets
 
--- gcc/go/Make-lang.in.jj	2014-09-14 21:09:05.744493198 +0200
+++ gcc/go/Make-lang.in	2014-09-14 21:25:17.767673249 +0200
@@ -131,11 +131,7 @@ go.srcman: doc/gccgo.1
 
 lang_checks += check-go
 lang_checks_parallelized += check-go
-check_go_parallelize = go-test.exp=*/test/\[0-57-9a-bd-hj-qs-zA-Z\]* \
-		       go-test.exp=*/test/c* \
-		       go-test.exp=*/test/i* \
-		       go-test.exp=*/test/r* \
-		       go-test.exp=*/test/6*
+check_go_parallelize = 10
 
 # Install hooks.
 
--- gcc/fortran/Make-lang.in.jj	2014-09-14 21:09:04.560499080 +0200
+++ gcc/fortran/Make-lang.in	2014-09-14 21:25:17.768673222 +0200
@@ -168,12 +168,7 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = 10000
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
--- gcc/cp/Make-lang.in.jj	2014-09-14 21:09:05.729493272 +0200
+++ gcc/cp/Make-lang.in	2014-09-14 21:25:17.768673222 +0200
@@ -156,11 +156,7 @@ check-c++-subtargets : check-g++-subtarg
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp \
-	dg.exp=g++.dg/[0-9A-Za-bd-su-z]* \
-	dg.exp=g++.dg/[ct]* \
-	dg.exp=c-c++-common/*,dg-torture.exp
-
+check_g++_parallelize = 10000
 #\f
 # Install hooks:
 # cc1plus is installed elsewhere as part of $(COMPILERS).
--- gcc/ada/gcc-interface/Make-lang.in.jj	2014-08-04 10:43:27.000000000 +0200
+++ gcc/ada/gcc-interface/Make-lang.in	2014-09-15 14:38:30.250714169 +0200
@@ -862,7 +862,18 @@ check-ada-subtargets: check-acats-subtar
 
 ACATSDIR = $(TESTSUITEDIR)/ada/acats
 
-check_acats_targets = $(patsubst %,check-acats%, 0 1 2)
+check_acats_numbers0:=1 2 3 4 5 6 7 8 9
+check_acats_numbers1:=0 $(check_acats_numbers0)
+check_acats_numbers2:=$(foreach i,$(check_acats_numbers0),$(addprefix $(i),$(check_acats_numbers1)))
+check_acats_numbers3:=$(addprefix 0,$(check_acats_numbers1)) $(check_acats_numbers2)
+check_acats_numbers4:=$(foreach i,$(check_acats_numbers0),$(addprefix $(i),$(check_acats_numbers3)))
+check_acats_numbers5:=$(addprefix 0,$(check_acats_numbers3)) $(check_acats_numbers4)
+check_acats_numbers6:=$(foreach i,$(check_acats_numbers0),$(addprefix $(i),$(check_acats_numbers5)))
+check_acats_numbers:=$(check_acats_numbers0) $(check_acats_numbers2) $(check_acats_numbers4) $(check_acats_numbers6)
+check_acats_subdirs=$(wordlist 1, \
+		    $(if $(GCC_TEST_PARALLEL_SLOTS),$(GCC_TEST_PARALLEL_SLOTS),128), \
+		    $(check_acats_numbers))
+check_acats_targets = $(addprefix check-acats, $(check_acats_subdirs))
 
 check-acats:
 	@test -d $(ACATSDIR) || mkdir -p $(ACATSDIR); \
@@ -870,35 +881,45 @@ check-acats:
 	EXPECT=$(EXPECT); export EXPECT; \
 	if [ -z "$(CHAPTERS)" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; \
 	then \
+	  rm -rf $(ACATSDIR)-parallel; \
+	  mkdir $(ACATSDIR)-parallel; \
+	  ( testdir=`cd ${srcdir}/${ACATSDIR} && ${PWD_COMMAND}`; \
+	    export testdir; \
+	    cd $(ACATSDIR) && $(SHELL) $${testdir}/run_acats NONE ) \
+	    || exit 1; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=$$rootme/$(ACATSDIR)-parallel; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR; \
 	  $(MAKE) $(check_acats_targets); \
-	  for idx in 0 1 2; do \
+	  rm -rf $(ACATSDIR)-parallel; \
+	  acats_sums= ; acats_logs= ; \
+	  for idx in "" $(check_acats_subdirs); do \
+	    [ -d $(ACATSDIR)$$idx ] || continue; \
 	    mv -f $(ACATSDIR)$$idx/acats.sum $(ACATSDIR)$$idx/acats.sum.sep; \
 	    mv -f $(ACATSDIR)$$idx/acats.log $(ACATSDIR)$$idx/acats.log.sep; \
+	    acats_sums="$$acats_sums $(ACATSDIR)$$idx/acats.sum.sep"; \
+	    acats_logs="$$acats_logs $(ACATSDIR)$$idx/acats.log.sep"; \
 	  done; \
 	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh \
-	    $(ACATSDIR)0/acats.sum.sep $(ACATSDIR)1/acats.sum.sep \
-	    $(ACATSDIR)2/acats.sum.sep > $(ACATSDIR)/acats.sum; \
+	    $$acats_sums > $(ACATSDIR)/acats.sum; \
 	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L \
-	    $(ACATSDIR)0/acats.log.sep $(ACATSDIR)1/acats.log.sep \
-	    $(ACATSDIR)2/acats.log.sep > $(ACATSDIR)/acats.log; \
+	    $$acats_logs > $(ACATSDIR)/acats.log; \
 	  exit 0; \
 	fi; \
 	testdir=`cd ${srcdir}/${ACATSDIR} && ${PWD_COMMAND}`; \
 	export testdir; cd $(ACATSDIR) && $(SHELL) $${testdir}/run_acats $(CHAPTERS)
 
 check-acats-subtargets:
-	@echo $(check_acats_targets)
+	@echo check-acats
 
 # Parallelized check-acats
 $(check_acats_targets): check-acats%:
+	if [ -f $$GCC_RUNTEST_PARALLELIZE_DIR/finished ]; then \
+	  rm -rf $(ACATSDIR)$*; exit 0; \
+	fi; \
 	test -d $(ACATSDIR)$* || mkdir -p $(ACATSDIR)$*; \
 	testdir=`cd ${srcdir}/${ACATSDIR} && ${PWD_COMMAND}`; \
-	case "$*" in \
-	  0) chapters="`cd $$testdir/tests; echo [a-b]* c[0-4]*`";; \
-	  1) chapters="`cd $$testdir/tests; echo c[5-9ab]*`";; \
-	  2) chapters="`cd $$testdir/tests; echo c[c-z]* [d-z]*`";; \
-	esac; \
-	export testdir; cd $(ACATSDIR)$* && $(SHELL) $${testdir}/run_acats $$chapters
+	export testdir; cd $(ACATSDIR)$* && $(SHELL) $${testdir}/run_acats
+	touch $$GCC_RUNTEST_PARALLELIZE_DIR/finished
 
 .PHONY: check-acats $(check_acats_targets)
 \f
--- gcc/testsuite/gcc.misc-tests/sieve.exp.jj	2014-09-14 21:09:05.668493574 +0200
+++ gcc/testsuite/gcc.misc-tests/sieve.exp	2014-09-14 21:25:17.773673082 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p sieve] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase sieve.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/sort2.exp.jj	2014-09-14 21:09:05.524494295 +0200
+++ gcc/testsuite/gcc.misc-tests/sort2.exp	2014-09-14 21:25:17.773673082 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p sort2] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase sort2.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/matrix1.exp.jj	2014-09-14 21:09:05.544494191 +0200
+++ gcc/testsuite/gcc.misc-tests/matrix1.exp	2014-09-14 21:25:17.772673116 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p matrix1] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase matrix1.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/dhry.exp.jj	2014-09-14 21:09:05.596493933 +0200
+++ gcc/testsuite/gcc.misc-tests/dhry.exp	2014-09-14 21:25:17.772673116 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p dhry] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase dhry.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/mg.exp.jj	2014-09-14 21:09:05.580494013 +0200
+++ gcc/testsuite/gcc.misc-tests/mg.exp	2014-09-14 21:25:17.772673116 +0200
@@ -17,8 +17,20 @@
 # Test the -MG flag.
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p mg] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions none-of-the-above
 set compiler_output "mg.o ?: .*mg.c \[ \\\\\n\]*nonexist.h"
 postbase mg.c "" "" "-MM -MG"
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/options.exp.jj	2014-09-14 21:09:05.563494097 +0200
+++ gcc/testsuite/gcc.misc-tests/options.exp	2014-09-14 21:25:17.773673082 +0200
@@ -19,6 +19,17 @@
 # match the patterns COMPILER_PATTERN, AS_PATTERN and LD_PATTERN,
 # respectively.
 
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p options] {
+    return
+}
+gcc_parallel_test_enable 0
+
 proc check_for_all_options {language gcc_options compiler_pattern as_pattern ld_pattern} {
     set filename test-[pid]
     set fd [open $filename.c w]
@@ -52,3 +63,5 @@ proc check_for_all_options {language gcc
 }
 
 check_for_all_options c {--coverage} {-fprofile-arcs -ftest-coverage} {} {-lgcov}
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/help.exp.jj	2014-09-14 21:09:05.555494137 +0200
+++ gcc/testsuite/gcc.misc-tests/help.exp	2014-09-14 21:25:17.774673051 +0200
@@ -18,6 +18,16 @@
 # documented in --help, and that the various --help* options work.
 
 load_lib options.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p help] {
+    return
+}
+gcc_parallel_test_enable 0
 
 # Document --version.  Ideally, there should be no undocumented switches
 # in --help.
@@ -78,3 +88,5 @@ check_for_options c "--help=joined,^sepa
 check_for_options c "--help=joined,undocumented" "" "" ""
 # Listing only excludes gives empty results.
 check_for_options c "--help=^joined,^separate" "" "" ""
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/linkage.exp.jj	2014-09-14 21:09:05.624493790 +0200
+++ gcc/testsuite/gcc.misc-tests/linkage.exp	2014-09-14 21:25:17.772673116 +0200
@@ -18,6 +18,17 @@
 # was written by Rob Savoye. (rob@cygnus.com)
 # All the other tests driven by that file have since been moved elsewhere.
 
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p linkage] {
+    return
+}
+gcc_parallel_test_enable 0
+
 if { [isnative] && ![is_remote host] } then {
     set lines [gcc_target_compile "$srcdir/$subdir/linkage-x.c" "linkage-x.o" object {additional_flags="-w"}]
     if ![string match "" $lines] then {
@@ -117,3 +128,5 @@ if { [isnative] && ![is_remote host] } t
 	file delete "linkage-x.o"
     }
 }
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/acker1.exp.jj	2014-09-14 21:09:05.586493983 +0200
+++ gcc/testsuite/gcc.misc-tests/acker1.exp	2014-09-14 21:25:17.772673116 +0200
@@ -20,9 +20,21 @@ if { ![info exists PERF_TEST] || "$PERF_
 }
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p acker1] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions run
 set compiler_output "^$"
 set program_output "^$"
 postbase acker1.c $run $groups
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.misc-tests/mg-2.exp.jj	2014-09-14 21:09:05.570494062 +0200
+++ gcc/testsuite/gcc.misc-tests/mg-2.exp	2014-09-14 21:25:17.773673082 +0200
@@ -17,8 +17,20 @@
 # Test the -MG flag with a system header file.
 
 load_lib mike-gcc.exp
+load_lib gcc-defs.exp
+
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p mg-2] {
+    return
+}
+gcc_parallel_test_enable 0
 
 prebase
 set actions none-of-the-above
 set compiler_output "mg-2.o ?: .*mg-2.c \[ \\\\\n\]*nonexist.h"
 postbase mg-2.c "" "" "-MM -MG"
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/g++.dg/guality/guality.exp.jj	2014-09-14 21:09:04.767498056 +0200
+++ gcc/testsuite/g++.dg/guality/guality.exp	2014-09-14 21:25:17.768673222 +0200
@@ -14,6 +14,11 @@ if { [istarget "powerpc-ibm-aix*"] } {
 }
 
 proc check_guality {args} {
+    # Don't count check_guality as PASS, or FAIL etc., that would make
+    # the total PASS count dependent on how many parallel runtest invocations
+    # ran guality.exp.  So save the counts first and restore them afterwards.
+    global test_counts
+    array set saved_test_counts [array get test_counts]
     set result [eval check_compile guality_check executable $args "-g -O0"]
     set lines [lindex $result 0]
     set output [lindex $result 1]
@@ -23,6 +28,7 @@ proc check_guality {args} {
       set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
     }
     remote_file build delete $output
+    array get test_counts [array get saved_test_counts]
     return $ret
 }
 
--- gcc/testsuite/g++.dg/plugin/plugin.exp.jj	2014-09-14 21:09:04.824497773 +0200
+++ gcc/testsuite/g++.dg/plugin/plugin.exp	2014-09-14 21:25:17.769673197 +0200
@@ -44,6 +44,15 @@ if $tracelevel then {
 # Load support procs.
 load_lib plugin-support.exp
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p plugin] {
+    return
+}
+gcc_parallel_test_enable 0
+
 # Specify the plugin source file and the associated test files in a list.
 # plugin_test_list={ {plugin1 test1 test2 ...} {plugin2 test1 ...} ... }
 set plugin_test_list [list \
@@ -68,3 +77,5 @@ foreach plugin_test $plugin_test_list {
     set plugin_input_tests [lreplace $plugin_test 0 0]
     plugin-test-execute $plugin_src $plugin_input_tests
 }
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/gcc.dg/guality/guality.exp.jj	2014-09-14 21:09:05.362495088 +0200
+++ gcc/testsuite/gcc.dg/guality/guality.exp	2014-09-14 21:25:17.769673197 +0200
@@ -14,6 +14,11 @@ if { [istarget "powerpc-ibm-aix*"] } {
 }
 
 proc check_guality {args} {
+    # Don't count check_guality as PASS, or FAIL etc., that would make
+    # the total PASS count dependent on how many parallel runtest invocations
+    # ran guality.exp.  So save the counts first and restore them afterwards.
+    global test_counts
+    array set saved_test_counts [array get test_counts]
     set result [eval check_compile guality_check executable $args "-g -O0"]
     set lines [lindex $result 0]
     set output [lindex $result 1]
@@ -23,6 +28,7 @@ proc check_guality {args} {
       set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
     }
     remote_file build delete $output
+    array set test_counts [array get saved_test_counts]
     return $ret
 }
 
--- gcc/testsuite/gcc.dg/plugin/plugin.exp.jj	2014-09-14 21:09:05.470494558 +0200
+++ gcc/testsuite/gcc.dg/plugin/plugin.exp	2014-09-14 21:25:17.770673173 +0200
@@ -44,6 +44,15 @@ if $tracelevel then {
 # Load support procs.
 load_lib plugin-support.exp
 
+# These tests don't run runtest_file_p consistently if it
+# doesn't return the same values, so disable parallelization
+# of this *.exp file.  The first parallel runtest to reach
+# this will run all the tests serially.
+if ![gcc_parallel_test_run_p plugin] {
+    return
+}
+gcc_parallel_test_enable 0
+
 # Specify the plugin source file and the associated test files in a list.
 # plugin_test_list={ {plugin1 test1 test2 ...} {plugin2 test1 ...} ... }
 set plugin_test_list [list \
@@ -80,3 +89,5 @@ dg-runtest [lsort [glob -nocomplain $src
 
 # All done.
 dg-finish
+
+gcc_parallel_test_enable 1
--- gcc/testsuite/go.test/go-test.exp.jj	2014-09-14 21:09:05.292495444 +0200
+++ gcc/testsuite/go.test/go-test.exp	2014-09-14 21:25:17.770673173 +0200
@@ -489,6 +489,12 @@ proc go-gc-tests { } {
 
 	close $fd
 
+	# runtest_file_p is already run above, and the code below can run
+	# runtest_file_p again, make sure everything for this test is
+	# performed if the above runtest_file_p decided this runtest
+	# instance should execute the test
+	gcc_parallel_test_enable 0
+
 	set go_compile_args ""
 	set go_execute_args ""
 	if { [regexp "// run (\[^|&>2\].*)\$" $test_line match progargs] \
@@ -1164,6 +1170,7 @@ proc go-gc-tests { } {
 	set go_compile_args ""
 	set go_execute_args ""
         set TORTURE_OPTIONS [list { -O2 -g }]
+	gcc_parallel_test_enable 1
     }
 
     set dg-do-what-default ${saved-dg-do-what-default}
--- gcc/testsuite/ada/acats/run_all.sh.jj	2013-06-01 10:28:28.100138045 +0200
+++ gcc/testsuite/ada/acats/run_all.sh	2014-09-15 15:33:15.793502393 +0200
@@ -18,6 +18,21 @@ target_run () {
 
 # End of customization section.
 
+# Perform arithmetic evaluation on the ARGs, and store the result in the
+# global $as_val. Take advantage of shells that can avoid forks. The arguments
+# must be portable across $(()) and expr.
+if (eval "test \$(( 1 + 1 )) = 2") 2>/dev/null; then :
+  eval 'as_fn_arith ()
+  {
+    as_val=$(( $* ))
+  }'
+else
+  as_fn_arith ()
+  {
+    as_val=`expr "$@" || test $? -eq 1`
+  }
+fi # as_fn_arith
+
 display_noeol () {
   printf "$@"
   printf "$@" >> $dir/acats.sum
@@ -90,6 +105,18 @@ display `type gnatmake`
 gnatls -v >> $dir/acats.log
 display ""
 
+if [ -n "$GCC_RUNTEST_PARALLELIZE_DIR" ]; then
+  dir_support=$dir/../acats/support
+
+  rm -rf $dir/run
+  mv $dir/tests $dir/tests.$$ 2> /dev/null
+  rm -rf $dir/tests.$$ &
+  mkdir -p $dir/run
+
+  cp -pr $dir/../acats/tests $dir/
+else
+  dir_support=$dir/support
+
 display "		=== acats support ==="
 display_noeol "Generating support files..."
 
@@ -186,6 +213,9 @@ target_gnatmake -c -gnato -gnatE *.adb >
 
 display " done."
 display ""
+
+fi
+
 display "		=== acats tests ==="
 
 if [ $# -eq 0 ]; then
@@ -197,8 +227,14 @@ fi
 glob_countn=0
 glob_countok=0
 glob_countu=0
+par_count=0
+par_countm=0
+par_last=
 
 for chapter in $chapters; do
+   # Used to generate support once and finish after that.
+   [ "$chapter" = "NONE" ] && continue
+
    display Running chapter $chapter ...
 
    if [ ! -d $dir/tests/$chapter ]; then
@@ -209,13 +245,43 @@ for chapter in $chapters; do
 
    cd $dir/tests/$chapter
    ls *.a *.ada *.adt *.am *.dep 2> /dev/null | sed -e 's/\(.*\)\..*/\1/g' | \
-   cut -c1-7 | sort | uniq | comm -23 - $dir/support/norun.lst \
+   cut -c1-7 | sort | uniq | comm -23 - $dir_support/norun.lst \
      > $dir/tests/$chapter/${chapter}.lst 
    countn=`wc -l < $dir/tests/$chapter/${chapter}.lst`
-   glob_countn=`expr $glob_countn + $countn`
-   counti=0
+   as_fn_arith $glob_countn + $countn
+   glob_countn=$as_val
    for i in `cat $dir/tests/$chapter/${chapter}.lst`; do 
-      counti=`expr $counti + 1`
+
+      # If running multiple run_all.sh jobs in parallel, decide
+      # if we should run this test in the current instance.
+      if [ -n "$GCC_RUNTEST_PARALLELIZE_DIR" ]; then
+	 case "$i" in
+	    # Ugh, some tests have inter-test dependencies, those
+	    # tests have to be scheduled on the same parallel instance
+	    # as previous test.
+	    ce2108f | ce2108h | ce3112d) ;;
+	    # All others can be hopefully scheduled freely.
+	    *)
+	       as_fn_arith $par_countm + 1
+	       par_countm=$as_val
+	       [ $par_countm -eq 10 ] && par_countm=0
+	       if [ $par_countm -eq 1 ]; then
+		  as_fn_arith $par_count + 1
+		  par_count=$as_val
+		  if mkdir $GCC_RUNTEST_PARALLELIZE_DIR/$par_count; then
+		     par_last=1
+		  else
+		     par_last=
+		  fi
+	       fi;;
+	 esac
+	 if [ -z "$par_last" ]; then
+	    as_fn_arith $glob_countn - 1
+	    glob_countn=$as_val
+	    continue
+	 fi
+      fi
+
       extraflags="-gnat95"
       grep $i $testdir/overflow.lst > /dev/null 2>&1
       if [ $? -eq 0 ]; then
@@ -254,7 +320,7 @@ for chapter in $chapters; do
       echo "BUILD $main" >> $dir/acats.log
       EXTERNAL_OBJECTS=""
       case $i in
-        cxb30*) EXTERNAL_OBJECTS="$dir/support/cxb30040.o $dir/support/cxb30060.o $dir/support/cxb30130.o $dir/support/cxb30131.o";;
+        cxb30*) EXTERNAL_OBJECTS="$dir_support/cxb30040.o $dir_support/cxb30060.o $dir_support/cxb30130.o $dir_support/cxb30131.o";;
         ca1020e) rm -f ca1020e_func1.adb ca1020e_func2.adb ca1020e_proc1.adb ca1020e_proc2.adb > /dev/null 2>&1;;
         ca14028) rm -f ca14028_func2.ads ca14028_func3.ads ca14028_proc1.ads ca14028_proc3.ads > /dev/null 2>&1;;
       esac
@@ -265,7 +331,7 @@ for chapter in $chapters; do
          continue
       fi
 
-      target_gnatmake $extraflags -I$dir/support $main >> $dir/acats.log 2>&1
+      target_gnatmake $extraflags -I$dir_support $main >> $dir/acats.log 2>&1
       if [ $? -ne 0 ]; then
          display "FAIL:	$i"
          failed="${failed}${i} "
@@ -290,12 +356,15 @@ for chapter in $chapters; do
             failed="${failed}${i} "
          else
             log "UNSUPPORTED:	$i"
-            glob_countn=`expr $glob_countn - 1`
-            glob_countu=`expr $glob_countu + 1`
+            as_fn_arith $glob_countn - 1
+            glob_countn=$as_val
+            as_fn_arith $glob_countu + 1
+            glob_countu=$as_val
          fi
       else
          log "PASS:	$i"
-         glob_countok=`expr $glob_countok + 1`
+         as_fn_arith $glob_countok + 1
+         glob_countok=$as_val
       fi
       clean_dir
    done
--- gcc/testsuite/lib/gcc-defs.exp.jj	2014-09-14 21:09:04.880497500 +0200
+++ gcc/testsuite/lib/gcc-defs.exp	2014-09-14 22:10:41.116165494 +0200
@@ -188,6 +188,81 @@ if { [info procs runtest_file_p] == "" }
     }
 }
 
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    global gcc_runtest_parallelize_counter
+    global gcc_runtest_parallelize_counter_minor
+    global gcc_runtest_parallelize_enable
+    global gcc_runtest_parallelize_dir
+    global gcc_runtest_parallelize_last
+
+    set gcc_runtest_parallelize_counter 0
+    set gcc_runtest_parallelize_counter_minor 0
+    set gcc_runtest_parallelize_enable 1
+    set gcc_runtest_parallelize_dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+    set gcc_runtest_parallelize_last 0
+
+    proc gcc_parallel_test_run_p { testcase } {
+	global gcc_runtest_parallelize_counter
+	global gcc_runtest_parallelize_counter_minor
+	global gcc_runtest_parallelize_enable
+	global gcc_runtest_parallelize_dir
+	global gcc_runtest_parallelize_last
+
+	if { $gcc_runtest_parallelize_enable == 0 } {
+	    return 1
+	}
+
+	# Only test the filesystem every 10th iteration
+	incr gcc_runtest_parallelize_counter_minor
+	if { $gcc_runtest_parallelize_counter_minor == 10 } {
+	    set gcc_runtest_parallelize_counter_minor 0
+	}
+	if { $gcc_runtest_parallelize_counter_minor != 1 } {
+	    #verbose -log "gcc_parallel_test_run_p $testcase $gcc_runtest_parallelize_counter $gcc_runtest_parallelize_last"
+	    return $gcc_runtest_parallelize_last
+	}
+
+	set path $gcc_runtest_parallelize_dir/$gcc_runtest_parallelize_counter
+
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    set gcc_runtest_parallelize_last 1
+	    #verbose -log "gcc_parallel_test_run_p $testcase $gcc_runtest_parallelize_counter 1"
+	    incr gcc_runtest_parallelize_counter
+	    return 1
+	}
+	set gcc_runtest_parallelize_last 0
+	#verbose -log "gcc_parallel_test_run_p $testcase $gcc_runtest_parallelize_counter 0"
+	incr gcc_runtest_parallelize_counter
+	return 0
+    }
+
+    proc gcc_parallel_test_enable { val } {
+	global gcc_runtest_parallelize_enable
+	set gcc_runtest_parallelize_enable $val
+    }
+
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    proc runtest_file_p { runtests testcase } {
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+	return [gcc_parallel_test_run_p $testcase]
+    }
+
+} else {
+
+    proc gcc_parallel_test_run_p { testcase } {
+	return 1
+    }
+
+    proc gcc_parallel_test_enable { val } {
+    }
+
+}
+
 # Like dg-options, but adds to the default options rather than replacing them.
 
 proc dg-additional-options { args } {
--- gcc/testsuite/objc.dg/gnu-encoding/gnu-encoding.exp.jj	2014-09-14 21:09:05.237495674 +0200
+++ gcc/testsuite/objc.dg/gnu-encoding/gnu-encoding.exp	2014-09-14 22:09:58.788381473 +0200
@@ -58,7 +58,9 @@ if { $status == 0 } then {
                 continue
             }
 
+	    gcc_parallel_test_enable 0
             dg-runtest $src "" $DEFAULT_CFLAGS
+	    gcc_parallel_test_enable 1
         }
     } else {
         warning "Could not execute objc.dg/gnu-encoding/struct-layout-encoding-1 generator"
--- gcc/objc/Make-lang.in.jj	2014-09-14 21:09:05.807492885 +0200
+++ gcc/objc/Make-lang.in	2014-09-14 21:25:17.771673151 +0200
@@ -96,7 +96,7 @@ lang_checks += check-objc
 # The following allows you to do 'make check-objc -j2'.  The
 # execute.exp tests will be run in parallel with all the other ones.
 lang_checks_parallelized += check-objc
-check_objc_parallelize = gnu-encoding.exp execute.exp exceptions.exp
+check_objc_parallelize = 6
 
 #\f
 # Install hooks:
--- libstdc++-v3/testsuite/libstdc++-abi/abi.exp.jj	2014-05-20 16:36:47.000000000 +0200
+++ libstdc++-v3/testsuite/libstdc++-abi/abi.exp	2014-09-15 14:23:06.308253078 +0200
@@ -18,6 +18,12 @@
 # this test. Or, hey, if we don't support this kind of symbol
 # versioning test: don't run it.
 
+# Only run whole abi.exp in one instance, serially.
+if ![gcc_parallel_test_run_p abi] {
+    return
+}
+gcc_parallel_test_enable 0
+
 if { [string match "*-*-darwin*" $target_triplet] } {
     set lib $blddir/src/.libs/libstdc++.dylib
 } else { 
@@ -30,6 +36,7 @@ v3-build_support
 if { (${v3-symver} == 0) || ![info exists baseline_dir] \
 	 || ![file exists $baseline_dir] \
 	 || ![file exists $lib] } {
+    gcc_parallel_test_enable 1
     return
 }
 
@@ -42,12 +49,13 @@ if ![file exists $baseline_file] {
     set baseline_file [file join $baseline_dir "baseline_symbols.txt"]
 }
 if ![file exists $baseline_file] {
+    gcc_parallel_test_enable 1
     return
 }
 send_log "Checking $lib against $baseline_file\n"
 
 # Figure out what symbols are defined by the active build of the library.
-remote_exec "build" "$objdir/../scripts/extract_symvers" \
+remote_exec "build" "$blddir/scripts/extract_symvers" \
     [list $lib "current_symbols.txt"]
 
 # Build the abi_check program.
@@ -62,3 +70,5 @@ set result [${tool}_load "./abi_check" \
 		[list "--check-verbose" "current_symbols.txt" \
 		     "baseline_symbols.txt"]]
 [lindex $result 0] "libstdc++-abi/abi_check"
+
+gcc_parallel_test_enable 1
--- libstdc++-v3/testsuite/libstdc++-xmethods/xmethods.exp.jj	2014-09-10 21:57:14.000000000 +0200
+++ libstdc++-v3/testsuite/libstdc++-xmethods/xmethods.exp	2014-09-15 15:31:50.277960368 +0200
@@ -16,6 +16,12 @@
 
 load_lib gdb-test.exp
 
+# Only run whole xmethods.exp in one instance, serially.
+if ![gcc_parallel_test_run_p xmethods] {
+    return
+}
+gcc_parallel_test_enable 0
+
 dg-init
 v3-build_support
 
@@ -30,7 +36,9 @@ if ![info exists ::env(GUALITY_GDB_NAME)
 }
 
 if {! [gdb_version_check_xmethods]} {
+    dg-finish
     unsupported "xmethods.exp"
+    gcc_parallel_test_enable 1
     return
 }
 
@@ -49,3 +57,4 @@ if [info exists guality_gdb_name] {
 }
 
 dg-finish
+gcc_parallel_test_enable 1
--- libstdc++-v3/testsuite/Makefile.am.jj	2014-09-14 21:08:57.067536354 +0200
+++ libstdc++-v3/testsuite/Makefile.am	2014-09-15 15:27:57.632110277 +0200
@@ -101,26 +101,37 @@ new-abi-baseline:
 	@test ! -f $*/site.exp || mv $*/site.exp $*/site.bak
 	@mv $*/site.exp.tmp $*/site.exp
 
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers1)))
+check_p_numbers3:=$(addprefix 0,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers3)))
+check_p_numbers5:=$(addprefix 0,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
+check_p_subdirs=$(wordlist 1,$(if $(GCC_TEST_PARALLEL_SLOTS),$(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers))
+check_DEJAGNU_normal_targets = $(addprefix check-DEJAGNUnormal,$(check_p_subdirs))
 $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
 check-DEJAGNU $(check_DEJAGNU_normal_targets): check-DEJAGNU%: site.exp
 	AR="$(AR)"; export AR; \
 	RANLIB="$(RANLIB)"; export RANLIB; \
-	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
-	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	if [ -z "$*" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  rm -rf normal-parallel || true; \
+	  mkdir normal-parallel; \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
-	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
-	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	  rm -rf normal-parallel || true; \
+	  for idx in $(check_p_subdirs); do \
+	    if [ -d normal$$idx ]; then \
+	      mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
+	      mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	    fi; \
 	  done; \
-	  mv -f libstdc++.sum libstdc++.sum.sep; \
-	  mv -f libstdc++.log libstdc++.log.sep; \
 	  $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \
-	    libstdc++.sum.sep normal[0-9]*/libstdc++.sum.sep > libstdc++.sum; \
+	    normal[0-9]*/libstdc++.sum.sep > libstdc++.sum; \
 	  $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh -L \
-	    libstdc++.log.sep normal[0-9]*/libstdc++.log.sep > libstdc++.log; \
+	    normal[0-9]*/libstdc++.log.sep > libstdc++.log; \
 	  exit 0; \
 	fi; \
 	srcdir=`$(am__cd) $(srcdir) && pwd`; export srcdir; \
@@ -128,47 +139,20 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	runtest=$(RUNTEST); \
 	if [ -z "$$runtest" ]; then runtest=runtest; fi; \
 	tool=libstdc++; \
-	dirs=; \
-	case "$*" in \
-	  normal0) \
-	    if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	      $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-			$(RUNTESTFLAGS) abi.exp prettyprinters.exp; \
-	    else echo "WARNING: could not find \`runtest'" 1>&2; :;\
-	    fi; \
-	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
-	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
-	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
-	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
-	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
-	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
-	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
-	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
-	  normal8) \
-	    dirs="`cd $$srcdir; echo 27_*/*`";; \
-	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
-	  normal10) \
-	    dirs="`cd $$srcdir; echo t*/*`";; \
-	esac; \
-	if [ -n "$*" ]; then cd "$*"; fi; \
+	if [ -n "$*" ]; then \
+	  if [ -f normal-parallel/finished ]; then rm -rf "$*"; exit 0; fi; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/normal-parallel; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR; \
+	  cd "$*"; \
+	fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	  if [ -n "$$dirs" ]; then \
-	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-		    $(RUNTESTFLAGS) \
-		    "conformance.exp=`echo $$dirs | sed 's/ /* /g;s/$$/*/'`"; \
-	  else \
-	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
+	  $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
 		    $(RUNTESTFLAGS); \
+	  if [ -n "$*" ]; then \
+	    touch $$GCC_RUNTEST_PARALLELIZE_DIR/finished; \
 	  fi; \
-	else echo "WARNING: could not find \`runtest'" 1>&2; :;\
+	else \
+	  echo "WARNING: could not find \`runtest'" 1>&2; :;\
 	fi
 
 check-am:
--- libstdc++-v3/testsuite/lib/libstdc++.exp.jj	2014-09-14 21:08:57.278535267 +0200
+++ libstdc++-v3/testsuite/lib/libstdc++.exp	2014-09-15 13:42:23.822366707 +0200
@@ -1823,3 +1823,78 @@ proc check_v3_target_little_endian { } {
 }
 
 set additional_prunes ""
+
+if { [info exists env(GCC_RUNTEST_PARALLELIZE_DIR)] \
+     && [info procs runtest_file_p] != [list] \
+     && [info procs gcc_parallelize_saved_runtest_file_p] == [list] } then {
+    global gcc_runtest_parallelize_counter
+    global gcc_runtest_parallelize_counter_minor
+    global gcc_runtest_parallelize_enable
+    global gcc_runtest_parallelize_dir
+    global gcc_runtest_parallelize_last
+
+    set gcc_runtest_parallelize_counter 0
+    set gcc_runtest_parallelize_counter_minor 0
+    set gcc_runtest_parallelize_enable 1
+    set gcc_runtest_parallelize_dir [getenv GCC_RUNTEST_PARALLELIZE_DIR]
+    set gcc_runtest_parallelize_last 0
+
+    proc gcc_parallel_test_run_p { testcase } {
+	global gcc_runtest_parallelize_counter
+	global gcc_runtest_parallelize_counter_minor
+	global gcc_runtest_parallelize_enable
+	global gcc_runtest_parallelize_dir
+	global gcc_runtest_parallelize_last
+
+	if { $gcc_runtest_parallelize_enable == 0 } {
+	    return 1
+	}
+
+	# Only test the filesystem every 10th iteration
+	incr gcc_runtest_parallelize_counter_minor
+	if { $gcc_runtest_parallelize_counter_minor == 10 } {
+	    set gcc_runtest_parallelize_counter_minor 0
+	}
+	if { $gcc_runtest_parallelize_counter_minor != 1 } {
+	    #verbose -log "gcc_parallel_test_run_p $testcase $gcc_runtest_parallelize_counter $gcc_runtest_parallelize_last"
+	    return $gcc_runtest_parallelize_last
+	}
+
+	set path $gcc_runtest_parallelize_dir/$gcc_runtest_parallelize_counter
+
+	if {![catch {open $path {RDWR CREAT EXCL} 0600} fd]} {
+	    close $fd
+	    set gcc_runtest_parallelize_last 1
+	    #verbose -log "gcc_parallel_test_run_p $testcase $gcc_runtest_parallelize_counter 1"
+	    incr gcc_runtest_parallelize_counter
+	    return 1
+	}
+	set gcc_runtest_parallelize_last 0
+	#verbose -log "gcc_parallel_test_run_p $testcase $gcc_runtest_parallelize_counter 0"
+	incr gcc_runtest_parallelize_counter
+	return 0
+    }
+
+    proc gcc_parallel_test_enable { val } {
+	global gcc_runtest_parallelize_enable
+	set gcc_runtest_parallelize_enable $val
+    }
+
+    rename runtest_file_p gcc_parallelize_saved_runtest_file_p
+    proc runtest_file_p { runtests testcase } {
+	if ![gcc_parallelize_saved_runtest_file_p $runtests $testcase] {
+	    return 0
+	}
+	return [gcc_parallel_test_run_p $testcase]
+    }
+
+} else {
+
+    proc gcc_parallel_test_run_p { testcase } {
+	return 1
+    }
+
+    proc gcc_parallel_test_enable { val } {
+    }
+
+}
--- libstdc++-v3/testsuite/Makefile.in.jj	2014-09-14 21:08:57.170535802 +0200
+++ libstdc++-v3/testsuite/Makefile.in	2014-09-15 15:29:57.323521363 +0200
@@ -301,7 +301,16 @@ lists_of_files = \
 
 extract_symvers = $(glibcxx_builddir)/scripts/extract_symvers
 baseline_subdir := $(shell $(CXX) $(baseline_subdir_switch))
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10)
+check_p_numbers0:=1 2 3 4 5 6 7 8 9
+check_p_numbers1:=0 $(check_p_numbers0)
+check_p_numbers2:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers1)))
+check_p_numbers3:=$(addprefix 0,$(check_p_numbers1)) $(check_p_numbers2)
+check_p_numbers4:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers3)))
+check_p_numbers5:=$(addprefix 0,$(check_p_numbers3)) $(check_p_numbers4)
+check_p_numbers6:=$(foreach i,$(check_p_numbers0),$(addprefix $(i),$(check_p_numbers5)))
+check_p_numbers:=$(check_p_numbers0) $(check_p_numbers2) $(check_p_numbers4) $(check_p_numbers6)
+check_p_subdirs=$(wordlist 1,$(if $(GCC_TEST_PARALLEL_SLOTS),$(GCC_TEST_PARALLEL_SLOTS),128),$(check_p_numbers))
+check_DEJAGNU_normal_targets = $(addprefix check-DEJAGNUnormal,$(check_p_subdirs))
 
 # Runs the testsuite, but in compile only mode.
 # Can be used to test sources with non-GNU FE's at various warning
@@ -559,19 +568,21 @@ $(check_DEJAGNU_normal_targets): check-D
 check-DEJAGNU $(check_DEJAGNU_normal_targets): check-DEJAGNU%: site.exp
 	AR="$(AR)"; export AR; \
 	RANLIB="$(RANLIB)"; export RANLIB; \
-	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
-	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	if [ -z "$*" ] && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  rm -rf normal-parallel || true; \
+	  mkdir normal-parallel; \
 	  $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \
-	  for idx in 0 1 2 3 4 5 6 7 8 9 10; do \
-	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
-	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	  rm -rf normal-parallel || true; \
+	  for idx in $(check_p_subdirs); do \
+	    if [ -d normal$$idx ]; then \
+	      mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
+	      mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
+	    fi; \
 	  done; \
-	  mv -f libstdc++.sum libstdc++.sum.sep; \
-	  mv -f libstdc++.log libstdc++.log.sep; \
 	  $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \
-	    libstdc++.sum.sep normal[0-9]*/libstdc++.sum.sep > libstdc++.sum; \
+	    normal[0-9]*/libstdc++.sum.sep > libstdc++.sum; \
 	  $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh -L \
-	    libstdc++.log.sep normal[0-9]*/libstdc++.log.sep > libstdc++.log; \
+	    normal[0-9]*/libstdc++.log.sep > libstdc++.log; \
 	  exit 0; \
 	fi; \
 	srcdir=`$(am__cd) $(srcdir) && pwd`; export srcdir; \
@@ -579,47 +590,20 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar
 	runtest=$(RUNTEST); \
 	if [ -z "$$runtest" ]; then runtest=runtest; fi; \
 	tool=libstdc++; \
-	dirs=; \
-	case "$*" in \
-	  normal0) \
-	    if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	      $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-			$(RUNTESTFLAGS) abi.exp prettyprinters.exp; \
-	    else echo "WARNING: could not find \`runtest'" 1>&2; :;\
-	    fi; \
-	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/*`";; \
-	  normal1) \
-	    dirs="`cd $$srcdir; echo [ab]* de* [ep]*/*`";; \
-	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[01]_*/*`";; \
-	  normal3) \
-	    dirs="`cd $$srcdir; echo 22_*/*`";; \
-	  normal4) \
-	    dirs="`cd $$srcdir; echo 23_*/[a-km-tw-z]*`";; \
-	  normal5) \
-	    dirs="`cd $$srcdir; echo 23_*/[luv]*`";; \
-	  normal6) \
-	    dirs="`cd $$srcdir; echo 2[459]_*/*`";; \
-	  normal7) \
-	    dirs="`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`";; \
-	  normal8) \
-	    dirs="`cd $$srcdir; echo 27_*/*`";; \
-	  normal9) \
-	    dirs="`cd $$srcdir; echo 28_*/[ab]*`";; \
-	  normal10) \
-	    dirs="`cd $$srcdir; echo t*/*`";; \
-	esac; \
-	if [ -n "$*" ]; then cd "$*"; fi; \
+	if [ -n "$*" ]; then \
+	  if [ -f normal-parallel/finished ]; then rm -rf "$*"; exit 0; fi; \
+	  GCC_RUNTEST_PARALLELIZE_DIR=`${PWD_COMMAND}`/normal-parallel; \
+	  export GCC_RUNTEST_PARALLELIZE_DIR; \
+	  cd "$*"; \
+	fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-	  if [ -n "$$dirs" ]; then \
-	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
-		    $(RUNTESTFLAGS) \
-		    "conformance.exp=`echo $$dirs | sed 's/ /* /g;s/$$/*/'`"; \
-	  else \
-	    $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
+	  $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) \
 		    $(RUNTESTFLAGS); \
+	  if [ -n "$*" ]; then \
+	    touch $$GCC_RUNTEST_PARALLELIZE_DIR/finished; \
 	  fi; \
-	else echo "WARNING: could not find \`runtest'" 1>&2; :;\
+	else \
+	  echo "WARNING: could not find \`runtest'" 1>&2; :;\
 	fi
 
 check-am:


	Jakub

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Sep 15, 2014, at 9:05 AM, Jakub Jelinek <jakub@redhat.com> wrote:

All the updates sound good.

> Regtested on x86_64-linux, without the patch toplevel make -k check
> took 8hrs3minutes (don't have time data for that run),

This confuses me, but, no matter.  Isn’t 8hrs time data?  :-)

> patch toplevel make -j48 -k check took:
> real    40m21.984s
> user    341m51.675s
> sys     112m46.993s
> and with the patch make -j48 -k check took:
> real    32m22.066s
> user    355m1.788s
> sys     117m5.809s

These numbers are useful to try and ensure the overhead (scaling factor) is reasonable, thanks.

> Is this version ok for trunk?

Ok.

Thanks for all your work.

Re: [PATCH] gcc parallel make check

[Richard Biener]

On Mon, Sep 15, 2014 at 7:44 PM, Mike Stump <mikestump@comcast.net> wrote:
> On Sep 15, 2014, at 9:05 AM, Jakub Jelinek <jakub@redhat.com> wrote:
>
> All the updates sound good.
>
>> Regtested on x86_64-linux, without the patch toplevel make -k check
>> took 8hrs3minutes (don't have time data for that run),
>
> This confuses me, but, no matter.  Isn’t 8hrs time data?  :-)
>
>> patch toplevel make -j48 -k check took:
>> real    40m21.984s
>> user    341m51.675s
>> sys     112m46.993s
>> and with the patch make -j48 -k check took:
>> real    32m22.066s
>> user    355m1.788s
>> sys     117m5.809s
>
> These numbers are useful to try and ensure the overhead (scaling factor) is reasonable, thanks.

A nice improvement indeed.  The patched result is 15 times faster
than the serial unpatched run.  So there is room for improvement
(I wouldn't say the scaling factor is reasonable - with accounting for
overhead I'd expect it should be possible to arrive at a factor of 32
here at least).

So - where's the "serial" parts of the testing run?

Thanks for the improvements btw!

Richard.

>> Is this version ok for trunk?
>
> Ok.
>
> Thanks for all your work.

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Tue, Sep 16, 2014 at 11:20:37AM +0200, Richard Biener wrote:
> > This confuses me, but, no matter.  Isn’t 8hrs time data?  :-)

It is, but not time(1) data, just wall clock computed from subtracting
mtimes of my make check output log and make -j48 bootstrap log.

> >> patch toplevel make -j48 -k check took:
> >> real    40m21.984s
> >> user    341m51.675s
> >> sys     112m46.993s
> >> and with the patch make -j48 -k check took:
> >> real    32m22.066s
> >> user    355m1.788s
> >> sys     117m5.809s
> >
> > These numbers are useful to try and ensure the overhead (scaling factor) is reasonable, thanks.
> 
> A nice improvement indeed.  The patched result is 15 times faster
> than the serial unpatched run.  So there is room for improvement

Note, the box used was oldish AMD 16-core, no ht, box, haven't tried it on anything
more parallel, also it was normal hard disk, etc.  No idea whether anything
from this is relevant to that though.
Some CPU time goes into the expect processes, I can retry the build tonight
and grab also time(1) info from make -k check to see the user/sys times for
serial testing.

	Jakub

Re: [PATCH] gcc parallel make check

[Richard Biener]

On Tue, Sep 16, 2014 at 11:28 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Tue, Sep 16, 2014 at 11:20:37AM +0200, Richard Biener wrote:
>> > This confuses me, but, no matter.  Isn’t 8hrs time data?  :-)
>
> It is, but not time(1) data, just wall clock computed from subtracting
> mtimes of my make check output log and make -j48 bootstrap log.
>
>> >> patch toplevel make -j48 -k check took:
>> >> real    40m21.984s
>> >> user    341m51.675s
>> >> sys     112m46.993s
>> >> and with the patch make -j48 -k check took:
>> >> real    32m22.066s
>> >> user    355m1.788s
>> >> sys     117m5.809s
>> >
>> > These numbers are useful to try and ensure the overhead (scaling factor) is reasonable, thanks.
>>
>> A nice improvement indeed.  The patched result is 15 times faster
>> than the serial unpatched run.  So there is room for improvement
>
> Note, the box used was oldish AMD 16-core, no ht, box, haven't tried it on anything

Ah, I assumed -j48 testing means you have 48 cores.  I usually test
with -j12 on my 6-core HT-enabled box.  A factor 15 scaling for 16
CPUs is of course close to the best we can achieve.

Richard.

> more parallel, also it was normal hard disk, etc.  No idea whether anything
> from this is relevant to that though.
> Some CPU time goes into the expect processes, I can retry the build tonight
> and grab also time(1) info from make -k check to see the user/sys times for
> serial testing.
>
>         Jakub

RE: [PATCH] gcc parallel make check

[VandeVondele Joost]

>> > These numbers are useful to try and ensure the overhead (scaling factor) is reasonable, thanks.
>>
>> A nice improvement indeed.  The patched result is 15 times faster
>> than the serial unpatched run.  So there is room for improvement
>
> Note, the box used was oldish AMD 16-core, no ht, box, haven't tried it on anything

on a 32 core box, no ht, I see these timings:

time make -j32 -k check >& log.check32 ; time make -j8 -k check >& log.check8

real	18m14.562s
user	260m21.578s
sys	264m26.042s

real	41m33.210s
user	233m4.563s
sys	72m11.429s

so it is not quite reaching the ideal 4x speedup. Counting the number of 'expect' processes they are nicely at around 32 and 8 for the full test, with only a very short tail near the end. So, there might be some overhead somewhere. Total user time is similar, but time in sys goes up.

Re: [PATCH] gcc parallel make check

[Christophe Lyon]

Hi Jakub,


On 15 September 2014 18:05, Jakub Jelinek <jakub@redhat.com> wrote:
[...]
>  # For parallelized check-% targets, this decides whether parallelization
>  # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
>  # but optional --target_board or --extra_opts arguments).  If desirable,
>  # recursive make is run with check-parallel-$lang{,1,2,3,4,5} etc. goals,
>  # which can be executed in parallel, as they are run in separate directories.
> -# check-parallel-$lang{1,2,3,4,5} etc. goals invoke runtest with the longest
> -# running *.exp files from the testsuite, as determined by check_$lang_parallelize
> -# variable.  The check-parallel-$lang goal in that case invokes runtest with
> -# all the remaining *.exp files not handled by the separate goals.
> +# check-parallel-$lang{,1,2,3,4,5} etc. goals invoke runtest with
> +# GCC_RUNTEST_PARALLELIZE_DIR var in the environment and runtest_file_p
> +# dejaGNU procedure is overridden to additionally synchronize through
> +# a $lang-parallel directory which tests will be run by which runtest instance.
>  # Afterwards contrib/dg-extract-results.sh is used to merge the sum and log
>  # files.  If parallelization isn't desirable, only one recursive make
>  # is run with check-parallel-$lang goal and check_$lang_parallelize variable
> @@ -3662,76 +3645,60 @@ check_p_subdirs=$(wordlist 1,$(words $(c
>  # to lang_checks_parallelized variable and define check_$lang_parallelize
>  # variable (see above check_gcc_parallelize description).
>  $(lang_checks_parallelized): check-% : site.exp
> -       @if [ -z "$(filter-out --target_board=%,$(filter-out --extra_opts%,$(RUNTESTFLAGS)))" ] \

Since you removed this test, the comment above is not longer accurate:
setting RUNTESTFLAGS to whatever value no longer disables
parallelization.

Which leads me to discuss a bug I faced after you committed this
change: I am testing a patch which bring a series of new tests.
$ RUNTESTFLAGS=my.exp make -jN check (in fact the 'make -j' is
embedded in a larger build script)

my.exp contains the following construct which is often used in the testsuite:
==========
foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]] {
    # If we're only testing specific files and this isn't one of them,
skip it.
    if ![runtest_file_p $runtests $src] then {
        continue
    }
    c-torture-execute $src $additional_flags
    gcc-dg-runtest $src "" $additional_flags
}
==========
Note that gcc-dg-runtest calls runtest_file_p too.

What I observed is that if I use -j1, all my .c files get tested,
while with N>2 some of them are silently skipped.

It took me a while to figure out that it's because gcc-dg-runtest
calls runtest_file_p, which means that runtest_file_p is called twice
when the 1st invocation returns 1, and only once when the 1st
invocation returns 0.

For example, if we have pid0, pid1 the concurrent runtest processes,
and file0.c, file1.c, .... the testcases, then:
* pid0 decides to keep file0.c file1.c file2.c file3.c file4.c. Since
the above loop calls runtest_file_p twice for each, we reach the
"minor" counter of 10.
* in the mean time, pid1 decides to skip file0.c, file1.c ... file9.c
since it calls runtest_file_p only once for each
* pid1 increments its parallel counter to 1, and create the new testing subdir
* pid1 decides to keep file10, file11, file12, file13 and file14
(again, 2 calls to runtest_file_p per testcase)
* pid0 increments its parallel counter to 1, and decides it has to skip it
* pid0 thus decides to skip file5, file6, file7, ... file14, calling
runtest_file_p once for each
* etc...

In the end, we have ignored file5...file9

I'm not sure why you have made special cases for some of the existing
*.exp when you forced them to disable parallelization.
Was it to handle such cases?

I'm not sure about the next step:
- should I modify my .exp file?
- should you modify gcc_parallel_test_run_p?

Even if I have to modify my .exp file, I think this is error prone,
and others could introduce a similar construct in the future.

Thanks,

Christophe.

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Oct 10, 2014 at 04:09:39PM +0200, Christophe Lyon wrote:
> my.exp contains the following construct which is often used in the testsuite:
> ==========
> foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]] {
>     # If we're only testing specific files and this isn't one of them,
> skip it.
>     if ![runtest_file_p $runtests $src] then {
>         continue
>     }
>     c-torture-execute $src $additional_flags
>     gcc-dg-runtest $src "" $additional_flags
> }
> ==========
> Note that gcc-dg-runtest calls runtest_file_p too.

Such my.exp is invalid, you need to guarantee gcc_parallel_test_run_p
is run the same number of times in all instances unless
gcc_parallel_test_enable has been disabled.

See the patches I've posted when adding the fine-grained parallelization,
e.g. go testsuite has been fixed that way, etc.
So, in your above example, you'd need:
    gcc_parallel_test_enable 0
line before c-torture-execute and
    gcc_parallel_test_enable 1
line after gcc-dg-runtest.  That way, if runtest_file_p says the test should
be scheduled by current instance, all the subtests will be run there.

If my.exp is part of gcc/testsuite, I'm sorry for missing it, if it is
elsewhere, just fix it up.

Note, there are #verbose lines in gcc_parallel_test_run_p, you can uncomment
them and through sed on the log files verify that each instance performs the
same parallelization checks (same strings).

	Jakub

Re: [PATCH] gcc parallel make check

[Christophe Lyon]

On 10 October 2014 16:19, Jakub Jelinek <jakub@redhat.com> wrote:
> On Fri, Oct 10, 2014 at 04:09:39PM +0200, Christophe Lyon wrote:
>> my.exp contains the following construct which is often used in the testsuite:
>> ==========
>> foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]] {
>>     # If we're only testing specific files and this isn't one of them,
>> skip it.
>>     if ![runtest_file_p $runtests $src] then {
>>         continue
>>     }
>>     c-torture-execute $src $additional_flags
>>     gcc-dg-runtest $src "" $additional_flags
>> }
>> ==========
>> Note that gcc-dg-runtest calls runtest_file_p too.
>
> Such my.exp is invalid, you need to guarantee gcc_parallel_test_run_p
> is run the same number of times in all instances unless
> gcc_parallel_test_enable has been disabled.

Thanks for your prompt answer.

Is this documented somewhere, so that such cases do not happen in the future?


> See the patches I've posted when adding the fine-grained parallelization,
> e.g. go testsuite has been fixed that way, etc.
> So, in your above example, you'd need:
>     gcc_parallel_test_enable 0
> line before c-torture-execute and
>     gcc_parallel_test_enable 1
> line after gcc-dg-runtest.  That way, if runtest_file_p says the test should
> be scheduled by current instance, all the subtests will be run there.
>
> If my.exp is part of gcc/testsuite, I'm sorry for missing it, if it is
> elsewhere, just fix it up.

It's in a patch which has been under review for quite some time
(started before your change), that's why you missed it.

> Note, there are #verbose lines in gcc_parallel_test_run_p, you can uncomment
> them and through sed on the log files verify that each instance performs the
> same parallelization checks (same strings).
Yep, I saw those and also added other traces of my own :-)


What about my remark about:
>  # For parallelized check-% targets, this decides whether parallelization
>  # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
>  # but optional --target_board or --extra_opts arguments).  If desirable,
I think it should be removed from gcc/Makefile.in

Thanks,

Christophe.

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Fri, Oct 10, 2014 at 04:50:47PM +0200, Christophe Lyon wrote:
> On 10 October 2014 16:19, Jakub Jelinek <jakub@redhat.com> wrote:
> > On Fri, Oct 10, 2014 at 04:09:39PM +0200, Christophe Lyon wrote:
> >> my.exp contains the following construct which is often used in the testsuite:
> >> ==========
> >> foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]] {
> >>     # If we're only testing specific files and this isn't one of them,
> >> skip it.
> >>     if ![runtest_file_p $runtests $src] then {
> >>         continue
> >>     }
> >>     c-torture-execute $src $additional_flags
> >>     gcc-dg-runtest $src "" $additional_flags
> >> }
> >> ==========
> >> Note that gcc-dg-runtest calls runtest_file_p too.
> >
> > Such my.exp is invalid, you need to guarantee gcc_parallel_test_run_p
> > is run the same number of times in all instances unless
> > gcc_parallel_test_enable has been disabled.
> 
> Thanks for your prompt answer.
> 
> Is this documented somewhere, so that such cases do not happen in the future?

Feel free to submit a documentation patch.

> It's in a patch which has been under review for quite some time
> (started before your change), that's why you missed it.

Ah, ok.

> What about my remark about:
> >  # For parallelized check-% targets, this decides whether parallelization
> >  # is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
> >  # but optional --target_board or --extra_opts arguments).  If desirable,
> I think it should be removed from gcc/Makefile.in

Only the " and RUNTESTFLAGS ... arguments" part of that.  Patch preapproved.

	Jakub

[PATCH] gcc/testsuite: guality.exp: Fix `test_counts' restoration

[Maciej W. Rozycki]

On Mon, 15 Sep 2014, Jakub Jelinek wrote:

> 2014-09-14  Jakub Jelinek  <jakub@redhat.com>
> 
> gcc/testsuite/
> 	* g++.dg/guality/guality.exp (check_guality): Save/restore
> 	test_counts array around the body of the procedure.
> 	* gcc.dg/guality/guality.exp (check_guality): Likewise.
[...]
> --- gcc/testsuite/g++.dg/guality/guality.exp.jj	2014-09-14 21:09:04.767498056 +0200
> +++ gcc/testsuite/g++.dg/guality/guality.exp	2014-09-14 21:25:17.768673222 +0200
> @@ -14,6 +14,11 @@ if { [istarget "powerpc-ibm-aix*"] } {
>  }
>  
>  proc check_guality {args} {
> +    # Don't count check_guality as PASS, or FAIL etc., that would make
> +    # the total PASS count dependent on how many parallel runtest invocations
> +    # ran guality.exp.  So save the counts first and restore them afterwards.
> +    global test_counts
> +    array set saved_test_counts [array get test_counts]
>      set result [eval check_compile guality_check executable $args "-g -O0"]
>      set lines [lindex $result 0]
>      set output [lindex $result 1]
> @@ -23,6 +28,7 @@ proc check_guality {args} {
>        set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
>      }
>      remote_file build delete $output
> +    array get test_counts [array get saved_test_counts]
>      return $ret
>  }
>  

 ^-- typo here I believe, compare the corresponding change below:

> --- gcc/testsuite/gcc.dg/guality/guality.exp.jj	2014-09-14 21:09:05.362495088 +0200
> +++ gcc/testsuite/gcc.dg/guality/guality.exp	2014-09-14 21:25:17.769673197 +0200
> @@ -14,6 +14,11 @@ if { [istarget "powerpc-ibm-aix*"] } {
>  }
>  
>  proc check_guality {args} {
> +    # Don't count check_guality as PASS, or FAIL etc., that would make
> +    # the total PASS count dependent on how many parallel runtest invocations
> +    # ran guality.exp.  So save the counts first and restore them afterwards.
> +    global test_counts
> +    array set saved_test_counts [array get test_counts]
>      set result [eval check_compile guality_check executable $args "-g -O0"]
>      set lines [lindex $result 0]
>      set output [lindex $result 1]
> @@ -23,6 +28,7 @@ proc check_guality {args} {
>        set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
>      }
>      remote_file build delete $output
> +    array set test_counts [array get saved_test_counts]
>      return $ret
>  }
>  

 OK to apply?

2014-11-14  Maciej W. Rozycki  <macro@codesourcery.com>

	gcc/testsuite/ 
	* g++.dg/guality/guality.exp (check_guality): Fix `test_counts' 
	restoration.

  Maciej

gcc-test-guality.patch
Index: gcc-fsf-trunk-quilt/gcc/testsuite/g++.dg/guality/guality.exp
===================================================================
--- gcc-fsf-trunk-quilt.orig/gcc/testsuite/g++.dg/guality/guality.exp	2014-11-14 18:33:47.000000000 +0000
+++ gcc-fsf-trunk-quilt/gcc/testsuite/g++.dg/guality/guality.exp	2014-11-14 20:18:35.038856372 +0000
@@ -28,7 +28,7 @@ proc check_guality {args} {
       set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
     }
     remote_file build delete $output
-    array get test_counts [array get saved_test_counts]
+    array set test_counts [array get saved_test_counts]
     return $ret
 }
 

Re: [PATCH] gcc/testsuite: guality.exp: Fix `test_counts' restoration

[Jakub Jelinek]

On Fri, Nov 14, 2014 at 09:01:25PM +0000, Maciej W. Rozycki wrote:
> 2014-11-14  Maciej W. Rozycki  <macro@codesourcery.com>
> 
> 	gcc/testsuite/ 
> 	* g++.dg/guality/guality.exp (check_guality): Fix `test_counts' 
> 	restoration.

Ok, thanks.

> --- gcc-fsf-trunk-quilt.orig/gcc/testsuite/g++.dg/guality/guality.exp	2014-11-14 18:33:47.000000000 +0000
> +++ gcc-fsf-trunk-quilt/gcc/testsuite/g++.dg/guality/guality.exp	2014-11-14 20:18:35.038856372 +0000
> @@ -28,7 +28,7 @@ proc check_guality {args} {
>        set ret [string match "*1 PASS, 0 FAIL, 0 UNRESOLVED*" $execout]
>      }
>      remote_file build delete $output
> -    array get test_counts [array get saved_test_counts]
> +    array set test_counts [array get saved_test_counts]
>      return $ret
>  }
>  

	Jakub

Re: [PATCH] gcc/testsuite: guality.exp: Fix `test_counts' restoration

[Maciej W. Rozycki]

On Fri, 14 Nov 2014, Jakub Jelinek wrote:

> > 	gcc/testsuite/ 
> > 	* g++.dg/guality/guality.exp (check_guality): Fix `test_counts' 
> > 	restoration.
> 
> Ok, thanks.

 Applied, thanks.

  Maciej

Re: [PATCH] gcc parallel make check

[Tom de Vries]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

On 15-09-14 18:05, Jakub Jelinek wrote:
> libstdc++-v3/
> 	* testsuite/Makefile.am (check_p_numbers0, check_p_numbers1,
> 	check_p_numbers2, check_p_numbers3, check_p_numbers4,
> 	check_p_numbers5, check_p_numbers6, check_p_numbers,
> 	check_p_subdirs): New variables.
> 	(check_DEJAGNU_normal_targets): Use check_p_subdirs.
> 	(check-DEJAGNU): Rewritten so that for parallelized
> 	testing each job runs all the *.exp files, with
> 	GCC_RUNTEST_PARALLELIZE_DIR set in environment.
> 	* testsuite/Makefile.in: Regenerated.
> 	* testsuite/lib/libstdc++.exp (gcc_parallel_test_run_p,
> 	gcc_parallel_test_enable): New procedures.  If
> 	GCC_RUNTEST_PARALLELIZE_DIR is set in environment, override
> 	runtest_file_p to invoke also gcc_parallel_test_run_p.
> 	* testsuite/libstdc++-abi/abi.exp: Run all the tests serially
> 	by the first parallel runtest encountering it.  Fix up path
> 	of the extract_symvers script.
> 	* testsuite/libstdc++-xmethods/xmethods.exp: Run all the tests
> 	serially by the first parallel runtest encountering it.  Run
> 	dg-finish even in case of error.

When comparing test results of patch builds with test results of reference 
builds, the only differences I'm seeing are random differences in amount of 
'UNSUPPORTED: prettyprinter.exp'.

This patch fixes that by ensuring that we print that unsupported message only once.

The resulting test result comparison diff is:
...
--- without/FAIL  2014-11-24 17:46:32.202673282 +0100
+++ with/FAIL     2014-11-25 13:45:15.636131571 +0100
  libstdc++-v3/testsuite/libstdc++.sum:UNSUPPORTED: prettyprinters.exp
-libstdc++-v3/testsuite/libstdc++.sum:UNSUPPORTED: prettyprinters.exp
-libstdc++-v3/testsuite/libstdc++.sum:UNSUPPORTED: prettyprinters.exp
-libstdc++-v3/testsuite/libstdc++.sum:UNSUPPORTED: prettyprinters.exp
-libstdc++-v3/testsuite/libstdc++.sum:UNSUPPORTED: prettyprinters.exp
  libstdc++-v3/testsuite/libstdc++.sum:UNSUPPORTED: xmethods.exp
...

Furthermore, the patch adds a dg-finish in case the prettyprinters.exp file is 
unsupported, which AFAIU is also required in that case.

Bootstrapped and reg-tested on x86_64.

OK for trunk/stage3?

Thanks,
- Tom



[-- Attachment #2: 0001-Ensure-single-UNSUPPORTED-prettyprinters.exp-message.patch --]
[-- Type: text/x-patch, Size: 979 bytes --]

2014-11-25  Tom de Vries  <tom@codesourcery.com>

	* testsuite/libstdc++-prettyprinters/prettyprinters.exp: Add missing
	dg-finish.  Only print unsupported message once.
---
 libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp b/libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp
index a57660f..e5be5b5 100644
--- a/libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp
+++ b/libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp
@@ -30,7 +30,14 @@ if ![info exists ::env(GUALITY_GDB_NAME)] {
 }
 
 if {! [gdb_version_check]} {
+    dg-finish
+    # Only print unsupported message in one instance.
+    if ![gcc_parallel_test_run_p prettyprinters] {
+	return
+    }
+    gcc_parallel_test_enable 0
     unsupported "prettyprinters.exp"
+    gcc_parallel_test_enable 1
     return
 }
 
-- 
1.9.1

Re: [PATCH] gcc parallel make check

[Jakub Jelinek]

On Tue, Nov 25, 2014 at 03:27:40PM +0100, Tom de Vries wrote:
> This patch fixes that by ensuring that we print that unsupported message only once.
> 
> The resulting test result comparison diff is:
> 2014-11-25  Tom de Vries  <tom@codesourcery.com>
> 
> 	* testsuite/libstdc++-prettyprinters/prettyprinters.exp: Add missing
> 	dg-finish.  Only print unsupported message once.

LGTM.

> --- a/libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp
> +++ b/libstdc++-v3/testsuite/libstdc++-prettyprinters/prettyprinters.exp
> @@ -30,7 +30,14 @@ if ![info exists ::env(GUALITY_GDB_NAME)] {
>  }
>  
>  if {! [gdb_version_check]} {
> +    dg-finish
> +    # Only print unsupported message in one instance.
> +    if ![gcc_parallel_test_run_p prettyprinters] {
> +	return
> +    }
> +    gcc_parallel_test_enable 0
>      unsupported "prettyprinters.exp"
> +    gcc_parallel_test_enable 1
>      return
>  }
>  
> -- 
> 1.9.1
> 


	Jakub

Re: [PATCH] gcc parallel make check

[Mike Stump]

On Nov 25, 2014, at 12:15 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Tue, Nov 25, 2014 at 03:27:40PM +0100, Tom de Vries wrote:
>> This patch fixes that by ensuring that we print that unsupported message only once.
>> 
>> The resulting test result comparison diff is:
>> 2014-11-25  Tom de Vries  <tom@codesourcery.com>
>> 
>>    * testsuite/libstdc++-prettyprinters/prettyprinters.exp: Add missing
>>    dg-finish.  Only print unsupported message once.
> 
> LGTM.

That is the approval.  :-)

Re: [PATCH] gcc parallel make check

[Tom Tromey]

>>>>> "Jakub" == Jakub Jelinek <jakub@redhat.com> writes:

Jakub> I fear that is going to be too expensive, because e.g. all the
Jakub> caching that dejagnu and our tcl stuff does would be gone, all
Jakub> the tests for lp64 etc.  would need to be repeated for each test.

In gdb I arranged to have this stuff saved in a special cache directory.
See gdb/testsuite/lib/cache.exp for the mechanism.

Tom