[contrib] dg-extract-test-results.sh, to combine or extract .sum files

[contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Janis Johnson]

This proposed addition to contrib/ extracts data for a particular tool
and one or more test variants from one or more test summary (.sum) files
and combines them into a new test summary file.  It's meant to be used
for combining the results of tests that are run separately so those
combined results can be used for comparisons or summaries, but it can
also be used to separate a single set of results for multiple test
variants in order to compare results across those variants.

I've been using this quite a bit with results from my attempts to
paralellize the compiler tests on GNU/Linux.  I've tried to make it
portable  but it's quite possible that it isn't.  I'd appreicate having
other people try it out, perhaps just by running a .sum file through it
to make sure the result is the same as the original file.

2008-10-15  Janis Johnson  <janis187@us.ibm.com>

	* dg-extract-results.sh: New.

Index: contrib/dg-extract-results.sh
===================================================================
--- contrib/dg-extract-results.sh	(revision 0)
+++ contrib/dg-extract-results.sh	(revision 0)
@@ -0,0 +1,279 @@
+#! /bin/sh
+
+# For a specified tool and optional list of test variants, extract
+# test results from one or more test summary (.sum) files and combine
+# the results into a new test summary file, sent to the standard output.
+# The resulting file can be used with test result comparison scripts for
+# results from tests that were run in parallel.  See usage() below.
+
+# Copyright (C) 2008 Free Software Foundation
+# Contributed by Janis Johnson <janis187@us.ibm.com>
+#
+# This file is part of GCC. 
+# 
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version. 
+# 
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+PROGNAME=dg-extract-results.sh
+
+usage() {
+  cat <<EOF >&2
+Usage: $PROGNAME [-t tool] [-l variant-list] sum-file ...
+
+    tool           The tool (e.g. g++, libffi) for which to create a
+                   new test summary file.  If not specified then all
+                   specified sum files must be for the same tool.
+    variant-list   One or more test variant names.  If the list is
+                   not specified then one is constructed from all
+                   variants in the files for <tool>.
+    sum-file       A test summary file with the format of those
+                   created by runtest from DejaGnu.
+EOF
+}
+
+# Write a message to the standard error.
+
+msg() {
+  echo "$@" >&2
+}
+
+# Parse the command-line options.
+
+VARIANTS=""
+TOOL=""
+
+while getopts "l:t:" ARG; do
+  case $ARG in
+  l)  VARIANTS="${VARIANTS} ${OPTARG}";;
+  t)  test -z "$TOOL" || (msg "${PROGNAME}: only one tool can be specified"; exit 1);
+      TOOL="${OPTARG}";;
+  \?) usage; exit 0;;
+  esac
+done
+shift `expr ${OPTIND} - 1`
+
+if test $# -lt 1 ; then
+  usage
+  exit 1
+fi
+
+TMPDIR=${TMPDIR-/tmp}
+SUM_FILES="$@"
+FIRST_SUM=$1
+TMP=${TMPDIR}/dg-combine-results-$$
+
+# Find a good awk.
+
+if test -z "$AWK" ; then
+  for AWK in gawk nawk awk
+  do 
+    if type $AWK 2>&1 | grep 'not found' > /dev/null 2>&1 ; then
+      :
+    else
+      break
+    fi
+  done
+fi
+
+# Verify that the specified summary files exist.
+
+ERROR=0
+for FILE in $SUM_FILES
+do
+  if ! test -f $FILE ; then
+    msg "${PROGNAME}: file $FILE does not exist."
+    ERROR=1
+  fi
+done
+test $ERROR -eq 0 || exit 1
+
+if [ -z "$TOOL" ]; then
+  # If no tool was specified, all specified summary files must be for
+  # the same tool.
+
+  CNT=`grep '=== .* tests ===' $SUM_FILES | awk '{ print $3 }' | sort -u | wc -l`
+  if [ $CNT -eq 1 ]; then
+    TOOL=`grep '=== .* tests ===' $FIRST_SUM | awk '{ print $2 }'`
+  else
+    msg "${PROGNAME}: sum files are for multiple tools, specify a tool"
+    msg ""
+    usage
+    exit 1
+  fi
+else 
+  # Ignore the specified summary files that are not for this tool.  This
+  # should keep the relevant files in the same order.
+
+  SUM_FILES=`grep -l "=== $TOOL" $SUM_FILES`
+  if test -z "$SUM_FILES" ; then
+    msg "${PROGNAME}: none of the specified files are results for $TOOL"
+    exit 1
+  fi
+fi
+
+# If no variants were specified, find all variants in the remaining
+# summary files.  Otherwise, ignore specified variants that aren't in
+# any of those summary files.
+
+if test -z "$VARIANTS" ; then
+  VAR_AWK=${TMP}-variants.awk
+  cat <<EOF > $VAR_AWK
+/^Schedule of variations:/      { in_vars=1; next }
+/^$/                            { in_vars=0 }
+/^Running target/               { exit }
+{ if (in_vars==1) print \$1; else next }
+EOF
+
+  rm -f ${TMP}-list
+  touch ${TMP}-list
+  for FILE in $SUM_FILES; do
+    $AWK -f $VAR_AWK $FILE >> ${TMP}-list
+  done
+  VARIANTS="`sort -u ${TMP}-list`"
+else
+  VARS="$VARIANTS"
+  VARIANTS=""
+  for VAR in $VARS
+  do
+    grep -q "Running target $VAR" $SUM_FILES && VARIANTS="$VARIANTS $VAR"
+  done
+fi
+
+# Find out if we have more than one variant, or any at all.
+
+VARIANT_COUNT=0
+for VAR in $VARIANTS
+do
+  VARIANT_COUNT=`expr $VARIANT_COUNT + 1`
+done
+
+if test $VARIANT_COUNT -eq 0 ; then
+  msg "${PROGNAME}: no file for $TOOL has results for the specified variants"
+  exit 1
+fi
+
+# Write the begining of the combined summary file.
+
+head -n 2 $FIRST_SUM
+echo
+echo "		=== $TOOL tests ==="
+echo
+echo "Schedule of variations:"
+for VAR in $VARIANTS
+do
+  echo "    $VAR"
+done
+echo
+
+# For each test variant for the tool, copy test reports from each of the
+# summary files.  Set up two awk scripts from within the loop to
+# initialize VAR and TOOL with the script, rather than assuming that the
+# available version of awk can pass variables from the command line.
+
+for VAR in $VARIANTS
+do
+  GUTS_AWK=${TMP}-guts.awk
+  cat << EOF > $GUTS_AWK
+BEGIN {
+  variant="$VAR"
+  firstvar=1
+}
+/^Running target /	{ curvar = \$3;
+			  if (variant == curvar && firstvar == 1)
+			    { print; firstvar = 0 }
+			  next
+			}
+/\===/			{ curvar = ""; next }
+{ if (variant == curvar) print; else next }
+EOF
+
+  SUMS_AWK=${TMP}-sums.awk
+  rm -f $SUMS_AWK
+  cat << EOF > $SUMS_AWK
+BEGIN {
+  variant="$VAR"
+  tool="$TOOL"
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; unsupcnt=0
+  curvar=""; insummary=0
+}
+/^Running target /		{ curvar = \$3; next }
+/^# of /			{ if (variant == curvar) insummary = 1 }
+/^# of expected passes/		{ if (insummary == 1) passcnt += \$5; next; }
+/^# of unexpected successes/	{ if (insummary == 1) xpasscnt += \$5; next; }
+/^# of unexpected failures/	{ if (insummary == 1) failcnt += \$5; next; }
+/^# of expected failures/	{ if (insummary == 1) xfailcnt += \$5; next; }
+/^# of untested testcases/	{ if (insummary == 1) untstcnt += \$5; next; }
+/^# of unsupported tests/	{ if (insummary == 1) unsupcnt += \$5; next; }
+/^$/				{ if (insummary == 1)
+				    { insummary = 0; curvar = "" }
+				  next
+				}
+{ next }
+END {
+  printf ("\t\t=== %s Summary for %s ===\n\n", tool, variant)
+  if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
+  if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
+  if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
+  if (xfailcnt != 0) printf ("# of expected failures\t\t%d\n", xfailcnt)
+  if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt)
+  if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt)
+}
+EOF
+
+  PVAR=`echo $VAR | sed 's,/,.,g'`
+  TMPFILE=${TMP}-var-$PVAR
+  rm -f $TMPFILE
+  cat $SUM_FILES | $AWK -f $GUTS_AWK
+  cat $SUM_FILES | $AWK -f $SUMS_AWK > $TMPFILE
+  # If there are multiple variants, output the counts for this one;
+  # otherwise there will just be the final counts at the end.
+  test $VARIANT_COUNT -eq 1 || cat $TMPFILE
+done
+
+# Set up an awk script to get the combined summary counts for the tool.
+
+TOTAL_AWK=${TMP}-total.awk
+cat << EOF > $TOTAL_AWK
+BEGIN {
+  tool="$TOOL"
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; unsupcnt=0
+}
+/^# of expected passes/		{ passcnt += \$5 }
+/^# of unexpected failures/	{ failcnt += \$5 }
+/^# of unexpected successes/	{ xpasscnt += \$5 }
+/^# of expected failures/	{ xfailcnt += \$5 }
+/^# of untested testcases/	{ untstcnt += \$5 }
+/^# of unsupported tests/	{ unsupcnt += \$5 }
+END {
+  printf ("\n\t\t=== %s Summary ===\n\n", tool)
+  if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
+  if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
+  if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
+  if (xfailcnt != 0) printf ("# of expected failures\t\t%d\n", xfailcnt)
+  if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt)
+  if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt)
+}
+EOF
+
+# Find the total summaries for the tool and add to the end of the output.
+cat ${TMP}-var-* | $AWK -f $TOTAL_AWK
+
+# This is ugly, but if there's version output from the compiler under test
+# at the end of the file, we want it.  The other thing that might be there
+# is the final summary counts.
+tail -n 2 $FIRST_SUM | grep -q '^#' || tail -n 2 $FIRST_SUM
+
+rm -f ${TMP}-*
+exit 0

Property changes on: contrib/dg-extract-results.sh
___________________________________________________________________
Name: svn:executable
   + *

Re: [contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Jakub Jelinek]

On Wed, Oct 15, 2008 at 10:56:13AM -0700, Janis Johnson wrote:
> I've been using this quite a bit with results from my attempts to
> paralellize the compiler tests on GNU/Linux.  I've tried to make it
> portable  but it's quite possible that it isn't.  I'd appreicate having
> other people try it out, perhaps just by running a .sum file through it
> to make sure the result is the same as the original file.

Here is a WIP what I've been playing with today.  Some tuning
is needed in check_gcc_parallelize (guess tree-ssa.exp and maybe
dg-torture.exp and/or vect.exp could be kicked out and replaced by some *.exp
that takes a long time, measuring times right now (likely i386.exp
(or generally $(target).exp?), unsorted.exp and stackalign.exp),
but even with this I got make -j8 -k check-gcc finished in 19 minutes
instead of the usual 60 minutes.  The merging is something I've left
out for the time being, so your merge script could be very handy.

If you use make -j4 check-gcc RUNTESTFLAGS=dg.exp etc., it doesn't
parallelize and similarly if make check-gcc, parallelization is only done
if RUNTESTFLAGS is empty or contains only --target_board and -j was used.

I believe check-gcc is the longest one, check-target-libstdc++-v3 the
second and check-gfortran the third.  check-g++ and check-gfortran speedup
is harder if we don't want to restructure the exp files, most of
check-g++ time is in dg.exp and most of check-gfortran time is in dg.exp
too.  Perhaps something like running dg.exp=[a-h]* in one case and
dg.exp=[^a-h]* in check-gfortran and something similar for check-g++
could help.

--- gcc/Makefile.in.jj	2008-10-14 13:58:50.000000000 +0200
+++ gcc/Makefile.in	2008-10-15 20:02:21.000000000 +0200
@@ -442,6 +442,10 @@ GTHREAD_FLAGS=@gthread_flags@
 extra_modes_file=@extra_modes_file@
 extra_opt_files=@extra_opt_files@
 host_hook_obj=@out_host_hook_obj@
+# This lists a couple of test files that take most time during check-gcc.
+# When doing parallelized check-gcc, these can run in parallel with the
+# remaining tests.
+check_gcc_parallelize=compile.exp execute.exp dg.exp dg-torture.exp tree-ssa.exp vect.exp
 
 # ------------------------
 # Installation directories
@@ -4498,7 +4502,7 @@ $(TESTSUITEDIR)/site.exp: site.exp
 	-rm -f $@
 	sed '/set tmpdir/ s|testsuite|$(TESTSUITEDIR)|' < site.exp > $@
 
-$(lang_checks): check-% : site.exp
+$(filter-out check-gcc,$(lang_checks)): check-% : site.exp
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$* || mkdir $(TESTSUITEDIR)/$*
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
@@ -4515,6 +4519,47 @@ $(lang_checks): check-% : site.exp
 	GCC_EXEC_PREFIX="$(libdir)/gcc/" ; export GCC_EXEC_PREFIX ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
+$(filter check-gcc, $(lang_checks)): check-% : site.exp
+	@if [ -z "$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
+	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  $(MAKE) RUNTESTFLAGS=$(RUNTESTFLAGS) \
+	    check-parallel-$* $(patsubst %,check-parallel-$*_%,$(check_$*_parallelize)); \
+	  echo "Merge it now"; \
+	else \
+	  $(MAKE) RUNTESTFLAGS=$(RUNTESTFLAGS) check_$*_parallelize= check-parallel-$*; \
+	fi
+
+check-parallel-% : site.exp
+	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
+	test -d $(TESTSUITEDIR)/$* || mkdir $(TESTSUITEDIR)/$*
+	-(rootme=`${PWD_COMMAND}`; export rootme; \
+	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
+	cd $(TESTSUITEDIR)/$*; \
+	rm -f tmp-site.exp; \
+	sed '/set tmpdir/ s|testsuite|$(TESTSUITEDIR)/$*|' \
+		< ../../site.exp > tmp-site.exp; \
+	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	EXPECT=${EXPECT} ; export EXPECT ; \
+	if [ -f $${rootme}/../expect/expect ] ; then  \
+	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; fi ; \
+	GCC_EXEC_PREFIX="$(libdir)/gcc/" ; export GCC_EXEC_PREFIX ; \
+	tool=`echo "$*" | sed 's/_.*$$//'` ; \
+	part=`echo "$*" | sed -n 's/^[^_]*_//p'` ; \
+	runtestflags= ; \
+	if [ -n "$$part" ] ; then \
+	  runtestflags="$$part" ; \
+	elif [ -n "$(check_$*_parallelize)" ] ; then \
+	  for part in `find $$srcdir/testsuite/$$tool* -name \*.exp` ; do \
+	    part=`basename $$part` ; \
+	    case " $(strip $(check_$*_parallelize)) $$runtestflags " in \
+	      *" $$part "*)  ;; \
+	      *) runtestflags="$$runtestflags $$part" ;; \
+	    esac ; \
+	  done ; \
+	fi ; \
+	$(RUNTEST) --tool $$tool $(RUNTESTFLAGS) $$runtestflags)
+
 check-consistency: testsuite/site.exp
 	-rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
@@ -4552,7 +4597,7 @@ QMTEST_DIR=qmtestsuite
 ${QMTEST_DIR} stamp-qmtest:
 	${QMTEST} -D ${QMTEST_DIR} create-tdb \
 	    -c gcc_database.GCCDatabase \
-            -a srcdir=`cd ${srcdir}/testsuite && ${PWD_COMMAND}` && \
+	    -a srcdir=`cd ${srcdir}/testsuite && ${PWD_COMMAND}` && \
 	    $(STAMP) stamp-qmtest
 
 # Create the QMTest context file.


	Jakub

Re: [contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Jakub Jelinek]

On Wed, Oct 15, 2008 at 08:30:16PM +0200, Jakub Jelinek wrote:
> On Wed, Oct 15, 2008 at 10:56:13AM -0700, Janis Johnson wrote:
> > I've been using this quite a bit with results from my attempts to
> > paralellize the compiler tests on GNU/Linux.  I've tried to make it
> > portable  but it's quite possible that it isn't.  I'd appreicate having
> > other people try it out, perhaps just by running a .sum file through it
> > to make sure the result is the same as the original file.
> 
> Here is a WIP what I've been playing with today.

Updated patch, with wich make -j13 -k check-{gcc,g++,gfortran} finishes
in less than 14 minutes for me (instead of the usual 60).
Compared to the older patch, it parallelizes check-gfortran and check-g++
and adds a syntax to run a bunch of *.exp's in one make and also to
run just part of one *.exp in one make and another part of the same *.exp
in another make.  To my surprise dejagnu (particularly runtest_file_p
and string match it uses) doesn't handle [^a-z] style wildcards, only
[a-z] ones, so I had to always use inclusive ranges.

The patch still doesn't integrate your sum file merger, will test
it tomorrow, it would go into the spot where echo "Merge it now"
is executed.  I think it would be desirable not to merge just
sum files, but also log files and once merged, rename the auxiliary
*.log and *.sum files, so that they aren't picked by test_summary etc.
As the tests in separate make invocations are executed in the same
directory depth as when not parallelizing, cut'n'paste of commands from
the merged log file and executing in the non-suffixed testsuite/XXX
directory should work just fine.

If this is acceptable solution, I guess I can look at parallelizing
check-acats, check-libstdc++-v3 and maybe check-libjava tomorrow.

--- gcc/cp/Make-lang.in.jj	2008-09-30 16:56:10.000000000 +0200
+++ gcc/cp/Make-lang.in	2008-10-15 23:30:53.000000000 +0200
@@ -140,6 +140,8 @@ c++.srcman: doc/g++.1
 check-c++ : check-g++
 # List of targets that can use the generic check- rule and its // variant.
 lang_checks += check-g++
+lang_checks_parallelized += check-g++
+check_g++_parallelize = old-deja.exp dg.exp
 
 #\f
 # Install hooks:
--- gcc/fortran/Make-lang.in.jj	2008-09-30 16:56:44.000000000 +0200
+++ gcc/fortran/Make-lang.in	2008-10-16 00:19:08.000000000 +0200
@@ -146,6 +146,8 @@ fortran.srcextra:
 check-f95 : check-gfortran
 check-fortran : check-gfortran
 lang_checks += check-gfortran
+lang_checks_parallelized += check-gfortran
+check_gfortran_parallelize = dg.exp~gfortran.dg@a-cA-C dg.exp~gfortran.dg@d-mD-M dg.exp~gfortran.dg@n-zN-Z0-9
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
--- gcc/Makefile.in.jj	2008-10-14 13:58:50.000000000 +0200
+++ gcc/Makefile.in	2008-10-16 00:08:47.000000000 +0200
@@ -432,6 +432,11 @@ xm_file_list=@xm_file_list@
 xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=check-gcc
+lang_checks_parallelized=check-gcc
+# This lists a couple of test files that take most time during check-gcc.
+# When doing parallelized check-gcc, these can run in parallel with the
+# remaining tests.
+check_gcc_parallelize=execute.exp~execute@2 execute.exp~execute@013-9a-zA-Z compile.exp dg.exp struct-layout-1.exp,unsorted.exp,stackalign.exp,i386.exp
 lang_opt_files=@lang_opt_files@ $(srcdir)/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -4498,7 +4503,7 @@ $(TESTSUITEDIR)/site.exp: site.exp
 	-rm -f $@
 	sed '/set tmpdir/ s|testsuite|$(TESTSUITEDIR)|' < site.exp > $@
 
-$(lang_checks): check-% : site.exp
+$(filter-out $(lang_checks_parallelized),$(lang_checks)): check-% : site.exp
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$* || mkdir $(TESTSUITEDIR)/$*
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
@@ -4515,6 +4520,50 @@ $(lang_checks): check-% : site.exp
 	GCC_EXEC_PREFIX="$(libdir)/gcc/" ; export GCC_EXEC_PREFIX ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
+$(lang_checks_parallelized): check-% : site.exp
+	@if [ -z "$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
+	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  $(MAKE) RUNTESTFLAGS=$(RUNTESTFLAGS) \
+	    check-parallel-$* $(patsubst %,check-parallel-$*_%,$(check_$*_parallelize)); \
+	  echo "Merge it now"; \
+	else \
+	  $(MAKE) RUNTESTFLAGS=$(RUNTESTFLAGS) check_$*_parallelize= check-parallel-$*; \
+	fi
+
+check-parallel-% : site.exp
+	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
+	test -d $(TESTSUITEDIR)/'$*' || mkdir $(TESTSUITEDIR)/'$*'
+	-(rootme=`${PWD_COMMAND}`; export rootme; \
+	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
+	cd $(TESTSUITEDIR)/'$*'; \
+	rm -f tmp-site.exp; \
+	sed '/set tmpdir/ s|testsuite|$(TESTSUITEDIR)/$*|' \
+		< ../../site.exp > tmp-site.exp; \
+	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	EXPECT=${EXPECT} ; export EXPECT ; \
+	if [ -f $${rootme}/../expect/expect ] ; then  \
+	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; fi ; \
+	GCC_EXEC_PREFIX="$(libdir)/gcc/" ; export GCC_EXEC_PREFIX ; \
+	tool=`echo '$*' | sed 's/_.*$$//'` ; \
+	part=`echo '$*' | sed -n 's/^[^_]*_//p'` ; \
+	runtestflags= ; \
+	if [ -n "$$part" ] ; then \
+	  runtestflags=`echo "$$part " | tr '@~,' '/= ' \
+			| sed 's,/\([^/\\][^/]*\) ,/\\\[\1\\\]\* ,g;s,/ ,/\* ,g'`; \
+	elif [ -n "$(check_$*_parallelize)" ] ; then \
+	  parts="`echo ' $(strip $(check_$*_parallelize)) ' | tr ',' ' ' \
+		  | sed 's/~[^ ]* / /g'`"; \
+	  for part in `find $$srcdir/testsuite/$$tool* -name \*.exp` ; do \
+	    part=`basename $$part` ; \
+	    case " $$parts $$runtestflags " in \
+	      *" $$part "*)  ;; \
+	      *) runtestflags="$$runtestflags $$part" ;; \
+	    esac ; \
+	  done ; \
+	fi ; \
+	$(RUNTEST) --tool $$tool $(RUNTESTFLAGS) $$runtestflags)
+
 check-consistency: testsuite/site.exp
 	-rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
@@ -4552,7 +4601,7 @@ QMTEST_DIR=qmtestsuite
 ${QMTEST_DIR} stamp-qmtest:
 	${QMTEST} -D ${QMTEST_DIR} create-tdb \
 	    -c gcc_database.GCCDatabase \
-            -a srcdir=`cd ${srcdir}/testsuite && ${PWD_COMMAND}` && \
+	    -a srcdir=`cd ${srcdir}/testsuite && ${PWD_COMMAND}` && \
 	    $(STAMP) stamp-qmtest
 
 # Create the QMTest context file.


	Jakub

Re: [contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Jakub Jelinek]

On Thu, Oct 16, 2008 at 12:47:42AM +0200, Jakub Jelinek wrote:
> Updated patch, with wich make -j13 -k check-{gcc,g++,gfortran} finishes
> in less than 14 minutes for me (instead of the usual 60).

Here is the same, with added comments, using nicer testsuite/* directory
names for the separate goals, integrated your merging script and
changes to the script to make the merged *.sum files identical
(except for timestamps) between parallelized and non-parallelized
testing.  It also merges log files, though the log files aren't 100%
identical (it is difficult to maintain order in that case
if some *.exp file has been invoked in parts, and the actual gccN etc.
directories are embedded in the commands).  The order for the *.sum
files matters a lot (as that's used to compare test_summary results etc.),
while the *.log files are usually just grepped for the needed command
to reproduce some failure.  For this task the merged log files are perfectly
usable.  I've also added mktemp -d (copied code to do it from config.status code,
so it should be portable), and a few other tweaks.

I'll now look at check-target-libstdc++-v3 and check-acats parallelization.

--- gcc/cp/Make-lang.in.jj	2008-10-16 01:06:28.000000000 +0200
+++ gcc/cp/Make-lang.in	2008-10-16 15:39:26.000000000 +0200
@@ -140,6 +140,9 @@ c++.srcman: doc/g++.1
 check-c++ : check-g++
 # List of targets that can use the generic check- rule and its // variant.
 lang_checks += check-g++
+lang_checks_parallelized += check-g++
+# For description see comment above check_gcc_parallelize in gcc/Makefile.in.
+check_g++_parallelize = old-deja.exp dg.exp
 
 #\f
 # Install hooks:
--- gcc/fortran/Make-lang.in.jj	2008-10-16 01:06:28.000000000 +0200
+++ gcc/fortran/Make-lang.in	2008-10-16 15:39:08.000000000 +0200
@@ -146,6 +146,11 @@ fortran.srcextra:
 check-f95 : check-gfortran
 check-fortran : check-gfortran
 lang_checks += check-gfortran
+lang_checks_parallelized += check-gfortran
+# For description see comment above check_gcc_parallelize in gcc/Makefile.in.
+check_gfortran_parallelize = dg.exp~gfortran.dg@a-cA-C \
+			     dg.exp~gfortran.dg@d-mD-M \
+			     dg.exp~gfortran.dg@n-zN-Z0-9
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
--- gcc/Makefile.in.jj	2008-10-16 01:06:28.000000000 +0200
+++ gcc/Makefile.in	2008-10-16 16:19:15.000000000 +0200
@@ -432,6 +432,28 @@ xm_file_list=@xm_file_list@
 xm_include_list=@xm_include_list@
 xm_defines=@xm_defines@
 lang_checks=check-gcc
+lang_checks_parallelized=check-gcc
+# This lists a couple of test files that take most time during check-gcc.
+# When doing parallelized check-gcc, these can run in parallel with the
+# remaining tests.  Each word in this variable stands for work for one
+# make goal and one extra make goal is added to handle all the *.exp
+# files not handled explicitly already.  Because of make syntax, the
+# work is specially encoded.
+# foo.exp		means the whole foo.exp will be run by one goal.
+# foo.exp~dir@		this is decoded to foo.exp=dir/* argument to runtest,
+#			run all tests in a given directory.
+# foo.exp~dir@chars	this is decoded as foo.exp=dir/\[chars\]* argument
+#			to runtest, run all tests that begin with the
+#			listed chars, char ranges are possible too,
+#			but beware that dejagnu doesn't support dir/\[^a-z\]*
+#			patterns.
+# foo.exp,bar.exp	run both foo.exp and bar.exp in one runtest invocation.
+#			Useful if the tests aren't very long running, but still
+#			desirable to split them of the check-parallel-$lang
+#			remaining tests runtest invocation.
+check_gcc_parallelize=execute.exp~execute@2 execute.exp~execute@013-9a-zA-Z \
+		      compile.exp dg.exp \
+		      struct-layout-1.exp,unsorted.exp,stackalign.exp,i386.exp
 lang_opt_files=@lang_opt_files@ $(srcdir)/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -4498,7 +4520,8 @@ $(TESTSUITEDIR)/site.exp: site.exp
 	-rm -f $@
 	sed '/set tmpdir/ s|testsuite|$(TESTSUITEDIR)|' < site.exp > $@
 
-$(lang_checks): check-% : site.exp
+# This is only used for check-% targets that aren't parallelized.
+$(filter-out $(lang_checks_parallelized),$(lang_checks)): check-% : site.exp
 	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
 	test -d $(TESTSUITEDIR)/$* || mkdir $(TESTSUITEDIR)/$*
 	-(rootme=`${PWD_COMMAND}`; export rootme; \
@@ -4515,6 +4538,97 @@ $(lang_checks): check-% : site.exp
 	GCC_EXEC_PREFIX="$(libdir)/gcc/" ; export GCC_EXEC_PREFIX ; \
 	$(RUNTEST) --tool $* $(RUNTESTFLAGS))
 
+check_p_tool = $(firstword $(subst _, ,$*))
+check_p_vars = $(check_$(check_p_tool)_parallelize)
+check_p_cond = $(filter $*,$(patsubst %,$(check_p_tool)_%,$(word $(check_p_idx),$(check_p_vars))))
+check_p_numbers = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_subdir = $(check_p_tool)$(strip $(foreach check_p_idx, $(check_p_numbers),$(if $(check_p_cond),$(check_p_idx))))
+check_p_subdirs = $(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
+
+# For parallelized check-% targets, this decides whether parallelization
+# is desirable (if -jN is used and RUNTESTFLAGS doesn't contain anything
+# but optionally --target_board argument).  If it is desirable,
+# recursive make is run with check-parallel-$lang{,1,2,3,4,5} etc. goals,
+# which can be executed in parallel, as they are run in separate directories.
+# check-parallel-$lang{1,2,3,4,5} etc. goals invoke runtest with the longest
+# running *.exp files from the testsuite, as determined by check_$lang_parallelize
+# variable.  The check-parallel-$lang goal in that case invokes runtest with
+# all the remaining *.exp files not handled by the separate goals.
+# Afterwards contrib/dg-extract-results.sh is used to merge the sum and log
+# files.  If parallelization isn't desirable, only one recursive make
+# is run with check-parallel-$lang goal and check_$lang_parallelize variable
+# cleared to say that no additional arguments beyond $(RUNTESTFLAGS)
+# should be passed to runtest.
+#
+# To parallelize some language check, add the corresponding check-$lang
+# to lang_checks_parallelized variable and define check_$lang_parallelize
+# variable (see above check_gcc_parallelize description).
+$(lang_checks_parallelized): check-% : site.exp
+	@if [ -z "$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
+	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
+	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
+	    check-parallel-$* $(patsubst %,check-parallel-$*_%,$(check_$*_parallelize)); \
+	  for file in $(TESTSUITEDIR)/$*/$* \
+		      $(patsubst %,$(TESTSUITEDIR)/$*%/$*,$(check_p_subdirs));\
+	  do \
+	    mv -f $$file.sum $$file.sum.sep; mv -f $$file.log $$file.log.sep; \
+	  done; \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh \
+	    $(TESTSUITEDIR)/$*/$*.sum.sep \
+	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.sum.sep,$(check_p_subdirs)) \
+	    > $(TESTSUITEDIR)/$*/$*.sum; \
+	  $(SHELL) $(srcdir)/../contrib/dg-extract-results.sh -L \
+	    $(TESTSUITEDIR)/$*/$*.log.sep \
+	    $(patsubst %,$(TESTSUITEDIR)/$*%/$*.log.sep,$(check_p_subdirs)) \
+	    > $(TESTSUITEDIR)/$*/$*.log; \
+	else \
+	  $(MAKE) TESTSUITEDIR="$(TESTSUITEDIR)" RUNTESTFLAGS="$(RUNTESTFLAGS)" \
+	    check_$*_parallelize= check-parallel-$*; \
+	fi
+
+check-parallel-% : site.exp
+	-test -d $(TESTSUITEDIR) || mkdir $(TESTSUITEDIR)
+	test -d $(TESTSUITEDIR)/$(check_p_subdir) || mkdir $(TESTSUITEDIR)/$(check_p_subdir)
+	-(rootme=`${PWD_COMMAND}`; export rootme; \
+	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
+	cd $(TESTSUITEDIR)/$(check_p_subdir); \
+	rm -f tmp-site.exp; \
+	sed '/set tmpdir/ s|testsuite|$(TESTSUITEDIR)/$(check_p_subdir)|' \
+		< ../../site.exp > tmp-site.exp; \
+	$(SHELL) $${srcdir}/../move-if-change tmp-site.exp site.exp; \
+	EXPECT=${EXPECT} ; export EXPECT ; \
+	if [ -f $${rootme}/../expect/expect ] ; then  \
+	   TCL_LIBRARY=`cd .. ; cd $${srcdir}/../tcl/library ; ${PWD_COMMAND}` ; \
+	    export TCL_LIBRARY ; fi ; \
+	GCC_EXEC_PREFIX="$(libdir)/gcc/" ; export GCC_EXEC_PREFIX ; \
+	part=`echo '$*' | sed -n 's/^[^_]*_//p'` ; \
+	runtestflags= ; \
+	if [ -n "$$part" ] ; then \
+	  # In this case runtest should be given the name of the give \
+	  # *.exp file(s).  As make doesn't like certain characters in goal's \
+	  # names, check_$lang_parallelize variable is specially encoded, see \
+	  # comment above check_gcc_parallelize variable for details. \
+	  # This command decodes it. \
+	  runtestflags=`echo "$$part " | tr '@~,' '/= ' \
+			| sed 's,/\([^/\\][^/]*\) ,/\\\[\1\\\]\* ,g;s,/ ,/\* ,g'`; \
+	elif [ -n "$(check_$*_parallelize)" ] ; then \
+	  # In this case runtest should be given names of all the *.exp \
+	  # files for this tool that aren't already handled by other goals. \
+	  parts="`echo ' $(strip $(check_$*_parallelize)) ' | tr ',' ' ' \
+		  | sed 's/~[^ ]* / /g'`"; \
+	  # Find all the *.exp files for this tool. \
+	  for part in `find $$srcdir/testsuite/$(check_p_tool)* -name \*.exp` ; do \
+	    part=`basename $$part` ; \
+	    # And prune those already specified in check_$lang_parallelize \
+	    # or duplicates. \
+	    case " $$parts $$runtestflags " in \
+	      *" $$part "*)  ;; \
+	      *) runtestflags="$$runtestflags $$part" ;; \
+	    esac ; \
+	  done ; \
+	fi ; \
+	$(RUNTEST) --tool $(check_p_tool) $(RUNTESTFLAGS) $$runtestflags)
+
 check-consistency: testsuite/site.exp
 	-rootme=`${PWD_COMMAND}`; export rootme; \
 	srcdir=`cd ${srcdir}; ${PWD_COMMAND}` ; export srcdir ; \
@@ -4552,7 +4666,7 @@ QMTEST_DIR=qmtestsuite
 ${QMTEST_DIR} stamp-qmtest:
 	${QMTEST} -D ${QMTEST_DIR} create-tdb \
 	    -c gcc_database.GCCDatabase \
-            -a srcdir=`cd ${srcdir}/testsuite && ${PWD_COMMAND}` && \
+	    -a srcdir=`cd ${srcdir}/testsuite && ${PWD_COMMAND}` && \
 	    $(STAMP) stamp-qmtest
 
 # Create the QMTest context file.
--- contrib/dg-extract-results.sh.jj	2008-10-16 14:33:30.000000000 +0200
+++ contrib/dg-extract-results.sh	2008-10-16 16:17:57.000000000 +0200
@@ -0,0 +1,360 @@
+#! /bin/sh
+
+# For a specified tool and optional list of test variants, extract
+# test results from one or more test summary (.sum) files and combine
+# the results into a new test summary file, sent to the standard output.
+# The resulting file can be used with test result comparison scripts for
+# results from tests that were run in parallel.  See usage() below.
+
+# Copyright (C) 2008 Free Software Foundation
+# Contributed by Janis Johnson <janis187@us.ibm.com>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+PROGNAME=dg-extract-results.sh
+
+usage() {
+  cat <<EOF >&2
+Usage: $PROGNAME [-t tool] [-l variant-list] [-L] sum-file ...
+
+    tool           The tool (e.g. g++, libffi) for which to create a
+                   new test summary file.  If not specified then all
+                   specified sum files must be for the same tool.
+    variant-list   One or more test variant names.  If the list is
+                   not specified then one is constructed from all
+                   variants in the files for <tool>.
+    sum-file       A test summary file with the format of those
+                   created by runtest from DejaGnu.
+    If -L is used, merge *.log files instead of *.sum.  In this
+    mode the exact order of lines may not be preserved, just different
+    Running *.exp chunks should be in correct order.
+EOF
+}
+
+# Write a message to the standard error.
+
+msg() {
+  echo "$@" >&2
+}
+
+# Parse the command-line options.
+
+VARIANTS=""
+TOOL=""
+MODE="sum"
+
+while getopts "l:t:L" ARG; do
+  case $ARG in
+  l)  VARIANTS="${VARIANTS} ${OPTARG}";;
+  t)  test -z "$TOOL" || (msg "${PROGNAME}: only one tool can be specified"; exit 1);
+      TOOL="${OPTARG}";;
+  L)  MODE="log";;
+  \?) usage; exit 0;;
+  esac
+done
+shift `expr ${OPTIND} - 1`
+
+if test $# -lt 1 ; then
+  usage
+  exit 1
+fi
+
+TMPDIR=${TMPDIR-/tmp}
+SUM_FILES="$@"
+FIRST_SUM=$1
+TMP=
+trap 'EXIT_STATUS=$?; rm -rf $TMP && exit $EXIT_STATUS' 0
+# Create a (secure) tmp directory for tmp files.
+{
+  TMP=`(umask 077 && mktemp -d -q "${TMPDIR}/dg-combine-results-$$-XXXXXX") 2>/dev/null` &&
+  test -n "$TMP" && test -d "$TMP"
+} ||
+{
+  TMP=${TMPDIR}/dg-combine-results-$$-$RANDOM
+  (umask 077 && mkdir $TMP)
+} ||
+{
+  msg "${PROGNAME}: cannot create a temporary directory"
+  { (exit 1); exit 1; }
+}
+
+# Find a good awk.
+
+if test -z "$AWK" ; then
+  for AWK in gawk nawk awk
+  do
+    if type $AWK 2>&1 | grep 'not found' > /dev/null 2>&1 ; then
+      :
+    else
+      break
+    fi
+  done
+fi
+
+# Verify that the specified summary files exist.
+
+ERROR=0
+for FILE in $SUM_FILES
+do
+  if ! test -f $FILE ; then
+    msg "${PROGNAME}: file $FILE does not exist."
+    ERROR=1
+  fi
+done
+test $ERROR -eq 0 || exit 1
+
+if [ -z "$TOOL" ]; then
+  # If no tool was specified, all specified summary files must be for
+  # the same tool.
+
+  CNT=`grep '=== .* tests ===' $SUM_FILES | $AWK '{ print $3 }' | sort -u | wc -l`
+  if [ $CNT -eq 1 ]; then
+    TOOL=`grep '=== .* tests ===' $FIRST_SUM | $AWK '{ print $2 }'`
+  else
+    msg "${PROGNAME}: sum files are for multiple tools, specify a tool"
+    msg ""
+    usage
+    exit 1
+  fi
+else
+  # Ignore the specified summary files that are not for this tool.  This
+  # should keep the relevant files in the same order.
+
+  SUM_FILES=`grep -l "=== $TOOL" $SUM_FILES`
+  if test -z "$SUM_FILES" ; then
+    msg "${PROGNAME}: none of the specified files are results for $TOOL"
+    exit 1
+  fi
+fi
+
+# If no variants were specified, find all variants in the remaining
+# summary files.  Otherwise, ignore specified variants that aren't in
+# any of those summary files.
+
+if test -z "$VARIANTS" ; then
+  VAR_AWK=${TMP}/variants.awk
+  cat <<EOF > $VAR_AWK
+/^Schedule of variations:/      { in_vars=1; next }
+/^$/                            { in_vars=0 }
+/^Running target/               { exit }
+{ if (in_vars==1) print \$1; else next }
+EOF
+
+  touch ${TMP}/varlist
+  for FILE in $SUM_FILES; do
+    $AWK -f $VAR_AWK $FILE >> ${TMP}/varlist
+  done
+  VARIANTS="`sort -u ${TMP}/varlist`"
+else
+  VARS="$VARIANTS"
+  VARIANTS=""
+  for VAR in $VARS
+  do
+    grep -q "Running target $VAR" $SUM_FILES && VARIANTS="$VARIANTS $VAR"
+  done
+fi
+
+# Find out if we have more than one variant, or any at all.
+
+VARIANT_COUNT=0
+for VAR in $VARIANTS
+do
+  VARIANT_COUNT=`expr $VARIANT_COUNT + 1`
+done
+
+if test $VARIANT_COUNT -eq 0 ; then
+  msg "${PROGNAME}: no file for $TOOL has results for the specified variants"
+  exit 1
+fi
+
+cat $SUM_FILES \
+  | $AWK '/^Running/ { if ($2 != "target" && $3 == "...") print "EXPFILE: "$2 } ' \
+  | sort -u > ${TMP}/expfiles
+
+# Write the begining of the combined summary file.
+
+head -n 2 $FIRST_SUM
+echo
+echo "		=== $TOOL tests ==="
+echo
+echo "Schedule of variations:"
+for VAR in $VARIANTS
+do
+  echo "    $VAR"
+done
+echo
+
+# For each test variant for the tool, copy test reports from each of the
+# summary files.  Set up two awk scripts from within the loop to
+# initialize VAR and TOOL with the script, rather than assuming that the
+# available version of awk can pass variables from the command line.
+
+for VAR in $VARIANTS
+do
+  GUTS_AWK=${TMP}/guts.awk
+  cat << EOF > $GUTS_AWK
+BEGIN {
+  variant="$VAR"
+  firstvar=1
+  expfileno=1
+  cnt=0
+  print_using=0
+}
+/^EXPFILE: / {
+  expfiles[expfileno] = \$2
+  expfilesr[\$2] = expfileno
+  expfileno = expfileno + 1
+}
+/^Running target / {
+  curvar = \$3
+  if (variant == curvar && firstvar == 1) { print; print_using=1; firstvar = 0 }
+  next
+}
+/^Using / {
+  if (variant == curvar && print_using) { print; next }
+}
+/^Running / {
+  print_using=0
+  if (variant == curvar) {
+    curfile="${TMP}/list"expfilesr[\$2]
+    expfileseen[\$2]=expfileseen[\$2] + 1
+    testname="00"
+    next
+  }
+}
+/\===/ { curvar = ""; next }
+/^(PASS|XPASS|FAIL|XFAIL|UNRESOLVED|WARNING|ERROR|UNSUPPORTED|UNTESTED|KFAIL):/ {
+  testname=\$2
+  # Ugly hack for gfortran.dg/dg.exp
+  if ("$TOOL" == "gfortran" && testname ~ /^gfortran.dg\/g77\//)
+    testname="h"testname
+}
+/^$/ { if ("$MODE" == "sum") next }
+{ if (variant == curvar && curfile) {
+    if ("$MODE" == "sum") {
+      printf "%s %08d|", testname, cnt > curfile
+      cnt = cnt + 1
+    }
+    filewritten[curfile]=1
+    print > curfile
+  } else
+    next
+}
+END {
+  n=1
+  while (n < expfileno) {
+    if (expfileseen[expfiles[n]]) {
+      print "Running "expfiles[n]" ..."
+      if (filewritten["${TMP}/list"n]) {
+	if (expfileseen[expfiles[n]] == 1)
+	  cmd="cat"
+	else
+	  cmd="LC_ALL=C sort"
+	if ("$MODE" == "sum")
+	  system (cmd" ${TMP}/list"n" | sed -n 's/^[^ ]* [^ |]*|//p'")
+	else
+	  system ("cat ${TMP}/list"n)
+      }
+    }
+    n = n + 1
+  }
+}
+EOF
+
+  SUMS_AWK=${TMP}/sums.awk
+  rm -f $SUMS_AWK
+  cat << EOF > $SUMS_AWK
+BEGIN {
+  variant="$VAR"
+  tool="$TOOL"
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; unsupcnt=0; unrescnt=0;
+  curvar=""; insummary=0
+}
+/^Running target /		{ curvar = \$3; next }
+/^# of /			{ if (variant == curvar) insummary = 1 }
+/^# of expected passes/		{ if (insummary == 1) passcnt += \$5; next; }
+/^# of unexpected successes/	{ if (insummary == 1) xpasscnt += \$5; next; }
+/^# of unexpected failures/	{ if (insummary == 1) failcnt += \$5; next; }
+/^# of expected failures/	{ if (insummary == 1) xfailcnt += \$5; next; }
+/^# of untested testcases/	{ if (insummary == 1) untstcnt += \$5; next; }
+/^# of unresolved testcases/	{ if (insummary == 1) unrescnt += \$5; next; }
+/^# of unsupported tests/	{ if (insummary == 1) unsupcnt += \$5; next; }
+/^$/				{ if (insummary == 1)
+				    { insummary = 0; curvar = "" }
+				  next
+				}
+{ next }
+END {
+  printf ("\t\t=== %s Summary for %s ===\n\n", tool, variant)
+  if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
+  if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
+  if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
+  if (xfailcnt != 0) printf ("# of expected failures\t\t%d\n", xfailcnt)
+  if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt)
+  if (unrescnt != 0) printf ("# of unresolved testcases\t%d\n", unrescnt)
+  if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt)
+}
+EOF
+
+  PVAR=`echo $VAR | sed 's,/,.,g'`
+  TMPFILE=${TMP}/var-$PVAR
+  rm -f $TMPFILE
+  rm -f ${TMP}/list*
+  cat ${TMP}/expfiles $SUM_FILES | $AWK -f $GUTS_AWK
+  cat $SUM_FILES | $AWK -f $SUMS_AWK > $TMPFILE
+  # If there are multiple variants, output the counts for this one;
+  # otherwise there will just be the final counts at the end.
+  test $VARIANT_COUNT -eq 1 || cat $TMPFILE
+done
+
+# Set up an awk script to get the combined summary counts for the tool.
+
+TOTAL_AWK=${TMP}/total.awk
+cat << EOF > $TOTAL_AWK
+BEGIN {
+  tool="$TOOL"
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; unsupcnt=0; unrescnt=0
+}
+/^# of expected passes/		{ passcnt += \$5 }
+/^# of unexpected failures/	{ failcnt += \$5 }
+/^# of unexpected successes/	{ xpasscnt += \$5 }
+/^# of expected failures/	{ xfailcnt += \$5 }
+/^# of untested testcases/	{ untstcnt += \$5 }
+/^# of unresolved testcases/	{ unrescnt += \$5 }
+/^# of unsupported tests/	{ unsupcnt += \$5 }
+END {
+  printf ("\n\t\t=== %s Summary ===\n\n", tool)
+  if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
+  if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
+  if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
+  if (xfailcnt != 0) printf ("# of expected failures\t\t%d\n", xfailcnt)
+  if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt)
+  if (unrescnt != 0) printf ("# of unresolved testcases\t%d\n", unrescnt)
+  if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt)
+}
+EOF
+
+# Find the total summaries for the tool and add to the end of the output.
+cat ${TMP}/var-* | $AWK -f $TOTAL_AWK
+
+# This is ugly, but if there's version output from the compiler under test
+# at the end of the file, we want it.  The other thing that might be there
+# is the final summary counts.
+tail -n 2 $FIRST_SUM | grep -q '^#' || tail -n 2 $FIRST_SUM
+
+exit 0


	Jakub

Re: [contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Janis Johnson]

On Thu, 2008-10-16 at 16:47 +0200, Jakub Jelinek wrote:
> On Thu, Oct 16, 2008 at 12:47:42AM +0200, Jakub Jelinek wrote:
> > Updated patch, with wich make -j13 -k check-{gcc,g++,gfortran} finishes
> > in less than 14 minutes for me (instead of the usual 60).
> 
> Here is the same, with added comments, using nicer testsuite/* directory
> names for the separate goals, integrated your merging script and
> changes to the script to make the merged *.sum files identical
> (except for timestamps) between parallelized and non-parallelized
> testing.  It also merges log files, though the log files aren't 100%
> identical (it is difficult to maintain order in that case
> if some *.exp file has been invoked in parts, and the actual gccN etc.
> directories are embedded in the commands).  The order for the *.sum
> files matters a lot (as that's used to compare test_summary results etc.),
> while the *.log files are usually just grepped for the needed command
> to reproduce some failure.  For this task the merged log files are perfectly
> usable.  I've also added mktemp -d (copied code to do it from config.status code,
> so it should be portable), and a few other tweaks.

It doesn't work to put comments in the middle of a block of shell
code, but other than that the changes look, and work, great.  I ran
the compiler tests several different ways and so far haven't found
anything that's broken. 

> I'll now look at check-target-libstdc++-v3 and check-acats parallelization.

Yippee!  Thanks for doing this!  Now we just need to get other
people to try it and and see how they like it.

Janis

Re: [contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Jakub Jelinek]

On Thu, Oct 16, 2008 at 04:27:11PM -0700, Janis Johnson wrote:
> It doesn't work to put comments in the middle of a block of shell
> code, but other than that the changes look, and work, great.

You mean portably?  My bash certainly accepts that just fine.

	Jakub

Re: [contrib] dg-extract-test-results.sh, to combine or extract .sum files

[Janis Johnson]

On Fri, 2008-10-17 at 09:54 +0200, Jakub Jelinek wrote:
> On Thu, Oct 16, 2008 at 04:27:11PM -0700, Janis Johnson wrote:
> > It doesn't work to put comments in the middle of a block of shell
> > code, but other than that the changes look, and work, great.
> 
> You mean portably?  My bash certainly accepts that just fine.

That's odd, on SLES 10 I got shell errors until I removed all of
the comments within the big block that gets turned into a single
line ("; \" at the end of the lines in the Makefile).

Janis