[PATCH] make -j10 check TESTS=gdb.foo/*.exp

[PATCH] make -j10 check TESTS=gdb.foo/*.exp

[Doug Evans]

Hi.

I couldn't find an easy way to run a subset of tests using the new
parallel machinery.  I find it very convenient from time to time
to be able to specify "run all these tests in this directory in parallel".

gdb 7.6 could let me run a subset of directories in parallel
using TEST_DIRS (hacky, but it worked),
but with the new machinery we can take this further and run
any random set of tests in parallel.

This patch implements this idea.

E.g.,
bash$ make -j10 check TESTS="gdb.server/*.exp gdb.threads/*.exp"

2014-02-12  Doug Evans  <dje@google.com>

	* Makefile.in (TESTS): New variable.
	(check-single): Pass $(TESTS) to runtest.
	(check-parallel): Only run tests in $(TESTS) if non-empty.

diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in
index 7a44660..d48587c 100644
--- a/gdb/testsuite/Makefile.in
+++ b/gdb/testsuite/Makefile.in
@@ -160,8 +160,17 @@ DO_RUNTEST = \
 	  export TCL_LIBRARY ; fi ; \
 	$(RUNTEST)
 
+# TESTS exists for the user to pass on the command line to easily
+# say "Only run these tests."  With check-single it's not necessary, but
+# with check-parallel there's no other way to (easily) specify a subset
+# of tests.  For consistency we support it for check-single as well.
+# To specify all tests in a subdirectory, use TESTS=gdb.subdir/*.exp.
+# E.g., make check TESTS="gdb.server/*.exp gdb.threads/*.exp".
+@GMAKE_TRUE@TESTS :=
+@GMAKE_FALSE@TESTS =
+
 check-single:
-	$(DO_RUNTEST) $(RUNTESTFLAGS)
+	$(DO_RUNTEST) $(RUNTESTFLAGS) $(TESTS)
 
 check-parallel:
 	-rm -rf cache
@@ -180,11 +189,16 @@ check-parallel:
 # them to the front of the list to try to lessen the overall time
 # taken by the test suite -- if one of these tests happens to be run
 # late, it will cause the overall time to increase.
+@GMAKE_TRUE@ifeq ($(strip $(TESTS)),)
 slow_tests = gdb.base/break-interp.exp gdb.base/interp.exp \
 	gdb.base/multi-forks.exp
 @GMAKE_TRUE@all_tests := $(shell cd $(srcdir) && find gdb.* -name '*.exp' -print)
 @GMAKE_TRUE@reordered_tests := $(slow_tests) $(filter-out $(slow_tests),$(all_tests))
 @GMAKE_TRUE@TEST_TARGETS := $(addprefix check/,$(reordered_tests))
+@GMAKE_TRUE@else
+@GMAKE_TRUE@all_tests := $(shell cd $(srcdir) && echo $(TESTS))
+@GMAKE_TRUE@TEST_TARGETS := $(addprefix check/,$(all_tests))
+@GMAKE_TRUE@endif
 
 do-check-parallel: $(TEST_TARGETS)
 	@:

Re: [PATCH] make -j10 check TESTS=gdb.foo/*.exp

[Yao Qi]

On 02/13/2014 09:25 AM, Doug Evans wrote:
> This patch implements this idea.
> 
> E.g.,
> bash$ make -j10 check TESTS="gdb.server/*.exp gdb.threads/*.exp"

This is good.  I am using RUNTESTFLAGS='--directory=gdb.server', but
impossible to specify two directories in one run.  We also need to
update README in testsuite directory to document it.

-- 
Yao (齐尧)

Re: [PATCH] make -j10 check TESTS=gdb.foo/*.exp

[Pedro Alves]

On 02/13/2014 01:25 AM, Doug Evans wrote:

> E.g.,
> bash$ make -j10 check TESTS="gdb.server/*.exp gdb.threads/*.exp"

Brilliant!

-- 
Pedro Alves

[PATCH v2] make -j10 check TESTS=gdb.foo/*.exp

[Doug Evans]

Yao Qi writes:
 > On 02/13/2014 09:25 AM, Doug Evans wrote:
 > > This patch implements this idea.
 > > 
 > > E.g.,
 > > bash$ make -j10 check TESTS="gdb.server/*.exp gdb.threads/*.exp"
 > 
 > This is good.  I am using RUNTESTFLAGS='--directory=gdb.server', but
 > impossible to specify two directories in one run.  We also need to
 > update README in testsuite directory to document it.

This patch improves things a little bit.
It makes TESTS=foo work for non-parallel mode as well,
and handles the "no matching tests found" case a little cleaner.
[At some point I think one can get carried away with polishing
up this stuff, but I'm happy to take it this far.]

2014-02-13  Doug Evans  <dje@google.com>

	* Makefile.in (TESTS): New variable.
	(expanded_tests, expanded_tests_or_none): New variables
	(check-single): Pass $(expanded_tests_or_none) to runtest.
	(check-parallel): Only run tests in $(TESTS) if non-empty.
	(check/no-matching-tests-found): New rule.
	* README: Document TESTS makefile variable.

diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in
index 7a44660..cbe39db 100644
--- a/gdb/testsuite/Makefile.in
+++ b/gdb/testsuite/Makefile.in
@@ -160,8 +160,25 @@ DO_RUNTEST = \
 	  export TCL_LIBRARY ; fi ; \
 	$(RUNTEST)
 
+# TESTS exists for the user to pass on the command line to easily
+# say "Only run these tests."  With check-single it's not necessary, but
+# with check-parallel there's no other way to (easily) specify a subset
+# of tests.  For consistency we support it for check-single as well.
+# To specify all tests in a subdirectory, use TESTS=gdb.subdir/*.exp.
+# E.g., make check TESTS="gdb.server/*.exp gdb.threads/*.exp".
+@GMAKE_TRUE@TESTS :=
+@GMAKE_FALSE@TESTS =
+
+@GMAKE_TRUE@ifeq ($(strip $(TESTS)),)
+@GMAKE_TRUE@expanded_tests_or_none :=
+@GMAKE_TRUE@else
+@GMAKE_TRUE@expanded_tests := $(patsubst $(srcdir)/%,%,$(wildcard $(addprefix $(srcdir)/,$(TESTS))))
+@GMAKE_TRUE@expanded_tests_or_none := $(or $(expanded_tests),no-matching-tests-found)
+@GMAKE_TRUE@endif
+@GMAKE_FALSE@expanded_tests_or_none = $(TESTS)
+
 check-single:
-	$(DO_RUNTEST) $(RUNTESTFLAGS)
+	$(DO_RUNTEST) $(RUNTESTFLAGS) $(expanded_tests_or_none)
 
 check-parallel:
 	-rm -rf cache
@@ -180,11 +197,15 @@ check-parallel:
 # them to the front of the list to try to lessen the overall time
 # taken by the test suite -- if one of these tests happens to be run
 # late, it will cause the overall time to increase.
+@GMAKE_TRUE@ifeq ($(strip $(TESTS)),)
 slow_tests = gdb.base/break-interp.exp gdb.base/interp.exp \
 	gdb.base/multi-forks.exp
 @GMAKE_TRUE@all_tests := $(shell cd $(srcdir) && find gdb.* -name '*.exp' -print)
 @GMAKE_TRUE@reordered_tests := $(slow_tests) $(filter-out $(slow_tests),$(all_tests))
 @GMAKE_TRUE@TEST_TARGETS := $(addprefix check/,$(reordered_tests))
+@GMAKE_TRUE@else
+@GMAKE_TRUE@TEST_TARGETS := $(addprefix check/,$(expanded_tests_or_none))
+@GMAKE_TRUE@endif
 
 do-check-parallel: $(TEST_TARGETS)
 	@:
@@ -193,6 +214,11 @@ do-check-parallel: $(TEST_TARGETS)
 @GMAKE_TRUE@	-mkdir -p outputs/$*
 @GMAKE_TRUE@	@$(DO_RUNTEST) GDB_PARALLEL=yes --outdir=outputs/$* $*.exp $(RUNTESTFLAGS)
 
+check/no-matching-tests-found:
+	@echo ""
+	@echo "No matching tests found."
+	@echo ""
+
 check-perf: all $(abs_builddir)/site.exp
 	@if test ! -d gdb.perf; then mkdir gdb.perf; fi
 	$(DO_RUNTEST) --directory=gdb.perf --outdir gdb.perf GDB_PERFTEST_MODE=both $(RUNTESTFLAGS)
diff --git a/gdb/testsuite/README b/gdb/testsuite/README
index c17bd34..e6d5318 100644
--- a/gdb/testsuite/README
+++ b/gdb/testsuite/README
@@ -151,6 +151,30 @@ them, both to stdout and in the test suite log file.
 
 This setting is only meaningful in conjunction with GDB_PARALLEL.
 
+TESTS
+
+This variable is used to specify which set of tests to run.
+It is passed to make (not runtest) and its contents are a space separated
+list of tests to run.
+
+If using GNU make then the contents are wildcard-expanded using
+GNU make's $(wildcard) function.  Test paths must be fully specified,
+relative to the "testsuite" subdirectory.  This allows one to run all
+tests in a subdirectory by passing "gdb.subdir/*.exp".
+If for some strange reason one wanted to run all tests that begin with
+the letter "d" that is also possible: TESTS="*/d*.exp".
+
+Do not write */*.exp to specify all tests (assuming all tests are only
+nested one level deep, which is not necessarily true).  This will pick up
+.exp files in ancillary directories like "lib" and "config".
+Instead write gdb.*/*.exp.
+
+Example:
+
+	make -j10 check TESTS="gdb.server/[s-w]*.exp */x*.exp"
+
+If not using GNU make then the value is passed directly to runtest.
+If not specified, all tests are run.
 
 Testsuite Configuration
 ***********************

Re: [PATCH v2] make -j10 check TESTS=gdb.foo/*.exp

[Doug Evans]

On Thu, Feb 13, 2014 at 12:29 PM, Doug Evans <dje@google.com> wrote:
> Yao Qi writes:
>  > On 02/13/2014 09:25 AM, Doug Evans wrote:
>  > > This patch implements this idea.
>  > >
>  > > E.g.,
>  > > bash$ make -j10 check TESTS="gdb.server/*.exp gdb.threads/*.exp"
>  >
>  > This is good.  I am using RUNTESTFLAGS='--directory=gdb.server', but
>  > impossible to specify two directories in one run.  We also need to
>  > update README in testsuite directory to document it.
>
> This patch improves things a little bit.
> It makes TESTS=foo work for non-parallel mode as well,
> and handles the "no matching tests found" case a little cleaner.
> [At some point I think one can get carried away with polishing
> up this stuff, but I'm happy to take it this far.]
>
> 2014-02-13  Doug Evans  <dje@google.com>
>
>         * Makefile.in (TESTS): New variable.
>         (expanded_tests, expanded_tests_or_none): New variables
>         (check-single): Pass $(expanded_tests_or_none) to runtest.
>         (check-parallel): Only run tests in $(TESTS) if non-empty.
>         (check/no-matching-tests-found): New rule.
>         * README: Document TESTS makefile variable.

I've committed this.