[PATCH 0/2] Make make-check-all.sh more accessible

[PATCH 0/2] Make make-check-all.sh more accessible

[Andrew Burgess]

Love the make-check-all.sh script.  This series aims to make the
script easier to use.

---

Andrew Burgess (2):
  gdb/testsuite: log 'make check' command in make-check-all.sh
  gdb/testsuite: add a new check-all-boards target

 gdb/Makefile.in                 |  8 +++++++
 gdb/testsuite/Makefile.in       | 19 +++++++++++++++
 gdb/testsuite/README            | 42 +++++++++++++++++++++++++++++++++
 gdb/testsuite/make-check-all.sh |  4 ++++
 4 files changed, 73 insertions(+)


base-commit: 2029e13917d53d2289d3ebb390c4f40bd2112d21
-- 
2.25.4

[PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh

[Andrew Burgess]

I have been making more use of the make-check-all.sh script to run
tests against all boards.

But one thing is pretty annoying.  When a test fails on some random
board, I have to run make-check-all.sh with --verbose and --dry-run in
order to see what RUNTESTFLAGS I should be using.

I always run with --keep-results on, so, in this commit, I propose
that, when --keep-results is on the 'make check' command will be
written out to a file within the stored results directory, like:

  check-all/BOARD_NAME/make-check.sh

then, if I want to rerun a test, I can just:

  sh check-all/BOARD_NAME/make-check.sh

and the test will be re-run for me.
---
 gdb/testsuite/make-check-all.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/gdb/testsuite/make-check-all.sh b/gdb/testsuite/make-check-all.sh
index 7d8adb58e87..6d67567758f 100755
--- a/gdb/testsuite/make-check-all.sh
+++ b/gdb/testsuite/make-check-all.sh
@@ -186,6 +186,10 @@ do_tests ()
 
 	mkdir -p "$dir"
 	cp gdb.sum gdb.log "$dir"
+
+	# Record the 'make check' command to enable easy re-running.
+	echo "make check RUNTESTFLAGS=\"${rtf[*]} ${tests[*]}\"" \
+	     > "$dir/make-check.sh"
     fi
 }
 
-- 
2.25.4

[PATCH 2/2] gdb/testsuite: add a new check-all-boards target

[Andrew Burgess]

The make-check-all.sh script (gdb/testsuite/make-check-all.sh) is
great, it makes it super easy to run some test(s) using all the
available board files.

This commit aims to make this script even easier to access by adding a
check-all-boards target to the GDB Makefile.  This new target checks
for (and requires) a number of environment variables, so the target
should be used like this:

  make check-all-boards GDB_TARGET_USERNAME=remote-target \
                        GDB_HOST_USERNAME=remote-host \
			TESTS="gdb.base/break.exp"

Where GDB_TARGET_USERNAME and GDB_HOST_USERNAME are the user names
that should be passed to the make-check-all.sh --target-user and
--host-user command line options respectively.

My personal intention is to set these variables in my environment, so
all I'll need to do is:

  make check-all-boards TESTS="gdb.base/break.exp"

The make rule always passes --keep-results to the make-check-all.sh
script, as I find that the most useful.  It's super frustrating to run
the tests and realise you forgot that option and the results have been
discarded.
---
 gdb/Makefile.in           |  8 ++++++++
 gdb/testsuite/Makefile.in | 19 ++++++++++++++++++
 gdb/testsuite/README      | 42 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 69 insertions(+)

diff --git a/gdb/Makefile.in b/gdb/Makefile.in
index 9c0a0bff2cd..491003c7543 100644
--- a/gdb/Makefile.in
+++ b/gdb/Makefile.in
@@ -1971,6 +1971,14 @@ check-parallel: force
 	  $(MAKE) $(TARGET_FLAGS_TO_PASS) check-parallel; \
 	else true; fi
 
+check-all-boards: force
+	@if [ -f testsuite/Makefile ]; then \
+	  rootme=`pwd`; export rootme; \
+	  rootsrc=`cd $(srcdir); pwd`; export rootsrc; \
+	  cd testsuite; \
+	  $(MAKE) $(TARGET_FLAGS_TO_PASS) check-all-boards; \
+	else true; fi
+
 # The idea is to parallelize testing of multilibs, for example:
 #   make -j3 check//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu}
 # will run 3 concurrent sessions of check, eventually testing all 10
diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in
index 3fc9a907f5b..3488e6acc48 100644
--- a/gdb/testsuite/Makefile.in
+++ b/gdb/testsuite/Makefile.in
@@ -368,6 +368,25 @@ check-perf: all $(abs_builddir)/site.exp
 	@if test ! -d gdb.perf; then mkdir gdb.perf; fi
 	$(DO_RUNTEST) --directory=gdb.perf --outdir gdb.perf GDB_PERFTEST_MODE=$(GDB_PERFTEST_MODE) $(RUNTESTFLAGS) $(TIMESTAMP)
 
+
+check-all-boards: all $(abs_builddir)/site.exp
+	@if [ -z "$(GDB_TARGET_USERNAME)" ]; then \
+		echo "Must define GDB_TARGET_USERNAME"; \
+		false; \
+	fi
+	@if [ -z "$(GDB_HOST_USERNAME)" ]; then \
+		echo "Must define GDB_HOST_USERNAME"; \
+		false; \
+	fi
+	@if [ -z "$(TESTS)" ]; then \
+		echo "Must define TESTS"; \
+		false; \
+	fi
+	${abs_srcdir}/make-check-all.sh --keep-results \
+		--host-user "$(GDB_HOST_USERNAME)" \
+		--target-user "$(GDB_TARGET_USERNAME)" \
+		"$(TESTS)"
+
 force:;
 
 clean mostlyclean:
diff --git a/gdb/testsuite/README b/gdb/testsuite/README
index 330a385f349..5ad0293649e 100644
--- a/gdb/testsuite/README
+++ b/gdb/testsuite/README
@@ -328,6 +328,48 @@ For example, to turn on gdbserver debugging, you can do:
 
 	make check GDBSERVER_DEBUG="debug,replay"
 
+GDB_TARGET_USERNAME
+GDB_HOST_USERNAME
+
+These settings are only used with the check-all-boards target, and
+should be the usernames of two separate users on the local machine,
+both of which the current user can be ssh to without a password.
+
+These users will be used when by board files that simulate remote
+targets by switching to a different user on the same machine.  These
+users will have random files copied into their $HOME directories, so
+it is a good idea to setup new users just for this purpose.
+
+Testing All Simple Boards
+*************************
+
+There are a number of boards that ship with GDB that simulate common
+debug scenarios.  For example by sshing to a different user on the
+local machine and running gdbserver as this alternative user we aim to
+simulate a true remote debug experience.
+
+There is a script binutils-gdb/gdb/testssuite/make-check-all.sh which
+can be used to run a defined set of tests using all of the available
+simple board files.  Support for using this script is also included in
+GDB's makefile, and can be used as:
+
+  make check-all-boards GDB_TARGET_USERNAME=remote-target \
+			GDB_HOST_USERNAME=remote-host \
+			TESTS="gdb.base/break.exp"
+
+The 'remote-target' and 'remote-host' can be replaced with any user
+names on the local machine, the only requirements are that the current
+user must be able to ssh to these users without a password, and these
+users must be happy to have arbitrary files copied into their $HOME
+directory.  Ideally, these users would be setup just for GDB testing.
+
+The check-all-boards target requires that TESTS be defined, though it
+is fine to include multiple tests.
+
+There results are preserved and can be found in
+gdb/testsuite/check-all/, within this directory the results are split
+by the board file used.
+
 Architecture-specific Parameters
 ******************************
 
-- 
2.25.4

Re: [PATCH 0/2] Make make-check-all.sh more accessible

[Tom Tromey]

>>>>> "Andrew" == Andrew Burgess <aburgess@redhat.com> writes:

Andrew> Love the make-check-all.sh script.  This series aims to make the
Andrew> script easier to use.

FWIW I looked at these & didn't see anything amiss.  I have never used
this script so I don't know whether I ought to really be approving it.

Tom

Re: [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh

[Tom de Vries]

On 11/7/23 19:18, Andrew Burgess wrote:
> I have been making more use of the make-check-all.sh script to run
> tests against all boards.
> 

Hi Andrew,

Nice :)

> But one thing is pretty annoying.  When a test fails on some random
> board, I have to run make-check-all.sh with --verbose and --dry-run in
> order to see what RUNTESTFLAGS I should be using.
> 

I wondered why I haven't run into this problem, so I thought I describe 
the answer here, in case that's useful.

I have a setup where I have a test-case.sh file containing for instance:
...
rtf=$(cd src/gdb/testsuite; echo gdb.tui/*.exp gdb.python/tui*.exp)
...
and all my test scripts source it to determine what test-cases to run.

So, rerunning the same set of test-cases is trivial that way.

Then about the host/target board: my regular test script take the board 
files as argument, so if I see after running make-check-all.sh:
...
TARGET BOARD: dwarf4-gdb-index
...
and I want to rerun that configuration, I just run:
...
$ ./test.sh -dwarf4-gdb-index
...

I thought at some point about adding a similar syntax to 
make-check-all.sh, which could be a different way of handling the 
problem you describe.

> I always run with --keep-results on,

Me too.  Maybe we should make that the default?

> so, in this commit, I propose
> that, when --keep-results is on the 'make check' command will be
> written out to a file within the stored results directory, like:
> 
>    check-all/BOARD_NAME/make-check.sh
> 
> then, if I want to rerun a test, I can just:
> 
>    sh check-all/BOARD_NAME/make-check.sh
> 
> and the test will be re-run for me.

I think it's useful, both to use and as log file.

I also wonder whether it would be a good idea to:
- generate the file, and
- run it from make-check-all.sh.
That way we don't have to worry about keeping the "make check" command 
and the record of the "make check" command in sync.

Anyway, as is, LGTM.

Thanks,
- Tom

> ---
>   gdb/testsuite/make-check-all.sh | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/gdb/testsuite/make-check-all.sh b/gdb/testsuite/make-check-all.sh
> index 7d8adb58e87..6d67567758f 100755
> --- a/gdb/testsuite/make-check-all.sh
> +++ b/gdb/testsuite/make-check-all.sh
> @@ -186,6 +186,10 @@ do_tests ()
>   
>   	mkdir -p "$dir"
>   	cp gdb.sum gdb.log "$dir"
> +
> +	# Record the 'make check' command to enable easy re-running.
> +	echo "make check RUNTESTFLAGS=\"${rtf[*]} ${tests[*]}\"" \
> +	     > "$dir/make-check.sh"
>       fi
>   }
>   

Re: [PATCH 2/2] gdb/testsuite: add a new check-all-boards target

[Tom de Vries]

On 11/7/23 19:18, Andrew Burgess wrote:

Hi Andrew,

just a review of the documentation in this reply.

> +GDB_TARGET_USERNAME
> +GDB_HOST_USERNAME
> +
> +These settings are only used with the check-all-boards target, and
> +should be the usernames of two separate users on the local machine,
> +both of which the current user can be ssh to without a password.

can ssh to

> +
> +These users will be used when by board files that simulate remote

used by board files

> +targets by switching to a different user on the same machine.  These
> +users will have random files copied into their $HOME directories, so
> +it is a good idea to setup new users just for this purpose.
> +
> +Testing All Simple Boards
> +*************************
> +
> +There are a number of boards that ship with GDB that simulate common
> +debug scenarios.  For example by sshing to a different user on the
> +local machine and running gdbserver as this alternative user we aim to
> +simulate a true remote debug experience.
> +
> +There is a script binutils-gdb/gdb/testssuite/make-check-all.sh which
> +can be used to run a defined set of tests using all of the available
> +simple board files.  Support for using this script is also included in
> +GDB's makefile, and can be used as:
> +
> +  make check-all-boards GDB_TARGET_USERNAME=remote-target \
> +			GDB_HOST_USERNAME=remote-host \
> +			TESTS="gdb.base/break.exp"
> +
> +The 'remote-target' and 'remote-host' can be replaced with any user
> +names on the local machine, the only requirements are that the current
> +user must be able to ssh to these users without a password, and these
> +users must be happy to have arbitrary files copied into their $HOME
> +directory.  Ideally, these users would be setup just for GDB testing.
> +
> +The check-all-boards target requires that TESTS be defined, though it
> +is fine to include multiple tests.
> +
> +There results are preserved and can be found in

The results

> +gdb/testsuite/check-all/, within this directory the results are split
> +by the board file used.
> +

Thanks,
- Tom

Re: [PATCH 0/2] Make make-check-all.sh more accessible

[Andrew Burgess]

Andrew Burgess <aburgess@redhat.com> writes:

> Love the make-check-all.sh script.  This series aims to make the
> script easier to use.

Tom & Tom,

Thanks for your feedback.  I made the doc fixes Tom suggested in patch
#2 and pushed this series.

I agree with Tom that there are likely more improvements that we could
make to the make-check-all.sh script, but as this is not user facing,
and having these changes in would make my life easier, I'm going to push
what I have for now ... in time we might bring more improvements to this
script, and hopefully get more folk using it.

Thanks,
Andrew


>
> ---
>
> Andrew Burgess (2):
>   gdb/testsuite: log 'make check' command in make-check-all.sh
>   gdb/testsuite: add a new check-all-boards target
>
>  gdb/Makefile.in                 |  8 +++++++
>  gdb/testsuite/Makefile.in       | 19 +++++++++++++++
>  gdb/testsuite/README            | 42 +++++++++++++++++++++++++++++++++
>  gdb/testsuite/make-check-all.sh |  4 ++++
>  4 files changed, 73 insertions(+)
>
>
> base-commit: 2029e13917d53d2289d3ebb390c4f40bd2112d21
> -- 
> 2.25.4