* [PATCH 0/2] Make make-check-all.sh more accessible @ 2023-11-07 18:18 Andrew Burgess 2023-11-07 18:18 ` [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh Andrew Burgess ` (3 more replies) 0 siblings, 4 replies; 7+ messages in thread From: Andrew Burgess @ 2023-11-07 18:18 UTC (permalink / raw) To: gdb-patches; +Cc: Andrew Burgess Love the make-check-all.sh script. This series aims to make the script easier to use. --- Andrew Burgess (2): gdb/testsuite: log 'make check' command in make-check-all.sh gdb/testsuite: add a new check-all-boards target gdb/Makefile.in | 8 +++++++ gdb/testsuite/Makefile.in | 19 +++++++++++++++ gdb/testsuite/README | 42 +++++++++++++++++++++++++++++++++ gdb/testsuite/make-check-all.sh | 4 ++++ 4 files changed, 73 insertions(+) base-commit: 2029e13917d53d2289d3ebb390c4f40bd2112d21 -- 2.25.4 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh 2023-11-07 18:18 [PATCH 0/2] Make make-check-all.sh more accessible Andrew Burgess @ 2023-11-07 18:18 ` Andrew Burgess 2023-11-24 15:36 ` Tom de Vries 2023-11-07 18:18 ` [PATCH 2/2] gdb/testsuite: add a new check-all-boards target Andrew Burgess ` (2 subsequent siblings) 3 siblings, 1 reply; 7+ messages in thread From: Andrew Burgess @ 2023-11-07 18:18 UTC (permalink / raw) To: gdb-patches; +Cc: Andrew Burgess I have been making more use of the make-check-all.sh script to run tests against all boards. But one thing is pretty annoying. When a test fails on some random board, I have to run make-check-all.sh with --verbose and --dry-run in order to see what RUNTESTFLAGS I should be using. I always run with --keep-results on, so, in this commit, I propose that, when --keep-results is on the 'make check' command will be written out to a file within the stored results directory, like: check-all/BOARD_NAME/make-check.sh then, if I want to rerun a test, I can just: sh check-all/BOARD_NAME/make-check.sh and the test will be re-run for me. --- gdb/testsuite/make-check-all.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/gdb/testsuite/make-check-all.sh b/gdb/testsuite/make-check-all.sh index 7d8adb58e87..6d67567758f 100755 --- a/gdb/testsuite/make-check-all.sh +++ b/gdb/testsuite/make-check-all.sh @@ -186,6 +186,10 @@ do_tests () mkdir -p "$dir" cp gdb.sum gdb.log "$dir" + + # Record the 'make check' command to enable easy re-running. + echo "make check RUNTESTFLAGS=\"${rtf[*]} ${tests[*]}\"" \ + > "$dir/make-check.sh" fi } -- 2.25.4 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh 2023-11-07 18:18 ` [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh Andrew Burgess @ 2023-11-24 15:36 ` Tom de Vries 0 siblings, 0 replies; 7+ messages in thread From: Tom de Vries @ 2023-11-24 15:36 UTC (permalink / raw) To: Andrew Burgess, gdb-patches On 11/7/23 19:18, Andrew Burgess wrote: > I have been making more use of the make-check-all.sh script to run > tests against all boards. > Hi Andrew, Nice :) > But one thing is pretty annoying. When a test fails on some random > board, I have to run make-check-all.sh with --verbose and --dry-run in > order to see what RUNTESTFLAGS I should be using. > I wondered why I haven't run into this problem, so I thought I describe the answer here, in case that's useful. I have a setup where I have a test-case.sh file containing for instance: ... rtf=$(cd src/gdb/testsuite; echo gdb.tui/*.exp gdb.python/tui*.exp) ... and all my test scripts source it to determine what test-cases to run. So, rerunning the same set of test-cases is trivial that way. Then about the host/target board: my regular test script take the board files as argument, so if I see after running make-check-all.sh: ... TARGET BOARD: dwarf4-gdb-index ... and I want to rerun that configuration, I just run: ... $ ./test.sh -dwarf4-gdb-index ... I thought at some point about adding a similar syntax to make-check-all.sh, which could be a different way of handling the problem you describe. > I always run with --keep-results on, Me too. Maybe we should make that the default? > so, in this commit, I propose > that, when --keep-results is on the 'make check' command will be > written out to a file within the stored results directory, like: > > check-all/BOARD_NAME/make-check.sh > > then, if I want to rerun a test, I can just: > > sh check-all/BOARD_NAME/make-check.sh > > and the test will be re-run for me. I think it's useful, both to use and as log file. I also wonder whether it would be a good idea to: - generate the file, and - run it from make-check-all.sh. That way we don't have to worry about keeping the "make check" command and the record of the "make check" command in sync. Anyway, as is, LGTM. Thanks, - Tom > --- > gdb/testsuite/make-check-all.sh | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/gdb/testsuite/make-check-all.sh b/gdb/testsuite/make-check-all.sh > index 7d8adb58e87..6d67567758f 100755 > --- a/gdb/testsuite/make-check-all.sh > +++ b/gdb/testsuite/make-check-all.sh > @@ -186,6 +186,10 @@ do_tests () > > mkdir -p "$dir" > cp gdb.sum gdb.log "$dir" > + > + # Record the 'make check' command to enable easy re-running. > + echo "make check RUNTESTFLAGS=\"${rtf[*]} ${tests[*]}\"" \ > + > "$dir/make-check.sh" > fi > } > ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/2] gdb/testsuite: add a new check-all-boards target 2023-11-07 18:18 [PATCH 0/2] Make make-check-all.sh more accessible Andrew Burgess 2023-11-07 18:18 ` [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh Andrew Burgess @ 2023-11-07 18:18 ` Andrew Burgess 2023-11-24 16:14 ` Tom de Vries 2023-11-13 17:29 ` [PATCH 0/2] Make make-check-all.sh more accessible Tom Tromey 2023-11-28 10:46 ` Andrew Burgess 3 siblings, 1 reply; 7+ messages in thread From: Andrew Burgess @ 2023-11-07 18:18 UTC (permalink / raw) To: gdb-patches; +Cc: Andrew Burgess The make-check-all.sh script (gdb/testsuite/make-check-all.sh) is great, it makes it super easy to run some test(s) using all the available board files. This commit aims to make this script even easier to access by adding a check-all-boards target to the GDB Makefile. This new target checks for (and requires) a number of environment variables, so the target should be used like this: make check-all-boards GDB_TARGET_USERNAME=remote-target \ GDB_HOST_USERNAME=remote-host \ TESTS="gdb.base/break.exp" Where GDB_TARGET_USERNAME and GDB_HOST_USERNAME are the user names that should be passed to the make-check-all.sh --target-user and --host-user command line options respectively. My personal intention is to set these variables in my environment, so all I'll need to do is: make check-all-boards TESTS="gdb.base/break.exp" The make rule always passes --keep-results to the make-check-all.sh script, as I find that the most useful. It's super frustrating to run the tests and realise you forgot that option and the results have been discarded. --- gdb/Makefile.in | 8 ++++++++ gdb/testsuite/Makefile.in | 19 ++++++++++++++++++ gdb/testsuite/README | 42 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 69 insertions(+) diff --git a/gdb/Makefile.in b/gdb/Makefile.in index 9c0a0bff2cd..491003c7543 100644 --- a/gdb/Makefile.in +++ b/gdb/Makefile.in @@ -1971,6 +1971,14 @@ check-parallel: force $(MAKE) $(TARGET_FLAGS_TO_PASS) check-parallel; \ else true; fi +check-all-boards: force + @if [ -f testsuite/Makefile ]; then \ + rootme=`pwd`; export rootme; \ + rootsrc=`cd $(srcdir); pwd`; export rootsrc; \ + cd testsuite; \ + $(MAKE) $(TARGET_FLAGS_TO_PASS) check-all-boards; \ + else true; fi + # The idea is to parallelize testing of multilibs, for example: # make -j3 check//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu} # will run 3 concurrent sessions of check, eventually testing all 10 diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in index 3fc9a907f5b..3488e6acc48 100644 --- a/gdb/testsuite/Makefile.in +++ b/gdb/testsuite/Makefile.in @@ -368,6 +368,25 @@ check-perf: all $(abs_builddir)/site.exp @if test ! -d gdb.perf; then mkdir gdb.perf; fi $(DO_RUNTEST) --directory=gdb.perf --outdir gdb.perf GDB_PERFTEST_MODE=$(GDB_PERFTEST_MODE) $(RUNTESTFLAGS) $(TIMESTAMP) + +check-all-boards: all $(abs_builddir)/site.exp + @if [ -z "$(GDB_TARGET_USERNAME)" ]; then \ + echo "Must define GDB_TARGET_USERNAME"; \ + false; \ + fi + @if [ -z "$(GDB_HOST_USERNAME)" ]; then \ + echo "Must define GDB_HOST_USERNAME"; \ + false; \ + fi + @if [ -z "$(TESTS)" ]; then \ + echo "Must define TESTS"; \ + false; \ + fi + ${abs_srcdir}/make-check-all.sh --keep-results \ + --host-user "$(GDB_HOST_USERNAME)" \ + --target-user "$(GDB_TARGET_USERNAME)" \ + "$(TESTS)" + force:; clean mostlyclean: diff --git a/gdb/testsuite/README b/gdb/testsuite/README index 330a385f349..5ad0293649e 100644 --- a/gdb/testsuite/README +++ b/gdb/testsuite/README @@ -328,6 +328,48 @@ For example, to turn on gdbserver debugging, you can do: make check GDBSERVER_DEBUG="debug,replay" +GDB_TARGET_USERNAME +GDB_HOST_USERNAME + +These settings are only used with the check-all-boards target, and +should be the usernames of two separate users on the local machine, +both of which the current user can be ssh to without a password. + +These users will be used when by board files that simulate remote +targets by switching to a different user on the same machine. These +users will have random files copied into their $HOME directories, so +it is a good idea to setup new users just for this purpose. + +Testing All Simple Boards +************************* + +There are a number of boards that ship with GDB that simulate common +debug scenarios. For example by sshing to a different user on the +local machine and running gdbserver as this alternative user we aim to +simulate a true remote debug experience. + +There is a script binutils-gdb/gdb/testssuite/make-check-all.sh which +can be used to run a defined set of tests using all of the available +simple board files. Support for using this script is also included in +GDB's makefile, and can be used as: + + make check-all-boards GDB_TARGET_USERNAME=remote-target \ + GDB_HOST_USERNAME=remote-host \ + TESTS="gdb.base/break.exp" + +The 'remote-target' and 'remote-host' can be replaced with any user +names on the local machine, the only requirements are that the current +user must be able to ssh to these users without a password, and these +users must be happy to have arbitrary files copied into their $HOME +directory. Ideally, these users would be setup just for GDB testing. + +The check-all-boards target requires that TESTS be defined, though it +is fine to include multiple tests. + +There results are preserved and can be found in +gdb/testsuite/check-all/, within this directory the results are split +by the board file used. + Architecture-specific Parameters ****************************** -- 2.25.4 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 2/2] gdb/testsuite: add a new check-all-boards target 2023-11-07 18:18 ` [PATCH 2/2] gdb/testsuite: add a new check-all-boards target Andrew Burgess @ 2023-11-24 16:14 ` Tom de Vries 0 siblings, 0 replies; 7+ messages in thread From: Tom de Vries @ 2023-11-24 16:14 UTC (permalink / raw) To: Andrew Burgess, gdb-patches On 11/7/23 19:18, Andrew Burgess wrote: Hi Andrew, just a review of the documentation in this reply. > +GDB_TARGET_USERNAME > +GDB_HOST_USERNAME > + > +These settings are only used with the check-all-boards target, and > +should be the usernames of two separate users on the local machine, > +both of which the current user can be ssh to without a password. can ssh to > + > +These users will be used when by board files that simulate remote used by board files > +targets by switching to a different user on the same machine. These > +users will have random files copied into their $HOME directories, so > +it is a good idea to setup new users just for this purpose. > + > +Testing All Simple Boards > +************************* > + > +There are a number of boards that ship with GDB that simulate common > +debug scenarios. For example by sshing to a different user on the > +local machine and running gdbserver as this alternative user we aim to > +simulate a true remote debug experience. > + > +There is a script binutils-gdb/gdb/testssuite/make-check-all.sh which > +can be used to run a defined set of tests using all of the available > +simple board files. Support for using this script is also included in > +GDB's makefile, and can be used as: > + > + make check-all-boards GDB_TARGET_USERNAME=remote-target \ > + GDB_HOST_USERNAME=remote-host \ > + TESTS="gdb.base/break.exp" > + > +The 'remote-target' and 'remote-host' can be replaced with any user > +names on the local machine, the only requirements are that the current > +user must be able to ssh to these users without a password, and these > +users must be happy to have arbitrary files copied into their $HOME > +directory. Ideally, these users would be setup just for GDB testing. > + > +The check-all-boards target requires that TESTS be defined, though it > +is fine to include multiple tests. > + > +There results are preserved and can be found in The results > +gdb/testsuite/check-all/, within this directory the results are split > +by the board file used. > + Thanks, - Tom ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] Make make-check-all.sh more accessible 2023-11-07 18:18 [PATCH 0/2] Make make-check-all.sh more accessible Andrew Burgess 2023-11-07 18:18 ` [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh Andrew Burgess 2023-11-07 18:18 ` [PATCH 2/2] gdb/testsuite: add a new check-all-boards target Andrew Burgess @ 2023-11-13 17:29 ` Tom Tromey 2023-11-28 10:46 ` Andrew Burgess 3 siblings, 0 replies; 7+ messages in thread From: Tom Tromey @ 2023-11-13 17:29 UTC (permalink / raw) To: Andrew Burgess; +Cc: gdb-patches >>>>> "Andrew" == Andrew Burgess <aburgess@redhat.com> writes: Andrew> Love the make-check-all.sh script. This series aims to make the Andrew> script easier to use. FWIW I looked at these & didn't see anything amiss. I have never used this script so I don't know whether I ought to really be approving it. Tom ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] Make make-check-all.sh more accessible 2023-11-07 18:18 [PATCH 0/2] Make make-check-all.sh more accessible Andrew Burgess ` (2 preceding siblings ...) 2023-11-13 17:29 ` [PATCH 0/2] Make make-check-all.sh more accessible Tom Tromey @ 2023-11-28 10:46 ` Andrew Burgess 3 siblings, 0 replies; 7+ messages in thread From: Andrew Burgess @ 2023-11-28 10:46 UTC (permalink / raw) To: gdb-patches; +Cc: Tom de Vries, Tom Tromey Andrew Burgess <aburgess@redhat.com> writes: > Love the make-check-all.sh script. This series aims to make the > script easier to use. Tom & Tom, Thanks for your feedback. I made the doc fixes Tom suggested in patch #2 and pushed this series. I agree with Tom that there are likely more improvements that we could make to the make-check-all.sh script, but as this is not user facing, and having these changes in would make my life easier, I'm going to push what I have for now ... in time we might bring more improvements to this script, and hopefully get more folk using it. Thanks, Andrew > > --- > > Andrew Burgess (2): > gdb/testsuite: log 'make check' command in make-check-all.sh > gdb/testsuite: add a new check-all-boards target > > gdb/Makefile.in | 8 +++++++ > gdb/testsuite/Makefile.in | 19 +++++++++++++++ > gdb/testsuite/README | 42 +++++++++++++++++++++++++++++++++ > gdb/testsuite/make-check-all.sh | 4 ++++ > 4 files changed, 73 insertions(+) > > > base-commit: 2029e13917d53d2289d3ebb390c4f40bd2112d21 > -- > 2.25.4 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-11-28 10:46 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-11-07 18:18 [PATCH 0/2] Make make-check-all.sh more accessible Andrew Burgess 2023-11-07 18:18 ` [PATCH 1/2] gdb/testsuite: log 'make check' command in make-check-all.sh Andrew Burgess 2023-11-24 15:36 ` Tom de Vries 2023-11-07 18:18 ` [PATCH 2/2] gdb/testsuite: add a new check-all-boards target Andrew Burgess 2023-11-24 16:14 ` Tom de Vries 2023-11-13 17:29 ` [PATCH 0/2] Make make-check-all.sh more accessible Tom Tromey 2023-11-28 10:46 ` Andrew Burgess
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).