[PATCH 0/1] [gdb/testsuite] Run test-case with all host/target boards

[PATCH 0/1] [gdb/testsuite] Run test-case with all host/target boards

[Tom de Vries]

Directory gdb/testsuite/boards contains a number of host/target boards, which
run a test-case (or test-cases) in a different way.

The benefits of using these boards are:
- improving test coverage of gdb,
- making the testsuite more robust, and
- making sure the test-cases work for non-native and remote setups.

Each board is slightly different, and developers need to learn how to use each
one, what parameters to pass and how, which ones can be using in combination
with each other.  This is a threshold to start using them.

And then there quite a few, so typically only a few will be used by each
developer (I think).

This patch makes it possible to run a test-case with all the boards, using a
single invocation.  The script is designed as a make check drop-in
replacement (though in the strictest sense it's not, because it doesn't use
RUNTESTFLAGS).

An example make check use is:
...
$ cd build/gdb/testsuite
$ make check RUNTESTFLAGS=gdb.base/advance.exp
  ...
                 === gdb Summary ===

 # of expected passes            8
 gdb version  14.0.50.20230403-git -nw -nx -iex "set height 0" \
   -iex "set width 0" -data-directory ./build/gdb/data-directory
...

And an example of make-check-all.sh for the same test-case is:
...
 $  ~/gdb/src/gdb/contrib/make-check-all.sh gdb.base/advance.exp
 LOCAL:
 # of expected passes            8
 TARGET BOARD: cc-with-gdb-index
 # of expected passes            8
 TARGET BOARD: cc-with-debug-names
 # of expected passes            8
 TARGET BOARD: cc-with-dwz
 # of expected passes            8
 TARGET BOARD: cc-with-dwz-m
 # of expected passes            8
 TARGET BOARD: cc-with-gnu-debuglink
 # of expected passes            8
 TARGET BOARD: debug-types
 # of expected passes            8
 TARGET BOARD: dwarf4-gdb-index
 # of expected passes            8
 TARGET BOARD: dwarf64
 # of untested testcases         1
 TARGET BOARD: fission
 # of expected passes            8
 TARGET BOARD: fission-dwp
 # of expected passes            8
 TARGET BOARD: gold
 # of expected passes            8
 TARGET BOARD: gold-gdb-index
 # of expected passes            8
 TARGET BOARD: readnow
 # of expected passes            8
 TARGET BOARD: stabs
 # of expected passes            8
 TARGET BOARD: native-extended-gdbserver
 # of expected passes            8
 TARGET BOARD: native-gdbserver
 # of expected passes            8
 TARGET BOARD: native-stdio-gdbserver
 # of expected passes            8
 TARGET BOARD: remote-gdbserver-on-localhost
 # of expected passes            8
 TARGET BOARD: remote-stdio-gdbserver
 # of expected passes            8
 HOST BOARD: local-remote-host, TARGET BOARD: remote-gdbserver-on-localhost
 # of expected passes            8
 HOST BOARD: local-remote-host, TARGET BOARD: remote-stdio-gdbserver
 # of expected passes            8
 HOST BOARD: local-remote-host-notty, TARGET BOARD: remote-gdbserver-on-localhost
 # of expected passes            8
 HOST BOARD: local-remote-host-notty, TARGET BOARD: remote-stdio-gdbserver
 # of expected passes            8
 HOST/TARGET BOARD: local-remote-host-native
 # of expected passes            8
...

Note that the dwarf64 board doesn't work, because my system compiler is gcc
7.5.0, which doesn't support -gdwarf64.

An annoying property of the remote host/target boards is that they dump
files into the home dir.  This problem can be worked around by using the
--host-user and --target-user switches, in combination with test accounts.

Using the test accounts has the additional benefit that it triggers more strict
checking of build/host/target file manipulations.

Tom de Vries (1):
  [gdb/contrib] Add make-check-all.sh

 gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100755 gdb/contrib/make-check-all.sh


base-commit: 2e411b8c68eb2b035b31d5b00d940d4be1a0928b
-- 
2.35.3

[PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Tom de Vries]

Add script gdb/contrib/make-check-all.sh, that's intended to function as a
drop-in replacement of make check, but excercising all host/target boards in
gdb/testsuite/boards.

Shell-checked and tested on x86_64-linux.
---
 gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100755 gdb/contrib/make-check-all.sh

diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
new file mode 100755
index 00000000000..1befe418b81
--- /dev/null
+++ b/gdb/contrib/make-check-all.sh
@@ -0,0 +1,255 @@
+#!/bin/bash
+
+# Copyright (C) 2023 Free Software Foundation, Inc.
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Run make check with all boards from gdb/testsuite/boards.
+
+host_boards=(
+    local-remote-host
+    local-remote-host-notty
+)
+
+gdbserver_boards=(
+    native-extended-gdbserver
+    native-gdbserver
+    native-stdio-gdbserver
+)
+
+remote_gdbserver_boards=(
+    remote-gdbserver-on-localhost
+    remote-stdio-gdbserver
+)
+
+host_target_boards=(
+    local-remote-host-native
+)
+
+target_boards=(
+    cc-with-gdb-index
+    cc-with-debug-names
+    cc-with-dwz
+    cc-with-dwz-m
+    cc-with-gnu-debuglink
+    debug-types
+    dwarf4-gdb-index
+    dwarf64
+    fission
+    fission-dwp
+    gold
+    gold-gdb-index
+    readnow
+    stabs
+)
+
+rtf_for_board ()
+{
+    local b
+    b="$1"
+
+    case $b in
+	local-remote-host-native)
+	    mkdir -p "$tmpdir/$b"
+	    rtf=(
+		"${rtf[@]}"
+		"HOST_DIR=$tmpdir/$b"
+	    )
+	    ;;
+	remote-stdio-gdbserver)
+	    rtf=(
+		"${rtf[@]}"
+		"REMOTE_HOSTNAME=localhost"
+	    )
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$target_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	remote-gdbserver-on-localhost)
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_TARGET_USERNAME=$target_user"
+		)
+	    fi
+	    ;;
+	local-remote-host|local-remote-host-notty)
+	    if [ "$host_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$host_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	*)
+	    ;;
+    esac
+}
+
+summary ()
+{
+    if $verbose; then
+	cat
+    else
+	grep -E "^(#|FAIL:|ERROR:|WARNING:)" \
+	    | sort -u
+    fi
+}
+
+do_tests ()
+{
+    if $debug; then
+	echo "RTF: ${rtf[*]}"
+    fi
+
+    if $dry_run; then
+	return
+    fi
+
+    make check \
+	 RUNTESTFLAGS="${rtf[*]} ${tests[*]}" \
+	 2>&1 \
+	| summary
+}
+
+parse_args ()
+{
+    debug=false
+    keep=false
+    verbose=false
+    dry_run=false
+
+    host_user=""
+    target_user=""
+
+    while [ $# -gt 0 ]; do
+	case "$1" in
+	    --debug)
+		debug=true
+		;;
+	    --keep)
+		keep=true
+		;;
+	    --verbose)
+		verbose=true
+		;;
+	    --dry-run)
+		dry_run=true
+		;;
+	    --host-user)
+		shift
+		host_user="$1"
+		;;
+	    --target-user)
+		shift
+		target_user="$1"
+		;;
+	    *)
+		break
+		;;
+	esac
+	shift
+    done
+
+    tests=("$@")
+}
+
+main ()
+{
+    local b
+    local h
+
+    parse_args "$@"
+
+    tmpdir=$(mktemp -d)
+
+    if $debug; then
+	echo "TESTS: ${tests[*]}"
+    fi
+
+    if true; then
+	echo "LOCAL:"
+	rtf=()
+	do_tests
+    fi
+
+    if true; then
+	for b in "${target_boards[@]}"; do
+	    echo "TARGET BOARD: $b"
+	    rtf=(
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if true; then
+	for b in "${gdbserver_boards[@]}" "${remote_gdbserver_boards[@]}"; do
+	    echo "TARGET BOARD: $b"
+	    rtf=(
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if true; then
+	for h in "${host_boards[@]}"; do
+	    for b in "${remote_gdbserver_boards[@]}"; do
+		echo "HOST BOARD: $h, TARGET BOARD: $b"
+		rtf=(
+		    --host_board="$h"
+		    --target_board="$b"
+		)
+		rtf_for_board "$h"
+		rtf_for_board "$b"
+		do_tests
+	    done
+	done
+    fi
+
+    if true; then
+	for b in "${host_target_boards[@]}"; do
+	    echo "HOST/TARGET BOARD: $b"
+	    rtf=(
+		--host_board="$b"
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if $keep; then
+	echo "keeping tmp dir $tmpdir"
+    else
+	rm -Rf "$tmpdir"
+    fi
+}
+
+main "$@"
-- 
2.35.3

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Andrew Burgess]

Tom de Vries via Gdb-patches <gdb-patches@sourceware.org> writes:

> Add script gdb/contrib/make-check-all.sh, that's intended to function as a
> drop-in replacement of make check, but excercising all host/target boards in
> gdb/testsuite/boards.
>
> Shell-checked and tested on x86_64-linux.

Hi Tom,

Thanks for putting this together, I think this could be really useful.

I'm not a fan of the way you've split the patch description into email
0/1 and not included it with this commit.  I think there's lots of
useful information in there, and I'd much rather have the whole
description included in the commit message -- it's much easier to find
then rather than having to hunt on the mailing list in the future.

> ---
>  gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
>  1 file changed, 255 insertions(+)
>  create mode 100755 gdb/contrib/make-check-all.sh
>
> diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
> new file mode 100755
> index 00000000000..1befe418b81
> --- /dev/null
> +++ b/gdb/contrib/make-check-all.sh
> @@ -0,0 +1,255 @@
> +#!/bin/bash
> +
> +# Copyright (C) 2023 Free Software Foundation, Inc.
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Run make check with all boards from gdb/testsuite/boards.

I think you need some kind of usage test here too.  Imagine a new
developer finds this file and looks inside.  Ideally, I think they
should know how to use it, and what it will do for them without having
to read any of the actual script content.

On implementation, I wonder if it would be useful to provide a mechanism
by which the gdb.sum and gdb.log files for each test run could be
preserved?

Thanks,
Andrew

> +
> +host_boards=(
> +    local-remote-host
> +    local-remote-host-notty
> +)
> +
> +gdbserver_boards=(
> +    native-extended-gdbserver
> +    native-gdbserver
> +    native-stdio-gdbserver
> +)
> +
> +remote_gdbserver_boards=(
> +    remote-gdbserver-on-localhost
> +    remote-stdio-gdbserver
> +)
> +
> +host_target_boards=(
> +    local-remote-host-native
> +)
> +
> +target_boards=(
> +    cc-with-gdb-index
> +    cc-with-debug-names
> +    cc-with-dwz
> +    cc-with-dwz-m
> +    cc-with-gnu-debuglink
> +    debug-types
> +    dwarf4-gdb-index
> +    dwarf64
> +    fission
> +    fission-dwp
> +    gold
> +    gold-gdb-index
> +    readnow
> +    stabs
> +)
> +
> +rtf_for_board ()
> +{
> +    local b
> +    b="$1"
> +
> +    case $b in
> +	local-remote-host-native)
> +	    mkdir -p "$tmpdir/$b"
> +	    rtf=(
> +		"${rtf[@]}"
> +		"HOST_DIR=$tmpdir/$b"
> +	    )
> +	    ;;
> +	remote-stdio-gdbserver)
> +	    rtf=(
> +		"${rtf[@]}"
> +		"REMOTE_HOSTNAME=localhost"
> +	    )
> +	    if [ "$target_user" != "" ]; then
> +		rtf=(
> +		    "${rtf[@]}"
> +		    "REMOTE_USERNAME=$target_user"
> +		)
> +	    else
> +		rtf=(
> +		    "${rtf[@]}"
> +		    "REMOTE_USERNAME=$USER"
> +		)
> +	    fi
> +	    ;;
> +	remote-gdbserver-on-localhost)
> +	    if [ "$target_user" != "" ]; then
> +		rtf=(
> +		    "${rtf[@]}"
> +		    "REMOTE_TARGET_USERNAME=$target_user"
> +		)
> +	    fi
> +	    ;;
> +	local-remote-host|local-remote-host-notty)
> +	    if [ "$host_user" != "" ]; then
> +		rtf=(
> +		    "${rtf[@]}"
> +		    "REMOTE_HOST_USERNAME=$host_user"
> +		)
> +	    else
> +		rtf=(
> +		    "${rtf[@]}"
> +		    "REMOTE_HOST_USERNAME=$USER"
> +		)
> +	    fi
> +	    ;;
> +	*)
> +	    ;;
> +    esac
> +}
> +
> +summary ()
> +{
> +    if $verbose; then
> +	cat
> +    else
> +	grep -E "^(#|FAIL:|ERROR:|WARNING:)" \
> +	    | sort -u
> +    fi
> +}
> +
> +do_tests ()
> +{
> +    if $debug; then
> +	echo "RTF: ${rtf[*]}"
> +    fi
> +
> +    if $dry_run; then
> +	return
> +    fi
> +
> +    make check \
> +	 RUNTESTFLAGS="${rtf[*]} ${tests[*]}" \
> +	 2>&1 \
> +	| summary
> +}
> +
> +parse_args ()
> +{
> +    debug=false
> +    keep=false
> +    verbose=false
> +    dry_run=false
> +
> +    host_user=""
> +    target_user=""
> +
> +    while [ $# -gt 0 ]; do
> +	case "$1" in
> +	    --debug)
> +		debug=true
> +		;;
> +	    --keep)
> +		keep=true
> +		;;
> +	    --verbose)
> +		verbose=true
> +		;;
> +	    --dry-run)
> +		dry_run=true
> +		;;
> +	    --host-user)
> +		shift
> +		host_user="$1"
> +		;;
> +	    --target-user)
> +		shift
> +		target_user="$1"
> +		;;
> +	    *)
> +		break
> +		;;
> +	esac
> +	shift
> +    done
> +
> +    tests=("$@")
> +}
> +
> +main ()
> +{
> +    local b
> +    local h
> +
> +    parse_args "$@"
> +
> +    tmpdir=$(mktemp -d)
> +
> +    if $debug; then
> +	echo "TESTS: ${tests[*]}"
> +    fi
> +
> +    if true; then
> +	echo "LOCAL:"
> +	rtf=()
> +	do_tests
> +    fi
> +
> +    if true; then
> +	for b in "${target_boards[@]}"; do
> +	    echo "TARGET BOARD: $b"
> +	    rtf=(
> +		--target_board="$b"
> +	    )
> +	    rtf_for_board "$b"
> +	    do_tests
> +	done
> +    fi
> +
> +    if true; then
> +	for b in "${gdbserver_boards[@]}" "${remote_gdbserver_boards[@]}"; do
> +	    echo "TARGET BOARD: $b"
> +	    rtf=(
> +		--target_board="$b"
> +	    )
> +	    rtf_for_board "$b"
> +	    do_tests
> +	done
> +    fi
> +
> +    if true; then
> +	for h in "${host_boards[@]}"; do
> +	    for b in "${remote_gdbserver_boards[@]}"; do
> +		echo "HOST BOARD: $h, TARGET BOARD: $b"
> +		rtf=(
> +		    --host_board="$h"
> +		    --target_board="$b"
> +		)
> +		rtf_for_board "$h"
> +		rtf_for_board "$b"
> +		do_tests
> +	    done
> +	done
> +    fi
> +
> +    if true; then
> +	for b in "${host_target_boards[@]}"; do
> +	    echo "HOST/TARGET BOARD: $b"
> +	    rtf=(
> +		--host_board="$b"
> +		--target_board="$b"
> +	    )
> +	    rtf_for_board "$b"
> +	    do_tests
> +	done
> +    fi
> +
> +    if $keep; then
> +	echo "keeping tmp dir $tmpdir"
> +    else
> +	rm -Rf "$tmpdir"
> +    fi
> +}
> +
> +main "$@"
> -- 
> 2.35.3

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Tom de Vries]

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

On 4/4/23 13:35, Andrew Burgess wrote:
> Tom de Vries via Gdb-patches <gdb-patches@sourceware.org> writes:
> 
>> Add script gdb/contrib/make-check-all.sh, that's intended to function as a
>> drop-in replacement of make check, but excercising all host/target boards in
>> gdb/testsuite/boards.
>>
>> Shell-checked and tested on x86_64-linux.
> 
> Hi Tom,
> 
> Thanks for putting this together, I think this could be really useful.
> 

That's great to hear :)

> I'm not a fan of the way you've split the patch description into email
> 0/1 and not included it with this commit.  I think there's lots of
> useful information in there, and I'd much rather have the whole
> description included in the commit message -- it's much easier to find
> then rather than having to hunt on the mailing list in the future.
> 

Ack, I've reworked this into a patch rather than patch series.

>> ---
>>   gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 255 insertions(+)
>>   create mode 100755 gdb/contrib/make-check-all.sh
>>
>> diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
>> new file mode 100755
>> index 00000000000..1befe418b81
>> --- /dev/null
>> +++ b/gdb/contrib/make-check-all.sh
>> @@ -0,0 +1,255 @@
>> +#!/bin/bash
>> +
>> +# Copyright (C) 2023 Free Software Foundation, Inc.
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 3 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> +
>> +# Run make check with all boards from gdb/testsuite/boards.
> 
> I think you need some kind of usage test here too.  Imagine a new
> developer finds this file and looks inside.  Ideally, I think they
> should know how to use it, and what it will do for them without having
> to read any of the actual script content.
> 

Added.

> On implementation, I wonder if it would be useful to provide a mechanism
> by which the gdb.sum and gdb.log files for each test run could be
> preserved?
> 

Done, I've renamed --keep to --keep-tmp, and added this new 
functionality under --keep.

I've also added "set -e" to make sure ^C aborts the script rather than 
just one make check invocation, and moved the tmpdir removal to a 
cleanup function that's executed on exit, to make sure ^C doesn't leave 
tmpdirs behind.

Thanks,
- Tom


[-- Attachment #2: 0001-gdb-contrib-Add-make-check-all.sh.patch --]
[-- Type: text/x-patch, Size: 7218 bytes --]

From 8647ef0c3dd076ee18d1060882552a20a016b2d2 Mon Sep 17 00:00:00 2001
From: Tom de Vries <tdevries@suse.de>
Date: Mon, 3 Apr 2023 03:47:46 +0200
Subject: [PATCH] [gdb/contrib] Add make-check-all.sh

Directory gdb/testsuite/boards contains a number of host/target boards, which
run a test-case (or test-cases) in a different way.

The benefits of using these boards are:
- improving test coverage of gdb,
- making the testsuite more robust, and
- making sure the test-cases work for non-native and remote setups, if
  possible.

Each board is slightly different, and developers need to learn how to use each
one, what parameters to pass and how, and which ones can be using in
combination with each other.  This is a threshold to start using them.

And then there quite a few, so I suppose typically only a few will be used by
each developer.

Add script gdb/contrib/make-check-all.sh, that's intended to function as a
drop-in replacement of make check, while excercising all host/target boards in
gdb/testsuite/boards.

An an example of make-check-all.sh for one test-case is:
...
 $  ~/gdb/src/gdb/contrib/make-check-all.sh gdb.base/advance.exp
 LOCAL:
 # of expected passes            8
 TARGET BOARD: cc-with-gdb-index
 # of expected passes            8
   ...
 HOST BOARD: local-remote-host-notty, TARGET BOARD: remote-stdio-gdbserver
 # of expected passes            8
 HOST/TARGET BOARD: local-remote-host-native
 # of expected passes            8
...

Shell-checked and tested on x86_64-linux.
---
 gdb/contrib/make-check-all.sh | 299 ++++++++++++++++++++++++++++++++++
 1 file changed, 299 insertions(+)
 create mode 100755 gdb/contrib/make-check-all.sh

diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
new file mode 100755
index 00000000000..0d9f983302f
--- /dev/null
+++ b/gdb/contrib/make-check-all.sh
@@ -0,0 +1,299 @@
+#!/bin/bash
+
+# Copyright (C) 2023 Free Software Foundation, Inc.
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Run make check with all boards from gdb/testsuite/boards.
+
+# Note that running with the --host-user and --target-user options is
+# recommended, because:
+# - remote host/target boards will use $HOME and leave (potentially lots of)
+#   files behind, and
+# - it enabled more strict checking of build/host/target file manipulations.
+#
+# Recommended usage example:
+#
+# bash$ cd $objdir/gdb/testsuite
+# bash$ $srcdir/contrib/gdb/make-check-all.sh --host-user remote-host \
+#           --target-user remote-target gdb.base/advance.exp
+
+set -e
+
+host_boards=(
+    local-remote-host
+    local-remote-host-notty
+)
+
+gdbserver_boards=(
+    native-extended-gdbserver
+    native-gdbserver
+    native-stdio-gdbserver
+)
+
+remote_gdbserver_boards=(
+    remote-gdbserver-on-localhost
+    remote-stdio-gdbserver
+)
+
+host_target_boards=(
+    local-remote-host-native
+)
+
+target_boards=(
+    cc-with-gdb-index
+    cc-with-debug-names
+    cc-with-dwz
+    cc-with-dwz-m
+    cc-with-gnu-debuglink
+    debug-types
+    dwarf4-gdb-index
+    dwarf64
+    fission
+    fission-dwp
+    gold
+    gold-gdb-index
+    readnow
+    stabs
+)
+
+rtf_for_board ()
+{
+    local b
+    b="$1"
+
+    case $b in
+	local-remote-host-native)
+	    mkdir -p "$tmpdir/$b"
+	    rtf=(
+		"${rtf[@]}"
+		"HOST_DIR=$tmpdir/$b"
+	    )
+	    ;;
+	remote-stdio-gdbserver)
+	    rtf=(
+		"${rtf[@]}"
+		"REMOTE_HOSTNAME=localhost"
+	    )
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$target_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	remote-gdbserver-on-localhost)
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_TARGET_USERNAME=$target_user"
+		)
+	    fi
+	    ;;
+	local-remote-host|local-remote-host-notty)
+	    if [ "$host_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$host_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	*)
+	    ;;
+    esac
+}
+
+summary ()
+{
+    if $verbose; then
+	cat
+    else
+	grep -E "^(#|FAIL:|ERROR:|WARNING:)" \
+	    | sort -u
+    fi
+}
+
+do_tests ()
+{
+    if $debug; then
+	echo "RTF: ${rtf[*]}"
+    fi
+
+    if $dry_run; then
+	return
+    fi
+
+    make check \
+	 RUNTESTFLAGS="${rtf[*]} ${tests[*]}" \
+	 2>&1 \
+	| summary
+
+    if [ "$h" = "" ]; then
+	if [ "$b" = "" ]; then
+	    cfg=local
+	else
+	    cfg=$b
+	fi
+    else
+	cfg=$h-$b
+    fi
+
+    local dir
+    dir="check-all/$cfg"
+
+    if $keep; then
+	mkdir -p "$dir"
+	cp gdb.sum gdb.log "$dir"
+    fi
+}
+
+parse_args ()
+{
+    debug=false
+    keep=false
+    keep_tmp=false
+    verbose=false
+    dry_run=false
+
+    host_user=""
+    target_user=""
+
+    while [ $# -gt 0 ]; do
+	case "$1" in
+	    --debug)
+		debug=true
+		;;
+	    --keep)
+		keep=true
+		;;
+	    --keep-tmp)
+		keep_tmp=true
+		;;
+	    --verbose)
+		verbose=true
+		;;
+	    --dry-run)
+		dry_run=true
+		;;
+	    --host-user)
+		shift
+		host_user="$1"
+		;;
+	    --target-user)
+		shift
+		target_user="$1"
+		;;
+	    *)
+		break
+		;;
+	esac
+	shift
+    done
+
+    tests=("$@")
+}
+
+cleanup ()
+{
+    if [ "$tmpdir" != "" ]; then
+        if $keep_tmp; then
+	    echo "keeping tmp dir $tmpdir"
+	else	
+	    rm -Rf "$tmpdir"
+	fi
+    fi
+}
+
+main ()
+{
+    parse_args "$@"
+
+    tmpdir=""
+    trap cleanup EXIT
+    tmpdir=$(mktemp -d)
+
+    if $debug; then
+	echo "TESTS: ${tests[*]}"
+    fi
+
+    h=""
+    b=""
+
+    if true; then
+	echo "LOCAL:"
+	rtf=()
+	do_tests
+    fi
+
+    if true; then
+	for b in "${target_boards[@]}"; do
+	    echo "TARGET BOARD: $b"
+	    rtf=(
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if true; then
+	for b in "${gdbserver_boards[@]}" "${remote_gdbserver_boards[@]}"; do
+	    echo "TARGET BOARD: $b"
+	    rtf=(
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if true; then
+	for h in "${host_boards[@]}"; do
+	    for b in "${remote_gdbserver_boards[@]}"; do
+		echo "HOST BOARD: $h, TARGET BOARD: $b"
+		rtf=(
+		    --host_board="$h"
+		    --target_board="$b"
+		)
+		rtf_for_board "$h"
+		rtf_for_board "$b"
+		do_tests
+	    done
+	done
+	h=""
+    fi
+
+    if true; then
+	for b in "${host_target_boards[@]}"; do
+	    echo "HOST/TARGET BOARD: $b"
+	    rtf=(
+		--host_board="$b"
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+}
+
+main "$@"

base-commit: 9278aa6171edd477326f1a73a112d98b8b785954
-- 
2.35.3

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Tom de Vries]

On 4/5/23 11:01, Tom de Vries via Gdb-patches wrote:
> On 4/4/23 13:35, Andrew Burgess wrote:
>> Tom de Vries via Gdb-patches <gdb-patches@sourceware.org> writes:
>>
>>> Add script gdb/contrib/make-check-all.sh, that's intended to function 
>>> as a
>>> drop-in replacement of make check, but excercising all host/target 
>>> boards in
>>> gdb/testsuite/boards.
>>>
>>> Shell-checked and tested on x86_64-linux.
>>
>> Hi Tom,
>>
>> Thanks for putting this together, I think this could be really useful.
>>
> 
> That's great to hear :)
> 
>> I'm not a fan of the way you've split the patch description into email
>> 0/1 and not included it with this commit.  I think there's lots of
>> useful information in there, and I'd much rather have the whole
>> description included in the commit message -- it's much easier to find
>> then rather than having to hunt on the mailing list in the future.
>>
> 
> Ack, I've reworked this into a patch rather than patch series.
> 
>>> ---
>>>   gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
>>>   1 file changed, 255 insertions(+)
>>>   create mode 100755 gdb/contrib/make-check-all.sh
>>>
>>> diff --git a/gdb/contrib/make-check-all.sh 
>>> b/gdb/contrib/make-check-all.sh
>>> new file mode 100755
>>> index 00000000000..1befe418b81
>>> --- /dev/null
>>> +++ b/gdb/contrib/make-check-all.sh
>>> @@ -0,0 +1,255 @@
>>> +#!/bin/bash
>>> +
>>> +# Copyright (C) 2023 Free Software Foundation, Inc.
>>> +# This program is free software; you can redistribute it and/or modify
>>> +# it under the terms of the GNU General Public License as published by
>>> +# the Free Software Foundation; either version 3 of the License, or
>>> +# (at your option) any later version.
>>> +#
>>> +# This program is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> +
>>> +# Run make check with all boards from gdb/testsuite/boards.
>>
>> I think you need some kind of usage test here too.  Imagine a new
>> developer finds this file and looks inside.  Ideally, I think they
>> should know how to use it, and what it will do for them without having
>> to read any of the actual script content.
>>
> 
> Added.
> 
>> On implementation, I wonder if it would be useful to provide a mechanism
>> by which the gdb.sum and gdb.log files for each test run could be
>> preserved?
>>
> 
> Done, I've renamed --keep to --keep-tmp, and added this new 
> functionality under --keep.
> 
> I've also added "set -e" to make sure ^C aborts the script rather than 
> just one make check invocation, and moved the tmpdir removal to a 
> cleanup function that's executed on exit, to make sure ^C doesn't leave 
> tmpdirs behind.
> 

I'd like to commit this at the end of the week, if there are no further 
review comments.

Simon, I remember discussing something similar with you a few years ago 
( https://sourceware.org/pipermail/gdb-patches/2020-May/169003.html ), 
so I wondered if you have any comments on this approach.

Thanks,
- Tom

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Simon Marchi]

On 4/18/23 08:43, Tom de Vries wrote:
> On 4/5/23 11:01, Tom de Vries via Gdb-patches wrote:
>> On 4/4/23 13:35, Andrew Burgess wrote:
>>> Tom de Vries via Gdb-patches <gdb-patches@sourceware.org> writes:
>>>
>>>> Add script gdb/contrib/make-check-all.sh, that's intended to function as a
>>>> drop-in replacement of make check, but excercising all host/target boards in
>>>> gdb/testsuite/boards.
>>>>
>>>> Shell-checked and tested on x86_64-linux.
>>>
>>> Hi Tom,
>>>
>>> Thanks for putting this together, I think this could be really useful.
>>>
>>
>> That's great to hear :)
>>
>>> I'm not a fan of the way you've split the patch description into email
>>> 0/1 and not included it with this commit.  I think there's lots of
>>> useful information in there, and I'd much rather have the whole
>>> description included in the commit message -- it's much easier to find
>>> then rather than having to hunt on the mailing list in the future.
>>>
>>
>> Ack, I've reworked this into a patch rather than patch series.
>>
>>>> ---
>>>>   gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
>>>>   1 file changed, 255 insertions(+)
>>>>   create mode 100755 gdb/contrib/make-check-all.sh
>>>>
>>>> diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
>>>> new file mode 100755
>>>> index 00000000000..1befe418b81
>>>> --- /dev/null
>>>> +++ b/gdb/contrib/make-check-all.sh
>>>> @@ -0,0 +1,255 @@
>>>> +#!/bin/bash
>>>> +
>>>> +# Copyright (C) 2023 Free Software Foundation, Inc.
>>>> +# This program is free software; you can redistribute it and/or modify
>>>> +# it under the terms of the GNU General Public License as published by
>>>> +# the Free Software Foundation; either version 3 of the License, or
>>>> +# (at your option) any later version.
>>>> +#
>>>> +# This program is distributed in the hope that it will be useful,
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>> +# GNU General Public License for more details.
>>>> +#
>>>> +# You should have received a copy of the GNU General Public License
>>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>>> +
>>>> +# Run make check with all boards from gdb/testsuite/boards.
>>>
>>> I think you need some kind of usage test here too.  Imagine a new
>>> developer finds this file and looks inside.  Ideally, I think they
>>> should know how to use it, and what it will do for them without having
>>> to read any of the actual script content.
>>>
>>
>> Added.
>>
>>> On implementation, I wonder if it would be useful to provide a mechanism
>>> by which the gdb.sum and gdb.log files for each test run could be
>>> preserved?
>>>
>>
>> Done, I've renamed --keep to --keep-tmp, and added this new functionality under --keep.

Maybe call it "keep-results"?  Otherwise it's not clear what --keep keeps.

>>
>> I've also added "set -e" to make sure ^C aborts the script rather than just one make check invocation, and moved the tmpdir removal to a cleanup function that's executed on exit, to make sure ^C doesn't leave tmpdirs behind.
>>
> 
> I'd like to commit this at the end of the week, if there are no further review comments.
> 
> Simon, I remember discussing something similar with you a few years ago ( https://sourceware.org/pipermail/gdb-patches/2020-May/169003.html ), so I wondered if you have any comments on this approach.

I don't recall this discussion at all :).  But I like the idea of your
script, it lets people run tests on all target boards without having to
specifically know about each board.  I never really test on boards other
than unix, native-gdbserver and native-extended-gdbserver, but I might
now with your script (probably not whole testsuite runs, but at least
when working on a test).

When applying:

    Applying: Add make-check-all.sh
    .git/rebase-apply/patch:232: trailing whitespace.
            else
    warning: 1 line adds whitespace errors.

Would it make more sense to have this file under gdb/testsuite?

I don't understand the comment about using --host-user and
--target-user.  Are you suggesting that I create new users on my machine
(e.g. gdb-test-host and gdb-test-target), so that running with the
remote host and remote target boards pollute those users' home
directories instead of mine?

I also don't understand those `if true` conditions in main.

Finally, I'd appreciate a few comments here and there in the script, to
make further modifications easier.  But other than that, it looks good
to me, it can't hurt to have that in the tree.

Simon

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Tom de Vries]

[-- Attachment #1: Type: text/plain, Size: 7187 bytes --]

On 4/19/23 16:04, Simon Marchi wrote:
> On 4/18/23 08:43, Tom de Vries wrote:
>> On 4/5/23 11:01, Tom de Vries via Gdb-patches wrote:
>>> On 4/4/23 13:35, Andrew Burgess wrote:
>>>> Tom de Vries via Gdb-patches <gdb-patches@sourceware.org> writes:
>>>>
>>>>> Add script gdb/contrib/make-check-all.sh, that's intended to function as a
>>>>> drop-in replacement of make check, but excercising all host/target boards in
>>>>> gdb/testsuite/boards.
>>>>>
>>>>> Shell-checked and tested on x86_64-linux.
>>>>
>>>> Hi Tom,
>>>>
>>>> Thanks for putting this together, I think this could be really useful.
>>>>
>>>
>>> That's great to hear :)
>>>
>>>> I'm not a fan of the way you've split the patch description into email
>>>> 0/1 and not included it with this commit.  I think there's lots of
>>>> useful information in there, and I'd much rather have the whole
>>>> description included in the commit message -- it's much easier to find
>>>> then rather than having to hunt on the mailing list in the future.
>>>>
>>>
>>> Ack, I've reworked this into a patch rather than patch series.
>>>
>>>>> ---
>>>>>    gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
>>>>>    1 file changed, 255 insertions(+)
>>>>>    create mode 100755 gdb/contrib/make-check-all.sh
>>>>>
>>>>> diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
>>>>> new file mode 100755
>>>>> index 00000000000..1befe418b81
>>>>> --- /dev/null
>>>>> +++ b/gdb/contrib/make-check-all.sh
>>>>> @@ -0,0 +1,255 @@
>>>>> +#!/bin/bash
>>>>> +
>>>>> +# Copyright (C) 2023 Free Software Foundation, Inc.
>>>>> +# This program is free software; you can redistribute it and/or modify
>>>>> +# it under the terms of the GNU General Public License as published by
>>>>> +# the Free Software Foundation; either version 3 of the License, or
>>>>> +# (at your option) any later version.
>>>>> +#
>>>>> +# This program is distributed in the hope that it will be useful,
>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>>> +# GNU General Public License for more details.
>>>>> +#
>>>>> +# You should have received a copy of the GNU General Public License
>>>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>>>> +
>>>>> +# Run make check with all boards from gdb/testsuite/boards.
>>>>
>>>> I think you need some kind of usage test here too.  Imagine a new
>>>> developer finds this file and looks inside.  Ideally, I think they
>>>> should know how to use it, and what it will do for them without having
>>>> to read any of the actual script content.
>>>>
>>>
>>> Added.
>>>
>>>> On implementation, I wonder if it would be useful to provide a mechanism
>>>> by which the gdb.sum and gdb.log files for each test run could be
>>>> preserved?
>>>>
>>>
>>> Done, I've renamed --keep to --keep-tmp, and added this new functionality under --keep.
> 
> Maybe call it "keep-results"?  Otherwise it's not clear what --keep keeps.
> 

Done.

>>>
>>> I've also added "set -e" to make sure ^C aborts the script rather than just one make check invocation, and moved the tmpdir removal to a cleanup function that's executed on exit, to make sure ^C doesn't leave tmpdirs behind.
>>>
>>
>> I'd like to commit this at the end of the week, if there are no further review comments.
>>
>> Simon, I remember discussing something similar with you a few years ago ( https://sourceware.org/pipermail/gdb-patches/2020-May/169003.html ), so I wondered if you have any comments on this approach.
> 
> I don't recall this discussion at all :).  But I like the idea of your
> script, it lets people run tests on all target boards without having to
> specifically know about each board.  I never really test on boards other
> than unix, native-gdbserver and native-extended-gdbserver, but I might
> now with your script (probably not whole testsuite runs, but at least
> when working on a test).
> 
> When applying:
> 
>      Applying: Add make-check-all.sh
>      .git/rebase-apply/patch:232: trailing whitespace.
>              else
>      warning: 1 line adds whitespace errors.

Ack, fixed.

TIL: I can reproduce this using git show --check.

> Would it make more sense to have this file under gdb/testsuite?

Fine by me.

[ FWIW, I usually look for scripts in a contrib dir, so my idea for the 
most logical place for it would be gdb/testsuite/contrib.

If we'd add such a dir, we'd probably move the gdb/testsuite/*.py 
scripts there.  Also, some from gdb/contrib, but I suppose it would 
leave gdb/contrib rather empty, that is only words.sh, ari/ and 
gdb-add-index.sh left AFAICT. ]

Moved to gdb/testsuite.

> I don't understand the comment about using --host-user and
> --target-user.  Are you suggesting that I create new users on my machine
> (e.g. gdb-test-host and gdb-test-target), so that running with the
> remote host and remote target boards pollute those users' home
> directories instead of mine?

To demonstrate the problem of polluting the home dir, after running 
gdb.base/advance.exp without --host-user and --target-user, I have these 
additional files in the home dir:
...
advance0.o
advance.c
ccopts.c
compiler.c
...
And if I just use --target-user but not the --host-user one, we get the 
advance executable in addition.

Yes, I do suggest creating new users.  There are several reasons:
- it prevents polluting your home dir, as some boards do, which on the
   flip side makes it easier to clean up if something was left, and make
   it easier to identify what was left.  It also makes it easier to clean
   up inbetween boards, to prevent files copied by one board being used
   by the next board.
- it prevents clashes in your home dir (I had a dir ~/dwz, which clashes
   with the exec from the gdb.dwarf2/dwz.exp test-case)
- it's a good simulation of an actual remote host/target setup:
   - files from build can be made inaccessible from target/host
     and vice versa
   - pids from one user cannot be killed by other users, so errors like
     getting a pid on build and killing it on target/host will show up

So, running the script with --host-user and --target-user gives you 
better checking, but for unsuspecting first-time users, the polluting 
home dir part will be the biggest thing I imagine, hence the comment.

You say you don't understand the comment, but to me it seems you do so 
I'm not sure I understand what your concern is.  Should the comment be 
improved, or do you want a change in functionality?

FWIW, I've also written a script to add such test users, which I might 
submit after this gets committed, with the idea to both facilitate 
creation and standardize the result.

> I also don't understand those `if true` conditions in main.

Yeah, it was in preparation for an eventual ./make-check-all.sh 
--remote-target etc, but currently there's no need.  Removed.

> Finally, I'd appreciate a few comments here and there in the script, to
> make further modifications easier.  But other than that, it looks good
> to me, it can't hurt to have that in the tree.

I've tried adding a bit more comments, updated version attached.

Thanks,
- Tom

[-- Attachment #2: 0001-gdb-testsuite-Add-make-check-all.sh.patch --]
[-- Type: text/x-patch, Size: 8830 bytes --]

From 6f1e26aa560fef1cc1b97939fee83feabbf0d3aa Mon Sep 17 00:00:00 2001
From: Tom de Vries <tdevries@suse.de>
Date: Mon, 3 Apr 2023 03:47:46 +0200
Subject: [PATCH] [gdb/testsuite] Add make-check-all.sh

Directory gdb/testsuite/boards contains a number of host/target boards, which
run a test-case (or test-cases) in a different way.

The benefits of using these boards are:
- improving test coverage of gdb,
- making the testsuite more robust, and
- making sure the test-cases work for non-native and remote setups, if
  possible.

Each board is slightly different, and developers need to learn how to use each
one, what parameters to pass and how, and which ones can be used in
combination with each other.  This is a threshold to start using them.

And then there quite a few, so I suppose typically only a few will be used by
each developer.

Add script gdb/testsuite/make-check-all.sh, that's intended to function as a
drop-in replacement of make check, while excercising all host/target boards in
gdb/testsuite/boards.

An example of make-check-all.sh for one test-case is:
...
 $  ~/gdb/src/gdb/testsuite/make-check-all.sh gdb.base/advance.exp
 LOCAL:
 # of expected passes            8
 TARGET BOARD: cc-with-gdb-index
 # of expected passes            8
   ...
 HOST BOARD: local-remote-host-notty, TARGET BOARD: remote-stdio-gdbserver
 # of expected passes            8
 HOST/TARGET BOARD: local-remote-host-native
 # of expected passes            8
...

Shell-checked and tested on x86_64-linux.
---
 gdb/testsuite/make-check-all.sh | 323 ++++++++++++++++++++++++++++++++
 1 file changed, 323 insertions(+)
 create mode 100755 gdb/testsuite/make-check-all.sh

diff --git a/gdb/testsuite/make-check-all.sh b/gdb/testsuite/make-check-all.sh
new file mode 100755
index 00000000000..d54f9aaae2c
--- /dev/null
+++ b/gdb/testsuite/make-check-all.sh
@@ -0,0 +1,323 @@
+#!/bin/bash
+
+# Copyright (C) 2023 Free Software Foundation, Inc.
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Run make check with all boards from gdb/testsuite/boards.
+
+# Note that running with the --host-user and --target-user options is
+# recommended, because:
+# - remote host/target boards will use $HOME and leave (potentially lots of)
+#   files behind, and
+# - it enables more strict checking of build/host/target file manipulations.
+#
+# Recommended usage example:
+#
+# bash$ cd $objdir/gdb/testsuite
+# bash$ $srcdir/testsuite/gdb/make-check-all.sh \
+#           --host-user remote-host \
+#           --target-user remote-target \
+#           gdb.base/advance.exp
+
+set -e
+
+# Boards that run the host tools (compiler, gdb) on a remote host.
+remote_host_boards=(
+    local-remote-host
+    local-remote-host-notty
+)
+
+# Boards that use gdbserver to launch target executables on local target.
+gdbserver_boards=(
+    native-extended-gdbserver
+    native-gdbserver
+    native-stdio-gdbserver
+)
+
+# Boards that use gdbserver to launch target executables on a remote target.
+remote_gdbserver_boards=(
+    remote-gdbserver-on-localhost
+    remote-stdio-gdbserver
+)
+
+# Boards that run compiler, gdb and target executables on a remote machine
+# that serves both as host and target.
+host_target_boards=(
+    local-remote-host-native
+)
+
+# Boards that run everything on local target and local host.
+target_boards=(
+    cc-with-gdb-index
+    cc-with-debug-names
+    cc-with-dwz
+    cc-with-dwz-m
+    cc-with-gnu-debuglink
+    debug-types
+    dwarf4-gdb-index
+    dwarf64
+    fission
+    fission-dwp
+    gold
+    gold-gdb-index
+    readnow
+    stabs
+)
+
+# Get RUNTESTFLAGS needed for specific boards.
+rtf_for_board ()
+{
+    local b
+    b="$1"
+
+    case $b in
+	local-remote-host-native)
+	    mkdir -p "$tmpdir/$b"
+	    rtf=(
+		"${rtf[@]}"
+		"HOST_DIR=$tmpdir/$b"
+	    )
+	    ;;
+	remote-stdio-gdbserver)
+	    rtf=(
+		"${rtf[@]}"
+		"REMOTE_HOSTNAME=localhost"
+	    )
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$target_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	remote-gdbserver-on-localhost)
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_TARGET_USERNAME=$target_user"
+		)
+	    fi
+	    ;;
+	local-remote-host|local-remote-host-notty)
+	    if [ "$host_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$host_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	*)
+	    ;;
+    esac
+}
+
+# Summarize make check output.
+summary ()
+{
+    if $verbose; then
+	cat
+    else
+	# We need the sort -u, because some items, for instance "# of expected
+	# passes" are output twice.
+	grep -E "^(#|FAIL:|ERROR:|WARNING:)" \
+	    | sort -u
+    fi
+}
+
+# Run make check, and possibly save test results.
+do_tests ()
+{
+    if $debug; then
+	echo "RTF: ${rtf[*]}"
+    fi
+
+    if $dry_run; then
+	return
+    fi
+
+    # Run make check.
+    make check \
+	 RUNTESTFLAGS="${rtf[*]} ${tests[*]}" \
+	 2>&1 \
+	| summary
+
+    # Save test results.
+    if $keep_results; then
+	# Set cfg to identifier unique to host/target board combination.
+	if [ "$h" = "" ]; then
+	    if [ "$b" = "" ]; then
+		cfg=local
+	    else
+		cfg=$b
+	    fi
+	else
+	    cfg=$h-$b
+	fi
+
+	local dir
+	dir="check-all/$cfg"
+
+	mkdir -p "$dir"
+	cp gdb.sum gdb.log "$dir"
+    fi
+}
+
+# Set default values for global vars and modify according to command line
+# arguments.
+parse_args ()
+{
+    # Default values.
+    debug=false
+    keep_results=false
+    keep_tmp=false
+    verbose=false
+    dry_run=false
+
+    host_user=""
+    target_user=""
+
+    # Parse command line arguments.
+    while [ $# -gt 0 ]; do
+	case "$1" in
+	    --debug)
+		debug=true
+		;;
+	    --keep-results)
+		keep_results=true
+		;;
+	    --keep-tmp)
+		keep_tmp=true
+		;;
+	    --verbose)
+		verbose=true
+		;;
+	    --dry-run)
+		dry_run=true
+		;;
+	    --host-user)
+		shift
+		host_user="$1"
+		;;
+	    --target-user)
+		shift
+		target_user="$1"
+		;;
+	    *)
+		break
+		;;
+	esac
+	shift
+    done
+
+    tests=("$@")
+}
+
+# Cleanup function, scheduled to run on exit.
+cleanup ()
+{
+    if [ "$tmpdir" != "" ]; then
+	if $keep_tmp; then
+	    echo "keeping tmp dir $tmpdir"
+	else
+	    rm -Rf "$tmpdir"
+	fi
+    fi
+}
+
+# Top-level function, called with command line arguments of the script.
+main ()
+{
+    # Parse command line arguments.
+    parse_args "$@"
+
+    # Create tmpdir and schedule cleanup.
+    tmpdir=""
+    trap cleanup EXIT
+    tmpdir=$(mktemp -d)
+
+    if $debug; then
+	echo "TESTS: ${tests[*]}"
+    fi
+
+    # Variables that point to current host (h) and target board (b) when
+    # executing do_tests.
+    h=""
+    b=""
+
+    # For reference, run the tests without any explicit host or target board.
+    echo "LOCAL:"
+    rtf=()
+    do_tests
+
+    # Run the boards for local host and local target.
+    for b in "${target_boards[@]}"; do
+	echo "TARGET BOARD: $b"
+	rtf=(
+	    --target_board="$b"
+	)
+	rtf_for_board "$b"
+	do_tests
+    done
+
+    # Run the boards that use gdbserver, for local host, and for both local and
+    # remote target.
+    for b in "${gdbserver_boards[@]}" "${remote_gdbserver_boards[@]}"; do
+	echo "TARGET BOARD: $b"
+	rtf=(
+	    --target_board="$b"
+	)
+	rtf_for_board "$b"
+	do_tests
+    done
+
+    # Run the boards that use remote host, in combination with boards that use
+    # gdbserver on remote target.
+    for h in "${remote_host_boards[@]}"; do
+	for b in "${remote_gdbserver_boards[@]}"; do
+	    echo "HOST BOARD: $h, TARGET BOARD: $b"
+	    rtf=(
+		--host_board="$h"
+		--target_board="$b"
+	    )
+	    rtf_for_board "$h"
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    done
+    h=""
+
+    # Run the boards that function as both remote host and remote target.
+    for b in "${host_target_boards[@]}"; do
+	echo "HOST/TARGET BOARD: $b"
+	rtf=(
+	    --host_board="$b"
+	    --target_board="$b"
+	)
+	rtf_for_board "$b"
+	do_tests
+    done
+}
+
+# Call top-level function with command line arguments.
+main "$@"

base-commit: 99eca30b2c14898b2596bd5d84ec2228f6a38ad2
-- 
2.35.3

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Simon Marchi]

On 4/20/23 07:43, Tom de Vries wrote:
> You say you don't understand the comment, but to me it seems you do so I'm not sure I understand what your concern is.  Should the comment be improved, or do you want a change in functionality?

I just wasn't sure which username I was expected to pass to those
options, I wanted to confirm with you that the intent is to have
dedicated systems users for that.  So the comment could say that:

  It is recommended to create users on the local system that will act as
  "remote host" and "remote target", for the boards that use them.
  Pass their usernames to --host-user and --target-user.  This helps
  because:

    - remote host/target boards will use $HOME and leave (potentially
      lots of) files behind
    - it enables more strict checking of build/host/target file
      manipulations.
    - it prevents a command running on one "machine" to kill or send a
      signal to a process on another machine

> I've tried adding a bit more comments, updated version attached.

Thanks, that LGTM.

Simon

Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Tom de Vries]

On 4/20/23 16:30, Simon Marchi wrote:
> On 4/20/23 07:43, Tom de Vries wrote:
>> You say you don't understand the comment, but to me it seems you do so I'm not sure I understand what your concern is.  Should the comment be improved, or do you want a change in functionality?
> 
> I just wasn't sure which username I was expected to pass to those
> options, I wanted to confirm with you that the intent is to have
> dedicated systems users for that.  So the comment could say that:
> 
>    It is recommended to create users on the local system that will act as
>    "remote host" and "remote target", for the boards that use them.
>    Pass their usernames to --host-user and --target-user.  This helps
>    because:
> 
>      - remote host/target boards will use $HOME and leave (potentially
>        lots of) files behind
>      - it enables more strict checking of build/host/target file
>        manipulations.
>      - it prevents a command running on one "machine" to kill or send a
>        signal to a process on another machine
> 

Ack, I've copied that and added a Co-Authored-By tag.

>> I've tried adding a bit more comments, updated version attached.
> 
> Thanks, that LGTM.

Committed now.

Thanks,
- Tom