* [PATCH 0/3] Improve and document stdx::simd testsuite
@ 2021-06-08 8:57 Matthias Kretz
2021-06-08 8:58 ` [PATCH 1/3] libstdc++: Remove -fno-tree-vrp after PR98834 was resolved Matthias Kretz
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Matthias Kretz @ 2021-06-08 8:57 UTC (permalink / raw)
To: gcc-patches, libstdc++
As discussed a long time ago on IRC, this improves (i.e. decreases by default)
the verbosity of make check-simd, gives more verbosity options, and finally
documents how the simd testsuite is used and how it works. In addition, after
PR98834 was resolved, remove the -fno-tree-vrp workaround.
Tested on x86_64-linux (and more).
Matthias Kretz (3):
libstdc++: Remove -fno-tree-vrp after PR98834 was resolved
libstdc++: Improve output verbosity options and default
libstdc++: Document simd testsuite
libstdc++-v3/testsuite/Makefile.am | 3 +-
libstdc++-v3/testsuite/Makefile.in | 3 +-
.../testsuite/experimental/simd/README.md | 257 ++++++++++++++++++
.../testsuite/experimental/simd/driver.sh | 137 +++++++---
.../experimental/simd/generate_makefile.sh | 33 ++-
5 files changed, 380 insertions(+), 53 deletions(-)
create mode 100644 libstdc++-v3/testsuite/experimental/simd/README.md
--
──────────────────────────────────────────────────────────────────────────
Dr. Matthias Kretz https://mattkretz.github.io
GSI Helmholtz Centre for Heavy Ion Research https://gsi.de
std::experimental::simd https://github.com/VcDevel/std-simd
──────────────────────────────────────────────────────────────────────────
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/3] libstdc++: Remove -fno-tree-vrp after PR98834 was resolved
2021-06-08 8:57 [PATCH 0/3] Improve and document stdx::simd testsuite Matthias Kretz
@ 2021-06-08 8:58 ` Matthias Kretz
2021-06-08 8:59 ` [PATCH 2/3] libstdc++: Improve output verbosity options and default Matthias Kretz
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Matthias Kretz @ 2021-06-08 8:58 UTC (permalink / raw)
To: gcc-patches, libstdc++
[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]
libstdc++-v3/ChangeLog:
* testsuite/Makefile.am (check-simd): Remove -fno-tree-vrp flag
and associated warning.
* testsuite/Makefile.in: Regenerate.
Signed-off-by: Matthias Kretz <m.kretz@gsi.de>
---
libstdc++-v3/testsuite/Makefile.am | 3 +--
libstdc++-v3/testsuite/Makefile.in | 3 +--
2 files changed, 2 insertions(+), 4 deletions(-)
--
──────────────────────────────────────────────────────────────────────────
Dr. Matthias Kretz https://mattkretz.github.io
GSI Helmholtz Centre for Heavy Ion Research https://gsi.de
std::experimental::simd https://github.com/VcDevel/std-simd
──────────────────────────────────────────────────────────────────────────
[-- Attachment #2: 0001-libstdc-Remove-fno-tree-vrp-after-PR98834-was-resolv.patch --]
[-- Type: text/x-patch, Size: 1798 bytes --]
diff --git a/libstdc++-v3/testsuite/Makefile.am b/libstdc++-v3/testsuite/Makefile.am
index ba5023a8b54..d2011f03c64 100644
--- a/libstdc++-v3/testsuite/Makefile.am
+++ b/libstdc++-v3/testsuite/Makefile.am
@@ -191,10 +191,9 @@ check-simd: $(srcdir)/experimental/simd/generate_makefile.sh \
${glibcxx_srcdir}/scripts/check_simd \
testsuite_files_simd \
${glibcxx_builddir}/scripts/testsuite_flags
- @echo "WARNING: Adding -fno-tree-vrp to CXXFLAGS to work around PR98834."
@rm -f .simd.summary
@echo "Generating simd testsuite subdirs and Makefiles ..."
- @${glibcxx_srcdir}/scripts/check_simd "${glibcxx_srcdir}" "${glibcxx_builddir}" "$(CXXFLAGS) -fno-tree-vrp" | \
+ @${glibcxx_srcdir}/scripts/check_simd "${glibcxx_srcdir}" "${glibcxx_builddir}" "$(CXXFLAGS)" | \
while read subdir; do \
$(MAKE) -C "$${subdir}"; \
tail -n20 $${subdir}/simd_testsuite.sum | \
diff --git a/libstdc++-v3/testsuite/Makefile.in b/libstdc++-v3/testsuite/Makefile.in
index c9dd7f5da61..c65cdaf2015 100644
--- a/libstdc++-v3/testsuite/Makefile.in
+++ b/libstdc++-v3/testsuite/Makefile.in
@@ -716,10 +716,9 @@ check-simd: $(srcdir)/experimental/simd/generate_makefile.sh \
${glibcxx_srcdir}/scripts/check_simd \
testsuite_files_simd \
${glibcxx_builddir}/scripts/testsuite_flags
- @echo "WARNING: Adding -fno-tree-vrp to CXXFLAGS to work around PR98834."
@rm -f .simd.summary
@echo "Generating simd testsuite subdirs and Makefiles ..."
- @${glibcxx_srcdir}/scripts/check_simd "${glibcxx_srcdir}" "${glibcxx_builddir}" "$(CXXFLAGS) -fno-tree-vrp" | \
+ @${glibcxx_srcdir}/scripts/check_simd "${glibcxx_srcdir}" "${glibcxx_builddir}" "$(CXXFLAGS)" | \
while read subdir; do \
$(MAKE) -C "$${subdir}"; \
tail -n20 $${subdir}/simd_testsuite.sum | \
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/3] libstdc++: Improve output verbosity options and default
2021-06-08 8:57 [PATCH 0/3] Improve and document stdx::simd testsuite Matthias Kretz
2021-06-08 8:58 ` [PATCH 1/3] libstdc++: Remove -fno-tree-vrp after PR98834 was resolved Matthias Kretz
@ 2021-06-08 8:59 ` Matthias Kretz
2021-06-08 8:59 ` [PATCH 3/3] libstdc++: Document simd testsuite Matthias Kretz
2021-06-23 15:46 ` [PATCH 0/3] Improve and document stdx::simd testsuite Jonathan Wakely
3 siblings, 0 replies; 6+ messages in thread
From: Matthias Kretz @ 2021-06-08 8:59 UTC (permalink / raw)
To: gcc-patches, libstdc++
[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]
For most uses --quiet was too quiet while the default was too noisy. Now
the default output, if stdout is a tty, shows the last successful test
on the same line. With --percentage it adds a percentage at the start of
the line. --percentage is not default because it requires more resources
and might not be 100% compatible to all environments.
If stdout is not a tty the default is quiet output like for dejagnu.
Additionally, argument parsing now recognizes contracted short options
which is easier to use with e.g. DRIVEROPTS=-pxk.
libstdc++-v3/ChangeLog:
* testsuite/experimental/simd/driver.sh: Rewrite output
verbosity logic. Add -p/--percentage option. Allow -v/--verbose
to be used twice. Add -x and -o short options. Parse long
options with = instead of separating space generically. Parce
contracted short options. Make unrecognized options an error.
If same-line output is active, trap on EXIT to increment the
progress (only with --percentage), erase the line and print the
current status.
* testsuite/experimental/simd/generate_makefile.sh: Initialize
helper files for progress account keeping. Update help target
for changes to DRIVEROPTS.
Signed-off-by: Matthias Kretz <m.kretz@gsi.de>
---
.../testsuite/experimental/simd/driver.sh | 137 +++++++++++++-----
.../experimental/simd/generate_makefile.sh | 33 +++--
2 files changed, 121 insertions(+), 49 deletions(-)
--
──────────────────────────────────────────────────────────────────────────
Dr. Matthias Kretz https://mattkretz.github.io
GSI Helmholtz Centre for Heavy Ion Research https://gsi.de
std::experimental::simd https://github.com/VcDevel/std-simd
──────────────────────────────────────────────────────────────────────────
[-- Attachment #2: 0002-libstdc-Improve-output-verbosity-options-and-default.patch --]
[-- Type: text/x-patch, Size: 10318 bytes --]
diff --git a/libstdc++-v3/testsuite/experimental/simd/driver.sh b/libstdc++-v3/testsuite/experimental/simd/driver.sh
index f2d31c70bd0..5ae9905e3a3 100755
--- a/libstdc++-v3/testsuite/experimental/simd/driver.sh
+++ b/libstdc++-v3/testsuite/experimental/simd/driver.sh
@@ -5,8 +5,22 @@ abi=0
name=
srcdir="$(cd "${0%/*}" && pwd)/tests"
sim="$GCC_TEST_SIMULATOR"
-quiet=false
-verbose=false
+
+# output_mode values:
+# print only failures with minimal context
+readonly really_quiet=0
+# as above plus same-line output of last successful test
+readonly same_line=1
+# as above plus percentage
+readonly percentage=2
+# print one line per finished test with minimal context on failure
+readonly verbose=3
+# print one line per finished test with full output of the compiler and test
+readonly really_verbose=4
+
+output_mode=$really_quiet
+[ -t 1 ] && output_mode=$same_line
+
timeout=180
run_expensive=false
if [ -n "$GCC_TEST_RUN_EXPENSIVE" ]; then
@@ -21,8 +35,12 @@ Usage: $0 [Options] <g++ invocation>
Options:
-h, --help Print this message and exit.
- -q, --quiet Only print failures.
- -v, --verbose Print compiler and test output on failure.
+ -q, --quiet Disable same-line progress output (default if stdout is
+ not a tty).
+ -p, --percentage Add percentage to default same-line progress output.
+ -v, --verbose Print one line per test and minimal extra information on
+ failure.
+ -vv Print all compiler and test output.
-t <type>, --type <type>
The value_type to test (default: $type).
-a [0-9], --abi [0-9]
@@ -36,9 +54,10 @@ Options:
GCC_TEST_SIMULATOR).
--timeout-factor <x>
Multiply the default timeout with x.
- --run-expensive Compile and run tests marked as expensive (default:
+ -x, --run-expensive Compile and run tests marked as expensive (default:
true if GCC_TEST_RUN_EXPENSIVE is set, false otherwise).
- --only <pattern> Compile and run only tests matching the given pattern.
+ -o <pattern>, --only <pattern>
+ Compile and run only tests matching the given pattern.
EOF
}
@@ -49,71 +68,74 @@ while [ $# -gt 0 ]; do
exit
;;
-q|--quiet)
- quiet=true
+ output_mode=$really_quiet
+ ;;
+ -p|--percentage)
+ output_mode=$percentage
;;
-v|--verbose)
- verbose=true
+ if [ $output_mode -lt $verbose ]; then
+ output_mode=$verbose
+ else
+ output_mode=$really_verbose
+ fi
;;
- --run-expensive)
+ -x|--run-expensive)
run_expensive=true
;;
-k|--keep-failed)
keep_failed=true
;;
- --only)
+ -o|--only)
only="$2"
shift
;;
- --only=*)
- only="${1#--only=}"
- ;;
-t|--type)
type="$2"
shift
;;
- --type=*)
- type="${1#--type=}"
- ;;
-a|--abi)
abi="$2"
shift
;;
- --abi=*)
- abi="${1#--abi=}"
- ;;
-n|--name)
name="$2"
shift
;;
- --name=*)
- name="${1#--name=}"
- ;;
--srcdir)
srcdir="$2"
shift
;;
- --srcdir=*)
- srcdir="${1#--srcdir=}"
- ;;
--sim)
sim="$2"
shift
;;
- --sim=*)
- sim="${1#--sim=}"
- ;;
--timeout-factor)
timeout=$(awk "BEGIN { print int($timeout * $2) }")
shift
;;
- --timeout-factor=*)
- x=${1#--timeout-factor=}
- timeout=$(awk "BEGIN { print int($timeout * $x) }")
- ;;
--)
shift
break
;;
+ --*=*)
+ opt="$1"
+ shift
+ value=${opt#*=}
+ set -- ${opt%=$value} "$value" ${1+"$@"}
+ continue
+ ;;
+ -[ahknopqtvx][ahknopqtvx]*)
+ opt="$1"
+ shift
+ next=${opt#??}
+ set -- ${opt%$next} "-$next" ${1+"$@"}
+ continue
+ ;;
+ -*)
+ echo "Error: Unrecognized option '$1'" >&2
+ exit 1
+ ;;
*)
break
;;
@@ -121,6 +143,17 @@ while [ $# -gt 0 ]; do
shift
done
+if [ $output_mode = $percentage ]; then
+ inc_progress() {
+ {
+ flock -n 9
+ n=$(($(cat .progress) + 1))
+ echo $n >&9
+ echo $n
+ } 9<>.progress
+ }
+fi
+
CXX="$1"
shift
CXXFLAGS="$@"
@@ -133,6 +166,7 @@ sum="${testname}.sum"
if [ -n "$only" ]; then
if echo "$testname"|awk "{ exit /$only/ }"; then
touch "$log" "$sum"
+ [ $output_mode = $percentage ] && inc_progress >/dev/null
exit 0
fi
fi
@@ -146,35 +180,58 @@ else
exit 1
fi
+if [ $output_mode = $percentage ]; then
+ show_progress() {
+ n=$(inc_progress)
+ read total < .progress_total
+ total=${total}0
+ printf "\e[1G\e[K[%3d %%] ${src##*/} $type $abiflag" \
+ $((n * 1005 / total))
+ }
+ trap 'show_progress' EXIT
+ prefix="\e[1G\e[K"
+elif [ $output_mode = $same_line ]; then
+ show_progress() {
+ printf "\e[1G\e[K${src##*/} $type $abiflag"
+ }
+ trap 'show_progress' EXIT
+ prefix="\e[1G\e[K"
+else
+ prefix=""
+fi
+
fail() {
+ printf "$prefix"
echo "FAIL: $src $type $abiflag ($*)" | tee -a "$sum" "$log"
}
xpass() {
+ printf "$prefix"
echo "XPASS: $src $type $abiflag ($*)" | tee -a "$sum" "$log"
}
xfail() {
- $quiet || echo "XFAIL: $src $type $abiflag ($*)"
+ [ $output_mode -ge $verbose ] && echo "XFAIL: $src $type $abiflag ($*)"
echo "XFAIL: $src $type $abiflag ($*)" >> "$sum"
echo "XFAIL: $src $type $abiflag ($*)" >> "$log"
}
pass() {
- $quiet || echo "PASS: $src $type $abiflag ($*)"
+ [ $output_mode -ge $verbose ] && echo "PASS: $src $type $abiflag ($*)"
echo "PASS: $src $type $abiflag ($*)" >> "$sum"
echo "PASS: $src $type $abiflag ($*)" >> "$log"
}
unsupported() {
- $quiet || echo "UNSUPPORTED: $src $type $abiflag ($*)"
+ test
+ [ $output_mode -ge $verbose ] && echo "UNSUPPORTED: $src $type $abiflag ($*)"
echo "UNSUPPORTED: $src $type $abiflag ($*)" >> "$sum"
echo "UNSUPPORTED: $src $type $abiflag ($*)" >> "$log"
}
write_log_and_verbose() {
echo "$*" >> "$log"
- if $verbose; then
+ if [ $output_mode = $really_verbose ]; then
if [ -z "$COLUMNS" ] || ! type fmt>/dev/null; then
echo "$*"
else
@@ -265,7 +322,7 @@ if read_src_option timeout-factor factor; then
fi
log_output() {
- if $verbose; then
+ if [ $output_mode = $really_verbose ]; then
maxcol=${1:-1024}
awk "
BEGIN { count = 0 }
@@ -323,7 +380,7 @@ verify_compilation() {
warnings=$(grep -ic 'warning:' "$log")
if [ $warnings -gt 0 ]; then
fail "excess warnings:" $warnings
- if ! $verbose && ! $quiet; then
+ if [ $output_mode = $verbose ]; then
grep -i 'warning:' "$log" | head -n5
fi
elif [ "$xfail" = "compile" ]; then
@@ -344,7 +401,7 @@ verify_compilation() {
fail "excess errors:" $errors
fi
fi
- if ! $verbose && ! $quiet; then
+ if [ $output_mode = $verbose ]; then
grep -i 'error:' "$log" | head -n5
fi
return 1
@@ -365,7 +422,7 @@ verify_test() {
return 0
else
$keep_failed || rm "$exe"
- if ! $verbose && ! $quiet; then
+ if [ $output_mode = $verbose ]; then
grep -i fail "$log" | head -n5
fi
if [ $exitstatus -eq 124 ]; then
diff --git a/libstdc++-v3/testsuite/experimental/simd/generate_makefile.sh b/libstdc++-v3/testsuite/experimental/simd/generate_makefile.sh
index 4fb710c7767..ce5162a6ceb 100755
--- a/libstdc++-v3/testsuite/experimental/simd/generate_makefile.sh
+++ b/libstdc++-v3/testsuite/experimental/simd/generate_makefile.sh
@@ -97,7 +97,7 @@ driveroptions := \$(DRIVEROPTS)
all: simd_testsuite.sum
-simd_testsuite.sum: simd_testsuite.log
+simd_testsuite.sum: .progress .progress_total simd_testsuite.log
@printf "\n\t\t=== simd_testsuite \$(test_flags) Summary ===\n\n"\\
"# of expected passes:\t\t\$(shell grep -c '^PASS:' \$@)\n"\\
"# of unexpected passes:\t\t\$(shell grep -c '^XPASS:' \$@)\n"\\
@@ -255,7 +255,7 @@ EOF
done
cat <<EOF
run-%: export GCC_TEST_RUN_EXPENSIVE=yes
-run-%: driveroptions += -v
+run-%: driveroptions += -vv
run-%: %.log
@rm \$^ \$(^:log=sum)
@@ -266,17 +266,22 @@ EOF
dsthelp="${dst%Makefile}.make_help.txt"
cat <<EOF > "$dsthelp"
use DRIVEROPTS=<options> to pass the following options:
--q, --quiet Only print failures.
--v, --verbose Print compiler and test output on failure.
+-q, --quiet Disable same-line progress output (default if stdout is
+ not a tty).
+-p, --percentage Add percentage to default same-line progress output.
+-v, --verbose Print one line per test and minimal extra information on
+ failure.
+-vv Print all compiler and test output.
-k, --keep-failed Keep executables of failed tests.
--sim <executable> Path to an executable that is prepended to the test
execution binary (default: the value of
GCC_TEST_SIMULATOR).
--timeout-factor <x>
Multiply the default timeout with x.
---run-expensive Compile and run tests marked as expensive (default:
+-x, --run-expensive Compile and run tests marked as expensive (default:
true if GCC_TEST_RUN_EXPENSIVE is set, false otherwise).
---only <pattern> Compile and run only tests matching the given pattern.
+-o <pattern>, --only <pattern>
+ Compile and run only tests matching the given pattern.
use TESTFLAGS=<flags> to pass additional compiler flags
@@ -285,9 +290,13 @@ The following are some of the valid targets for this Makefile:
... clean
... help"
EOF
+ N=$(((0$(
+ all_tests | while read file && read name; do
+ all_types "$file" | printf " + %d" $(wc -l)
+ done) ) * 5))
all_tests | while read file && read name; do
echo "... run-${name}"
- all_types | while read t && read type; do
+ all_types "$file" | while read t && read type; do
echo "... run-${name}-${type}"
for i in $(seq 0 9); do
echo "... run-${name}-${type}-$i"
@@ -296,10 +305,16 @@ EOF
done >> "$dsthelp"
cat <<EOF
+.progress:
+ @echo 0 > .progress
+
+.progress_total:
+ @echo $N > .progress_total
+
clean:
- rm -f -- *.sum *.log *.exe
+ rm -f -- *.sum *.log *.exe .progress .progress_total
-.PHONY: clean help
+.PHONY: all clean help .progress .progress_total
.PRECIOUS: %.log %.sum
EOF
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 3/3] libstdc++: Document simd testsuite
2021-06-08 8:57 [PATCH 0/3] Improve and document stdx::simd testsuite Matthias Kretz
2021-06-08 8:58 ` [PATCH 1/3] libstdc++: Remove -fno-tree-vrp after PR98834 was resolved Matthias Kretz
2021-06-08 8:59 ` [PATCH 2/3] libstdc++: Improve output verbosity options and default Matthias Kretz
@ 2021-06-08 8:59 ` Matthias Kretz
2021-06-23 15:46 ` [PATCH 0/3] Improve and document stdx::simd testsuite Jonathan Wakely
3 siblings, 0 replies; 6+ messages in thread
From: Matthias Kretz @ 2021-06-08 8:59 UTC (permalink / raw)
To: gcc-patches, libstdc++
[-- Attachment #1: Type: text/plain, Size: 981 bytes --]
libstdc++-v3/ChangeLog:
* testsuite/experimental/simd/README.md: New file.
Signed-off-by: Matthias Kretz <m.kretz@gsi.de>
---
.../testsuite/experimental/simd/README.md | 257 ++++++++++++++++++
1 file changed, 257 insertions(+)
create mode 100644 libstdc++-v3/testsuite/experimental/simd/README.md
--
──────────────────────────────────────────────────────────────────────────
Dr. Matthias Kretz https://mattkretz.github.io
GSI Helmholtz Centre for Heavy Ion Research https://gsi.de
std::experimental::simd https://github.com/VcDevel/std-simd
──────────────────────────────────────────────────────────────────────────
[-- Attachment #2: 0003-libstdc-Document-simd-testsuite.patch --]
[-- Type: text/x-patch, Size: 9851 bytes --]
diff --git a/libstdc++-v3/testsuite/experimental/simd/README.md b/libstdc++-v3/testsuite/experimental/simd/README.md
new file mode 100644
index 00000000000..db0d71f8d43
--- /dev/null
+++ b/libstdc++-v3/testsuite/experimental/simd/README.md
@@ -0,0 +1,257 @@
+# SIMD Tests
+
+To execute the simd testsuite, call `make check-simd`, typically with `-j N`
+argument.
+
+For more control over verbosity, compiler flags, and use of a simulator, use
+the environment variables documented below.
+
+## Environment variables
+
+### `target_list`
+
+Similar to dejagnu target lists: E.g.
+`target_list="unix{-march=sandybridge,-march=native/-ffast-math,-march=native/-ffinite-math-only}"
+would create three subdirs in `testsuite/simd/` to run the complete simd
+testsuite first with `-march=sandybridge`, then with `-march=native
+-ffast-math`, and finally with `-march=native -ffinite-math-only`.
+
+
+### `CHECK_SIMD_CONFIG`
+
+This variable can be set to a path to a file which is equivalent to a dejagnu
+board. The file needs to be a valid `sh` script since it is sourced from the
+`scripts/check_simd` script. It's purpose is to set the `target_list` variable
+depending on `$target_triplet` (or whatever else makes sense for you). Example:
+
+```sh
+case "$target_triplet" in
+x86_64-*)
+ target_list="unix{-march=sandybridge,-march=skylake-avx512,-march=native/-ffast-math,-march=athlon64,-march=core2,-march=nehalem,-march=skylake,-march=native/-ffinite-math-only,-march=knl}"
+ ;;
+
+powerpc64le-*)
+ define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc112"
+ define_target power8 "-mcpu=power8 -static" "$HOME/bin/run_on_gccfarm gcc112"
+ define_target power9 "-mcpu=power9 -static" "$HOME/bin/run_on_gccfarm gcc135"
+ target_list="power7 power8 power9{,-ffast-math}"
+ ;;
+
+powerpc64-*)
+ define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc110"
+ define_target power8 "-mcpu=power8 -static" "$HOME/bin/run_on_gccfarm gcc110"
+ target_list="power7 power8{,-ffast-math}"
+ ;;
+esac
+```
+
+The `unix` target is pre-defined to have no initial flags and no simulator. Use
+the `define_target(name, flags, sim)` function to define your own targets for
+the `target_list` variable. In the example above `define_target power7
+"-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc112"` defines the target
+`power7` which always uses the flags `-mcpu=power7` and `-static` when
+compiling tests and prepends `$HOME/bin/run_on_gccfarm gcc112` to test
+executables. In `target_list` you can now use the name `power7`. E.g.
+`target_list="power7 power7/-ffast-math"` or it's shorthand
+`target_list="power7{,-ffast-math}"`.
+
+
+### `DRIVEROPTS`
+
+This variable affects the `Makefile`s generated per target (as defined above).
+It's a string of flags that are prepended to the `driver.sh` invocation which
+builds and runs the tests. You `cd` into a simd test subdir and use `make help`
+to see possible options and a list of all valid targets.
+
+```
+use DRIVEROPTS=<options> to pass the following options:
+-q, --quiet Disable same-line progress output (default if stdout is
+ not a tty).
+-p, --percentage Add percentage to default same-line progress output.
+-v, --verbose Print one line per test and minimal extra information on
+ failure.
+-vv Print all compiler and test output.
+-k, --keep-failed Keep executables of failed tests.
+--sim <executable> Path to an executable that is prepended to the test
+ execution binary (default: the value of
+ GCC_TEST_SIMULATOR).
+--timeout-factor <x>
+ Multiply the default timeout with x.
+-x, --run-expensive Compile and run tests marked as expensive (default:
+ true if GCC_TEST_RUN_EXPENSIVE is set, false otherwise).
+-o <pattern>, --only <pattern>
+ Compile and run only tests matching the given pattern.
+```
+
+
+### `TESTFLAGS`
+
+This variable also affects the `Makefile`s generated per target. It's a list of
+compiler flags that are appended to `CXXFLAGS`.
+
+
+### `GCC_TEST_SIMULATOR`
+
+If `--sim` is not passed via `DRIVEROPTS`, then this variable is prepended to
+test invocations. If a simulator was defined via the `CHECK_SIMD_CONFIG`
+script, then then generated `Makefile` sets the `GCC_TEST_SIMULATOR` variable.
+
+
+### `GCC_TEST_RUN_EXPENSIVE`
+
+If set to any non-empty string, run tests marked as expensive, otherwise treat
+these tests as `UNSUPPORTED`.
+
+
+## Writing new tests
+
+A test starts with the copyright header, directly followed by directives
+influencing the set of tests to generate and whether the test driver should
+expect a failure.
+
+Then the test must at least `#include "bits/verify.h", which provides `main`
+and declares a `template <typename V> void test()` function, which the test has
+to define. The template parameter is set to `simd<T, Abi>` type where `T` and
+`Abi` are determined by the type and ABI subset dimensions.
+
+The `test()` functions are typically implemented using the `COMPARE(x,
+reference)`, `VERIFY(boolean)`, and `ULP_COMPARE(x, reference,
+allowed_distance)` macros.
+
+### Directives
+
+* `// skip: <type pattern> <ABI subset pattern> <target triplet pattern>
+ <CXXFLAGS pattern>`
+ If all patterns match, the test is silently skipped.
+
+* `// only: <type pattern> <ABI subset pattern> <target triplet pattern>
+ <CXXFLAGS pattern>`
+ If any pattern doesn't match, the test is silently skipped.
+
+* `// expensive: <type pattern> <ABI subset pattern> <target triplet pattern>
+ <CXXFLAGS pattern>`
+ If all patterns match, the test is `UNSUPPORTED` unless expensive tests are
+ enabled.
+
+* `// xfail: run|compile <type pattern> <ABI subset pattern> <target triplet
+ pattern> <CXXFLAGS pattern>`
+ If all patterns match, test compilation or execution is expected to fail. The
+ test then shows as "XFAIL: ...". If the test passes, the test shows "XPASS:
+ ...".
+
+All patterns are matched via
+```sh
+case '<test context>' in
+ <pattern>)
+ # treat as match
+ ;;
+esac
+```
+The `<CXXFLAGS pattern>` is implicitly adds a `*` wildcard before and after the
+pattern. Thus, the `CXXFLAGS` pattern matches a substring and all other
+patterns require a full match.
+
+Examples:
+```cpp
+// The test is only valid for floating-point types:
+// only: float|double|ldouble * * *
+
+// Skip the test for long double for all powerpc64* targets:
+// skip: ldouble * powerpc64* *
+
+// The test is expected to unconditionally fail on execution:
+// xfail: run * * * *
+
+// ABI subsets 1-9 are considered expensive:
+// expensive: * [1-9] * *
+```
+
+
+## Implementation sketch
+
+* `scripts/create_testsuite_files` collects all `*.c` and `*.cc` files with
+ `simd/tests/` in their path into the file `testsuite_file_simd` (and at the
+ same time removes them from `testsuite_files`.
+
+* The `check-simd` target in `testsuite/Makefile.am` calls
+ `scripts/check_simd`. This script calls
+ `testsuite/experimental/simd/generate_makefile.sh` to generate `Makefile`s in
+ all requested subdirectories. The subdirectories are communicated back to the
+ make target via a `stdout` pipe. The `check-simd` rule then spawns sub-make
+ in these subdirectories. Finally it collects all summaries
+ (`simd_testsuite.sum`) to present them at the end of the rule.
+
+* The generated Makefiles define targets for each file in `testsuite_file_simd`
+ (you can edit this file after it was generated, though that's not
+ recommended) while adding two test dimensions: type and ABI subset. The type
+ is a list of all arithmetic types, potentially reduced via `only` and/or
+ `skip` directives in the test's source file. The ABI subset is a number
+ between 0 and 9 (inclusive) mapping to a set of `simd_abi`s in
+ `testsuite/experimental/simd/tests/bits/verify.h` (`iterate_abis()`). The
+ tests are thus potentially compiled 170 (17 arithmetic types * 10 ABI
+ subsets) times. This is necessary to limit the memory usage of GCC to
+ reasonable numbers and keep the compile time below 1 minute (per compiler
+ invocation).
+
+* When `make` executes in the generated subdir, the `all` target depends on
+ building and running all tests via `testsuite/experimental/simd/driver.sh`
+ and collecting their logs into a `simd_testsuite.log` and then extracting
+ `simd_testsuite.sum` from it.
+
+* The `driver.sh` script builds and runs the test, parses the compiler and test
+ output, and prints progress information to the terminal.
+
+## Appendix
+
+### `run_on_gccfarm` script
+
+```sh
+#!/bin/sh
+usage() {
+ cat <<EOF
+Usage $0 <hostname> <executable> [arguments]
+
+Copies <executable> to $host, executes it and cleans up again.
+EOF
+}
+
+[ $# -lt 2 ] && usage && exit 1
+case "$1" in
+ -h|--help)
+ usage
+ exit
+ ;;
+esac
+
+host="$1"
+exe="$2"
+shift 2
+
+# Copy executable locally to strip it before scp to remote host
+local_tmpdir=$(mktemp -d)
+cp "$exe" $local_tmpdir
+cd $local_tmpdir
+exe="${exe##*/}"
+powerpc64le-linux-gnu-strip "$exe"
+
+ssh_controlpath=~/.local/run_on_gccfarm/$host
+if [ ! -S $ssh_controlpath ]; then
+ mkdir -p ~/.local/run_on_gccfarm
+ (
+ flock -n 9
+ if [ ! -S $ssh_controlpath ]; then
+ ssh -o ControlMaster=yes -o ControlPath=$ssh_controlpath -o ControlPersist=10m $host.fsffrance.org true
+ fi
+ ) 9> ~/.local/run_on_gccfarm/lockfile
+fi
+opts="-o ControlPath=$ssh_controlpath"
+
+remote_tmpdir=$(ssh $opts $host.fsffrance.org mktemp -d -p .)
+scp $opts -C -q "$exe" $host.fsffrance.org:$remote_tmpdir/
+cd
+rm -r "$local_tmpdir" &
+ssh $opts $host.fsffrance.org $remote_tmpdir/$exe "$@"
+ret=$?
+ssh $opts $host.fsffrance.org rm -r $remote_tmpdir &
+exit $ret
+```
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/3] Improve and document stdx::simd testsuite
2021-06-08 8:57 [PATCH 0/3] Improve and document stdx::simd testsuite Matthias Kretz
` (2 preceding siblings ...)
2021-06-08 8:59 ` [PATCH 3/3] libstdc++: Document simd testsuite Matthias Kretz
@ 2021-06-23 15:46 ` Jonathan Wakely
2021-06-24 13:12 ` Jonathan Wakely
3 siblings, 1 reply; 6+ messages in thread
From: Jonathan Wakely @ 2021-06-23 15:46 UTC (permalink / raw)
To: Matthias Kretz; +Cc: gcc Patches, libstdc++
On Tue, 8 Jun 2021 at 09:57, Matthias Kretz <m.kretz@gsi.de> wrote:
>
> As discussed a long time ago on IRC, this improves (i.e. decreases by default)
> the verbosity of make check-simd, gives more verbosity options, and finally
> documents how the simd testsuite is used and how it works. In addition, after
> PR98834 was resolved, remove the -fno-tree-vrp workaround.
>
> Tested on x86_64-linux (and more).
Great, thanks, I'm about to push all 3 patches to trunk.
>
>
> Matthias Kretz (3):
> libstdc++: Remove -fno-tree-vrp after PR98834 was resolved
> libstdc++: Improve output verbosity options and default
> libstdc++: Document simd testsuite
>
> libstdc++-v3/testsuite/Makefile.am | 3 +-
> libstdc++-v3/testsuite/Makefile.in | 3 +-
> .../testsuite/experimental/simd/README.md | 257 ++++++++++++++++++
> .../testsuite/experimental/simd/driver.sh | 137 +++++++---
> .../experimental/simd/generate_makefile.sh | 33 ++-
> 5 files changed, 380 insertions(+), 53 deletions(-)
> create mode 100644 libstdc++-v3/testsuite/experimental/simd/README.md
>
> --
> ──────────────────────────────────────────────────────────────────────────
> Dr. Matthias Kretz https://mattkretz.github.io
> GSI Helmholtz Centre for Heavy Ion Research https://gsi.de
> std::experimental::simd https://github.com/VcDevel/std-simd
> ──────────────────────────────────────────────────────────────────────────
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/3] Improve and document stdx::simd testsuite
2021-06-23 15:46 ` [PATCH 0/3] Improve and document stdx::simd testsuite Jonathan Wakely
@ 2021-06-24 13:12 ` Jonathan Wakely
0 siblings, 0 replies; 6+ messages in thread
From: Jonathan Wakely @ 2021-06-24 13:12 UTC (permalink / raw)
To: Matthias Kretz; +Cc: gcc Patches, libstdc++
[-- Attachment #1: Type: text/plain, Size: 599 bytes --]
On Wed, 23 Jun 2021 at 16:46, Jonathan Wakely wrote:
>
> On Tue, 8 Jun 2021 at 09:57, Matthias Kretz <m.kretz@gsi.de> wrote:
> >
> > As discussed a long time ago on IRC, this improves (i.e. decreases by default)
> > the verbosity of make check-simd, gives more verbosity options, and finally
> > documents how the simd testsuite is used and how it works. In addition, after
> > PR98834 was resolved, remove the -fno-tree-vrp workaround.
> >
> > Tested on x86_64-linux (and more).
>
> Great, thanks, I'm about to push all 3 patches to trunk.
I've pushed this fix for some typos in the new README.md
[-- Attachment #2: patch.txt --]
[-- Type: text/plain, Size: 2455 bytes --]
commit 07ba52849ffca26a3d461f94921b23a9cdbaea7f
Author: Jonathan Wakely <jwakely@redhat.com>
Date: Thu Jun 24 13:49:19 2021
libstdc++: Fix typos and markdown errors in new simd/README.md
Signed-off-by: Jonathan Wakely <jwakely@redhat.com>
libstdc++-v3/ChangeLog:
* testsuite/experimental/simd/README.md: Fix typos.
diff --git a/libstdc++-v3/testsuite/experimental/simd/README.md b/libstdc++-v3/testsuite/experimental/simd/README.md
index db0d71f8d43..b82453df403 100644
--- a/libstdc++-v3/testsuite/experimental/simd/README.md
+++ b/libstdc++-v3/testsuite/experimental/simd/README.md
@@ -11,7 +11,7 @@
### `target_list`
Similar to dejagnu target lists: E.g.
-`target_list="unix{-march=sandybridge,-march=native/-ffast-math,-march=native/-ffinite-math-only}"
+`target_list="unix{-march=sandybridge,-march=native/-ffast-math,-march=native/-ffinite-math-only}"`
would create three subdirs in `testsuite/simd/` to run the complete simd
testsuite first with `-march=sandybridge`, then with `-march=native
-ffast-math`, and finally with `-march=native -ffinite-math-only`.
@@ -21,7 +21,7 @@
This variable can be set to a path to a file which is equivalent to a dejagnu
board. The file needs to be a valid `sh` script since it is sourced from the
-`scripts/check_simd` script. It's purpose is to set the `target_list` variable
+`scripts/check_simd` script. Its purpose is to set the `target_list` variable
depending on `$target_triplet` (or whatever else makes sense for you). Example:
```sh
@@ -52,7 +52,7 @@
`power7` which always uses the flags `-mcpu=power7` and `-static` when
compiling tests and prepends `$HOME/bin/run_on_gccfarm gcc112` to test
executables. In `target_list` you can now use the name `power7`. E.g.
-`target_list="power7 power7/-ffast-math"` or it's shorthand
+`target_list="power7 power7/-ffast-math"` or its shorthand
`target_list="power7{,-ffast-math}"`.
@@ -109,7 +109,7 @@
influencing the set of tests to generate and whether the test driver should
expect a failure.
-Then the test must at least `#include "bits/verify.h", which provides `main`
+Then the test must at least `#include "bits/verify.h"`, which provides `main`
and declares a `template <typename V> void test()` function, which the test has
to define. The template parameter is set to `simd<T, Abi>` type where `T` and
`Abi` are determined by the type and ABI subset dimensions.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-06-24 13:13 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-08 8:57 [PATCH 0/3] Improve and document stdx::simd testsuite Matthias Kretz
2021-06-08 8:58 ` [PATCH 1/3] libstdc++: Remove -fno-tree-vrp after PR98834 was resolved Matthias Kretz
2021-06-08 8:59 ` [PATCH 2/3] libstdc++: Improve output verbosity options and default Matthias Kretz
2021-06-08 8:59 ` [PATCH 3/3] libstdc++: Document simd testsuite Matthias Kretz
2021-06-23 15:46 ` [PATCH 0/3] Improve and document stdx::simd testsuite Jonathan Wakely
2021-06-24 13:12 ` Jonathan Wakely
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).