From: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
To: libc-alpha@sourceware.org
Cc: Carlos O'Donell <carlos@redhat.com>, DJ Delorie <dj@delorie.com>,
Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
Subject: [PATCH] Format test results closer to what DejaGnu does
Date: Thu, 1 Jun 2023 10:14:24 +0000 [thread overview]
Message-ID: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> (raw)
The years of dealing with Binutils, GCC and GDB test results
made the community create good tools for comparison and analysis
of DejaGnu test results. This change allows to use those tools
for Glibc's test results as well.
The motivation for this change is Linaro's pre-commit testers,
which use a modified version of GCC's validate_failures.py
to create test xfail lists with baseline failures and known
flaky tests. See below links for an example xfails file (only
one link is supposed to work at any given time):
- https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/
- https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/
Specifacally, this patch changes format of glibc's .sum files from ...
<cut>
FAIL: elf/test1
PASS: string/test2
</cut>
... to ...
<cut>
=== glibc tests ===
Running elf ...
FAIL: elf/test1
Running string ...
PASS: string/test2
</cut>.
And output of "make check" from ...
<cut>
FAIL: elf/test1
</cut>
... to ...
<cut>
=== glibc failures ===
FAIL: elf/test1
=== Summary of results ===
1 FAIL
1 PASS
</cut>.
Signed-off-by: Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
---
Makefile | 12 ++++++++----
scripts/merge-test-results.sh | 3 +++
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile
index 224c792185..779938f3b7 100644
--- a/Makefile
+++ b/Makefile
@@ -564,11 +564,15 @@ $(objpfx)check-wrapper-headers.out: scripts/check-wrapper-headers.py $(headers)
--generated $(common-generated) > $@; $(evaluate-test)
endif # $(headers)
+# Print test summary for tests in $1 .sum file;
+# $2 is optional test identifier.
+# Fail if there are unexpected failures in the test results.
define summarize-tests
-@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
-@echo "Summary of test results$2:"
-@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c
-@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1
+@echo " === glibc failures ==="
+@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true
+@echo " === Summary of results$2 ==="
+@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c
+@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):'
endef
# The intention here is to do ONE install of our build into the
diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
index e4dcc2520a..8f9d81f6eb 100755
--- a/scripts/merge-test-results.sh
+++ b/scripts/merge-test-results.sh
@@ -50,7 +50,10 @@ case $type in
-t)
subdir_file_name=$1
shift
+ echo " === glibc tests ==="
for d in "$@"; do
+ echo
+ echo "Running $d ..."
if [ -f "$objpfx$d/$subdir_file_name" ]; then
cat "$objpfx$d/$subdir_file_name"
else
--
2.34.1
next reply other threads:[~2023-06-01 10:14 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-01 10:14 Maxim Kuvyrkov [this message]
2023-06-05 13:35 ` Florian Weimer
2023-06-05 13:51 ` Maxim Kuvyrkov
2023-06-19 13:59 ` Florian Weimer
2023-06-19 17:55 ` Maxim Kuvyrkov
2023-11-01 10:28 ` [PATCH v2] " Maxim Kuvyrkov
2023-11-01 13:58 ` Adhemerval Zanella Netto
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230601101424.1848817-1-maxim.kuvyrkov@linaro.org \
--to=maxim.kuvyrkov@linaro.org \
--cc=carlos@redhat.com \
--cc=dj@delorie.com \
--cc=libc-alpha@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).