From: Alexandre Oliva <oliva@gnu.org>
To: Jeff Law <jeffreyalaw@gmail.com>
Cc: "gcc-patches@gcc.gnu.org" <gcc-patches@gcc.gnu.org>
Subject: [PATCH] compare_tests: distinguish c-c++-common results by tool
Date: Wed, 20 Dec 2023 03:31:26 -0300 [thread overview]
Message-ID: <or8r5p2xmp.fsf@lxoliva.fsfla.org> (raw)
In-Reply-To: <76a02442-290a-4929-b566-035753afd859@gmail.com> (Jeff Law's message of "Tue, 19 Dec 2023 21:40:32 -0700")
On Dec 20, 2023, Jeff Law <jeffreyalaw@gmail.com> wrote:
> So the strub tests in c-c++-common are problematical. They get run
> twice, once for C, once for C++. Yet the name of the test is the same
> in both runs. (by the name, I mean the name emitted into the dejagnu
> summary and log files).
> Thus if you have a test in there which passes in one context, but
> fails in the other, comparison tools like contrib/compare_tests may
> erroneously report the tests as both a test which now fails, but
> passed before and a test which now passes but failed before.
> It looks like some of the strub tests are currently known to fail with
> C++ and are triggering this problem
Yeah, type warnings/errors are different between C and C++, and this is
noticeable with permissible conversions between strub types.
> A third option would be to change the compare_tests tool to somehow
> distinguish between the C and C++ tests. Not sure how feasible that
> is.
Most feasible among the possibilities ;-)
I've tested the following by comparing my obj-x86_64-linux-gnu test tree
with itself. Ok to install?
When compare_tests compares both C and C++ tests in c-c++-common, they
get the same identifier, so expected differences in results across
languages become undesirably noisy.
This patch adds tool identifiers to tests, so that runs by different
tools are not confused by the compare logic.
It also fixes a bug in reporting differences, that would attempt to
print an undefined fname (the definitions are in subshell loops), and
adjusts the target insertion to match tabs in addition to blanks after
colons.
for contrib/ChangeLog
* compare_tests: Add tool to test lines. Match tabs besides
blanks to insert tool and target. Don't print undefined fname.
---
contrib/compare_tests | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/contrib/compare_tests b/contrib/compare_tests
index 2dfa8640756a0..e09fc4f113a3f 100755
--- a/contrib/compare_tests
+++ b/contrib/compare_tests
@@ -96,7 +96,7 @@ if [ -d "$1" -a -d "$2" ] ; then
ret=$?
if [ $ret -ne 0 ]; then
exit_status=`expr $exit_status + 1`
- echo "## Differences found: $fname"
+ echo "## Differences found"
fi
if [ $exit_status -ne 0 ]; then
echo "# $exit_status differences in $cmnsums common sum files found"
@@ -108,8 +108,8 @@ elif [ -d "$1" -o -d "$2" ] ; then
usage "Must specify either two directories or two files"
fi
-sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^Running target / {target = $3} { if (target != "unix") { sub(/: /, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp1
-sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^Running target / {target = $3} { if (target != "unix") { sub(/: /, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp2
+sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$1" | awk '/^[ ]*=== [^ ]* tests ===$/ {tool = $2} /^Running target / {target = $3} { if (tool != "") { sub(/:[ ]/, "&"tool": " ); }; if (target != "unix") { sub(/:[ ]/, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp1
+sed 's/^XFAIL/FAIL/; s/^ERROR/FAIL/; s/^XPASS/PASS/' < "$2" | awk '/^[ ]*=== [^ ]* tests ===$/ {tool = $2} /^Running target / {target = $3} { if (tool != "") { sub(/:[ ]/, "&"tool": " ); }; if (target != "unix") { sub(/:[ ]/, "&"target": " ); }; print $0; }' | cut -c1-2000 >$tmp2
before=$tmp1
now=$tmp2
--
Alexandre Oliva, happy hacker https://FSFLA.org/blogs/lxo/
Free Software Activist GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice but
very few check the facts. Think Assange & Stallman. The empires strike back
next prev parent reply other threads:[~2023-12-20 6:31 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-20 4:40 Problems with strub tests Jeff Law
2023-12-20 6:31 ` Alexandre Oliva [this message]
2023-12-20 17:17 ` [PATCH] compare_tests: distinguish c-c++-common results by tool Jeff Law
2024-01-07 1:51 ` Problems with strub tests Hans-Peter Nilsson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=or8r5p2xmp.fsf@lxoliva.fsfla.org \
--to=oliva@gnu.org \
--cc=gcc-patches@gcc.gnu.org \
--cc=jeffreyalaw@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).