[Bug testsuite/29772] New: [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[Bug testsuite/29772] New: [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

            Bug ID: 29772
           Summary: [gdb/testsuite] Detect duplicates for "FAIL: foo" and
                    "FAIL: foo (timeout)"
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: testsuite
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
  Target Milestone: ---

I noticed here ( https://sourceware.org/gdb/wiki/GDBTestcaseCookbook ):
...
Do not use "tail parentheses" on test messages

When you write a test, do not put text between parentheses preceded by a space
at the end of the text message. For example, this is incorrect:

gdb_test "some_test" "expected output" "I am a test message (123)"
...

I then noticed that when checking for duplicates, we do detect:
...
FAIL: gdb.testsuite/duplicate.exp: foo
FAIL: gdb.testsuite/duplicate.exp: foo
...
but not:
...
FAIL: gdb.testsuite/duplicate.exp: foo
FAIL: gdb.testsuite/duplicate.exp: foo (timeout)
...

I wrote a patch to fix this:
...
diff --git a/gdb/testsuite/lib/check-test-names.exp
b/gdb/testsuite/lib/check-test
-names.exp
index 752981c7888..fac93259464 100644
--- a/gdb/testsuite/lib/check-test-names.exp
+++ b/gdb/testsuite/lib/check-test-names.exp
@@ -64,6 +64,9 @@ namespace eval ::CheckTestNames {
     proc _check_duplicates { message } {
        variable all_test_names

+       # Remove the "extra information" part.
+       set message [regsub { \(.*\)$} $message ""]
+
        # Initialise a count, or increment the count for this test name.
        if {![info exists all_test_names($message)]} {
            set all_test_names($message) 0
...

I ran into 6453 duplicates in 134 test-cases:
...
$ grep DUPLICATE: gdb.sum | wc -l
6453
$ grep DUPLICATE: gdb.sum | awk '{print $2}' | sort -u | wc -l
134
...

For instance:
...
DUPLICATE: gdb.arch/amd64-osabi.exp: arch=i386:x86-64: print sizeof (long
double)
DUPLICATE: gdb.arch/amd64-osabi.exp: arch=i386:x64-32: print sizeof (long
double)
DUPLICATE: gdb.arch/amd64-osabi.exp: arch=i386: print sizeof (long double)
...

It may be worthwhile to override all pass / fail procs to do some automated
test name substitution:
...
-PASS: gdb.arch/amd64-osabi.exp: arch=i386:x86-64: print sizeof (long double)
+PASS: gdb.arch/amd64-osabi.exp: arch=i386:x86-64: print sizeof _(long double)_
...
and create extra versions fail_with_info to do:
...
-fail "$testname (timeout)"
+fail_with_info $testname "timeout"
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
Created attachment 14449
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14449&action=edit
tentative patch, fixes one test-case

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[pedro at palves dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

Pedro Alves <pedro at palves dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pedro at palves dot net

--- Comment #2 from Pedro Alves <pedro at palves dot net> ---
For tests like:

  print sizeof (long double)

or in general any "print func (type...)" test, we can just remove the space
before parens:

  print sizeof(long double)
  print func(type)

The advantage is that the testcase name is still valid syntax.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

Tom Tromey <tromey at sourceware dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at sourceware dot org

--- Comment #3 from Tom Tromey <tromey at sourceware dot org> ---
I tried this and ugh... I wonder if it would be simpler to
change the suffix for auxiliary information.  Really it
would be nicer if this were a dejagnu feature, like
fail -reason Text testname

... but meanwhile what if we made it something unlikely to appear
in real syntax, like fail "testname <<<timeout>>>"

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[tromey at sourceware dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

--- Comment #4 from Tom Tromey <tromey at sourceware dot org> ---
I fixed the buglets here in gdb.rust, but the process seems a bit
daunting for the entire test suite.  I didn't look at how hard it
would be to change the suffix.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

--- Comment #5 from Tom de Vries <vries at gcc dot gnu.org> ---
https://sourceware.org/pipermail/gdb-patches/2024-July/210510.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

--- Comment #6 from Sourceware Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Tom de Vries <vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=35f09cd5d7fdd1a64f4d1751e73c3495bef1ed99

commit 35f09cd5d7fdd1a64f4d1751e73c3495bef1ed99
Author: Tom de Vries <tdevries@suse.de>
Date:   Wed Jul 31 15:04:25 2024 +0200

    [gdb/testsuite] Detect trailing-text-in-parentheses duplicates

    When using a duplicate test name:
    ...
    fail foo
    fail foo
    ...
    we get:
    ...
    FAIL: $exp: foo
    FAIL: $exp: foo
    DUPLICATE: $exp: foo
    ...

    But when we do:
    ...
    fail foo
    fail "foo (timeout)"
    ...
    we get only:
    ...
    FAIL: $exp: foo
    FAIL: $exp: foo (timeout)
    ...

    Trailing text between parentheses prefixed with a space is interpreted as
    extra information, and not as part of the test name [1].

    Consequently, "foo" and "foo (timeout)" do count as duplicate test names,
    which should have been detected.  This is PR testsuite/29772.

    Fix this in CheckTestNames::_check_duplicates, such that we get:
    ...
    FAIL: $exp: foo
    FAIL: $exp: foo (timeout)
    DUPLICATE: $exp: foo (timeout)
    ...

    [ One note on the implementation: I used the regexp { \([^()]*\)$}. I don't
    know whether that covers all required cases, due to the fact that those are
    not unambiguousely specified.  It might be possible to reverse-engineer
that
    information by reading or running the "regression analysis tools" mentioned
on
    the wiki page [1], but I haven't been able to.  Regardless, the current
regexp
    covers a large amount of cases, which IMO should be sufficient to be
    acceptable. ]

    Doing so shows many new duplicates in the testsuite.

    A significant number of those is due to using a message which is a copy of
the
    command:
    ...
    gdb_test "print (1)"
    ...

    Fix this by handling those cases using test names "gdb-command<print (1)>"
and
    "gdb-command<print (2)>.

    Fix the remaining duplicates manually (split off as follow-up patch for
    readability of this patch).

    Tested on x86_64-linux and aarch64-linux.

    Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=29772

    [1]
https://sourceware.org/gdb/wiki/GDBTestcaseCookbook#Do_not_use_.22tail_parentheses.22_on_test_messages

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/29772] [gdb/testsuite] Detect duplicates for "FAIL: foo" and "FAIL: foo (timeout)"

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=29772

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |16.1
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #7 from Tom de Vries <vries at gcc dot gnu.org> ---
Fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.