[Bug testsuite/13443] New: test messages should be unique

[Bug testsuite/13443] New: test messages should be unique

[pedro at codesourcery dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13443

             Bug #: 13443
           Summary: test messages should be unique
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: testsuite
        AssignedTo: unassigned@sourceware.org
        ReportedBy: pedro@codesourcery.com
    Classification: Unclassified


There are many tests in the test suite that output the same message as other
tests in the same test file.

We should make each output unique, to facilitate test identification, as well
as helping with test case result analysis/comparison.

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq
-c | sort -n | grep -v "^\( \)*1 " | wc -l
1263

That means there are currently 1263 outputs in gdb.sum that are not unique. 
That was on amd64-linux-linux.

The top 20 of worse offenders is:

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq
-c | sort -n | tail -n 20

      7 gdb.python/py-value.exp: get value from history
      8 gdb.python/py-template.exp: print foo
      8 gdb.python/py-template.exp: python foo = gdb.history(0)
      8 gdb.python/py-type.exp: get value from history
      9 gdb.base/call-sc.exp: return foo; synchronize pc to main()
      9 gdb.base/call-sc.exp: set print address off
      9 gdb.base/call-sc.exp: set print sevenbit-strings
      9 gdb.base/call-sc.exp: set width 0
      9 gdb.python/py-breakpoint.exp: continue to breakpoint: Break at
multiply.
     10 gdb.base/checkpoint.exp: restart 1 three
     10 gdb.base/relational.exp: set variable x
     11 gdb.base/relational.exp: set variable z
     11 gdb.python/py-type.exp: print value
     15 gdb.base/dump.exp: print zero_all ()
     24 gdb.base/structs.exp: set print address off
     24 gdb.base/structs.exp: set print elements 300
     24 gdb.base/structs.exp: set print sevenbit-strings
     24 gdb.base/structs.exp: set width 0
     26 gdb.reverse/sigall-precsave.exp: reverse signal delivered
     26 gdb.reverse/sigall-reverse.exp: reverse signal delivered

The first column indicates how often is the output message repeated in gdb.sum.

And here's the distribution:

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq
-c | sort -n | awk ' { print $1 } ' | uniq -c
  18636 1
   1129 2
     55 3
     41 4
      8 5
      6 6
      5 7
      3 8
      5 9
      2 10
      2 11
      1 15
      4 24
      2 26

Unfortunately, by far, we have many cases of the same output appearing twice.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug testsuite/13443] test messages should be unique

[pedro at codesourcery dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13443

--- Comment #1 from Pedro Alves <pedro at codesourcery dot com> 2011-11-27 11:42:57 UTC ---
gcc's compare_tests script is one example of a tool that gets confused by the
non-unique test messages.  

E.g., comparing the same .sum with itself, should naturally not find any
regressions or new passes, but note:

$ ~/src/gcc/gcc/contrib/compare_tests testsuite/gdb.sum testsuite/gdb.sum 
Tests that now fail, but worked before:

gdb.base/jit.exp: one_jit_test-1: info function jit_function
gdb.base/jit.exp: one_jit_test-2: info function jit_function
gdb.mi/mi-break.exp: list of breakpoints
gdb.mi/mi2-break.exp: list of breakpoints
one_jit_test-1 info function jit_function
one_jit_test-2 info function jit_function

Tests that now work, but didn't before:

gdb.base/jit.exp: one_jit_test-1: info function jit_function
gdb.base/jit.exp: one_jit_test-2: info function jit_function
gdb.mi/mi-break.exp: list of breakpoints
gdb.mi/mi2-break.exp: list of breakpoints
one_jit_test-1 info function jit_function
one_jit_test-2 info function jit_function

This is because each of these tests outputs the same message twice, once with a
PASS and once with a FAIL.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug testsuite/13443] test messages should be unique

[cvs-commit at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=13443

--- Comment #2 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> 2012-12-04 03:56:03 UTC ---
CVSROOT:    /cvs/src
Module name:    src
Changes by:    qiyao@sourceware.org    2012-12-04 03:55:52

Modified files:
    gdb/testsuite  : ChangeLog 
    gdb/testsuite/gdb.trace: actions.exp 

Log message:
    gdb/testsuite
    2012-12-04  Yao Qi  <yao@codesourcery.com>

    PR gdb/13443
    * gdb.trace/actions.exp: Make test messages unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3471&r2=1.3472
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.trace/actions.exp.diff?cvsroot=src&r1=1.25&r2=1.26

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug testsuite/13443] test messages should be unique

[cvs-commit at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=13443

--- Comment #3 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> 2013-01-18 10:50:21 UTC ---
CVSROOT:    /cvs/src
Module name:    src
Changes by:    abidh@sourceware.org    2013-01-18 10:50:15

Modified files:
    gdb/testsuite  : ChangeLog 
    gdb/testsuite/gdb.base: checkpoint.exp 

Log message:
    2013-01-18  Hafiz Abid Qadeer<abidh@codesourcery.com>

    PR gdb/13443
    * gdb.base/checkpoint.exp: Update test messages to make them
    unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3517&r2=1.3518
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.base/checkpoint.exp.diff?cvsroot=src&r1=1.20&r2=1.21

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug testsuite/13443] test messages should be unique

[cvs-commit at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=13443

--- Comment #4 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> 2013-01-24 10:29:13 UTC ---
CVSROOT:    /cvs/src
Module name:    src
Changes by:    abidh@sourceware.org    2013-01-24 10:29:09

Modified files:
    gdb/testsuite  : ChangeLog 
    gdb/testsuite/gdb.mi: mi-var-block.exp 

Log message:
    2013-01-24  Hafiz Abid Qadeer  <abidh@codesourcery.com>

    PR gdb/13443
    * gdb.mi/mi-var-block.exp: Make test messages unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3530&r2=1.3531
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.mi/mi-var-block.exp.diff?cvsroot=src&r1=1.29&r2=1.30

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug testsuite/13443] test messages should be unique

[cvs-commit at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=13443

--- Comment #5 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
CVSROOT:    /cvs/src
Module name:    src
Changes by:    qiyao@sourceware.org    2013-07-31 00:38:40

Modified files:
    gdb/testsuite  : ChangeLog 
    gdb/testsuite/gdb.trace: backtrace.exp 

Log message:
    gdb/testsuite/

    PR gdb/13443
    * gdb.trace/backtrace.exp (gdb_backtrace_tdp_3): Add parameter
    'traceframe'.  Wrap test with with_test_prefix.
    (top level): Update.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3752&r2=1.3753
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.trace/backtrace.exp.diff?cvsroot=src&r1=1.28&r2=1.29

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/13443] test messages should be unique

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=13443

--- Comment #6 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "gdb and binutils".

The branch, master has been updated
       via  5792e8e37be2f848152115fd4783b0e28b5253ad (commit)
      from  26278bb8717a16139b3a4da9ba1cd2aed435f1ee (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=5792e8e37be2f848152115fd4783b0e28b5253ad

commit 5792e8e37be2f848152115fd4783b0e28b5253ad
Author: Yao Qi <yao@codesourcery.com>
Date:   Thu Aug 7 15:01:22 2014 +0800

    Make test messages in gdb.mi/mi-var-display.exp unique

    When I fix a bug in gdb.mi/mi-var-display.exp, I find its test
    messages aren't unique, which makes some confusions for me.

    $ cat testsuite/gdb.sum | grep "PASS" | sort | uniq -c | sort -n
    ...
     2 PASS: gdb.mi/mi-var-display.exp: set format variable bar
     2 PASS: gdb.mi/mi-var-display.exp: set format variable foo
     2 PASS: gdb.mi/mi-var-display.exp: set format variable weird.func_ptr
     2 PASS: gdb.mi/mi-var-display.exp: set format variable weird.func_ptr_ptr
     2 PASS: gdb.mi/mi-var-display.exp: show format variable foo
     3 PASS: gdb.mi/mi-var-display.exp: eval variable foo

    This patch is to make test messages in mi-var-display.exp unique.

    gdb/testsuite:

    2014-08-09  Yao Qi  <yao@codesourcery.com>

        PR testsuite/13443
        * gdb.mi/mi-var-display.exp: Make test messages unique.

-----------------------------------------------------------------------

Summary of changes:
 gdb/testsuite/ChangeLog                 |    5 ++++
 gdb/testsuite/gdb.mi/mi-var-display.exp |   32 +++++++++++++++---------------
 2 files changed, 21 insertions(+), 16 deletions(-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.