[Bug testsuite/26110] New: invalid command name in one test-case aborts entire test run

[Bug testsuite/26110] New: invalid command name in one test-case aborts entire test run

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26110

            Bug ID: 26110
           Summary: invalid command name in one test-case aborts entire
                    test run
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: testsuite
          Assignee: unassigned at sourceware dot org
          Reporter: vries at gcc dot gnu.org
  Target Milestone: ---

Say we mistakenly do:
...
diff --git a/gdb/testsuite/gdb.ada/O2_float_param.exp
b/gdb/testsuite/gdb.ada/O2_float_param.exp
index 09ebeec405..87f344b1e8 100644
--- a/gdb/testsuite/gdb.ada/O2_float_param.exp
+++ b/gdb/testsuite/gdb.ada/O2_float_param.exp
@@ -29,3 +29,5 @@ runto "increment"

 gdb_test "frame" \
          "#0\\s+callee\\.increment \\(val(=val@entry)?=99\\.0,
msg=\\.\\.\\.\\).*"
+
+foobar
...

and try to run the test-suite, we have:
...
Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for
target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using /data/gdb_versions/devel/src/gdb/testsuite/config/unix.exp as
tool-and-target-specific interface file.
Running /data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp
...
SOURCING: /data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp
ERROR: (DejaGnu) proc "foobar" does not exist.
The error code is TCL LOOKUP COMMAND foobar
The info on the error is:
invalid command name "foobar"
    while executing
"::tcl_unknown foobar"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 ::tcl_unknown $args"

                === gdb Summary ===

# of expected passes            2
/data/gdb_versions/devel/build/gdb/gdb version  10.0.50.20200610-git -nw -nx
-data-directory /data/gdb_versions/devel/build/gdb/testsuite/../data-directory 

...

So, the entire subsequent test run is aborted because of a mistake inside a
single test-case.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/26110] invalid command name in one test-case aborts entire test run

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26110

--- Comment #1 from Tom de Vries <vries at gcc dot gnu.org> ---
In the dejagnu proc runtest, we source each test-case inside a catch call:
...
        if { [catch "uplevel #0 source $test_file_name"] == 1 } {
...
but that doesn't catch this particular error, because in the dejagnu
framework.exp file we have:
...
# unknown -- called by expect if a proc is called that doesn't exist            
#                                                                               
# Rename unknown to tcl_unknown so that we can wrap tcl_unknown.                
# This allows Tcl package autoloading to work in the modern age.                

rename ::unknown ::tcl_unknown
proc unknown args {
    if {[catch {uplevel 1 ::tcl_unknown $args} msg]} {
        global errorCode
        global errorInfo
        global exit_status

        clone_output "ERROR: (DejaGnu) proc \"$args\" does not exist."
        if {[info exists errorCode]} {
            send_error "The error code is $errorCode\n"
        }
        if {[info exists errorInfo]} {
            send_error "The info on the error is:\n$errorInfo\n"
        }
        set exit_status 2
        log_and_exit
    }
}
...

If in f.i. gdb.exp we undo this dejagnu change:
...
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index 9a0620a2bf..bbb9988d9a 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -7200,5 +7200,8 @@ proc hex_in_list { val hexlist } {
     return [expr $index != -1]
 }

+rename ::unknown ::dejagnu_unknown
+rename ::tcl_unknown ::unknown
+
 # Always load compatibility stuff.
 load_lib future.exp
...
we have instead:
...
Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for
target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using /data/gdb_versions/devel/src/gdb/testsuite/config/unix.exp as
tool-and-target-specific interface file.
Running /data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp
...
ERROR: tcl error sourcing
/data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp.
ERROR: invalid command name "foobar"
    while executing
"foobar"
    (file
"/data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp" line
33)
    invoked from within
"source /data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 source
/data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp"
    invoked from within
"catch "uplevel #0 source $test_file_name""
Running
/data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/access_tagged_param.exp ...
Running
/data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/access_to_packed_array.exp
...
Running
/data/gdb_versions/devel/src/gdb/testsuite/gdb.ada/access_to_unbounded_array.exp
...
  ...
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/26110] invalid command name in one test-case aborts entire test run

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26110

--- Comment #2 from Tom de Vries <vries at gcc dot gnu.org> ---
This is a more specific fix:
...
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index 9a0620a2bf..4e5c43e3c9 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -7200,5 +7200,29 @@ proc hex_in_list { val hexlist } {
     return [expr $index != -1]
 }

+rename ::unknown ::dejagnu_unknown
+proc unknown { args } {
+    set script [info script]
+
+    set script [file dirname $script]
+    set subdir3 [file tail $script]
+
+    set script [file dirname $script]
+    set subdir2 [file tail $script]
+
+    set script [file dirname $script]
+    set subdir1 [file tail $script]
+
+    if { $subdir1 == "gdb"
+        && $subdir2 == "testsuite"
+        && [regexp {^gdb[.]} $subdir3] } {
+       # If we're executing a gdb test-case, skip dejagnu_unknown to prevent
+       # it from exiting and aborting the entire test run.
+       return [uplevel 1 ::tcl_unknown $args]
+    }
+
+    return return [uplevel 1 ::dejagnu_unknown $args]
+}
+
 # Always load compatibility stuff.
 load_lib future.exp
...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/26110] invalid command name in one test-case aborts entire test run

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26110

--- Comment #3 from Tom de Vries <vries at gcc dot gnu.org> ---
https://sourceware.org/pipermail/gdb-patches/2020-June/169404.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/26110] invalid command name in one test-case aborts entire test run

[cvs-commit at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26110

--- Comment #4 from cvs-commit at gcc dot gnu.org <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Tom de Vries <vries@sourceware.org>:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=26783bce15adc0316f9167a216519cebcf1ccac3

commit 26783bce15adc0316f9167a216519cebcf1ccac3
Author: Tom de Vries <tdevries@suse.de>
Date:   Fri Jun 12 09:13:17 2020 +0200

    [gdb/testsuite] Don't abort testrun for invalid command in test-case

    Say we add a call to foobar at the end of a test-case, and run the
    test-suite.  We'll run into a dejagnu error:
    ...
    ERROR: (DejaGnu) proc "foobar" does not exist.
    ...
    and the test-suite run is aborted.

    It's reasonable that the test-case is aborted, but it's not reasonable that
    the testsuite run is aborted.

    Problems in one test-case should not leak into other test-cases, and they
    generally don't.  The exception is the "invalid command name" problem due
to
    an override of ::unknown in dejagnu's framework.exp.

    Fix this by reverting dejagnu's ::unknown override for the duration of each
    test-case, using the gdb_init/gdb_finish hooks.

    Tested on x86_64-linux.

    gdb/testsuite/ChangeLog:

    2020-06-12  Tom de Vries  <tdevries@suse.de>

            PR testsuite/26110
            * lib/gdb.exp (gdb_init): Revert dejagnu's override of ::unknown.
            (gdb_finish): Reinstall dejagnu's override of ::unknown.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug testsuite/26110] invalid command name in one test-case aborts entire test run

[vries at gcc dot gnu.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=26110

Tom de Vries <vries at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
   Target Milestone|---                         |10.1
         Resolution|---                         |FIXED

--- Comment #5 from Tom de Vries <vries at gcc dot gnu.org> ---
Patch adding workaround to test-suite committed, marking resolved-fixed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.