[RFC][gdb/testsuite] Add gdbfail/envfail aliases for kfail/xfail

[RFC][gdb/testsuite] Add gdbfail/envfail aliases for kfail/xfail

[Tom de Vries]

Hi,

The difference between an xfail and a kfail (in the context of gdb testing) is
that:
- a kfail marks a problem in gdb
- an xfail marks a problem in the environment (compiler, libc, kernel)

However, this difference is not immediately obvious from the contrast between
"known" and "expected".

Add aliases gdbfail for kfail, and envfail for xfail, that are more intuitive
to use.

Tested on x86_64-linux.

Any comments?

Thanks,
- Tom

[gdb/testsuite] Add gdbfail/envfail aliases for kfail/xfail

gdb/testsuite/ChangeLog:

2020-04-30  Tom de Vries  <tdevries@suse.de>

	* lib/gdb.exp (gdbfail, envfail, setup_gdbfail, setup_envfail): New
	proc.
	* gdb.ada/packed_tagged.exp: Fix incorrectly marked kfails, by using
	envfail.

---
 gdb/testsuite/gdb.ada/packed_tagged.exp | 12 ++++++------
 gdb/testsuite/lib/gdb.exp               | 26 ++++++++++++++++++++++++++
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/gdb/testsuite/gdb.ada/packed_tagged.exp b/gdb/testsuite/gdb.ada/packed_tagged.exp
index d6ee8454c5a..1e965cffda1 100644
--- a/gdb/testsuite/gdb.ada/packed_tagged.exp
+++ b/gdb/testsuite/gdb.ada/packed_tagged.exp
@@ -38,16 +38,16 @@ foreach_with_prefix scenario {all minimal} {
     set pass_re \
 	"= \\(exists => true, value => 10\\)"
     # There is a compiler bug that causes this output.
-    set kfail_re \
+    set envfail_re \
 	"= \\(exists => true\\)"
 
     gdb_test_multiple "print x" "" {
 	-re -wrap $pass_re {
 	    pass $gdb_test_name
 	}
-	-re -wrap $kfail_re {
+	-re -wrap $envfail_re {
 	    if {$scenario == "minimal"} {
-		setup_kfail "gnat compiler bug" *-*-*
+		setup_envfail "gnat compiler bug" *-*-*
 	    }
 	    fail $gdb_test_name
 	}
@@ -63,7 +63,7 @@ foreach_with_prefix scenario {all minimal} {
 	     "    end case;" \
 	     "end record" ]
     # There is a compiler bug that causes this output.
-    set kfail_re \
+    set envfail_re \
 	[multi_line "type = record" \
 	     "    exists: (boolean|range false \\.\\. true);" \
 	     "    case \\? is" \
@@ -77,9 +77,9 @@ foreach_with_prefix scenario {all minimal} {
 	-re -wrap $pass_re {
 	    pass $gdb_test_name
 	}
-	-re -wrap $kfail_re {
+	-re -wrap $envfail_re {
 	    if {$scenario == "minimal"} {
-		setup_kfail "gnat compiler bug" *-*-*
+		setup_envfail "gnat compiler bug" *-*-*
 	    }
 	    fail $gdb_test_name
 	}
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index b72ce0cda7f..00efe280a43 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -7124,5 +7124,31 @@ proc debug_types { } {
     return 0
 }
 
+# Alias for kfail, to make the distinction between kfail and xfail clearer.
+
+proc gdbfail { args } {
+    kfail {*}$args
+}
+
+# Alias for xfail, to make the distinction between kfail and xfail clearer.
+
+proc envfail { args } {
+    xfail {*}$args
+}
+
+# Alias for setup_kfail, to make the distinction between kfail and xfail
+# clearer.
+
+proc setup_gdbfail { args } {
+    setup_kfail {*}$args
+}
+
+# Alias for setup_xfail, to make the distinction between kfail and xfail
+# clearer.
+
+proc setup_envfail { args } {
+    setup_xfail {*}$args
+}
+
 # Always load compatibility stuff.
 load_lib future.exp

Re: [RFC][gdb/testsuite] Add gdbfail/envfail aliases for kfail/xfail

[Pedro Alves]

On 4/30/20 7:43 AM, Tom de Vries wrote:
> Hi,
> 
> The difference between an xfail and a kfail (in the context of gdb testing) is
> that:
> - a kfail marks a problem in gdb
> - an xfail marks a problem in the environment (compiler, libc, kernel)
> 
> However, this difference is not immediately obvious from the contrast between
> "known" and "expected".

I memorized the the X in xfail as "external", not "expected".
So I always think of them as "known" vs "external", which makes the
difference much more obvious, IMO.

I propose that the documentation references the "external" mnemonic to help.

KFAIL and XFAIL are documented in testsuite/README:

 XFAIL

 Short for "expected failure", this indicates a known problem with the
 environment.  This could include limitations of the operating system,
 compiler version, and other components.

Maybe some new entry on KFAIL/XFAIL at:

 https://sourceware.org/gdb/wiki/GDBTestcaseCookbook

would help too.

> 
> Add aliases gdbfail for kfail, and envfail for xfail, that are more intuitive
> to use.
> 
> Tested on x86_64-linux.
> 
> Any comments?

I think this isn't a good idea:

 - We still end up with XFAIL / KFAIL in result logs anyway.  People
   still need to know the difference.

 - It adds different ways to do the same thing.  I.e., more things
   to learn, when you still need to know about the old way anyway.

Thanks,
Pedro Alves