[PATCH] testsuite, jit: Stabilize error output.

[PATCH] testsuite, jit: Stabilize error output.

[Iain Sandoe]

Tested on x86_64, i686 Darwin, x86_64 Linux,
OK for trunk? When?
thanks
Iain

--- 8< ---

Currently when a test fails, we print out a lot of information,
this includes items that are not stable between invocations (e.g.
the PID for the executable).  That makes automated comparisons
between test runs flag any persistent fails as new ones each time
which is not usually what is wanted.

This patch amends the error output to drop the variable portion
of the message and retain items that should only change if the
failure mode changes.

gcc/testsuite/ChangeLog:

	* jit.dg/jit.exp: Filter error output to remove per-run
	variable content.

Signed-off-by: Iain Sandoe <iain@sandoe.co.uk>
---
 gcc/testsuite/jit.dg/jit.exp | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/gcc/testsuite/jit.dg/jit.exp b/gcc/testsuite/jit.dg/jit.exp
index 286cfa8192a..893ff5f6dd0 100644
--- a/gcc/testsuite/jit.dg/jit.exp
+++ b/gcc/testsuite/jit.dg/jit.exp
@@ -94,25 +94,34 @@ proc parse_valgrind_logfile {name logfile} {
 # unexpected exits.
 
 proc verify_exit_status { executable wres } {
-    lassign $wres pid spawnid os_error_flag value
+    set extra [lassign $wres pid spawnid os_error_flag value]
     verbose "pid: $pid" 3
     verbose "spawnid: $spawnid" 3
     verbose "os_error_flag: $os_error_flag" 3
     verbose "value: $value" 3
 
     # Detect segfaults etc:
-    if { [llength $wres] > 4 } {
-	if { [lindex $wres 4] == "CHILDKILLED" } {
-	    fail "$executable killed: $wres"
+    set len [llength $extra]
+    if { $len >= 1 } {
+	if { [lindex $extra 0] == "CHILDKILLED" } {
+	    set reason "Unknown Reason"
+	    set detail "No Details"
+	    if { $len >= 2 } {
+		set reason [lindex $extra 1]
+		if { $len >= 3 } {
+		    set detail [lindex $extra 2]
+		}
+	    }
+	    fail "$executable killed: $reason $detail"
 	    return
 	}
     }
     if { $os_error_flag != 0 } {
-	fail "$executable: OS error: $wres"
+	fail "$executable: OS error: $os_error_flag $extra"
 	return
     }
     if { $value != 0 } {
-	fail "$executable: non-zero exit code: $wres"
+	fail "$executable: non-zero exit code: $value $extra"
 	return
     }
     pass "$executable exited cleanly"
-- 
2.39.2 (Apple Git-143)

Re: [PATCH] testsuite, jit: Stabilize error output.

[Iain Sandoe]

gentle ping,
with the increasing use of CI, it seems an idea to tackle this sooner rather than later.
thanks
Iain

> On 16 Jan 2024, at 11:12, Iain Sandoe <iains.gcc@gmail.com> wrote:
> 
> Tested on x86_64, i686 Darwin, x86_64 Linux,
> OK for trunk? When?
> thanks
> Iain
> 
> --- 8< ---
> 
> Currently when a test fails, we print out a lot of information,
> this includes items that are not stable between invocations (e.g.
> the PID for the executable).  That makes automated comparisons
> between test runs flag any persistent fails as new ones each time
> which is not usually what is wanted.
> 
> This patch amends the error output to drop the variable portion
> of the message and retain items that should only change if the
> failure mode changes.
> 
> gcc/testsuite/ChangeLog:
> 
> 	* jit.dg/jit.exp: Filter error output to remove per-run
> 	variable content.
> 
> Signed-off-by: Iain Sandoe <iain@sandoe.co.uk>
> ---
> gcc/testsuite/jit.dg/jit.exp | 21 +++++++++++++++------
> 1 file changed, 15 insertions(+), 6 deletions(-)
> 
> diff --git a/gcc/testsuite/jit.dg/jit.exp b/gcc/testsuite/jit.dg/jit.exp
> index 286cfa8192a..893ff5f6dd0 100644
> --- a/gcc/testsuite/jit.dg/jit.exp
> +++ b/gcc/testsuite/jit.dg/jit.exp
> @@ -94,25 +94,34 @@ proc parse_valgrind_logfile {name logfile} {
> # unexpected exits.
> 
> proc verify_exit_status { executable wres } {
> -    lassign $wres pid spawnid os_error_flag value
> +    set extra [lassign $wres pid spawnid os_error_flag value]
>     verbose "pid: $pid" 3
>     verbose "spawnid: $spawnid" 3
>     verbose "os_error_flag: $os_error_flag" 3
>     verbose "value: $value" 3
> 
>     # Detect segfaults etc:
> -    if { [llength $wres] > 4 } {
> -	if { [lindex $wres 4] == "CHILDKILLED" } {
> -	    fail "$executable killed: $wres"
> +    set len [llength $extra]
> +    if { $len >= 1 } {
> +	if { [lindex $extra 0] == "CHILDKILLED" } {
> +	    set reason "Unknown Reason"
> +	    set detail "No Details"
> +	    if { $len >= 2 } {
> +		set reason [lindex $extra 1]
> +		if { $len >= 3 } {
> +		    set detail [lindex $extra 2]
> +		}
> +	    }
> +	    fail "$executable killed: $reason $detail"
> 	    return
> 	}
>     }
>     if { $os_error_flag != 0 } {
> -	fail "$executable: OS error: $wres"
> +	fail "$executable: OS error: $os_error_flag $extra"
> 	return
>     }
>     if { $value != 0 } {
> -	fail "$executable: non-zero exit code: $wres"
> +	fail "$executable: non-zero exit code: $value $extra"
> 	return
>     }
>     pass "$executable exited cleanly"
> -- 
> 2.39.2 (Apple Git-143)
> 

Re: [PATCH] testsuite, jit: Stabilize error output.

[David Malcolm]

On Tue, 2024-01-16 at 11:12 +0000, Iain Sandoe wrote:
> Tested on x86_64, i686 Darwin, x86_64 Linux,
> OK for trunk? When?
> thanks
> Iain

Thanks; looks good to me for trunk.

Given that the scope is just the jit testsuite and that you've tested
it on 3 configurations (and presumably made use of this when debugging
the other issue), I think this is OK to go in now.

Dave

> 
> --- 8< ---
> 
> Currently when a test fails, we print out a lot of information,
> this includes items that are not stable between invocations (e.g.
> the PID for the executable).  That makes automated comparisons
> between test runs flag any persistent fails as new ones each time
> which is not usually what is wanted.
> 
> This patch amends the error output to drop the variable portion
> of the message and retain items that should only change if the
> failure mode changes.
> 
> gcc/testsuite/ChangeLog:
> 
>         * jit.dg/jit.exp: Filter error output to remove per-run
>         variable content.
> 
> Signed-off-by: Iain Sandoe <iain@sandoe.co.uk>
> ---
>  gcc/testsuite/jit.dg/jit.exp | 21 +++++++++++++++------
>  1 file changed, 15 insertions(+), 6 deletions(-)
> 
> diff --git a/gcc/testsuite/jit.dg/jit.exp
> b/gcc/testsuite/jit.dg/jit.exp
> index 286cfa8192a..893ff5f6dd0 100644
> --- a/gcc/testsuite/jit.dg/jit.exp
> +++ b/gcc/testsuite/jit.dg/jit.exp
> @@ -94,25 +94,34 @@ proc parse_valgrind_logfile {name logfile} {
>  # unexpected exits.
>  
>  proc verify_exit_status { executable wres } {
> -    lassign $wres pid spawnid os_error_flag value
> +    set extra [lassign $wres pid spawnid os_error_flag value]
>      verbose "pid: $pid" 3
>      verbose "spawnid: $spawnid" 3
>      verbose "os_error_flag: $os_error_flag" 3
>      verbose "value: $value" 3
>  
>      # Detect segfaults etc:
> -    if { [llength $wres] > 4 } {
> -       if { [lindex $wres 4] == "CHILDKILLED" } {
> -           fail "$executable killed: $wres"
> +    set len [llength $extra]
> +    if { $len >= 1 } {
> +       if { [lindex $extra 0] == "CHILDKILLED" } {
> +           set reason "Unknown Reason"
> +           set detail "No Details"
> +           if { $len >= 2 } {
> +               set reason [lindex $extra 1]
> +               if { $len >= 3 } {
> +                   set detail [lindex $extra 2]
> +               }
> +           }
> +           fail "$executable killed: $reason $detail"
>             return
>         }
>      }
>      if { $os_error_flag != 0 } {
> -       fail "$executable: OS error: $wres"
> +       fail "$executable: OS error: $os_error_flag $extra"
>         return
>      }
>      if { $value != 0 } {
> -       fail "$executable: non-zero exit code: $wres"
> +       fail "$executable: non-zero exit code: $value $extra"
>         return
>      }
>      pass "$executable exited cleanly"