[PATCH] jit: Fix random truncation of testsuite output

[PATCH] jit: Fix random truncation of testsuite output

[Alex Coplan]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

Hello,

This patch fixes a bug in jit.exp which causes the DejaGnu output of the
libgccjit testsuite to be nondeterministically truncated. This bug was
copied from DejaGnu's own implementation of the host_execute function.
See the upstream bug report [0] where the maintainers point out that the
regex patterns in host_execute should (but don't currently) explicitly
match newlines to avoid relying on DejaGnu not reading more than one
line of the output (which is not guaranteed).

To reproduce the bug, run:

$ make check-jit RUNTESTFLAGS="jit.exp=test-arith-overflow.c"
$ grep -v iteration testsuite/jit/jit.sum

and you should see some lines that have been truncated (I see the word
iteration partially or fully truncated). Alternatively, simply run the
testsuite twice (saving a copy of testsuite/jit/jit.sum from the first
run) and diff the two jit.sum files to observe the random truncations to
the output.

This patch should make it easier to test jit patches in the future,
since it makes it possible to reliably compare the output of two jit.sum
files (as with the other tests in GCC).

Testing:
 * Ran the testsuite before and after the patch, observing that the only
   differences in jit.sum were in test-threads.c (nondeterministic test)
   and where the truncated output from the first run was no longer
   truncated.
 * Ran the testsuite twice after the patch, observing that the only
   differences in jit.sum between the two runs were in test-threads.c.

OK for master?

Thanks,
Alex

---

2020-07-20  Alex Coplan  <alex.coplan@arm.com>

gcc/testsuite/ChangeLog:

	* jit.dg/jit.exp (fixed_host_execute): Fix regex patterns to
	always explicitly match newlines.


[0] : https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42399

[-- Attachment #2: patch.txt --]
[-- Type: text/plain, Size: 1860 bytes --]

diff --git a/gcc/testsuite/jit.dg/jit.exp b/gcc/testsuite/jit.dg/jit.exp
index 2f54681713b..2d8c884b6b8 100644
--- a/gcc/testsuite/jit.dg/jit.exp
+++ b/gcc/testsuite/jit.dg/jit.exp
@@ -202,37 +202,37 @@ proc fixed_host_execute {args} {
 	    set timetol 0
 	    exp_continue
 	}
-	-re "^$prefix\tNOTE:${text}*" {
+	-re "^$prefix\tNOTE:\[^\r\n\]+\r\n" {
 	    regsub "\[\n\r\t\]*NOTE: $text\r\n" $expect_out(0,string) "" output
-	    set output [string range $output 6 end]
+	    set output [string range $output 6 end-2]
 	    verbose "$output" 2
 	    set timetol 0
 	    exp_continue
 	}
-	-re "^$prefix\tPASSED:${text}*" {
+	-re "^$prefix\tPASSED:\[^\r\n\]+\r\n" {
 	    regsub "\[\n\r\t\]*PASSED: $text\r\n" $expect_out(0,string) "" output
-	    set output [string range $output 8 end]
+	    set output [string range $output 8 end-2]
 	    pass "$output"
 	    set timetol 0
 	    exp_continue
 	}
-	-re "^$prefix\tFAILED:${text}*" {
+	-re "^$prefix\tFAILED:\[^\r\n\]+\r\n" {
 	    regsub "\[\n\r\t\]*FAILED: $text\r\n" $expect_out(0,string) "" output
-	    set output [string range $output 8 end]
+	    set output [string range $output 8 end-2]
 	    fail "$output"
 	    set timetol 0
 	    exp_continue
 	}
-	-re "^$prefix\tUNTESTED:${text}*" {
+	-re "^$prefix\tUNTESTED:\[^\r\n\]+\r\n" {
 	    regsub "\[\n\r\t\]*TESTED: $text\r\n" $expect_out(0,string) "" output
-	    set output [string range $output 8 end]
+	    set output [string range $output 8 end-2]
 	    untested "$output"
 	    set timetol 0
 	    exp_continue
 	}
-	-re "^$prefix\tUNRESOLVED:${text}*" {
+	-re "^$prefix\tUNRESOLVED:\[^\r\n\]+\r\n" {
 	    regsub "\[\n\r\t\]*UNRESOLVED: $text\r\n" $expect_out(0,string) "" output
-	    set output [string range $output 8 end]
+	    set output [string range $output 8 end-2]
 	    unresolved "$output"
 	    set timetol 0
 	    exp_continue

Re: [PATCH] jit: Fix random truncation of testsuite output

[David Malcolm]

On Mon, 2020-07-20 at 12:10 +0100, Alex Coplan wrote:
> Hello,
> 
> This patch fixes a bug in jit.exp which causes the DejaGnu output of
> the
> libgccjit testsuite to be nondeterministically truncated. This bug
> was
> copied from DejaGnu's own implementation of the host_execute
> function.
> See the upstream bug report [0] where the maintainers point out that
> the
> regex patterns in host_execute should (but don't currently)
> explicitly
> match newlines to avoid relying on DejaGnu not reading more than one
> line of the output (which is not guaranteed).
> 
> To reproduce the bug, run:
> 
> $ make check-jit RUNTESTFLAGS="jit.exp=test-arith-overflow.c"
> $ grep -v iteration testsuite/jit/jit.sum
> 
> and you should see some lines that have been truncated (I see the
> word
> iteration partially or fully truncated). Alternatively, simply run
> the
> testsuite twice (saving a copy of testsuite/jit/jit.sum from the
> first
> run) and diff the two jit.sum files to observe the random truncations
> to
> the output.
> 
> This patch should make it easier to test jit patches in the future,
> since it makes it possible to reliably compare the output of two
> jit.sum
> files (as with the other tests in GCC).
> 
> Testing:
>  * Ran the testsuite before and after the patch, observing that the
> only
>    differences in jit.sum were in test-threads.c (nondeterministic
> test)
>    and where the truncated output from the first run was no longer
>    truncated.
>  * Ran the testsuite twice after the patch, observing that the only
>    differences in jit.sum between the two runs were in test-
> threads.c.
> 
> OK for master?
> 
> Thanks,
> Alex
> 
> ---
> 
> 2020-07-20  Alex Coplan  <alex.coplan@arm.com>
> 
> gcc/testsuite/ChangeLog:
> 
> 	* jit.dg/jit.exp (fixed_host_execute): Fix regex patterns to
> 	always explicitly match newlines.
> 
> 
> [0] : https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42399
`
Thanks for chasing this up. This looks a lot like the issues being
tracked in PR jit/69435, so please add that to the ChangeLog when
committing.

OK for master.
Dave