[PATCH] testsuite/lib/multline.exp: show test name and line numbers

[PATCH] testsuite/lib/multline.exp: show test name and line numbers

[David Malcolm]

The output from multiline.exp doesn't scale well when reviewing
and comparing .sum files.

Lines in e.g. gcc.sum from a dg-{begin|end}-multiline-output pair are
currently of the form:
 PASS: expected multiline pattern 0 was found: "\s*myvar = myvar\.x;.*\n           ~~~~~\^~\n"

as compared to e.g. this result line for a dg-warning:
 PASS: gcc.dg/plugin/diagnostic-test-show-locus-bw.c -fplugin=./diagnostic_plugin_test_show_locus.so  (test for warnings, line 198)

In particular the line doesn't identify the filename/options of the test,
which means in a .sum file with a FAIL of one of these we can't easily
locate the testcase of interest.

The following patch updates multiline.exp to use the global
  $testname_with_flags
as a prefix in such results.

I also dropped the printing of the index in favor of printing the line
numbers enclosed within dg-{begin|end}-multiline-output.

After the patch, we get result lines like this:
 PASS: gcc.dg/plugin/diagnostic-test-show-locus-bw.c -fplugin=./diagnostic_plugin_test_show_locus.so  expected multiline pattern lines 17-18 was found: "\s*myvar = myvar\.x;.*\n           ~~~~~\^~\n"

(note the presence of the source file and options and the "lines 17-18")

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu;
I compared the results against a control build (of r231445), and
the results were unchanged, other than the expected changes from
the above, leading to
- 92 PASS results changing name within g++.sum
- 7 PASS results changing name within each of obj-c++.sum
  and objc.sum, and
- 125 PASS results changing name within gcc.sum.

OK for trunk for gcc 6?

gcc/testsuite/ChangeLog:
	* lib/multiline.exp (_multiline_expected_outputs): Update comment.
	(dg-end-multiline-output): Capture line numbers within
	_multiline_expected_outputs.
	(handle-multiline-outputs): Access global $testname_with_flags
	and add it as a prefix to pass/fail results.  Extract line numbers
	from $_multiline_expected_outputs and print them within pass/fail
	results, replacing the printing of $index.  Consolidate the
	string prefix shared between pass/fail into a new local: $title.
---
 gcc/testsuite/lib/multiline.exp | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/gcc/testsuite/lib/multiline.exp b/gcc/testsuite/lib/multiline.exp
index c3d0506..5367437 100644
--- a/gcc/testsuite/lib/multiline.exp
+++ b/gcc/testsuite/lib/multiline.exp
@@ -54,7 +54,9 @@
 # The line number of the last dg-begin-multiline-output directive.
 set _multiline_last_beginning_line -1
 
-# A list of lists of strings.
+# A list of
+#   first-line-number, last-line-number, lines
+# where each "lines" is a list of strings.
 set _multiline_expected_outputs []
 
 ############################################################################
@@ -88,12 +90,15 @@ proc dg-end-multiline-output { args } {
     # Load it and split it into lines
 
     set lines [_get_lines $prog $_multiline_last_beginning_line $line]
-    set _multiline_last_beginning_line -1
 
     verbose "lines: $lines" 3
+    # Create an entry of the form:  first-line, last-line, lines
+    set entry [list $_multiline_last_beginning_line $line $lines]
     global _multiline_expected_outputs
-    lappend _multiline_expected_outputs $lines
+    lappend _multiline_expected_outputs $entry
     verbose "within dg-end-multiline-output: _multiline_expected_outputs: $_multiline_expected_outputs" 3
+
+    set _multiline_last_beginning_line -1
 }
 
 # Hook to be called by prune.exp's prune_gcc_output to
@@ -107,9 +112,14 @@ proc dg-end-multiline-output { args } {
 
 proc handle-multiline-outputs { text } {
     global _multiline_expected_outputs
+    global testname_with_flags
     set index 0
-    foreach multiline $_multiline_expected_outputs {
-	verbose "  multiline: $multiline" 4
+    foreach entry $_multiline_expected_outputs {
+	verbose "  entry: $entry" 3
+	set start_line [lindex $entry 0]
+	set end_line   [lindex $entry 1]
+	set multiline  [lindex $entry 2]
+	verbose "  multiline: $multiline" 3
 	set rexp [_build_multiline_regex $multiline $index]
 	verbose "rexp: ${rexp}" 4
 	# Escape newlines in $rexp so that we can print them in
@@ -117,12 +127,14 @@ proc handle-multiline-outputs { text } {
 	set escaped_regex [string map {"\n" "\\n"} $rexp]
 	verbose "escaped_regex: ${escaped_regex}" 4
 
+	set title "$testname_with_flags expected multiline pattern lines $start_line-$end_line"
+
 	# Use "regsub" to attempt to prune the pattern from $text
 	if {[regsub -line $rexp $text "" text]} {
 	    # Success; the multiline pattern was pruned.
-	    pass "expected multiline pattern $index was found: \"$escaped_regex\""
+	    pass "$title was found: \"$escaped_regex\""
 	} else {
-	    fail "expected multiline pattern $index not found: \"$escaped_regex\""
+	    fail "$title not found: \"$escaped_regex\""
 	}
 
 	set index [expr $index + 1]
-- 
1.8.5.3

Re: [PATCH] testsuite/lib/multline.exp: show test name and line numbers

[Bernd Schmidt]

On 12/10/2015 03:56 PM, David Malcolm wrote:
> The following patch updates multiline.exp to use the global
>    $testname_with_flags
> as a prefix in such results.
>
> I also dropped the printing of the index in favor of printing the line
> numbers enclosed within dg-{begin|end}-multiline-output.
>
> After the patch, we get result lines like this:
>   PASS: gcc.dg/plugin/diagnostic-test-show-locus-bw.c -fplugin=./diagnostic_plugin_test_show_locus.so  expected multiline pattern lines 17-18 was found: "\s*myvar = myvar\.x;.*\n           ~~~~~\^~\n"

This seems to be an improvement, so OK. I'm not sure what the value of 
printing the regexp in the output is, I tend to see it as clutter and 
would favour another patch to remove it.


Bernd

Re: [PATCH] testsuite/lib/multline.exp: show test name and line numbers

[Uros Bizjak]

On Thu, Jan 7, 2016 at 1:10 PM, Bernd Schmidt <bschmidt@redhat.com> wrote:
> On 12/29/2015 10:17 AM, Uros Bizjak wrote:
>>
>> It looks that this new functionality doesn't handle conditional
>> compilation, when
>>
>> /* { dg-do compile { target { ! ia32 } } } */
>>
>> is added to the testcase, such as in recently changed
>> gcc.target/i386/pr68473-1.c.
>>
>> The directive is passed to the next testcase, leading to spurious
>> testsuite failures [1] in unrelated testcases.
>
>
> Please open a PR if you haven't already. David, could you investigate?

I have opened PR 69181 [1].

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69181

Uros.

Re: [PATCH] testsuite/lib/multline.exp: show test name and line numbers

[Bernd Schmidt]

On 12/29/2015 10:17 AM, Uros Bizjak wrote:
> It looks that this new functionality doesn't handle conditional
> compilation, when
>
> /* { dg-do compile { target { ! ia32 } } } */
>
> is added to the testcase, such as in recently changed
> gcc.target/i386/pr68473-1.c.
>
> The directive is passed to the next testcase, leading to spurious
> testsuite failures [1] in unrelated testcases.

Please open a PR if you haven't already. David, could you investigate?


Bernd

Re: [PATCH] testsuite/lib/multline.exp: show test name and line numbers

[Uros Bizjak]

Hello!

> Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu;
> I compared the results against a control build (of r231445), and
> the results were unchanged, other than the expected changes from
> the above, leading to
> - 92 PASS results changing name within g++.sum
> - 7 PASS results changing name within each of obj-c++.sum
>   and objc.sum, and
> - 125 PASS results changing name within gcc.sum.
>
> OK for trunk for gcc 6?
>
> gcc/testsuite/ChangeLog:
> * lib/multiline.exp (_multiline_expected_outputs): Update comment.
> (dg-end-multiline-output): Capture line numbers within
> _multiline_expected_outputs.
> (handle-multiline-outputs): Access global $testname_with_flags
> and add it as a prefix to pass/fail results.  Extract line numbers
> from $_multiline_expected_outputs and print them within pass/fail
> results, replacing the printing of $index.  Consolidate the
> string prefix shared between pass/fail into a new local: $title.

It looks that this new functionality doesn't handle conditional
compilation, when

/* { dg-do compile { target { ! ia32 } } } */

is added to the testcase, such as in recently changed
gcc.target/i386/pr68473-1.c.

The directive is passed to the next testcase, leading to spurious
testsuite failures [1] in unrelated testcases.

[1] https://gcc.gnu.org/ml/gcc-testresults/2015-12/msg02761.html

Uros.