[PATCH] testsuite: always use UTF-8 in scan-sarif-file[-not] [PR105959]

[PATCH] testsuite: always use UTF-8 in scan-sarif-file[-not] [PR105959]

[David Malcolm]

c-c++-common/diagnostic-format-sarif-file-4.c is a test case for
quoting non-ASCII source code in a SARIF diagnostic log.

The SARIF standard mandates that .sarif files are UTF-8 encoded.

PR testsuite/105959 notes that the test case fails when the system
encoding is not UTF-8, such as when the "make" invocation is prefixed
with LC_ALL=C, whereas it works with in a UTF-8-locale.

The root cause is that dg-scan opens the file for reading using the
"system" encoding; I believe it is falling back to treating all files as
effectively ISO 8859-1 in a non-UTF-8 locale.

This patch fixes things by adding a mechanism to dg-scan to allow
callers to (optionally) specify an encoding to use when reading the
file, and updating scan-sarif-file (and the -not variant) to always
use UTF-8 when calling dg-scan, fixing the test case with LC_ALL=C.

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Manually verified the problematic testcases work with this
with both LC_ALL=C and LC_ALL=en_US.UTF-8

OK for trunk?

gcc/testsuite/ChangeLog:
	PR testsuite/105959
	* gcc.dg-selftests/dg-final.exp
	(dg_final_directive_check_num_args): Update expected maximum
	number of args for the various directives using dg-scan.
	* lib/scanasm.exp (append_encoding_arg): New procedure.
	(dg-scan): Add optional 3rd argument: the encoding to use when
	reading from the file.
	* lib/scansarif.exp (scan-sarif-file): Treat the file as UTF-8
	encoded when reading it.
	(scan-sarif-file-not): Likewise.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>
---
 gcc/testsuite/gcc.dg-selftests/dg-final.exp | 22 ++++++++++-----------
 gcc/testsuite/lib/scanasm.exp               | 22 +++++++++++++++++++--
 gcc/testsuite/lib/scansarif.exp             | 13 +++++++++++-
 3 files changed, 43 insertions(+), 14 deletions(-)

diff --git a/gcc/testsuite/gcc.dg-selftests/dg-final.exp b/gcc/testsuite/gcc.dg-selftests/dg-final.exp
index e75fb45f44a..e34f29ab13a 100644
--- a/gcc/testsuite/gcc.dg-selftests/dg-final.exp
+++ b/gcc/testsuite/gcc.dg-selftests/dg-final.exp
@@ -104,17 +104,17 @@ proc dg_final_directive_check_num_args {} {
 
     global testname_with_flags
     set testname_with_flags "test.c"
-    verify_args scan-assembler 1 2
-    verify_args scan-assembler-not 1 2
-    verify_args scan-hidden 1 2
-    verify_args scan-not-hidden 1 2
-    verify_args scan-file 2 3
-    verify_args scan-file-not 2 3
-    verify_args scan-stack-usage 1 2
-    verify_args scan-stack-usage-not 1 2
-    verify_args scan-ada-spec 1 2
-    verify_args scan-ada-spec-not 1 2
-    verify_args scan-lto-assembler 1 2
+    verify_args scan-assembler 1 3
+    verify_args scan-assembler-not 1 3
+    verify_args scan-hidden 1 3
+    verify_args scan-not-hidden 1 3
+    verify_args scan-file 2 4
+    verify_args scan-file-not 2 5
+    verify_args scan-stack-usage 1 3
+    verify_args scan-stack-usage-not 1 3
+    verify_args scan-ada-spec 1 3
+    verify_args scan-ada-spec-not 1 3
+    verify_args scan-lto-assembler 1 3
     unset testname_with_flags
 }
 
diff --git a/gcc/testsuite/lib/scanasm.exp b/gcc/testsuite/lib/scanasm.exp
index 4b018abcf3d..fb53544d40c 100644
--- a/gcc/testsuite/lib/scanasm.exp
+++ b/gcc/testsuite/lib/scanasm.exp
@@ -24,19 +24,33 @@ proc make_pattern_printable { pattern } {
     return [string map {\t \\t \n \\n \r \\r \\ \\\\} $pattern]
 }
 
+# Append to ARGS to make it suitable for use by dg-scan to indicate
+# that encoding ENC should be used when reading from the file.
+
+proc append_encoding_arg { args enc } {
+    if { [llength $args] < 2 } {
+	# Add target selector.
+	lappend args { target "*-*-*" }
+    }
+    # Add encoding ENC.
+    lappend args $enc
+    return $args
+}
+
 # Scan the OUTPUT_FILE for a pattern.  If it is present and POSITIVE
 # is non-zero, or it is not present and POSITIVE is zero, the test
 # passes.  The ORIG_ARGS is the list of arguments provided by dg-final
 # to scan-assembler.  The first element in ORIG_ARGS is the regular
 # expression to look for in the file.  The second element, if present,
-# is a DejaGNU target selector.
+# is a DejaGNU target selector.  The third element, if present, is the
+# encoding to use when reading from the file.
 
 proc dg-scan { name positive testcase output_file orig_args } {
     if { [llength $orig_args] < 1 } {
 	error "$name: too few arguments"
         return
     }
-    if { [llength $orig_args] > 2 } {
+    if { [llength $orig_args] > 3 } {
 	error "$name: too many arguments"
 	return
     }
@@ -59,6 +73,10 @@ proc dg-scan { name positive testcase output_file orig_args } {
 	return
     }
     set fd [open $output_file r]
+    if { [llength $orig_args] >= 3 } {
+	set file_encoding [lindex $orig_args 2]
+	fconfigure $fd -encoding $file_encoding
+    }
     set text [read $fd]
     close $fd
 
diff --git a/gcc/testsuite/lib/scansarif.exp b/gcc/testsuite/lib/scansarif.exp
index d06390b854f..05524aa6e7a 100644
--- a/gcc/testsuite/lib/scansarif.exp
+++ b/gcc/testsuite/lib/scansarif.exp
@@ -17,7 +17,10 @@
 # Various utilities for scanning SARIF output, used by gcc-dg.exp and
 # g++-dg.exp.
 #
-# This is largely borrowed from scanasm.exp.
+# This is largely borrowed from scanasm.exp, but tweaked to force Tcl
+# to treat the file as UTF-8: section 3.1 of SARIF 2.1.0
+# ("File Format" > "General") specifies: "A SARIF log file SHALL be
+# encoded in UTF-8 [RFC3629])".
 
 # Look for a pattern in the .sarif file produced by the compiler.  See
 # dg-scan for details.
@@ -27,6 +30,10 @@ proc scan-sarif-file { args } {
     # The name might include a list of options; extract the file name.
     set filename [lindex $testcase 0]
     set output_file "[file tail $filename].sarif"
+
+    # Treat the file as UTF-8 encoded when reading it.
+    set args [append_encoding_arg $args "utf-8"]
+
     dg-scan "scan-sarif-file" 1 $testcase $output_file $args
 }
 
@@ -38,5 +45,9 @@ proc scan-sarif-file-not { args } {
     # The name might include a list of options; extract the file name.
     set filename [lindex $testcase 0]
     set output_file "[file tail $filename].sarif"
+
+    # Treat the file as UTF-8 encoded when reading it.
+    set args [append_encoding_arg $args "utf-8"]
+
     dg-scan "scan-sarif-file-not" 0 $testcase $output_file $args
 }
-- 
2.26.3

Re: [PATCH] testsuite: always use UTF-8 in scan-sarif-file[-not] [PR105959]

[Mike Stump]

On Mar 20, 2023, at 3:06 PM, David Malcolm via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
> 
> c-c++-common/diagnostic-format-sarif-file-4.c is a test case for
> quoting non-ASCII source code in a SARIF diagnostic log.
> 
> The SARIF standard mandates that .sarif files are UTF-8 encoded.
> 
> PR testsuite/105959 notes that the test case fails when the system
> encoding is not UTF-8, such as when the "make" invocation is prefixed
> with LC_ALL=C, whereas it works with in a UTF-8-locale.
> 
> The root cause is that dg-scan opens the file for reading using the
> "system" encoding; I believe it is falling back to treating all files as
> effectively ISO 8859-1 in a non-UTF-8 locale.
> 
> This patch fixes things by adding a mechanism to dg-scan to allow
> callers to (optionally) specify an encoding to use when reading the
> file, and updating scan-sarif-file (and the -not variant) to always
> use UTF-8 when calling dg-scan, fixing the test case with LC_ALL=C.

> OK for trunk?

Ok.