public inbox for gcc-patches@gcc.gnu.org
 help / color / mirror / Atom feed
From: David Malcolm <dmalcolm@redhat.com>
To: gcc-patches@gcc.gnu.org, Dimitar Dimitrov <dimitar@dinux.eu>
Cc: David Malcolm <dmalcolm@redhat.com>
Subject: [pushed] testsuite: use check-jsonschema for validating .sarif files [PR109360]
Date: Tue, 25 Jun 2024 20:31:31 -0400	[thread overview]
Message-ID: <20240626003131.1428584-1-dmalcolm@redhat.com> (raw)
In-Reply-To: <5ced30f68e42da0bf2f975a17ba3ca32c92654d5.camel@redhat.com>

As reported here:
  https://gcc.gnu.org/pipermail/gcc-patches/2024-June/655434.html
the schema validation I added for generated .sarif files in
r15-1541-ga84fe222029ff2 used the "jsonschema" command line tool, which
has been deprecated by more recent versions of the Python 3 "jsonschema"
module.

This patch updates the validation to use the more recent
"check-jsonschema" command line tool, from the Python 3 "check-jsonschema"
module, fixing the testsuite FAILs due to the deprecation message.

As an added bonus, the output on validation failures is *much* nicer, e.g.
if I undo r15-1540-g9f4fdc3acebcf6, the error messages begin like this:
verify-sarif-file: res: Schema validation errors were encountered.
  diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].locations[0].physicalLocation.region.startColumn: 0 is less than the minimum of 1
  diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].relatedLocations[0].physicalLocation.region.startColumn: 0 is less than the minimum of 1
  diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].relatedLocations[1].physicalLocation.region.startColumn: 0 is less than the minimum of 1
  diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].relatedLocations[2].physicalLocation.region.startColumn: 0 is less than the minimum of 1
child process exited abnormally
FAIL: c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-1.c  -Wc++-compat   (test .sarif output against SARIF schema)

Tested with Python 3.8 with check_jsonschema 0.28.6

Successfully bootstrapped & regrtested on x86_64-pc-linux-gnu.
Successful run of analyzer integration tests on x86_64-pc-linux-gnu.
Pushed to trunk as r15-1633-g17967907102099.

gcc/ChangeLog:
	PR testsuite/109360
	* doc/install.texi (Python3 modules): Update SARIF validation
	requirement to use check-jsonschema rather than jsonschema.

gcc/testsuite/ChangeLog:
	PR testsuite/109360
	* lib/scansarif.exp (verify-sarif-file): Use check-jsonschema
	rather than jsonschema, updating the invocation accordingly.
	* lib/target-supports.exp (check_effective_target_jsonschema): Convert
	to...
	(check_effective_target_check_jsonschema): ...this.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>
---
 gcc/doc/install.texi                  | 7 ++++---
 gcc/testsuite/lib/scansarif.exp       | 8 ++++----
 gcc/testsuite/lib/target-supports.exp | 6 +++---
 3 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index 0c7691651466..b54569925837 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -461,9 +461,10 @@ is shown below:
 @code{gcov}, @code{gzip}, @code{json}, @code{os} and @code{pytest}.
 
 @item SARIF testsuite
-Tests of SARIF output will use the @code{jsonschema} program from the
-@code{jsonschema} module (if available) to validate generated .sarif files.
-If this tool is not found, the validation parts of those tests are skipped.
+Tests of SARIF output will use the @code{check-jsonschema} program from
+the @code{check-jsonschema} module (if available) to validate generated
+.sarif files.  If this tool is not found, the validation parts of those
+tests are skipped.
 
 @item c++ cxx api generation
 @code{csv}, @code{os}, @code{sys} and @code{time}.
diff --git a/gcc/testsuite/lib/scansarif.exp b/gcc/testsuite/lib/scansarif.exp
index 3eb38b8102e9..cc0890ef5d8b 100644
--- a/gcc/testsuite/lib/scansarif.exp
+++ b/gcc/testsuite/lib/scansarif.exp
@@ -57,7 +57,7 @@ proc scan-sarif-file-not { args } {
 # Assuming python3 is available, use verify-sarif-file.py to check
 # that the .sarif file is UTF-8 encoded and is parseable as JSON.
 #
-# Assuming "jsonschema" is available, use it to verify that the .sarif
+# Assuming "check-jsonschema" is available, use it to verify that the .sarif
 # file complies with the SARIF schema.
 
 proc verify-sarif-file { args } {
@@ -86,8 +86,8 @@ proc verify-sarif-file { args } {
     # Verify that the file complies with the SARIF schema.
 
     # Check that jsonschema is installed.
-    if { ![check_effective_target_jsonschema] } {
-	unsupported "$testcase verify-sarif-file: jsonschema is missing"
+    if { ![check_effective_target_check_jsonschema] } {
+	unsupported "$testcase verify-sarif-file: check-jsonschema is missing"
 	return	
     }
 
@@ -95,7 +95,7 @@ proc verify-sarif-file { args } {
     verbose "schema_file: $schema_file" 2
 
     set what "$testcase (test .sarif output against SARIF schema)"
-    if [catch {exec jsonschema --instance $output_file $schema_file} res ] {
+    if [catch {exec check-jsonschema --schemafile $schema_file $output_file} res ] {
 	verbose "verify-sarif-file: res: $res" 2
 	fail "$what"
 	return
diff --git a/gcc/testsuite/lib/target-supports.exp b/gcc/testsuite/lib/target-supports.exp
index ed30cd18ad69..828c46fb15e5 100644
--- a/gcc/testsuite/lib/target-supports.exp
+++ b/gcc/testsuite/lib/target-supports.exp
@@ -13506,10 +13506,10 @@ proc check_effective_target_heap_trampoline {} {
     return 0
 }
 
-# Return 1 if jsonschema is available.
+# Return 1 if check-jsonschema is available.
 
-proc check_effective_target_jsonschema { } {
-    set result [remote_exec host "jsonschema --version"]
+proc check_effective_target_check_jsonschema { } {
+    set result [remote_exec host "check-jsonschema --version"]
     set status [lindex $result 0]
     if { $status == 0 } then {
         return 1;
-- 
2.26.3


      reply	other threads:[~2024-06-26  0:31 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-06-21 12:55 [pushed 1/2] diagnostics: fixes to SARIF output [PR109360] David Malcolm
2024-06-21 12:55 ` [pushed 2/2] testsuite: check that generated .sarif files validate against the SARIF schema [PR109360] David Malcolm
2024-06-22  8:26   ` Dimitar Dimitrov
2024-06-24 22:27     ` David Malcolm
2024-06-26  0:31       ` David Malcolm [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240626003131.1428584-1-dmalcolm@redhat.com \
    --to=dmalcolm@redhat.com \
    --cc=dimitar@dinux.eu \
    --cc=gcc-patches@gcc.gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).