[Bug testsuite/109360] New: RFE: check that generated .sarif files validate against the SARIF schema

[Bug testsuite/109360] New: RFE: check that generated .sarif files validate against the SARIF schema

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109360

            Bug ID: 109360
           Summary: RFE: check that generated .sarif files validate
                    against the SARIF schema
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: testsuite
          Assignee: unassigned at gcc dot gnu.org
          Reporter: dmalcolm at gcc dot gnu.org
  Target Milestone: ---

Created attachment 54798
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=54798&action=edit
WIP progress patch

I have a partially-working patch to scansarif.exp that validates the .sarif
files generated by -fdiagnostics-format=sarif-file against the sarif schema.

Filing here to keep track of merging this to trunk

[Bug testsuite/109360] RFE: check that generated .sarif files validate against the SARIF schema

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109360

--- Comment #1 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by David Malcolm <dmalcolm@gcc.gnu.org>:

https://gcc.gnu.org/g:9f4fdc3acebcf6b045edea1361570658da4bc0ab

commit r15-1540-g9f4fdc3acebcf6b045edea1361570658da4bc0ab
Author: David Malcolm <dmalcolm@redhat.com>
Date:   Fri Jun 21 08:46:13 2024 -0400

    diagnostics: fixes to SARIF output [PR109360]

    When adding validation of .sarif files against the schema
    (PR testsuite/109360) I discovered various issues where we were
    generating invalid .sarif files.

    Specifically, in
      c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-1.c
    the relatedLocations for the "note" diagnostics were missing column
    numbers, leading to validation failure due to non-unique elements,
    such as multiple:
            "message": {"text": "invalid UTF-8 character <bf>"}},
    on line 25 with no column information.

    Root cause is that for some diagnostics in libcpp we have a location_t
    representing the line as a whole, setting a column_override on the
    rich_location (since the line hasn't been fully read yet).  We were
    handling this column override for plain text output, but not for .sarif
    output.

    Similarly, in diagnostic-format-sarif-file-pr111700.c there is a warning
    emitted on "line 0" of the file, whereas SARIF requires line numbers to
    be positive.

    We also use column == 0 internally to mean "the line as a whole",
    whereas SARIF required column numbers to be positive.

    This patch fixes these various issues.

    gcc/ChangeLog:
            PR testsuite/109360
            * diagnostic-format-sarif.cc
            (sarif_builder::make_location_object): Pass any column override
            from rich_loc to maybe_make_physical_location_object.
            (sarif_builder::maybe_make_physical_location_object): Add
            "column_override" param and pass it to maybe_make_region_object.
            (sarif_builder::maybe_make_region_object): Add "column_override"
            param and use it when the location has 0 for a column.  Don't
            add "startLine", "startColumn", "endLine", or "endColumn" if
            the values aren't positive.
            (sarif_builder::maybe_make_region_object_for_context): Don't
            add "startLine" or "endLine" if the values aren't positive.

    libcpp/ChangeLog:
            PR testsuite/109360
            * include/rich-location.h (rich_location::get_column_override):
            New accessor.

    Signed-off-by: David Malcolm <dmalcolm@redhat.com>

[Bug testsuite/109360] RFE: check that generated .sarif files validate against the SARIF schema

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109360

--- Comment #2 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by David Malcolm <dmalcolm@gcc.gnu.org>:

https://gcc.gnu.org/g:a84fe222029ff21903283cc8ee4bc760ebf80ec2

commit r15-1541-ga84fe222029ff21903283cc8ee4bc760ebf80ec2
Author: David Malcolm <dmalcolm@redhat.com>
Date:   Fri Jun 21 08:46:14 2024 -0400

    testsuite: check that generated .sarif files validate against the SARIF
schema [PR109360]

    This patch extends the dg directive verify-sarif-file so that if
    the "jsonschema" tool is available, it will be used to validate the
    generated .sarif file.

    Tested with jsonschema 3.2 with Python 3.8

    gcc/ChangeLog:
            PR testsuite/109360
            * doc/install.texi: Mention optional usage of "jsonschema" tool.

    gcc/testsuite/ChangeLog:
            PR testsuite/109360
            * lib/sarif-schema-2.1.0.json: New file, downloaded from
           
https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/schemas/sarif-schema-2.1.0.json
            Licensing information can be seen at
            https://github.com/oasis-tcs/sarif-spec/issues/583
            which states "They are free to incorporate it into their
            implementation. No need for special permission or paperwork from
            OASIS."
            * lib/scansarif.exp (verify-sarif-file): If "jsonschema" is
            available, use it to verify that the .sarif file complies with the
            SARIF schema.
            * lib/target-supports.exp (check_effective_target_jsonschema):
            New.

    Signed-off-by: David Malcolm <dmalcolm@redhat.com>

[Bug testsuite/109360] RFE: check that generated .sarif files validate against the SARIF schema

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109360

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2024-06-21
             Status|UNCONFIRMED                 |ASSIGNED
     Ever confirmed|0                           |1

--- Comment #3 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Should be fixed on trunk for GCC 15 by the above patche.

Keeping open to track backporting to GCC 14 and GCC 13.

[Bug testsuite/109360] RFE: check that generated .sarif files validate against the SARIF schema

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109360

--- Comment #4 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by David Malcolm <dmalcolm@gcc.gnu.org>:

https://gcc.gnu.org/g:17967907102099806dc80c71ee7665ffb22ffa23

commit r15-1633-g17967907102099806dc80c71ee7665ffb22ffa23
Author: David Malcolm <dmalcolm@redhat.com>
Date:   Tue Jun 25 20:26:21 2024 -0400

    testsuite: use check-jsonschema for validating .sarif files [PR109360]

    As reported here:
      https://gcc.gnu.org/pipermail/gcc-patches/2024-June/655434.html
    the schema validation I added for generated .sarif files in
    r15-1541-ga84fe222029ff2 used the "jsonschema" command line tool, which
    has been deprecated by more recent versions of the Python 3 "jsonschema"
    module.

    This patch updates the validation to use the more recent
    "check-jsonschema" command line tool, from the Python 3 "check-jsonschema"
    module, fixing the testsuite FAILs due to the deprecation message.

    As an added bonus, the output on validation failures is *much* nicer, e.g.
    if I undo r15-1540-g9f4fdc3acebcf6, the error messages begin like this:
    verify-sarif-file: res: Schema validation errors were encountered.
     
diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].locations[0].physicalLocation.region.startColumn:
0 is less than the minimum of 1
     
diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].relatedLocations[0].physicalLocation.region.startColumn:
0 is less than the minimum of 1
     
diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].relatedLocations[1].physicalLocation.region.startColumn:
0 is less than the minimum of 1
     
diagnostic-format-sarif-file-bad-utf8-pr109098-1.c.sarif::$.runs[0].results[0].relatedLocations[2].physicalLocation.region.startColumn:
0 is less than the minimum of 1
    child process exited abnormally
    FAIL: c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-1.c 
-Wc++-compat   (test .sarif output against SARIF schema)

    Tested with Python 3.8 with check_jsonschema 0.28.6

    gcc/ChangeLog:
            PR testsuite/109360
            * doc/install.texi (Python3 modules): Update SARIF validation
            requirement to use check-jsonschema rather than jsonschema.

    gcc/testsuite/ChangeLog:
            PR testsuite/109360
            * lib/scansarif.exp (verify-sarif-file): Use check-jsonschema
            rather than jsonschema, updating the invocation accordingly.
            * lib/target-supports.exp (check_effective_target_jsonschema):
Convert
            to...
            (check_effective_target_check_jsonschema): ...this.

    Signed-off-by: David Malcolm <dmalcolm@redhat.com>