From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id 8F8B03858410; Sat, 11 Mar 2023 00:22:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8F8B03858410 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1678494150; bh=KEcwCl4FubeyFavKKqVm2R02X4DpFkK8QkaTcfTDmT8=; h=From:To:Subject:Date:From; b=oeWlTIgFJiJE/E6DhceFy9Xla1Tzer5+lUINr4Wcio7RCcszwAFA4g52lgbsQHJdf EFEu2YOn93JmcVQUsT8aW/X+dF/8Z84IVF2uftG1+cOT/q+rmt/j7WRqTOW0KmsQvd QU9iNlIt8NfvDOHH0NG7Q87iRY8TwM2NVfnXC7IY= From: "dmalcolm at gcc dot gnu.org" To: gcc-bugs@gcc.gnu.org Subject: [Bug analyzer/109098] New: Encoding errors on SARIF output for non-UTF-8 source files Date: Sat, 11 Mar 2023 00:22:30 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: gcc X-Bugzilla-Component: analyzer X-Bugzilla-Version: 13.0 X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: dmalcolm at gcc dot gnu.org X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P3 X-Bugzilla-Assigned-To: dmalcolm at gcc dot gnu.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://gcc.gnu.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 List-Id: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D109098 Bug ID: 109098 Summary: Encoding errors on SARIF output for non-UTF-8 source files Product: gcc Version: 13.0 Status: UNCONFIRMED Severity: normal Priority: P3 Component: analyzer Assignee: dmalcolm at gcc dot gnu.org Reporter: dmalcolm at gcc dot gnu.org Target Milestone: --- I've run into issues with -fanalyzer integration testing when a diagnostic occurs in a source file with non-UTF-8 bytes in it. Specifically, haproxy-2.7.1's include/import/ist.h has a comment within function "istalloc" that isn't quite UTF-8. When a diagnostic occurs on this, with -fdiagnostics-format=3Dsarif-file, = the file is quoted in the SARIF output, but the individual bytes are copied directly into the JSON, leading to e.g.: $ sarif ls integration-tests/haproxy-2.7.1/haproxy-2.7.1/http_client.c.sarif Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/sarif/loader.py", line 57, in load_sarif_file data =3D json.load(file_in) File "/usr/lib64/python3.8/json/__init__.py", line 293, in load return loads(fp.read(), File "/usr/lib64/python3.8/codecs.py", line 322, in decode (result, consumed) =3D self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 96006: invalid start byte The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/bin/sarif", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/sarif/cmdline/main.py", line= 40, in main exitcode =3D args.func(args) File "/usr/local/lib/python3.8/site-packages/sarif/cmdline/main.py", line 384, in _ls ls_op.print_ls(args.files_or_dirs, args.output) File "/usr/local/lib/python3.8/site-packages/sarif/operations/ls_op.py", = line 17, in print_ls sarif_files =3D loader.load_sarif_files(path) File "/usr/local/lib/python3.8/site-packages/sarif/loader.py", line 30, in load_sarif_files path_exists =3D _add_path_to_sarif_file_set(path, ret) File "/usr/local/lib/python3.8/site-packages/sarif/loader.py", line 17, in _add_path_to_sarif_file_set sarif_file_set.add_file(load_sarif_file(path)) File "/usr/local/lib/python3.8/site-packages/sarif/loader.py", line 60, in load_sarif_file raise IOError(f"Cannot load {file_path}") from exception OSError: integration-tests/haproxy-2.7.1/haproxy-2.7.1/http_client.c.sarif I'm working on a fix.=