* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
@ 2021-09-24 6:15 ` rguenther at suse dot de
2021-09-24 6:33 ` rguenth at gcc dot gnu.org
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: rguenther at suse dot de @ 2021-09-24 6:15 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
--- Comment #1 from rguenther at suse dot de <rguenther at suse dot de> ---
On Thu, 23 Sep 2021, dmalcolm at gcc dot gnu.org wrote:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
>
> Bug ID: 102471
> Summary: RFE: add support to analyzer testsuite for running
> SAMATE/SARD tests (e.g. Juliet Test Suite)
> Product: gcc
> Version: 12.0
> Status: UNCONFIRMED
> Severity: normal
> Priority: P3
> Component: analyzer
> Assignee: dmalcolm at gcc dot gnu.org
> Reporter: dmalcolm at gcc dot gnu.org
> CC: rguenth at gcc dot gnu.org
> Target Milestone: ---
>
> See:
> https://www.nist.gov/itl/ssd/software-quality-group/samate
> https://samate.nist.gov/SARD/testsuite.php
>
> The links above have various promising-looking testsuites e.g.
> - Juliet Test Suite
> - Klocwork test suite
> - ITC-Benchmarks
> etc
>
> It would be good to be able to (somehow) automatically run them as part of
> regression testing of the analyzer - either by turning them directly into
> DejaGnu tests, or by wrapping the suite's own harness in a way that we can
> invoke it during "make check".
It might be also feasible to think of on-the-side CI runs of
external testsuites on buildbots or similar and have results
reported on gcc.gnu.org.
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
2021-09-24 6:15 ` [Bug analyzer/102471] " rguenther at suse dot de
@ 2021-09-24 6:33 ` rguenth at gcc dot gnu.org
2021-10-26 0:16 ` pinskia at gcc dot gnu.org
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: rguenth at gcc dot gnu.org @ 2021-09-24 6:33 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
Richard Biener <rguenth at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |NEW
Last reconfirmed| |2021-09-24
Ever confirmed|0 |1
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
2021-09-24 6:15 ` [Bug analyzer/102471] " rguenther at suse dot de
2021-09-24 6:33 ` rguenth at gcc dot gnu.org
@ 2021-10-26 0:16 ` pinskia at gcc dot gnu.org
2021-12-01 15:52 ` dmalcolm at gcc dot gnu.org
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: pinskia at gcc dot gnu.org @ 2021-10-26 0:16 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
Andrew Pinski <pinskia at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Severity|normal |enhancement
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
` (2 preceding siblings ...)
2021-10-26 0:16 ` pinskia at gcc dot gnu.org
@ 2021-12-01 15:52 ` dmalcolm at gcc dot gnu.org
2021-12-01 17:39 ` dmalcolm at gcc dot gnu.org
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: dmalcolm at gcc dot gnu.org @ 2021-12-01 15:52 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
--- Comment #2 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
There's also
https://www.nist.gov/itl/ssd/software-quality-group/other-assurance-tool-test-collections
Currently the only non-Java collection on the list is:
https://sir.csc.ncsu.edu/portal/index.php
However that repository has:
https://sir.csc.ncsu.edu/portal/sir-license.php
which seems to be a non-Open Source licence ("solely for non-commercial,
educational, evaluation and/or personal use").
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
` (3 preceding siblings ...)
2021-12-01 15:52 ` dmalcolm at gcc dot gnu.org
@ 2021-12-01 17:39 ` dmalcolm at gcc dot gnu.org
2021-12-01 23:21 ` cvs-commit at gcc dot gnu.org
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: dmalcolm at gcc dot gnu.org @ 2021-12-01 17:39 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
David Malcolm <dmalcolm at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
--- Comment #3 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
I've got a primitive workflow of running the Juliet 1.3 testsuite, and a script
for testing results here:
https://github.com/davidmalcolm/juliet-harness
Currently I've only run it on the:
C/testcases/CWE415_Double_Free/s01
sudirectory, with:
# of passes 130
# of failures 103
I'm looking through the failures.
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
` (4 preceding siblings ...)
2021-12-01 17:39 ` dmalcolm at gcc dot gnu.org
@ 2021-12-01 23:21 ` cvs-commit at gcc dot gnu.org
2022-05-17 18:04 ` dmalcolm at gcc dot gnu.org
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: cvs-commit at gcc dot gnu.org @ 2021-12-01 23:21 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
--- Comment #4 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by David Malcolm <dmalcolm@gcc.gnu.org>:
https://gcc.gnu.org/g:860c56b5bc356960a4d0445dadc43ceddbe3c7e2
commit r12-5701-g860c56b5bc356960a4d0445dadc43ceddbe3c7e2
Author: David Malcolm <dmalcolm@redhat.com>
Date: Wed Dec 1 14:12:33 2021 -0500
analyzer: fix false leak seen in Juliet 1.3 [PR102471]
Juliet 1.3's CWE415_Double_Free__malloc_free_*_67a.c
were showing leak false positives in non-LTO builds; fixed thusly.
gcc/analyzer/ChangeLog:
PR analyzer/102471
* region-model-reachability.cc (reachable_regions::handle_parm):
Treat all svalues within a compound parm has reachable, and those
wrapped in a cast.
gcc/testsuite/ChangeLog:
PR analyzer/102471
* gcc.dg/analyzer/leak-3.c: New test.
Signed-off-by: David Malcolm <dmalcolm@redhat.com>
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
` (5 preceding siblings ...)
2021-12-01 23:21 ` cvs-commit at gcc dot gnu.org
@ 2022-05-17 18:04 ` dmalcolm at gcc dot gnu.org
2023-01-18 16:17 ` dmalcolm at gcc dot gnu.org
2023-01-19 19:01 ` dmalcolm at gcc dot gnu.org
8 siblings, 0 replies; 10+ messages in thread
From: dmalcolm at gcc dot gnu.org @ 2022-05-17 18:04 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
--- Comment #5 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Another source of possible benchmarks:
https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks
>From SV-COMP: https://sv-comp.sosy-lab.org/
This embeds the Juliet testsuite, but also many other tests.
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
` (6 preceding siblings ...)
2022-05-17 18:04 ` dmalcolm at gcc dot gnu.org
@ 2023-01-18 16:17 ` dmalcolm at gcc dot gnu.org
2023-01-19 19:01 ` dmalcolm at gcc dot gnu.org
8 siblings, 0 replies; 10+ messages in thread
From: dmalcolm at gcc dot gnu.org @ 2023-01-18 16:17 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
--- Comment #6 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
I've created
https://github.com/davidmalcolm/gcc-analyzer-integration-tests
which builds Juliet plus various real-world C projects with a candidate build
of GCC with -fanalyzer and captures the diagnostics in SARIF format.
https://github.com/microsoft/sarif-tools has a command for comparing SARIF
files (or pairs of directories full of SARIF files) which is what I'm currently
using to compare diagnostics emitted before/after a candidate GCC patch.
I'm working through the diagnostics seen with GCC trunk's -fanalyzer,
classifying them into true vs false positives and will probably write my own
comparison script to show the effect of a GCC patch on false vs true positives.
^ permalink raw reply [flat|nested] 10+ messages in thread* [Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
2021-09-23 13:26 [Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite) dmalcolm at gcc dot gnu.org
` (7 preceding siblings ...)
2023-01-18 16:17 ` dmalcolm at gcc dot gnu.org
@ 2023-01-19 19:01 ` dmalcolm at gcc dot gnu.org
8 siblings, 0 replies; 10+ messages in thread
From: dmalcolm at gcc dot gnu.org @ 2023-01-19 19:01 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
David Malcolm <dmalcolm at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution|--- |MOVED
--- Comment #7 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Marking this one as RESOLVED MOVED, since I'm now doing this in:
https://github.com/davidmalcolm/gcc-analyzer-integration-tests
^ permalink raw reply [flat|nested] 10+ messages in thread