[Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[Bug analyzer/102471] New: RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

            Bug ID: 102471
           Summary: RFE: add support to analyzer testsuite for running
                    SAMATE/SARD tests (e.g. Juliet Test Suite)
           Product: gcc
           Version: 12.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: analyzer
          Assignee: dmalcolm at gcc dot gnu.org
          Reporter: dmalcolm at gcc dot gnu.org
                CC: rguenth at gcc dot gnu.org
  Target Milestone: ---

See:
  https://www.nist.gov/itl/ssd/software-quality-group/samate
  https://samate.nist.gov/SARD/testsuite.php

The links above have various promising-looking testsuites e.g.
- Juliet Test Suite
- Klocwork test suite
- ITC-Benchmarks
etc

It would be good to be able to (somehow) automatically run them as part of
regression testing of the analyzer - either by turning them directly into
DejaGnu tests, or by wrapping the suite's own harness in a way that we can
invoke it during "make check".

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

--- Comment #1 from rguenther at suse dot de <rguenther at suse dot de> ---
On Thu, 23 Sep 2021, dmalcolm at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
> 
>             Bug ID: 102471
>            Summary: RFE: add support to analyzer testsuite for running
>                     SAMATE/SARD tests (e.g. Juliet Test Suite)
>            Product: gcc
>            Version: 12.0
>             Status: UNCONFIRMED
>           Severity: normal
>           Priority: P3
>          Component: analyzer
>           Assignee: dmalcolm at gcc dot gnu.org
>           Reporter: dmalcolm at gcc dot gnu.org
>                 CC: rguenth at gcc dot gnu.org
>   Target Milestone: ---
> 
> See:
>   https://www.nist.gov/itl/ssd/software-quality-group/samate
>   https://samate.nist.gov/SARD/testsuite.php
> 
> The links above have various promising-looking testsuites e.g.
> - Juliet Test Suite
> - Klocwork test suite
> - ITC-Benchmarks
> etc
> 
> It would be good to be able to (somehow) automatically run them as part of
> regression testing of the analyzer - either by turning them directly into
> DejaGnu tests, or by wrapping the suite's own harness in a way that we can
> invoke it during "make check".

It might be also feasible to think of on-the-side CI runs of
external testsuites on buildbots or similar and have results
reported on gcc.gnu.org.

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2021-09-24
     Ever confirmed|0                           |1

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

--- Comment #2 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
There's also
 
https://www.nist.gov/itl/ssd/software-quality-group/other-assurance-tool-test-collections

Currently the only non-Java collection on the list is:
  https://sir.csc.ncsu.edu/portal/index.php
However that repository has:
  https://sir.csc.ncsu.edu/portal/sir-license.php
which seems to be a non-Open Source licence ("solely for non-commercial,
educational, evaluation and/or personal use").

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #3 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
I've got a primitive workflow of running the Juliet 1.3 testsuite, and a script
for testing results here:
  https://github.com/davidmalcolm/juliet-harness

Currently I've only run it on the:
  C/testcases/CWE415_Double_Free/s01
sudirectory, with:
  # of passes   130
  # of failures 103

I'm looking through the failures.

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

--- Comment #4 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by David Malcolm <dmalcolm@gcc.gnu.org>:

https://gcc.gnu.org/g:860c56b5bc356960a4d0445dadc43ceddbe3c7e2

commit r12-5701-g860c56b5bc356960a4d0445dadc43ceddbe3c7e2
Author: David Malcolm <dmalcolm@redhat.com>
Date:   Wed Dec 1 14:12:33 2021 -0500

    analyzer: fix false leak seen in Juliet 1.3 [PR102471]

    Juliet 1.3's CWE415_Double_Free__malloc_free_*_67a.c
    were showing leak false positives in non-LTO builds; fixed thusly.

    gcc/analyzer/ChangeLog:
            PR analyzer/102471
            * region-model-reachability.cc (reachable_regions::handle_parm):
            Treat all svalues within a compound parm has reachable, and those
            wrapped in a cast.

    gcc/testsuite/ChangeLog:
            PR analyzer/102471
            * gcc.dg/analyzer/leak-3.c: New test.

    Signed-off-by: David Malcolm <dmalcolm@redhat.com>

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

--- Comment #5 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Another source of possible benchmarks:
  https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks
>From SV-COMP: https://sv-comp.sosy-lab.org/

This embeds the Juliet testsuite, but also many other tests.

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

--- Comment #6 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
I've created
  https://github.com/davidmalcolm/gcc-analyzer-integration-tests
which builds Juliet plus various real-world C projects with a candidate build
of GCC with -fanalyzer and captures the diagnostics in SARIF format.

https://github.com/microsoft/sarif-tools has a command for comparing SARIF
files (or pairs of directories full of SARIF files) which is what I'm currently
using to compare diagnostics emitted before/after a candidate GCC patch.

I'm working through the diagnostics seen with GCC trunk's -fanalyzer,
classifying them into true vs false positives and will probably write my own
comparison script to show the effect of a GCC patch on false vs true positives.

[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |MOVED

--- Comment #7 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Marking this one as RESOLVED MOVED, since I'm now doing this in:
  https://github.com/davidmalcolm/gcc-analyzer-integration-tests