[Bug analyzer/106441] New: Analyzer has some issues with nested functions extension to C

[Bug analyzer/106441] New: Analyzer has some issues with nested functions extension to C

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106441

            Bug ID: 106441
           Summary: Analyzer has some issues with nested functions
                    extension to C
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: analyzer
          Assignee: dmalcolm at gcc dot gnu.org
          Reporter: dmalcolm at gcc dot gnu.org
  Target Milestone: ---

See: https://gcc.gnu.org/onlinedocs/gcc/Nested-Functions.html

Given:

#include "analyzer-decls.h"

static int __attribute__((noinline))
__analyzer_callee_test_1 (int a, int b)
{
  int square (int z) { return z * z; }

  return square (a) + square (b);
}

void test_1 (void)
{
  __analyzer_describe (0, __analyzer_callee_test_1 (3, 4)); /* { dg-warning
"'\\(int\\)25'" } */
}

void test_2 (void *p)
{
  void call_free ()
  {
    __builtin_free (p); /* { dg-warning "double-'free'" } */
  }
  call_free (p);
  call_free (p);
}

static int __attribute__((noinline))
__analyzer_callee_test_3 (int a, int b, int (**out_fn) (int))
{
  int square (int z) { return z * z; }

  *out_fn = square;

  return square (a) + square (b);
}

void test_3 (void)
{
  int (*nested_fn) (int);
  __analyzer_describe (0, __analyzer_callee_test_3 (3, 4, &nested_fn)); /* {
dg-warning "'\\(int\\)25'" } */

  __analyzer_describe (0, nested_fn (4));  /* { dg-warning "'\\(int\\)16'" "" {
xfail *-*-* } } */
}

...it kind of works, but there are some issues.

The double-free is reported as:
nested-fn-1.c:20:5: warning: double-‘free’ of ‘*CHAIN.p’ [CWE-415]
[-Wanalyzer-double-free]
   20 |     __builtin_free (p);
      |     ^~~~~~~~~~~~~~~~~~
where the "CHAIN" implementation detail leaks through.


The square of 4 via a function pointer is reported as:
nested-fn-1.c:41:3: warning: svalue: ‘CONJURED(_3 = nested_fn.6_2 (4);, _3)’
   41 |   __analyzer_describe (0, nested_fn (4));

rather than as 16.

Looking at the gimple dump I see uses of __builtin_dwarf_cfa,
__builtin_init_trampoline, __builtin_adjust_trampoline, and references to a
"static-chain" at calls.  The analyzer doesn't know anything about any of this
(and neither do I, right now :) )

[Bug analyzer/106441] Analyzer has some issues with nested functions extension to C

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106441

--- Comment #1 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Above example on Compiler Explorer:
  https://godbolt.org/z/Pv1GoonE6

[Bug analyzer/106441] Analyzer has some issues with nested functions extension to C

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106441

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
__analyzer_callee_test_3 (int a, int b, int (**out_fn) (int))
{
  int square (int z) { return z * z; }

  *out_fn = square;

This really should be warning that the assignment of the nested function is
escaping the current scope because the chain value goes out of the scope at the
end.

[Bug analyzer/106441] Analyzer has some issues with nested functions extension to C

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106441

--- Comment #3 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Andrew Pinski from comment #2)
> __analyzer_callee_test_3 (int a, int b, int (**out_fn) (int))
> {
>   int square (int z) { return z * z; }
> 
>   *out_fn = square;
> 
> This really should be warning that the assignment of the nested function is
> escaping the current scope because the chain value goes out of the scope at
> the end.

Sorry the FRAME (in this case FRAME.4).

If we look at the IR:

  __builtin_init_trampoline (&FRAME.4.square, square, &FRAME.4);
  _8 = __builtin_adjust_trampoline (&FRAME.4.square);
  _9 = (int (*<T34d>) (int)) _8;
  *out_fn_10(D) = _9;

__builtin_init_trampoline is setting up the trampoline (new function ptr) into
FRAME.4.square for the nested function square with the frame of FRAME.4.

__builtin_adjust_trampoline adjusts the trampoline if needed (usually just a
call to flush the icache) and will return a function pointer.

So _9/_8 will refer to the trampoline and/or frame that is an local variable
which goes out of scope when the function returns too.

[Bug analyzer/106441] Analyzer has some issues with nested functions extension to C

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106441

--- Comment #4 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Oh I didn't mention what __builtin_dwarf_cfa  was for. It is used for debugging
really. So it is easy to find where the function was nested into.