[Bug analyzer/98599] New: fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[Bug analyzer/98599] New: fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[acoplan at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

            Bug ID: 98599
           Summary: fatal error: Cgraph edge statement index out of range
                    with -Os -flto -fanalyzer
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: analyzer
          Assignee: dmalcolm at gcc dot gnu.org
          Reporter: acoplan at gcc dot gnu.org
  Target Milestone: ---

The following fails:

$ cat test.c
int b(int);
int a() { b(5); }
int main() { a(); }
$ aarch64-elf-gcc -Os -flto -fanalyzer test.c
test.c: In function 'a':
test.c:3:5: fatal error: Cgraph edge statement index out of range
    3 | int main() { a(); }
      |     ^
compilation terminated.
lto-wrapper: fatal error: aarch64-elf-gcc returned 1 exit status
compilation terminated.
/home/alecop01/toolchain/build-aarch64-elf/install/lib/gcc/aarch64-elf/11.0.0/../../../../aarch64-elf/bin/ld:
error: lto-wrapper failed
collect2: error: ld returned 1 exit status

Removing any one of these options prevents the error.

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-01-08

--- Comment #1 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Thanks for filing this.  Confirmed (on x86_64-pc-linux-gnu); am debugging it.

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #2 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
As far as I can tell, there are two invocations of lto1: wpa, then ltrans.

The analyzer is run in the first invocation.

The analyzer updates the gimple stmt uids; if I disable this updating the crash
doesn't happen.

The crash happens in the 2nd invocation of lto1 at:
    1200          for (cedge = node->indirect_calls; cedge; cedge =
cedge->next_callee)
    1201            {
    1202              if (STMT_UID_NOT_IN_RANGE (cedge->lto_stmt_uid))
    1203                fatal_error (input_location,
    1204                             "Cgraph edge statement index out of
range");

I haven't managed to fully debug this yet, but it looks like the stmt uids are
stored into the callgraph edge's lto_stmt_uid, and somewhere between the cgraph
and LTO infrastructure it doesn't expect an IPA pass (the analyzer) to change
the uids in the stmts from under it, even though gimple.h has this for the stmt
field:

  /* UID of this statement.  This is used by passes that want to
     assign IDs to statements.  It must be assigned and used by each
     pass.  By default it should be assumed to contain garbage.  */
  unsigned uid;

and gimple_set_uid has:

   Please note that this UID property is supposed to be undefined at
   pass boundaries.  This means that a given pass should not assume it
   contains any useful value when the pass starts and thus can set it
   to any value it sees fit.

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #3 from Martin Liška <marxin at gcc dot gnu.org> ---
> 
> The analyzer updates the gimple stmt uids; if I disable this updating the
> crash doesn't happen.

Hm, why do you update them in LTO WPA phase?
Note that they are used by cgraph_edges to point to a proper GIMPLE statement.

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #4 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
I set them so that each stmt has a unique id, unique across all functions.  I
was assuming from the comments I quoted above in gimple.h that this is safe to
do, but it sounds like from your comment that WPA makes assumptions that the
uids don't change.  If that's the case, then I need to rethink things, and the
gimple.h comments probably need updating (or WPA needs to allow for uids to
change, but maybe that's hard to do?)

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org

--- Comment #5 from Martin Liška <marxin at gcc dot gnu.org> ---
@Honza: Can you please judge here?

@David: Can you really reproduce that on x86_64-linux-gnu (I can't for some
reason)?

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #6 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
(In reply to Martin Liška from comment #5)
> @David: Can you really reproduce that on x86_64-linux-gnu (I can't for some
> reason)?
Yes (with current master e.g. cfaaa6a1ca744c1a93fa08a3e7ab2a821383cac1), using
  -Os -flto -fanalyzer
and adding
  -v --save-temps
to debug it

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #7 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to David Malcolm from comment #4)
> I set them so that each stmt has a unique id, unique across all functions. 
> I was assuming from the comments I quoted above in gimple.h that this is
> safe to do, but it sounds like from your comment that WPA makes assumptions
> that the uids don't change.  If that's the case, then I need to rethink
> things, and the gimple.h comments probably need updating (or WPA needs to
> allow for uids to change, but maybe that's hard to do?)

So looking into the code, in WPA when we stream out cgraph_edge, we record to
which GIMPLE statement an edge points to:

gcc/lto-cgraph.c:264-265

and the info is later used in LTRANS where you see the fatal_error.

So I would recommend using a separate table where you will assign a unique ID
to statements. And please update the comment in gimple.h.

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #8 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Saving and restoring the uids fixes the issue, so I'm working on a patch to the
analyzer pass to do that.

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |WAITING

--- Comment #9 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Candidate patch:
https://gcc.gnu.org/pipermail/gcc-patches/2021-January/563416.html

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #10 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
I no longer believe the patch in comment #9 is correct.

My latest analysis of this issue is:
  https://gcc.gnu.org/pipermail/gcc-patches/2021-January/564070.html
(I was hoping for a response from Honza on that latter post)

[Bug analyzer/98599] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jseward at acm dot org

--- Comment #11 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
*** Bug 99854 has been marked as a duplicate of this bug. ***

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|fatal error: Cgraph edge    |[11 Regression] fatal
                   |statement index out of      |error: Cgraph edge
                   |range with -Os -flto        |statement index out of
                   |-fanalyzer                  |range with -Os -flto
                   |                            |-fanalyzer

--- Comment #12 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Am fairly sure this is a regression relative to GCC 10, so updating Summary
accordingly; I'd like -fanalyzer to be usable with -flto again (even with its
current limitations)

Honza; any thoughts on the stuff in comment #10?

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #13 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by David Malcolm <dmalcolm@gcc.gnu.org>:

https://gcc.gnu.org/g:17f3c2b8ac477b07ca0aafbc7d74ba305dc1ee33

commit r11-8142-g17f3c2b8ac477b07ca0aafbc7d74ba305dc1ee33
Author: David Malcolm <dmalcolm@redhat.com>
Date:   Mon Apr 12 21:13:40 2021 -0400

    gimple UIDs, LTO and -fanalyzer [PR98599]

    gimple.h has this comment for gimple's uid field:

      /* UID of this statement.  This is used by passes that want to
         assign IDs to statements.  It must be assigned and used by each
         pass.  By default it should be assumed to contain garbage.  */
      unsigned uid;

    and gimple_set_uid has:

       Please note that this UID property is supposed to be undefined at
       pass boundaries.  This means that a given pass should not assume it
       contains any useful value when the pass starts and thus can set it
       to any value it sees fit.

    which suggests that any pass can use the uid field as an arbitrary
    scratch space.

    PR analyzer/98599 reports a case where this error occurs in LTO mode:
      fatal error: Cgraph edge statement index out of range
    on certain inputs with -fanalyzer.

    The error occurs in the LTRANS phase after -fanalyzer runs in the
    WPA phase.  The analyzer pass writes to the uid fields of all stmts.

    The error occurs when LTRANS is streaming callgraph edges back in.
    The LTO format uses stmt uids to associate call stmts with callgraph
    edges between WPA and LTRANS.
    For example, in lto-cgraph.c, lto_output_edge writes out the
    gimple_uid, and input_edge reads it back in.

    lto_prepare_function_for_streaming has code to renumber the stmt UIDs
    when the code is streamed back out, but for some reason this isn't
    called for clones:
        307   /* Do body modifications needed for streaming before we fork out
        308      worker processes.  */
        309   FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (node)
        310     if (!node->clone_of && gimple_has_body_p (node->decl))
        311       lto_prepare_function_for_streaming (node);

    Hence the combination of -fanalyzer and -flto will fail in LTRANS's
    stream-in if any function clones are encountered.

    It's not fully clear to me why this isn't done for clones, and what the
    correct fix should be to allow arbitrary changes to uids within WPA
    passes.

    In the meantime, this patch works around the issue by updating the
    analyzer to save and restore the UIDs, fixing the error.

    gcc/analyzer/ChangeLog:
            PR analyzer/98599
            * supergraph.cc (saved_uids::make_uid_unique): New.
            (saved_uids::restore_uids): New.
            (supergraph::supergraph): Replace assignments to stmt->uid with
            calls to m_stmt_uids.make_uid_unique.
            (supergraph::~supergraph): New.
            * supergraph.h (class saved_uids): New.
            (supergraph::~supergraph): New decl.
            (supergraph::m_stmt_uids): New field.

    gcc/testsuite/ChangeLog:
            PR analyzer/98599
            * gcc.dg/analyzer/pr98599-a.c: New test.
            * gcc.dg/analyzer/pr98599-b.c: New test.

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

David Malcolm <dmalcolm at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|WAITING                     |RESOLVED

--- Comment #14 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
Should be fixed by the above patch, though it's more of a workaround for now;
am still not sure about what's going on with clones.

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #15 from Jan Hubicka <hubicka at ucw dot cz> ---
> Should be fixed by the above patch, though it's more of a workaround for now;
> am still not sure about what's going on with clones.

I undestand it now.  The problem is that fixup is missed for one gimple
body after statements was renumbered by analyzer.  The reason is
confused check for has_gimple_body_p.   We do not really get out of sync
between uids of functions and its clone, since we stream only clone, but
with the fact that stream in code expects uids to be continuously
increasing.

So apparenlty analyzer is first pass that does use UIDs of statements at
WPA time.

Sorry for taking so long time to understand this.

diff --git a/gcc/lto/lto.c b/gcc/lto/lto.c
index ceb61bb300b..5903f75ac23 100644
--- a/gcc/lto/lto.c
+++ b/gcc/lto/lto.c
@@ -306,7 +306,7 @@ lto_wpa_write_files (void)
   cgraph_node *node;
   /* Do body modifications needed for streaming before we fork out
      worker processes.  */
-  FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (node)
+  FOR_EACH_FUNCTION (node)
     if (!node->clone_of && gimple_has_body_p (node->decl))
       lto_prepare_function_for_streaming (node);

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[acoplan at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #16 from Alex Coplan <acoplan at gcc dot gnu.org> ---
*** Bug 100042 has been marked as a duplicate of this bug. ***

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[dmalcolm at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #17 from David Malcolm <dmalcolm at gcc dot gnu.org> ---
(In reply to Jan Hubicka from comment #15)

Thanks.

[...]
> So apparenlty analyzer is first pass that does use UIDs of statements at
> WPA time.

I wonder if there should be a debug flag that trashes all UIDs to a garbage
value at the end of every pass, to avoid reliance on UIDs surviving a pass.

[Bug analyzer/98599] [11 Regression] fatal error: Cgraph edge statement index out of range with -Os -flto -fanalyzer

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98599

--- Comment #18 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jan Hubicka <hubicka@gcc.gnu.org>:

https://gcc.gnu.org/g:b5f644a98b3f3543d3a8d2dfea7785c22879013f

commit r11-8190-gb5f644a98b3f3543d3a8d2dfea7785c22879013f
Author: Jan Hubicka <jh@suse.cz>
Date:   Thu Apr 15 11:40:40 2021 +0200

    Fix handling of clones in lto_wpa_write_files [PR98599]

    2021-04-15  Jan Hubicka  <hubicka@ucw.cz>

            PR lto/98599
            * lto.c (lto_wpa_write_files): Fix handling of clones.