[Bug other/111559] New: [14 regression] ICE when building Python with PGO

[Bug other/111559] New: [14 regression] ICE when building Python with PGO

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

            Bug ID: 111559
           Summary: [14 regression] ICE when building Python with PGO
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
          Assignee: unassigned at gcc dot gnu.org
          Reporter: sjames at gcc dot gnu.org
                CC: arsen at gcc dot gnu.org
  Target Milestone: ---

Unfortunately, Python PGO strikes again.

Originally reported downstream at
https://bugs.gentoo.org/show_bug.cgi?id=914578.

```
gcc-14 -c -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1
-DNDEBUG     -O2 -pipe -march=native -fdiagnostics-color=always
-frecord-gcc-switches -Wreturn-type -ggdb3 -fwrapv -fno-semantic-interposition
-std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers
-Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden
-fprofile-use -fprofile-correction -I./Include/internal  -I. -I./Include
-I/usr/include/ncursesw  -fPIC -DPy_BUILD_CORE -o Parser/parser.o
Parser/parser.c
Parser/parser.c: In function 'simple_stmt_rule':
Parser/parser.c:1706:1: error: probability of edge 613->614 not initialized
 1706 | simple_stmt_rule(Parser *p)
      | ^~~~~~~~~~~~~~~~
Parser/parser.c:1706:1: error: probability of edge 615->621 not initialized
during IPA pass: inline
Parser/parser.c:1706:1: internal compiler error: verify_flow_info failed
0x55c9cced2153 verify_flow_info()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cfghooks.cc:287
0x55c9cd4da214 checking_verify_flow_info()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cfghooks.h:214
0x55c9cd4da214 cleanup_tree_cfg_noloop
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/tree-cfgcleanup.cc:1154
0x55c9cd4da214 cleanup_tree_cfg(unsigned int)
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/tree-cfgcleanup.cc:1205
0x55c9cd330924 execute_function_todo
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/passes.cc:2057
0x55c9cd330e10 execute_todo
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/passes.cc:2142
0x55c9cd333bef execute_one_ipa_transform_pass
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/passes.cc:2336
0x55c9cd333bef execute_all_ipa_transforms(bool)
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/passes.cc:2396
0x55c9ccf13238 cgraph_node::expand()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cgraphunit.cc:1834
0x55c9ccf13238 cgraph_node::expand()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cgraphunit.cc:1794
0x55c9ccf141dc expand_all_functions
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cgraphunit.cc:2000
0x55c9ccf141dc symbol_table::compile()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cgraphunit.cc:2398
0x55c9ccf17a0b symbol_table::compile()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cgraphunit.cc:2311
0x55c9ccf17a0b symbol_table::finalize_compilation_unit()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cgraphunit.cc:2583
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
make[1]: *** [Makefile:2717: Parser/parser.o] Error 1
```

I can reproduce it manually, but still with a large reproducer:
* wget https://www.python.org/ftp/python/3.11.5/Python-3.11.5.tar.xz
* tar xvf Python-3.11.5.tar.xz
* ./configure CC=gcc-14 --enable-optimizations
* make -j$(nproc)

[Bug other/111559] [14 regression] ICE when building Python with PGO

[sjames at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #1 from Sam James <sjames at gcc dot gnu.org> ---
I don't think I'm going to be able to get a minimal reproducer for this, but
trying to at least get something a bit smaller/precise.

If I take inspiration from PR79587, I can do a bit better:
* wget https://www.python.org/ftp/python/3.11.5/Python-3.11.5.tar.xz
* tar xvf Python-3.11.5.tar.xz
* ./configure CC=gcc-14 --enable-optimizations
* make clean
* make profile-removal
* make build_all_generate_profile
* make profile-removal
* make run_profile_task # unfortunately this runs something like 44 different
Python tests
* make build_all_merge_profile
* gcc-14 -c -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1
-DNDEBUG     -O2 -pipe -march=native -fdiagnostics-color=always
-frecord-gcc-switches -Wreturn-type -ggdb3 -fwrapv -fno-semantic-interposition
-std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers
-Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden
-fprofile-use -fprofile-correction -I./Include/internal  -I. -I./Include
-I/usr/include/ncursesw  -fPIC -DPy_BUILD_CORE -o Parser/parser.o
Parser/parser.c -save-temps


```
gcc-14 -c -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1
-DNDEBUG     -O2 -pipe -fdiagnostics-color=always -frecord-gcc-switches
-Wreturn-type -ggdb3 -fwrapv -fno-semantic-interposition -std=c11 -Wextra
-Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes
-Werror=implicit-function-declaration -fvisibility=hidden -fprofile-use
-fprofile-correction -I./Include/internal  -I. -I./Include
-I/usr/include/ncursesw  -fPIC -DPy_BUILD_CORE -o Parser/parser.o
Parser/parser.c -save-temps
gcc-14: warning: ‘-pipe’ ignored because ‘-save-temps’ specified
Parser/parser.c: In function ‘_tmp_28_rule’:
Parser/parser.c:25179:1: error: probability of edge 3->4 not initialized
25179 | _tmp_28_rule(Parser *p)
      | ^~~~~~~~~~~~
Parser/parser.c:25179:1: error: probability of edge 5->11 not initialized
during GIMPLE pass: fixup_cfg
Parser/parser.c:25179:1: internal compiler error: verify_flow_info failed
0x55b70ce37153 verify_flow_info()
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/cfghooks.cc:287
0x55b70d295897 execute_function_todo
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/passes.cc:2100
0x55b70d295e10 execute_todo
       
/usr/src/debug/sys-devel/gcc-14.0.0_pre20230917/gcc-14-20230917/gcc/passes.cc:2142
Please submit a full bug report, with preprocessed source (by using
-freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
```

```
$ gcc-14 -v
Using built-in specs.
COLLECT_GCC=gcc-14
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/14/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with:
/var/tmp/portage/sys-devel/gcc-14.0.0_pre20230917/work/gcc-14-20230917/configure
--host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr
--bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/14
--includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/14/include
--datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14
--mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14/man
--infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/14/info
--with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14
--disable-silent-rules --disable-dependency-tracking
--with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/14/python
--enable-languages=c,c++,fortran,rust --enable-obsolete --enable-secureplt
--disable-werror --with-system-zlib --enable-nls --without-included-gettext
--disable-libunwind-exceptions --enable-checking=yes,extra,rtl
--with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo Hardened
14.0.0_pre20230917 p2' --with-gcc-major-version-only --enable-libstdcxx-time
--enable-lto --disable-libstdcxx-pch --enable-shared --enable-threads=posix
--enable-__cxa_atexit --enable-clocale=gnu --enable-multilib
--with-multilib-list=m32,m64 --disable-fixed-point --enable-targets=all
--enable-libgomp --disable-libssp --disable-libada --enable-cet
--disable-systemtap --enable-valgrind-annotations --disable-vtable-verify
--disable-libvtv --with-zstd --with-isl --disable-isl-version-check
--enable-default-pie --enable-host-pie --enable-host-bind-now
--enable-default-ssp --with-build-config=bootstrap-cet
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.0 20230917 (experimental) (Gentoo Hardened 14.0.0_pre20230917
p2)
```

I've uploaded build dir after running that last command at
https://dev.gentoo.org/~sam/bugs/gcc/gcc-python-pgo-x86/Python-3.11.5-pgo-crash.tar.xz.

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org
   Target Milestone|---                         |14.0

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

Sergei Trofimovich <slyfox at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |slyfox at gcc dot gnu.org

--- Comment #2 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
Possibly exposed by r14-3459-g0c78240fd7d519 "Check that passes do not forget
to define profile"

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[sirl at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

Franz Sirl <sirl at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sirl at gcc dot gnu.org

--- Comment #3 from Franz Sirl <sirl at gcc dot gnu.org> ---
Looks like a dup of the profiledbootstrap fail bug 111283

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #4 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
Looks like identical code folding creates uninitialized profile counters if
there are any edges in folded functions.

I think cvise did a decent job extracting the reproducer below. Here is a
single-file trigger on `--enable-checking=yes` `gcc` from `master`:

```
// $ cat bug.c
__attribute__((noipa)) static void edge(void) {}

static void rule1(int *p) {
    edge();
    if (*p) edge();
}

static void rule1_same(int *p) {
    edge();
    if (*p) edge();
}

__attribute__((noipa)) int main(void) {
    int p = 0;
    rule1(&p);
    rule1_same(&p);
}
```

Trigger:

```
$ echo PG
$ gcc -O2 -fprofile-generate bug.c -o b -fopt-info
$ echo RUN
$ ./b
$ echo PU
$ gcc -O2 -fprofile-use -fprofile-correction bug.c -o b -fopt-info
```

Running:

```
PG
$ gcc -O2 -fprofile-generate bug.c -o b -fopt-info
bug.c:15:5: optimized:  Inlined rule1.constprop/28 into main/3 which now has
time 75.280000 and size 51, net change of -6.
bug.c:16:5: optimized:  Inlined rule1_same.constprop/27 into main/3 which now
has time 94.560000 and size 72, net change of -6.

RUN
$ ./b

PU
$ gcc -O2 -fprofile-use -fprofile-correction bug.c -o b -fopt-info
bug.c:3:13: optimized: Semantic equality hit:rule1/1->rule1_same/2
bug.c:3:13: optimized: Assembler symbol names:rule1/1->rule1_same/2
bug.c:15:5: optimized:  Inlined rule1.constprop/5 into main/3 which now has
time 26.000000 and size 10, net change of +2.
bug.c:16:5: optimized:  Inlined rule1.constprop/4 into main/3 which now has
time 27.000000 and size 12, net change of -6.

bug.c: In function 'main':
bug.c:13:28: error: probability of edge 3->4 not initialized
   13 | __attribute__((noipa)) int main(void) {
      |                            ^~~~
bug.c:13:28: error: probability of edge 5->6 not initialized
during IPA pass: inline
bug.c:13:28: internal compiler error: verify_flow_info failed
```

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #5 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
Slightly shorter example that does not rely on inline:

// $ cat bug.c
__attribute__((noipa)) static void edge(void) {}

int p = 0;

__attribute__((noinline))
static void rule1(void) { if (p) edge(); }

__attribute__((noinline))
static void rule1_same(void) { if (p) edge(); }

__attribute__((noipa)) int main(void) {
    rule1();
    rule1_same();
}



bug.c: In function 'rule1':
bug.c:6:13: error: probability of edge 3->4 not initialized
    6 | static void rule1(void) { if (p) edge(); }
      |             ^~~~~
during GIMPLE pass: fixup_cfg
bug.c:6:13: internal compiler error: verify_flow_info failed

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #6 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
Uninitialized value comes from `ipa_merge_profiles()` for our `rule1_same()`
alias and `rule1()` functions:

  // in gcc/ipa-icf.cc:

  else if (create_alias)
    {
      alias->icf_merged = true;

      /* Remove the function's body.  */
      ipa_merge_profiles (original, alias);
      // ...

If I comment out `ipa_merge_profiles (original, alias);` call to leave
`original` as is then failure does not happen. Which means at least
`original`'s profile is fine.

Tracing through `ipa_merge_profiles()` we generate uninitialized probalility
profile when divide by zero at:

  // in gcc/ipa-utils.cc

void
ipa_merge_profiles (struct cgraph_node *dst,
                    struct cgraph_node *src,
                    bool preserve_body)
    // ...

      /* TODO: merge also statement histograms.  */
      FOR_ALL_BB_FN (srcbb, srccfun)
        {
          unsigned int i;

          if (copy_counts) { /* snip: ireelevant */ }
          else
            {
              for (i = 0; i < EDGE_COUNT (srcbb->succs); i++)
                {
                  edge srce = EDGE_SUCC (srcbb, i);
                  edge dste = EDGE_SUCC (dstbb, i);
                  dste->probability =
                    dste->probability * dstbb->count.ipa ().probability_in
                                                 (dstbb->count.ipa ()
                                                  + srccount.ipa ())
                    + srce->probability * srcbb->count.ipa ().probability_in
                                                 (dstbb->count.ipa ()
                                                  + srccount.ipa ());
                }
              dstbb->count = dstbb->count.ipa () + srccount.ipa ();
            }
        }
    // ...

Here `dstbb->count.ipa () + srccount.ipa ()` is zero.

This assert should expose it as well:

--- a/gcc/ipa-utils.cc
+++ b/gcc/ipa-utils.cc
@@ -651,13 +651,15 @@ ipa_merge_profiles (struct cgraph_node *dst,
                {
                  edge srce = EDGE_SUCC (srcbb, i);
                  edge dste = EDGE_SUCC (dstbb, i);
+
+                 profile_count den = dstbb->count.ipa () + srccount.ipa ();
+                 gcc_assert(den.nonzero_p());
+
                  dste->probability =
                    dste->probability * dstbb->count.ipa ().probability_in
-                                                (dstbb->count.ipa ()
-                                                 + srccount.ipa ())
+                                                (den)
                    + srce->probability * srcbb->count.ipa ().probability_in
-                                                (dstbb->count.ipa ()
-                                                 + srccount.ipa ());
+                                                (den);
                }
              dstbb->count = dstbb->count.ipa () + srccount.ipa ();
            }

If we attach `gdb` it agrees we exercise these edges 0 times.

    (gdb) call dstbb->count.debug()
    0 (precise)
    (gdb) call srccount.ipa ().debug()
    0 (precise)

For comparison we are trying to clobber `always` probability with `undefined`:

    (gdb) call dste->probability.debug()
    always

What edge is that?

    (gdb) call debug_edge(srce)
    edge (bb_3, bb_4)

__attribute__((noinline))
void rule1 ()
{
  int p.0_1;

  <bb 2> [count: 2]:
  p.0_1 = p;
  if (p.0_1 != 0)
    goto <bb 3>; [0.00%]
  else
    goto <bb 4>; [100.00%]

  <bb 3> [count: 0]:
  edge ();

  <bb 4> [count: 2]:
  return;

}

`always` should valid for `bb_3->bb_4`. But for our data input it's `never`.

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #7 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
Proposed conservative fix as
https://gcc.gnu.org/pipermail/gcc-patches/2023-September/631526.html

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[sirl at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #8 from Franz Sirl <sirl at gcc dot gnu.org> ---
The proposed patch on top of r14-4307 fixes the profiled bootstrap here.

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

--- Comment #9 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Sergei Trofimovich <slyfox@gcc.gnu.org>:

https://gcc.gnu.org/g:043a6fcbc27f8721301eb2f72a7839f54f393003

commit r14-4421-g043a6fcbc27f8721301eb2f72a7839f54f393003
Author: Sergei Trofimovich <siarheit@google.com>
Date:   Wed Sep 27 14:29:12 2023 +0100

    ipa-utils: avoid uninitialized probabilities on ICF [PR111559]

    r14-3459-g0c78240fd7d519 "Check that passes do not forget to define
profile"
    exposed check failures in cases when gcc produces uninitialized profile
    probabilities. In case of PR/111559 uninitialized profile is generated
    by edges executed 0 times reported by IPA profile:

        $ gcc -O2 -fprofile-generate pr111559.c -o b -fopt-info
        $ ./b
        $ gcc -O2 -fprofile-use -fprofile-correction pr111559.c -o b -fopt-info

        pr111559.c: In function 'rule1':
        pr111559.c:6:13: error: probability of edge 3->4 not initialized
            6 | static void rule1(void) { if (p) edge(); }
              |             ^~~~~
        during GIMPLE pass: fixup_cfg
        pr111559.c:6:13: internal compiler error: verify_flow_info failed

    The change conservatively ignores updates with zero execution counts and
    uses initially assigned probabilities (`always` probability in case of
    the example).

            PR ipa/111283
            PR gcov-profile/111559

    gcc/
            * ipa-utils.cc (ipa_merge_profiles): Avoid producing
            uninitialized probabilities when merging counters with zero
            denominators.

    gcc/testsuite/
            * gcc.dg/tree-prof/pr111559.c: New test.

[Bug gcov-profile/111559] [14 regression] ICE when building Python with PGO

[slyfox at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111559

Sergei Trofimovich <slyfox at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED
           Assignee|unassigned at gcc dot gnu.org      |slyfox at gcc dot gnu.org

--- Comment #10 from Sergei Trofimovich <slyfox at gcc dot gnu.org> ---
Fixed in master.

Thanks for the report!