[Bug lto/96343] New: LTO ICE on PPC64le

[Bug lto/96343] New: LTO ICE on PPC64le

[axel.huebl at web dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

            Bug ID: 96343
           Summary: LTO ICE on PPC64le
           Product: gcc
           Version: 9.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: axel.huebl at web dot de
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

Hi,

we are trying to compile a few larger HPC libraries and applications with GCC
and CMake using LTO support on Power9 (ppc64le). With the available GCC
versions we have on Summit (OLCF), 6.4.0 - 9.1.0, we reproducibly tap into an
ICE in the LTO step.


GCC 6.4.0:

lto1: internal compiler error: invalid resolution in the resolution file
0x101d0e93 lto_resolution_read
        /sw/summit/gcc/6.4.0/src/gcc-6.4.0/gcc/lto/lto.c:1923
0x101d0e93 lto_file_read
        /sw/summit/gcc/6.4.0/src/gcc-6.4.0/gcc/lto/lto.c:2092
0x101d0e93 read_cgraph_and_symbols
        /sw/summit/gcc/6.4.0/src/gcc-6.4.0/gcc/lto/lto.c:2806
0x101d0e93 lto_main()
        /sw/summit/gcc/6.4.0/src/gcc-6.4.0/gcc/lto/lto.c:3310

GCC 7.4.0:

lto1: internal compiler error: invalid resolution in the resolution file
0x101a589b lto_resolution_read
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/lto/lto.c:1939
0x101a589b lto_file_read
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/lto/lto.c:2108
0x101a589b read_cgraph_and_symbols
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/lto/lto.c:2824
0x101a589b lto_main()
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/lto/lto.c:3341

GCC 9.1.0:

lto1: internal compiler error: invalid line in the resolution file
0x101915bb lto_resolution_read
       
/tmp/belhorn/gcc-build-9.1.0-alpha+20190716/gcc-9.1.0/gcc/lto/lto.c:1957
0x101915bb lto_file_read
       
/tmp/belhorn/gcc-build-9.1.0-alpha+20190716/gcc-9.1.0/gcc/lto/lto.c:2142
0x10191c57 read_cgraph_and_symbols
       
/tmp/belhorn/gcc-build-9.1.0-alpha+20190716/gcc-9.1.0/gcc/lto/lto.c:2849
0x10191c57 lto_main()
       
/tmp/belhorn/gcc-build-9.1.0-alpha+20190716/gcc-9.1.0/gcc/lto/lto.c:3387

The problem occurs with the library https://github.com/ornladios/ADIOS2 or the
application https://github.com/ECP-WarpX/WarpX independently, given one adds

set_property(TARGET WarpX PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE)

to the latter's CMakeLists.txt, anywhere after add_executable(WarpX) .

We try to derive a smaller example but have not arrived at one yet. Maybe the
ICE line numbers above can hint us where to look?

Ref.: https://github.com/ornladios/ADIOS2/issues/1524#issuecomment-662783738

[Bug lto/96343] LTO ICE on PPC64le

[axel.huebl at web dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

--- Comment #1 from Axel <axel.huebl at web dot de> ---
I did verify that `gcc-ar` is used for intermediate libraries in the build.

[Bug lto/96343] LTO ICE on PPC64le

[axel.huebl at web dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

--- Comment #2 from Axel <axel.huebl at web dot de> ---
The flags the CMake adds for LTO are -flto -fno-fat-lto-objects

The projects are otherwise C++11 and C++14 projects.

[Bug lto/96343] LTO ICE on PPC64le

[segher at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

--- Comment #3 from Segher Boessenkool <segher at gcc dot gnu.org> ---
Thanks for the report.

Yes, not sure we can reproduce it like this.  We might need something
smaller, more self-contained.

In the meantime...  Can you check if this still happens with trunk,
or with GCC 10?  If it doesn't, we don't have to chase a bug that has
been solved already, and we can bisect to find the patch that fixed
this (hopefully actually fixed, it might be just a side effect).  Or,
if it is still occurring, it is something we need to know as well :-)

[Bug lto/96343] LTO ICE on PPC64le

[luoxhu at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

--- Comment #4 from luoxhu at gcc dot gnu.org ---
I tried to build both ADIOS2 and WarpX(with INTERPROCEDURAL_OPTIMIZATION) on a
Power8 machine with gcc 9.3.0&9.2.1, no LTO error seen.

/usr/bin/cmake ../ -DCMAKE_C_COMPILER=/opt/at12.0/bin/gcc
-DCMAKE_CXX+COMPILER=/opt/at12.0/bin/g++ -DADIOS2_USE_Fortran=OFF
-DADIOS2_USE_ZeroMQ=OFF -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release
-DCMAKE_POSITION_INDEPENDENT_CODE=ON -DADIOS2_USE_SST=OFF
-DCMAKE_CXX_FLAGS="-flto -fno-fat-lto-objects ${CMAKE_CXX_FLAGS}"
 make -j50

Not sure any difference with your configuration?

Anyway, it will be much better if you could try new GCC or reduce a smaller
test case, BTW, I see that someone mentioned that it may related to conda and
python https://github.com/ornladios/ADIOS2/issues/1524#issue-458229988?

[Bug lto/96343] LTO ICE on PPC64le

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2020-07-28
     Ever confirmed|0                           |1

--- Comment #5 from Richard Biener <rguenth at gcc dot gnu.org> ---
lto1: internal compiler error: invalid resolution in the resolution file

it would be interesting to see the actual resolution file.  When you
lto-link with -v -save-temps added you'll see a lto1 -fwpa invocation
with a -fresolution=FILENAME argument, attach FILENAME here.

[Bug lto/96343] LTO ICE on PPC64le

[axel.huebl at web dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96343

Axel <axel.huebl at web dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |INVALID

--- Comment #6 from Axel <axel.huebl at web dot de> ---
Thank you everyone for your help!

I started over and cannot reproduce the problem anymore. Uff, I am really sorry
for that.

I am took now the latest `development` branch as of commit
57cc9a3397f3e89e57b3bbc15ba8215ad2144b4f.
  https://github.com/ECP-WarpX/WarpX
I patched in
  set_property(TARGET WarpX PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE)
in line 152 of CMakeLists.txt

If I configure and compile like this on Power9:
  cmake ..
  make VERBOSE=ON

I CANNOT reproduce the ICE.

Next, I also add LTO to the static library target that we build on the fly,
AMReX, as I forgot to report initially. I pushed a branch to
https://github.com/ax3l/amrex/commit/facef9ac064ff638c7f37c726f65b4e465c532af
so one can do execute in a fresh build directory:
  cmake .. -DWarpX_amrex_repo=https://github.com/ax3l/amrex.git
-DWarpX_amrex_branch=topic-lto
  make VERBOSE=ON

I can still not reproduce the report, something must have been off in my
environment... I am also building not in TMPFS now, as I was last time due to a
laggy FS in $HOME.

I can even build with microarch tuning now:
  CXXFLAGS="-mcpu=native -mtune=native" cmake ..
-DWarpX_amrex_repo=https://github.com/ax3l/amrex.git
-DWarpX_amrex_branch=topic-lto

@Richard @Segher Thank you for the hints as well. Sorry as well to you.

Pew, I am glad this works!

I did a little non-representative GCC 6.4.0 benchmark on Power9 with
time-to-solution (in s):

notune     406.8 (-O3) | 485.9 (-O2) 
  tune     400.3 (-O3) | 476.3 (-O2)
notune_lto 407.5 (-O3)
  tune_lto 404.2 (-O3)