ICE with precompiled headers (GCC 8.1)

ICE with precompiled headers (GCC 8.1)

[Paul Smith]

I have GCC 8.1.0 / binutils 2.30 on GNU/Linux 64bit (built locally). 
My codebase is almost all C++.

I'm implementing precompiled headers and it was going well.  However,
sometimes a PCH file is generated that causes the compiler to ICE when
trying to use it during a source file compilation:

  <command-line>: internal compiler error: Segmentation fault
  Please submit a full bug report,
  with preprocessed source if appropriate.
  See <https://gcc.gnu.org/bugs/> for instructions.

If I rename the .gch file and compile the source it works; if I put
back the .gch file and compile the source it will ICE again.  So
clearly there's something corrupt about the PCH file itself.

Also this does not happen every time, even using the identical code. 
Sometimes the PCH file is fine and the compile is fine.  It has never
happened on my local system but it happens quite often on some of our
build servers (typically these are a bit older/slower with slower
disks).  And, it doesn't happen to all PCH files in the same build (I
create different PCH files for different libraries), even ones that
include the same files!

Since it's not reproducible and the code is proprietary I haven't tried
to produce a publishable case so far.

Here's the thing: I added the "-MD -MF foo.d" options to the PCH file
compile (I wanted to get a manifest of which files were included in the
PCH file).  Without those options I cannot reproduce this at all (in a
number of tries).  With those options it happens most of the time on
some systems (but again, never on my local system).

Unfortunately my GCC is heavily optimized and stripped so backtraces
are useless.  I will generate a debuggable GCC and see if I can get
more info on the ICE.

In the meantime, does this remind anyone of an existing bug, hopefully
one that was fixed in 8.2 or 8.3?

Re: ICE with precompiled headers (GCC 8.1)

[Zan Lynx]

On April 30, 2019 7:43:47 AM MDT, Paul Smith <paul@mad-scientist.net> wrote:
>I have GCC 8.1.0 / binutils 2.30 on GNU/Linux 64bit (built locally). 
>My codebase is almost all C++.
>
>I'm implementing precompiled headers and it was going well.  However,
>sometimes a PCH file is generated that causes the compiler to ICE when
>trying to use it during a source file compilation:
>
>  <command-line>: internal compiler error: Segmentation fault
>  Please submit a full bug report,
>  with preprocessed source if appropriate.
>  See <https://gcc.gnu.org/bugs/> for instructions.
>
>If I rename the .gch file and compile the source it works; if I put
>back the .gch file and compile the source it will ICE again.  So
>clearly there's something corrupt about the PCH file itself.
>
>Also this does not happen every time, even using the identical code. 
>Sometimes the PCH file is fine and the compile is fine.  It has never
>happened on my local system but it happens quite often on some of our
>build servers (typically these are a bit older/slower with slower
>disks).  And, it doesn't happen to all PCH files in the same build (I
>create different PCH files for different libraries), even ones that
>include the same files!
>
>Since it's not reproducible and the code is proprietary I haven't tried
>to produce a publishable case so far.
>
>Here's the thing: I added the "-MD -MF foo.d" options to the PCH file
>compile (I wanted to get a manifest of which files were included in the
>PCH file).  Without those options I cannot reproduce this at all (in a
>number of tries).  With those options it happens most of the time on
>some systems (but again, never on my local system).
>
>Unfortunately my GCC is heavily optimized and stripped so backtraces
>are useless.  I will generate a debuggable GCC and see if I can get
>more info on the ICE.
>
>In the meantime, does this remind anyone of an existing bug, hopefully
>one that was fixed in 8.2 or 8.3?

It does remind me of a race condition bug with a Makefile I wrote years ago.

One or two build tasks did not properly depend on the precompiled headers but used them anyway, and sometimes during make -j8 would get unlucky and use a partial header file.
-- 
                Knowledge is Power -- Power Corrupts
                        Study Hard -- Be Evil

Re: ICE with precompiled headers (GCC 8.1)

[Paul Smith]

On Tue, 2019-04-30 at 09:34 -0600, Zan Lynx wrote:
> > In the meantime, does this remind anyone of an existing bug,
> > hopefully one that was fixed in 8.2 or 8.3?
> 
> It does remind me of a race condition bug with a Makefile I wrote
> years ago.
> 
> One or two build tasks did not properly depend on the precompiled
> headers but used them anyway, and sometimes during make -j8 would get
> unlucky and use a partial header file.

Unfortunately once one of these corrupted PCH files is created every
compile fails, not just the first few.  Also after the build is fully
complete (failed) and I log into the build server and run the compiler
by hand, I continue to get the failure every time with that PCH file. 
So, it's not just a temporary use of a partially-created file.

I've checked the build logs and as far as I can see each precompiled
header is built only one time, so it's also not a matter of the same
PCH file being built multiple times and the commands stomping on each
other.

Thx for the note...

Re: ICE with precompiled headers (GCC 8.1)

[Paul Smith]

On Tue, 2019-04-30 at 09:43 -0400, Paul Smith wrote:
> Here's the thing: I added the "-MD -MF foo.d" options to the PCH file
> compile (I wanted to get a manifest of which files were included in the
> PCH file).  Without those options I cannot reproduce this at all (in a
> number of tries).  With those options it happens most of the time on
> some systems (but again, never on my local system).

I have made more discoveries.

In the compiler invocation that is ICEing, if I remove the -fpch-deps
option then it no longer ICEs (with the same PCH file that ICEd
before!)  I can remove/add that option to my .cpp compile line, and it
ICEs with it and works without it, no other changes.

But again, it works fine on my system always.  So there's _sometimes_
something odd inside the PCH file which is triggered by the source file
compilation adding -fpch-deps.

FWIW, here's the compile line I use to generate the PCH:

  g++ ...opts... -MD -MF dir/foo.h.d -o dir/foo.h.gch -x c++-header foo.h

And here's the compile line I use to compile source code:

  g++ ...opts... -fpch-deps -fpch-preprocess -Winvalid-pch \
      -iquote"dir" --include=foo.h

Adding/removing -fpch-preprocess doesn't seem to matter.  It's only the
-fpch-deps option during the compile, combined with the -MD/-MF options
when creating the PCH file, that seems to trigger the ICE.

> Unfortunately my GCC is heavily optimized and stripped so backtraces
> are useless.  I will generate a debuggable GCC and see if I can get
> more info on the ICE.

I have created a GCC with debug enabled so I'll see what I find.

Re: ICE with precompiled headers (GCC 8.1)

[Paul Smith]

On Wed, 2019-05-01 at 09:35 -0400, Paul Smith wrote:
> > Unfortunately my GCC is heavily optimized and stripped so backtraces
> > are useless.  I will generate a debuggable GCC and see if I can get
> > more info on the ICE.
> 
> I have created a GCC with debug enabled so I'll see what I find.

I was able to reproduce this ICE quite readily with my debuggable GCC
8.1.0.  Here's the failure:

<command-line>: internal compiler error: Segmentation fault
0x9cae61 crash_signal
        /work/src/cc/gcc-8.1.0/gcc/toplev.c:325
0x1293778 apply_vpath
        /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:127
0x1293acc deps_add_dep(deps*, char const*)
        /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:258
0x1293fe3 deps_restore(deps*, _IO_FILE*, char const*)
        /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:432
0x129535b cpp_read_state(cpp_reader*, char const*, _IO_FILE*, save_macro_data*)
        /work/src/cc/gcc-8.1.0/libcpp/pch.c:884
0x596d59 c_common_read_pch(cpp_reader*, char const*, int, char const*)
        /work/src/cc/gcc-8.1.0/gcc/c-family/c-pch.c:373
0x12872fe should_stack_file
        /work/src/cc/gcc-8.1.0/libcpp/files.c:814
0x12874f1 _cpp_stack_file
        /work/src/cc/gcc-8.1.0/libcpp/files.c:900
0x12876a7 _cpp_stack_include
        /work/src/cc/gcc-8.1.0/libcpp/files.c:1049
0x1287b22 cpp_push_include(cpp_reader*, char const*)
        /work/src/cc/gcc-8.1.0/libcpp/files.c:1484
0x5943ec push_command_line_include
        /work/src/cc/gcc-8.1.0/gcc/c-family/c-opts.c:1483
0x594615 c_finish_options
        /work/src/cc/gcc-8.1.0/gcc/c-family/c-opts.c:1452
0x5963a2 c_common_parse_file()
        /work/src/cc/gcc-8.1.0/gcc/c-family/c-opts.c:1126

Unsurprisingly the problem is that the "deps" data member in
cpp_reader* is null:

#0  apply_vpath (d=d@entry=0x0,
    t=t@entry=0x2174860 "/src/foo_pch.h") at /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:127
#1  0x0000000001293acd in deps_add_dep (d=d@entry=0x0,
    t=t@entry=0x2174860 "/src/foo_pch.h") at /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:258
#2  0x0000000001293fe4 in deps_restore (deps=0x0, fd=fd@entry=0x2171750,
    self=self@entry=0x2106810 "/src/obj/foo_pch.h.gch")
    at /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:432
#3  0x000000000129535c in cpp_read_state (r=r@entry=0x20f4d60,
    name=name@entry=0x2106810 "/src/obj/foo_pch.h.gch", f=f@entry=0x2171750, data=0x210bce0)
    at /work/src/cc/gcc-8.1.0/libcpp/pch.c:884

I have no idea whether the problem is that it should never be possible
for "deps" to be null, or whether the problem is that we're not
checking for that possibility when we should be.

I'm building the current GCC 9.0.1 prerelease to see if I can reproduce
it there.

Once again removing -fpch-deps solves the problem.  I can only assume
that without that flag we never bother to walk the deps data member at
all.

Re: ICE with precompiled headers (GCC 8.1)

[Paul Smith]

I was able to reproduce this with GCC 9.0.1 20190430 as well.

It appears that adding -MD to the PCH build sometimes causes the "deps"
data in the PCH file to be empty, but this is never noticed unless you
use the -fpch-deps option on the source compile line: if you don't do
this then nothing in GCC will try to access that pointer.

I feel like I've hit the end of what I can do with tracking this down,
so I filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90306

Although I couldn't create a repro case, I can reproduce it here on
some systems so if someone wants me to investigate further I'm happy to
do that: please provide some pointers/advice for debugging.


On Wed, 2019-05-01 at 10:56 -0400, Paul Smith wrote:
> On Wed, 2019-05-01 at 09:35 -0400, Paul Smith wrote:
> > > Unfortunately my GCC is heavily optimized and stripped so backtraces
> > > are useless.  I will generate a debuggable GCC and see if I can get
> > > more info on the ICE.
> > 
> > I have created a GCC with debug enabled so I'll see what I find.
> 
> I was able to reproduce this ICE quite readily with my debuggable GCC
> 8.1.0.  Here's the failure:

Re: ICE with precompiled headers (GCC 8.1)

[Nathan Sidwell]

On 5/1/19 10:56 AM, Paul Smith wrote:
> On Wed, 2019-05-01 at 09:35 -0400, Paul Smith wrote:
>>> Unfortunately my GCC is heavily optimized and stripped so backtraces
>>> are useless.  I will generate a debuggable GCC and see if I can get
>>> more info on the ICE.
>>
>> I have created a GCC with debug enabled so I'll see what I find.
> 
> I was able to reproduce this ICE quite readily with my debuggable GCC
> 8.1.0.  Here's the failure:
> 
> <command-line>: internal compiler error: Segmentation fault
> 0x9cae61 crash_signal
>          /work/src/cc/gcc-8.1.0/gcc/toplev.c:325
> 0x1293778 apply_vpath
>          /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:127
> 0x1293acc deps_add_dep(deps*, char const*)
>          /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:258
> 0x1293fe3 deps_restore(deps*, _IO_FILE*, char const*)
>          /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:432
> 0x129535b cpp_read_state(cpp_reader*, char const*, _IO_FILE*, save_macro_data*)
>          /work/src/cc/gcc-8.1.0/libcpp/pch.c:884
> 0x596d59 c_common_read_pch(cpp_reader*, char const*, int, char const*)
>          /work/src/cc/gcc-8.1.0/gcc/c-family/c-pch.c:373
> 0x12872fe should_stack_file
>          /work/src/cc/gcc-8.1.0/libcpp/files.c:814
> 0x12874f1 _cpp_stack_file
>          /work/src/cc/gcc-8.1.0/libcpp/files.c:900
> 0x12876a7 _cpp_stack_include
>          /work/src/cc/gcc-8.1.0/libcpp/files.c:1049
> 0x1287b22 cpp_push_include(cpp_reader*, char const*)
>          /work/src/cc/gcc-8.1.0/libcpp/files.c:1484
> 0x5943ec push_command_line_include
>          /work/src/cc/gcc-8.1.0/gcc/c-family/c-opts.c:1483
> 0x594615 c_finish_options
>          /work/src/cc/gcc-8.1.0/gcc/c-family/c-opts.c:1452
> 0x5963a2 c_common_parse_file()
>          /work/src/cc/gcc-8.1.0/gcc/c-family/c-opts.c:1126
> 
> Unsurprisingly the problem is that the "deps" data member in
> cpp_reader* is null:
> 
> #0  apply_vpath (d=d@entry=0x0,
>      t=t@entry=0x2174860 "/src/foo_pch.h") at /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:127
> #1  0x0000000001293acd in deps_add_dep (d=d@entry=0x0,
>      t=t@entry=0x2174860 "/src/foo_pch.h") at /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:258
> #2  0x0000000001293fe4 in deps_restore (deps=0x0, fd=fd@entry=0x2171750,
>      self=self@entry=0x2106810 "/src/obj/foo_pch.h.gch")
>      at /work/src/cc/gcc-8.1.0/libcpp/mkdeps.c:432
> #3  0x000000000129535c in cpp_read_state (r=r@entry=0x20f4d60,
>      name=name@entry=0x2106810 "/src/obj/foo_pch.h.gch", f=f@entry=0x2171750, data=0x210bce0)
>      at /work/src/cc/gcc-8.1.0/libcpp/pch.c:884
> 
> I have no idea whether the problem is that it should never be possible
> for "deps" to be null, or whether the problem is that we're not
> checking for that possibility when we should be.
> 
> I'm building the current GCC 9.0.1 prerelease to see if I can reproduce
> it there.
> 
> Once again removing -fpch-deps solves the problem.  I can only assume
> that without that flag we never bother to walk the deps data member at
> all.

I've been playing with dep generation on the modules branch, and noticed 
there did appear to be something funky going on with its interaction 
with PCH.  I didn't investigate, but have some patches that I'll be 
merging in the not too distant future.

nathan
-- 
Nathan Sidwell