Inconsistent segmentation fault in GCC

Inconsistent segmentation fault in GCC

[Alessandro Baretta]

Hello GCC hackers, and thank you very much for your precious work here.

I've been observing somewhat random GCC segfaults in my C++ 20
codebase for a while. By "random" I mean that if I rerun the cmake
build after a failure, 75% out of the time it will succeed, and it
might fail again the next time I try running the build. I have every
reason to believe that the issue is not with my code because:
1) At the second (or more rarely third) attempt at running the build,
it succeeds.
2) Clang does not segfault on the same codebase, and indeed produces
correct code.

The codebase is fairly large and heavily templatized, so I know that
the the compiler allocates a seemingly inordinate amount of memory
(using top, I've seen it reach a RES size of 1.6GB and a VIRT size of
4GB), so I initially attributed the segfaults to lack of enough memory
on my computer (16GB, with chrome running...). Now I turned on swap
(32GB), and I see that even when gcc is running, there is plenty of
available memory, so this is not a plausible cause.

The possibility of a hardware glitch in my system also does not hold
much water, as neither clang nor any other piece of software seems to
exhibit any instability.

It'd be great if I could get some guidance on how to investigate this
issue further. If it were of value to the developers, I'd be happy to
file a bug report, but given the randomness of the issue, I doubt that
they'd be able to reproduce the problem even with a preprocessed file.
What then is my path forward?

Here's some information that might be relevant:

$ cat /etc/debian_version
11.1
$ gcc --version
gcc (Debian 10.2.1-6) 10.2.1 20210110
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Thanks for any help!

-- 
Alex Baretta
t: 650-383-0227

Re: Inconsistent segmentation fault in GCC

[Andrew Pinski]

On Wed, Dec 22, 2021 at 5:07 PM Alessandro Baretta via Gcc
<gcc@gcc.gnu.org> wrote:
>
> Hello GCC hackers, and thank you very much for your precious work here.
>
> I've been observing somewhat random GCC segfaults in my C++ 20
> codebase for a while. By "random" I mean that if I rerun the cmake
> build after a failure, 75% out of the time it will succeed, and it
> might fail again the next time I try running the build. I have every
> reason to believe that the issue is not with my code because:
> 1) At the second (or more rarely third) attempt at running the build,
> it succeeds.
> 2) Clang does not segfault on the same codebase, and indeed produces
> correct code.

Maybe https://gcc.gnu.org/PR99831 Which was fixed in GCC 10.4 but was
broken in GCC 10.2.x.
This is a garbage collector related Internal compiler error which can
seem to randomly appear and disappear even.

Thanks,
Andrew Pinski

>
> The codebase is fairly large and heavily templatized, so I know that
> the the compiler allocates a seemingly inordinate amount of memory
> (using top, I've seen it reach a RES size of 1.6GB and a VIRT size of
> 4GB), so I initially attributed the segfaults to lack of enough memory
> on my computer (16GB, with chrome running...). Now I turned on swap
> (32GB), and I see that even when gcc is running, there is plenty of
> available memory, so this is not a plausible cause.
>
> The possibility of a hardware glitch in my system also does not hold
> much water, as neither clang nor any other piece of software seems to
> exhibit any instability.
>
> It'd be great if I could get some guidance on how to investigate this
> issue further. If it were of value to the developers, I'd be happy to
> file a bug report, but given the randomness of the issue, I doubt that
> they'd be able to reproduce the problem even with a preprocessed file.
> What then is my path forward?
>
> Here's some information that might be relevant:
>
> $ cat /etc/debian_version
> 11.1
> $ gcc --version
> gcc (Debian 10.2.1-6) 10.2.1 20210110
> Copyright (C) 2020 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
> Thanks for any help!
>
> --
> Alex Baretta
> t: 650-383-0227

Re: Inconsistent segmentation fault in GCC

[Alessandro Baretta]

Andrew,

Thank you for your kind reply. You alerted me to the fact that the bug
I am seeing is likely limited to a version or a small range of
versions of GCC. I have thus tinkered with my project's CMakeLists.txt
and discovered the following:
* The compiler was originally set to `gcc-11`, whereas in my previous
email I incorrectly reported the version of the default `gcc`
executable, which happens to be version 11.2.
* Switching the compiler in CMakeLists.txt to `gcc-10` fixed the issue for me.

Here is the output of the --version option for both compilers.
$ gcc-10 --version
gcc-10 (Debian 10.2.1-6) 10.2.1 20210110

$ gcc-11 --version
gcc-11 (Debian 11.2.0-12) 11.2.0

I have thus concluded that the bug causing the segfaults that I
described affects version 11.2 but not 10.2. I was unable to find deb
packages for neither version 10.4, which you suggested might have a
fix for this bug, nor 10.3. Hence, I cannot report on the presence of
this bug in any other version.

How I might help diagnose and fix this bug? For instance, how does one
run gcc from inside gdb? I know that gcc is just the driver and
cc1plus is the actual compiler, so I presume I'd have to run cc1plus
inside gdb, but as far as I know cc1plus cannot be called directly.
Suggestions?

-- Alex


On Wed, Dec 22, 2021 at 5:16 PM Andrew Pinski <pinskia@gmail.com> wrote:
>
> On Wed, Dec 22, 2021 at 5:07 PM Alessandro Baretta via Gcc
> <gcc@gcc.gnu.org> wrote:
> >
> > Hello GCC hackers, and thank you very much for your precious work here.
> >
> > I've been observing somewhat random GCC segfaults in my C++ 20
> > codebase for a while. By "random" I mean that if I rerun the cmake
> > build after a failure, 75% out of the time it will succeed, and it
> > might fail again the next time I try running the build. I have every
> > reason to believe that the issue is not with my code because:
> > 1) At the second (or more rarely third) attempt at running the build,
> > it succeeds.
> > 2) Clang does not segfault on the same codebase, and indeed produces
> > correct code.
>
> Maybe https://gcc.gnu.org/PR99831 Which was fixed in GCC 10.4 but was
> broken in GCC 10.2.x.
> This is a garbage collector related Internal compiler error which can
> seem to randomly appear and disappear even.
>
> Thanks,
> Andrew Pinski
>
> >
> > The codebase is fairly large and heavily templatized, so I know that
> > the the compiler allocates a seemingly inordinate amount of memory
> > (using top, I've seen it reach a RES size of 1.6GB and a VIRT size of
> > 4GB), so I initially attributed the segfaults to lack of enough memory
> > on my computer (16GB, with chrome running...). Now I turned on swap
> > (32GB), and I see that even when gcc is running, there is plenty of
> > available memory, so this is not a plausible cause.
> >
> > The possibility of a hardware glitch in my system also does not hold
> > much water, as neither clang nor any other piece of software seems to
> > exhibit any instability.
> >
> > It'd be great if I could get some guidance on how to investigate this
> > issue further. If it were of value to the developers, I'd be happy to
> > file a bug report, but given the randomness of the issue, I doubt that
> > they'd be able to reproduce the problem even with a preprocessed file.
> > What then is my path forward?
> >
> > Here's some information that might be relevant:
> >
> > $ cat /etc/debian_version
> > 11.1
> > $ gcc --version
> > gcc (Debian 10.2.1-6) 10.2.1 20210110
> > Copyright (C) 2020 Free Software Foundation, Inc.
> > This is free software; see the source for copying conditions.  There is NO
> > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> >
> >
> > Thanks for any help!
> >
> > --
> > Alex Baretta
> > t: 650-383-0227



-- 
Alex Baretta
t: 650-383-0227

Re: Inconsistent segmentation fault in GCC

[Jonathan Wakely]

On Thu, 23 Dec 2021, 06:07 Alessandro Baretta via Gcc, <gcc@gcc.gnu.org>
wrote:

>
> How I might help diagnose and fix this bug? For instance, how does one
> run gcc from inside gdb? I know that gcc is just the driver and
> cc1plus is the actual compiler, so I presume I'd have to run cc1plus
> inside gdb, but as far as I know cc1plus cannot be called directly.
>

That's not true, it can be run directly. The full cc1plus command is shown
if you add -v to your GCC command.

https://gcc.gnu.org/wiki/DebuggingGCC has some more tips.