[Bug pch/61250] New: Random pch failures on x86_64-apple-darwin13.

[Bug pch/61250] New: Random pch failures on x86_64-apple-darwin13.

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

            Bug ID: 61250
           Summary: Random pch failures on x86_64-apple-darwin13.
           Product: gcc
           Version: 4.10.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: pch
          Assignee: unassigned at gcc dot gnu.org
          Reporter: dominiq at lps dot ens.fr
                CC: iains at gcc dot gnu.org
              Host: x86_64-apple-darwin13
            Target: x86_64-apple-darwin13
             Build: x86_64-apple-darwin13

When running the test suite on x86_64-apple-darwin13 with -j8 (4 cores, 8
threads), I see random failures for the tests

g++.dg/pch/pch.C 
gcc.dg/pch/save-temps-1.c

They are either an ICE:

Internal compiler error: Error reporting routines re-entered.

(may be related to pr59877), or a "program timed out" leading to an "assembly
comparison":

WARNING: program timed out.
FAIL: g++.dg/pch/pch.C  -g -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C -g assembly comparison

AFAICT these failures occur for any set of options or -m32/-m64. The number of
failures ranges from 0 to 4/5 and seems (weakly) correlated to the machine load
(-j8+backup).

[Bug pch/61250] Random pch failures on x86_64-apple-darwin13.

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2014-11-11
                 CC|                            |fxcoudert at gmail dot com
     Ever confirmed|0                           |1

--- Comment #1 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
In https://gcc.gnu.org/ml/gcc-testresults/2014-11/msg01024.html I saw

FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C -O2 assembly comparison

FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (internal compiler error)
FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C  -O2 assembly comparison

FAIL: gcc.dg/pch/save-temps-1.c   -O2  -I. -Dwith_PCH (test for excess errors)
FAIL: gcc.dg/pch/save-temps-1.c  -O2  assembly comparison
WARNING: program timed out.

FAIL: gcc.dg/pch/save-temps-1.c   -O0  -I. -Dwith_PCH (test for excess errors)
FAIL: gcc.dg/pch/save-temps-1.c   -O1  -I. -Dwith_PCH (internal compiler error)
FAIL: gcc.dg/pch/save-temps-1.c   -O1  -I. -Dwith_PCH (test for excess errors)
FAIL: gcc.dg/pch/save-temps-1.c   -O1  assembly comparison
FAIL: gcc.dg/pch/save-temps-1.c   -O3 -fomit-frame-pointer  -I. -Dwith_PCH
(test for excess errors)
FAIL: gcc.dg/pch/save-temps-1.c  -O0  assembly comparison
FAIL: gcc.dg/pch/save-temps-1.c  -O3 -fomit-frame-pointer  assembly comparison

FX

Are the tests done in parallel? If yes with which value? Are you using "Time
Machine" during the tests?

[Bug pch/61250] Random pch failures on x86_64-apple-darwin13.

[fxcoudert at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

Francois-Xavier Coudert <fxcoudert at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fxcoudert at gcc dot gnu.org

--- Comment #2 from Francois-Xavier Coudert <fxcoudert at gcc dot gnu.org> ---
(In reply to Dominique d'Humieres from comment #1)
> Are the tests done in parallel? If yes with which value? Are you using "Time
> Machine" during the tests?

Yes, tests done in parallel (typically -j6), no Time Machine backup going on
(I'm at home). I know they are random failures, and I have confirmed that they
work well if serial testing is done. I plan to investigate it at some point,
but for now I focused on the main issues :)

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Target|x86_64-apple-darwin13       |x86_64-apple-darwin1(3|4)
               Host|x86_64-apple-darwin13       |x86_64-apple-darwin1(3|4)
            Summary|Random pch failures on      |Random pch failures on
                   |x86_64-apple-darwin13.      |x86_64-apple-darwin1(3|4).
              Build|x86_64-apple-darwin13       |x86_64-apple-darwin1(3|4)

--- Comment #3 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Also seen on x86_64-apple-darwin14.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

howarth at bromo dot med.uc.edu changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |howarth at bromo dot med.uc.edu

--- Comment #4 from howarth at bromo dot med.uc.edu ---
I see the failure of...

FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (internal compiler error)
FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (test for excess errors)
Excess errors:
Internal compiler error: Error reporting routines re-entered.
xg++: internal compiler error: Abort trap: 6 (program cc1plus)

randomly with a serial...

make -k check RUNTESTFLAGS="pch.exp=pch.C --target_board=unix'{-m32,-m64}'"

on x86_64-apple-darwin14.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[egall at gwmail dot gwu.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

Eric Gallager <egall at gwmail dot gwu.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |egall at gwmail dot gwu.edu

--- Comment #5 from Eric Gallager <egall at gwmail dot gwu.edu> ---
Does the -Winvalid-pch flag give any extra information?

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #6 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
> Does the -Winvalid-pch flag give any extra information?

AFAICT no.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #7 from howarth at bromo dot med.uc.edu ---
Is there any sensible approach we can use to try to obtain a backtrace?

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #8 from howarth at bromo dot med.uc.edu ---
I also noticed that the ICE...

FAIL: g++.dg/pch/pch.C  -O2 -g -I. -Dwith_PCH (internal compiler error)
FAIL: g++.dg/pch/pch.C  -O2 -g -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C  -O2 -g assembly comparison

doesn't seem to produce a log for the crashing compiler in
~/Library/Logs/CrashReporter whereas the crashing testcase executables do. Is
there a trick to building the compiler so that when it crashes a CrashLog will
be produced?

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #9 from howarth at bromo dot med.uc.edu ---
Unfortunately it doesn't look like the -d option...

`H'
Produce a core dump whenever an error occurs. 

is functional on darwin.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #10 from howarth at bromo dot med.uc.edu ---
The crashing compiler does leave a crash log in
~/Library/Logs/DiagnosticReports but unfortunately these show...

Backtrace not available

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #11 from howarth at bromo dot med.uc.edu ---
It appears that -dH does indeed produce a core in /cores. Using...

gdb /sw/src/fink.build/gcc5-5.0.0-1/darwin_objdir/gcc/cc1plus /cores/core.92484

the bt shows...

#0  0x00007fff96238286 in __pthread_kill ()
#1  0x00007fff9488042f in pthread_kill ()
#2  0x00007fff8e9c1b53 in abort ()
#3  0x0000000100f31530 in linemap_lookup (set=0x10175d000, line=651) at
../../gcc-5-20150325/libcpp/line-map.c:763

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #12 from howarth at bromo dot med.uc.edu ---
To clarify, I am able to debug the core file by executing...

make -k -j8 check RUNTESTFLAGS="pch.exp=pch.C --target_board=unix'{-dH}'"

until I see...

FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (internal compiler error)
FAIL: g++.dg/pch/pch.C  -O2 -I. -Dwith_PCH (test for excess errors)
FAIL: g++.dg/pch/pch.C  -O2 assembly comparison

at which time I find the expected core file added to /cores for use with...

gdb /sw/src/fink.build/gcc5-5.0.0-1/darwin_objdir/gcc/cc1plus /cores/core.4746

which uniformly shows...

#0  0x00007fff96238286 in __pthread_kill ()
#1  0x00007fff9488042f in pthread_kill ()
#2  0x00007fff8e9c1b53 in abort ()
#3  0x0000000100f31530 in linemap_lookup (set=0x1017590f0, line=523) at
../../gcc-5-20150325/libcpp/line-map.c:763

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #13 from howarth at bromo dot med.uc.edu ---
The offending line appears to be at...

static const struct line_map*
linemap_macro_map_lookup (struct line_maps *set, source_location line)
{
  unsigned int md, mn, mx;
  const struct line_map *cached, *result;

  if (IS_ADHOC_LOC (line))
    line = set->location_adhoc_data_map.data[line & MAX_SOURCE_LOCATION].locus;

  linemap_assert (line >= LINEMAPS_MACRO_LOWEST_LOCATION (set));  <=====

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #14 from howarth at bromo dot med.uc.edu ---
Testing https://gcc.gnu.org/ml/gcc-patches/2015-03/msg01334.html in case
correction of the overflow tests eliminates this bug.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #15 from howarth at bromo dot med.uc.edu ---
(In reply to howarth from comment #14)
> Testing https://gcc.gnu.org/ml/gcc-patches/2015-03/msg01334.html in case
> correction of the overflow tests eliminates this bug.

The proposed patch for correcting overflows doesn't eliminate this bug.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #16 from howarth at bromo dot med.uc.edu ---
(In reply to howarth from comment #15)
> (In reply to howarth from comment #14)
> > Testing https://gcc.gnu.org/ml/gcc-patches/2015-03/msg01334.html in case
> > correction of the overflow tests eliminates this bug.
> 
> The proposed patch for correcting overflows doesn't eliminate this bug.

With Jan's patch the failure still appears as...

#0  0x00007fff96238286 in __pthread_kill ()
#1  0x00007fff9488042f in pthread_kill ()
#2  0x00007fff8e9c1b53 in abort ()
#3  0x0000000100f32170 in linemap_lookup (set=0x10175e0f0, line=651) at
../../gcc-5-20150326/libcpp/line-map.c:806

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #17 from howarth at bromo dot med.uc.edu ---
Bug 65550 may be a duplicate of this one.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[howarth at bromo dot med.uc.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #18 from howarth at bromo dot med.uc.edu ---
(In reply to howarth from comment #17)
> Bug 65550 may be a duplicate of this one.

Like Bug 65550, the random ICEs in pch.exp are suppressed by building gcc trunk
with --enable-checking=release. In an unrelated issue, I see the parallel make
for this test randomly hang for both the stock build and
--enable-checking=release. I suspect that the pch.exp test harness isn't really
parallel safe.

[Bug pch/61250] Random pch failures on x86_64-apple-darwin1(3|4).

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=61250

--- Comment #19 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
> Like Bug 65550, the random ICEs in pch.exp are suppressed by building
> gcc trunk with --enable-checking=release. 

This is not true: I have seen the failure with gcc trunk configured with
--enable-checking=release.

> In an unrelated issue, I see the parallel make for this test randomly
> hang for both the stock build and --enable-checking=release.
> I suspect that the pch.exp test harness isn't really parallel safe.

I have also seen the tests hanging in serial test.

I have checked my archives and it seems that the PR appears only on darwin13
and 14.