[Bug lto/65559] New: [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[Bug lto/65559] New: [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

            Bug ID: 65559
           Summary: [5 Regression] lto1.exe: internal compiler error: in
                    read_cgraph_and_symbols, at lto/lto.c:2947
           Product: gcc
           Version: 5.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rainer@emrich-ebersheim.de

Running the testsuite I get ICEs for lto in several places, but the ICE is
always the same:

lto1.exe: internal compiler error: in read_cgraph_and_symbols, at
lto/lto.c:2947
libbacktrace could not find executable to open

Here one example:

Executing on host:
/opt/devel/SCRATCH/tmp.5jnZ8G4weh/gcc-5.0.0/gcc-5.0.0/gcc/xgcc
-B/opt/devel/SCRATCH/tmp.5jnZ8G4weh/gcc-5.0.0/gcc-5.0.0/gcc/
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/gcc/testsuite/gcc.c-torture/execute/builtins/fprintf.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/gcc/testsuite/gcc.c-torture/execute/builtins/fprintf-lib.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/gcc/testsuite/gcc.c-torture/execute/builtins/lib/main.c
 -fno-diagnostics-show-caret -fdiagnostics-color=never  -w  -O2 -flto
-flto-partition=none  -fno-tree-loop-distribute-patterns
-Wl,--allow-multiple-definition  -lm    -o
/opt/devel/SCRATCH/tmp.5jnZ8G4weh/gcc-5.0.0/gcc-5.0.0/gcc/testsuite/gcc/fprintf.x7
   (timeout = 300)
spawn /opt/devel/SCRATCH/tmp.5jnZ8G4weh/gcc-5.0.0/gcc-5.0.0/gcc/xgcc
-B/opt/devel/SCRATCH/tmp.5jnZ8G4weh/gcc-5.0.0/gcc-5.0.0/gcc/
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/gcc/testsuite/gcc.c-torture/execute/builtins/fprintf.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/gcc/testsuite/gcc.c-torture/execute/builtins/fprintf-lib.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/gcc/testsuite/gcc.c-torture/execute/builtins/lib/main.c
-fno-diagnostics-show-caret -fdiagnostics-color=never -w -O2 -flto
-flto-partition=none -fno-tree-loop-distribute-patterns
-Wl,--allow-multiple-definition -lm -o
/opt/devel/SCRATCH/tmp.5jnZ8G4weh/gcc-5.0.0/gcc-5.0.0/gcc/testsuite/gcc/fprintf.x7
lto1.exe: internal compiler error: in read_cgraph_and_symbols, at
lto/lto.c:2947
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper.exe: fatal error:
D:\opt\devel\SCRATCH\tmp.5jnZ8G4weh\gcc-5.0.0\gcc-5.0.0\gcc\xgcc.exe returned 1
exit status
compilation terminated.
D:\opt\devel\gnu\gcc\MINGW_NT\x86_64-w64-mingw32\mingw-w64-runtime-trunk-svn\gcc-5.0.0\bin\ld.exe:
error: lto-wrapper failed
collect2.exe: error: ld returned 1 exit status
compiler exited with status 1
output is:
lto1.exe: internal compiler error: in read_cgraph_and_symbols, at
lto/lto.c:2947
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper.exe: fatal error:
D:\opt\devel\SCRATCH\tmp.5jnZ8G4weh\gcc-5.0.0\gcc-5.0.0\gcc\xgcc.exe returned 1
exit status
compilation terminated.
D:\opt\devel\gnu\gcc\MINGW_NT\x86_64-w64-mingw32\mingw-w64-runtime-trunk-svn\gcc-5.0.0\bin\ld.exe:
error: lto-wrapper failed
collect2.exe: error: ld returned 1 exit status

FAIL: gcc.c-torture/execute/builtins/fprintf.c compilation,  -O2 -flto
-flto-partition=none  (internal compiler error)

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Kai Tietz <ktietz at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2015-03-25
                 CC|                            |ktietz at gcc dot gnu.org
     Ever confirmed|0                           |1

--- Comment #1 from Kai Tietz <ktietz at gcc dot gnu.org> ---
Yeah, I noticed such failures too.
AFAI researched them, they are related to out-of-stack issues.

The problem seems to be that within lto a lot of vec-s are passed on stack and
lead for stack-limited targets easily to out-of-stack issues.

Not sure if this is here really the reason, but my gut feeling would say so.

Nevertheless I can confirm this ice

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #2 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
(In reply to Kai Tietz from comment #1)
> Yeah, I noticed such failures too.
> AFAI researched them, they are related to out-of-stack issues.
> 
> The problem seems to be that within lto a lot of vec-s are passed on stack
> and lead for stack-limited targets easily to out-of-stack issues.
> 
> Not sure if this is here really the reason, but my gut feeling would say so.
> 
> Nevertheless I can confirm this ice
This issue is new, last time I ran the whole testsuite, end of october last
year, I didn't get such ICEs. And they are present in the g++ testsuite too.
And there are a lot of them.

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[hubicka at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Jan Hubicka <hubicka at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hubicka at gcc dot gnu.org

--- Comment #3 from Jan Hubicka <hubicka at gcc dot gnu.org> ---
lto1.exe: internal compiler error: in read_cgraph_and_symbols, at
lto/lto.c:2947
does not seem like stack limit issue.

      /* True, since the plugin splits the archives.  */                        
      gcc_assert (num_objects == nfiles);                                       

so it seems there is some confussion in handling the object files.  num_objects
is the number of objects stored by linker to resolution file, while nfiles is
number of objects seen by GCC.  How they differ?

This may be either linker or simple-object bug I guess.
with --save-temps you can check the .res file and see if it looks sane.

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |5.0

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #4 from Richard Biener <rguenth at gcc dot gnu.org> ---
Needs more analysis from folks that can reproduce - see Honzas comment.

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #5 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
Created attachment 35237
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=35237&action=edit
reproducer with temporaries

$ gcc fprintf.c fprintf-lib.c main.c -fno-diagnostics-show-caret
-fdiagnostics-color=never -w -O2 -flto -flto-partition=none
-fno-tree-loop-distribute-patterns --save-temps -Wl,--allow-multiple-definition
-lm -o fprintf.x7
lto1.exe: internal compiler error: in read_cgraph_and_symbols, at
lto/lto.c:2960
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper.exe: fatal error:
D:\opt\devel\gnu\gcc\MINGW_NT\x86_64-w64-mingw32\mingw-w64-runtime-trunk-svn\gcc-5.0.0\bin\gcc.exe
returned 1 exit status
compilation terminated.
d:/opt/devel/gnu/gcc/mingw_nt/x86_64-w64-mingw32/mingw-w64-runtime-trunk-svn/gcc-5.0.0/bin/../lib/gcc/x86_64-w64-mingw32/5.0.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
lto-wrapper failed
collect2.exe: error: ld returned 1 exit status

$ gcc -v
Using built-in specs.
COLLECT_GCC=D:\opt\devel\gnu\gcc\MINGW_NT\x86_64-w64-mingw32\mingw-w64-runtime-trunk-svn\gcc-5.0.0\bin\gcc.exe
COLLECT_LTO_WRAPPER=d:/opt/devel/gnu/gcc/mingw_nt/x86_64-w64-mingw32/mingw-w64-runtime-trunk-svn/gcc-5.0.0/bin/../libexec/gcc/x86_64-w64-mingw32/5.0.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with:
../../../../../../opt/devel/gnu/src/gcc-mingw-w64/gcc-5.0.0/configure
--prefix=/opt/devel/gnu/gcc/MINGW_NT/x86_64-w64-mingw32/mingw-w64-runtime-trunk-svn/gcc-5.0.0
--with-gnu-as
--with-as=/opt/devel/gnu/gcc/MINGW_NT/x86_64-w64-mingw32/mingw-w64-runtime-trunk-svn/gcc-5.0.0/bin/as
--with-gnu-ld
--with-ld=/opt/devel/gnu/gcc/MINGW_NT/x86_64-w64-mingw32/mingw-w64-runtime-trunk-svn/gcc-5.0.0/bin/ld
--build=x86_64-w64-mingw32 --enable-threads=posix
--enable-languages=c,c++,fortran,java,lto,objc,obj-c++
--with-gmp-include=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/include
--with-gmp-lib=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/lib64
--with-mpfr-include=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/include
--with-mpfr-lib=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/lib64
--with-mpc-include=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/include
--with-mpc-lib=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/lib64
--with-isl-include=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/include
--with-isl-lib=/opt/devel/SCRATCH/tmp.ZMSUJS8SO8/install/lib64
--with-local-prefix=/opt/devel/tec/devel/MINGW_NT/x86_64-w64-mingw32/mingw-w64-runtime-trunk-svn/gcc-5.0.0
--enable-libgomp --enable-fully-dynamic-string --disable-multilib
--enable-checking=release --disable-werror --with-sysroot=/x86_64-w64-trunk
Thread model: posix
gcc version 5.0.0 20150403 (experimental) [trunk revision 221852] (GCC)

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[hubicka at ucw dot cz]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #6 from Jan Hubicka <hubicka at ucw dot cz> ---
Can you please compile with --verbose --save-temps and attach the output +
temporary files produced?
(in particular I wonder about resolution file that should be named *.res)

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #7 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
Created attachment 35239
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=35239&action=edit
reproducer with temporaries and verbose gcc output

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #8 from Richard Biener <rguenth at gcc dot gnu.org> ---
3
fprintf.o 2
195 905fe68a PREVAILING_DEF_IRONLY main_test
233 905fe68a RESOLVED_EXEC __iob_func
fprintf-lib.o 1
263 905f8622 RESOLVED_IR inside_main
main.o 3
172 905f8490 PREVAILING_DEF main
179 905f8490 PREVAILING_DEF_IRONLY inside_main
175 905f8490 RESOLVED_IR main_test

but we have only

fprintf.o
main.o

as inputs (from ccm7oRxV).  That file is generated from lto-wrapper which
already gets fprintf-lib.o stripped somehow (cc8SwjwK).  collect2.c still
sees all inputs.

Unfortunately -Wl,-debug is missing ;)

It would be interesting to see the lto-wrapper invocation (is there sth like
strace on mingw?) and to eventually attach to lto-wrapper with a debugger...

We seem to use a linker plugin, so the error may even start there.

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #9 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
Created attachment 35244
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=35244&action=edit
reproducer with temporaries and verbose gcc output including -Wl,-debug

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #11 from Richard Biener <rguenth at gcc dot gnu.org> ---
Ok, the ld invocation still looks correct.  For some reason we don't see the
debug output from lto-wrapper or the linker plugin.  Ah - it looks for '-v',
so can you try with -Wl,-debug -Wl,-v?

Debugging the linker-plugin should work by debugging the ld.exe invocation.
>From that gdb should be able to follow forks(?)

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #12 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
Created attachment 35245
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=35245&action=edit
yet another more verbose reproducer

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #13 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
(In reply to Richard Biener from comment #11)
> Ok, the ld invocation still looks correct.  For some reason we don't see the
> debug output from lto-wrapper or the linker plugin.  Ah - it looks for '-v',
> so can you try with -Wl,-debug -Wl,-v?
The real trick is -Wl,--verbose=2.

I'm really sorry, for the next few days I don't have the time to debug this.
For me it's really time consuming, because I don't know what to look for and
even can't really operate gdb.

[Bug lto/65559] [5 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #15 from Kai Tietz <ktietz at gcc dot gnu.org> ---
That is my issue too.  I try to reproduce this issue with cross and native.
But I see some issues only in combination with upstream binutils, and here only
in native case, which is the curious part ...
I am still on to find the underlying issue ...

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|5.0                         |5.2

--- Comment #16 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
GCC 5.1 has been released.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[daniel.f.starke at freenet dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Daniel Starke <daniel.f.starke at freenet dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |daniel.f.starke at freenet dot de

--- Comment #17 from Daniel Starke <daniel.f.starke at freenet dot de> ---
I have used a different example to debug this issue. This was easier for me to
reconstruct the problem. The reason for this problem is probably the same.
Using gcc 5.1.0 and binutils 2.25 on mingw32-w64 (Windows host) I observed that
the first object file passed to lto-wrapper.exe fails to be included in the
list of object files. This happens because find_and_merge_options() in
run_gcc() fails when executing simple_object_find_section() for the section
.gnu.lto_.opts which appears to be inside the object file in question (checked
with objdump). Further investigation seems to be needed to track down why it
only fails to read the section only for some object files. Maybe some
incompatibility between the new binutils and gcc?

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #18 from Kai Tietz <ktietz at gcc dot gnu.org> ---
Does the following patch fixes your problem?

Index: lto-wrapper.c
===================================================================
--- lto-wrapper.c       (Revision 222269)
+++ lto-wrapper.c       (Arbeitskopie)
@@ -934,7 +934,7 @@ run_gcc (unsigned argc, char *argv[])
          filename[p - argv[i]] = '\0';
          file_offset = (off_t) loffset;
        }
-      fd = open (argv[i], O_RDONLY);
+      fd = open (argv[i], O_RDONLY|O_BINARY);
       if (fd == -1)
        {
          lto_argv[lto_argc++] = argv[i];

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[breedlove.matt at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Matt Breedlove <breedlove.matt at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |breedlove.matt at gmail dot com

--- Comment #19 from Matt Breedlove <breedlove.matt at gmail dot com> ---
Kai,

This corrects issue #2 from my ML post
(https://gcc.gnu.org/ml/gcc/2015-04/msg00355.html) but should this not be
something like the following instead?

fd = open (filename, O_RDONLY|O_BINARY);

With how things are currently written in the code, archive libraries will
always fail to be opened properly since the code used to parse their file names
and offsets will always chuck out the work that is being done.  Toward the top
of the loop, we're setting "char *filename = argv[i];" and then overriding it
when we hit a filename with an offset but then using the original un-parsed
command-line parameter when trying to open the file to see if it exists.

Ex:

Parameter: libarchive.a@0x42e20
Parsed as:

filename = libarchive.a
offset = 0x42e20

File opened: libarchive.a@0x42e20 (instead of libarchive.a)

>From looking at the code, this has been this way for quite awhile.  I tested
the change after modifying it to use the filename variable instead and it works
successfully rather than simply failing to open any archive passed in with an
offset.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #20 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
Kai,

(In reply to Kai Tietz from comment #18)
> Does the following patch fixes your problem?
> 
> Index: lto-wrapper.c
> ===================================================================
> --- lto-wrapper.c       (Revision 222269)
> +++ lto-wrapper.c       (Arbeitskopie)
> @@ -934,7 +934,7 @@ run_gcc (unsigned argc, char *argv[])
>           filename[p - argv[i]] = '\0';
>           file_offset = (off_t) loffset;
>         }
> -      fd = open (argv[i], O_RDONLY);
> +      fd = open (argv[i], O_RDONLY|O_BINARY);
>        if (fd == -1)
>         {
>           lto_argv[lto_argc++] = argv[i];

the following as Matt suggested

Index: lto-wrapper.c
===================================================================
--- lto-wrapper.c       (Revision 222611)
+++ lto-wrapper.c       (Arbeitskopie)
@@ -934,7 +934,7 @@ run_gcc (unsigned argc, char *argv[])
          filename[p - argv[i]] = '\0';
          file_offset = (off_t) loffset;
        }
-      fd = open (argv[i], O_RDONLY);
+      fd = open (filename, O_RDONLY|O_BINARY);
       if (fd == -1)
        {
          lto_argv[lto_argc++] = argv[i];


fixes the issue and not only this.

I did a full native x86_64-w64-mingw32 bootstrap for c,c++ and ran the
testsuite. A quick inspection shows that most lto failures are gone.

gcc-5.1.0
                === gcc Summary ===

# of expected passes            106113
# of unexpected failures        1809
# of unexpected successes       20
# of expected failures          282
# of unresolved testcases       1242
# of unsupported tests          1940 

gcc-5.1.1 revision 222611 patch applied
                === gcc Summary ===

# of expected passes            108435
# of unexpected failures        839
# of unexpected successes       21
# of expected failures          283
# of unresolved testcases       21
# of unsupported tests          1907 

I have to look at the related PRs still.

I can't test on Linux at the moment. Has somebody the time to do a regression
test for the patch on Linux? Kai?

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #21 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
*** Bug 65582 has been marked as a duplicate of this bug. ***

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #22 from Kai Tietz <ktietz at gcc dot gnu.org> ---
I will be able to test this tomorrow (or this evening) for a linux bootstrap.

Patch tested is:

Index: lto-wrapper.c
===================================================================
--- lto-wrapper.c       (Revision 222269)
+++ lto-wrapper.c       (Arbeitskopie)
@@ -934,7 +934,7 @@ run_gcc (unsigned argc, char *argv[])
          filename[p - argv[i]] = '\0';
          file_offset = (off_t) loffset;
        }
-      fd = open (argv[i], O_RDONLY);
+      fd = open (filename, O_RDONLY | O_BINARY);
       if (fd == -1)
        {
          lto_argv[lto_argc++] = argv[i];

Honza, Jakub, when regression-test passes is it ok to apply?
The O_BINARY change is pretty obvious, but the filename vs argv[i] change
should indeed affect other targets using the @<n> decoration, too.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #23 from Richard Biener <rguenth at gcc dot gnu.org> ---
The patch looks pretty obvious to me - though I wonder if archives still work
on x86_64-linux after it (or rather I wonder how they worked before...).

I'm not aware of @ doing any magic for filenames - do they?

So - please test this with boostrap-lto on x86_64-linux as well.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #24 from Richard Biener <rguenth at gcc dot gnu.org> ---
Note that the issue should only cause option merging to be skipped for files in
archives (and that, too, on x86_64-linux).  Though compared to the 4.9 branch
we do

      fd = open (argv[i], O_RDONLY);
      if (fd == -1)
        {
          lto_argv[lto_argc++] = argv[i];
          continue;
        }

vs.

     fd = open (argv[i], O_RDONLY);
      if (fd == -1)
        continue;

so we add the file to later processing even if we failed to open it.  Thus,
does removing _that_ also fix the issue?

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #25 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
(In reply to Richard Biener from comment #24)
> Note that the issue should only cause option merging to be skipped for files
> in archives (and that, too, on x86_64-linux).  Though compared to the 4.9
> branch
> we do
> 
>       fd = open (argv[i], O_RDONLY);
>       if (fd == -1)
>         {
>           lto_argv[lto_argc++] = argv[i];
>           continue;
>         }
> 
> vs.
> 
>      fd = open (argv[i], O_RDONLY);
>       if (fd == -1)
>         continue;
> 
> so we add the file to later processing even if we failed to open it.  Thus,
> does removing _that_ also fix the issue?

Native bootstrap for c,c++ started on x86_64-w64-mingw32. I will run the
testsuite afterwards. Results expected in about 4 hours.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #26 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
(In reply to Rainer Emrich from comment #25)
> (In reply to Richard Biener from comment #24)
> > Note that the issue should only cause option merging to be skipped for files
> > in archives (and that, too, on x86_64-linux).  Though compared to the 4.9
> > branch
> > we do
> > 
> >       fd = open (argv[i], O_RDONLY);
> >       if (fd == -1)
> >         {
> >           lto_argv[lto_argc++] = argv[i];
> >           continue;
> >         }
> > 
> > vs.
> > 
> >      fd = open (argv[i], O_RDONLY);
> >       if (fd == -1)
> >         continue;
> > 
> > so we add the file to later processing even if we failed to open it.  Thus,
> > does removing _that_ also fix the issue?
> 
> Native bootstrap for c,c++ started on x86_64-w64-mingw32. I will run the
> testsuite afterwards. Results expected in about 4 hours.

Tested patch:

Index: gcc/lto-wrapper.c
===================================================================
--- gcc/lto-wrapper.c   (Revision 222611)
+++ gcc/lto-wrapper.c   (Arbeitskopie)
@@ -936,10 +936,7 @@ run_gcc (unsigned argc, char *argv[])
        }
       fd = open (argv[i], O_RDONLY);
       if (fd == -1)
-       {
-         lto_argv[lto_argc++] = argv[i];
-         continue;
-       }
+       continue;

       if (find_and_merge_options (fd, file_offset, LTO_SECTION_NAME_PREFIX,
                                  &fdecoded_options, &fdecoded_options_count,



Doesn't help, lto failures back again!

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[breedlove.matt at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #27 from Matt Breedlove <breedlove.matt at gmail dot com> ---
The primary differences between the 4.9 branch and 5.1.0 are only that the
underlying issue now results in a fatal error.  In the 4.9 branch, the objects
and archives (even "-fresolution=libarchive.res" verbatim) are opened to see
whether they exist and if so, then are parsed for LTO options.  In 4.9, if they
fail to be opened as files *or* if they fail have parse-able LTO section, they
are still passed along the command-line.

Currently in 5.1.0 (without patches listed here), if a file does not exist
(such as an archive with an offset since argv[i] is used) then it is not parsed
for LTO sections and passed along as-in.  If an object is found, then it is
parsed for LTO sections and passed along only if it has those sections.  The
real question is what behavior should be given if a non-LTO object is passed
along with an LTO build.  This wasn't a problem before because in 4.9 and
prior, even when it failed to parse LTO sections for that file, the file would
still be passed along on the command-line.  With the current functionality, I
believe failing to parse the file and failing to find LTO sections within the
file are essentially equivalent in which case the file is not passed along any
longer.

The problem with Richard's patch seems opposite to me.  Since lto_argv is used
as a container to hold the objects/archives that are passed along into the
output, these should likely contain all objects/archives whether they have LTO
options with them or not or the result is that we're simply dropping objects. 
The reason that there weren't failures on archive files previously is precisely
because we are adding files that do not exist  explicitly to that array when we
fail to open them which has hidden the underlying issue in previous versions. 
The lines below show how these objects are being added to the output:

  /* Append the input objects and possible preceding arguments.  */
  for (i = 0; i < lto_argc; ++i)
    obstack_ptr_grow (&argv_obstack, lto_argv[i]);

Where do the objects that don't have LTO sections go?  Nowhere.  These issues
are present in previous versions but went unnoticed for the only reason that
there was no temporary lto_argv array.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[breedlove.matt at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #28 from Matt Breedlove <breedlove.matt at gmail dot com> ---
(In reply to Rainer Emrich from comment #26)
> (In reply to Rainer Emrich from comment #25)
> > (In reply to Richard Biener from comment #24)
> > > Note that the issue should only cause option merging to be skipped for files
> > > in archives (and that, too, on x86_64-linux).  Though compared to the 4.9
> > > branch
> > > we do
> > > 
> > >       fd = open (argv[i], O_RDONLY);
> > >       if (fd == -1)
> > >         {
> > >           lto_argv[lto_argc++] = argv[i];
> > >           continue;
> > >         }
> > > 
> > > vs.
> > > 
> > >      fd = open (argv[i], O_RDONLY);
> > >       if (fd == -1)
> > >         continue;
> > > 
> > > so we add the file to later processing even if we failed to open it.  Thus,
> > > does removing _that_ also fix the issue?
> > 
> > Native bootstrap for c,c++ started on x86_64-w64-mingw32. I will run the
> > testsuite afterwards. Results expected in about 4 hours.
> 
> Tested patch:
> 
> Index: gcc/lto-wrapper.c
> ===================================================================
> --- gcc/lto-wrapper.c   (Revision 222611)
> +++ gcc/lto-wrapper.c   (Arbeitskopie)
> @@ -936,10 +936,7 @@ run_gcc (unsigned argc, char *argv[])
>         }
>        fd = open (argv[i], O_RDONLY);
>        if (fd == -1)
> -       {
> -         lto_argv[lto_argc++] = argv[i];
> -         continue;
> -       }
> +       continue;
> 
>        if (find_and_merge_options (fd, file_offset, LTO_SECTION_NAME_PREFIX,
>                                   &fdecoded_options, &fdecoded_options_count,
> 
> 
> 
> Doesn't help, lto failures back again!

Rainer,

If you don't mind, what were the failures you were getting on this one or did
the original reported errors simply return?

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #29 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
(In reply to Matt Breedlove from comment #28)
> If you don't mind, what were the failures you were getting on this one or
> did the original reported errors simply return?

The failures are different now, for example:

Executing on host:
/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/xgcc
-B/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.1.1/gcc/testsuite/gcc.c-torture/execute/builtins/strcpy-chk.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.1.1/gcc/testsuite/gcc.c-torture/execute/builtins/strcpy-chk-lib.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.1.1/gcc/testsuite/gcc.c-torture/execute/builtins/lib/main.c
 -fno-diagnostics-show-caret -fdiagnostics-color=never  -w  -O2 -flto 
-fno-tree-loop-distribute-patterns -Wl,--allow-multiple-definition  -lm    -o
/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/testsuite/gcc/strcpy-chk.x10
   (timeout = 300)
spawn /opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/xgcc
-B/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.1.1/gcc/testsuite/gcc.c-torture/execute/builtins/strcpy-chk.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.1.1/gcc/testsuite/gcc.c-torture/execute/builtins/strcpy-chk-lib.c
/opt/devel/gnu/src/gcc-mingw-w64/gcc-5.1.1/gcc/testsuite/gcc.c-torture/execute/builtins/lib/main.c
-fno-diagnostics-show-caret -fdiagnostics-color=never -w -O2 -flto
-fno-tree-loop-distribute-patterns -Wl,--allow-multiple-definition -lm -o
/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/testsuite/gcc/strcpy-chk.x10
`main_test' referenced in section `.text.startup' of
D:\Users\RAINER~1.EMR\AppData\Local\Temp\ccWjFUZA.ltrans0.ltrans.o: defined in
discarded section `.text' of D:\msys64\tmp\cc0Zb6D8.o (symbol from plugin)
collect2.exe: error: ld returned 1 exit status
compiler exited with status 1
output is:
`main_test' referenced in section `.text.startup' of
D:\Users\RAINER~1.EMR\AppData\Local\Temp\ccWjFUZA.ltrans0.ltrans.o: defined in
discarded section `.text' of D:\msys64\tmp\cc0Zb6D8.o (symbol from plugin)
collect2.exe: error: ld returned 1 exit status

FAIL: gcc.c-torture/execute/builtins/strcpy-chk.c compilation,  -O2 -flto  


and a lot of failures of the following form:

Executing on host:
/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/xgcc
-B/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/
linker_plugin14720.c  -fno-diagnostics-show-caret -fdiagnostics-color=never 
-flto -fuse-linker-plugin  -lm    -o linker_plugin14720.exe    (timeout = 300)
spawn /opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/xgcc
-B/opt/devel/SCRATCH/tmp.SFHKmXSZ3g/gcc-5.1.1/gcc-5.1.1/gcc/
linker_plugin14720.c -fno-diagnostics-show-caret -fdiagnostics-color=never
-flto -fuse-linker-plugin -lm -o linker_plugin14720.exe
xgcc.exe: warning: '-x lto' after last input file has no effect
xgcc.exe: fatal error: no input files
compilation terminated.
lto-wrapper.exe: fatal error:
D:\opt\devel\SCRATCH\tmp.SFHKmXSZ3g\gcc-5.1.1\gcc-5.1.1\gcc\xgcc.exe returned 1
exit status
compilation terminated.
D:\opt\devel\gnu\gcc\MINGW_NT\x86_64-w64-mingw32\mingw-w64-runtime-trunk-svn\gcc-5.1.1\bin\ld.exe:
lto-wrapper failed
collect2.exe: error: ld returned 1 exit status
compiler exited with status 1
output is:
xgcc.exe: warning: '-x lto' after last input file has no effect
xgcc.exe: fatal error: no input files
compilation terminated.
lto-wrapper.exe: fatal error:
D:\opt\devel\SCRATCH\tmp.SFHKmXSZ3g\gcc-5.1.1\gcc-5.1.1\gcc\xgcc.exe returned 1
exit status
compilation terminated.
D:\opt\devel\gnu\gcc\MINGW_NT\x86_64-w64-mingw32\mingw-w64-runtime-trunk-svn\gcc-5.1.1\bin\ld.exe:
lto-wrapper failed
collect2.exe: error: ld returned 1 exit status


This kind of failures were also present without any patch.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rainer@emrich-ebersheim.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #30 from Rainer Emrich <rainer@emrich-ebersheim.de> ---
I'm testing the following instead:

Index: gcc/lto-wrapper.c
===================================================================
--- gcc/lto-wrapper.c   (Revision 222611)
+++ gcc/lto-wrapper.c   (Arbeitskopie)
@@ -934,12 +934,9 @@ run_gcc (unsigned argc, char *argv[])
          filename[p - argv[i]] = '\0';
          file_offset = (off_t) loffset;
        }
-      fd = open (argv[i], O_RDONLY);
+      fd = open (argv[i], O_RDONLY|O_BINARY);
       if (fd == -1)
-       {
-         lto_argv[lto_argc++] = argv[i];
-         continue;
-       }
+       continue;

       if (find_and_merge_options (fd, file_offset, LTO_SECTION_NAME_PREFIX,
                                  &fdecoded_options, &fdecoded_options_count,

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #31 from Kai Tietz <ktietz at gcc dot gnu.org> ---
(In reply to Richard Biener from comment #23)
> The patch looks pretty obvious to me - though I wonder if archives still
> work on x86_64-linux after it (or rather I wonder how they worked before...).
> 
> I'm not aware of @ doing any magic for filenames - do they?

No, me neither.

> So - please test this with boostrap-lto on x86_64-linux as well.

I did bootstrap for all languages (including lto) for x86_64-unknown-linux-gnu
without any new regressions.

Ok to apply to trunk?  Ok for release-branch 5, too?

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[breedlove.matt at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #32 from Matt Breedlove <breedlove.matt at gmail dot com> ---
I've been building with just the corrected open call and have successfully
built LTO versions of binutils, mpc, mpfr, isl, libiconv, bzip2, zlib, etc
without any issue but have had issues on doing a bootstrap-lto with it.

I'll try the last patch and do a native bootstrap-lto build with it and report
back shortly.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[breedlove.matt at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #33 from Matt Breedlove <breedlove.matt at gmail dot com> ---
bootstrap-lto fails to build:

`isl_union_map_read_from_str' referenced in section `.text' of
M:\msys64\tmp\cceOtKWH.ltrans0.ltrans.o: defined in discarded section `.text'
of isl_input.o (symbol from plugin)
`isl_union_map_read_from_str' referenced in section `.text' of
M:\msys64\tmp\cceOtKWH.ltrans0.ltrans.o: defined in discarded section `.text'
of isl_input.o (symbol from plugin)
`isl_union_map_copy' referenced in section `.text' of
M:\msys64\tmp\cceOtKWH.ltrans0.ltrans.o: defined in discarded section 
...


Replacing the lto-wrapper with the one from my previous build makes these
errors disappear.  The patch from my previous build was the following:

--- gcc-5.1.0/gcc/lto-wrapper.c.orig    2015-05-01 16:38:03.348243300 -0400
+++ gcc-5.1.0/gcc/lto-wrapper.c 2015-05-02 00:30:58.345729900 -0400
@@ -934,7 +934,7 @@
          filename[p - argv[i]] = '\0';
          file_offset = (off_t) loffset;
        }
-      fd = open (argv[i], O_RDONLY);
+      fd = open (filename, O_RDONLY|O_BINARY);
       if (fd == -1)
        {
          lto_argv[lto_argc++] = argv[i];


Rainer hasn't tested this yet.  I'm going to do a full native bootstrap-lto
with this, however it was working fine on everything else I compiled.  The
issue with the last tested patch was archive files with offsets were getting
discarded from the build since we were back to not using the parsed archive
filename.

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #34 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Rainer Emrich from comment #30)
> I'm testing the following instead:
> 
> Index: gcc/lto-wrapper.c
> ===================================================================
> --- gcc/lto-wrapper.c   (Revision 222611)
> +++ gcc/lto-wrapper.c   (Arbeitskopie)
> @@ -934,12 +934,9 @@ run_gcc (unsigned argc, char *argv[])
>           filename[p - argv[i]] = '\0';
>           file_offset = (off_t) loffset;
>         }
> -      fd = open (argv[i], O_RDONLY);
> +      fd = open (argv[i], O_RDONLY|O_BINARY);
>        if (fd == -1)
> -       {
> -         lto_argv[lto_argc++] = argv[i];
> -         continue;
> -       }
> +       continue;

Note that passing filename instead of argv[i] is most certainly also good
(though I was trying to figure out if that's a separate issue or not).

> 
>        if (find_and_merge_options (fd, file_offset, LTO_SECTION_NAME_PREFIX,
>                                   &fdecoded_options, &fdecoded_options_count,

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #35 from Kai Tietz <ktietz at gcc dot gnu.org> ---
Author: ktietz
Date: Mon May  4 10:16:23 2015
New Revision: 222759

URL: https://gcc.gnu.org/viewcvs?rev=222759&root=gcc&view=rev
Log:
        PR target/65559
        * lto-wrapper.c (run_gcc): Open filename
        with in binary-mode.


Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/lto-wrapper.c

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

--- Comment #36 from Kai Tietz <ktietz at gcc dot gnu.org> ---
Author: ktietz
Date: Mon May  4 10:23:55 2015
New Revision: 222761

URL: https://gcc.gnu.org/viewcvs?rev=222761&root=gcc&view=rev
Log:
        Backmerge from trunk.

        PR lto/65559
        * lto-wrapper.c (run_gcc): Open filename
        in binary-mode.

Modified:
    branches/gcc-5-branch/gcc/ChangeLog
    branches/gcc-5-branch/gcc/lto-wrapper.c

[Bug lto/65559] [5/6 Regression] lto1.exe: internal compiler error: in read_cgraph_and_symbols, at lto/lto.c:2947

[ktietz at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65559

Kai Tietz <ktietz at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |FIXED

--- Comment #37 from Kai Tietz <ktietz at gcc dot gnu.org> ---
Fixed on trunk and on 5-branch.