lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Toon Moene]

While compiling our Weather Forecasting code with the latest trunk, I 
got the following (don't know how long this has been a problem, as I 
haven't tried -flto recently):

lto1: internal compiler error: in lto_symtab_merge_decls_1, at 
lto-symtab.c:549
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
lto-wrapper: gfortran returned 1 exit status
/usr/snp/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/../../../../x86_64-unknown-linux-gnu/bin/ld: 
fatal error: lto-wrapper failed
collect2: ld returned 1 exit status

lto-symtab.c:549:

     524
     525 /* Helper to process the decl chain for the symbol table entry 
*SLOT.  */
     526
     527 static int
     528 lto_symtab_merge_decls_1 (void **slot, void *data ATTRIBUTE_UNUSED)
     ....
     545   /* Assert it's the only one.  */
     546   if (prevailing)
     547     for (e = prevailing->next; e; e = e->next)
     548       gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
     549                   && e->resolution != LDPR_PREVAILING_DEF);

Of course, I'd like to make a test case out of this - but what is this 
assert checking ?  Reducing from several 100's of thousands of lines of 
Fortran might be more difficult than to reason from first principles 
about how this assert might be hit.

Thanks in advance,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/
Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Richard Guenther]

On Sat, Apr 24, 2010 at 3:28 PM, Toon Moene <toon@moene.org> wrote:
> While compiling our Weather Forecasting code with the latest trunk, I got
> the following (don't know how long this has been a problem, as I haven't
> tried -flto recently):
>
> lto1: internal compiler error: in lto_symtab_merge_decls_1, at
> lto-symtab.c:549
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <http://gcc.gnu.org/bugs.html> for instructions.
> lto-wrapper: gfortran returned 1 exit status
> /usr/snp/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/../../../../x86_64-unknown-linux-gnu/bin/ld:
> fatal error: lto-wrapper failed
> collect2: ld returned 1 exit status
>
> lto-symtab.c:549:
>
>    524
>    525 /* Helper to process the decl chain for the symbol table entry *SLOT.
>  */
>    526
>    527 static int
>    528 lto_symtab_merge_decls_1 (void **slot, void *data ATTRIBUTE_UNUSED)
>    ....
>    545   /* Assert it's the only one.  */
>    546   if (prevailing)
>    547     for (e = prevailing->next; e; e = e->next)
>    548       gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
>    549                   && e->resolution != LDPR_PREVAILING_DEF);
>
> Of course, I'd like to make a test case out of this - but what is this
> assert checking ?

It is checking that for one symbol we only have one definition.

You are using -fuse-linker-plugin?

Richard.

>  Reducing from several 100's of thousands of lines of
> Fortran might be more difficult than to reason from first principles about
> how this assert might be hit.
>
> Thanks in advance,
>
> --
> Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
> Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
> At home: http://moene.org/~toon/
> Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran
>

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Toon Moene]

Richard Guenther wrote:

> On Sat, Apr 24, 2010 at 3:28 PM, Toon Moene <toon@moene.org> wrote:

>> lto-symtab.c:549:
>>
>>    524
>>    525 /* Helper to process the decl chain for the symbol table entry *SLOT.
>>  */
>>    526
>>    527 static int
>>    528 lto_symtab_merge_decls_1 (void **slot, void *data ATTRIBUTE_UNUSED)
>>    ....
>>    545   /* Assert it's the only one.  */
>>    546   if (prevailing)
>>    547     for (e = prevailing->next; e; e = e->next)
>>    548       gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
>>    549                   && e->resolution != LDPR_PREVAILING_DEF);
>>
>> Of course, I'd like to make a test case out of this - but what is this
>> assert checking ?
> 
> It is checking that for one symbol we only have one definition.
> 
> You are using -fuse-linker-plugin?

Indeed, I do (all of our code ends up in libraries - .a files - so I 
have to, to make -flto -fwhole-program be meaningful).

Is it a problem with COMMON ?  Those typically have umpteen definitions, 
which all have to match ...

Thanks in advance,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Richard Guenther]

On Sun, Apr 25, 2010 at 1:24 PM, Toon Moene <toon@moene.org> wrote:
> Richard Guenther wrote:
>
>> On Sat, Apr 24, 2010 at 3:28 PM, Toon Moene <toon@moene.org> wrote:
>
>>> lto-symtab.c:549:
>>>
>>>   524
>>>   525 /* Helper to process the decl chain for the symbol table entry
>>> *SLOT.
>>>  */
>>>   526
>>>   527 static int
>>>   528 lto_symtab_merge_decls_1 (void **slot, void *data ATTRIBUTE_UNUSED)
>>>   ....
>>>   545   /* Assert it's the only one.  */
>>>   546   if (prevailing)
>>>   547     for (e = prevailing->next; e; e = e->next)
>>>   548       gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
>>>   549                   && e->resolution != LDPR_PREVAILING_DEF);
>>>
>>> Of course, I'd like to make a test case out of this - but what is this
>>> assert checking ?
>>
>> It is checking that for one symbol we only have one definition.
>>
>> You are using -fuse-linker-plugin?
>
> Indeed, I do (all of our code ends up in libraries - .a files - so I have
> to, to make -flto -fwhole-program be meaningful).
>
> Is it a problem with COMMON ?  Those typically have umpteen definitions,
> which all have to match ...

No, gold should choose a single prevailing definition.  The issue is that
gold and the linker-plugin seem to be unmaintained.  Can you make sure
you have an up-to-date gold?  There was a gold bugfix related to the
above this month.

Richard.

> Thanks in advance,
>
> --
> Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
> Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
> At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
> Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran
>

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Steven Bosscher]

On Sun, Apr 25, 2010 at 9:26 PM, Richard Guenther
<richard.guenther@gmail.com> wrote:
>
> No, gold should choose a single prevailing definition.  The issue is that
> gold and the linker-plugin seem to be unmaintained.

Looking at the binutils list, there seems to be a lot of gold patching
going on. Why do you believe it is unmaintained?

For the plugin: is this not covered in the test suite (and if that's
the problem, then what can we do about it)?

Ciao!
Steven

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Richard Guenther]

On Sun, Apr 25, 2010 at 9:37 PM, Steven Bosscher <stevenb.gcc@gmail.com> wrote:
> On Sun, Apr 25, 2010 at 9:26 PM, Richard Guenther
> <richard.guenther@gmail.com> wrote:
>>
>> No, gold should choose a single prevailing definition.  The issue is that
>> gold and the linker-plugin seem to be unmaintained.
>
> Looking at the binutils list, there seems to be a lot of gold patching
> going on. Why do you believe it is unmaintained?

I meant that the combination of LTO + gold/-fuse-linker-plugin is
unmaintained.  Nobody is actively looking at bugreports with
that combination.

> For the plugin: is this not covered in the test suite (and if that's
> the problem, then what can we do about it)?

The testsuite does not cover -fuse-linker-plugin at all - it lacks at least
a feature test for it (not all configurations support it).  Thus, it's not
unfortunately possible to write a testcase excercising the
-fuse-linker-plugin path.

I'm somewhat uncomfortable that we have the two paths into LTO,
by using collect2 or the linker plugin.  Unfortunately the linker-plugin
is currently the only path that supports LTOing static libs.  As soon
as that issue is solved we should IMHO remove the linker-plugin path
again.

Richard.

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Dave Korn]

On 25/04/2010 21:41, Richard Guenther wrote:

> I'm somewhat uncomfortable that we have the two paths into LTO,
> by using collect2 or the linker plugin.  Unfortunately the linker-plugin
> is currently the only path that supports LTOing static libs.  As soon
> as that issue is solved we should IMHO remove the linker-plugin path
> again.

  Is there a PR open about this, or any notes anywhere?  Being as I use a
non-ELF platform and so gold is not an option, I'd be pleased to help with
making this work.

    cheers,
      DaveK

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Steven Bosscher]

On Mon, Apr 26, 2010 at 12:27 AM, Dave Korn
<dave.korn.cygwin@googlemail.com> wrote:

>  Is there a PR open about this, or any notes anywhere?  Being as I use a
> non-ELF platform and so gold is not an option, I'd be pleased to help with
> making this work.

See http://gcc.gnu.org/ml/gcc-patches/2009-11/msg00015.html for some
first steps.

I don't understand this, actually. When I look at lto-elf/lto-coff,
there seems to be AR support already. But this doesn't work,
apparently?

Ciao!
Steven

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Dave Korn]

On 25/04/2010 23:16, Steven Bosscher wrote:
> On Mon, Apr 26, 2010 at 12:27 AM, Dave Korn wrote:
> 
>>  Is there a PR open about this, or any notes anywhere?  Being as I use a
>> non-ELF platform and so gold is not an option, I'd be pleased to help with
>> making this work.
> 
> See http://gcc.gnu.org/ml/gcc-patches/2009-11/msg00015.html for some
> first steps.
> 
> I don't understand this, actually. When I look at lto-elf/lto-coff,
> there seems to be AR support already. But this doesn't work,
> apparently?

  There is support for offsetting an arbitrary way into a file before
beginning reading, which in theory would let us read an archive member given
some parsing of the archive header, but I think that we still lack any means
of reading and parsing the archive, knowing what's there, and selecting the
member we want.  So, we have support for reading a single archive member as an
object for LTOing, but we need guidance about which ones and where from.

  If I understand correctly, what we farm out to gold/lto-plugin is the task
of identifying a) which archive members are required to be pulled into the
final link depending on the undefined references found in the existing object
files and b) what offsets those members begin at.

  On the other hand, I may not understand correctly.

    cheers,
      DaveK

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Richard Guenther]

On Mon, Apr 26, 2010 at 4:25 AM, Dave Korn
<dave.korn.cygwin@googlemail.com> wrote:
> On 25/04/2010 23:16, Steven Bosscher wrote:
>> On Mon, Apr 26, 2010 at 12:27 AM, Dave Korn wrote:
>>
>>>  Is there a PR open about this, or any notes anywhere?  Being as I use a
>>> non-ELF platform and so gold is not an option, I'd be pleased to help with
>>> making this work.
>>
>> See http://gcc.gnu.org/ml/gcc-patches/2009-11/msg00015.html for some
>> first steps.
>>
>> I don't understand this, actually. When I look at lto-elf/lto-coff,
>> there seems to be AR support already. But this doesn't work,
>> apparently?
>
>  There is support for offsetting an arbitrary way into a file before
> beginning reading, which in theory would let us read an archive member given
> some parsing of the archive header, but I think that we still lack any means
> of reading and parsing the archive, knowing what's there, and selecting the
> member we want.  So, we have support for reading a single archive member as an
> object for LTOing, but we need guidance about which ones and where from.
>
>  If I understand correctly, what we farm out to gold/lto-plugin is the task
> of identifying a) which archive members are required to be pulled into the
> final link depending on the undefined references found in the existing object
> files and b) what offsets those members begin at.
>
>  On the other hand, I may not understand correctly.

That's correct.  Now I delayed hacking this all into collect2
(because I don't think that's the correct place to do it).  I first
wanted to cleanup the driver <-> lto1 interface so we do _not_
rely on collect2 to identify LTO objects but instead have lto1
do that work and communicate final link objects to the driver
back via a response file (same for -fwhopr WPA / LTRANS
stage - do not exec lto1 from lto1 but rather tell the driver
the set of LTRANS files).

That's also the only easy way to get rid of the .comm __gnu_lto_v2
marker and simply check the availability of one of the always
present LTO sections.  For ar archieves it is easy to iterate
through them and we need to re-write them anyway if we want
to support partly LTO / non-LTO objects inside an archive.

Now of course if ar support ends up to look easy and sane in
the collect2 framework then we can choose to not wait for
somebody doing the above suggested cleanups ...

Richard.

>    cheers,
>      DaveK
>
>

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Richard Guenther]

On Thu, Apr 29, 2010 at 6:11 AM, Dave Korn
<dave.korn.cygwin@googlemail.com> wrote:
> On 26/04/2010 10:46, Richard Guenther wrote:
>> On Mon, Apr 26, 2010 at 4:25 AM, Dave Korn wrote:
>
>>>  If I understand correctly, what we farm out to gold/lto-plugin is the task
>>> of identifying a) which archive members are required to be pulled into the
>>> final link depending on the undefined references found in the existing object
>>> files and b) what offsets those members begin at.
>
>> That's correct.  Now I delayed hacking this all into collect2
>> (because I don't think that's the correct place to do it).  I first
>> wanted to cleanup the driver <-> lto1 interface so we do _not_
>> rely on collect2 to identify LTO objects but instead have lto1
>> do that work and communicate final link objects to the driver
>> back via a response file (same for -fwhopr WPA / LTRANS
>> stage - do not exec lto1 from lto1 but rather tell the driver
>> the set of LTRANS files).
>>
>> That's also the only easy way to get rid of the .comm __gnu_lto_v2
>> marker and simply check the availability of one of the always
>> present LTO sections.  For ar archieves it is easy to iterate
>> through them and we need to re-write them anyway if we want
>> to support partly LTO / non-LTO objects inside an archive.
>>
>> Now of course if ar support ends up to look easy and sane in
>> the collect2 framework then we can choose to not wait for
>> somebody doing the above suggested cleanups ...
>
>  Actually, I'm about to argue that that's the correct place to do it, anyway.

Correct (I'll be working on that soon).

>  Isn't there going to be a problem that if we teach lto1 to look inside
> archives and extract members, it doesn't have the knowledge that the linker
> would have (cf. gold plugin) of which archive members will actually be needed
> in the final link, and that therefore it would have to assume any of the
> member objects might be needed and effectively recompile the entire library
> every link time?

Well, we'd then need to re-architect the symbol merging and
LTO unit read-in to properly honor linking semantics (drop
a LTO unit from an archive if it doesn't resolve any unresolved
symbols).  I don't know how easy that will be, but it shouldn't
be impossible at least.

>  I'm sketching a plan where collect2 invokes 'ld' as if to do an ordinary
> non-LTO link, but passes it a flag like "--lto-assist" which causes it to
> output a list of just the archive members that it actually needs to complete
> the link, in response file "filename.a@offset" format.  ISTM that this is the
> simplest method to avoid recompiling entire archives (sort of building a
> linker into the compiler!), and I guess I should also make it check for an LTO
> marker (whether symbol or section) and only output those members that actually
> contain any LTO data.

Yes - that would be basically a linker plugin without plugin support.
And I'd go even further and have LD provide a complete symbol
resolution set like we get from the gold linker-plugin.

That wouldn't help for old or non-gnu LDs of course.

>  Making lto1 understand archives seems logical at first, but I don't think
> it's much use without knowing which archive members we want in advance, and in
> that case the existing code that reads a single archive member by pretending
> it's an ordinary object file with a constant offset from the start of file
> marker already does all we need, or so it seems to me.

I think we should try without lto1 understanding archives first
(or we are basically re-implementing a linker in lto1).

Richard.

>    cheers,
>      DaveK
>
>

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Steven Bosscher]

On Thu, Apr 29, 2010 at 10:57 AM, Richard Guenther
<richard.guenther@gmail.com> wrote:
> Yes - that would be basically a linker plugin without plugin support.
> And I'd go even further and have LD provide a complete symbol
> resolution set like we get from the gold linker-plugin.
>
> That wouldn't help for old or non-gnu LDs of course.

Right. The way this seems to be going, we're looking at LTO support
for archives only for targets where GNU binutils is used. But what are
the alternatives? You have to somehow know what symbols you want to
extract from an archive, without implementing ld again.

What would be helpful, is when things get set up in such a way that
binutils ld is just one tool that can give you this resolution file,
but leave the option open to call another tool. That would allow us to
write a special tool for targets without binutils. I'm thinking of
course of my latest pet project, LTO for Mach-O. There is no working
Mach-O linker in binutils (or at least it's not the standard ld) but
it may be possible to just write a separate tool for Mach-O that
generates the resolution file.

Users would still need to install the extra tool, but at least it
would be possible to make things work.

Ciao!
Steven

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Richard Guenther]

On Thu, Apr 29, 2010 at 11:19 AM, Steven Bosscher <stevenb.gcc@gmail.com> wrote:
> On Thu, Apr 29, 2010 at 10:57 AM, Richard Guenther
> <richard.guenther@gmail.com> wrote:
>> Yes - that would be basically a linker plugin without plugin support.
>> And I'd go even further and have LD provide a complete symbol
>> resolution set like we get from the gold linker-plugin.
>>
>> That wouldn't help for old or non-gnu LDs of course.
>
> Right. The way this seems to be going, we're looking at LTO support
> for archives only for targets where GNU binutils is used. But what are
> the alternatives? You have to somehow know what symbols you want to
> extract from an archive, without implementing ld again.
>
> What would be helpful, is when things get set up in such a way that
> binutils ld is just one tool that can give you this resolution file,
> but leave the option open to call another tool. That would allow us to
> write a special tool for targets without binutils. I'm thinking of
> course of my latest pet project, LTO for Mach-O. There is no working
> Mach-O linker in binutils (or at least it's not the standard ld) but
> it may be possible to just write a separate tool for Mach-O that
> generates the resolution file.
>
> Users would still need to install the extra tool, but at least it
> would be possible to make things work.

Indeed.  That extra tool is usually collect2 though (so if you
can write such tool in a very portable way ...)

Richard.

> Ciao!
> Steven
>

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Jan Hubicka]

> Well, we'd then need to re-architect the symbol merging and
> LTO unit read-in to properly honor linking semantics (drop
> a LTO unit from an archive if it doesn't resolve any unresolved
> symbols).  I don't know how easy that will be, but it shouldn't
> be impossible at least.

We also should keep in mind that we really ought to be able to produce LTO .o files
without actual assembly in, so either we should not tie this too much with linking
process anyway or we need to output fake symbols into the LTO .o file when assembly
is not done.
(I guess one can just output empty variables and functions, but then .o will link
without LTO and lead to wrong code).

This is IMO quite important feature, we don't want to double compile times forever.

Honza
> 
> >  I'm sketching a plan where collect2 invokes 'ld' as if to do an ordinary
> > non-LTO link, but passes it a flag like "--lto-assist" which causes it to
> > output a list of just the archive members that it actually needs to complete
> > the link, in response file "filename.a@offset" format.  ISTM that this is the
> > simplest method to avoid recompiling entire archives (sort of building a
> > linker into the compiler!), and I guess I should also make it check for an LTO
> > marker (whether symbol or section) and only output those members that actually
> > contain any LTO data.
> 
> Yes - that would be basically a linker plugin without plugin support.
> And I'd go even further and have LD provide a complete symbol
> resolution set like we get from the gold linker-plugin.
> 
> That wouldn't help for old or non-gnu LDs of course.
> 
> >  Making lto1 understand archives seems logical at first, but I don't think
> > it's much use without knowing which archive members we want in advance, and in
> > that case the existing code that reads a single archive member by pretending
> > it's an ordinary object file with a constant offset from the start of file
> > marker already does all we need, or so it seems to me.
> 
> I think we should try without lto1 understanding archives first
> (or we are basically re-implementing a linker in lto1).
> 
> Richard.
> 
> >    cheers,
> >      DaveK
> >
> >

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Richard Guenther]

2010/4/29 Jan Hubicka <hubicka@ucw.cz>:
>> Well, we'd then need to re-architect the symbol merging and
>> LTO unit read-in to properly honor linking semantics (drop
>> a LTO unit from an archive if it doesn't resolve any unresolved
>> symbols).  I don't know how easy that will be, but it shouldn't
>> be impossible at least.
>
> We also should keep in mind that we really ought to be able to produce LTO .o files
> without actual assembly in, so either we should not tie this too much with linking
> process anyway or we need to output fake symbols into the LTO .o file when assembly
> is not done.
> (I guess one can just output empty variables and functions, but then .o will link
> without LTO and lead to wrong code).
>
> This is IMO quite important feature, we don't want to double compile times forever.

Well, what we should do anyway is short-cut compilation after
LTO bytecode output and go directly to expansion.  Otherwise
we risk to have different sets of symbols with the intermediate
object files and thus symbol resultion with GOLD does not
work compeltely (there are bugs about this already).

So compile-time wouldn't be _that_ bad (just RTL opts and
assembling).

Richard.

> Honza
>>
>> >  I'm sketching a plan where collect2 invokes 'ld' as if to do an ordinary
>> > non-LTO link, but passes it a flag like "--lto-assist" which causes it to
>> > output a list of just the archive members that it actually needs to complete
>> > the link, in response file "filename.a@offset" format.  ISTM that this is the
>> > simplest method to avoid recompiling entire archives (sort of building a
>> > linker into the compiler!), and I guess I should also make it check for an LTO
>> > marker (whether symbol or section) and only output those members that actually
>> > contain any LTO data.
>>
>> Yes - that would be basically a linker plugin without plugin support.
>> And I'd go even further and have LD provide a complete symbol
>> resolution set like we get from the gold linker-plugin.
>>
>> That wouldn't help for old or non-gnu LDs of course.
>>
>> >  Making lto1 understand archives seems logical at first, but I don't think
>> > it's much use without knowing which archive members we want in advance, and in
>> > that case the existing code that reads a single archive member by pretending
>> > it's an ordinary object file with a constant offset from the start of file
>> > marker already does all we need, or so it seems to me.
>>
>> I think we should try without lto1 understanding archives first
>> (or we are basically re-implementing a linker in lto1).
>>
>> Richard.
>>
>> >    cheers,
>> >      DaveK
>> >
>> >
>

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Jan Hubicka]

> 2010/4/29 Jan Hubicka <hubicka@ucw.cz>:
> >> Well, we'd then need to re-architect the symbol merging and
> >> LTO unit read-in to properly honor linking semantics (drop
> >> a LTO unit from an archive if it doesn't resolve any unresolved
> >> symbols).  I don't know how easy that will be, but it shouldn't
> >> be impossible at least.
> >
> > We also should keep in mind that we really ought to be able to produce LTO .o files
> > without actual assembly in, so either we should not tie this too much with linking
> > process anyway or we need to output fake symbols into the LTO .o file when assembly
> > is not done.
> > (I guess one can just output empty variables and functions, but then .o will link
> > without LTO and lead to wrong code).
> >
> > This is IMO quite important feature, we don't want to double compile times forever.
> 
> Well, what we should do anyway is short-cut compilation after
> LTO bytecode output and go directly to expansion.  Otherwise
> we risk to have different sets of symbols with the intermediate
> object files and thus symbol resultion with GOLD does not
> work compeltely (there are bugs about this already).
> 
> So compile-time wouldn't be _that_ bad (just RTL opts and
> assembling).

Well, but in that case the resulting assembly would be completely useless anyway.
RTL opts and expansion play important factor, so it would be nice to avoid those.

Honza

Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Ian Lance Taylor]

Richard Guenther <richard.guenther@gmail.com> writes:

> Well, we'd then need to re-architect the symbol merging and
> LTO unit read-in to properly honor linking semantics (drop
> a LTO unit from an archive if it doesn't resolve any unresolved
> symbols).  I don't know how easy that will be, but it shouldn't
> be impossible at least.

Yes, as I understand it, it was the need to replicate linking
semantics that led to the gold plugin framework in the first place.
(There is nothing stopping anybody from adding the same plugin
framework to GNU ld, by the way, it's just that nobody has done the
work.)  Of course all the linker semantics can be copied into the
compiler.  But it's moderately complex.

If you go this route, note that different object file formats have
different rules for when to include an object from an archive, so make
the code flexible.  E.g., in BFD, the code to select an object from an
archive is completely different for COFF and for ELF.

The Darwin linker supports a different plugin framework, by the way.
As far as I know it would be possible to adapt lto-plugin for that
framework.

Ian

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Toon Moene]

On 04/25/2010 01:24 PM, Toon Moene wrote:

> Richard Guenther wrote:

[ Concerning this assert ]

>> It is checking that for one symbol we only have one definition.
>>
>> You are using -fuse-linker-plugin?
>
> Indeed, I do (all of our code ends up in libraries - .a files - so I
> have to, to make -flto -fwhole-program be meaningful).
>
> Is it a problem with COMMON ? Those typically have umpteen definitions,
> which all have to match ...

I decided to change the following in lto-symtab.c to help me pinpoint 
this culprit:

Index: lto-symtab.c
===================================================================
--- lto-symtab.c        (revision 159389)
+++ lto-symtab.c        (working copy)
@@ -603,8 +603,15 @@
    /* Assert it's the only one.  */
    if (prevailing)
      for (e = prevailing->next; e; e = e->next)
-      gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
-                 && e->resolution != LDPR_PREVAILING_DEF);
+      {
+      if (!(e->resolution != LDPR_PREVAILING_DEF_IRONLY
+                 && e->resolution != LDPR_PREVAILING_DEF))
+        {
+         debug_tree(prevailing->decl);
+         debug_tree(e->decl);
+         gcc_assert(0);
+        }
+      }

    /* If there's not a prevailing symbol yet it's an external reference.
       Happens a lot during ltrans.  Choose the first symbol with a

That resulted in the following messages in my compile log:

  <function_decl 0x2b9d3a4af900 convect_satmixratio
     type <function_type 0x2b9d3a471000
         type <void_type 0x2b9d3548dd20 VOID
             align 8 symtab 0 alias set -1 canonical type 0x2b9d3548dd20
             pointer_to_this <pointer_type 0x2b9d3548ddc8>>
         QI
         size <integer_cst 0x2b9d3547e758 constant 8>
         unit size <integer_cst 0x2b9d3547e780 constant 1>
         align 8 symtab 0 alias set -1 canonical type 0x2b9d3a471000>
     public external QI file bkfconv.f90 line 7712 col 0 align 8>
  <function_decl 0x2b9d3a4af200 convect_satmixratio
     type <function_type 0x2b9d3a471000
         type <void_type 0x2b9d3548dd20 VOID
             align 8 symtab 0 alias set -1 canonical type 0x2b9d3548dd20
             pointer_to_this <pointer_type 0x2b9d3548ddc8>>
         QI
         size <integer_cst 0x2b9d3547e758 constant 8>
         unit size <integer_cst 0x2b9d3547e780 constant 1>
         align 8 symtab 0 alias set -1 canonical type 0x2b9d3a471000>
     public external QI file bkfconv.f90 line 6525 col 0 align 8>
lto1: internal compiler error: in lto_symtab_merge_decls_1, at 
lto-symtab.c:612

bkconv.f90:7712:

    7711 !
    7712        CALL CONVECT_SATMIXRATIO( KLON, ZWORK1, ZTELCL, ZWORK3, 
ZLV, ZLS, ZCPH )

bkfconv.f90:6525:
    6524 !
    6525      CALL CONVECT_SATMIXRATIO( KLON, ZPLCL, ZTLCL, ZWORK1, ZLV, 
ZWORK2, ZCPH )

The only thing I can see is that CONVECT_SATMIXRATIO is called from two 
different routines in one file (bkfconv.f90:6260 CONVECT_TRIGGER_SHAL 
and bkfconv.f90:7258 CONVECT_CLOSURE_SHAL)

Any suggestions how I can further chase this down ?  Are the arguments 
to the subroutine an issue (kind, array size, etc.) ?

Thanks in advance,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Richard Guenther]

On Fri, May 14, 2010 at 3:34 PM, Toon Moene <toon@moene.org> wrote:
> On 04/25/2010 01:24 PM, Toon Moene wrote:
>
>> Richard Guenther wrote:
>
> [ Concerning this assert ]
>
>>> It is checking that for one symbol we only have one definition.
>>>
>>> You are using -fuse-linker-plugin?
>>
>> Indeed, I do (all of our code ends up in libraries - .a files - so I
>> have to, to make -flto -fwhole-program be meaningful).
>>
>> Is it a problem with COMMON ? Those typically have umpteen definitions,
>> which all have to match ...
>
> I decided to change the following in lto-symtab.c to help me pinpoint this
> culprit:
>
> Index: lto-symtab.c
> ===================================================================
> --- lto-symtab.c        (revision 159389)
> +++ lto-symtab.c        (working copy)
> @@ -603,8 +603,15 @@
>   /* Assert it's the only one.  */
>   if (prevailing)
>     for (e = prevailing->next; e; e = e->next)
> -      gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
> -                 && e->resolution != LDPR_PREVAILING_DEF);
> +      {
> +      if (!(e->resolution != LDPR_PREVAILING_DEF_IRONLY
> +                 && e->resolution != LDPR_PREVAILING_DEF))
> +        {
> +         debug_tree(prevailing->decl);
> +         debug_tree(e->decl);
> +         gcc_assert(0);
> +        }
> +      }
>
>   /* If there's not a prevailing symbol yet it's an external reference.
>      Happens a lot during ltrans.  Choose the first symbol with a
>
> That resulted in the following messages in my compile log:
>
>  <function_decl 0x2b9d3a4af900 convect_satmixratio
>    type <function_type 0x2b9d3a471000
>        type <void_type 0x2b9d3548dd20 VOID
>            align 8 symtab 0 alias set -1 canonical type 0x2b9d3548dd20
>            pointer_to_this <pointer_type 0x2b9d3548ddc8>>
>        QI
>        size <integer_cst 0x2b9d3547e758 constant 8>
>        unit size <integer_cst 0x2b9d3547e780 constant 1>
>        align 8 symtab 0 alias set -1 canonical type 0x2b9d3a471000>
>    public external QI file bkfconv.f90 line 7712 col 0 align 8>
>  <function_decl 0x2b9d3a4af200 convect_satmixratio
>    type <function_type 0x2b9d3a471000
>        type <void_type 0x2b9d3548dd20 VOID
>            align 8 symtab 0 alias set -1 canonical type 0x2b9d3548dd20
>            pointer_to_this <pointer_type 0x2b9d3548ddc8>>
>        QI
>        size <integer_cst 0x2b9d3547e758 constant 8>
>        unit size <integer_cst 0x2b9d3547e780 constant 1>
>        align 8 symtab 0 alias set -1 canonical type 0x2b9d3a471000>
>    public external QI file bkfconv.f90 line 6525 col 0 align 8>
> lto1: internal compiler error: in lto_symtab_merge_decls_1, at
> lto-symtab.c:612
>
> bkconv.f90:7712:
>
>   7711 !
>   7712        CALL CONVECT_SATMIXRATIO( KLON, ZWORK1, ZTELCL, ZWORK3, ZLV,
> ZLS, ZCPH )
>
> bkfconv.f90:6525:
>   6524 !
>   6525      CALL CONVECT_SATMIXRATIO( KLON, ZPLCL, ZTLCL, ZWORK1, ZLV,
> ZWORK2, ZCPH )
>
> The only thing I can see is that CONVECT_SATMIXRATIO is called from two
> different routines in one file (bkfconv.f90:6260 CONVECT_TRIGGER_SHAL and
> bkfconv.f90:7258 CONVECT_CLOSURE_SHAL)
>
> Any suggestions how I can further chase this down ?  Are the arguments to
> the subroutine an issue (kind, array size, etc.) ?

No, the lto-plugin or gold provides you with broken symbol resolution.

Try 1) update gold, 2) file a bugreport and CC Ian and Cary.

Richard.

> Thanks in advance,
>
> --
> Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
> Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
> At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
> Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran
>

Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549

[Toon Moene]

On 05/14/2010 03:40 PM, Richard Guenther wrote:

> On Fri, May 14, 2010 at 3:34 PM, Toon Moene<toon@moene.org>  wrote:

>> On 04/25/2010 01:24 PM, Toon Moene wrote:
>>
>>> Richard Guenther wrote:
>>
>> [ Concerning this assert ]
>>
>>>> It is checking that for one symbol we only have one definition.
>>>>
>>>> You are using -fuse-linker-plugin?
>>>
>>> Indeed, I do (all of our code ends up in libraries - .a files - so I
>>> have to, to make -flto -fwhole-program be meaningful).
>>>
>>> Is it a problem with COMMON ? Those typically have umpteen definitions,
>>> which all have to match ...
>>
>> I decided to change the following in lto-symtab.c to help me pinpoint this
>> culprit:
>>
>> Index: lto-symtab.c
>> ===================================================================
>> --- lto-symtab.c        (revision 159389)
>> +++ lto-symtab.c        (working copy)
>> @@ -603,8 +603,15 @@
>>    /* Assert it's the only one.  */
>>    if (prevailing)
>>      for (e = prevailing->next; e; e = e->next)
>> -      gcc_assert (e->resolution != LDPR_PREVAILING_DEF_IRONLY
>> -&&  e->resolution != LDPR_PREVAILING_DEF);
>> +      {
>> +      if (!(e->resolution != LDPR_PREVAILING_DEF_IRONLY
>> +&&  e->resolution != LDPR_PREVAILING_DEF))
>> +        {
>> +         debug_tree(prevailing->decl);
>> +         debug_tree(e->decl);
>> +         gcc_assert(0);
>> +        }
>> +      }
>>
>>    /* If there's not a prevailing symbol yet it's an external reference.
>>       Happens a lot during ltrans.  Choose the first symbol with a
>>
>> That resulted in the following messages in my compile log:
>>
>>   <function_decl 0x2b9d3a4af900 convect_satmixratio
>>     type<function_type 0x2b9d3a471000
>>         type<void_type 0x2b9d3548dd20 VOID
>>             align 8 symtab 0 alias set -1 canonical type 0x2b9d3548dd20
>>             pointer_to_this<pointer_type 0x2b9d3548ddc8>>
>>         QI
>>         size<integer_cst 0x2b9d3547e758 constant 8>
>>         unit size<integer_cst 0x2b9d3547e780 constant 1>
>>         align 8 symtab 0 alias set -1 canonical type 0x2b9d3a471000>
>>     public external QI file bkfconv.f90 line 7712 col 0 align 8>
>>   <function_decl 0x2b9d3a4af200 convect_satmixratio
>>     type<function_type 0x2b9d3a471000
>>         type<void_type 0x2b9d3548dd20 VOID
>>             align 8 symtab 0 alias set -1 canonical type 0x2b9d3548dd20
>>             pointer_to_this<pointer_type 0x2b9d3548ddc8>>
>>         QI
>>         size<integer_cst 0x2b9d3547e758 constant 8>
>>         unit size<integer_cst 0x2b9d3547e780 constant 1>
>>         align 8 symtab 0 alias set -1 canonical type 0x2b9d3a471000>
>>     public external QI file bkfconv.f90 line 6525 col 0 align 8>
>> lto1: internal compiler error: in lto_symtab_merge_decls_1, at
>> lto-symtab.c:612
>>
>> bkconv.f90:7712:
>>
>>    7711 !
>>    7712        CALL CONVECT_SATMIXRATIO( KLON, ZWORK1, ZTELCL, ZWORK3, ZLV,
>> ZLS, ZCPH )
>>
>> bkfconv.f90:6525:
>>    6524 !
>>    6525      CALL CONVECT_SATMIXRATIO( KLON, ZPLCL, ZTLCL, ZWORK1, ZLV,
>> ZWORK2, ZCPH )
>>
>> The only thing I can see is that CONVECT_SATMIXRATIO is called from two
>> different routines in one file (bkfconv.f90:6260 CONVECT_TRIGGER_SHAL and
>> bkfconv.f90:7258 CONVECT_CLOSURE_SHAL)
>>
>> Any suggestions how I can further chase this down ?  Are the arguments to
>> the subroutine an issue (kind, array size, etc.) ?
>
> No, the lto-plugin or gold provides you with broken symbol resolution.
>
> Try 1) update gold, 2) file a bugreport and CC Ian and Cary.

The gold is relatively new (5/6/10), so I did 2 - it's PR lto/44149.

Kind regards,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/gcc-4.5/changes.html#Fortran