Re: LTO vs static library archives [was Re: lto1: internal compiler error: in lto_symtab_merge_decls_1, at lto-symtab.c:549]

[Richard Guenther <richard.guenther@gmail.com>]

On Thu, Apr 29, 2010 at 6:11 AM, Dave Korn
<dave.korn.cygwin@googlemail.com> wrote:
> On 26/04/2010 10:46, Richard Guenther wrote:
>> On Mon, Apr 26, 2010 at 4:25 AM, Dave Korn wrote:
>
>>>  If I understand correctly, what we farm out to gold/lto-plugin is the task
>>> of identifying a) which archive members are required to be pulled into the
>>> final link depending on the undefined references found in the existing object
>>> files and b) what offsets those members begin at.
>
>> That's correct.  Now I delayed hacking this all into collect2
>> (because I don't think that's the correct place to do it).  I first
>> wanted to cleanup the driver <-> lto1 interface so we do _not_
>> rely on collect2 to identify LTO objects but instead have lto1
>> do that work and communicate final link objects to the driver
>> back via a response file (same for -fwhopr WPA / LTRANS
>> stage - do not exec lto1 from lto1 but rather tell the driver
>> the set of LTRANS files).
>>
>> That's also the only easy way to get rid of the .comm __gnu_lto_v2
>> marker and simply check the availability of one of the always
>> present LTO sections.  For ar archieves it is easy to iterate
>> through them and we need to re-write them anyway if we want
>> to support partly LTO / non-LTO objects inside an archive.
>>
>> Now of course if ar support ends up to look easy and sane in
>> the collect2 framework then we can choose to not wait for
>> somebody doing the above suggested cleanups ...
>
>  Actually, I'm about to argue that that's the correct place to do it, anyway.

Correct (I'll be working on that soon).

>  Isn't there going to be a problem that if we teach lto1 to look inside
> archives and extract members, it doesn't have the knowledge that the linker
> would have (cf. gold plugin) of which archive members will actually be needed
> in the final link, and that therefore it would have to assume any of the
> member objects might be needed and effectively recompile the entire library
> every link time?

Well, we'd then need to re-architect the symbol merging and
LTO unit read-in to properly honor linking semantics (drop
a LTO unit from an archive if it doesn't resolve any unresolved
symbols).  I don't know how easy that will be, but it shouldn't
be impossible at least.

>  I'm sketching a plan where collect2 invokes 'ld' as if to do an ordinary
> non-LTO link, but passes it a flag like "--lto-assist" which causes it to
> output a list of just the archive members that it actually needs to complete
> the link, in response file "filename.a@offset" format.  ISTM that this is the
> simplest method to avoid recompiling entire archives (sort of building a
> linker into the compiler!), and I guess I should also make it check for an LTO
> marker (whether symbol or section) and only output those members that actually
> contain any LTO data.

Yes - that would be basically a linker plugin without plugin support.
And I'd go even further and have LD provide a complete symbol
resolution set like we get from the gold linker-plugin.

That wouldn't help for old or non-gnu LDs of course.

>  Making lto1 understand archives seems logical at first, but I don't think
> it's much use without knowing which archive members we want in advance, and in
> that case the existing code that reads a single archive member by pretending
> it's an ordinary object file with a constant offset from the start of file
> marker already does all we need, or so it seems to me.

I think we should try without lto1 understanding archives first
(or we are basically re-implementing a linker in lto1).

Richard.

>    cheers,
>      DaveK
>
>