[Bug lto/41376] New: collect2 (and maybe lto1) do not handle static libraries

[Bug lto/41376] New: collect2 (and maybe lto1) do not handle static libraries

[rguenth at gcc dot gnu dot org]

collect2 does not handle static libraries to maybe_run_lto_and_relink so it
doesn't get a chance to optimize using its content.  This for example means
that when bootstrapping with BOOT_CFLAGS=-flto libbackend.a does not benefit
from lto.


-- 
           Summary: collect2 (and maybe lto1) do not handle static libraries
           Product: gcc
           Version: lto
            Status: UNCONFIRMED
          Keywords: missed-optimization
          Severity: enhancement
          Priority: P3
         Component: lto
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: rguenth at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 (and maybe lto1) do not handle static libraries

[rguenth at gcc dot gnu dot org]

------- Comment #1 from rguenth at gcc dot gnu dot org  2009-12-11 16:50 -------
It works when you use the linker plugin


-- 

rguenth at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|0                           |1
   Last reconfirmed|0000-00-00 00:00:00         |2009-12-11 16:50:03
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[davek at gcc dot gnu dot org]

------- Comment #2 from davek at gcc dot gnu dot org  2010-04-28 13:38 -------
Quoting RG from the gcc list:

"[ ... ] Or you fix collect2 to do processing of archives and hand
lto1 the required information (it expects archive components
with LTO bytecode like archive.a@offset with offset being the
offset of the .o file with LTO bytecode inside the archive).  See
lto/lto-elf.c:lto_obj_file_open for "details"."


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[davek at gcc dot gnu dot org]

------- Comment #3 from davek at gcc dot gnu dot org  2010-04-28 23:49 -------
(In reply to comment #2)
> Quoting RG from the gcc list:
> 
> "[ ... ] Or you fix collect2 to do processing of archives and hand
> lto1 the required information (it expects archive components
> with LTO bytecode like archive.a@offset with offset being the
> offset of the .o file with LTO bytecode inside the archive).  See
> lto/lto-elf.c:lto_obj_file_open for "details"."


  Of course, without unresolved symbol info to guide us, as the linker has, we
won't know which archive members would be required in a link.  So if we do try
this in collect2, it would need to effectively recompile the entire library at
link time.

  Ow.  I think we need to get LD to help us out here.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[davek at gcc dot gnu dot org]

------- Comment #4 from davek at gcc dot gnu dot org  2010-04-29 05:28 -------
Created an attachment (id=20512)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=20512&action=view)
Extends GNU LD to parse archives for LTO.

(In reply to comment #3)
>   Ow.  I think we need to get LD to help us out here.

So this is a rough first draft of the-kind-of-thing-i-was-thinking-of.  We get
collect2 to run a dummy link early, and extract the output from the
--lto-assist flag to get a list of archive members that we need lto to
recompile for us.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[rguenther at suse dot de]

------- Comment #5 from rguenther at suse dot de  2010-04-29 08:29 -------
Subject: Re:  collect2 does not handle static libraries

On Thu, 29 Apr 2010, davek at gcc dot gnu dot org wrote:

> ------- Comment #4 from davek at gcc dot gnu dot org  2010-04-29 05:28 -------
> Created an attachment (id=20512)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=20512&action=view)
>  --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=20512&action=view)
> Extends GNU LD to parse archives for LTO.
> 
> (In reply to comment #3)
> >   Ow.  I think we need to get LD to help us out here.
> 
> So this is a rough first draft of the-kind-of-thing-i-was-thinking-of.  We get
> collect2 to run a dummy link early, and extract the output from the
> --lto-assist flag to get a list of archive members that we need lto to
> recompile for us.

If we go that way (extending LD), then it might be worthwhile to go
the full way and have it dump a complete symbol resolution file
like we get from the gold linker plugin.

Note that collect2 was intended to work with any linker (and it does
already query symbols of everything via nm), so I think we have to
emulate static archive linking properly.  Or alternatively move
static archive support completely into lto1 (though that would require
to make symbol resolution follow incremental linking semantics).

Richard.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[bmei at broadcom dot com]

------- Comment #6 from bmei at broadcom dot com  2010-05-04 16:54 -------
> So this is a rough first draft of the-kind-of-thing-i-was-thinking-of.  We get
> collect2 to run a dummy link early, and extract the output from the
> --lto-assist flag to get a list of archive members that we need lto to
> recompile for us.
> 

Well I spent some time to read into collect2/lto code and understand pro/cons
of different approaches. So far, adding --lto-assist to ld/hacking collect2
approach looks reasonable to me, though it does require gnu ld. What extra info
should be in a complete symbol resolution file?


-- 

bmei at broadcom dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bmei at broadcom dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[rguenther at suse dot de]

------- Comment #7 from rguenther at suse dot de  2010-05-05 08:55 -------
Subject: Re:  collect2 does not handle static libraries

On Tue, 4 May 2010, bmei at broadcom dot com wrote:

> ------- Comment #6 from bmei at broadcom dot com  2010-05-04 16:54 -------
> > So this is a rough first draft of the-kind-of-thing-i-was-thinking-of.  We get
> > collect2 to run a dummy link early, and extract the output from the
> > --lto-assist flag to get a list of archive members that we need lto to
> > recompile for us.
> > 
> 
> Well I spent some time to read into collect2/lto code and understand pro/cons
> of different approaches. So far, adding --lto-assist to ld/hacking collect2
> approach looks reasonable to me, though it does require gnu ld. What extra info
> should be in a complete symbol resolution file?

Basically what lto-plugin puts there.  A list of symbols for each
object files that is linked (thus, all required ar archive members)
together with the symbol resolution the linker chose (whether the
symbol was overridden, resolved to sth external, or used as the
prevailing definition).

Richard.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[bmei at broadcom dot com]

------- Comment #8 from bmei at broadcom dot com  2010-05-24 09:31 -------
I integrated Dave's patch into LD with some modification (only emit those with
LTO sections) and hacked collect2 to support that. The size gain of LTO, our
main concern, is quite limited for our application. Large amount of functions
called only once cannot be inlined across files because compiler doesn't know
whether they are referred in non-LTO compiled code (mostly hand-code assembly
in our cases). We really need full resolution file, especially
LDPR_PREVAILING_DEF_IRONLY type. I will try next to make LD emit full
resolution file. 

Since GNU LD doesn't have plugin support like GOLD. Won't any changes here be
too invasive/specific for LTO purpose to be accepted by LD? We are fine to live
with that in our private port. 


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[rguenth at gcc dot gnu dot org]

------- Comment #9 from rguenth at gcc dot gnu dot org  2010-05-24 12:15 -------
(In reply to comment #8)
> I integrated Dave's patch into LD with some modification (only emit those with
> LTO sections) and hacked collect2 to support that. The size gain of LTO, our
> main concern, is quite limited for our application. Large amount of functions
> called only once cannot be inlined across files because compiler doesn't know
> whether they are referred in non-LTO compiled code (mostly hand-code assembly
> in our cases). We really need full resolution file, especially
> LDPR_PREVAILING_DEF_IRONLY type. I will try next to make LD emit full
> resolution file.

Note that for a notable size-gain you likely need to build with
-fwhole-program (which means GCC needs to see all references to functions
or you need to mark functions which are accessed from assembly code
with the externally_visible attribute).

> Since GNU LD doesn't have plugin support like GOLD. Won't any changes here be
> too invasive/specific for LTO purpose to be accepted by LD? We are fine to live
> with that in our private port. 

I guess that outputting a full resolution file might be ok with the LD
folks as it can be a useful thing for debugging as well.  But of course
it would be their call, including details of the format.

I would also expect that a very simple approach for a collect2-only patch
would work most of the times (but when it wouldn't work it could result
in wrong-code or final linker failures).  Simply decompose lib*.a linker
arguments to their respective lib*.a@offset members for the lto1
invocation (and make lto1 ignore, not barf on non-LTO objects).

I will also try to see what incrementally doing the symbol resolution
would take.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[bmei at broadcom dot com]

------- Comment #10 from bmei at broadcom dot com  2010-05-24 13:29 -------
annotating functions with externally_visible sounds a bit difficult to
maintain. Programmer needs to know whether a function is used outside of LTO
objects. This can change over time and extra efforts are needed to keep it
correct.  It would be better if GCC can derive that info with -fwhole-program,
whether it is deal with LTO-object file only or LTO/Regular object files, since
it should have all the symbol reference information by then. 


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376

[Bug lto/41376] collect2 does not handle static libraries

[rguenther at suse dot de]

------- Comment #11 from rguenther at suse dot de  2010-05-24 17:27 -------
Subject: Re:  collect2 does not handle static libraries

On Mon, 24 May 2010, bmei at broadcom dot com wrote:

> ------- Comment #10 from bmei at broadcom dot com  2010-05-24 13:29 -------
> annotating functions with externally_visible sounds a bit difficult to
> maintain. Programmer needs to know whether a function is used outside of LTO
> objects. This can change over time and extra efforts are needed to keep it
> correct.  It would be better if GCC can derive that info with -fwhole-program,
> whether it is deal with LTO-object file only or LTO/Regular object files, since
> it should have all the symbol reference information by then. 

Indeed - at least if you are not dealing with building a shared
library or a program which needs to expose symbols to plugins
that are loaded at runtime.  Of course at the moment - if at all - the
symbol reference information is only available when using something
like the linker-plugin.

Richard.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41376