Resolving LTO symbols for static library boundary

Resolving LTO symbols for static library boundary

[Allan Sandfeld Jensen]

Hello GCC

In trying to make it possible to use LTO for distro-builds of Qt, I have again 
hit the problem of static libraries. In Qt in general we for LTO rely on a 
library boundary, where LTO gets resolved when generating the library but no 
LTO-symbols are exported in the shared library. This ensure the library has a 
well defined binary compatible interface and gets LTO optimizations within 
each library. For some private libraries we use static libraries however, 
because we don't need binary compatibility, but though we don't need BC 
between Qt versions, the libraries should still be linkable with different gcc 
versions (and with different compilers). However when LTO is enabled the 
static libraries will contain definitions that depend on a single gcc version 
making it unsuitable for distribution.

One solution is to enable fat-lto object files for static libraries but that 
is both a waste of space and compile time, and disables any LTO optimization 
within the library. Ideally I would like to have the static library do LTO 
optimizations internally just like a shared library, but then exported as 
static library.

I suspect this is more of gcc task than ar/ld task, as it basically boils down 
to gcc doing for a static library what it does for shared library, but maybe 
export the result as a single combined .o file, that can then be ar'ed into a 
compatible static library.

Is this possible?

Best regards
'Allan Jensen

Re: Resolving LTO symbols for static library boundary

[Richard Biener]

On February 5, 2018 12:26:58 PM GMT+01:00, Allan Sandfeld Jensen <linux@carewolf.com> wrote:
>Hello GCC
>
>In trying to make it possible to use LTO for distro-builds of Qt, I
>have again 
>hit the problem of static libraries. In Qt in general we for LTO rely
>on a 
>library boundary, where LTO gets resolved when generating the library
>but no 
>LTO-symbols are exported in the shared library. This ensure the library
>has a 
>well defined binary compatible interface and gets LTO optimizations
>within 
>each library. For some private libraries we use static libraries
>however, 
>because we don't need binary compatibility, but though we don't need BC
>
>between Qt versions, the libraries should still be linkable with
>different gcc 
>versions (and with different compilers). However when LTO is enabled
>the 
>static libraries will contain definitions that depend on a single gcc
>version 
>making it unsuitable for distribution.
>
>One solution is to enable fat-lto object files for static libraries but
>that 
>is both a waste of space and compile time, and disables any LTO
>optimization 
>within the library. Ideally I would like to have the static library do
>LTO 
>optimizations internally just like a shared library, but then exported
>as 
>static library.
>
>I suspect this is more of gcc task than ar/ld task, as it basically
>boils down 
>to gcc doing for a static library what it does for shared library, but
>maybe 
>export the result as a single combined .o file, that can then be ar'ed
>into a 
>compatible static library.
>
>Is this possible?

Hmm. I think you could partially link the static archive contents into a single relocatable object. Or we could add a mode where you do a 1to1 LTO link of the objects and stop at the ltrans object files. You could stuff those into an archive again. 

I'm not sure how far Honza got partial LTO linking to work? 

Richard. 

>Best regards
>'Allan Jensen

Re: Resolving LTO symbols for static library boundary

[Jan Hubicka]

Dne 2018-02-05 18:44, Richard Biener napsal:
> On February 5, 2018 12:26:58 PM GMT+01:00, Allan Sandfeld Jensen
> <linux@carewolf.com> wrote:
>> Hello GCC
>> 
>> In trying to make it possible to use LTO for distro-builds of Qt, I
>> have again
>> hit the problem of static libraries. In Qt in general we for LTO rely
>> on a
>> library boundary, where LTO gets resolved when generating the library
>> but no
>> LTO-symbols are exported in the shared library. This ensure the 
>> library
>> has a
>> well defined binary compatible interface and gets LTO optimizations
>> within
>> each library. For some private libraries we use static libraries
>> however,
>> because we don't need binary compatibility, but though we don't need 
>> BC
>> 
>> between Qt versions, the libraries should still be linkable with
>> different gcc
>> versions (and with different compilers). However when LTO is enabled
>> the
>> static libraries will contain definitions that depend on a single gcc
>> version
>> making it unsuitable for distribution.
>> 
>> One solution is to enable fat-lto object files for static libraries 
>> but
>> that
>> is both a waste of space and compile time, and disables any LTO
>> optimization
>> within the library. Ideally I would like to have the static library do
>> LTO
>> optimizations internally just like a shared library, but then exported
>> as
>> static library.
>> 
>> I suspect this is more of gcc task than ar/ld task, as it basically
>> boils down
>> to gcc doing for a static library what it does for shared library, but
>> maybe
>> export the result as a single combined .o file, that can then be ar'ed
>> into a
>> compatible static library.
>> 
>> Is this possible?
> 
> Hmm. I think you could partially link the static archive contents into
> a single relocatable object. Or we could add a mode where you do a
> 1to1 LTO link of the objects and stop at the ltrans object files. You
> could stuff those into an archive again.
> 
> I'm not sure how far Honza got partial LTO linking to work?

Parital linking of lto .o files into single non-lto .o file should work 
and it will get you cross-module optimization done. The problem is that 
without resolution info compiler needs to assume that all symbols 
exported by object files are possibly referneced by the later 
incremental link and thus the code quality will definitly not be 
comparable with what you get for LTO on final binary or DSO. Still 
should be better than non-lto build.
I would be curious if it is useful for you in practice.

Honza
> 
> Richard.
> 
>> Best regards
>> 'Allan Jensen

Re: Resolving LTO symbols for static library boundary

[Allan Sandfeld Jensen]

On Dienstag, 6. Februar 2018 01:01:29 CET Jan Hubicka wrote:
> Dne 2018-02-05 18:44, Richard Biener napsal:
> > On February 5, 2018 12:26:58 PM GMT+01:00, Allan Sandfeld Jensen
> > 
> > <linux@carewolf.com> wrote:
> >> Hello GCC
> >> 
> >> In trying to make it possible to use LTO for distro-builds of Qt, I
> >> have again
> >> hit the problem of static libraries. In Qt in general we for LTO rely
> >> on a
> >> library boundary, where LTO gets resolved when generating the library
> >> but no
> >> LTO-symbols are exported in the shared library. This ensure the
> >> library
> >> has a
> >> well defined binary compatible interface and gets LTO optimizations
> >> within
> >> each library. For some private libraries we use static libraries
> >> however,
> >> because we don't need binary compatibility, but though we don't need
> >> BC
> >> 
> >> between Qt versions, the libraries should still be linkable with
> >> different gcc
> >> versions (and with different compilers). However when LTO is enabled
> >> the
> >> static libraries will contain definitions that depend on a single gcc
> >> version
> >> making it unsuitable for distribution.
> >> 
> >> One solution is to enable fat-lto object files for static libraries
> >> but
> >> that
> >> is both a waste of space and compile time, and disables any LTO
> >> optimization
> >> within the library. Ideally I would like to have the static library do
> >> LTO
> >> optimizations internally just like a shared library, but then exported
> >> as
> >> static library.
> >> 
> >> I suspect this is more of gcc task than ar/ld task, as it basically
> >> boils down
> >> to gcc doing for a static library what it does for shared library, but
> >> maybe
> >> export the result as a single combined .o file, that can then be ar'ed
> >> into a
> >> compatible static library.
> >> 
> >> Is this possible?
> > 
> > Hmm. I think you could partially link the static archive contents into
> > a single relocatable object. Or we could add a mode where you do a
> > 1to1 LTO link of the objects and stop at the ltrans object files. You
> > could stuff those into an archive again.
> > 
> > I'm not sure how far Honza got partial LTO linking to work?
> 
> Parital linking of lto .o files into single non-lto .o file should work
> and it will get you cross-module optimization done. The problem is that
> without resolution info compiler needs to assume that all symbols
> exported by object files are possibly referneced by the later
> incremental link and thus the code quality will definitly not be
> comparable with what you get for LTO on final binary or DSO. Still
> should be better than non-lto build.
> I would be curious if it is useful for you in practice.
> 
How would I do that partial link, and what are the requirements?

Best regards
'Allan