Can LTO minor version be updated in backward compatible way ?

Can LTO minor version be updated in backward compatible way ?

[Romain Geissler]

Hi,

SuSE (Martin) annonunced today that fromw now on SuSE Tumbleweed will
ship with LTO-built packages by default [1].

That's a good news, however I have a question wrt how you expect to
support LTO in the future. I have been enabling it in my company for
just few selected components and I run into trouble several times these
last years. In the LTO section you define both a major version and a
minor version, however changing any of them will result in LTO build to
fail if all binaries involved in the link don't strictly have the exact
same version. Recently in gcc 9 we went from version 8.0 to 8.1. In the
past in gcc 8 I recall I also hit a problem when it went from 7.0 to
7.1. In my case, it meant recompiling a set of let's say 100 open source
libraries, and around 30 different proprietary libraries (we use static
linking, that's why all libs have to be rebuilt each time we upgrade gcc
to the next minor version). This is still bearable at my level, I don't
have too many dependencies.

However at scale, I think this can become a problem. What will happen
when in gcc 9.3 we change the version to 8.2 ? Will Tumbleweed recompile
100% of the static libraris it ships ? What about all users of
Tumbleweed having their own private libs with LTO as well ? In my company,
I don't advocate LTO at scale (yet) because of this problem in particular:
re-building everything when we release a toolchain with an updated gcc
would be too complex.

I am totally fine with having the major version mismatch as a
showstopper for the link. People will usually not combine a gcc 8 built
binary with a gcc 9 one. However if we have made a distinction with
major vs minor, is it possible to adopt a backward compatible policy in
the minor version ? Let's say I have a shiny new gcc 9, it can combine
both LTO binaries of version 8.0 and 8.1. Maybe it can emit a warning
saying it will work in degraded mode, but at least allow the build to go
on.

If having format backward compatible constraints is too hard inside a
given major gcc release, may we can consider another alternative to
failure. If fat objects were used, and if really the two minor
versions are really incompatible, maybe we can fallback on the non-LTO
part for the old library and still the link will be successful (but not
as optimized as we would like too, most likely warnings will notify
about that).

I have no idea of the LTO format and if indeed it can easily be updated
in a backward compatible way. But I would say it would be nice if it
could, and would allow adoption for projects spread on many teams
depending on each others and unable to re-build everything at each
toolchain update.

Cheers,
Romain

[1] https://lists.opensuse.org/opensuse-factory/2019-07/msg00240.html

Re: Can LTO minor version be updated in backward compatible way ?

[Michael Matz]

Hi,

On Wed, 17 Jul 2019, Romain Geissler wrote:

> However at scale, I think this can become a problem. What will happen
> when in gcc 9.3 we change the version to 8.2 ? Will Tumbleweed recompile
> 100% of the static libraris it ships ?

Every compiler change causes the whole distro to be rebuilt.  So for us 
the LTO byte stream instability is no problem.

> What about all users of Tumbleweed having their own private libs with 
> LTO as well ?

LTO is currently not designed for this use case, you can use fat objects 
to get around the limitation, as you say, but a stable LTO byte stream is 
currently not a focus.  But with time I indeed also hope that some 
backward compatiblity can be achieved, with degrading modes like you 
suggested.

> I am totally fine with having the major version mismatch as a 
> showstopper for the link. People will usually not combine a gcc 8 built 
> binary with a gcc 9 one.

That's actually not too far off from what people will want to do in the 
future.  Say some HPC vendor ships their libs as static archives, 
containing LTO byte code compiled by gcc 9.  Then a distro user might get 
gcc 10 at some point later, and it's reasonable to expect that the HPC 
libs still are working.  We aren't there yet, but we eventually want to be 
there.


Ciao,
Michael.

Re: Can LTO minor version be updated in backward compatible way ?

[Andi Kleen]

Romain Geissler <romain.geissler@amadeus.com> writes:
>
> I have no idea of the LTO format and if indeed it can easily be updated
> in a backward compatible way. But I would say it would be nice if it
> could, and would allow adoption for projects spread on many teams
> depending on each others and unable to re-build everything at each
> toolchain update.

Right now any change to an compiler option breaks the LTO format
in subtle ways. In fact even the minor changes that are currently
done are not frequent enough to catch all such cases.

So it's unlikely to really work.

-Andi

Re: Can LTO minor version be updated in backward compatible way ?

[Jeff Law]

On 7/17/19 11:29 AM, Andi Kleen wrote:
> Romain Geissler <romain.geissler@amadeus.com> writes:
>>
>> I have no idea of the LTO format and if indeed it can easily be updated
>> in a backward compatible way. But I would say it would be nice if it
>> could, and would allow adoption for projects spread on many teams
>> depending on each others and unable to re-build everything at each
>> toolchain update.
> 
> Right now any change to an compiler option breaks the LTO format
> in subtle ways. In fact even the minor changes that are currently
> done are not frequent enough to catch all such cases.
> 
> So it's unlikely to really work.
Right and stable LTO bytecode really isn't on the radar at this time.

IMHO it's more important right now to start pushing LTO into the
mainstream for the binaries shipped by the vendors (and stripping the
LTO bits out of any static libraries/.o's shipped by the vendors).


SuSE's announcement today is quite ironic.  Red Hat's toolchain team is
planning to propose switching to LTO by default for Fedora 32 and were
working through various details yesterday.  Our proposal will almost
certainly include stripping out the LTO bits from .o's and any static
libraries.

Jeff

Re: Can LTO minor version be updated in backward compatible way ?

[Florian Weimer]

* Jeff Law:

> On 7/17/19 11:29 AM, Andi Kleen wrote:
>> Romain Geissler <romain.geissler@amadeus.com> writes:
>>>
>>> I have no idea of the LTO format and if indeed it can easily be updated
>>> in a backward compatible way. But I would say it would be nice if it
>>> could, and would allow adoption for projects spread on many teams
>>> depending on each others and unable to re-build everything at each
>>> toolchain update.
>> 
>> Right now any change to an compiler option breaks the LTO format
>> in subtle ways. In fact even the minor changes that are currently
>> done are not frequent enough to catch all such cases.
>> 
>> So it's unlikely to really work.

> Right and stable LTO bytecode really isn't on the radar at this time.

Maybe it's better to serialize the non-preprocessed source code instead.
It would need some (hash-based?) deduplication.  For #include
directives, the hash of the file would be captured for reproducibility.
Then if the initial #defines are known, the source code after processing
can be reproduced exactly.

Compressed source code is a surprisingly compact representation of a
program, usually smaller than any (compressed) IR dump.

Thanks,
Florian

Re: [EXT] Re: Can LTO minor version be updated in backward compatible way ?

[Romain Geissler]

On Thu, 18 Jul 2019, Florian Weimer wrote:

> > Right and stable LTO bytecode really isn't on the radar at this time.
>
> Maybe it's better to serialize the non-preprocessed source code instead.
> It would need some (hash-based?) deduplication.  For #include
> directives, the hash of the file would be captured for reproducibility.
> Then if the initial #defines are known, the source code after processing
> can be reproduced exactly.
>
> Compressed source code is a surprisingly compact representation of a
> program, usually smaller than any (compressed) IR dump.

Hi,

That may fly in the open source world, however I expect some vendors
shipping proprietary code might be fine with assembly/LTO representation
of their product, but not source.

It looks like from your different answers that for now it's hopeless to
expect good compatibility between minor releases. With that in mind, do
you think it might be worth implementing some kind of flag
-flto-fallback-to-fat-objects={error,warning,silent} where the default
value would be "error" (just say that we have an LTO version mismatch),
"warning" would just print the version mismatch, but fallback to fat
assembly for the conflicting libraries, and "silent" would do that same
fallback silently ? Or are we really the only users of fat LTO objects
and the only ones to face these kind of issues where rebuilding
everything all the time is not easy/possible ?

Cheers,
Romain

Re: [EXT] Re: Can LTO minor version be updated in backward compatible way ?

[Florian Weimer]

* Romain Geissler:

> That may fly in the open source world, however I expect some vendors
> shipping proprietary code might be fine with assembly/LTO
> representation of their product, but not source.

They can't ship LTO today anyway due to the format incompatibility, so
that's not really an argument against source-based LTO.

Thanks,
Florian

Re: [EXT] Re: Can LTO minor version be updated in backward compatible way ?

[Richard Biener]

On Fri, Jul 19, 2019 at 10:30 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Romain Geissler:
>
> > That may fly in the open source world, however I expect some vendors
> > shipping proprietary code might be fine with assembly/LTO
> > representation of their product, but not source.
>
> They can't ship LTO today anyway due to the format incompatibility, so
> that's not really an argument against source-based LTO.

Source-based LTO doesn't really work unless you can re-synthesize
source from the IL.  At least I don't see how you can do whole-program
analysis on source and then cut it into appropriate pieces, duplicating
some things and some not to make up for the parallel final compile step.

Richard.

> Thanks,
> Florian

Re: [EXT] Re: Can LTO minor version be updated in backward compatible way ?

[Florian Weimer]

* Richard Biener:

> On Fri, Jul 19, 2019 at 10:30 AM Florian Weimer <fweimer@redhat.com> wrote:
>>
>> * Romain Geissler:
>>
>> > That may fly in the open source world, however I expect some vendors
>> > shipping proprietary code might be fine with assembly/LTO
>> > representation of their product, but not source.
>>
>> They can't ship LTO today anyway due to the format incompatibility, so
>> that's not really an argument against source-based LTO.
>
> Source-based LTO doesn't really work unless you can re-synthesize
> source from the IL.  At least I don't see how you can do whole-program
> analysis on source and then cut it into appropriate pieces, duplicating
> some things and some not to make up for the parallel final compile step.

Oh, I meant using source code only as a portable serialization of the
program, instead of serializing unstable, compiler-specific IR.  If the
whole program does not fit into memory, the compiler will still have to
maintain on-disk data structures, but at least there wouldn't a
compatibility aspect to those anymore.

Thanks,
Florian

Re: [EXT] Re: Can LTO minor version be updated in backward compatible way ?

[Richard Biener]

On Mon, Jul 22, 2019 at 1:15 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Richard Biener:
>
> > On Fri, Jul 19, 2019 at 10:30 AM Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >> * Romain Geissler:
> >>
> >> > That may fly in the open source world, however I expect some vendors
> >> > shipping proprietary code might be fine with assembly/LTO
> >> > representation of their product, but not source.
> >>
> >> They can't ship LTO today anyway due to the format incompatibility, so
> >> that's not really an argument against source-based LTO.
> >
> > Source-based LTO doesn't really work unless you can re-synthesize
> > source from the IL.  At least I don't see how you can do whole-program
> > analysis on source and then cut it into appropriate pieces, duplicating
> > some things and some not to make up for the parallel final compile step.
>
> Oh, I meant using source code only as a portable serialization of the
> program, instead of serializing unstable, compiler-specific IR.  If the
> whole program does not fit into memory, the compiler will still have to
> maintain on-disk data structures, but at least there wouldn't a
> compatibility aspect to those anymore.

OK, but then we'd need to re-do the compile and IPA analysis stage
at each link with the appropriate frontend.  But sure, that would be
possible.

Richard.

> Thanks,
> Florian

Re: Can LTO minor version be updated in backward compatible way ?

[Martin Liška]

On 7/17/19 8:10 PM, Jeff Law wrote:
> On 7/17/19 11:29 AM, Andi Kleen wrote:
>> Romain Geissler <romain.geissler@amadeus.com> writes:
>>>
>>> I have no idea of the LTO format and if indeed it can easily be updated
>>> in a backward compatible way. But I would say it would be nice if it
>>> could, and would allow adoption for projects spread on many teams
>>> depending on each others and unable to re-build everything at each
>>> toolchain update.
>>
>> Right now any change to an compiler option breaks the LTO format
>> in subtle ways. In fact even the minor changes that are currently
>> done are not frequent enough to catch all such cases.
>>
>> So it's unlikely to really work.
> Right and stable LTO bytecode really isn't on the radar at this time.
> 
> IMHO it's more important right now to start pushing LTO into the
> mainstream for the binaries shipped by the vendors (and stripping the
> LTO bits out of any static libraries/.o's shipped by the vendors).
> 
> 
> SuSE's announcement today is quite ironic. 

Why and what is ironic about it?

> Red Hat's toolchain team is
> planning to propose switching to LTO by default for Fedora 32 and were
> working through various details yesterday.

Great!

>  Our proposal will almost
> certainly include stripping out the LTO bits from .o's and any static
> libraries.

Yes, we do it as well for now.

Martin

> 
> Jeff
> 

Re: Can LTO minor version be updated in backward compatible way ?

[Jeff Law]

On 7/22/19 8:25 AM, Martin Liška wrote:
> On 7/17/19 8:10 PM, Jeff Law wrote:
>> On 7/17/19 11:29 AM, Andi Kleen wrote:
>>> Romain Geissler <romain.geissler@amadeus.com> writes:
>>>>
>>>> I have no idea of the LTO format and if indeed it can easily be updated
>>>> in a backward compatible way. But I would say it would be nice if it
>>>> could, and would allow adoption for projects spread on many teams
>>>> depending on each others and unable to re-build everything at each
>>>> toolchain update.
>>>
>>> Right now any change to an compiler option breaks the LTO format
>>> in subtle ways. In fact even the minor changes that are currently
>>> done are not frequent enough to catch all such cases.
>>>
>>> So it's unlikely to really work.
>> Right and stable LTO bytecode really isn't on the radar at this time.
>>
>> IMHO it's more important right now to start pushing LTO into the
>> mainstream for the binaries shipped by the vendors (and stripping the
>> LTO bits out of any static libraries/.o's shipped by the vendors).
>>
>>
>> SuSE's announcement today is quite ironic. 
> 
> Why and what is ironic about it?
Sorry, you'd have to have internal context -- we'd been discussing it
within the Red Hat team for Fedora 32 the previous day.  One of the
questions that came up was whether or not any other major distributor
was shipping with LTO enabled :-)


Jeff

Re: Can LTO minor version be updated in backward compatible way ?

[Martin Liška]

On 7/22/19 4:31 PM, Jeff Law wrote:
> On 7/22/19 8:25 AM, Martin Liška wrote:
>> On 7/17/19 8:10 PM, Jeff Law wrote:
>>> On 7/17/19 11:29 AM, Andi Kleen wrote:
>>>> Romain Geissler <romain.geissler@amadeus.com> writes:
>>>>>
>>>>> I have no idea of the LTO format and if indeed it can easily be updated
>>>>> in a backward compatible way. But I would say it would be nice if it
>>>>> could, and would allow adoption for projects spread on many teams
>>>>> depending on each others and unable to re-build everything at each
>>>>> toolchain update.
>>>>
>>>> Right now any change to an compiler option breaks the LTO format
>>>> in subtle ways. In fact even the minor changes that are currently
>>>> done are not frequent enough to catch all such cases.
>>>>
>>>> So it's unlikely to really work.
>>> Right and stable LTO bytecode really isn't on the radar at this time.
>>>
>>> IMHO it's more important right now to start pushing LTO into the
>>> mainstream for the binaries shipped by the vendors (and stripping the
>>> LTO bits out of any static libraries/.o's shipped by the vendors).
>>>
>>>
>>> SuSE's announcement today is quite ironic. 
>>
>> Why and what is ironic about it?
> Sorry, you'd have to have internal context -- we'd been discussing it
> within the Red Hat team for Fedora 32 the previous day.  One of the
> questions that came up was whether or not any other major distributor
> was shipping with LTO enabled :-)

According to what I know, OpenMandriva should be using LTO:
https://www.openmandriva.org/en/news/article/openmandriva-lx-4-0-rc-released
but the package building machinery is closed, so I can't confirm that.

One another example is Gentoo:
https://github.com/InBetweenNames/gentooLTO
but it's up to users preferences.

So that I consider openSUSE Tumbleweed as a first one :P

Martin


> 
> 
> Jeff
> 

Re: Can LTO minor version be updated in backward compatible way ?

[Jeff Law]

On 7/23/19 2:49 AM, Martin Liška wrote:
> On 7/22/19 4:31 PM, Jeff Law wrote:
>> On 7/22/19 8:25 AM, Martin Liška wrote:
>>> On 7/17/19 8:10 PM, Jeff Law wrote:
>>>> On 7/17/19 11:29 AM, Andi Kleen wrote:
>>>>> Romain Geissler <romain.geissler@amadeus.com> writes:
>>>>>>
>>>>>> I have no idea of the LTO format and if indeed it can easily be updated
>>>>>> in a backward compatible way. But I would say it would be nice if it
>>>>>> could, and would allow adoption for projects spread on many teams
>>>>>> depending on each others and unable to re-build everything at each
>>>>>> toolchain update.
>>>>>
>>>>> Right now any change to an compiler option breaks the LTO format
>>>>> in subtle ways. In fact even the minor changes that are currently
>>>>> done are not frequent enough to catch all such cases.
>>>>>
>>>>> So it's unlikely to really work.
>>>> Right and stable LTO bytecode really isn't on the radar at this time.
>>>>
>>>> IMHO it's more important right now to start pushing LTO into the
>>>> mainstream for the binaries shipped by the vendors (and stripping the
>>>> LTO bits out of any static libraries/.o's shipped by the vendors).
>>>>
>>>>
>>>> SuSE's announcement today is quite ironic. 
>>>
>>> Why and what is ironic about it?
>> Sorry, you'd have to have internal context -- we'd been discussing it
>> within the Red Hat team for Fedora 32 the previous day.  One of the
>> questions that came up was whether or not any other major distributor
>> was shipping with LTO enabled :-)
> 
> According to what I know, OpenMandriva should be using LTO:
> https://www.openmandriva.org/en/news/article/openmandriva-lx-4-0-rc-released
> but the package building machinery is closed, so I can't confirm that.
> 
> One another example is Gentoo:
> https://github.com/InBetweenNames/gentooLTO
> but it's up to users preferences.>
> So that I consider openSUSE Tumbleweed as a first one :P
As do I.

Jeff

Re: Can LTO minor version be updated in backward compatible way ?

[Jeffrey Walton]

On Wed, Jul 17, 2019 at 2:10 PM Jeff Law <law@redhat.com> wrote:
>
> ...
> SuSE's announcement today is quite ironic.  Red Hat's toolchain team is
> planning to propose switching to LTO by default for Fedora 32 and were
> working through various details yesterday.  Our proposal will almost
> certainly include stripping out the LTO bits from .o's and any static
> libraries.

Be sure to include an ARMv7 test case where on source file uses a the
default arch flags, and one source file uses -march=armv7-a
-mfpu=neon. (With runtime feature checking):

for example:

    a.cpp - default flags
    b.cpp - -march=armv7-a -mfpu=neon

We can't seem to get around errors like this during link driven through GCC:

[  303s] /usr/lib/gcc/armv7hl-suse-linux-gnueabi/9/include/arm_neon.h:4835:48:
fatal error: You must enable NEON instructions (e.g.
'-mfloat-abi=softfp' '-mfpu=neon') to use these intrinsics.
[  303s]  4835 |   return (uint32x4_t)__builtin_neon_vshl_nv4si
((int32x4_t) __a, __b);
[  303s]       |                                                ^
[  303s] compilation terminated.

The only thing we have found to sidestep the problem is, disable LTO for ARM.

Jeff

Re: Can LTO minor version be updated in backward compatible way ?

[Richard Biener]

On Wed, Jul 17, 2019 at 7:30 PM Andi Kleen <ak@linux.intel.com> wrote:
>
> Romain Geissler <romain.geissler@amadeus.com> writes:
> >
> > I have no idea of the LTO format and if indeed it can easily be updated
> > in a backward compatible way. But I would say it would be nice if it
> > could, and would allow adoption for projects spread on many teams
> > depending on each others and unable to re-build everything at each
> > toolchain update.
>
> Right now any change to an compiler option breaks the LTO format
> in subtle ways.

Indeed - that one is quite awkward.  I wonder if we could try mitigating
that by streaming some hash in front of the actual data for optimization/target
nodes we can verify is correct.  Such change would be local to
optc-save-gen.awk and materialize in cl_*_stream_{in,out}.

I think that's the only place where streaming is auto-generated.

Richard.

> In fact even the minor changes that are currently
> done are not frequent enough to catch all such cases.
>
> So it's unlikely to really work.
>
> -Andi
>