Fwd: LLVM collaboration?

Fwd: LLVM collaboration?

[Renato Golin]

Folks,

I'm about to do something I've been advised against, but since I
normally don't have good judgement, I'll risk it, because I think it's
worth it. I know some people here share my views and this is the
reason I'm writing this.


The problem

For a long time already I've been hearing on the LLVM list people
saying: "oh, ld should not accept this deprecated instruction, but we
can't change that", "that would be a good idea, but we need to talk to
the GCC guys first", and to be honest, nobody ever does.

Worst still, with Clang and LLVM getting more traction recently, and
with a lot of very interesting academic work being done, a lot of new
things are getting into LLVM first (like the sanitizers, or some
specialized pragmas) and we're dangerously close to start having
clang-extensions, which in my humble opinion, would be a nightmare.

We, on the other side of the fence, know very well how hard it is to
keep with legacy undocumented gcc-extensions, and the ARM side is
particularly filled with magical things, so I know very well how you
guys would feel if you, one day, had to start implementing clang stuff
without even participating in the original design just because someone
relies on it.

So, as far as I can see (please, correct me if I'm wrong), there are
two critical problems that we're facing right now:

1. There IS an unnecessary fence between GCC and LLVM.

License arguments are one reason why we can't share code as easily as
we would like, but there is no argument against sharing ideas,
cross-reporting bugs, helping each other implement a better
compiler/linker/assembler/libraries just because of an artificial
wall. We need to break this wall.

I rarely see GCC folks reporting bugs on our side, or people saying
"we should check with the GCC folks" actually doing it. We're not
contagious folks, you know. Talking to GCC engineers won't make me a
lesser LLVM engineer, and vice-versa.

I happen to have a very deep respect for GCC *and* for my preferred
personal license (GPLv3), but I also happen to work with LLVM, and I
like it a lot. There is no contradiction on those statements, and I
wish more people could share my opinion.

2. There are decisions that NEED to be shared.

In the past, GCC implemented a lot of extensions because the standards
weren't good enough. This has changed, but the fact that there will
always be things that don't belong on any other standard, and are very
specific to the toolchain inner workings, hasn't.

It would be beneficial to both toolchains to have a shared forum where
we could not only discuss how to solve problems better, but also keep
track of the results, so we can use it as guidelines when implementing
those features.

Further still, other compilers would certainly benefit from such
guidelines, if they want to interact with our toolchains. So, this
wouldn't be just for our sake, but also for future technologies. We
had a hard time figuring out why GCC would do this or that, and in the
end, there was always a reason (mostly good, sometimes, not so much),
but we wasted a lot of time following problems lost in translation.


The Open Source Compiler Initiative

My view is that we're unnecessarily duplicating a lot of the work to
create a powerful toolchain. The license problems won't go away, so I
don't think LLVM will ever disappear. But we're engineers, not
lawyers, so we should solve the bigger technical problem in a way that
we know how: by making things work.

For the last year or two, Clang and GCC are approaching an asymptote
as to what people believe a toolchain should be, but we won't converge
to the same solution unless we talk. If we keep our ideas enclosed
inside our own communities (who has the time to follow both gcc and
llvm lists?), we'll forever fly around the expected target and never
reach it.

To solve the technical problem of duplicated work we just need to
start talking to each other. This mailing list (or LLVM's) is not a
good place, since the traffic is huge and not every one is interested,
so I think we should have something else (another list? a web page? a
bugzilla?) where we'd record all common problems and proposals for new
features (not present in any standards), so that at least we know what
the problems are.

Getting to fix a problem or accepting a proposal would go a long way
of having them as kosher on both compilers, and that could be
considered as the standard compiler implementation, so other
compilers, even the closed source ones, should follow suit.

I'll be at the GNU Cauldron this year, feel free to come and discuss
this and other ideas. I hope to participate more in the GCC side of
things, and I wish some of you guys would do the same on our side. And
hopefully, in a few years, we'll all be on the same side.

I'll stop here, TL;DR; wise. Please, reply copying me, as I'm not
(yet) subscribing to this list.

Best Regards,
--renato

Re: LLVM collaboration?

[Diego Novillo]

On Fri, Feb 7, 2014 at 4:33 PM, Renato Golin <renato.golin@linaro.org> wrote:

> I'll be at the GNU Cauldron this year, feel free to come and discuss
> this and other ideas. I hope to participate more in the GCC side of
> things, and I wish some of you guys would do the same on our side. And
> hopefully, in a few years, we'll all be on the same side.

I think this would be worth a BoF, at the very least. Would you be
willing to propose one? I just need an abstract to get it in the
system. We still have some room left for presentations.

I think the friendly competition we have going between the two
compilers has done nothing but improve both toolchains. I agree that
we should keep it at this level. Any kind of abrasive interaction
between the two communities is a waste of everyone's time.

Both compilers have a lot to learn from each other.


Diego.

Re: LLVM collaboration?

[Renato Golin]

On 7 February 2014 21:53, Diego Novillo <dnovillo@google.com> wrote:
> I think this would be worth a BoF, at the very least. Would you be
> willing to propose one? I just need an abstract to get it in the
> system. We still have some room left for presentations.

Hi Diego,

Thanks, that'd be great!

A BoF would give us more time to discuss the issue, even though I'd
like to start the conversation a lot earlier. Plus, I have a lot more
to learn than to talk about. ;)

Something along the lines of...

* GCC and LLVM collaboration / The Open Source Compiler Initiative

With LLVM mature enough to feature as the default toolchain in some
Unix distributions, and with the inherent (and profitable) share of
solutions, ideas and code between the two, we need to start talking at
a more profound level. There will always be problems that can't be
included in any standard (language, extension, or machine-specific)
and are intrinsic to the compilation infrastructure. For those, and
other common problems, we need common solutions to at least both LLVM
and GCC, but ideally any open source (and even closed source)
toolchain. In this BoF session, we shall discuss to what extent this
collaboration can take us, how we should start and what are the next
steps to make this happen.

cheers,
--renato

Re: LLVM collaboration?

[Andrew Pinski]

On Fri, Feb 7, 2014 at 2:07 PM, Renato Golin <renato.golin@linaro.org> wrote:
> On 7 February 2014 21:53, Diego Novillo <dnovillo@google.com> wrote:
>> I think this would be worth a BoF, at the very least. Would you be
>> willing to propose one? I just need an abstract to get it in the
>> system. We still have some room left for presentations.
>
> Hi Diego,
>
> Thanks, that'd be great!
>
> A BoF would give us more time to discuss the issue, even though I'd
> like to start the conversation a lot earlier. Plus, I have a lot more
> to learn than to talk about. ;)
>
> Something along the lines of...
>
> * GCC and LLVM collaboration / The Open Source Compiler Initiative

I think it is going to called anything, it should be GNU and LLVM
collaboration since GCC does not include binutils/gdb while LLVM
includes the assembler/etc.
Again I think the problem is not knowing who to talk with rather than
an issue of having another issue or a BoF.  Many of the GCC developers
don't talk with LLVM developers and don't feel the need to.  While on
the other hand LLVM folks say report it to the GCC (though should have
been the GNU folks) but never follow through; so this is a LLVM issue
rather than a GCC issue.  GCC developers depend on the GNU binutils
and when there is bug in the linker/assembler; we report them but from
the sound of it, the LLVM developers don't.

Thanks,
Andrew

>
> With LLVM mature enough to feature as the default toolchain in some
> Unix distributions, and with the inherent (and profitable) share of
> solutions, ideas and code between the two, we need to start talking at
> a more profound level. There will always be problems that can't be
> included in any standard (language, extension, or machine-specific)
> and are intrinsic to the compilation infrastructure. For those, and
> other common problems, we need common solutions to at least both LLVM
> and GCC, but ideally any open source (and even closed source)
> toolchain. In this BoF session, we shall discuss to what extent this
> collaboration can take us, how we should start and what are the next
> steps to make this happen.
>
> cheers,
> --renato

Re: LLVM collaboration?

[Renato Golin]

On 7 February 2014 22:33, Andrew Pinski <pinskia@gmail.com> wrote:
> I think it is going to called anything, it should be GNU and LLVM
> collaboration since GCC does not include binutils/gdb while LLVM
> includes the assembler/etc.

Good point. I do mean the whole toolchain.

cheers,
--renato

Re: LLVM collaboration?

[Diego Novillo]

On Fri, Feb 7, 2014 at 5:07 PM, Renato Golin <renato.golin@linaro.org> wrote:

> * GCC and LLVM collaboration / The Open Source Compiler Initiative
>
> With LLVM mature enough to feature as the default toolchain in some
> Unix distributions, and with the inherent (and profitable) share of
> solutions, ideas and code between the two, we need to start talking at
> a more profound level. There will always be problems that can't be
> included in any standard (language, extension, or machine-specific)
> and are intrinsic to the compilation infrastructure. For those, and
> other common problems, we need common solutions to at least both LLVM
> and GCC, but ideally any open source (and even closed source)
> toolchain. In this BoF session, we shall discuss to what extent this
> collaboration can take us, how we should start and what are the next
> steps to make this happen.

Looks good.  Registered.


Thanks. Diego.

Re: LLVM collaboration?

[Andrew Pinski]

On Fri, Feb 7, 2014 at 1:53 PM, Diego Novillo <dnovillo@google.com> wrote:
> On Fri, Feb 7, 2014 at 4:33 PM, Renato Golin <renato.golin@linaro.org> wrote:
>
>> I'll be at the GNU Cauldron this year, feel free to come and discuss
>> this and other ideas. I hope to participate more in the GCC side of
>> things, and I wish some of you guys would do the same on our side. And
>> hopefully, in a few years, we'll all be on the same side.
>
> I think this would be worth a BoF, at the very least. Would you be
> willing to propose one? I just need an abstract to get it in the
> system. We still have some room left for presentations.

I still don't see any need for this extra BoF really.  They should be
handled at the sources of the extensions rather than the destination
of the extensions.  In fact I see this as weaking the computition
between the two compilers.  Things like the new attributes being added
for the kernel to use (in fact they are already implemented in sparse
is a thing which should be mentioned here) should have been talked
about the source.  HPA filed the bugs to GCC as he is an user of GCC
but not an user of LLVM, if someone in the kernel community wanted
LLVM support they would have filed the bugs there.

And then again the original message here is that GCC is not
controlling binutils (ld) and " ld should not accept this deprecated
instruction, but we
can't change that"  but you should have talked with the binutils
community rather than the GCC one since they are two separate projects
(though most folks work on both).

Thanks,
Andrew Pinski

>
> I think the friendly competition we have going between the two
> compilers has done nothing but improve both toolchains. I agree that
> we should keep it at this level. Any kind of abrasive interaction
> between the two communities is a waste of everyone's time.
>
> Both compilers have a lot to learn from each other.
>
>
> Diego.

Re: LLVM collaboration?

[Andrew Pinski]

On Fri, Feb 7, 2014 at 1:33 PM, Renato Golin <renato.golin@linaro.org> wrote:
> Folks,
>
> I'm about to do something I've been advised against, but since I
> normally don't have good judgement, I'll risk it, because I think it's
> worth it. I know some people here share my views and this is the
> reason I'm writing this.
>
>
> The problem
>
> For a long time already I've been hearing on the LLVM list people
> saying: "oh, ld should not accept this deprecated instruction, but we
> can't change that", "that would be a good idea, but we need to talk to
> the GCC guys first", and to be honest, nobody ever does.

Well you should also be talking to the binutils folks for ld issue.
GCC is less an issue here.


>
> Worst still, with Clang and LLVM getting more traction recently, and
> with a lot of very interesting academic work being done, a lot of new
> things are getting into LLVM first (like the sanitizers, or some
> specialized pragmas) and we're dangerously close to start having
> clang-extensions, which in my humble opinion, would be a nightmare.
>
> We, on the other side of the fence, know very well how hard it is to
> keep with legacy undocumented gcc-extensions, and the ARM side is
> particularly filled with magical things, so I know very well how you
> guys would feel if you, one day, had to start implementing clang stuff
> without even participating in the original design just because someone
> relies on it.

Can you give an example?  We have been cleaning these undocumented
extensions.  I think this is not a big issue but without examples, it
is harder to figure out what needs to be done.

>
> So, as far as I can see (please, correct me if I'm wrong), there are
> two critical problems that we're facing right now:
>
> 1. There IS an unnecessary fence between GCC and LLVM.

I don't see that.  What I see a GCC folks are working on GCC and the
rest of the GNU tools (including glibc) but the LLVM only work on LLVM
and when they find an issue they don't bring it up to the GCC list.

>
> License arguments are one reason why we can't share code as easily as
> we would like, but there is no argument against sharing ideas,
> cross-reporting bugs, helping each other implement a better
> compiler/linker/assembler/libraries just because of an artificial
> wall. We need to break this wall.
>
> I rarely see GCC folks reporting bugs on our side, or people saying
> "we should check with the GCC folks" actually doing it. We're not
> contagious folks, you know. Talking to GCC engineers won't make me a
> lesser LLVM engineer, and vice-versa.

That is because most of us don't need to figure out what LLVM is doing
for most of the time.   It should be up to the people who say check
with the GCC folks first to actually ask the GCC folks rather than the
other way around to figure out what LLVM needs.

>
> I happen to have a very deep respect for GCC *and* for my preferred
> personal license (GPLv3), but I also happen to work with LLVM, and I
> like it a lot. There is no contradiction on those statements, and I
> wish more people could share my opinion.
>
> 2. There are decisions that NEED to be shared.
>
> In the past, GCC implemented a lot of extensions because the standards
> weren't good enough. This has changed, but the fact that there will
> always be things that don't belong on any other standard, and are very
> specific to the toolchain inner workings, hasn't.

It depends on the extensions.  Most of people working on GCC are
working on GCC because that is the only compiler they need to work on;
LLVM is not even in most of their minds when they bring up an
extension.

An good example is asm goto, it is brought up by the Linux kernel
folks to the GCC folks.  Us as a project should not say please also
bring it up to the clang folks for discussion  GCC should implement a
good extension which is useful for the kernel folks.  The same is true
of the ifunc extension, it was brought up by the glibc folks to GCC.

>
> It would be beneficial to both toolchains to have a shared forum where
> we could not only discuss how to solve problems better, but also keep
> track of the results, so we can use it as guidelines when implementing
> those features.

Again it depends on the issue.  I don't some issues don't need an
extra discussion list and in fact get in the way of implementing
features.

>
> Further still, other compilers would certainly benefit from such
> guidelines, if they want to interact with our toolchains. So, this
> wouldn't be just for our sake, but also for future technologies. We
> had a hard time figuring out why GCC would do this or that, and in the
> end, there was always a reason (mostly good, sometimes, not so much),
> but we wasted a lot of time following problems lost in translation.
>
>
> The Open Source Compiler Initiative
>
> My view is that we're unnecessarily duplicating a lot of the work to
> create a powerful toolchain. The license problems won't go away, so I
> don't think LLVM will ever disappear. But we're engineers, not
> lawyers, so we should solve the bigger technical problem in a way that
> we know how: by making things work.
>
> For the last year or two, Clang and GCC are approaching an asymptote
> as to what people believe a toolchain should be, but we won't converge
> to the same solution unless we talk. If we keep our ideas enclosed
> inside our own communities (who has the time to follow both gcc and
> llvm lists?), we'll forever fly around the expected target and never
> reach it.

I don't see a reason why you really should not bring these issues up
to the list that are proposing the extension in the first place like
the LKML since the current new extension are being added to support
the kernel rather than anything else.  I don't think it should be
another list to solve problems that are can be solved at the source
rather.

Thanks,
Andrew Pinski

>
> To solve the technical problem of duplicated work we just need to
> start talking to each other. This mailing list (or LLVM's) is not a
> good place, since the traffic is huge and not every one is interested,
> so I think we should have something else (another list? a web page? a
> bugzilla?) where we'd record all common problems and proposals for new
> features (not present in any standards), so that at least we know what
> the problems are.
>
> Getting to fix a problem or accepting a proposal would go a long way
> of having them as kosher on both compilers, and that could be
> considered as the standard compiler implementation, so other
> compilers, even the closed source ones, should follow suit.
>
> I'll be at the GNU Cauldron this year, feel free to come and discuss
> this and other ideas. I hope to participate more in the GCC side of
> things, and I wish some of you guys would do the same on our side. And
> hopefully, in a few years, we'll all be on the same side.
>
> I'll stop here, TL;DR; wise. Please, reply copying me, as I'm not
> (yet) subscribing to this list.
>
> Best Regards,
> --renato

Re: LLVM collaboration?

[Jonathan Wakely]

On 7 February 2014 21:33, Renato Golin wrote:
>
> Worst still, with Clang and LLVM getting more traction recently, and
> with a lot of very interesting academic work being done, a lot of new
> things are getting into LLVM first (like the sanitizers, or some
> specialized pragmas) and we're dangerously close to start having
> clang-extensions, which in my humble opinion, would be a nightmare.

The sanitizers are IMHO an impressive example of collaboration. The
process may not be perfect, but the fact is that those powerful tools
are available in both compilers - I think that's amazing!

> We, on the other side of the fence, know very well how hard it is to
> keep with legacy undocumented gcc-extensions, and the ARM side is
> particularly filled with magical things, so I know very well how you
> guys would feel if you, one day, had to start implementing clang stuff
> without even participating in the original design just because someone
> relies on it.

Like the Blocks extension? :-)


> So, as far as I can see (please, correct me if I'm wrong), there are
> two critical problems that we're facing right now:
>
> 1. There IS an unnecessary fence between GCC and LLVM.
>
> License arguments are one reason why we can't share code as easily as
> we would like, but there is no argument against sharing ideas,
> cross-reporting bugs, helping each other implement a better
> compiler/linker/assembler/libraries just because of an artificial
> wall. We need to break this wall.

If there's a wall I agree we should break it (I don't see one in the
areas I work on, which I think is great).


> I rarely see GCC folks reporting bugs on our side,

For my part, I report them when I find them, but I just don't use
Clang or LLVM that much, so I don't find many (also the few things I
test often work correctly anyway!)

I expect that many GCC devs aren't reporting bugs because they're just
not using LLVM.  I don't report OpenBSD bugs either, not because I
dislike OpenBSD, I just don't use it.


> I happen to have a very deep respect for GCC *and* for my preferred
> personal license (GPLv3), but I also happen to work with LLVM, and I
> like it a lot. There is no contradiction on those statements, and I
> wish more people could share my opinion.

I'm sure many of us do.

IMHO more collaboration should help both projects, but I think there
is already more collaboration than some people realise. Development of
OpenMP, DWARF, the C++ Itanium ABI, the psABI etc. happens in the open
and is not limited to GNU devs.

For things that don't belong in any standard, such as warning options,
that's an area where the compilers may be in competition to provide a
better user-experience, so it's unsurprising that options get added to
one compiler first without discussing it with the other project. What
tends to happen with warnings is someone says "hey, clang has this
warning, we should add it too" e.g. -Wdelete-non-virtual-dtor or
-Winclude-guard, so we may end up agreeing eventually anyway.

Re: LLVM collaboration?

[Renato Golin]

On 7 February 2014 22:42, Jonathan Wakely <jwakely.gcc@gmail.com> wrote:
> The sanitizers are IMHO an impressive example of collaboration. The
> process may not be perfect, but the fact is that those powerful tools
> are available in both compilers - I think that's amazing!

I agree.


> Like the Blocks extension? :-)

So, as an example, I started a discussion about our internal
vectorizer and how we could control it from pragmas to test and report
errors. It turned out a lot bigger than I imagined, with people
defending inclusion in openMP pragmas, or implementing as C++11
annotations, and even talking about back-porting annotations for C89
code, as an extension. Seriously, that gave me the chills.

Working in the ARM debugger, compiler and now with LLVM, I had to work
around and understand GNU-isms and the contract that the kernel has
with the toolchain, that I don't think is entirely healthy. We should,
yes, have a close relationship with them, but some proposals are
easier to implement in one compiler than another, others are just
implemented because it was the quickest implementation, or generated
the smallest code, or whatever. Things that I was expecting to see in
closed-source toolchains (as I have), but not on an open source one.

At the very least, some discussion could point to defects on one or
another toolchain, as well as the kernel. We've seen a fair number of
bad code that GCC accepts in the kernel just because it can (VLAIS,
nested functions), not because it's sensible, and that's actually
making the kernel code worse in respect with the standard. Opinions
will vary, and I don't expect everyone to agree with me that those are
nasty (nor I want flame from it, please), but some consensus would be
good.


> I expect that many GCC devs aren't reporting bugs because they're just
> not using LLVM.  I don't report OpenBSD bugs either, not because I
> dislike OpenBSD, I just don't use it.

I understand that, and I take your point. I wasn't requesting every
one to use it, but to enquire about new extensions when they come your
way, as we should do when it comes our way. I'm guilty of this being
my first email to the gcc list (and I have been publicly bashed at
FOSDEM because of it, which I appreciate).


> For things that don't belong in any standard, such as warning options,
> that's an area where the compilers may be in competition to provide a
> better user-experience, so it's unsurprising that options get added to
> one compiler first without discussing it with the other project. What
> tends to happen with warnings is someone says "hey, clang has this
> warning, we should add it too" e.g. -Wdelete-non-virtual-dtor or
> -Winclude-guard, so we may end up agreeing eventually anyway.

I think you have touched a very good point: "competition to provide
the best user experience". Do we really need that?

Front-end warnings are quite easy to replicate, but some other flags
may have slightly different semantics on each compiler, and having the
user to tell the difference is cruel. Inline assembly magic and new
ASM directives are another issue that populate the kernel (and we've
been implementing all of them in our assembler to compile the kernel).
That simply won't go away ever.

I question some of the decisions, as I have questioned some of ARM's
decisions on its ABIs, as something that had a purpose, but the core
reason is gone, and we can move along. Some consensus would have
probably helped to design a better, long-lasting solution, a lot more
consensus would have halted any progress, so we have to be careful.
But I specifically don't think that extensions required by
third-parties (like the kernel) should be discussed directly with any
specific compiler, as that will perpetuate this problem.

Some kernel developers, including Linus, are very receptive to
compiling it with Clang, so new extensions will probably be discussed
with both. Now, if we require them to discuss with each community in
separate, I'm sure the user experience will be terrible when trying to
consolidate it.

I don't want us to compete, I want us to collaborate. I don't believe
LLVM is ever going to steal GCC's shine, but both will coexist, and
having a friendly coexistence would be a lot better for everyone.
Aren't we doing open source for a better world? I can't see where
competition fits into this.

As I said before, one of my main goals of working with LLVM is to make
*GCC* better. I believe that having two toolchains is better than one
for that very reason, but maintaining two completely separate and
competing toolchains is not sustainable, even for open source
projects.

cheers,
--renato

Re: Fwd: LLVM collaboration?

[Joseph S. Myers]

On Fri, 7 Feb 2014, Renato Golin wrote:

> For a long time already I've been hearing on the LLVM list people
> saying: "oh, ld should not accept this deprecated instruction, but we
> can't change that", "that would be a good idea, but we need to talk to
> the GCC guys first", and to be honest, nobody ever does.

I think there are other closely related issues, as GCC people try to work 
around issues with glibc, or vice versa, rather than coordinating what 
might be the best solution involving changes to both components, as people 
in the glibc context complain about some Linux kernel decision but have 
difficulty getting any conclusion in conjunction with Linux kernel people 
about the right way forward (or, historically, have difficulty getting 
agreement there is a problem at all - the Linux kernel community has 
tended to have less interest in supporting the details of standards than 
the people now trying to do things in GCC and glibc), as Linux kernel 
people complain about any compiler that optimizes C as a high-level 
language in ways conflicting with its use as something more like a 
portable assembler for kernel code, and as people from the various 
communities complain about issues with underlying standards such as ISO C 
and POSIX but rather less reliably engage with the standards process to 
solve those issues.

Maybe the compiler context is sufficiently separate from the others 
mentioned that there should be multiple collaboration routes for 
(compilers), (libraries / kernel), ... - but people need to be aware that 
just because something is being discussed in a compiler context doesn't 
mean that a C language extension is the right solution; it's possible 
something involving both language and library elements is right, it's 
possible collaboration with the standards process is right at an early 
stage.

(The libraries / kernel collaboration venue exists - the linux-api list, 
which was meant to be for anything about the kernel/userspace interface.  
Unfortunately, it's rarely used - most relevant kernel discussion doesn't 
go there - and I don't have time to follow linux-kernel.  We have recently 
seen several feature requests from the Linux kernel side reported to GCC 
Bugzilla, which is good - at least if there are people on the GCC side 
working on getting such things of use to the kernel implemented in a 
suitably clean way that works for what the kernel wants.)

> 2. There are decisions that NEED to be shared.
> 
> In the past, GCC implemented a lot of extensions because the standards
> weren't good enough. This has changed, but the fact that there will
> always be things that don't belong on any other standard, and are very
> specific to the toolchain inner workings, hasn't.

There are also lots of things where either (a) it would make sense to get 
something in a standard - it can be defined sensibly at the level ISO C / 
C++ deals with, or (b) the standard exists, but what gets implemented 
ignores the standard.  Some of this may be because economic incentives 
seem to get things done one way rather than another way that would 
ultimately be better for users of the languages.

To expand on (a): for a recent GCC patch there was a use for having 
popcount on the host, and I noted 
<http://gcc.gnu.org/ml/gcc-patches/2014-02/msg00305.html> how that's one 
of many integer manipulation operations lacking any form of standard C 
bindings.  Sometimes for these things we do at least have 
target-independent GCC extensions - but sometimes just target-specific 
ones, with multiple targets having built-in functions for similar things, 
nominally mapping to particular instructions, when it would be better to 
have a standard C binding for a given operation.

To expand on (b): consider the recent AVX512 GCC patches.  As is typical 
for patches enabling support for new instruction set features, they added 
a large pile of intrinsics (intrinsic headers, mapping to built-in 
functions).  The intrinsics implement a standard of sorts - shared with 
Intel's compiler, at least.  But are they really the right approach in all 
cases?  The features of AVX512 include, in particular, floating-point 
operations with rounding modes embedded in the instruction (in support of 
IEEE 754-2008 saying language standards should support a way to apply 
rounding modes to particular blocks, not just dynamic rounding modes).

There's a proposed specification for C bindings to such a feature - draft 
TS 18661-1 (WG14 N1778; JTC1 ballot ends 2014-03-05, so may well be 
published later this year).  There was some discussion of this starting 
with <http://gcc.gnu.org/ml/gcc/2012-12/msg00129.html> (discussion 
continued into Jan 2013), which I presume was motivated by the AVX512 
feature, but in the end the traditional intrinsics were the approach taken 
for supporting this feature, not anything that would allow 
architecture-independent source code to be written.  (The AVX512 feature 
combines constant rounding modes with disabling exceptions, which isn't 
ideal, but could still be used for FENV_ROUND with -fno-trapping-math.)

Now, implementation and standards can go in either order.  When you start 
with the implementation, even with a couple of implementations working 
together, there's a risk of some cases ending up as just what the 
implementation accidentally does.  When you start with a specification, 
there's a risk of other corner cases not being considered if they are only 
apparent when you try to implement something - and also that the result of 
a design-by-committee is too removed from user requirements, or otherwise 
impractical.  So it can be helpful to have feedback as features are 
implemented during standardization - but if the features go in mainline of 
the relevant tool / library, then you have compatibility issues when the 
standard changes but people were using earlier versions.  So there isn't 
one clear right answer there.

It can also sometimes be useful to have multiple implementation approaches 
providing input for a standards committee later trying to adopt the best 
from both (consider the current CPLEX work, trying to draw on both OpenMP 
and Cilkplus).  So if you do coordinate on some feature, that doesn't mean 
you need to reach agreed conclusions - it may be perfectly reasonable for 
different preferences to arise in different communities, resulting in 
different features for the same purpose being implemented (hopefully not 
actually using the same syntax for different incompatible things), so user 
experience can help determine what a future standard solution might look 
like.

As for the economic incentives point: given a new CPU feature, there's an 
obvious incentive for the CPU company to make the feature available to 
their users from C.  Doing rather more work to make such a language / 
library feature available on all architectures - but optimized on theirs 
to take advantage of the CPU feature - may be less attractive.  But it's 
probably better long-term for the C users for their programs to be more 
portable, and if there's a standard involved that can mean portability 
among implementations as well as among processor architectures.

So: we could discuss what some feature in C should look like, whether as a 
pure extension, or something to end up as a standards proposal; having 
more people looking at the feature earlier might well help spot potential 
issues.  But until someone actually tries implementing it and people start 
using it, it's quite likely the discussion could go off in directions that 
are not particularly helpful.  And if only one tool implements it, 
feedback from the other may well be of limited value.

I think an important question then is how to get more of the larger pieces 
of work done where there aren't the immediate incentives.  It would be 
useful to get something (say integer manipulation) into ISO C, but there's 
a lot of work and ad hoc extensions already exist in compilers.  I'd like 
to get more of the floating-point features (both C99/C11 Annex F/G, and 
the new draft IEEE 754-2008 bindings) implemented in GCC and glibc (and 
would be doubtful about integrating the new bindings in a future version 
of the C standard without some implementation experience) - again, it's a 
lot of work, both in time and in terms of the range of expertise required 
that would probably need multiple people doing different bits.  I'd like 
more future ISO C features implemented as they enter the draft standard 
rather than afterwards - again, significant work to be done at particular 
times, along with compatibility risks.

One thing that may limit collaboration taking place (at present, and in 
future) is simply the time involved to keep on top of the various 
discussions.  Many, perhaps most, people can barely keep up with the 
discussions for the projects they are actively involved in.  Keeping up 
with GCC, glibc, ISO C, ..., skimming things for POSIX, Linux kernel, 
binutils, GDB, ... takes up a lot of my time, even without implementing 
things for the various projects (predominantly glibc over the past couple 
of years) and other responsibilities.  I'd like to do more in various of 
these areas, to get missing standard / proposed standard features 
implemented in GCC and glibc, to ensure proposed standards and extensions 
are high quality, to get the standards to better address areas in which 
they are weak.  But there are limits to both total time available and time 
I can actually justify spending on these things.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Fwd: LLVM collaboration?

[Renato Golin]

On 7 February 2014 23:30, Joseph S. Myers <joseph@codesourcery.com> wrote:
> I think there are other closely related issues, as GCC people try to work
> around issues with glibc, or vice versa, rather than coordinating what
> might be the best solution involving changes to both components,

Hi Joseph,

Thanks for the huge email, all of it (IMHO) was spot on. I agree with
your arguments, and one of the reasons why I finally sent the email,
is that I'm starting to see all this on LLVM, too.

Because of licenses, we have to replicate libgcc, libstdc++, the
linker, etc. And in many ways, features get added to random places
because it's the easiest route, or because it's the right place to be,
even though there isn't anything controlling or monitoring the feature
in the grand scheme of things. This will, in the end, invariably take
us through the route that GNU crossed a few years back, when people
had to use radioactive suites to work on some parts of GCC.

So, I guess my email was more of a cry for help, than a request to
play nice (as some would infer). I don't think we should repeat the
same mistakes you guys did, but I also think that we have a lot to
offer, as you mention, in looking at extensions and proposing to
standards, or keeping kernel requests sane, and having a unison
argument on specific changes, and so on.

The perfect world would be if any compiler could use any assembler,
linker and libraries, interchangeably. While that may never happen, as
a long term goal, this would at least draw us a nice asymptote to
follow. As every one here and there, I don't have enough time to work
through every detail and follow all lists, but if we encourage the
cross over, or even cross posting between the two lists, we might
solve common problems without incurring in additional time wasted.

--renato

Re: Fwd: LLVM collaboration?

[Jan Hubicka]

> 1. There IS an unnecessary fence between GCC and LLVM.
> 
> License arguments are one reason why we can't share code as easily as
> we would like, but there is no argument against sharing ideas,
> cross-reporting bugs, helping each other implement a better
> compiler/linker/assembler/libraries just because of an artificial
> wall. We need to break this wall.
> 
> I rarely see GCC folks reporting bugs on our side, or people saying
> "we should check with the GCC folks" actually doing it. We're not
> contagious folks, you know. Talking to GCC engineers won't make me a
> lesser LLVM engineer, and vice-versa.

One practical experience I have with LLVM developers is sharing experiences
about getting Firefox to work with LTO with Rafael Espindola and I think it was
useful for both of us. I am definitly open to more discussion.

Lets try a specific topic that is on my TODO list for some time.

I would like to make it possible for mutliple compilers to be used to LTO a
single binary. As we are all making LTO more useful, I think it is matter of
time until people will start shipping LTO object files by default and users
will end up feeding them into different compilers or incompatible version of
the same compiler. We probably want to make this work, even thought the
cross-module optimization will not happen in this case.

The plugin interface in binutils seems to do its job well both for GCC and LLVM
and I hope that open64 and ICC will eventually join, too.

The trouble however is that one needs to pass explicit --plugin argument
specifying the particular plugin to load and so GCC ships with its own wrappers
(gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.

It may be smoother if binutils was able to load multiple plugins at once and
grab plugins from system and user installed compilers without explicit --plugin
argument.

Binutils probably should also have a way to detect LTO object files and produce
more useful diagnostic than they do now, when there is no plugin claiming them.

There are some PRs filled on the topic
http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
but not much progress on them.

I wonder if we can get this designed and implemented.

On the other hand, GCC current maintains non-plugin path for LTO that is now
only used by darwin port due to lack of plugin enabled LD there.  It seems
that liblto used by darwin is losely compatible with the plugin API, but it makes
it harder to have different compilers share it (one has to LD_PRELOAD liblto
to different one prior executing the linker?)

I wonder, is there chance to implement linker plugin API to libLTO glue or add
plugin support to native Darwin tools?

Honza

Re: Fwd: LLVM collaboration?

[Renato Golin]

Hi Jan,

I think this is a very good example where we could all collaborate
(including binutils).

I'll leave your reply intact, so that Chandler (CC'd) can get a bit
more context. I'm copying him because he (and I believe Diego) had
more contact with LTO than I had.

If I got it right, LTO today:

- needs the drivers to explicitly declare the plugin
- needs the library available somewhere
- may have to change the library loading semantics (via LD_PRELOAD)

Since both toolchains do the magic, binutils has no incentive to
create any automatic detection of objects.

The part that I didn't get is when you say about backward
compatibility. Would LTO work on a newer binutils with the liblto but
on an older compiler that knew nothing about LTO?

Your proposal is, then, to get binutils:

- recognizing LTO logic in the objects
- automatically loading liblto if recognized
- warning if not

I'm assuming the extra symbols would be discarded if no library is
found, together with the warning, right? Maybe an error if -Wall or
whatever.

Can we get someone from the binutils community to opine on that?

cheers,
--renato

On 11 February 2014 02:29, Jan Hubicka <hubicka@ucw.cz> wrote:
> One practical experience I have with LLVM developers is sharing experiences
> about getting Firefox to work with LTO with Rafael Espindola and I think it was
> useful for both of us. I am definitly open to more discussion.
>
> Lets try a specific topic that is on my TODO list for some time.
>
> I would like to make it possible for mutliple compilers to be used to LTO a
> single binary. As we are all making LTO more useful, I think it is matter of
> time until people will start shipping LTO object files by default and users
> will end up feeding them into different compilers or incompatible version of
> the same compiler. We probably want to make this work, even thought the
> cross-module optimization will not happen in this case.
>
> The plugin interface in binutils seems to do its job well both for GCC and LLVM
> and I hope that open64 and ICC will eventually join, too.
>
> The trouble however is that one needs to pass explicit --plugin argument
> specifying the particular plugin to load and so GCC ships with its own wrappers
> (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
>
> It may be smoother if binutils was able to load multiple plugins at once and
> grab plugins from system and user installed compilers without explicit --plugin
> argument.
>
> Binutils probably should also have a way to detect LTO object files and produce
> more useful diagnostic than they do now, when there is no plugin claiming them.
>
> There are some PRs filled on the topic
> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
> but not much progress on them.
>
> I wonder if we can get this designed and implemented.
>
> On the other hand, GCC current maintains non-plugin path for LTO that is now
> only used by darwin port due to lack of plugin enabled LD there.  It seems
> that liblto used by darwin is losely compatible with the plugin API, but it makes
> it harder to have different compilers share it (one has to LD_PRELOAD liblto
> to different one prior executing the linker?)
>
> I wonder, is there chance to implement linker plugin API to libLTO glue or add
> plugin support to native Darwin tools?
>
> Honza

Re: Fwd: LLVM collaboration?

[Uday Khedker]

On Tuesday 11 February 2014 03:25 PM, Renato Golin wrote:
> Hi Jan,
>
> I think this is a very good example where we could all collaborate
> (including binutils).
>
> I'll leave your reply intact, so that Chandler (CC'd) can get a bit
> more context. I'm copying him because he (and I believe Diego) had
> more contact with LTO than I had.
>
> If I got it right, LTO today:
>
> - needs the drivers to explicitly declare the plugin
> - needs the library available somewhere
> - may have to change the library loading semantics (via LD_PRELOAD)

There is another need that I have felt in LTO for quite some time. 
Currently, it has a non-partitioned mode or a partitioned mode but this 
decision is taken before the compilation begins. It would be nice to 
have a mode that allows dynamic loading of function bodies so that a 
flow and context sensitive IPA could load functions bodies on demand, 
and unload them when they are not needed.

Uday.

>
> Since both toolchains do the magic, binutils has no incentive to
> create any automatic detection of objects.
>
> The part that I didn't get is when you say about backward
> compatibility. Would LTO work on a newer binutils with the liblto but
> on an older compiler that knew nothing about LTO?
>
> Your proposal is, then, to get binutils:
>
> - recognizing LTO logic in the objects
> - automatically loading liblto if recognized
> - warning if not
>
> I'm assuming the extra symbols would be discarded if no library is
> found, together with the warning, right? Maybe an error if -Wall or
> whatever.
>
> Can we get someone from the binutils community to opine on that?
>
> cheers,
> --renato
>
> On 11 February 2014 02:29, Jan Hubicka <hubicka@ucw.cz> wrote:
>> One practical experience I have with LLVM developers is sharing experiences
>> about getting Firefox to work with LTO with Rafael Espindola and I think it was
>> useful for both of us. I am definitly open to more discussion.
>>
>> Lets try a specific topic that is on my TODO list for some time.
>>
>> I would like to make it possible for mutliple compilers to be used to LTO a
>> single binary. As we are all making LTO more useful, I think it is matter of
>> time until people will start shipping LTO object files by default and users
>> will end up feeding them into different compilers or incompatible version of
>> the same compiler. We probably want to make this work, even thought the
>> cross-module optimization will not happen in this case.
>>
>> The plugin interface in binutils seems to do its job well both for GCC and LLVM
>> and I hope that open64 and ICC will eventually join, too.
>>
>> The trouble however is that one needs to pass explicit --plugin argument
>> specifying the particular plugin to load and so GCC ships with its own wrappers
>> (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
>>
>> It may be smoother if binutils was able to load multiple plugins at once and
>> grab plugins from system and user installed compilers without explicit --plugin
>> argument.
>>
>> Binutils probably should also have a way to detect LTO object files and produce
>> more useful diagnostic than they do now, when there is no plugin claiming them.
>>
>> There are some PRs filled on the topic
>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
>> but not much progress on them.
>>
>> I wonder if we can get this designed and implemented.
>>
>> On the other hand, GCC current maintains non-plugin path for LTO that is now
>> only used by darwin port due to lack of plugin enabled LD there.  It seems
>> that liblto used by darwin is losely compatible with the plugin API, but it makes
>> it harder to have different compilers share it (one has to LD_PRELOAD liblto
>> to different one prior executing the linker?)
>>
>> I wonder, is there chance to implement linker plugin API to libLTO glue or add
>> plugin support to native Darwin tools?
>>
>> Honza

Re: Fwd: LLVM collaboration?

[Jan Hubicka]

> 
> 
> 
> 
> On Tuesday 11 February 2014 03:25 PM, Renato Golin wrote:
> >Hi Jan,
> >
> >I think this is a very good example where we could all collaborate
> >(including binutils).
> >
> >I'll leave your reply intact, so that Chandler (CC'd) can get a bit
> >more context. I'm copying him because he (and I believe Diego) had
> >more contact with LTO than I had.
> >
> >If I got it right, LTO today:
> >
> >- needs the drivers to explicitly declare the plugin

Yes.
> >- needs the library available somewhere
> >- may have to change the library loading semantics (via LD_PRELOAD)

Not in binutils implementation (I believe it is the case for darwin's libLTO).
With binutils you only need to pass explicit --plugin argument into all
tools that care (ld/ar/nm/ranlib)
> 
> There is another need that I have felt in LTO for quite some time.
> Currently, it has a non-partitioned mode or a partitioned mode but
> this decision is taken before the compilation begins. It would be
> nice to have a mode that allows dynamic loading of function bodies
> so that a flow and context sensitive IPA could load functions bodies
> on demand, and unload them when they are not needed.

I implemented on-demand loading of function bodies into GCC-4.8 if I recall
correctly. Currently I thinko only Martin Liska's code unification pass uses it
to verify that two function bdoes it thinks are equivalent are actually
equivalent. Hopefully it will be merged into 4.10.
> 
> Uday.
> 
> >
> >Since both toolchains do the magic, binutils has no incentive to
> >create any automatic detection of objects.
> >
> >The part that I didn't get is when you say about backward
> >compatibility. Would LTO work on a newer binutils with the liblto but
> >on an older compiler that knew nothing about LTO?
> >
> >Your proposal is, then, to get binutils:
> >
> >- recognizing LTO logic in the objects
> >- automatically loading liblto if recognized
> >- warning if not

I basically think that binutils should have a way for installed compiler to
register a plugin and load all plugins by default (or perhaps for performance
or upon detecking an compatible LTO object file in some way, perhaps also by
information given in the config file) and let them claim the LTO objects they
understand to.

With the backward compatibility I mean that if we release a new version of
compiler that can no longer read the LTO objects of older compiler, one can
just install both versions and have their plugins to claim only LTO objects
they understand. Just if they were two different compilers.

Finally I think we can make binutils to recognize GCC/LLVM LTO objects
as a special case and produce friendly message when user try to handle
them witout plugin as oposed to today strange errors about file formats
or missing symbols.

Honza
> >
> >I'm assuming the extra symbols would be discarded if no library is
> >found, together with the warning, right? Maybe an error if -Wall or
> >whatever.
> >
> >Can we get someone from the binutils community to opine on that?
> >
> >cheers,
> >--renato
> >
> >On 11 February 2014 02:29, Jan Hubicka <hubicka@ucw.cz> wrote:
> >>One practical experience I have with LLVM developers is sharing experiences
> >>about getting Firefox to work with LTO with Rafael Espindola and I think it was
> >>useful for both of us. I am definitly open to more discussion.
> >>
> >>Lets try a specific topic that is on my TODO list for some time.
> >>
> >>I would like to make it possible for mutliple compilers to be used to LTO a
> >>single binary. As we are all making LTO more useful, I think it is matter of
> >>time until people will start shipping LTO object files by default and users
> >>will end up feeding them into different compilers or incompatible version of
> >>the same compiler. We probably want to make this work, even thought the
> >>cross-module optimization will not happen in this case.
> >>
> >>The plugin interface in binutils seems to do its job well both for GCC and LLVM
> >>and I hope that open64 and ICC will eventually join, too.
> >>
> >>The trouble however is that one needs to pass explicit --plugin argument
> >>specifying the particular plugin to load and so GCC ships with its own wrappers
> >>(gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
> >>
> >>It may be smoother if binutils was able to load multiple plugins at once and
> >>grab plugins from system and user installed compilers without explicit --plugin
> >>argument.
> >>
> >>Binutils probably should also have a way to detect LTO object files and produce
> >>more useful diagnostic than they do now, when there is no plugin claiming them.
> >>
> >>There are some PRs filled on the topic
> >>http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
> >>http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
> >>but not much progress on them.
> >>
> >>I wonder if we can get this designed and implemented.
> >>
> >>On the other hand, GCC current maintains non-plugin path for LTO that is now
> >>only used by darwin port due to lack of plugin enabled LD there.  It seems
> >>that liblto used by darwin is losely compatible with the plugin API, but it makes
> >>it harder to have different compilers share it (one has to LD_PRELOAD liblto
> >>to different one prior executing the linker?)
> >>
> >>I wonder, is there chance to implement linker plugin API to libLTO glue or add
> >>plugin support to native Darwin tools?
> >>
> >>Honza

Re: Fwd: LLVM collaboration?

[Uday Khedker]

On Tuesday 11 February 2014 09:30 PM, Jan Hubicka wrote:
>>
>>
>>
>>
>> On Tuesday 11 February 2014 03:25 PM, Renato Golin wrote:
>>> Hi Jan,
>>>
>>> I think this is a very good example where we could all collaborate
>>> (including binutils).
>>>
>>> I'll leave your reply intact, so that Chandler (CC'd) can get a bit
>>> more context. I'm copying him because he (and I believe Diego) had
>>> more contact with LTO than I had.
>>>
>>> If I got it right, LTO today:
>>>
>>> - needs the drivers to explicitly declare the plugin
>
> Yes.
>>> - needs the library available somewhere
>>> - may have to change the library loading semantics (via LD_PRELOAD)
>
> Not in binutils implementation (I believe it is the case for darwin's libLTO).
> With binutils you only need to pass explicit --plugin argument into all
> tools that care (ld/ar/nm/ranlib)
>>
>> There is another need that I have felt in LTO for quite some time.
>> Currently, it has a non-partitioned mode or a partitioned mode but
>> this decision is taken before the compilation begins. It would be
>> nice to have a mode that allows dynamic loading of function bodies
>> so that a flow and context sensitive IPA could load functions bodies
>> on demand, and unload them when they are not needed.
>
> I implemented on-demand loading of function bodies into GCC-4.8 if I recall
> correctly. Currently I thinko only Martin Liska's code unification pass uses it
> to verify that two function bdoes it thinks are equivalent are actually
> equivalent. Hopefully it will be merged into 4.10.

Great. We will experiment with it.

Uday.

>>
>> Uday.
>>
>>>
>>> Since both toolchains do the magic, binutils has no incentive to
>>> create any automatic detection of objects.
>>>
>>> The part that I didn't get is when you say about backward
>>> compatibility. Would LTO work on a newer binutils with the liblto but
>>> on an older compiler that knew nothing about LTO?
>>>
>>> Your proposal is, then, to get binutils:
>>>
>>> - recognizing LTO logic in the objects
>>> - automatically loading liblto if recognized
>>> - warning if not
>
> I basically think that binutils should have a way for installed compiler to
> register a plugin and load all plugins by default (or perhaps for performance
> or upon detecking an compatible LTO object file in some way, perhaps also by
> information given in the config file) and let them claim the LTO objects they
> understand to.
>
> With the backward compatibility I mean that if we release a new version of
> compiler that can no longer read the LTO objects of older compiler, one can
> just install both versions and have their plugins to claim only LTO objects
> they understand. Just if they were two different compilers.
>
> Finally I think we can make binutils to recognize GCC/LLVM LTO objects
> as a special case and produce friendly message when user try to handle
> them witout plugin as oposed to today strange errors about file formats
> or missing symbols.
>
> Honza
>>>
>>> I'm assuming the extra symbols would be discarded if no library is
>>> found, together with the warning, right? Maybe an error if -Wall or
>>> whatever.
>>>
>>> Can we get someone from the binutils community to opine on that?
>>>
>>> cheers,
>>> --renato
>>>
>>> On 11 February 2014 02:29, Jan Hubicka <hubicka@ucw.cz> wrote:
>>>> One practical experience I have with LLVM developers is sharing experiences
>>>> about getting Firefox to work with LTO with Rafael Espindola and I think it was
>>>> useful for both of us. I am definitly open to more discussion.
>>>>
>>>> Lets try a specific topic that is on my TODO list for some time.
>>>>
>>>> I would like to make it possible for mutliple compilers to be used to LTO a
>>>> single binary. As we are all making LTO more useful, I think it is matter of
>>>> time until people will start shipping LTO object files by default and users
>>>> will end up feeding them into different compilers or incompatible version of
>>>> the same compiler. We probably want to make this work, even thought the
>>>> cross-module optimization will not happen in this case.
>>>>
>>>> The plugin interface in binutils seems to do its job well both for GCC and LLVM
>>>> and I hope that open64 and ICC will eventually join, too.
>>>>
>>>> The trouble however is that one needs to pass explicit --plugin argument
>>>> specifying the particular plugin to load and so GCC ships with its own wrappers
>>>> (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
>>>>
>>>> It may be smoother if binutils was able to load multiple plugins at once and
>>>> grab plugins from system and user installed compilers without explicit --plugin
>>>> argument.
>>>>
>>>> Binutils probably should also have a way to detect LTO object files and produce
>>>> more useful diagnostic than they do now, when there is no plugin claiming them.
>>>>
>>>> There are some PRs filled on the topic
>>>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
>>>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
>>>> but not much progress on them.
>>>>
>>>> I wonder if we can get this designed and implemented.
>>>>
>>>> On the other hand, GCC current maintains non-plugin path for LTO that is now
>>>> only used by darwin port due to lack of plugin enabled LD there.  It seems
>>>> that liblto used by darwin is losely compatible with the plugin API, but it makes
>>>> it harder to have different compilers share it (one has to LD_PRELOAD liblto
>>>> to different one prior executing the linker?)
>>>>
>>>> I wonder, is there chance to implement linker plugin API to libLTO glue or add
>>>> plugin support to native Darwin tools?
>>>>
>>>> Honza

Re: Fwd: LLVM collaboration?

[Renato Golin]

On 11 February 2014 16:00, Jan Hubicka <hubicka@ucw.cz> wrote:
> I basically think that binutils should have a way for installed compiler to
> register a plugin and load all plugins by default (or perhaps for performance
> or upon detecking an compatible LTO object file in some way, perhaps also by
> information given in the config file) and let them claim the LTO objects they
> understand to.

Right, so this would be not necessarily related to LTO, but with the
binutils plugin system. In my very limited experience with LTO and
binutils, I can't see how that would be different from just adding a
--plugin option on the compiler, unless it's something that the linker
would detect automatically without the interference of any compiler.


> With the backward compatibility I mean that if we release a new version of
> compiler that can no longer read the LTO objects of older compiler, one can
> just install both versions and have their plugins to claim only LTO objects
> they understand. Just if they were two different compilers.

Yes, this makes total sense.


> Finally I think we can make binutils to recognize GCC/LLVM LTO objects
> as a special case and produce friendly message when user try to handle
> them witout plugin as oposed to today strange errors about file formats
> or missing symbols.

Yes, that as well seems pretty obvious, and mostly orthogonal to the
other two proposals.

cheers,
--renato

PS: Removing Chandler, as he was not the right person to look at this.
I'll check with others in the LLVM list to chime in on this thread.

Re: Fwd: LLVM collaboration?

[Renato Golin]

Now copying Rafael, which can give us some more insight on the LLVM LTO side.

cheers,
--renato

On 11 February 2014 09:55, Renato Golin <renato.golin@linaro.org> wrote:
> Hi Jan,
>
> I think this is a very good example where we could all collaborate
> (including binutils).
>
> I'll leave your reply intact, so that Chandler (CC'd) can get a bit
> more context. I'm copying him because he (and I believe Diego) had
> more contact with LTO than I had.
>
> If I got it right, LTO today:
>
> - needs the drivers to explicitly declare the plugin
> - needs the library available somewhere
> - may have to change the library loading semantics (via LD_PRELOAD)
>
> Since both toolchains do the magic, binutils has no incentive to
> create any automatic detection of objects.
>
> The part that I didn't get is when you say about backward
> compatibility. Would LTO work on a newer binutils with the liblto but
> on an older compiler that knew nothing about LTO?
>
> Your proposal is, then, to get binutils:
>
> - recognizing LTO logic in the objects
> - automatically loading liblto if recognized
> - warning if not
>
> I'm assuming the extra symbols would be discarded if no library is
> found, together with the warning, right? Maybe an error if -Wall or
> whatever.
>
> Can we get someone from the binutils community to opine on that?
>
> cheers,
> --renato
>
> On 11 February 2014 02:29, Jan Hubicka <hubicka@ucw.cz> wrote:
>> One practical experience I have with LLVM developers is sharing experiences
>> about getting Firefox to work with LTO with Rafael Espindola and I think it was
>> useful for both of us. I am definitly open to more discussion.
>>
>> Lets try a specific topic that is on my TODO list for some time.
>>
>> I would like to make it possible for mutliple compilers to be used to LTO a
>> single binary. As we are all making LTO more useful, I think it is matter of
>> time until people will start shipping LTO object files by default and users
>> will end up feeding them into different compilers or incompatible version of
>> the same compiler. We probably want to make this work, even thought the
>> cross-module optimization will not happen in this case.
>>
>> The plugin interface in binutils seems to do its job well both for GCC and LLVM
>> and I hope that open64 and ICC will eventually join, too.
>>
>> The trouble however is that one needs to pass explicit --plugin argument
>> specifying the particular plugin to load and so GCC ships with its own wrappers
>> (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
>>
>> It may be smoother if binutils was able to load multiple plugins at once and
>> grab plugins from system and user installed compilers without explicit --plugin
>> argument.
>>
>> Binutils probably should also have a way to detect LTO object files and produce
>> more useful diagnostic than they do now, when there is no plugin claiming them.
>>
>> There are some PRs filled on the topic
>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
>> but not much progress on them.
>>
>> I wonder if we can get this designed and implemented.
>>
>> On the other hand, GCC current maintains non-plugin path for LTO that is now
>> only used by darwin port due to lack of plugin enabled LD there.  It seems
>> that liblto used by darwin is losely compatible with the plugin API, but it makes
>> it harder to have different compilers share it (one has to LD_PRELOAD liblto
>> to different one prior executing the linker?)
>>
>> I wonder, is there chance to implement linker plugin API to libLTO glue or add
>> plugin support to native Darwin tools?
>>
>> Honza

Re: Fwd: LLVM collaboration?

[Rafael Espíndola]

On 11 February 2014 12:28, Renato Golin <renato.golin@linaro.org> wrote:
> Now copying Rafael, which can give us some more insight on the LLVM LTO side.

Thanks.

> On 11 February 2014 09:55, Renato Golin <renato.golin@linaro.org> wrote:
>> Hi Jan,
>>
>> I think this is a very good example where we could all collaborate
>> (including binutils).

It is. Both LTO models (LLVM and GCC) were considered form the start
of the API design and I think we got a better plugin model as a
result.

>> If I got it right, LTO today:
>>
>> - needs the drivers to explicitly declare the plugin
>> - needs the library available somewhere

True.

>> - may have to change the library loading semantics (via LD_PRELOAD)

That depends on the library being loaded. RPATH works just fine too.

>> Since both toolchains do the magic, binutils has no incentive to
>> create any automatic detection of objects.

It is mostly a historical decision. At the time the design was for the
plugin to be matched to the compiler, and so the compiler could pass
that information down to the linker.

> The trouble however is that one needs to pass explicit --plugin argument
> specifying the particular plugin to load and so GCC ships with its own wrappers
> (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.

These wrappers should not be necessary. While the linker currently
requires a command line option, bfd has support for searching for a
plugin. It will search <inst>/lib/bfd-plugin. See for example the
instructions at http://llvm.org/docs/GoldPlugin.html.

This was done because ar and nm are not normally bound to any
compiler. Had we realized this issue earlier we would probably have
supported searching for plugins in the linker too.

So it seems that what you want could be done by

* having bfd-ld and gold search bfd-plugins (maybe rename the directory?)
* support loading multiple plugins, and asking each to see if it
supports a given file. That ways we could LTO when having a part GCC
and part LLVM build.
* maybe be smart about version and load new ones first? (libLLVM-3.4
before libLLVM-3.3 for example). Probably the first one should always
be the one given in the command line.

For OS X the situation is a bit different. There instead of a plugin
the linker loads a library: libLTO.dylib. When doing LTO with a newer
llvm, one needs to set DYLD_LIBRARY_PATH. I think I proposed setting
that from clang some time ago, but I don't remember the outcome.

In theory GCC could implement a libLTO.dylib and set
DYLD_LIBRARY_PATH. The gold/bfd plugin that LLVM uses is basically a
API mapping the other way, so the job would be inverting it. The LTO
model ld64 is a bit more strict about knowing all symbol definitions
and uses (including inline asm), so there would be work to be done to
cover that, but the simple cases shouldn't be too hard.

Cheers,
Rafael

Re: Fwd: LLVM collaboration?

[Markus Trippelsdorf]

On 2014.02.11 at 13:02 -0500, Rafael Espíndola wrote:
> On 11 February 2014 12:28, Renato Golin <renato.golin@linaro.org> wrote:
> > Now copying Rafael, which can give us some more insight on the LLVM LTO side.
> 
> Thanks.
> 
> > On 11 February 2014 09:55, Renato Golin <renato.golin@linaro.org> wrote:
> >> Hi Jan,
> >>
> >> I think this is a very good example where we could all collaborate
> >> (including binutils).
> 
> It is. Both LTO models (LLVM and GCC) were considered form the start
> of the API design and I think we got a better plugin model as a
> result.
> 
> >> If I got it right, LTO today:
> >>
> >> - needs the drivers to explicitly declare the plugin
> >> - needs the library available somewhere
> 
> True.
> 
> >> - may have to change the library loading semantics (via LD_PRELOAD)
> 
> That depends on the library being loaded. RPATH works just fine too.
> 
> >> Since both toolchains do the magic, binutils has no incentive to
> >> create any automatic detection of objects.
> 
> It is mostly a historical decision. At the time the design was for the
> plugin to be matched to the compiler, and so the compiler could pass
> that information down to the linker.
> 
> > The trouble however is that one needs to pass explicit --plugin argument
> > specifying the particular plugin to load and so GCC ships with its own wrappers
> > (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
> 
> These wrappers should not be necessary. While the linker currently
> requires a command line option, bfd has support for searching for a
> plugin. It will search <inst>/lib/bfd-plugin. See for example the
> instructions at http://llvm.org/docs/GoldPlugin.html.

Please note that this automatic loading of the plugin only happens for
non-ELF files. So the LLVM GoldPlugin gets loaded fine, but automatic
loading of gcc's liblto_plugin.so doesn't work at the moment.

A basic implementation to support both plugins seamlessly should be
pretty straightforward, because LLVM's bitstream file format (non-ELF)
is easily distinguishable from gcc's output (standard ELF with special
sections).

-- 
Markus

Re: Fwd: LLVM collaboration?

[Jan Hubicka]

> On 2014.02.11 at 13:02 -0500, Rafael EspĂ­ndola wrote:
> > On 11 February 2014 12:28, Renato Golin <renato.golin@linaro.org> wrote:
> > > Now copying Rafael, which can give us some more insight on the LLVM LTO side.
> > 
> > Thanks.
> > 
> > > On 11 February 2014 09:55, Renato Golin <renato.golin@linaro.org> wrote:
> > >> Hi Jan,
> > >>
> > >> I think this is a very good example where we could all collaborate
> > >> (including binutils).
> > 
> > It is. Both LTO models (LLVM and GCC) were considered form the start
> > of the API design and I think we got a better plugin model as a
> > result.
> > 
> > >> If I got it right, LTO today:
> > >>
> > >> - needs the drivers to explicitly declare the plugin
> > >> - needs the library available somewhere
> > 
> > True.
> > 
> > >> - may have to change the library loading semantics (via LD_PRELOAD)
> > 
> > That depends on the library being loaded. RPATH works just fine too.
> > 
> > >> Since both toolchains do the magic, binutils has no incentive to
> > >> create any automatic detection of objects.
> > 
> > It is mostly a historical decision. At the time the design was for the
> > plugin to be matched to the compiler, and so the compiler could pass
> > that information down to the linker.
> > 
> > > The trouble however is that one needs to pass explicit --plugin argument
> > > specifying the particular plugin to load and so GCC ships with its own wrappers
> > > (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
> > 
> > These wrappers should not be necessary. While the linker currently
> > requires a command line option, bfd has support for searching for a
> > plugin. It will search <inst>/lib/bfd-plugin. See for example the
> > instructions at http://llvm.org/docs/GoldPlugin.html.
> 
> Please note that this automatic loading of the plugin only happens for
> non-ELF files. So the LLVM GoldPlugin gets loaded fine, but automatic
> loading of gcc's liblto_plugin.so doesn't work at the moment.

Hmm, something that ought to be fixed.  Binutils can probably know about GCC's
LTO symbols it uses as a distniguisher.  Is there a PR about this?
> 
> A basic implementation to support both plugins seamlessly should be
> pretty straightforward, because LLVM's bitstream file format (non-ELF)
> is easily distinguishable from gcc's output (standard ELF with special
> sections).

I think it is easy even with two plugins for same file format - all ld need is
to load the plugins and then do the file claiming for each of them.
GCC plugin then should not claim files from LLVM or incompatible GCC version
and vice versa.

Honza
> 
> -- 
> Markus

Re: Fwd: LLVM collaboration?

[Jan Hubicka]

> >> Since both toolchains do the magic, binutils has no incentive to
> >> create any automatic detection of objects.
> 
> It is mostly a historical decision. At the time the design was for the
> plugin to be matched to the compiler, and so the compiler could pass
> that information down to the linker.
> 
> > The trouble however is that one needs to pass explicit --plugin argument
> > specifying the particular plugin to load and so GCC ships with its own wrappers
> > (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
> 
> These wrappers should not be necessary. While the linker currently
> requires a command line option, bfd has support for searching for a
> plugin. It will search <inst>/lib/bfd-plugin. See for example the
> instructions at http://llvm.org/docs/GoldPlugin.html.

My reading of bfd/plugin.c is that it basically walks the directory and looks
for first plugin that returns OK for onload. (that is always the case for
GCC/LLVM plugins).  So if I instlal GCC and llvm plugin there it will
depend who will end up being first and only that plugin will be used.

We need multiple plugin support as suggested by the directory name ;)

Also it sems that currently plugin is not used if file is ELF for ar/nm/ranlib
(as mentioned by Markus) and also GNU-ld seems to choke on LLVM object files
even if it has plugin.

This probably needs ot be sanitized.

> 
> This was done because ar and nm are not normally bound to any
> compiler. Had we realized this issue earlier we would probably have
> supported searching for plugins in the linker too.
> 
> So it seems that what you want could be done by
> 
> * having bfd-ld and gold search bfd-plugins (maybe rename the directory?)
> * support loading multiple plugins, and asking each to see if it
> supports a given file. That ways we could LTO when having a part GCC
> and part LLVM build.

Yes, that is what I have in mind.

Plus perhaps additional configuration file to avoid loading everything.  Say
user instealls 3 versions of LLVM, open64 and ICC. If all of them loads as a
shared library, like LLVM does, it will probably slow down the tools
measurably.

> * maybe be smart about version and load new ones first? (libLLVM-3.4
> before libLLVM-3.3 for example). Probably the first one should always
> be the one given in the command line.

Yes, i think we may want to prioritize the list.  So user can prevail
his own version of GCC over the system one, for example.
> 
> For OS X the situation is a bit different. There instead of a plugin
> the linker loads a library: libLTO.dylib. When doing LTO with a newer
> llvm, one needs to set DYLD_LIBRARY_PATH. I think I proposed setting
> that from clang some time ago, but I don't remember the outcome.
> 
> In theory GCC could implement a libLTO.dylib and set
> DYLD_LIBRARY_PATH. The gold/bfd plugin that LLVM uses is basically a
> API mapping the other way, so the job would be inverting it. The LTO
> model ld64 is a bit more strict about knowing all symbol definitions
> and uses (including inline asm), so there would be work to be done to
> cover that, but the simple cases shouldn't be too hard.

I would not care that much about symbols in asm definitions to start with.
Even if we will force users to non-LTO those object files, it would be an
improvement over what we have now.

One problem is that we need a volunteer to implement the reverse glue
(libLTO->plugin API), since I do not have an OS X box (well, have an old G5,
but even that is quite far from me right now)

Why complete symbol tables are required? Can't ld64 be changed to ignore
unresolved symbols in the first stage just like gold/gnu-ld does?

Honza
> 
> Cheers,
> Rafael

Re: Fwd: LLVM collaboration?

[Rafael Espíndola]

> My reading of bfd/plugin.c is that it basically walks the directory and looks
> for first plugin that returns OK for onload. (that is always the case for
> GCC/LLVM plugins).  So if I instlal GCC and llvm plugin there it will
> depend who will end up being first and only that plugin will be used.
>
> We need multiple plugin support as suggested by the directory name ;)
>
> Also it sems that currently plugin is not used if file is ELF for ar/nm/ranlib
> (as mentioned by Markus) and also GNU-ld seems to choke on LLVM object files
> even if it has plugin.
>
> This probably needs ot be sanitized.

CCing Hal Finkel. He got this to work some time ago. Not sure if he
ever ported the patches to bfd trunk.

>> For OS X the situation is a bit different. There instead of a plugin
>> the linker loads a library: libLTO.dylib. When doing LTO with a newer
>> llvm, one needs to set DYLD_LIBRARY_PATH. I think I proposed setting
>> that from clang some time ago, but I don't remember the outcome.
>>
>> In theory GCC could implement a libLTO.dylib and set
>> DYLD_LIBRARY_PATH. The gold/bfd plugin that LLVM uses is basically a
>> API mapping the other way, so the job would be inverting it. The LTO
>> model ld64 is a bit more strict about knowing all symbol definitions
>> and uses (including inline asm), so there would be work to be done to
>> cover that, but the simple cases shouldn't be too hard.
>
> I would not care that much about symbols in asm definitions to start with.
> Even if we will force users to non-LTO those object files, it would be an
> improvement over what we have now.
>
> One problem is that we need a volunteer to implement the reverse glue
> (libLTO->plugin API), since I do not have an OS X box (well, have an old G5,
> but even that is quite far from me right now)
>
> Why complete symbol tables are required? Can't ld64 be changed to ignore
> unresolved symbols in the first stage just like gold/gnu-ld does?

I am not sure about this. My *guess* is that it does dead stripping
computation before asking libLTO for the object file. I noticed the
issue while trying to LTO firefox some time ago.

Cheers,
Rafael

Re: LLVM collaboration?

[Hal Finkel]

[-- Attachment #1: Type: text/plain, Size: 3079 bytes --]

----- Original Message -----
> From: "Rafael Espíndola" <rafael.espindola@gmail.com>
> To: "Jan Hubicka" <hubicka@ucw.cz>
> Cc: "Renato Golin" <renato.golin@linaro.org>, "gcc" <gcc@gcc.gnu.org>, "Hal Finkel" <hfinkel@anl.gov>
> Sent: Tuesday, February 11, 2014 3:38:40 PM
> Subject: Re: Fwd: LLVM collaboration?
> 
> > My reading of bfd/plugin.c is that it basically walks the directory
> > and looks
> > for first plugin that returns OK for onload. (that is always the
> > case for
> > GCC/LLVM plugins).  So if I instlal GCC and llvm plugin there it
> > will
> > depend who will end up being first and only that plugin will be
> > used.
> >
> > We need multiple plugin support as suggested by the directory name
> > ;)
> >
> > Also it sems that currently plugin is not used if file is ELF for
> > ar/nm/ranlib
> > (as mentioned by Markus) and also GNU-ld seems to choke on LLVM
> > object files
> > even if it has plugin.
> >
> > This probably needs ot be sanitized.
> 
> CCing Hal Finkel. He got this to work some time ago. Not sure if he
> ever ported the patches to bfd trunk.

I have a patch for binutils 2.24 (attached -- I think this works, I hand isolated it from my BG/Q patchset). I would not consider it to be of upstream quality, but I'd obviously appreciate any assistance on making everything clean and proper ;)

 -Hal

> 
> >> For OS X the situation is a bit different. There instead of a
> >> plugin
> >> the linker loads a library: libLTO.dylib. When doing LTO with a
> >> newer
> >> llvm, one needs to set DYLD_LIBRARY_PATH. I think I proposed
> >> setting
> >> that from clang some time ago, but I don't remember the outcome.
> >>
> >> In theory GCC could implement a libLTO.dylib and set
> >> DYLD_LIBRARY_PATH. The gold/bfd plugin that LLVM uses is basically
> >> a
> >> API mapping the other way, so the job would be inverting it. The
> >> LTO
> >> model ld64 is a bit more strict about knowing all symbol
> >> definitions
> >> and uses (including inline asm), so there would be work to be done
> >> to
> >> cover that, but the simple cases shouldn't be too hard.
> >
> > I would not care that much about symbols in asm definitions to
> > start with.
> > Even if we will force users to non-LTO those object files, it would
> > be an
> > improvement over what we have now.
> >
> > One problem is that we need a volunteer to implement the reverse
> > glue
> > (libLTO->plugin API), since I do not have an OS X box (well, have
> > an old G5,
> > but even that is quite far from me right now)
> >
> > Why complete symbol tables are required? Can't ld64 be changed to
> > ignore
> > unresolved symbols in the first stage just like gold/gnu-ld does?
> 
> I am not sure about this. My *guess* is that it does dead stripping
> computation before asking libLTO for the object file. I noticed the
> issue while trying to LTO firefox some time ago.
> 
> Cheers,
> Rafael
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: binutils-2.24-ld-plugin.patch --]
[-- Type: text/x-patch; name="binutils-2.24-ld-plugin.patch", Size: 3087 bytes --]

diff --git a/bfd/elflink.c b/bfd/elflink.c
index 99b7ca1..c2bf9c3 100644
--- a/bfd/elflink.c
+++ b/bfd/elflink.c
@@ -5054,7 +5054,9 @@ elf_link_add_archive_symbols (bfd *abfd, struct bfd_link_info *info)
 	    goto error_return;
 
 	  if (! bfd_check_format (element, bfd_object))
-	    goto error_return;
+	    /* goto error_return; */
+            /* this might be an object understood only by an LTO plugin */
+            bfd_elf_make_object (element);
 
 	  /* Doublecheck that we have not included this object
 	     already--it should be impossible, but there may be
diff --git a/ld/ldfile.c b/ld/ldfile.c
index 16baef8..159a60c 100644
--- a/ld/ldfile.c
+++ b/ld/ldfile.c
@@ -38,6 +38,7 @@
 #ifdef ENABLE_PLUGINS
 #include "plugin-api.h"
 #include "plugin.h"
+#include "elf-bfd.h"
 #endif /* ENABLE_PLUGINS */
 
 bfd_boolean  ldfile_assumed_script = FALSE;
@@ -124,6 +125,7 @@ bfd_boolean
 ldfile_try_open_bfd (const char *attempt,
 		     lang_input_statement_type *entry)
 {
+  int is_obj = 0;
   entry->the_bfd = bfd_openr (attempt, entry->target);
 
   if (verbose)
@@ -168,6 +170,34 @@ ldfile_try_open_bfd (const char *attempt,
 	{
 	  if (! bfd_check_format (check, bfd_object))
 	    {
+#ifdef ENABLE_PLUGINS
+	      if (check == entry->the_bfd
+		  && bfd_get_error () == bfd_error_file_not_recognized
+		  && ! ldemul_unrecognized_file (entry))
+		{
+                  if (plugin_active_plugins_p ()
+                      && !no_more_claiming)
+                    {
+                      int fd = open (attempt, O_RDONLY | O_BINARY);
+                      if (fd >= 0)
+                        {
+                          struct ld_plugin_input_file file;
+
+                          bfd_elf_make_object (entry->the_bfd);
+ 
+                          file.name = attempt;
+                          file.offset = 0;
+                          file.filesize = lseek (fd, 0, SEEK_END);
+                          file.fd = fd;
+                          plugin_maybe_claim (&file, entry);
+
+                          if (entry->flags.claimed)
+                            return TRUE;
+                        }
+                    }
+                }
+#endif /* ENABLE_PLUGINS */
+
 	      if (check == entry->the_bfd
 		  && entry->flags.search_dirs
 		  && bfd_get_error () == bfd_error_file_not_recognized
@@ -303,7 +333,9 @@ success:
      bfd_object that it sets the bfd's arch and mach, which
      will be needed when and if we want to bfd_create a new
      one using this one as a template.  */
-  if (bfd_check_format (entry->the_bfd, bfd_object)
+  if (((is_obj = bfd_check_format (entry->the_bfd, bfd_object))
+       || (bfd_get_format(entry->the_bfd) == bfd_unknown
+           && bfd_get_error () == bfd_error_file_not_recognized))
       && plugin_active_plugins_p ()
       && !no_more_claiming)
     {
@@ -312,6 +344,9 @@ success:
 	{
 	  struct ld_plugin_input_file file;
 
+          if (!is_obj)
+            bfd_elf_make_object (entry->the_bfd);
+
 	  file.name = attempt;
 	  file.offset = 0;
 	  file.filesize = lseek (fd, 0, SEEK_END);

Re: LLVM collaboration?

[Hal Finkel]

[-- Attachment #1: Type: text/plain, Size: 3996 bytes --]

Rafael,

As I mentioned earlier on IRC, the current trunk LLVM plugin requires get_view and get/release_input_file (which had not previously been the case). I've attached a version of the patch that also provides an implementation of those functions.

 -Hal

----- Original Message -----
> From: "Hal Finkel" <hfinkel@anl.gov>
> To: "Rafael Espíndola" <rafael.espindola@gmail.com>
> Cc: "Renato Golin" <renato.golin@linaro.org>, "gcc" <gcc@gcc.gnu.org>, "Jan Hubicka" <hubicka@ucw.cz>
> Sent: Tuesday, February 11, 2014 4:35:59 PM
> Subject: Re: LLVM collaboration?
> 
> ----- Original Message -----
> > From: "Rafael Espíndola" <rafael.espindola@gmail.com>
> > To: "Jan Hubicka" <hubicka@ucw.cz>
> > Cc: "Renato Golin" <renato.golin@linaro.org>, "gcc"
> > <gcc@gcc.gnu.org>, "Hal Finkel" <hfinkel@anl.gov>
> > Sent: Tuesday, February 11, 2014 3:38:40 PM
> > Subject: Re: Fwd: LLVM collaboration?
> > 
> > > My reading of bfd/plugin.c is that it basically walks the
> > > directory
> > > and looks
> > > for first plugin that returns OK for onload. (that is always the
> > > case for
> > > GCC/LLVM plugins).  So if I instlal GCC and llvm plugin there it
> > > will
> > > depend who will end up being first and only that plugin will be
> > > used.
> > >
> > > We need multiple plugin support as suggested by the directory
> > > name
> > > ;)
> > >
> > > Also it sems that currently plugin is not used if file is ELF for
> > > ar/nm/ranlib
> > > (as mentioned by Markus) and also GNU-ld seems to choke on LLVM
> > > object files
> > > even if it has plugin.
> > >
> > > This probably needs ot be sanitized.
> > 
> > CCing Hal Finkel. He got this to work some time ago. Not sure if he
> > ever ported the patches to bfd trunk.
> 
> I have a patch for binutils 2.24 (attached -- I think this works, I
> hand isolated it from my BG/Q patchset). I would not consider it to
> be of upstream quality, but I'd obviously appreciate any assistance
> on making everything clean and proper ;)
> 
>  -Hal
> 
> > 
> > >> For OS X the situation is a bit different. There instead of a
> > >> plugin
> > >> the linker loads a library: libLTO.dylib. When doing LTO with a
> > >> newer
> > >> llvm, one needs to set DYLD_LIBRARY_PATH. I think I proposed
> > >> setting
> > >> that from clang some time ago, but I don't remember the outcome.
> > >>
> > >> In theory GCC could implement a libLTO.dylib and set
> > >> DYLD_LIBRARY_PATH. The gold/bfd plugin that LLVM uses is
> > >> basically
> > >> a
> > >> API mapping the other way, so the job would be inverting it. The
> > >> LTO
> > >> model ld64 is a bit more strict about knowing all symbol
> > >> definitions
> > >> and uses (including inline asm), so there would be work to be
> > >> done
> > >> to
> > >> cover that, but the simple cases shouldn't be too hard.
> > >
> > > I would not care that much about symbols in asm definitions to
> > > start with.
> > > Even if we will force users to non-LTO those object files, it
> > > would
> > > be an
> > > improvement over what we have now.
> > >
> > > One problem is that we need a volunteer to implement the reverse
> > > glue
> > > (libLTO->plugin API), since I do not have an OS X box (well, have
> > > an old G5,
> > > but even that is quite far from me right now)
> > >
> > > Why complete symbol tables are required? Can't ld64 be changed to
> > > ignore
> > > unresolved symbols in the first stage just like gold/gnu-ld does?
> > 
> > I am not sure about this. My *guess* is that it does dead stripping
> > computation before asking libLTO for the object file. I noticed the
> > issue while trying to LTO firefox some time ago.
> > 
> > Cheers,
> > Rafael
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: binutils-2.24-ld-plugin-v2.patch --]
[-- Type: text/x-patch; name="binutils-2.24-ld-plugin-v2.patch", Size: 8067 bytes --]

diff --git a/bfd/elflink.c b/bfd/elflink.c
index 99b7ca1..c2bf9c3 100644
--- a/bfd/elflink.c
+++ b/bfd/elflink.c
@@ -5054,7 +5054,9 @@ elf_link_add_archive_symbols (bfd *abfd, struct bfd_link_info *info)
 	    goto error_return;
 
 	  if (! bfd_check_format (element, bfd_object))
-	    goto error_return;
+	    /* goto error_return; */
+            /* this might be an object understood only by an LTO plugin */
+            bfd_elf_make_object (element);
 
 	  /* Doublecheck that we have not included this object
 	     already--it should be impossible, but there may be
diff --git a/ld/ldfile.c b/ld/ldfile.c
index 16baef8..159a60c 100644
--- a/ld/ldfile.c
+++ b/ld/ldfile.c
@@ -38,6 +38,7 @@
 #ifdef ENABLE_PLUGINS
 #include "plugin-api.h"
 #include "plugin.h"
+#include "elf-bfd.h"
 #endif /* ENABLE_PLUGINS */
 
 bfd_boolean  ldfile_assumed_script = FALSE;
@@ -124,6 +125,7 @@ bfd_boolean
 ldfile_try_open_bfd (const char *attempt,
 		     lang_input_statement_type *entry)
 {
+  int is_obj = 0;
   entry->the_bfd = bfd_openr (attempt, entry->target);
 
   if (verbose)
@@ -168,6 +170,34 @@ ldfile_try_open_bfd (const char *attempt,
 	{
 	  if (! bfd_check_format (check, bfd_object))
 	    {
+#ifdef ENABLE_PLUGINS
+	      if (check == entry->the_bfd
+		  && bfd_get_error () == bfd_error_file_not_recognized
+		  && ! ldemul_unrecognized_file (entry))
+		{
+                  if (plugin_active_plugins_p ()
+                      && !no_more_claiming)
+                    {
+                      int fd = open (attempt, O_RDONLY | O_BINARY);
+                      if (fd >= 0)
+                        {
+                          struct ld_plugin_input_file file;
+
+                          bfd_elf_make_object (entry->the_bfd);
+ 
+                          file.name = attempt;
+                          file.offset = 0;
+                          file.filesize = lseek (fd, 0, SEEK_END);
+                          file.fd = fd;
+                          plugin_maybe_claim (&file, entry);
+
+                          if (entry->flags.claimed)
+                            return TRUE;
+                        }
+                    }
+                }
+#endif /* ENABLE_PLUGINS */
+
 	      if (check == entry->the_bfd
 		  && entry->flags.search_dirs
 		  && bfd_get_error () == bfd_error_file_not_recognized
@@ -303,7 +333,9 @@ success:
      bfd_object that it sets the bfd's arch and mach, which
      will be needed when and if we want to bfd_create a new
      one using this one as a template.  */
-  if (bfd_check_format (entry->the_bfd, bfd_object)
+  if (((is_obj = bfd_check_format (entry->the_bfd, bfd_object))
+       || (bfd_get_format(entry->the_bfd) == bfd_unknown
+           && bfd_get_error () == bfd_error_file_not_recognized))
       && plugin_active_plugins_p ()
       && !no_more_claiming)
     {
@@ -312,6 +344,9 @@ success:
 	{
 	  struct ld_plugin_input_file file;
 
+          if (!is_obj)
+            bfd_elf_make_object (entry->the_bfd);
+
 	  file.name = attempt;
 	  file.offset = 0;
 	  file.filesize = lseek (fd, 0, SEEK_END);
diff --git a/ld/ldlang.h b/ld/ldlang.h
index 2dbec5a..8f05f75 100644
--- a/ld/ldlang.h
+++ b/ld/ldlang.h
@@ -305,6 +305,8 @@ typedef struct lang_input_statement_struct
   const char *target;
 
   struct lang_input_statement_flags flags;
+
+  struct ld_plugin_input_file *plugin_input_file;
 } lang_input_statement_type;
 
 typedef struct
diff --git a/ld/plugin.c b/ld/plugin.c
index 0d5339f..024e109 100644
--- a/ld/plugin.c
+++ b/ld/plugin.c
@@ -36,6 +36,8 @@
 #include <windows.h>
 #endif
 
+#include <sys/mman.h>
+
 /* Report plugin symbols.  */
 bfd_boolean report_plugin_symbols;
 
@@ -114,6 +116,7 @@ static const enum ld_plugin_tag tv_header_tags[] =
   LDPT_REGISTER_CLEANUP_HOOK,
   LDPT_ADD_SYMBOLS,
   LDPT_GET_INPUT_FILE,
+  LDPT_GET_VIEW,
   LDPT_RELEASE_INPUT_FILE,
   LDPT_GET_SYMBOLS,
   LDPT_GET_SYMBOLS_V2,
@@ -429,19 +432,75 @@ add_symbols (void *handle, int nsyms, const struct ld_plugin_symbol *syms)
 /* Get the input file information with an open (possibly re-opened)
    file descriptor.  */
 static enum ld_plugin_status
-get_input_file (const void *handle ATTRIBUTE_UNUSED,
-                struct ld_plugin_input_file *file ATTRIBUTE_UNUSED)
+get_input_file (const void *handle,
+                struct ld_plugin_input_file *file)
 {
+  bfd *abfd = (bfd*) handle;
+  struct ld_plugin_input_file *f;
+
   ASSERT (called_plugin);
-  return LDPS_ERR;
+
+  if (!abfd->usrdata)
+    return LDPS_BAD_HANDLE;
+  f = ((lang_input_statement_type *)(abfd->usrdata))->plugin_input_file;
+  if (!f)
+    return LDPS_BAD_HANDLE;
+
+  if (f->fd < 0)
+    f->fd = open (f->name, O_RDONLY | O_BINARY);
+  memcpy(file, f, sizeof(struct ld_plugin_input_file));
+
+  return LDPS_OK;
 }
 
-/* Release the input file.  */
 static enum ld_plugin_status
-release_input_file (const void *handle ATTRIBUTE_UNUSED)
+get_view (const void *handle, const void **viewp)
 {
+  bfd *abfd = (bfd*) handle;
+  struct ld_plugin_input_file *f;
+  off_t map_offset, offset_diff;
+
   ASSERT (called_plugin);
-  return LDPS_ERR;
+
+  if (!abfd->usrdata)
+    return LDPS_BAD_HANDLE;
+  f = ((lang_input_statement_type *)(abfd->usrdata))->plugin_input_file;
+  if (!f || f->fd <= 0)
+    return LDPS_BAD_HANDLE;
+
+  map_offset = f->offset & ~(sysconf(_SC_PAGE_SIZE) - 1);
+  offset_diff = f->offset - map_offset;
+  /* FIXME: We leak this? */
+  *viewp = (const void*)
+    (((char *) mmap(NULL, f->filesize + offset_diff, PROT_READ, MAP_PRIVATE,
+                    f->fd, map_offset)) + offset_diff);
+  if (!*viewp)
+    return LDPS_ERR;
+
+  return LDPS_OK;
+}
+
+/* Release the input file. */
+static enum ld_plugin_status
+release_input_file (const void *handle)
+{
+  bfd *abfd = (bfd*) handle;
+  struct ld_plugin_input_file *f;
+
+  ASSERT (called_plugin);
+
+  if (!abfd->usrdata)
+    return LDPS_BAD_HANDLE;
+  f = ((lang_input_statement_type *)(abfd->usrdata))->plugin_input_file;
+  if (!f)
+    return LDPS_BAD_HANDLE;
+
+  if (f->fd >= 0) {
+    close(f->fd);
+    f->fd = -1;
+  }
+
+  return LDPS_OK;
 }
 
 /* Return TRUE if a defined symbol might be reachable from outside the
@@ -724,6 +783,9 @@ set_tv_header (struct ld_plugin_tv *tv)
 	case LDPT_GET_INPUT_FILE:
 	  TVU(get_input_file) = get_input_file;
 	  break;
+	case LDPT_GET_VIEW:
+	  TVU(get_view) = get_view;
+	  break;
 	case LDPT_RELEASE_INPUT_FILE:
 	  TVU(release_input_file) = release_input_file;
 	  break;
@@ -867,9 +929,21 @@ plugin_maybe_claim (struct ld_plugin_input_file *file,
      the plugin may want to add.  */
   file->handle = plugin_get_ir_dummy_bfd (entry->the_bfd->filename,
 					  entry->the_bfd);
+
+  /* The plugin might call get_view on the provided handle,
+     prepare for that. */
+  lang_input_statement_type dummy_entry;
+  memset(&dummy_entry, 0, sizeof (lang_input_statement_type));
+  dummy_entry.plugin_input_file = file;
+  ((bfd *) file->handle)->usrdata = &dummy_entry;
+
   if (plugin_call_claim_file (file, &claimed))
     einfo (_("%P%F: %s: plugin reported error claiming file\n"),
 	   plugin_error_plugin ());
+
+  /* Make sure the dummy_entry is not used outside of this scope. */
+  ((bfd *) file->handle)->usrdata = NULL;
+
   /* fd belongs to us, not the plugin; but we don't need it.  */
   close (file->fd);
   if (claimed)
@@ -883,6 +957,17 @@ plugin_maybe_claim (struct ld_plugin_input_file *file,
       entry->the_bfd = file->handle;
       entry->flags.claimed = TRUE;
       bfd_make_readable (entry->the_bfd);
+
+      /* Save a copy of the input file information for use by get_input_file
+       * later. */
+      /* FIXME: We leak this? */
+      entry->plugin_input_file = (struct ld_plugin_input_file *)
+        xcalloc (1, sizeof (struct ld_plugin_input_file));
+      memcpy(entry->plugin_input_file, file,
+             sizeof (struct ld_plugin_input_file));
+      entry->plugin_input_file->fd = -1;
+      entry->plugin_input_file->name = xstrdup(file->name);
+      entry->plugin_input_file->handle = entry->the_bfd;
     }
   else
     {

Re: Fwd: LLVM collaboration?

[Richard Biener]

On Tue, Feb 11, 2014 at 10:20 PM, Jan Hubicka <hubicka@ucw.cz> wrote:
>> >> Since both toolchains do the magic, binutils has no incentive to
>> >> create any automatic detection of objects.
>>
>> It is mostly a historical decision. At the time the design was for the
>> plugin to be matched to the compiler, and so the compiler could pass
>> that information down to the linker.
>>
>> > The trouble however is that one needs to pass explicit --plugin argument
>> > specifying the particular plugin to load and so GCC ships with its own wrappers
>> > (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
>>
>> These wrappers should not be necessary. While the linker currently
>> requires a command line option, bfd has support for searching for a
>> plugin. It will search <inst>/lib/bfd-plugin. See for example the
>> instructions at http://llvm.org/docs/GoldPlugin.html.
>
> My reading of bfd/plugin.c is that it basically walks the directory and looks
> for first plugin that returns OK for onload. (that is always the case for
> GCC/LLVM plugins).  So if I instlal GCC and llvm plugin there it will
> depend who will end up being first and only that plugin will be used.
>
> We need multiple plugin support as suggested by the directory name ;)
>
> Also it sems that currently plugin is not used if file is ELF for ar/nm/ranlib
> (as mentioned by Markus) and also GNU-ld seems to choke on LLVM object files
> even if it has plugin.
>
> This probably needs ot be sanitized.
>
>>
>> This was done because ar and nm are not normally bound to any
>> compiler. Had we realized this issue earlier we would probably have
>> supported searching for plugins in the linker too.
>>
>> So it seems that what you want could be done by
>>
>> * having bfd-ld and gold search bfd-plugins (maybe rename the directory?)
>> * support loading multiple plugins, and asking each to see if it
>> supports a given file. That ways we could LTO when having a part GCC
>> and part LLVM build.
>
> Yes, that is what I have in mind.
>
> Plus perhaps additional configuration file to avoid loading everything.  Say
> user instealls 3 versions of LLVM, open64 and ICC. If all of them loads as a
> shared library, like LLVM does, it will probably slow down the tools
> measurably.

What about instead of our current odd way of identifying LTO objects
simply add a special ELF note telling the linker the plugin to use?

.note._linker_plugin '/...../libltoplugin.so'

that way the linker should try 1) loading that plugin, 2) register the
specific object with that plugin.

If a full path is undesired (depends on install setup) then specifying
the plugin SONAME might also work (we'd of course need to bump
our plugins SONAME for each release to allow parallel install
of multiple versions or make the plugin contain all the
dispatch-to-different-GCC-version-lto-wrapper code).

Richard.

>> * maybe be smart about version and load new ones first? (libLLVM-3.4
>> before libLLVM-3.3 for example). Probably the first one should always
>> be the one given in the command line.
>
> Yes, i think we may want to prioritize the list.  So user can prevail
> his own version of GCC over the system one, for example.
>>
>> For OS X the situation is a bit different. There instead of a plugin
>> the linker loads a library: libLTO.dylib. When doing LTO with a newer
>> llvm, one needs to set DYLD_LIBRARY_PATH. I think I proposed setting
>> that from clang some time ago, but I don't remember the outcome.
>>
>> In theory GCC could implement a libLTO.dylib and set
>> DYLD_LIBRARY_PATH. The gold/bfd plugin that LLVM uses is basically a
>> API mapping the other way, so the job would be inverting it. The LTO
>> model ld64 is a bit more strict about knowing all symbol definitions
>> and uses (including inline asm), so there would be work to be done to
>> cover that, but the simple cases shouldn't be too hard.
>
> I would not care that much about symbols in asm definitions to start with.
> Even if we will force users to non-LTO those object files, it would be an
> improvement over what we have now.
>
> One problem is that we need a volunteer to implement the reverse glue
> (libLTO->plugin API), since I do not have an OS X box (well, have an old G5,
> but even that is quite far from me right now)
>
> Why complete symbol tables are required? Can't ld64 be changed to ignore
> unresolved symbols in the first stage just like gold/gnu-ld does?
>
> Honza
>>
>> Cheers,
>> Rafael

Re: Fwd: LLVM collaboration?

[Rafael Espíndola]

> What about instead of our current odd way of identifying LTO objects
> simply add a special ELF note telling the linker the plugin to use?
>
> .note._linker_plugin '/...../libltoplugin.so'
>
> that way the linker should try 1) loading that plugin, 2) register the
> specific object with that plugin.
>
> If a full path is undesired (depends on install setup) then specifying
> the plugin SONAME might also work (we'd of course need to bump
> our plugins SONAME for each release to allow parallel install
> of multiple versions or make the plugin contain all the
> dispatch-to-different-GCC-version-lto-wrapper code).

Might be an interesting addition to what we have, but keep in mind
that LLVM uses thin non-ELF files. It is also able to load IR from
previous versions, so for LLVM at least, using the newest plugin is
probably the best default.

> Richard.

Cheers,
Rafael

Re: Fwd: LLVM collaboration?

[Joseph S. Myers]

On Wed, 12 Feb 2014, Richard Biener wrote:

> What about instead of our current odd way of identifying LTO objects
> simply add a special ELF note telling the linker the plugin to use?
> 
> .note._linker_plugin '/...../libltoplugin.so'
> 
> that way the linker should try 1) loading that plugin, 2) register the
> specific object with that plugin.

Unless this is only allowed for a whitelist of known-good plugins in 
known-good directories, it's a clear security hole for the linker to 
execute code in arbitrary files named by linker input.  The linker should 
be safe to run on untrusted input files.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Fwd: LLVM collaboration?

[Jan Hubicka]

> On Wed, 12 Feb 2014, Richard Biener wrote:
> 
> > What about instead of our current odd way of identifying LTO objects
> > simply add a special ELF note telling the linker the plugin to use?
> > 
> > .note._linker_plugin '/...../libltoplugin.so'
> > 
> > that way the linker should try 1) loading that plugin, 2) register the
> > specific object with that plugin.
> 
> Unless this is only allowed for a whitelist of known-good plugins in 
> known-good directories, it's a clear security hole for the linker to 
> execute code in arbitrary files named by linker input.  The linker should 
> be safe to run on untrusted input files.

Also I believe the flies should be independent of particular setup (that is not
contain a path) and probably host OS (that is not having .so extension) at least.
We need some versioning scheme for different versions of compilers.
Finally we need a solution for non-ELF LTO objects (like LLVM)

But yes, having an compiler independent way of declaring that plugin is needed
and what plugin should be uses seems possible.

Honza
> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com

Re: Fwd: LLVM collaboration?

[Richard Biener]

On Wed, Feb 12, 2014 at 5:22 PM, Jan Hubicka <hubicka@ucw.cz> wrote:
>> On Wed, 12 Feb 2014, Richard Biener wrote:
>>
>> > What about instead of our current odd way of identifying LTO objects
>> > simply add a special ELF note telling the linker the plugin to use?
>> >
>> > .note._linker_plugin '/...../libltoplugin.so'
>> >
>> > that way the linker should try 1) loading that plugin, 2) register the
>> > specific object with that plugin.
>>
>> Unless this is only allowed for a whitelist of known-good plugins in
>> known-good directories, it's a clear security hole for the linker to
>> execute code in arbitrary files named by linker input.  The linker should
>> be safe to run on untrusted input files.
>
> Also I believe the flies should be independent of particular setup (that is not
> contain a path) and probably host OS (that is not having .so extension) at least.
> We need some versioning scheme for different versions of compilers.
> Finally we need a solution for non-ELF LTO objects (like LLVM)
>
> But yes, having an compiler independent way of declaring that plugin is needed
> and what plugin should be uses seems possible.

Yeah, naming the plugin (and searching it in a ld specific trusted configurable
path only) would work as well, of course.

That also means that we should try to make the GCC side lto-plugin work for
older GCC versions as well (we pick the lto-wrapper to call from the environment
which would have to change if we'd try to support using multiple GCC versions
at the same time).

Richard.

> Honza
>>
>> --
>> Joseph S. Myers
>> joseph@codesourcery.com