Progress on GCC plugins ?

Progress on GCC plugins ?

[Emmanuel Fleury]

Hi,

Is there any progress in the gcc-plugin project ?

How far is the code and what about its integration in a future release ?

I still think this is a great idea and I'm quite impatient to try it out
(I'll have some time in January).

Regards
-- 
Emmanuel Fleury

Out the 10Base-T, through the router, down the T1, over the leased
line, off the bridge, past the firewall...nothing but Net.
  -- Tony Miller

Re: Progress on GCC plugins ?

[Joe Buck]

On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
> Is there any progress in the gcc-plugin project ?

Non-technical holdups.  RMS is worried that this will make it too easy
to integrate proprietary code directly with GCC.

If proponents can come up with good arguments about how the plugin
project can be structured to avoid this risk, that would help.

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Joe Buck wrote:
> On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
>> Is there any progress in the gcc-plugin project ?
> 
> Non-technical holdups.  RMS is worried that this will make it too easy
> to integrate proprietary code directly with GCC.
> 
> If proponents can come up with good arguments about how the plugin
> project can be structured to avoid this risk, that would help.


I thought (from my memory and understanding of discussions at last GCC 
summit in July 2O07 in Ottawa) that a major argument against this risk 
is that there is no stability in the API offered by GCC to plugins.

So any proprietary plugin would be a nightmare to maintain...


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Tom Tromey]

>>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:

Joe> On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
>> Is there any progress in the gcc-plugin project ?

Joe> Non-technical holdups.  RMS is worried that this will make it too easy
Joe> to integrate proprietary code directly with GCC.

Joe> If proponents can come up with good arguments about how the plugin
Joe> project can be structured to avoid this risk, that would help.

First, aren't we already in this situation?  There are at least 2
compilers out there that re-use parts of GCC by serializing trees and
then reading them into a different back end.

Second, how does the LTO project avoid this same worry?

Tom

Re: Progress on GCC plugins ?

[Robert Dewar]

Tom Tromey wrote:

> First, aren't we already in this situation?  There are at least 2
> compilers out there that re-use parts of GCC by serializing trees and
> then reading them into a different back end.

It's not obvious to me that this is consistent with the GPL ..
interesting issue ...
> 
> Second, how does the LTO project avoid this same worry?
> 
> Tom

Re: Progress on GCC plugins ?

[Chris Lattner]

On Nov 7, 2007, at 9:28 AM, Robert Dewar wrote:
> Tom Tromey wrote:
>
>> First, aren't we already in this situation?  There are at least 2
>> compilers out there that re-use parts of GCC by serializing trees and
>> then reading them into a different back end.
>
> It's not obvious to me that this is consistent with the GPL ..
> interesting issue ...

Assuming these projects are "linking" it in, it just means the  
resulting compiler is also be GPL.  That seems consistent.

-Chris

Re: Progress on GCC plugins ?

[Florian Weimer]

* Robert Dewar:

> Tom Tromey wrote:
>
>> First, aren't we already in this situation?  There are at least 2
>> compilers out there that re-use parts of GCC by serializing trees and
>> then reading them into a different back end.
>
> It's not obvious to me that this is consistent with the GPL ..

It requires a fairly expansionist view of copyright to make it
inconsistent.  The FSF has made similar claims in the past (for
instance, that all Emacs Lisp code must be GPLed), but I'm not convinced
that arguing for more Draconian copyright laws helps the free software
cause.  Copyleft requires compromises in this area, of course, but some
deals should be off limits.

Re: Progress on GCC plugins ?

[Robert Dewar]

Florian Weimer wrote:
> * Robert Dewar:
> 
>> Tom Tromey wrote:
>>
>>> First, aren't we already in this situation?  There are at least 2
>>> compilers out there that re-use parts of GCC by serializing trees and
>>> then reading them into a different back end.
>> It's not obvious to me that this is consistent with the GPL ..
> 
> It requires a fairly expansionist view of copyright to make it
> inconsistent. 

I disagree (based significantly on the proceedings of the Intergraph
vs Bentley trial, which unfortunately are not easily published 
anywhere), but I think this thread should not go too much farther here, 
it is really off topic.

RE: Progress on GCC plugins ?

[Dave Korn]

On 07 November 2007 16:41, Joe Buck wrote:

> On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
>> Is there any progress in the gcc-plugin project ?
> 
> Non-technical holdups.  RMS is worried that this will make it too easy
> to integrate proprietary code directly with GCC.
> 
> If proponents can come up with good arguments about how the plugin
> project can be structured to avoid this risk, that would help.

  I don't understand: why wouldn't designing it so that they have to be
implemented as DSOs and hence are covered by the
anything-directly-linked-in-is-GPL'd clause do the job?  Or is the concern
that people will write trivial marshalling plugins and ship all the data out
across a pipe or socket to some proprietary code and then only release source
for their shim layers?


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: Progress on GCC plugins ?

[David Edelsohn]

>>>>> Dave Korn writes:

Dave> I don't understand: why wouldn't designing it so that they have to be
Dave> implemented as DSOs and hence are covered by the
Dave> anything-directly-linked-in-is-GPL'd clause do the job?  Or is the concern
Dave> that people will write trivial marshalling plugins and ship all the data out
Dave> across a pipe or socket to some proprietary code and then only release source
Dave> for their shim layers?

	The concern is the many forms of shim layers that possibly could
be written more easily with a plug-in framework.

David

Re: Progress on GCC plugins ?

[Robert Dewar]

David Edelsohn wrote:
>>>>>> Dave Korn writes:
> 
> Dave> I don't understand: why wouldn't designing it so that they have to be
> Dave> implemented as DSOs and hence are covered by the
> Dave> anything-directly-linked-in-is-GPL'd clause do the job?  Or is the concern
> Dave> that people will write trivial marshalling plugins and ship all the data out
> Dave> across a pipe or socket to some proprietary code and then only release source
> Dave> for their shim layers?
> 
> 	The concern is the many forms of shim layers that possibly could
> be written more easily with a plug-in framework.

there is also a difference in these two scenarios:

1. a) Company X writes a modification to GCC to generate special
intermediate stuff with format Y.

   b) Company X writes propietary back end which can only
read format Y stuff written by that modification.

2. Same as 1, except that the GCC project does step a)
thus semi-standardizing the output.

TO me it is pretty clear that these are very different
situations from a license point of view.
> 
> David

Re: Progress on GCC plugins ?

[Brendon Costa]

>>     The concern is the many forms of shim layers that possibly could
>> be written more easily with a plug-in framework.
> 
> there is also a difference in these two scenarios:
> 
> 1. a) Company X writes a modification to GCC to generate special
> intermediate stuff with format Y.
> 
>   b) Company X writes propietary back end which can only
> read format Y stuff written by that modification.
> 
> 2. Same as 1, except that the GCC project does step a)
> thus semi-standardizing the output.
> 
> TO me it is pretty clear that these are very different
> situations from a license point of view.

I do exactly point 1 for my (Open Source) C++ exception static analysis
tool:
http://edoc.sourceforge.net/

I need a subset of the data in a format that is easy for me to process
and it must stay around for a long period of time so i dont want all the
information that GCC could generate in a more generic form. So in this
case i feel a non-standardized format is best suited for my project.
This same argument would apply for other projects too, though i can see
how it can be helpful to have a single more generic format. It just
might not be suitable in a lot of cases. If you already have a plugin
distributed with GCC that writes in a generic format, then most people
will use that anyway if they can, rather than write and maintain their
own plugin.

This is a problem with or without the plugin framework. By adding
plugins you have the same issues as before but just make it easier for
all developers whether open source or proprietary to develop new GCC
functionality.

My project is an example of using a patch against GCC in order to
achieve the "shim layer". I would much prefer to do this with a plugin,
but a lack of the plugin framework is not going to stop me doing it
whatever way i can.

Brendon.

Re: Progress on GCC plugins ?

[Robert Dewar]

Brendon Costa wrote:
>>>     The concern is the many forms of shim layers that possibly could
>>> be written more easily with a plug-in framework.
>> there is also a difference in these two scenarios:
>>
>> 1. a) Company X writes a modification to GCC to generate special
>> intermediate stuff with format Y.
>>
>>   b) Company X writes propietary back end which can only
>> read format Y stuff written by that modification.
>>
>> 2. Same as 1, except that the GCC project does step a)
>> thus semi-standardizing the output.
>>
>> TO me it is pretty clear that these are very different
>> situations from a license point of view.
> 
> I do exactly point 1 for my (Open Source) C++ exception static analysis
> tool:
> http://edoc.sourceforge.net/

Well assuming that your tool is GPL'ed there is no issue and
this is not a case of 1 above!

Re: Progress on GCC plugins ?

[Brendon Costa]

Robert Dewar wrote:
> Brendon Costa wrote:
>>>>     The concern is the many forms of shim layers that possibly could
>>>> be written more easily with a plug-in framework.
>>> there is also a difference in these two scenarios:
>>>
>>> 1. a) Company X writes a modification to GCC to generate special
>>> intermediate stuff with format Y.
>>>
>>>   b) Company X writes propietary back end which can only
>>> read format Y stuff written by that modification.
>>>
>>> 2. Same as 1, except that the GCC project does step a)
>>> thus semi-standardizing the output.
>>>
>>> TO me it is pretty clear that these are very different
>>> situations from a license point of view.
>>
>> I do exactly point 1 for my (Open Source) C++ exception static analysis
>> tool:
>> http://edoc.sourceforge.net/
> 
> Well assuming that your tool is GPL'ed there is no issue and
> this is not a case of 1 above!
> 

The patch against GCC is GPL, the main library that is capable of
manipulating the data exported by the patched GCC is LGPL and could
theoretically be under any license.

What i was trying to point out is that proprietary projects can
already (without plugins) make exporters for GCC which are GPL and
then create the majority of their code in other closed source apps
that use that data.

I don't see plugins as changing this except to make it easier. Which
is really a major reason for proving plugins isn't it? Making it
easier to provide additional GCC features?

My project was given as an example of how it could be done currently
without plugins even though this project is open source.

Brendon.

Re: Progress on GCC plugins ?

[Robert Dewar]

Brendon Costa wrote:

> The patch against GCC is GPL, the main library that is capable of
> manipulating the data exported by the patched GCC is LGPL and could
> theoretically be under any license.

Whose theory? You don't know that!
> 
> What i was trying to point out is that proprietary projects can
> already (without plugins) make exporters for GCC which are GPL and
> then create the majority of their code in other closed source apps
> that use that data.

You don't know that!
> 
> I don't see plugins as changing this except to make it easier. Which
> is really a major reason for proving plugins isn't it? Making it
> easier to provide additional GCC features?

No, it has other changes, since it establishes a standardized
interface. In my view (from the point of view of an expert
witness in copyright matters), this *does* make a big difference
from a licensing point of view. Of course that's just my view,
I don't know either, but I know that I don't know :-)
> 
> My project was given as an example of how it could be done currently
> without plugins even though this project is open source.

The issue is not can it be done, but what are the licensing issues?
And without litigation, no one really knows.
> 
> Brendon.

Re: Progress on GCC plugins ?

[David Edelsohn]

	If you want to have a legal discussion, please take this
conversation somewhere else.

Thanks, David

Re: Progress on GCC plugins ?

[Brendon Costa]

David Edelsohn wrote:
> 	If you want to have a legal discussion, please take this
> conversation somewhere else.
> 
> Thanks, David
> 

Sorry. I just posted another email before i got this. Is there a
suitable place to move this discussion besides private emails?

Thanks,
Brendon.

Re: Progress on GCC plugins ?

[Brendon Costa]

Robert Dewar wrote:
> Brendon Costa wrote:
> 
>> The patch against GCC is GPL, the main library that is capable of
>> manipulating the data exported by the patched GCC is LGPL and could
>> theoretically be under any license.
> 
> Whose theory? You don't know that!

I thought it was obvious :-) My Theory... A theory is not necessarily
true...

I will clarify where i am coming from... I am a software developer and
know very little about the legal side of things. What i have said is
really the way i understand things as i have tried ti have a basic
idea of what all this entails and how it affects my work.

If the license extends to the data generated by GPL apps (Which is
really what I think we are talking about), then wouldn't the binaries
or for that matter the intermediate source files generated by GCC (for
example using g++ -save-temps) also be covered under the GPL
regardless of the license of the source being compiled?

This data could include:
* object files
* intermediate source files (.i, .s)
* pre-compiled header files
* any other form of the original source serialized into some specific
format such as GIMPLE exports etc.

The problem with this is that each of these is really just a different
representation of the original source code. This complicates matters
even more if we have both the GPL of GCC and the license of the
original source code impacting on what is generated. Wouldn't it mean
that only certain types of licensed source code are allowed to be
compiled with GCC?


Where do you draw the limit of what is covered and what is not?

It seems to me from what you have said, nothing is safe from the GPL.
If an OS like BSD compiles itself with GCC and this mandates that
anything which uses that data must also be licensed under GPL, then
they have no choice but to say only GPL code is allowed to run on this
OS. I dont see that as being the current common view. Most of the
BSD's have a non-GPL license and still use GCC to compile.

Or are you saying that the FORMAT of the data exported is covered by
GPL, not the data itself? Does that mean if you design a particular
data format in a GPL app, that you are not allowed to write a non-GPL
application that uses data in that format? Does this also apply the
other way around?

I dont know if microsoft have some sort of license over the PExe
formation for binaries. But if so, then is GCC actually ALLOWED to
export data in that format? Look at GIF.

Then i guess it is worth asking, does the GPL applied to the code of
the plugin automatically then apply to the format of the data exported?

Sorry for the long email. I am just trying to understand the issues
involved.

>>
>> I don't see plugins as changing this except to make it easier. Which
>> is really a major reason for proving plugins isn't it? Making it
>> easier to provide additional GCC features?
> 
> No, it has other changes, since it establishes a standardized
> interface. In my view (from the point of view of an expert
> witness in copyright matters), this *does* make a big difference
> from a licensing point of view. Of course that's just my view,
> I don't know either, but I know that I don't know :-)

Are we talking about the standardized interface provided by GCC that
people can use to write plugins for? If so i would agree that anything
which uses this interface must be GPL. So the code for a plugin of GCC
must also be GPL.

If talking about the "interface" that is generated by the export of
data in a particular format. I still dont see how that affects the
tools that make use of that data format unless the particular format
has a license imposed on it.

Again, this is all just my view and I am *NOT* an expert in this
field. In fact, i have had almost NO experience in legal areas. I am
curious to know where i have mis-understood the application of the GPL
to the GCC project and how that might apply to others like my EDoc++
project.

Thanks,
Brendon.

Re: Progress on GCC plugins ?

[Alexandre Oliva]

On Nov  7, 2007, Brendon Costa <bcosta@avdat.com.au> wrote:

> If the license extends to the data generated by GPL apps

It doesn't, AFAIK, but IANAL.  But the issue is not so much the
specific data, but the format in which the data is and the reasons
behind choosing that format.

The FSF might successfully present to a court an argument such as:

  Look, Your Honor, it is quite obvious that the code in question was
  developed to work as a single program, and that the separation
  through the intermedia representation layer is just an attempt to
  escape the conditions of the license of the program they based their
  work on.  Please don't permit the defendant to keep on distributing
  that code without complying with the license conditions.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: Progress on GCC plugins ?

[Richard Kenner]

> The FSF might successfully present to a court an argument such as:
> 
>   Look, Your Honor, it is quite obvious that the code in question was
>   developed to work as a single program, and that the separation
>   through the intermedia representation layer is just an attempt to
>   escape the conditions of the license of the program they based their
>   work on.  Please don't permit the defendant to keep on distributing
>   that code without complying with the license conditions.

Yes, such an argument might be successful and indeed many people, including
myself, believe it *likely* would be successful, but going to court is in
many way a "crap shoot" and it might *not* be successful.  If that happened
it would set a precedent that could be extended on in a way that would be
harmful to Free Software.  That's why it's considered far better to try
to avoid such a case in the first place.

Moreover, if we did this in conjunction with a carefully-defined API, the
defendant could argue that the purpose of that API was precisely to *not*
have it be viewed as a single program, so the result of such a case would
be far less certain.

As was pointed out here numerous times, 90+% of the difficulty in
working with is the difficulty in understanding the inherently-complex
algorithmic structure of the compiler and its data structures (no
real-world compiler will ever be able to use algorithms unmodified from
a textbook) coupled with limited documentation.

Doing something to make the remaining few percent easier will not, in my
opinion, increase the participation in GCC enough to justify the legal
risk to Free Software.

RE: Progress on GCC plugins ?

[Dave Korn]

On 07 November 2007 17:52, David Edelsohn wrote:

> 
> 	The concern is the many forms of shim layers that possibly could
> be written more easily with a plug-in framework.

  I wonder if we could adapt some kind of privsep model, so that once the
compiler has finished init, it gives away its rights and can't even open files
or create pipes or sockets any more.  The gcc.c driver could remain at full
user prives and take care of opening everything (a bit like I guess it already
does when you're using -pipe?) and cc1 and pals could all be hobbled.


  Ooooor..... how about we define plugins in an interpreted or bytecoded
language and don't allow any IO whatsoever?


  Oooooorrrr.... if we were really clever we could maybe define an interface
that's entirely non-enumerable: it calls out to hooks, providing them with
just enough information and interfaces to do the job they have to do, but we
don't make it possible to derive information about the overall AST because we
don't provide any way to know 'what's out there'.


  #1 seems like there might too easily be loopholes even assuming it can
actually be made to work, that is to say that it would be very hard to really
prevent data escaping from the process boundary using the unix fs perms model.
Maybe the more modern unices with finergrained acls and kernel object
permissions would be able to make work, but I think that the lesson of chroot
jails is that they're to prevent fat-finger errors, not determined security
attackers.

  #3 isn't necessarily possible either.  It would probably need serious maths
formalisms and proofs to define.  Zero-knowledge secret problem sharing based
plugins, anyone?

  #2 might well be a goer.  It looks pretty plausible.

    cheers,
      DaveK [carefully avoiding the IANAL debate :-)]
-- 
Can't think of a witty .sigline today....

Re: Progress on GCC plugins ?

[Brendon Costa]

Joe Buck wrote:
> On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
>> Is there any progress in the gcc-plugin project ?
> 
> Non-technical holdups.  RMS is worried that this will make it too easy
> to integrate proprietary code directly with GCC.
> 
> If proponents can come up with good arguments about how the plugin
> project can be structured to avoid this risk, that would help.
> 

Is there anything that still needs to be done on the technical side at
all like testing or development? I am willing to help out a bit. I
have an interest in seeing this in a future release :-)

Thanks,
Brendon.

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Joe Buck <Joe.Buck@synopsys.COM> writes:

> On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
> > Is there any progress in the gcc-plugin project ?
> 
> Non-technical holdups.  RMS is worried that this will make it too easy
> to integrate proprietary code directly with GCC.
> 
> If proponents can come up with good arguments about how the plugin
> project can be structured to avoid this risk, that would help.

It seems very likely that it would be possible to write a plugin which
would generate IR to be fed into a proprietary backend.  Sun's GCCfss
is an existing example of what could be done in this area.  Of course,
GCCfss also shows that if people want to do this, they will.  Adding a
plugin framework doesn't even make it notably easier.  Once you've
written code to translate GIMPLE into some other IR, dropping it into
gcc is a matter of changing a few lines.  Providing a plugin interface
is not the issue.

More deeply, I think his concern is misplaced.  I think that gcc has
already demonstrated that the only widely used compilers are free
software.  Proprietary compilers don't keep up over time, outside of
niche markets.  Hooking proprietary code into gcc, one way or another,
is just going to create a dead end for the people who do it.
Certainly it's not a good thing, and certainly it would be preferable
to prevent it if possible.  But it is not the worst possible thing
that could happen; it is merely a cost.

I won't enumerate the benefits of plugins here.  But it is clear to me
that the benefits outweigh the costs.

Ian

Re: Progress on GCC plugins ?

[Emmanuel Fleury]

Ian Lance Taylor wrote:
> 
> More deeply, I think his concern is misplaced.  I think that gcc has
> already demonstrated that the only widely used compilers are free
> software.  Proprietary compilers don't keep up over time, outside of
> niche markets.  Hooking proprietary code into gcc, one way or another,
> is just going to create a dead end for the people who do it.
> Certainly it's not a good thing, and certainly it would be preferable
> to prevent it if possible.  But it is not the worst possible thing
> that could happen; it is merely a cost.
> 
> I won't enumerate the benefits of plugins here.  But it is clear to me
> that the benefits outweigh the costs.

I won't add anything to the comments of Ian because I fully agree with
him but I would like to clarify what I expect from GCC.

First of all, I'm coming from the verification community and I'm
currently very interested in '(automatic) code verification'. What our
community is really seeking for by now is a tool aiming at providing an
abstract (i.e. simplified) model of program sources. And actually, GCC
do it internally. Indeed, you can provide a gimplified CFG of C, C++,
Java, ... source files and many other informations, which overpass tools
such as CIL (C Intermediate Language, http://manju.cs.berkeley.edu/cil/)
because of the numerous front-end of GCC.

What I would like to have is just some mean to extract the last step
before the translation into RTL (usually 'final_cleanup').

Surely, having the possibility to decorate the code with some extra
information would be something more but we can actually work-around
without any problem considering the benefits we get from not having to
maintain all the front-ends.

What is providing the 'plugin scheme' is actually much more than we need
because it goes deep inside GCC and allow *modification* of GCC
internals (and therefore might break a lot of things, I guess) which (in
my humble opinion) is the real problem here.

Maybe defining some standardized outputs (very similar to -fdump-tree-*)
targeted for static-analysis and model-checking tools would be enough as
the main problem for us is just to get GCC to export its intermediate
internal representations of the program in a way which is reliable and
stable from one version to another.

For the software I'm thinking of I really just need the final CFG and
maybe a summary of the IPA would help as well.

Regards
-- 
Emmanuel Fleury

If you don't get everything you want,
think of the things you don't get that you don't want.
  -- Oscar Wilde

Re: Progress on GCC plugins ?

[Mark Mitchell]

Ian Lance Taylor wrote:

> It seems very likely that it would be possible to write a plugin which
> would generate IR to be fed into a proprietary backend. 
> 
> More deeply, I think his concern is misplaced.

For the record, I agree on both points.

There are some ways in which plugins might make getting proprietary code
in the loop easier.  For example, if someone plugs Guile into the plugin
interface, then it might be easy to write a pile of LISP code to poke at
GCC.  I understand that there are differing opinions on whether or not
these kinds of indirections affect the GPL.  But, that's the kind of
thing that the FSF gets to think about.

But, as you say, the whole idea of the GPL is that people can go build
what they want -- including plugin frameworks to GCC that might or might
not allow them to interface with proprietary code.  If it's really worth
it to them, then they will.  The FSF's best defense is to make it not
worth it by providing enough valuable free software.

Anyhow, in practical terms, debating this here probably will have zero
impact on the outcome.  The ball is in RMS' court, and SC members
(including myself) have made many of the arguments that have been made
in this thread.  If people want to influence the FSF, the best approach
is probably going to be sending mail directly to RMS, not discussing it
on this list.

Thanks,

-- 
Mark Mitchell
CodeSourcery
mark@codesourcery.com
(650) 331-3385 x713

RE: Progress on GCC plugins ?

[Gerald.Williams]

Mark Mitchell wrote:
> Anyhow, in practical terms, debating this here probably will have zero
> impact on the outcome.  The ball is in RMS' court, and SC members
> (including myself) have made many of the arguments that have been made
> in this thread.  If people want to influence the FSF, the best
approach
> is probably going to be sending mail directly to RMS, not discussing
it
> on this list.

I think your opinion probably carries a bit more weight
with RMS than mine. :-)

I don't want to prolong a pointless discussion either,
but I hope somebody has made the "forest through the
trees" argument. Proprietary extensions using a standard
plug-in interface will eventually get replaced by free
ones if they're of value. In the long term, this type of
direct assault on GPL would backfire in a big way.

-Jerry

Re: Progress on GCC plugins ?

[Diego Novillo]

Joe Buck wrote:
> On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
>> Is there any progress in the gcc-plugin project ?
> 
> Non-technical holdups.  RMS is worried that this will make it too easy
> to integrate proprietary code directly with GCC.

I don't believe this is a strong argument.  My contention is, and has 
always been, that GCC is _already_ trivial to integrate into a 
proprietary compiler.  There is at least one compiler I know that does this.

In fact, much/most of the effort we have been spending in making GCC 
easier to maintain, necessarily translates into a system that is easier 
to integrate into other code bases.

IMO, the benefits we gain in making GCC a more attractive code base, far 
outweigh the fears of someone co-opting it for their own proprietary uses.

We gain nothing by holding infrastructure advances in GCC.  While GCC 
still has the advantage of being widely used, its internal 
infrastructure is still relatively arcane and hard to deal with.  We 
have already kicked it into the mid 90s, but we still have a lot of 
ground to cover.  An antiquated and arcane infrastructure will only help 
turn new developers away.


Diego.

Re: Progress on GCC plugins ?

[Joe Buck]

On Thu, Nov 15, 2007 at 02:34:38PM -0500, Diego Novillo wrote:
> Joe Buck wrote:
> >On Wed, Nov 07, 2007 at 09:20:21AM +0100, Emmanuel Fleury wrote:
> >>Is there any progress in the gcc-plugin project ?
> >
> >Non-technical holdups.  RMS is worried that this will make it too easy
> >to integrate proprietary code directly with GCC.
> 
> I don't believe this is a strong argument.  My contention is, and has 
> always been, that GCC is _already_ trivial to integrate into a 
> proprietary compiler.  There is at least one compiler I know that does this.

I agree, but we still have the roadblock.

Maybe we need a group of volunteers to meet with RMS in person and work on
convincing him.  E-mail seems way too inefficient and frustrating a
mechanism.

RMS regularly points to the examples of C++ and Objective-C as an argument
for trying to force all extensions to be GPL (Mike Tiemann's employer
tried to figure out a way of making g++ proprietary; NeXT tried a
"user does the link" hack to get around the GPL for their original
Objective-C compiler).

Problem is, he hasn't really kept up; the problem with being as pure as
he is is that you can become isolated from what's going on.

Re: Progress on GCC plugins ?

[Richard Kenner]

> I don't believe this is a strong argument.  My contention is, and has 
> always been, that GCC is _already_ trivial to integrate into a 
> proprietary compiler.  There is at least one compiler I know that does this.

I believe that any such compiler would violate the GPL.  But I also believe
it's not in the best interest of the FSF to litigate that matter if the
linkage between the compiler is anything other than linked in a single
executable.  Therefore, I think it's important for us to make it as
technically hard as possible for people to do such a linkage by readin and
writing tree or communicating as different libraries or DLLs.  I'm very
much against any sort of "plug in" precisely for this reason.

> IMO, the benefits we gain in making GCC a more attractive code base, far 
> outweigh the fears of someone co-opting it for their own proprietary uses.

That depends on the importance you attach to the philosophy of free
software.  I suspect RMS attaches much more importance to that than anybody
on this list.

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:

> > I don't believe this is a strong argument.  My contention is, and has 
> > always been, that GCC is _already_ trivial to integrate into a 
> > proprietary compiler.  There is at least one compiler I know that does this.
> 
> I believe that any such compiler would violate the GPL.  But I also believe
> it's not in the best interest of the FSF to litigate that matter if the
> linkage between the compiler is anything other than linked in a single
> executable.  Therefore, I think it's important for us to make it as
> technically hard as possible for people to do such a linkage by readin and
> writing tree or communicating as different libraries or DLLs.  I'm very
> much against any sort of "plug in" precisely for this reason.

We can make it as technically hard as possible, but it's way too late
to make it technically hard.  In fact, it's easy.  You have to write
some code to translate from tree to your proprietary IR, and then you
have to plug that code into passes.c.

If gcc supports plugins, then all we've eliminated is the need to plug
that code into passes.c.  But that is the easiest part of the job.
Adding plugins is not going to require us to support a stable tree
interface or anything along those lines; if it did, I would oppose
that.

So this seems to me to be a very weak argument against plugins.
Adding plugins does not make it noticeably easier to integrate gcc's
frontend with a proprietary compiler.  And adding plugins would not
change the issue of whether such a combination violated the GPL.

Do you disagree with this assessment?


I think it's quite important for gcc's long-term health to permit and
even encourage academic researchers and students to use it.  And I see
plugins as directly supporting that goal.  Note that I don't see any
problem with requiring (or attempting to require) that any plugin be
covered by the GPL.

So from my perspective the downside of plugins is very small, and the
upside is very large.

Ian

Re: Progress on GCC plugins ?

[Richard Kenner]

> In fact, it's easy.  You have to write some code to translate from
> tree to your proprietary IR, and then you have to plug that code
> into passes.c.

Well first of all, that code becomes GPL so the IR isn't truely "proprietary".

> So this seems to me to be a very weak argument against plugins.
> Adding plugins does not make it noticeably easier to integrate gcc's
> frontend with a proprietary compiler.  And adding plugins would not
> change the issue of whether such a combination violated the GPL.
> 
> Do you disagree with this assessment?

No, not in that case, but I don't see that as the only case.  Another
case would be somebody who wanted to keep an optimizer proprietary by
making it a plug-in.  My view is that because of the linkage with the
GCC IR, it can't be proprietary in that case, but that's the harder argument
to make legally.

> I think it's quite important for gcc's long-term health to permit and
> even encourage academic researchers and students to use it.  And I see
> plugins as directly supporting that goal.  

I don't see that.  Why is it that much harder to link in with GCC than doing
it as a plugin?

Re: Progress on GCC plugins ?

[Diego Novillo]

Richard Kenner wrote:

> No, not in that case, but I don't see that as the only case.  Another
> case would be somebody who wanted to keep an optimizer proprietary by
> making it a plug-in.  My view is that because of the linkage with the
> GCC IR, it can't be proprietary in that case, but that's the harder argument
> to make legally.

I don't think we can argue legalities here.  All we can do is say that 
if they are using plug-ins, they are forced to include GPL'd header 
files and link against GPL'd code.  The rest is up to the FSF legal folks.

> I don't see that.  Why is it that much harder to link in with GCC than doing
> it as a plugin?

Limited time and steep learning curves.  Typically, researchers are 
interested in rapid-prototyping to keep the paper mill going.  Plug-ins 
offers a simple method for avoiding the latencies of repeated bootstrap 
cycles.

Several projects will survive the initial prototyping stages and become 
techniques we can apply in industrial settings.  We want to attract 
that.  Plus we want to attract the grad students that did the research 
and graduate with a favourable attitude towards using GCC in their 
future career.


Diego.

Re: Progress on GCC plugins ?

[Richard Kenner]

> Limited time and steep learning curves.  Typically, researchers are 
> interested in rapid-prototyping to keep the paper mill going.  Plug-ins 
> offers a simple method for avoiding the latencies of repeated bootstrap 
> cycles.

I don't follow.  If you're developing an optimizer, you need to do the
bootstrap to test the optimizer no matter how it connects to the rest
of the compiler.  All you save is that you do a smaller link, but that
time is measured in seconds on modern machines.

Re: Progress on GCC plugins ?

[Diego Novillo]

Richard Kenner wrote:

> I don't follow.  If you're developing an optimizer, you need to do the
> bootstrap to test the optimizer no matter how it connects to the rest
> of the compiler.  All you save is that you do a smaller link, but that
> time is measured in seconds on modern machines.

No, you don't.  All you need is an existing GCC binary installed 
somewhere in your path that accepts -fplugin=mypass.so.  All you compile 
is your pass, you don't need to even build GCC.

Plug-ins also facilitate other kinds of work that are not related to 
optimization passes.  Static analysis, refactoring tools, visualization, 
code navigation, etc.  Sean described a variety of different 
applications they had implemented with their plug-in framework at the 
last summit.  It was all pretty impressive.

I agree with your point of not getting into legal discussions here, so I 
won't comment on your previous posting.


Diego.

Re: Progress on GCC plugins ?

[Richard Kenner]

> > I don't follow.  If you're developing an optimizer, you need to do the
> > bootstrap to test the optimizer no matter how it connects to the rest
> > of the compiler.  All you save is that you do a smaller link, but that
> > time is measured in seconds on modern machines.
> 
> No, you don't.  All you need is an existing GCC binary installed 
> somewhere in your path that accepts -fplugin=mypass.so.  All you compile 
> is your pass, you don't need to even build GCC.

No, I mean for *testing* you need to do a bootstrap.  I'm not talking
about the minimum actually needed to build.

Re: Progress on GCC plugins ?

[Diego Novillo]

Richard Kenner wrote:

> No, I mean for *testing* you need to do a bootstrap.  I'm not talking
> about the minimum actually needed to build.

Nope, you don't.  If you are doing static analysis, for instance, you 
don't care nor need to bootstrap GCC.  You just need to load your module 
every time a file is compiled.

If you are doing a pass that you are testing and/or prototyping, you 
don't want to waste time rebuilding the whole compiler.  If/when your 
pass reaches certain maturity, you think it's ready for production, and 
people think it's a good idea to have it in the compiler, then you 
convert it into a static pass and you go through the traditional 
bootstrap process.

Similarly, for visualization plug-ins, you don't want/need to bootstrap 
the compiler.  That's the core idea with plug-ins, they allow more 
flexibility for experimentation.  They are not a means for implementing 
permanent features in the compiler.  We would not guarantee ABI nor API 
stability, there is just no point to doing that.


Diego.

Re: Progress on GCC plugins ?

[Alexandre Oliva]

On Nov 15, 2007, Diego Novillo <dnovillo@google.com> wrote:

> If/when your pass reaches certain maturity, you think it's ready for
> production, and people think it's a good idea to have it in the
> compiler, then you convert it into a static pass and you go through
> the traditional bootstrap process.

> That's the core idea with plug-ins, they allow more flexibility for
> experimentation.  They are not a means for implementing permanent
> features in the compiler.

I find the two paragraphs above contradictory.  If they're not means
for implementing permanent features, then converting a pass
implemented as a plugin that's ready for production into a static pass
is what?

It raises an issue of how important it is that this plugin
architecture, if it is to exist, be similar to the internals of GCC,
such that the conversion is as simple as possible.

OTOH, the more GCC internals it relies on, the less useful it is to
abstract away GCC's internal complexities.

Looks like conflicting goals to me :-(

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: Progress on GCC plugins ?

[Diego Novillo]

Alexandre Oliva wrote:
> On Nov 15, 2007, Diego Novillo <dnovillo@google.com> wrote:
> 
>> If/when your pass reaches certain maturity, you think it's ready for
>> production, and people think it's a good idea to have it in the
>> compiler, then you convert it into a static pass and you go through
>> the traditional bootstrap process.
> 
>> That's the core idea with plug-ins, they allow more flexibility for
>> experimentation.  They are not a means for implementing permanent
>> features in the compiler.
> 
> I find the two paragraphs above contradictory.  If they're not means
> for implementing permanent features, then converting a pass
> implemented as a plugin that's ready for production into a static pass
> is what?

Converting a pass implemented as a plug-in into a static pass is the way 
to add permanent feature in the compiler.  In principle, I don't think 
we'll want to implement permanent features as plug-ins.

> It raises an issue of how important it is that this plugin
> architecture, if it is to exist, be similar to the internals of GCC,
> such that the conversion is as simple as possible.

There is no conversion.  Read Sean Callanan's paper in this year's GCC 
Summit proceedings.  Plug-in code uses the same internal code as 
statically linked passes.


Diego.

Re: Progress on GCC plugins ?

[Alexandre Oliva]

On Nov 24, 2007, Diego Novillo <dnovillo@google.com> wrote:

> Alexandre Oliva wrote:
>> On Nov 15, 2007, Diego Novillo <dnovillo@google.com> wrote:
>> 
>>> If/when your pass reaches certain maturity, you think it's ready for
>>> production, and people think it's a good idea to have it in the
>>> compiler, then you convert it into a static pass and you go through
>>> the traditional bootstrap process.
>> 
>>> That's the core idea with plug-ins, they allow more flexibility for
>>> experimentation.  They are not a means for implementing permanent
>>> features in the compiler.
>> 
>> I find the two paragraphs above contradictory.  If they're not means
>> for implementing permanent features, then converting a pass
>> implemented as a plugin that's ready for production into a static pass
>> is what?

> Converting a pass implemented as a plug-in into a static pass is the
> way to add permanent feature in the compiler.  In principle, I don't
> think we'll want to implement permanent features as plug-ins.

Agreed.

>> It raises an issue of how important it is that this plugin
>> architecture, if it is to exist, be similar to the internals of GCC,
>> such that the conversion is as simple as possible.

> There is no conversion.

Then I guess I don't see any value whatsoever in this particular
plugin architecture in overcoming the difficulties unexperienced GCC
developers face :-(

> Plug-in code uses the same internal code as statically linked
> passes.

Then they can be means for implementing permanent features in the
compiler, after all.  I guess we can agree on that ;-)

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:

> > Limited time and steep learning curves.  Typically, researchers are 
> > interested in rapid-prototyping to keep the paper mill going.  Plug-ins 
> > offers a simple method for avoiding the latencies of repeated bootstrap 
> > cycles.
> 
> I don't follow.  If you're developing an optimizer, you need to do the
> bootstrap to test the optimizer no matter how it connects to the rest
> of the compiler.  All you save is that you do a smaller link, but that
> time is measured in seconds on modern machines.

But users who are not gcc developers have trouble doing a bootstrap.
Our build process is complicated, and it is not getting simpler.

And the sorts of researchers that dnovillo is talking about don't need
to do a bootstrap routinely.  They are experimenting with new
optimizations or with static analysis.  They aren't generating actual
code.  A bootstrap is beside the point.

Ian

RE: Progress on GCC plugins ?

[Alexander Lamaison]

Diego Novillo wrote:
 
> Richard Kenner wrote:
> 
> > I don't see that.  Why is it that much harder to link in with GCC
> than doing
> > it as a plugin?
> 
> Limited time and steep learning curves.  Typically, researchers are
> interested in rapid-prototyping to keep the paper mill going.  Plug-ins
> offers a simple method for avoiding the latencies of repeated bootstrap
> cycles.
> 
> Several projects will survive the initial prototyping stages and become
> techniques we can apply in industrial settings.  We want to attract
> that.  Plus we want to attract the grad students that did the research
> and graduate with a favourable attitude towards using GCC in their
> future career.

As a research student who spent 6 months working on an improvement to GCC, I
agree with all of Diego's remarks.  Out of the 6 months, 4 were spent
learning the GCC internals and fighting the GCC build process, 1 was spent
writing up leaving 1 month of actual productive research.  While not all of
this would be solved by a plugin system (a lot was down to documentation) it
would have significantly increased the amount of time I had to make useful
contributions.

I fully understand that this can seems strange to people who know GCC like
the back of their hand, but to a newcomer it is a huge task just to write a
single useful line of code.  I'm sure many give up before ever reaching that
point.

Alex Lamaison
Imperial College London

Re: Progress on GCC plugins ?

[Martin Jambor]

Hi,

On Nov 16, 2007 12:16 PM, Alexander Lamaison <awl03@doc.ic.ac.uk> wrote:
> Diego Novillo wrote:
> > Several projects will survive the initial prototyping stages and become
> > techniques we can apply in industrial settings.  We want to attract
> > that.  Plus we want to attract the grad students that did the research
> > and graduate with a favourable attitude towards using GCC in their
> > future career.
>
> As a research student who spent 6 months working on an improvement to GCC, I
> agree with all of Diego's remarks.  Out of the 6 months, 4 were spent
> learning the GCC internals and fighting the GCC build process, 1 was spent
> writing up leaving 1 month of actual productive research.  While not all of
> this would be solved by a plugin system (a lot was down to documentation) it
> would have significantly increased the amount of time I had to make useful
> contributions.

I have started  looking into GCC slightly more than  a year ago, since
then   I  have   successfully  finished   thesis   on  interprocedural
optimizations  which  was  largely  a  research project.  I  am  still
essentially a newcomer, yet I completely disagree.

When I  think what  a plugin  framework would help  me with,  I cannot
think  of  anything significant.  It  would  have  saved me  modifying
passes.c which was  not really an issue.  Everything  else would be as
complicated as it was or even more.

So as far as attracting new programmers, researchers and inexperienced
students  in  particular  is  concerned,  I  think  that  effort  that
implementing plugins would take would  be much better spent on keeping
documentation up to date,  possibly improving it (hey, Alexander, what
were your  problems, someone  might answer them  on Wiki  for others!)
and, in particular, staying as friendly and forgiving community as you
are (especially on IRC anyway :-).

IMHO 4 months  of learning how to work with GCC  internals seems to be
completely reasonable time for me. Compilers are complex and GCC is no
toy. (And plugins won't help with this, will they?)

Of  course,  I  understand  there  might be  other  and  perhaps  more
important uses of plugins.

Martin

Re: Progress on GCC plugins ?

[Alexander Lamaison]

Quoting Martin Jambor <jamborm@matfyz.cz>:

> So as far as attracting new programmers, researchers and inexperienced
> students  in  particular  is  concerned,  I  think  that  effort  that
> implementing plugins would take would  be much better spent on keeping
> documentation up to date,  possibly improving it (hey, Alexander, what
> were your  problems, someone  might answer them  on Wiki  for others!)
> and, in particular, staying as friendly and forgiving community as you
> are (especially on IRC anyway :-).

I certainly agree with this! The same effort spent on documentation  
rather than on a plugin system would, imho, give greater value. But,  
as there seems to be a willingness to work on plugins in contrast to  
the willingness to document, I am supporting progress wherever it is  
on offer.

The project is over and eventually I solved the problems through sweat  
and tears.  I have the solutions scribbled down somewhere and  
eventually I mean to put the up on the web somewhere to save others  
the same pain.

Re: Progress on GCC plugins ?

[Benjamin Smedberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Kenner wrote:

>> I think it's quite important for gcc's long-term health to permit and
>> even encourage academic researchers and students to use it.  And I see
>> plugins as directly supporting that goal.  
> 
> I don't see that.  Why is it that much harder to link in with GCC than doing
> it as a plugin?

To provide an example: Mozilla has been using elsa/oink to do static
analysis and rewriting of the codebase. We would love to use a more mature
and correct frontend such as GCC... and we would like to eventually have
this static analysis be a standard part of the build process when using the
GCC compiler.

To avoid requiring everyone who does Mozilla hacking to also do a
custom-built GCC would be to write just the static analysis as a plugin,
compile that to a .so, and then build with an extra flag such as g++
- -static-analyze=/custom/libmozstaticanalysis.so... in many cases we could
even pre-compile this file for common versions of GCC.

- --BDS

- --

Benjamin Smedberg
Platform Guru
Mozilla Corporation
benjamin@smedbergs.us
http://benjamin.smedbergs.us/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHPMKISSwGp5sTYNkRAtX2AKDa2OWDLgQkeXLQjzcI5BzqGf3b2ACgmm1r
jnbvtmAnq0GPPb19M/92lFo=
=af7b
-----END PGP SIGNATURE-----

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:

> > In fact, it's easy.  You have to write some code to translate from
> > tree to your proprietary IR, and then you have to plug that code
> > into passes.c.
> 
> Well first of all, that code becomes GPL so the IR isn't truely "proprietary".

I'm with you on that.  The fact remains, people have done this
already.  Whether the result is legally proprietary or not is almost
irrelevant; as you said earlier, the FSF is unlikely to actually sue.


> > So this seems to me to be a very weak argument against plugins.
> > Adding plugins does not make it noticeably easier to integrate gcc's
> > frontend with a proprietary compiler.  And adding plugins would not
> > change the issue of whether such a combination violated the GPL.
> > 
> > Do you disagree with this assessment?
> 
> No, not in that case, but I don't see that as the only case.  Another
> case would be somebody who wanted to keep an optimizer proprietary by
> making it a plug-in.  My view is that because of the linkage with the
> GCC IR, it can't be proprietary in that case, but that's the harder argument
> to make legally.

I can not deny the possibility of somebody writing an (effectively)
proprietary optimization pass.  However, I consider that to be both
unlikely and irrelevant.  People can always make private changes to
gcc.  We care about proprietary changes if they start distributing
them.  But that is an unlikely scenario.  There is no money to be made
in compiler optimization.

If we make it clear that we consider any plugin to be covered by the
GPL, then any attempt to sell a proprietary optimization will face
community opposition, thus making even less business sense.

And in the end, what will they have?  A marginal improvement over the
standard compiler.  Since they can not hide the results, we can figure
out what they are doing and recreate it.  In fact, it will be easier
for us to do so than it would be for a private port.

So I think this is a groundless fear.  When considering hypothetical
consequences, we need to consider both their likelihood and their
benefit or cost.  This one is both unlikely and has low cost.


> > I think it's quite important for gcc's long-term health to permit and
> > even encourage academic researchers and students to use it.  And I see
> > plugins as directly supporting that goal.  
> 
> I don't see that.  Why is it that much harder to link in with GCC than doing
> it as a plugin?

It is harder for people who are not used to the myriad details of
building gcc themselves.  Most people these days do not build gcc
themselves, and on gcc-help you will see the plaintive cries of those
who do.  We're talking about grad students focused on their research
here, not people who want to learn the arcana of our build system.

Ian

Re: Progress on GCC plugins ?

[Andrew Haley]

Ian Lance Taylor writes:
 > kenner@vlsi1.ultra.nyu.edu (Richard Kenner) writes:
 > 
 > > > I don't believe this is a strong argument.  My contention is,
 > > > and has always been, that GCC is _already_ trivial to integrate
 > > > into a proprietary compiler.  There is at least one compiler I
 > > > know that does this.
 > > 

 > > I believe that any such compiler would violate the GPL.  But I
 > > also believe it's not in the best interest of the FSF to litigate
 > > that matter if the linkage between the compiler is anything other
 > > than linked in a single executable.  Therefore, I think it's
 > > important for us to make it as technically hard as possible for
 > > people to do such a linkage by readin and writing tree or
 > > communicating as different libraries or DLLs.  I'm very much
 > > against any sort of "plug in" precisely for this reason.
 > 
 > We can make it as technically hard as possible, but it's way too late
 > to make it technically hard.  In fact, it's easy.  You have to write
 > some code to translate from tree to your proprietary IR, and then you
 > have to plug that code into passes.c.

Sure, but you then have to maintain your port forever, and there is a
substantial cost to this.  I am pretty sure that if there were a
stable API to get trees out of GCC, people would have bolted gcc into
proprietary compilers.  As there isn't a stable way to do it, it's
easier not to do it that way, and instead to contribute to gcc.

 > If gcc supports plugins, then all we've eliminated is the need to
 > plug that code into passes.c.  But that is the easiest part of the
 > job.  Adding plugins is not going to require us to support a stable
 > tree interface or anything along those lines; if it did, I would
 > oppose that.

Ahhhhhh.  I don't know about that: once we have a plugin
infrastructure, we have to document it and there will be pressure to
stabilize it.  I don't believe that an unstable plugin architecture
has any viability at all.

 > So this seems to me to be a very weak argument against plugins.
 > Adding plugins does not make it noticeably easier to integrate gcc's
 > frontend with a proprietary compiler.  And adding plugins would not
 > change the issue of whether such a combination violated the GPL.
 > 
 > Do you disagree with this assessment?

I think there is a real possibility that, had we had such a plugin
interface years ago, some of the gcc back-ends and optimization work
we have would never have been paid for by some companies, and so gcc
would be a worse compiler.

Andrew.

-- 
Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, UK
Registered in England and Wales No. 3798903

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Andrew Haley <aph@redhat.com> writes:

>  > We can make it as technically hard as possible, but it's way too late
>  > to make it technically hard.  In fact, it's easy.  You have to write
>  > some code to translate from tree to your proprietary IR, and then you
>  > have to plug that code into passes.c.
> 
> Sure, but you then have to maintain your port forever, and there is a
> substantial cost to this.  I am pretty sure that if there were a
> stable API to get trees out of GCC, people would have bolted gcc into
> proprietary compilers.  As there isn't a stable way to do it, it's
> easier not to do it that way, and instead to contribute to gcc.

I agree that there is a cost to maintaining your port.  I disagree
that plugins make it any cheaper.  See below.


>  > If gcc supports plugins, then all we've eliminated is the need to
>  > plug that code into passes.c.  But that is the easiest part of the
>  > job.  Adding plugins is not going to require us to support a stable
>  > tree interface or anything along those lines; if it did, I would
>  > oppose that.
> 
> Ahhhhhh.  I don't know about that: once we have a plugin
> infrastructure, we have to document it and there will be pressure to
> stabilize it.  I don't believe that an unstable plugin architecture
> has any viability at all.

I disagree.  In fact, if creating a plugin architecture comes with a
requirement to make a stable structure for trees, then I'm opposed to
it.  That would hurt us far more than it would help.  This is not a
slippery slope.

An unstable plugin architecture is still very useful for our users.
Correct installation of a patched gcc is an awkward affair that many
people get wrong.  Correct installation of a plugin requires no more
than a command line option.  Plugins make it easy for people to share
their gcc extensions across projects or across university departments.


>  > So this seems to me to be a very weak argument against plugins.
>  > Adding plugins does not make it noticeably easier to integrate gcc's
>  > frontend with a proprietary compiler.  And adding plugins would not
>  > change the issue of whether such a combination violated the GPL.
>  > 
>  > Do you disagree with this assessment?
> 
> I think there is a real possibility that, had we had such a plugin
> interface years ago, some of the gcc back-ends and optimization work
> we have would never have been paid for by some companies, and so gcc
> would be a worse compiler.

Most new gcc back-ends are private, so I don't buy that part of the
argument.  And in any case nobody is talking about plug-ins for gcc
backends.  We're talking about plugins at the tree/GIMPLE level.

And, frankly, very few people are paying for general new gcc
optimizations.  As far as I know, the only people doing so are
companies like IBM and Red Hat, and they would contribute their
changes anyhow.  Do you have any examples in mind?

When I was in the business of convincing people to pay for gcc work, I
had a laundry list of general gcc improvements to sell.  I was never
able to get a dime except for target specific improvements.  A plugin
architecture would not make any difference to that kind of work.

Ian

Re: Progress on GCC plugins ?

[Andrew Haley]

Ian Lance Taylor writes:
 > Andrew Haley <aph@redhat.com> writes:
 > 
 > >  > If gcc supports plugins, then all we've eliminated is the need to
 > >  > plug that code into passes.c.  But that is the easiest part of the
 > >  > job.  Adding plugins is not going to require us to support a stable
 > >  > tree interface or anything along those lines; if it did, I would
 > >  > oppose that.
 > > 
 > > Ahhhhhh.  I don't know about that: once we have a plugin
 > > infrastructure, we have to document it and there will be pressure to
 > > stabilize it.  I don't believe that an unstable plugin architecture
 > > has any viability at all.
 > 
 > I disagree.  In fact, if creating a plugin architecture comes with a
 > requirement to make a stable structure for trees, then I'm opposed to
 > it.  That would hurt us far more than it would help.  This is not a
 > slippery slope.
 > 
 > An unstable plugin architecture is still very useful for our users.
 > Correct installation of a patched gcc is an awkward affair that many
 > people get wrong.  Correct installation of a plugin requires no more
 > than a command line option.  Plugins make it easy for people to share
 > their gcc extensions across projects or across university departments.

Even if the interafce changes continuously?  Maybe.  I find it hard to
believe, but we'll just have to agree to differ.

 > >  > So this seems to me to be a very weak argument against plugins.
 > >  > Adding plugins does not make it noticeably easier to integrate gcc's
 > >  > frontend with a proprietary compiler.  And adding plugins would not
 > >  > change the issue of whether such a combination violated the GPL.
 > >  > 
 > >  > Do you disagree with this assessment?
 > > 
 > > I think there is a real possibility that, had we had such a plugin
 > > interface years ago, some of the gcc back-ends and optimization work
 > > we have would never have been paid for by some companies, and so gcc
 > > would be a worse compiler.
 > 
 > Most new gcc back-ends are private, so I don't buy that part of the
 > argument.  And in any case nobody is talking about plug-ins for gcc
 > backends.  We're talking about plugins at the tree/GIMPLE level.

Yeah, I know.  I'm thinking about proprietary compilers (not just 
back-ends, optimization passes) bolted on to a gcc front-end to get
Linux compatibility.

 > And, frankly, very few people are paying for general new gcc
 > optimizations.  As far as I know, the only people doing so are
 > companies like IBM and Red Hat, and they would contribute their
 > changes anyhow.  Do you have any examples in mind?

ISTR that at least part of if-conversion was paid for, but I can't
remember how much of what I know is confidential and how much is just
plain wrong.

 > When I was in the business of convincing people to pay for gcc
 > work, I had a laundry list of general gcc improvements to sell.  I
 > was never able to get a dime except for target specific
 > improvements.  A plugin architecture would not make any difference
 > to that kind of work.

No, but it might mean that entire gcc ports go away, as people who
already have in-house compilers use them with a gcc front-end for
Linux ports, rather than funding gcc ports.

Andrew.

-- 
Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, UK
Registered in England and Wales No. 3798903

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Andrew Haley <aph@redhat.com> writes:

>  > Most new gcc back-ends are private, so I don't buy that part of the
>  > argument.  And in any case nobody is talking about plug-ins for gcc
>  > backends.  We're talking about plugins at the tree/GIMPLE level.
> 
> Yeah, I know.  I'm thinking about proprietary compilers (not just 
> back-ends, optimization passes) bolted on to a gcc front-end to get
> Linux compatibility.

As we've discussed previously, we are already seeing that without
plugins: GCCfss.  Sun took gcc's frontend and attached it to their
proprietary backend.  So in my view introducing plugins will not make
a substantive difference here.


>  > When I was in the business of convincing people to pay for gcc
>  > work, I had a laundry list of general gcc improvements to sell.  I
>  > was never able to get a dime except for target specific
>  > improvements.  A plugin architecture would not make any difference
>  > to that kind of work.
> 
> No, but it might mean that entire gcc ports go away, as people who
> already have in-house compilers use them with a gcc front-end for
> Linux ports, rather than funding gcc ports.

But as you know, most gcc ports are never contributed anyhow.  Ports
that people hire Red Hat to do are contributed, but I can easily count
six gcc ports I've seen myself that were never contributed.  So again
I don't see a substantive difference here.

Ian

Re: Progress on GCC plugins ?

[Andrew Haley]

Ian Lance Taylor writes:
 > Andrew Haley <aph@redhat.com> writes:
 > 
 > >  > Most new gcc back-ends are private, so I don't buy that part of the
 > >  > argument.  And in any case nobody is talking about plug-ins for gcc
 > >  > backends.  We're talking about plugins at the tree/GIMPLE level.
 > > 
 > > Yeah, I know.  I'm thinking about proprietary compilers (not just 
 > > back-ends, optimization passes) bolted on to a gcc front-end to get
 > > Linux compatibility.
 > 
 > As we've discussed previously, we are already seeing that without
 > plugins: GCCfss.  Sun took gcc's frontend and attached it to their
 > proprietary backend.  So in my view introducing plugins will not make
 > a substantive difference here.

Well, yeah, but no-one ever said it wouldn't be possible without
plugins.

 > >  > When I was in the business of convincing people to pay for gcc
 > >  > work, I had a laundry list of general gcc improvements to sell.  I
 > >  > was never able to get a dime except for target specific
 > >  > improvements.  A plugin architecture would not make any difference
 > >  > to that kind of work.
 > > 
 > > No, but it might mean that entire gcc ports go away, as people who
 > > already have in-house compilers use them with a gcc front-end for
 > > Linux ports, rather than funding gcc ports.
 > 
 > But as you know, most gcc ports are never contributed anyhow.

Sure, but they are still free software: if the compiler gets
distributed, so does its source code.  Of couse, assigning copyright
to FSF is nice, but freedom is much more important.

 > Ports that people hire Red Hat to do are contributed, but I can
 > easily count six gcc ports I've seen myself that were never
 > contributed. 

 > So again I don't see a substantive difference here.

I guess it depends on what you mean by "substantive".  As I said, I
suspect that if it were easier to decouple the gcc front-end from the
back-end and to maintain the resulting compiler, there would be fewer
free compilers.  And no, neither of us can prove it without doing the
experiment.  I insist, however, that when it comes to a change that
potentially reduces freedom, the burden of proof -- or at least of
evidence -- is on those wanting to make the change.

Andrew.

-- 
Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, UK
Registered in England and Wales No. 3798903

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Andrew Haley wrote:
>  > 
>  > But as you know, most gcc ports are never contributed anyhow.
> 
> Sure, but they are still free software: if the compiler gets
> distributed, so does its source code.  Of couse, assigning copyright
> to FSF is nice, but freedom is much more important.

Oh I fully understand that now. So most GCC ports are GPL-ed but outside 
FSF source tree. Hence, a plugin machinery (including for backends) 
would make this easier!

I also expressed (orally) the concern (at previous GCC summit, Ottawa 
july 2007) that getting the copyright assignment signed is a lot of work 
in some organizations and I would hope that the FSF might consider 
making it easier (I don't know exactly how, as a non lawyer I would 
dream of simple GPLed contributions, with the copyright owner giving the 
FSF the right to sue on its behalf, ...) but apparently it is not a 
concern.

I still believe that GCC needs more developers, and attracting them 
could be a concern.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Andrew Haley <aph@redhat.com> writes:

> Ian Lance Taylor writes:
>  > Andrew Haley <aph@redhat.com> writes:
>  > 
>  > >  > Most new gcc back-ends are private, so I don't buy that part of the
>  > >  > argument.  And in any case nobody is talking about plug-ins for gcc
>  > >  > backends.  We're talking about plugins at the tree/GIMPLE level.
>  > > 
>  > > Yeah, I know.  I'm thinking about proprietary compilers (not just 
>  > > back-ends, optimization passes) bolted on to a gcc front-end to get
>  > > Linux compatibility.
>  > 
>  > As we've discussed previously, we are already seeing that without
>  > plugins: GCCfss.  Sun took gcc's frontend and attached it to their
>  > proprietary backend.  So in my view introducing plugins will not make
>  > a substantive difference here.
> 
> Well, yeah, but no-one ever said it wouldn't be possible without
> plugins.

I'm sorry, I've lost the sense of the argument here.  I thought you
were arguing that plugins would make this more likely.  I'm saying
that it's already happening, and that it's not noticeably easier with
plugins.  So can you repeat your point?


>  > >  > When I was in the business of convincing people to pay for gcc
>  > >  > work, I had a laundry list of general gcc improvements to sell.  I
>  > >  > was never able to get a dime except for target specific
>  > >  > improvements.  A plugin architecture would not make any difference
>  > >  > to that kind of work.
>  > > 
>  > > No, but it might mean that entire gcc ports go away, as people who
>  > > already have in-house compilers use them with a gcc front-end for
>  > > Linux ports, rather than funding gcc ports.
>  > 
>  > But as you know, most gcc ports are never contributed anyhow.
> 
> Sure, but they are still free software: if the compiler gets
> distributed, so does its source code.  Of couse, assigning copyright
> to FSF is nice, but freedom is much more important.

If I follow this, it seems that you are saying that if we have
plugins, some people will choose to use them to get a gcc frontend for
a proprietary compiler rather than doing a gcc port.  I'm sorry, I
don't buy this at all.  Again, people can already do this, and adding
plugins does not make it substantially easier.


>  > Ports that people hire Red Hat to do are contributed, but I can
>  > easily count six gcc ports I've seen myself that were never
>  > contributed. 
> 
>  > So again I don't see a substantive difference here.
> 
> I guess it depends on what you mean by "substantive".  As I said, I
> suspect that if it were easier to decouple the gcc front-end from the
> back-end and to maintain the resulting compiler, there would be fewer
> free compilers.  And no, neither of us can prove it without doing the
> experiment.  I insist, however, that when it comes to a change that
> potentially reduces freedom, the burden of proof -- or at least of
> evidence -- is on those wanting to make the change.

I'm very sorry to hear you take that position.  I think you are
letting an unlikely fear sacrifice a clear benefit.

I have a different fear: that gcc will become increasing irrelevant,
as more and more new programmers learn to work on alternative free
compilers instead.  That is neutral with regard to freedom, but it
will tend to lose the many years of experience which have been put
into gcc.  In my view, if we can't even get ourselves together to
permit something as simple as plugins with an unstable API, then we
deserve to lose.

Ian

Re: Progress on GCC plugins ?

[Diego Novillo]

Ian Lance Taylor wrote:

> I have a different fear: that gcc will become increasing irrelevant,
> as more and more new programmers learn to work on alternative free
> compilers instead.  That is neutral with regard to freedom, but it
> will tend to lose the many years of experience which have been put
> into gcc.  In my view, if we can't even get ourselves together to
> permit something as simple as plugins with an unstable API, then we
> deserve to lose.

That's precisely my concern.


Diego.

Re: Progress on GCC plugins ?

[Andrew Haley]

Ian Lance Taylor writes:
 > Andrew Haley <aph@redhat.com> writes:
 > 
 > > Ian Lance Taylor writes:
 > >  > Andrew Haley <aph@redhat.com> writes:
 > >  > 
 > >  > >  > Most new gcc back-ends are private, so I don't buy that part of the
 > >  > >  > argument.  And in any case nobody is talking about plug-ins for gcc
 > >  > >  > backends.  We're talking about plugins at the tree/GIMPLE level.
 > >  > > 
 > >  > > Yeah, I know.  I'm thinking about proprietary compilers (not just 
 > >  > > back-ends, optimization passes) bolted on to a gcc front-end to get
 > >  > > Linux compatibility.
 > >  > 
 > >  > As we've discussed previously, we are already seeing that without
 > >  > plugins: GCCfss.  Sun took gcc's frontend and attached it to their
 > >  > proprietary backend.  So in my view introducing plugins will not make
 > >  > a substantive difference here.
 > > 
 > > Well, yeah, but no-one ever said it wouldn't be possible without
 > > plugins.
 > 
 > I'm sorry, I've lost the sense of the argument here.  I thought you
 > were arguing that plugins would make this more likely.  I'm saying
 > that it's already happening, and that it's not noticeably easier
 > with plugins.  So can you repeat your point?

Ah, OK.  I think that it will be noticeably easier with plugins.  If
the plugin architecture really doesn't make it easier, my point falls.

 > >  > >  > When I was in the business of convincing people to pay for gcc
 > >  > >  > work, I had a laundry list of general gcc improvements to sell.  I
 > >  > >  > was never able to get a dime except for target specific
 > >  > >  > improvements.  A plugin architecture would not make any difference
 > >  > >  > to that kind of work.
 > >  > > 
 > >  > > No, but it might mean that entire gcc ports go away, as people who
 > >  > > already have in-house compilers use them with a gcc front-end for
 > >  > > Linux ports, rather than funding gcc ports.
 > >  > 
 > >  > But as you know, most gcc ports are never contributed anyhow.
 > > 
 > > Sure, but they are still free software: if the compiler gets
 > > distributed, so does its source code.  Of couse, assigning copyright
 > > to FSF is nice, but freedom is much more important.
 > 
 > If I follow this, it seems that you are saying that if we have
 > plugins, some people will choose to use them to get a gcc frontend
 > for a proprietary compiler rather than doing a gcc port.  I'm
 > sorry, I don't buy this at all.  Again, people can already do this,
 > and adding plugins does not make it substantially easier.

Well, that's where we differ.  I don't at all understand how adding
plugins won't make it very much easier.  It seems obvious to me that
if there is a reasonably well-defined plugin architecture it will be
vastly easier to export data from gcc's front-ends to a proprietary
compiler.  It is entirely beyond my understanding why this isn't also
obvious to you.

 > >  > Ports that people hire Red Hat to do are contributed, but I can
 > >  > easily count six gcc ports I've seen myself that were never
 > >  > contributed. 
 > > 
 > >  > So again I don't see a substantive difference here.
 > > 
 > > I guess it depends on what you mean by "substantive".  As I said,
 > > I suspect that if it were easier to decouple the gcc front-end
 > > from the back-end and to maintain the resulting compiler, there
 > > would be fewer free compilers.  And no, neither of us can prove
 > > it without doing the experiment.  I insist, however, that when it
 > > comes to a change that potentially reduces freedom, the burden of
 > > proof -- or at least of evidence -- is on those wanting to make
 > > the change.
 > 
 > I'm very sorry to hear you take that position.  I think you are
 > letting an unlikely fear sacrifice a clear benefit.
 > 
 > I have a different fear: that gcc will become increasing
 > irrelevant, as more and more new programmers learn to work on
 > alternative free compilers instead.  That is neutral with regard to
 > freedom, but it will tend to lose the many years of experience
 > which have been put into gcc.  In my view, if we can't even get
 > ourselves together to permit something as simple as plugins with an
 > unstable API, then we deserve to lose.

OK.  Well, that's your view.  I don't believe that the presence or
absence of plugins will make one iota of differebce to mainstream use
of gcc.

Andrew.

-- 
Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, UK
Registered in England and Wales No. 3798903

Re: Progress on GCC plugins ?

[Diego Novillo]

Andrew Haley wrote:

> Well, that's where we differ.  I don't at all understand how adding
> plugins won't make it very much easier.  It seems obvious to me that
> if there is a reasonably well-defined plugin architecture it will be
> vastly easier to export data from gcc's front-ends to a proprietary
> compiler.  It is entirely beyond my understanding why this isn't also
> obvious to you.

Because it is not at all easier.  It already is *trivial*.

Before plug-ins: put your gimple-to-myIR converter in passes.c
After plug-ins: dlopen gimple-to-myIR.so

Both represent the same effort.  Both require your converter to be kept 
up-to-date with GCC's ever shifting ABI/API.


Diego.

Re: Progress on GCC plugins ?

[Benjamin Smedberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Diego Novillo wrote:

> Before plug-ins: put your gimple-to-myIR converter in passes.c
> After plug-ins: dlopen gimple-to-myIR.so
> 
> Both represent the same effort.  Both require your converter to be kept
> up-to-date with GCC's ever shifting ABI/API.

They represent the same effort for somebody writing a plugin pass. They do
*not* represent the same effort for a Mozilla hacker who just wants to
compile Mozilla with the extra static-checking pass enabled. The fact that
they can use a stock GCC (and potentially a precompiled plugin specific to
their GCC version) is a huge advantage.

Note that we're talking about analysis passes that would never be
appropriate for integration with GCC directly: e.g. "statically enforce that
the internal typedef PRBool is only ever 0 or 1" or "statically enforce that
pointers to subclasses of MMgc::GCObject allocated on the heap are only
written through this particular writebarrier pattern"... so arguments about
whether we want the code to be integrated into GCC itself are irrelevant.

- --BDS

- --

Benjamin Smedberg
Platform Guru
Mozilla Corporation
benjamin@smedbergs.us
http://benjamin.smedbergs.us/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHPdQkSSwGp5sTYNkRAjbiAKCPnFBPN4wXswT35dSx3gpyZv+DWACg3L8X
0ntugTV0nMoJoMTXa1yX+FE=
=V32a
-----END PGP SIGNATURE-----

Re: Progress on GCC plugins ?

[Frank Ch. Eigler]

Benjamin Smedberg <benjamin@smedbergs.us> writes:

> Diego Novillo wrote:
>> Before plug-ins: put your gimple-to-myIR converter in passes.c
>> After plug-ins: dlopen gimple-to-myIR.so
>> 
>> Both represent the same effort.  Both require your converter to be kept
>> up-to-date with GCC's ever shifting ABI/API.
>
> They represent the same effort for somebody writing a plugin pass. They do
> *not* represent the same effort for a Mozilla hacker who just wants to
> compile Mozilla with the extra static-checking pass enabled. [...]

There may be a subtle connection between this and the debugging
information correctness thread by Alex Oliva a few weeks ago.  It may
be that if only gcc's debuginfo for deployed (-O2 -g) code were more
complete & more correct, that some such analysis code could work by
on-the-fly probing-based instrumentation rather than compiled-in
stuff.  At least, this applies to the kinds of assertions expressible
by something like the old GNU Nana tool.

- FChE

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Andrew Haley wrote:
> 
> Well, that's where we differ.  I don't at all understand how adding
> plugins won't make it very much easier.  It seems obvious to me that
> if there is a reasonably well-defined plugin architecture it will be
> vastly easier to export data from gcc's front-ends to a proprietary
> compiler.  It is entirely beyond my understanding why this isn't also
> obvious to you.


I beg to disagree. Exporting GCC data outside makes practical sense if 
the data is somehow stable (e.g. from one version of GCC to the next). 
But a plugin mechanism does not require any stability of any sort. And 
the current internal representations are not (in my view) stable (and 
this is good, I am not criticizing GCC here) in the same sense that the 
accepted languages & GCC options are stable.

For example, the "simple" plugin mechanism many people have implicitly 
in mind is just: something give you the ability to call a dlsymed 
function inside a dlopened plugin as a pass in a defined (& fixed) 
position in passes.c. I tend to believe it is not that difficult to 
implement (it seems to me that the issue is to get it accepted in the 
trunk). However, this does not guaranty at all that all the internal 
representation (e.g. the tree.h and other header files) is stable.

In other words, such a plugin yourplugin.so (which you coded for future 
gcc-4.4.3) might be usable from gcc 4.4.3 but not from gcc 4.4.5, and 
probably not from gcc-4.5 and certainly not from gcc-5.x

I'm probably naive, but I think that there are enough incentives (both 
technical compatibility as above, also legal requirements per GPL, and 
community & ethical standards of free software at large) that the author 
of the plugin has interest to publish his source under GPL together with 
the yourplugin.so file.

So a well defined plugin architecture does not mean any stability of 
internal representations (in their binary detail) or of the many GIMPLE 
transformations....

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Alexandre Oliva]

On Nov 16, 2007, Basile STARYNKEVITCH <basile@starynkevitch.net> wrote:

> For example, the "simple" plugin mechanism many people have implicitly
> in mind is just: something give you the ability to call a dlsymed
> function inside a dlopened plugin as a pass in a defined (& fixed)
> position in passes.c. I tend to believe it is not that difficult to
> implement (it seems to me that the issue is to get it accepted in the
> trunk). However, this does not guaranty at all that all the internal
> representation (e.g. the tree.h and other header files) is stable.

Exactly.  And I guess some of this could even be accomplished with
LD_PRELOAD.  But how useful would that be, really?  How far would this
go into addressing the real needs that are behind the requests for
plugin support?

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: Progress on GCC plugins ?

[Chris Lattner]

On Nov 24, 2007, at 1:54 PM, Alexandre Oliva wrote:

> On Nov 16, 2007, Basile STARYNKEVITCH <basile@starynkevitch.net>  
> wrote:
>
>> For example, the "simple" plugin mechanism many people have  
>> implicitly
>> in mind is just: something give you the ability to call a dlsymed
>> function inside a dlopened plugin as a pass in a defined (& fixed)
>> position in passes.c. I tend to believe it is not that difficult to
>> implement (it seems to me that the issue is to get it accepted in the
>> trunk). However, this does not guaranty at all that all the internal
>> representation (e.g. the tree.h and other header files) is stable.
>
> Exactly.  And I guess some of this could even be accomplished with
> LD_PRELOAD.  But how useful would that be, really?  How far would this
> go into addressing the real needs that are behind the requests for
> plugin support?

Full "plugin support" really encompasses more than having GCC just  
being able to dlopen a .so file.  It means providing (hopefully) well  
defined extension points for common operations.  For example, it  
should be relatively easy to load a plugin that defines a new tree-ssa  
pass, and have it get dropped naturally into the pass manager.   
Alternatively, you could have an extension point for "gimple  
consumers" that don't care about the optimizer or backend, but just  
want a way to parse the code and deal with the gimple or generic  
trees.  Examples of this would be static analysis tools or things like  
the "FFI generator" for scripting languages.

To me personally, I'm looking forward to this happening because it  
means that "llvm-gcc" could conceptually just become a plugin into a  
standard GCC: the llvm-gcc plugin would use exactly the same extension  
point as the "static analysis tools" (it doesn't care about the GCC  
optimizer or backend, so it just wouldn't use it).  This would make it  
potentially easier to deploy the llvm-gcc binary (it's a plugin for a  
specific gcc, instead of an entire copy of gcc itself), and it would  
potentially make it easier to keep the tree up to date as GCC moves  
forward.  For the record, I'm fine everything being GPL, the license  
is not the issue.

-Chris

Re: Progress on GCC plugins ?

[David Edelsohn]

>>>>> Andrew Haley writes:

>> I have a different fear: that gcc will become increasing
>> irrelevant, as more and more new programmers learn to work on
>> alternative free compilers instead.  That is neutral with regard to
>> freedom, but it will tend to lose the many years of experience
>> which have been put into gcc.  In my view, if we can't even get
>> ourselves together to permit something as simple as plugins with an
>> unstable API, then we deserve to lose.

Andrew> OK.  Well, that's your view.  I don't believe that the presence or
Andrew> absence of plugins will make one iota of differebce to mainstream use
Andrew> of gcc.

	The concern is not a first-order effect, but a second-order
effect.  GCC will improve more and faster if more developers are involved.
Plug-ins will encourage more research and development of GCC -- more
features and benefits.  An improved GCC will attract more users.

	In my experience, most users prefer GCC because it is free,
generates code with "good-enough" performance, supports many architectures
and languages, defines a uniform C language, and is distributed with an
"open source-compatible" license.  I do not believe that the GPL or the
Free Software Foundation's goals are near the top of the reasons for most
users.  If developers and users find that another free compiler satisfies
those requirements better, I suspect developers and users would start
migrating away.

	As Ian said, that ultimately would not hurt software freedom; it
might hurt Free Software, and it definitely would hurt the GNU Project and
the Free Software Foundation.

David

RE: Progress on GCC plugins ?

[Dep, Khushil (GE Money)]

 

-----Original Message-----
From: gcc-owner@gcc.gnu.org [mailto:gcc-owner@gcc.gnu.org] On Behalf Of
David Edelsohn
Sent: 16 November 2007 16:58
To: Andrew Haley
Cc: Ian Lance Taylor; Richard Kenner; dnovillo@google.com;
Joe.Buck@synopsys.com; fleury@labri.fr; gcc@gcc.gnu.org
Subject: Re: Progress on GCC plugins ?

--snip--

->	The concern is not a first-order effect, but a second-order
effect.  GCC will improve more and faster if more developers are
involved.
->Plug-ins will encourage more research and development of GCC -- more
features and benefits.  An improved GCC will attract more users.

--snip--

I'm not sure that a plugin system will encourage more research and
development. Anyone who even contemplates getting into the this field
isn't going to be someone who is easily disuaded by challenges and
obstacles.  Anyone new to a system looks for clear concise documentation
- something which the opensource world can lack!  I believe efforts to
clarify and expand documentation is much more likely to entice new
researchers and developers rather than a plugin system which no doubt
would be poorly documented!

My 2c - have a good weekend all!


-Khush.

Re: Progress on GCC plugins ?

[Tom Tromey]

>>>>> ">" == Dep, Khushil (GE Money) <Khushil.Dep@ge.com> writes:

>> I believe efforts to clarify and expand documentation is much more
>> likely to entice new researchers and developers rather than a
>> plugin system which no doubt would be poorly documented!

This idea comes up a lot.

I'm sympathetic to it -- it would have been really useful to me, a
couple years ago, if GCC had had better documentation.

However, the two things are not tradeable.  It isn't as if we are
deciding what to do with limited manpower.  GCC as a project basically
cannot do that.

Rather, we're deciding what changes to accept.  There are existing
patches to add plugins, and, presumably, the interest and manpower to
polish and maintain them.

Sadly, there aren't ready patches to radically upgrade the
documentation.

It doesn't make sense to reject plugin patches on the basis that
documentation would be preferable.  I find it unlikely that this will
cause more documentation to be written.  Instead, I think the likely
effect is that GCC will lose some potential developers.

Tom

Re: Progress on GCC plugins ?

[Gabriel Dos Reis]

"Dep, Khushil (GE Money)" <Khushil.Dep@ge.com> writes:

| I'm not sure that a plugin system will encourage more research and
| development. Anyone who even contemplates getting into the this field
| isn't going to be someone who is easily disuaded by challenges and
| obstacles.

I beg to disagree -- speaking from experience.  The issue isn't
`obstacle'; from my perspective, it is that of time and energy sunk
into `mere re-invention of pointless wheels with no research result'.  I
had no trouble getting students up to spead on well-structured
plugable proprietary compiler within weeks.  However, the same
experience on GCC has proven much less impressive results.
Consequently, we have been making more progress working with
proprietary compiler than with GCC -- something I find myself
embarrasing :-( 

-- Gaby

Re: Progress on GCC plugins ?

[Alexandre Oliva]

On Nov 17, 2007, Gabriel Dos Reis <gdr@cs.tamu.edu> wrote:

> I had no trouble getting students up to spead on well-structured
> plugable proprietary compiler within weeks.  However, the same
> experience on GCC has proven much less impressive results.
> Consequently, we have been making more progress working with
> proprietary compiler than with GCC -- something I find myself
> embarrasing :-(

That's sad, but adding to GCC the ability to support plugins (I assume
this is about dynamically-loaded GCC passes) won't change this at all.

The thing that people have historically found difficult in getting
started with GCC is that there was a lot of infrastructure that didn't
follow the patterns and algorithms found in compiler literature.  This
has changed a bit with the Tree-SSA infrastructure, not only because
SSA and GIMPLE-like representations are commonplace, but also because
there aren't as many inter-dependencies and implicit assumptions
between passes as there used to be with RTL.

But it's still true that one has to learn a lot about GCC internals
before one can write a new RTL pass, and even a new Tree-SSA pass.
Merely introducing means for people to dynamically load passes won't
change this.  Only designing a plugin API that hides a lot of GCC's
internal complexity, while still making room for useful experiments,
would make this plugin API useful to address this problem.

But then, this is much harder than just adding plugin support, and I'm
familiar with any actual proposals that would address the underlying
problem.

And then, once the underlying problem is addressed and we have an API
that is usable by regular users, maybe we will find out that we don't
need plugins, after all.  That, as Diego and Ian say, adding plugin
support is trivial, but I'm pretty sure that's not where the actual
problem lies.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: Progress on GCC plugins ?

[Richard Kenner]

> And then, once the underlying problem is addressed and we have an API
> that is usable by regular users, maybe we will find out that we don't
> need plugins, after all.  That, as Diego and Ian say, adding plugin
> support is trivial, but I'm pretty sure that's not where the actual
> problem lies.

That's my position as well.  AND that plugin (here meaning dynamic
link) support creates tricky legal issues that the RMS has not been
willing to deal with in the past.

Re: Progress on GCC plugins ?

[Tom Tromey]

>>>>> "Alexandre" == Alexandre Oliva <aoliva@redhat.com> writes:

Alexandre> And then, once the underlying problem is addressed and we
Alexandre> have an API that is usable by regular users, maybe we will
Alexandre> find out that we don't need plugins, after all.

Plugins are about deployment, not development.

Plugins make it possible to redistribute useful things which are not
in GCC.  They don't -- and as you rightly point out, can't -- make it
simpler to actually develop these things.

The canonical example, which has been covered many times, is a pass
that does extra checking for a large program (e.g., Mozilla).


LD_PRELOAD would work just as well as having gcc directly support
plugins, provided that certain internal things are never made
file-local.  Someone could write a helper library to make it
relatively simple to hook in.  But... I looked at this recently, and
since gcc is not linked with -rdynamic, it is a non-starter.

Tom

Re: Progress on GCC plugins ?

[Taras Glek]

Tom Tromey wrote:
>>>>>> "Alexandre" == Alexandre Oliva <aoliva@redhat.com> writes:
>>>>>>             
>
> Alexandre> And then, once the underlying problem is addressed and we
> Alexandre> have an API that is usable by regular users, maybe we will
> Alexandre> find out that we don't need plugins, after all.
>
> Plugins are about deployment, not development.
>
> Plugins make it possible to redistribute useful things which are not
> in GCC.  They don't -- and as you rightly point out, can't -- make it
> simpler to actually develop these things.
>   
> The canonical example, which has been covered many times, is a pass
> that does extra checking for a large program (e.g., Mozilla).
>   
There are a lot of checks that could be implemented as plugins to 
benefit Mozilla and other projects. For future Mozilla development 
certain features that we are looking at are not feasible in C++ until 
the compiler can help with enforcing correct API usage. It would be 
extremely awesome to be able to utilize GCC internals for static 
analysis and source refactoring. Currently that isn't realistic as these 
features do not belong in the GCC mainline and distributing a gcc fork 
would be very burdensome.
Plugins would also encourage projects such as Mozilla to contribute to 
gcc to implement various backend improvements to make various plugins 
possible. I think GCC could gain some "accidental" contributors this way.
i believe that this would also have an additional effect of making GCC 
the strongly suggested compiler for development as it would be able to 
provide development benefits not (yet?) available in most compilers.
>
> LD_PRELOAD would work just as well as having gcc directly support
> plugins, provided that certain internal things are never made
> file-local.  Someone could write a helper library to make it
> relatively simple to hook in.  But... I looked at this recently, and
> since gcc is not linked with -rdynamic, it is a non-starter.
>   
Tom, I don't know much about linkers and LD_PRELOAD. Would making 
LD_PRELOAD work be easier than making an unstable plugin API?

Taras

Re: Progress on GCC plugins ?

[Tom Tromey]

>>>>> "Taras" == Taras Glek <taras.judge@shaw.ca> writes:

Tom> LD_PRELOAD would work just as well as having gcc directly support
Tom> plugins, provided that certain internal things are never made
Tom> file-local.  Someone could write a helper library to make it
Tom> relatively simple to hook in.  But... I looked at this recently, and
Tom> since gcc is not linked with -rdynamic, it is a non-starter.

Taras> Tom, I don't know much about linkers and LD_PRELOAD. Would making
Taras> LD_PRELOAD work be easier than making an unstable plugin API?

Not really.

The difference would be that with LD_PRELOAD the gcc change would be
very small -- just linking with -rdynamic.  Maybe you could lobby a
Linux distro for this :-)

But even if the gcc change is small, you'd still want to write and
maintain about the same amount of code to let plugins interface with
the pass manager.

Tom

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Tom Tromey wrote:
>>>>>> "Taras" == Taras Glek <taras.judge@shaw.ca> writes:
> 
> Tom> LD_PRELOAD would work just as well as having gcc directly support
> Tom> plugins, provided that certain internal things are never made
> Tom> file-local.  Someone could write a helper library to make it
> Tom> relatively simple to hook in.  But... I looked at this recently, and
> Tom> since gcc is not linked with -rdynamic, it is a non-starter.
> 
> Taras> Tom, I don't know much about linkers and LD_PRELOAD. Would making
> Taras> LD_PRELOAD work be easier than making an unstable plugin API?
> 
> Not really.
> 
> The difference would be that with LD_PRELOAD the gcc change would be
> very small -- just linking with -rdynamic.  Maybe you could lobby a
> Linux distro for this :-)


I'm not fully convinced that just LD_PRELOAD is enough to add poor man's 
plugin into GCC.

Plugins into GCC are expected to add optimisation passes (see file 
gcc/passes.c function init_optimization_passes and I don't understand 
what exactly LD_PRELOAD trick (unless you also redefine this very 
function init_optimization_passes in your ld_preload-ed plugin) would 
enable this.

So Tom or Taras, could you please elaborate on this? What tricks are you 
thinking of?

However, you are right in the sense that implementing naive plugins is 
technically easy; apparently the issue is political, not technical (i.e. 
let RMS accept or bless it).

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Tom Tromey]

>>>>> "Basile" == Basile STARYNKEVITCH <basile@starynkevitch.net> writes:

Basile> Plugins into GCC are expected to add optimisation passes (see file
Basile> gcc/passes.c function init_optimization_passes and I don't understand
Basile> what exactly LD_PRELOAD trick (unless you also redefine this very
Basile> function init_optimization_passes in your ld_preload-ed plugin) would
Basile> enable this.

FWIW I don't think this is an especially fruitful avenue of
discussion.  Let's take future conversation about LD_PRELOAD off-list.

Basile> So Tom or Taras, could you please elaborate on this? What tricks are
Basile> you thinking of?

I think it isn't extremely hard for a preloaded .so to look up the
pass list and dynamically modify it.

This would be a big hack of course.  I don't really recommend it.

I do wonder if there is a platform out there that acts as if -rdynamic
were the default.  On such a platform you could already write plugins.

Tom

Re: Progress on GCC plugins ?

[Alexandre Oliva]

On Nov 26, 2007, Tom Tromey <tromey@redhat.com> wrote:

> I do wonder if there is a platform out there that acts as if -rdynamic
> were the default.

I'm pretty sure I was surprised when I first ran into the need for
-rdynamic on GNU/Linux, because other platforms I'd used didn't need
that.  I'd guess that would have been SunOS4.1.3 or maybe Solaris2.1,
back then.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
FSF Latin America Board Member         http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Alexandre Oliva <aoliva@redhat.com> writes:

> On Nov 26, 2007, Tom Tromey <tromey@redhat.com> wrote:
> 
> > I do wonder if there is a platform out there that acts as if -rdynamic
> > were the default.
> 
> I'm pretty sure I was surprised when I first ran into the need for
> -rdynamic on GNU/Linux, because other platforms I'd used didn't need
> that.  I'd guess that would have been SunOS4.1.3 or maybe Solaris2.1,
> back then.

-rdynamic was the default on SunOS and on SVR4.  It was not the
default on Solaris.  It is not the default for the GNU linker because
I was copying the Solaris linker.

Ian

Re: Progress on GCC plugins ?

[Richard Kenner]

> I have a different fear: that gcc will become increasing irrelevant,
> as more and more new programmers learn to work on alternative free
> compilers instead.  That is neutral with regard to freedom, but it
> will tend to lose the many years of experience which have been put
> into gcc.  In my view, if we can't even get ourselves together to
> permit something as simple as plugins with an unstable API, then we
> deserve to lose.

As was said before, the difficultly in people working with GCC is
primarily lack of adequate documentation.  Creating a "plugin" interface
is certainly much more fun than writing documentation, but doesn't help
this issue nearly as much.  Moreover, writing documentation is not a
potential legal threat while plugins are.  To me, that argues strongly
against plugins and in favor of much more documentation.

Re: Progress on GCC plugins ?

[Joe Buck]

On Fri, Nov 16, 2007 at 12:02:44PM -0500, Richard Kenner wrote:
> As was said before, the difficultly in people working with GCC is
> primarily lack of adequate documentation.  Creating a "plugin" interface
> is certainly much more fun than writing documentation, but doesn't help
> this issue nearly as much.  Moreover, writing documentation is not a
> potential legal threat while plugins are.  To me, that argues strongly
> against plugins and in favor of much more documentation.

More documentation: a good thing.  Contributions welcome.

Plugins a potential legal threat: you must be using "legal threat" in some
strange way I don't understand, but I don't see them as a threat at all.

In any case, the two issues are orthogonal.

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Richard Kenner wrote:
> 
> As was said before, the difficultly in people working with GCC is
> primarily lack of adequate documentation.  

I am not sure of that.

GCC is a huge piece of software. This is the major difficulty: grasping 
a 3MLOC software whose source is available, rather well commented, with 
some documentation (I agree it could be better; feel free to add it at 
least on the wiki).

Compare this to the Linux kernel (which I do not know in detail). While 
it might be more documented in the source tree, it is certainly more 
documented outside (there are several good books on kernel internals; 
I'm not sure that equivalent books for GCC exist, and I would spend 50 
euros -of my own money- for such a book if it existed).

And still, diving and contributing to the Linux kernel and/or to GCC is 
really hard (I admit I don't know what is harder, and probably nobody 
knows both software in the same details!).

The biggest barrier to working on GCC is a learning & working effort 
barrier. I don't think documentation would help a lot (and maintaining 
it is a nightmare).

I am not sure (I really don't know) if more documentation on GCC exist, 
even inside companies with teams of a dozen of talented GCC 
contributors. I don't believe (maybe I am wrong) that for instance IBM 
or Google or AMD (or any company with several GCC developers full time) 
have a lot of (maybe proprietary) internal documentation on GCC.

I even don't believe that competitor proprietary compilers are much more 
documented than GCC.

GCC is an incredibly complex software (because any similar compiler has 
to be complex and huge).

PS. My mentor on GCC has been Sebastian Pop. I'll never thank him enough!

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Joe Buck]

On Fri, Nov 16, 2007 at 06:15:50PM +0100, Basile STARYNKEVITCH wrote:
> I even don't believe that competitor proprietary compilers are much more 
> documented than GCC.

Depends.  Vendors of compiler front ends (those sold for extension by
others) provide very good documentation, much better than any that GCC
has; this is necessary since they are sold to be extended by the buyer.
I won't name names because it's inappropriate to plug proprietary
software on this list.

RE: Progress on GCC plugins ?

[Gerald.Williams]

Much as I hate prolonging a probably-pointless discussion...

I hope we aren't thinking about keeping things difficult for
everybody simply because everybody includes some people who
may want to take advantage of GCC in a proprietary way. In
the long term, this only benefits the folks you'd be trying
to exclude.

Think about it. You have nothing to fear from people writing
trivial add-ons (if they're useful, they'll be duplicated in
open source; if not, they'll fade away). The only people you
might need to worry about would be those writing significant
new compiler designs/enhancements using GCC as a starting
point (and possibly trying to get past the GPL by avoiding
direct linking/etc.). Yet as has already been pointed out,
they already can do this by creating their own interface to
plug into. If they use a standard interface (since the code
base would favor this), it would be easier to replace their
plug-ins with open-source alternatives later.

-Jerry

Re: Progress on GCC plugins ?

[Joe Buck]

On Fri, Nov 16, 2007 at 07:29:12PM +0100, Gerald.Williams@infineon.com wrote:
> I hope we aren't thinking about keeping things difficult for
> everybody simply because everybody includes some people who
> may want to take advantage of GCC in a proprietary way. In
> the long term, this only benefits the folks you'd be trying
> to exclude.

RMS believes that people who extend GCC, hoping to take their extensions
proprietary, and then finding that they can't, will then just decide to
contribute the code, if it is useful, since otherwise they can't
distribute and have to support it by themselves forever, or else they have
to risk legal problems.  And he has some evidence that this sometimes
happens (C++, Objective-C, many contributed back ends).  So the intent
isn't to prevent certain people from using it, but to have those people
contribute the changes back even if that isn't their preference.

Now that's fine as far as it goes, but when it becomes a defense
of an opaque, non-extendable architecture we have a problem.

RE: Progress on GCC plugins ?

[Gerald.Williams]

Joe Buck wrote:
> RMS believes that people who extend GCC, hoping to take their
extensions
> proprietary, and then finding that they can't, will then just decide
to
> contribute the code, if it is useful, since otherwise they can't
> distribute and have to support it by themselves forever, or else they
have
> to risk legal problems.  And he has some evidence that this sometimes
> happens (C++, Objective-C, many contributed back ends).  So the intent
> isn't to prevent certain people from using it, but to have those
people
> contribute the changes back even if that isn't their preference.
> 
> Now that's fine as far as it goes, but when it becomes a defense
> of an opaque, non-extendable architecture we have a problem.

Agreed. It can also make it harder to contribute changes back, thus
possibly precluding some contributions.

-Jerry

Re: Progress on GCC plugins ?

[Diego Novillo]

Richard Kenner wrote:
>> I have a different fear: that gcc will become increasing irrelevant,
>> as more and more new programmers learn to work on alternative free
>> compilers instead.  That is neutral with regard to freedom, but it
>> will tend to lose the many years of experience which have been put
>> into gcc.  In my view, if we can't even get ourselves together to
>> permit something as simple as plugins with an unstable API, then we
>> deserve to lose.
> 
> As was said before, the difficultly in people working with GCC is
> primarily lack of adequate documentation.  Creating a "plugin" interface
> is certainly much more fun than writing documentation, but doesn't help
> this issue nearly as much.  Moreover, writing documentation is not a
> potential legal threat while plugins are.  To me, that argues strongly
> against plugins and in favor of much more documentation.

No, this argument is fallacious.  Plug-ins and poor documentation are 
not, and should not be related.  Poor documentation is an orthogonal 
problem which ALSO needs to be addressed.

The existence of a plug-in framework with bad/no documentation does not 
make working with GCC any easier.


Diego.

Re: Progress on GCC plugins ?

[Martin Jambor]

Hi,

On Nov 16, 2007 6:45 PM, Diego Novillo <dnovillo@google.com> wrote:
> Richard Kenner wrote:
> > As was said before, the difficultly in people working with GCC is
> > primarily lack of adequate documentation.  Creating a "plugin" interface
> > is certainly much more fun than writing documentation, but doesn't help
> > this issue nearly as much.  Moreover, writing documentation is not a
> > potential legal threat while plugins are.  To me, that argues strongly
> > against plugins and in favor of much more documentation.
>
> No, this argument is fallacious.  Plug-ins and poor documentation are
> not, and should not be related.  Poor documentation is an orthogonal
> problem which ALSO needs to be addressed.

I brought up the documentation in this debate so let me clarify what I
meant.

I simply wanted to tell that  you cannot expect that plugins are going
to make life  easier for newcomers. If that is your  goal, you have to
do something else.

On the  contrary, I  have acknowledged there  might be other  and very
beneficial  reasons  for  having  such  a framework  like  the  static
analysis tools which I know nothing about.

The  conclusion is  that  the  arguments in  favor  of plugins  should
concentrate  on these  technical advantages  rather than  on research.
Such arguments,  if explained a  bit more thoroughly, might  indeed be
very interesting to hear (read).

Finally,  I also  do not  think plugins  would all  of a  sudden allow
anyone to hijack the compiler more easily than it is possible today.

Martin

Re: Progress on GCC plugins ?

[Robert Dewar]

Diego Novillo wrote:

> No, this argument is fallacious.  Plug-ins and poor documentation are 
> not, and should not be related.  Poor documentation is an orthogonal 
> problem which ALSO needs to be addressed.

Actually to me if you have plug-ins, good documentation of the plug-in
interface is absolutely essential, so I am a bit cooncerned to hear of
people eager to program plugin facilities, and at the same time hear
that people are reluctant to document.

It's interestinng to note that in the Ada world, there is an ISO
standard for plugins, which is compiler/vendor neutral (at least
in theory, in practice there are some implementation dependencies).
That's the ASIS interface (Ada Semantic Interface Specification).
> 
> The existence of a plug-in framework with bad/no documentation does not 
> make working with GCC any easier.
> 
> 
> Diego.

Re: Progress on GCC plugins ?

[Gabriel Dos Reis]

Robert Dewar <dewar@adacore.com> writes:

| It's interestinng to note that in the Ada world, there is an ISO
| standard for plugins, which is compiler/vendor neutral (at least
| in theory, in practice there are some implementation dependencies).
| That's the ASIS interface (Ada Semantic Interface Specification).

So, is it that plugins are good for Ada (and I presume the GNU Ada
front end) but not for GCC? 

-- Gaby

Re: Progress on GCC plugins ?

[Richard Kenner]

> Robert Dewar <dewar@adacore.com> writes:
> 
> | It's interestinng to note that in the Ada world, there is an ISO
> | standard for plugins, which is compiler/vendor neutral (at least
> | in theory, in practice there are some implementation dependencies).
> | That's the ASIS interface (Ada Semantic Interface Specification).
> 
> So, is it that plugins are good for Ada (and I presume the GNU Ada
> front end) but not for GCC? 

Robert is using the word "plugin" differently.  ASIS is an interface
and a library.  There are no plugins in the sense discussed here.  He
means it in a very generic sense, in the sense that we already have it
for GCC.

Re: Progress on GCC plugins ?

[Robert Dewar]

Richard Kenner wrote:
>> Robert Dewar <dewar@adacore.com> writes:
>>
>> | It's interestinng to note that in the Ada world, there is an ISO
>> | standard for plugins, which is compiler/vendor neutral (at least
>> | in theory, in practice there are some implementation dependencies).
>> | That's the ASIS interface (Ada Semantic Interface Specification).
>>
>> So, is it that plugins are good for Ada (and I presume the GNU Ada
>> front end) but not for GCC? 
> 
> Robert is using the word "plugin" differently.  ASIS is an interface
> and a library.  There are no plugins in the sense discussed here.  He
> means it in a very generic sense, in the sense that we already have it
> for GCC.

So I am not sure I understand Richard's points above, so let me be clear
about what ASIS is.

It is a set of libraries, and a well defined API, that allows generic
tools to be written that have full access to the semantic information
discovered by the compiler. This API is fully documented and defined
in a compiler-neutral form.

I am not at all clear that we have ANYTHING like that for GCC, so I
am completely puzzled by Richard's last remark.

Let's take an example, suppose we want to write a semantic analysis
tool (e.g. the Mozilla style checker mentioned earlier). For Ada,
we can write an ASIS application and we need to know NOTHING AT ALL
about the internals of the compiler we are using, we only read the
ASIS documentation. Indeed we could start writing that tool before
we decided which Ada commpiler we would use.

I know of nothing vaguely equivalent to this in the general gcc
context, am I really missing something that big?

On the other hand, if your mission is to plug in an extra optimization
pass, ASIS won't help since it is strictly a read-only interface with
no provision for feeding back information to the compiler.

It is theoretically possible to write a compiler back end using the
ASIS interface, but in practice I think this would be an extremely
difficult task. For sure it has never happened so far.

What is interesting is that ASIS provides a well-defined clearly
legally-ok interface from the commpiler to proprietary tools,
and there are a number of proprietary ASIS tools on the market.

Re: Progress on GCC plugins ?

[Richard Kenner]

> So I am not sure I understand Richard's points above, so let me be clear
> about what ASIS is.
> 
> It is a set of libraries, and a well defined API, that allows generic
> tools to be written that have full access to the semantic information
> discovered by the compiler. This API is fully documented and defined
> in a compiler-neutral form.
> 
> I am not at all clear that we have ANYTHING like that for GCC, so I
> am completely puzzled by Richard's last remark.

The discussion here is competely different.  The issue isn't the interface,
but the mechanism of how it's called.

A "plugin" here means a module that would be dynamically loaded by GCC, as
opposed to being linked in to the compiler statically.  In other words,
once a plugin mechanism exists, it's possible to add passes to GCC without
having to change the compiler at all.  The analogy are the plugins to Mozilla.

Re: Progress on GCC plugins ?

[Robert Dewar]

Richard Kenner wrote:
>> So I am not sure I understand Richard's points above, so let me be clear
>> about what ASIS is.
>>
>> It is a set of libraries, and a well defined API, that allows generic
>> tools to be written that have full access to the semantic information
>> discovered by the compiler. This API is fully documented and defined
>> in a compiler-neutral form.
>>
>> I am not at all clear that we have ANYTHING like that for GCC, so I
>> am completely puzzled by Richard's last remark.
> 
> The discussion here is competely different.  The issue isn't the interface,
> but the mechanism of how it's called.
> 
> A "plugin" here means a module that would be dynamically loaded by GCC, as
> opposed to being linked in to the compiler statically.  In other words,
> once a plugin mechanism exists, it's possible to add passes to GCC without
> having to change the compiler at all.  The analogy are the plugins to Mozilla.

Sure, that's the *mechanism* but the usage of such a plug-in will be 
varied. As is clear from this long thread, people have many ideas of
what might be done with such a plugin

a) separate (propietary?) back end .. could conceivably be done using
ASIS.

b) separate analysis tool (e.g. Mozilla style rules) could most 
certainly be done with ASIS.

c) new optimization pass -- out of range for ASIS.

Re: Progress on GCC plugins ?

[Gabriel Dos Reis]

On Sun, 18 Nov 2007, Robert Dewar wrote:

| Richard Kenner wrote:
| > > Robert Dewar <dewar@adacore.com> writes:
| > >
| > > | It's interestinng to note that in the Ada world, there is an ISO
| > > | standard for plugins, which is compiler/vendor neutral (at least
| > > | in theory, in practice there are some implementation dependencies).
| > > | That's the ASIS interface (Ada Semantic Interface Specification).
| > >
| > > So, is it that plugins are good for Ada (and I presume the GNU Ada
| > > front end) but not for GCC? 
| > 
| > Robert is using the word "plugin" differently.  ASIS is an interface
| > and a library.  There are no plugins in the sense discussed here.  He
| > means it in a very generic sense, in the sense that we already have it
| > for GCC.
| 
| So I am not sure I understand Richard's points above, so let me be clear
| about what ASIS is.
| 
| It is a set of libraries, and a well defined API, that allows generic
| tools to be written that have full access to the semantic information
| discovered by the compiler. This API is fully documented and defined
| in a compiler-neutral form.

Yes, I'm familiar with ASIS, ANNA, etc.  And Yes, Kenner's point sound a
bit strange to me, but then, this whole discussion seems so strange to
me to have given the current situation in 2007.

-- Gaby

Re: Progress on GCC plugins ?

[Gabriel Dos Reis]

On Sun, 18 Nov 2007, Robert Dewar wrote:

| Let's take an example, suppose we want to write a semantic analysis
| tool (e.g. the Mozilla style checker mentioned earlier). For Ada,
| we can write an ASIS application and we need to know NOTHING AT ALL
| about the internals of the compiler we are using, we only read the
| ASIS documentation. Indeed we could start writing that tool before
| we decided which Ada commpiler we would use.
| 
| I know of nothing vaguely equivalent to this in the general gcc
| context, am I really missing something that big?

I have been able to build similar tool for at least two radically
different C++ front ends -- one being proprietary, the other one 
being GCC (the most painful to work with).  We call the representation
`IPR' (for Internal Program Representation).  The interface is
completely compiler neutral.  I haven't tested it myself on another
important proprietary compiler but from feedback people we talked
with, we should not worry.  Yes, that is not an ISO-approved thing,
but just a data point something `vaguely equivalent to this in the
general g++' context has been done.  We have not released IPR yet, but
it be open source a fairly open license.  

| On the other hand, if your mission is to plug in an extra optimization
| pass, ASIS won't help since it is strictly a read-only interface with
| no provision for feeding back information to the compiler.

IPR doesn't attempt to replace compiler internal data structures for
writing optimizations.  Nor it is a plugin into GIMPLE or RTL -- as I
said, it is designed to be compiler neutral.

-- Gaby

Re: Progress on GCC plugins ?

[Robert Dewar]

Gabriel Dos Reis wrote:

> I have been able to build similar tool for at least two radically
> different C++ front ends -- one being proprietary, the other one 
> being GCC (the most painful to work with).  We call the representation
> `IPR' (for Internal Program Representation).  The interface is
> completely compiler neutral.  I haven't tested it myself on another
> important proprietary compiler but from feedback people we talked
> with, we should not worry.  Yes, that is not an ISO-approved thing,
> but just a data point something `vaguely equivalent to this in the
> general g++' context has been done.  We have not released IPR yet, but
> it be open source a fairly open license.  

It would be great if this could be the basis for an ASIS-similar
standard for C++, even if it is only a de factor semi-standard :-)
> 
> | On the other hand, if your mission is to plug in an extra optimization
> | pass, ASIS won't help since it is strictly a read-only interface with
> | no provision for feeding back information to the compiler.
> 
> IPR doesn't attempt to replace compiler internal data structures for
> writing optimizations.  Nor it is a plugin into GIMPLE or RTL -- as I
> said, it is designed to be compiler neutral.
> 
> -- Gaby

Re: Progress on GCC plugins ?

[Robert Dewar]

Gabriel Dos Reis wrote:
> Robert Dewar <dewar@adacore.com> writes:
> 
> | It's interestinng to note that in the Ada world, there is an ISO
> | standard for plugins, which is compiler/vendor neutral (at least
> | in theory, in practice there are some implementation dependencies).
> | That's the ASIS interface (Ada Semantic Interface Specification).
> 
> So, is it that plugins are good for Ada (and I presume the GNU Ada
> front end) but not for GCC? 

Well remember that the starting point in Ada that makes the plugin
useful is the ISO standard which achieves several things:

a) uniformity across different vendor implementations. It is possible
to write a semantic analysis tool that will run with GNAT or with
other Ada compilers implementing ASIS.

b) full formal documentation of the interface

Note that the existence of b) means clearly that the interface
is not proprietary, and in the case of GNAT not associated with
the GPL in any way in my non-lawyer, but offically-court-expert
view. There is a big difference between standard interfaces
of this kind and cooked up interfaces between two programs that
are dependent on one another to form a complete product.

To me it is premature to implement any kind of plug in without
complete documentation that gets extensively reviewed in advance
of any implementation, so I would hesitate to use the Ada ASIS
approach as justification for cooking up some idiosyncratic
not-completely-well-documented interface for gcc.

Re: Progress on GCC plugins ?

[Robert Dewar]

Gabriel Dos Reis wrote:
> Robert Dewar <dewar@adacore.com> writes:
> 
> | It's interestinng to note that in the Ada world, there is an ISO
> | standard for plugins, which is compiler/vendor neutral (at least
> | in theory, in practice there are some implementation dependencies).
> | That's the ASIS interface (Ada Semantic Interface Specification).
> 
> So, is it that plugins are good for Ada (and I presume the GNU Ada
> front end) but not for GCC? 

All versions of the GNAT front end can produce the trees needed as
INPUT to the ASIS library (these trees are reasonably documented,
see sinfo and einfo, but nothing like formally documented at the
standards level).

The ASIS library itself is not part of the FSF sources, because
that would seem contrary to the general GCC goal of discouraging
proprietary coupling.
> 
> -- Gaby

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Ian Lance Taylor wrote:
> But as you know, most gcc ports are never contributed anyhow.

Naively, I didn't know that!
I thought most ports were contributed, but some rejected because of code 
quality, lack of reviewers, etc....

But does these ports are published elsewhere, in the spirit of GPL, or 
are there distributed in a fully proprietary & binary only way (hence 
violating the GPL)?

 > Ports
> that people hire Red Hat to do are contributed, but I can easily count
> six gcc ports I've seen myself that were never contributed.  So again
> I don't see a substantive difference here.

Regarding GCC plugins, and in contrast to some, I still view them as a 
big progress (avoiding a make bootstrap is already significant). In 
particular, a GCC plugin machinery permit quicker experimentation of new 
stuff, and also would perhaps permit inclusion of some specialized code 
(like static analysis) which won't fit in the trunk easily. Of course it 
is bound by the GPL and should be GPL.

There is one (minor, & insignificant in my opinion) argument against 
dynamic plugins: they require a dynamic loading machinery (typically the 
libdl, dlopen or equivalent libtldl) which in principle could be 
unavailable on some bizarre hosts (but I don't know of anymore such 
host) In other words, they require additional features than the 
theoretical plain C ANSI compiler.

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Basile STARYNKEVITCH <basile@starynkevitch.net> writes:

> Ian Lance Taylor wrote:
> > But as you know, most gcc ports are never contributed anyhow.
> 
> Naively, I didn't know that!
> I thought most ports were contributed, but some rejected because of
> code quality, lack of reviewers, etc....
> 
> But does these ports are published elsewhere, in the spirit of GPL, or
> are there distributed in a fully proprietary & binary only way (hence
> violating the GPL)?

Fallacy of the excluded middle.  I'm not aware of anybody who violates
the GPL when distributing gcc.  But the GPL permits private
distribution: if you give people both the binaries and the sources,
you have satisfied the requirements of the GPL.  That is what happens
with private gcc ports: the companies distribute the modified compiler
to their customers, but not to the public at large.  Under the terms
of the GPL, their customers are of course free to distribute the
compiler to anybody they choose.  In practice, they don't bother; why
should they?


> There is one (minor, & insignificant in my opinion) argument against
> dynamic plugins: they require a dynamic loading machinery (typically
> the libdl, dlopen or equivalent libtldl) which in principle could be
> unavailable on some bizarre hosts (but I don't know of anymore such
> host) In other words, they require additional features than the
> theoretical plain C ANSI compiler.

Yes.  But in practice they can work on GNU/Linux, all Unix systems,
Windows, and MacOS, so I don't think this is a major concern.

Ian

Re: Progress on GCC plugins ?

[Martin Michlmayr]

* Ian Lance Taylor <iant@google.com> [2007-11-16 07:49]:
> But as you know, most gcc ports are never contributed anyhow.  Ports
> that people hire Red Hat to do are contributed, but I can easily
> count six gcc ports I've seen myself that were never contributed.

Can you list those six ports?  Has anyone tried to talk to those
people to get them to contribute?
-- 
Martin Michlmayr
http://www.cyrius.com/

Re: Progress on GCC plugins ?

[Ian Lance Taylor]

Martin Michlmayr <tbm@cyrius.com> writes:

> * Ian Lance Taylor <iant@google.com> [2007-11-16 07:49]:
> > But as you know, most gcc ports are never contributed anyhow.  Ports
> > that people hire Red Hat to do are contributed, but I can easily
> > count six gcc ports I've seen myself that were never contributed.
> 
> Can you list those six ports?  Has anyone tried to talk to those
> people to get them to contribute?

C2 Microsystems Jazz
C2 Microsystems ENE
Cray X-2
Cradle Technologies DSE
A processor from a Japanese company which I have forgotten, maybe Toshiba
and, OK, I thought I had seen six but now I can only remember five

There are also routinely messages on gcc-help referring to private
ports which I have not seen myself.

The truth is, there is no real advantage to anybody for these ports to
be contributed.  They often have machine specific changes to the
machine independent parts of gcc, which would have to be managed
somehow.  These are not popular processors.  Nobody outside the
manufacturer has the knowledge or interest to maintain the backends.
We've seen plenty of cases where a backend was contributed and then
dropped a couple of years later.  These would just be more of the
same.

Ian

Re: Progress on GCC plugins ?

[Emmanuel Fleury]

Ian Lance Taylor wrote:
> 
> I think it's quite important for gcc's long-term health to permit and
> even encourage academic researchers and students to use it.  And I see
> plugins as directly supporting that goal.  Note that I don't see any
> problem with requiring (or attempting to require) that any plugin be
> covered by the GPL.

Not only this but I'm also more and more concerned about closed source
static analysis and code verification tools such as Coverity
(http://www.coverity.com/), SLAM (http://research.microsoft.com/slam/)
and others (see the list on Wikipedia:
http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis).

If our concern in the 90's was to build code with an open source
compiler, our concern nowadays should be to produce better code inside
the open source community... Not having a common open source toolkit to
perform *advanced* analysis of code and to build software on it will
ends up having an unbalanced quality between proprietary and open source
projects in a very unfair manner.

We are now only at the beginning of the rise of these tools and yet it
makes quite a difference between the code quality of projects using it
or not... Having GCC to provide plug-ins would (in my humble opinion)
indirectly boost code quality in open source development by allowing
better tools to appears and help to fill the gap that started to appear
between proprietary and open source code.

Moreover, considering the GCC developers point-of-view, it becomes more
and more obvious (at least to me) that the next 'frontier' in modern
compiler design will be to export features to perform static analysis
and verification on code.

At last, I would say that I'm more concerned about the number of closed
source tools to perform code analysis (with or without the help of GCC)
than the hypothetical use of GCC by proprietary softwares because not
having this analysis tools will make us loose the race anyway...

Well, of course, this is only my very personal point of view on this
topic... :)

Regards
-- 
Emmanuel Fleury

Research is an organized method for keeping you
reasonably dissatisfied with what you have.
  -- Charles Kettering

Re: Progress on GCC plugins ?

[Bernd Schmidt]

Ian Lance Taylor wrote:

> I think it's quite important for gcc's long-term health to permit and
> even encourage academic researchers and students to use it.  And I see
> plugins as directly supporting that goal.  Note that I don't see any
> problem with requiring (or attempting to require) that any plugin be
> covered by the GPL.
> 
> So from my perspective the downside of plugins is very small, and the
> upside is very large.

I must admit I don't understand the upside.  I've always thought of
plugins as something proprietary programs need because their source
isn't open.

In my view, plugins will bitrot quickly as GCC's interface changes; and
they won't even help with the learning curve - does anyone believe for a
second you won't have to understand compiler internals to write a plugin?


Bernd
-- 
This footer brought to you by insane German lawmakers.
Analog Devices GmbH      Wilhelm-Wagenfeld-Str. 6      80807 Muenchen
Sitz der Gesellschaft Muenchen, Registergericht Muenchen HRB 40368
Geschaeftsfuehrer Thomas Wessel, William A. Martin, Margaret Seif

Re: Progress on GCC plugins ?

[Richard Kenner]

> I must admit I don't understand the upside.  I've always thought of
> plugins as something proprietary programs need because their source
> isn't open.
> 
> In my view, plugins will bitrot quickly as GCC's interface changes; and
> they won't even help with the learning curve - does anyone believe for a
> second you won't have to understand compiler internals to write a plugin?

I agree.  I don't understand the upside either, for the same reasons.

If I want to test some piece of code in the compiler, I don't have to
bootstrap with or without plugins (unless I need to for testing purposes).
The only difference is how I link, which seems a completely trivial
distinction to me.

RE: Progress on GCC plugins ?

[Dave Korn]

On 16 November 2007 17:25, Richard Kenner wrote:

> If I want to test some piece of code in the compiler, I don't have to
> bootstrap with or without plugins (unless I need to for testing purposes).
> The only difference is how I link, which seems a completely trivial
> distinction to me.

  That seems, on the face of it, a specious argument.  It's only true because
you have already in the past bootstrapped the compiler and kept the build
directory lying around.  Saying "I don't have to bootstrap because I've
already done it" doesn't justify your claim that it's no easier or harder for
new developers (who haven't got $objdir kicking around already).


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: Progress on GCC plugins ?

[Basile STARYNKEVITCH]

Bernd Schmidt wrote:
> Ian Lance Taylor wrote:
> 
>> I think it's quite important for gcc's long-term health to permit and
>> even encourage academic researchers and students to use it.  And I see
>> plugins as directly supporting that goal.  Note that I don't see any
>> problem with requiring (or attempting to require) that any plugin be
>> covered by the GPL.
>>
>> So from my perspective the downside of plugins is very small, and the
>> upside is very large.
> 
> I must admit I don't understand the upside.  I've always thought of
> plugins as something proprietary programs need because their source
> isn't open.


No. Many firefox plugins and linux kernel plugins (called modules) are 
opensource!

Every GCC plugin has to be open source (because GCC is GPL) because it 
has to be GPL!

> 
> In my view, plugins will bitrot quickly as GCC's interface changes; and
> they won't even help with the learning curve - does anyone believe for a
> second you won't have to understand compiler internals to write a plugin?

Plugin will help to experiment, and a major point of a plugin is that it 
can be removed or disabled without impacting GCC major ability to 
compile stuff (it might only decrease performance, additional 
functionalities, .. if removed).

Plugins should make daily work on GCC easier: you don't have to 
bootstrap everything! And you could release it (with its source code) 
even if it is incomplete, .... because it can be very easily disabled.

And plugins could even be dynamically produced (shameless ad for my 
paper & work).

The main point of a plugin is that it could be removed easily! Much 
harder for anything else.

Just hacking GCC for making a feature which can be robustly disabled is 
a significant work! Plugins get you that for free... And it is essential 
for every kind of "experimental" or "additional" features! You really 
want it to be easy to remove!

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Progress on GCC plugins ?

[Joe Buck]

On Fri, Nov 16, 2007 at 06:13:32PM +0100, Bernd Schmidt wrote:
> I must admit I don't understand the upside.  I've always thought of
> plugins as something proprietary programs need because their source
> isn't open.

On the contrary, many successful free programs have plugins.

Consider Emacs.  The user can extend the editor using a Turing-complete
extension language (elisp), and commands in the extension language have
the same status as native commands in C.

Consider Firefox.  Again, there are vast numbers of useful extensions.
They allow users to customize the tool in ways that the core developers
wouldn't necessarily want to support.

> In my view, plugins will bitrot quickly as GCC's interface changes; and
> they won't even help with the learning curve - does anyone believe for a
> second you won't have to understand compiler internals to write a plugin?

A poorly-designed plugin architecture would rot quickly.  A well-designed
architecture would require maintainance, but not as much work to keep up
(Firefox plugins often require changes to keep up with new versions).

Re: Progress on GCC plugins ?

[Diego Novillo]

Bernd Schmidt wrote:

> I must admit I don't understand the upside.  I've always thought of
> plugins as something proprietary programs need because their source
> isn't open.

On the contrary, the plug-in model is used in several large and complex 
open source projects (firefox, thunderbird, gimp, linux kernel, etc). 
It's precisely the complexity reduction features that make plug-ins so 
attractive.

For us this means attracting more developers, which in turn means bigger 
potential for attracting long-time contributors, which helps with the 
long-term survival of GCC.

> In my view, plugins will bitrot quickly as GCC's interface changes; and
> they won't even help with the learning curve - does anyone believe for a
> second you won't have to understand compiler internals to write a plugin?

Of course not.  But with a plug-in framework you get to interact with 
exactly the set of components important for your work, you don't have to 
deal with the whole compiler and its internal build machinery.


Diego.

Re: Progress on GCC plugins ?

[Tom Tromey]

>>>>> "Bernd" == Bernd Schmidt <bernds_cb1@t-online.de> writes:

Bernd> I must admit I don't understand the upside.  I've always thought of
Bernd> plugins as something proprietary programs need because their source
Bernd> isn't open.

Everybody explained about the existing free software that has plugins.
But, I thought I'd mention a few use cases for plugins.

The biggest benefit of a plugin system is that you can add things to
the compiler without requiring all your users to build their own
compiler.

E.g., Mozilla developers have said before (even earlier in this
thread) that they would like to be able to run Mozilla-specific
analysis passes over their code, say before checkin.  Probably this
consists of a bunch of warning checks that are suitable for Mozilla
but not suitable for inclusion in GCC.  With a plugin system they have
the option of providing "Mozilla GCC plugin for Fedora 9", or
whatever, and avoiding the mess of "to build Mozilla first you have to
build GCC with patch X".

Bernd> In my view, plugins will bitrot quickly as GCC's interface
Bernd> changes; and they won't even help with the learning curve -
Bernd> does anyone believe for a second you won't have to understand
Bernd> compiler internals to write a plugin?

Plugins are about deployment, not development.  They don't make
writing the code much simpler.  That is why we can argue that the risk
they pose is small: they don't make it significantly simpler to make a
proprietary GCC.

Tom

Re: Progress on GCC plugins ?

[Brendon Costa]

Tom Tromey wrote:
> Bernd> In my view, plugins will bitrot quickly as GCC's interface
> Bernd> changes; and they won't even help with the learning curve -
> Bernd> does anyone believe for a second you won't have to understand
> Bernd> compiler internals to write a plugin?
> 
> Plugins are about deployment, not development.  They don't make
> writing the code much simpler.  That is why we can argue that the risk
> they pose is small: they don't make it significantly simpler to make a
> proprietary GCC.
> 

In my situation this point is right on the mark. I follow with a
concrete example.

My project: "EDoc++", requires a patched version of GCC in order to
perform static analysis of source code. I approached the debian
maintainers list with a debian package for this project to see if they
would include it in the official repositories. It was not accepted and
the reason for that is because it includes another patched version of
GCC which takes up too much disk space. They don't want to accept
these sorts of projects because they all effectively require
duplicates of the same code(GCC). This problem of deployment could be
solved with plugins.

Development for my project would not be overly different with the
plugin framework included. I dont expect to be able to write plugins
without an understanding of the internals of GCC which the plugin is
being written for. What i expect from the framework is simplified,
standard method of deployment for additional GCC "features".


As an overall comment on the issues being discussed i think we should
not deliberately omit features that will be useful to the open source
community just to deny these same features to proprietary developers.
This is just shooting ourselves in the foot in order to do the same to
others. It makes no sense as we are restricting our own progress for a
possible misuse of the feature by others (That is already occurring
anyway).


Brendon.

Re: Progress on GCC plugins ?

[Diego Novillo]

Richard Kenner wrote:

> Therefore, I think it's important for us to make it as
> technically hard as possible for people to do such a linkage by readin and
> writing tree or communicating as different libraries or DLLs.  I'm very
> much against any sort of "plug in" precisely for this reason.

That's the line of reasoning that I find weak.  It is, in fact, 
extremely simple to integrate GCC into another compiler.  All you need 
to do is insert a gimple or RTL converter somewhere in passes.c.  Having 
plug-ins will not make this step any easier.

If a third party is willing to violate the GPL, the presence of a 
plug-in infrastructure will _not_ make their job significantly easier.

> That depends on the importance you attach to the philosophy of free
> software.

That's precisely the reason why I think it is imperative for us to keep 
improving GCC's internal architecture.  If we offer a solid compiler 
framework for people to use, we will increase its attractiveness to 
universities and research institutions, which are the very pools where 
future compiler engineers will come from.  That will help the long-term 
survival of the project.

It is very easy for us to make code developed using plug-ins to be 
covered by the GPL.  They will be using GCC header files, after all.


Diego.

Re: Progress on GCC plugins ?

[Richard Kenner]

> If a third party is willing to violate the GPL, the presence of a 
> plug-in infrastructure will _not_ make their job significantly easier.

The issue isn't the ease in which it violates the GPL, but the ease in
which you can show it *is* a violation!  If there's no plug-in and you
link directly with GCC, there's no question that such code is covered
by the GPL.  If you use a plug-in, there *is* such a question and people
might well believe it is *not* a GPL violation.

> It is very easy for us to make code developed using plug-ins to be 
> covered by the GPL.  They will be using GCC header files, after all.

I don't want to turn this into a legal discussion, but there is some legal
ambiguity about that statement too.