gcc extensibility

gcc extensibility

[Niels Möller]

I originally wrote this email as a reply to Ian Lance Taylor on a
different list, and he suggested that I send it also to the gcc list.
Please cc me on replies, since I'm not subscribed to the list. I hope
I'm not being too off-topic or off-the-mark.

Let me write down some reflections on gcc extensibility, even if I'm not
familiar at all with gcc internals.

1. I imagine the plugin API ought to stay in plain C, right?

2. Then there are at least two ways to think about the plugin API to,
   e.g., the gcc tree abstraction.

   Either one can define a C API one think the plugins will like, and
   then implement that on top of the internal C++ interfaces. These will
   be small wrapper functions, which lets the internal interfaces evolve
   without affecting the plugins.

   Or one sticks to a single "unified" tree API, to be used *both*
   internally and by plugins.

   I suspect the second option is the right one, because it brings some
   equality between plugin authors and gcc developers. It should make it
   easier to adopt a plugin into gcc proper. Together with (1), this
   forces the internal interface to be C rather than C++, which I guess
   you may see as a serious disadvantage.

   Going for a unified API, one gets less independence between plugins
   and gcc internals, but in return, one gets less clutter, and I think
   it may improve quality. Otherwise, it seems likely that one ends up
   with an internal interface which is powerful but somewhat ugly
   (internal use only, right?) and an external interface which may be
   beautiful on the surface, but in practice it's a second class citizen
   and awkward for doing interesting things with.

3. What is the purpose of the plugin API? I can see that it should make
   it easier to prototype new optimization passes. Both for educational
   purposes, and research, as well as by the gcc developers themselves.

   One of the goals you stated was "I think parts of GCC needs to move
   toward being a component of tools other than pure compilation, such
   as refactoring, profile analysis, debugging."

   I think we can all agree that's highly desirable. To be more
   concrete, I think it would be useful have access to information from
   the parse tree, from the symbol table (both for compiler and linker),
   dataflow analysis, etc, when editing C code in emacs. Is a plugin API
   the right tool for that type of integration? Or should one also have
   a gcc library, and have various other specialized tools link to that
   library?

   Maybe the organization of valgrind could provide some inspiration,
   with a couple of different tools on top of the same machinery.

Happy hacking,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

Re: gcc extensibility

[Richard Guenther]

On Thu, Mar 29, 2012 at 2:34 PM, Niels Möller <nisse@lysator.liu.se> wrote:
> I originally wrote this email as a reply to Ian Lance Taylor on a
> different list, and he suggested that I send it also to the gcc list.
> Please cc me on replies, since I'm not subscribed to the list. I hope
> I'm not being too off-topic or off-the-mark.
>
> Let me write down some reflections on gcc extensibility, even if I'm not
> familiar at all with gcc internals.
>
> 1. I imagine the plugin API ought to stay in plain C, right?
>
> 2. Then there are at least two ways to think about the plugin API to,
>   e.g., the gcc tree abstraction.
>
>   Either one can define a C API one think the plugins will like, and
>   then implement that on top of the internal C++ interfaces. These will
>   be small wrapper functions, which lets the internal interfaces evolve
>   without affecting the plugins.
>
>   Or one sticks to a single "unified" tree API, to be used *both*
>   internally and by plugins.
>
>   I suspect the second option is the right one, because it brings some
>   equality between plugin authors and gcc developers. It should make it
>   easier to adopt a plugin into gcc proper. Together with (1), this
>   forces the internal interface to be C rather than C++, which I guess
>   you may see as a serious disadvantage.

On the contrary - I think the first option is the right one.  Only that way
we can provide a stable ABI towards plugins.

>   Going for a unified API, one gets less independence between plugins
>   and gcc internals, but in return, one gets less clutter, and I think
>   it may improve quality. Otherwise, it seems likely that one ends up
>   with an internal interface which is powerful but somewhat ugly
>   (internal use only, right?) and an external interface which may be
>   beautiful on the surface, but in practice it's a second class citizen
>   and awkward for doing interesting things with.

It really depends on what plugins should be doing.  Or rather what
most plugins will end up doing.  In the end we probably will end up
with a stable plugin API/ABI for that common case (introspection
and simple instrumentation) and the awkward current one exporting
every GCC internal.

> 3. What is the purpose of the plugin API? I can see that it should make
>   it easier to prototype new optimization passes. Both for educational
>   purposes, and research, as well as by the gcc developers themselves.

No.  That's way easier to do if you are _not_ a plugin.

>   One of the goals you stated was "I think parts of GCC needs to move
>   toward being a component of tools other than pure compilation, such
>   as refactoring, profile analysis, debugging."

Right, and I see primarily those uses.

Richard.

>   I think we can all agree that's highly desirable. To be more
>   concrete, I think it would be useful have access to information from
>   the parse tree, from the symbol table (both for compiler and linker),
>   dataflow analysis, etc, when editing C code in emacs. Is a plugin API
>   the right tool for that type of integration? Or should one also have
>   a gcc library, and have various other specialized tools link to that
>   library?
>
>   Maybe the organization of valgrind could provide some inspiration,
>   with a couple of different tools on top of the same machinery.
>
> Happy hacking,
> /Niels
>
> --
> Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
> Internet email is subject to wholesale government surveillance.
>

Re: gcc extensibility

[Gabriel Dos Reis]

On Thu, Mar 29, 2012 at 7:34 AM, Niels Möller <nisse@lysator.liu.se> wrote:

> 1. I imagine the plugin API ought to stay in plain C, right?
>
> 2. Then there are at least two ways to think about the plugin API to,
>   e.g., the gcc tree abstraction.
>
>   Either one can define a C API one think the plugins will like, and
>   then implement that on top of the internal C++ interfaces. These will
>   be small wrapper functions, which lets the internal interfaces evolve
>   without affecting the plugins.
>
>   Or one sticks to a single "unified" tree API, to be used *both*
>   internally and by plugins.
>
>   I suspect the second option is the right one, because it brings some
>   equality between plugin authors and gcc developers.

It is a false equality.  The needs of plugins authors are not necessarily
the same as the need of GCC development itself.

You really do not want the existence of plugins to hinder (internal)
evolution of GCC itself.

-- Gaby

Re: gcc extensibility

[Romain Geissler]

Hi

Le 29 mars 2012 à 14:34, Niels Möller a écrit :

> 1. I imagine the plugin API ought to stay in plain C, right?

I don't know if this was already discussed and if the community
ended up with a clear answer for this question. If it's not the case
i would prefer a plugin interface in C++, for the same reasons it
was decided to slowly move the internals to C++.

Romain Geissler

Re: gcc extensibility

[Gabriel Dos Reis]

On Thu, Mar 29, 2012 at 10:34 AM, Romain Geissler
<romain.geissler@gmail.com> wrote:
> Hi
>
> Le 29 mars 2012 à 14:34, Niels Möller a écrit :
>
>> 1. I imagine the plugin API ought to stay in plain C, right?
>
> I don't know if this was already discussed and if the community
> ended up with a clear answer for this question. If it's not the case
> i would prefer a plugin interface in C++, for the same reasons it
> was decided to slowly move the internals to C++.
>

I do not think people working on plugins have come up with a
specification and an API they agree on.  Which makes any talk
of restricting GCC's own evolution premature if not pointless.

-- Gaby

Re: gcc extensibility

[Basile Starynkevitch]

On Thu, 29 Mar 2012 11:06:11 -0500
Gabriel Dos Reis <gdr@integrable-solutions.net> wrote:

> On Thu, Mar 29, 2012 at 10:34 AM, Romain Geissler
> <romain.geissler@gmail.com> wrote:
> > Hi
> >
> > Le 29 mars 2012 à 14:34, Niels Möller a écrit :
> >
> >> 1. I imagine the plugin API ought to stay in plain C, right?
> >
> > I don't know if this was already discussed and if the community
> > ended up with a clear answer for this question. If it's not the case
> > i would prefer a plugin interface in C++, for the same reasons it
> > was decided to slowly move the internals to C++.
> >
> 
> I do not think people working on plugins have come up with a
> specification and an API they agree on.


I believe plugin makers (if you can count me amongst them) don't have at all this
approach. By definition, a plugin should work with whatever interface a given GCC release
makes available. Plugins people by definition cannot alter or improve the interface that
GCC is giving to them (otherwise, they are no more plugins people, but GCC contributors).

What I mean is that it has absolutely no sense to make a plugin for something which is
derived from GCC (e.g. for an experimental branch or fork of GCC). The very idea of
plugins is that they should work with an unmodified distributed GCC.

To be more concrete, I am right now passing some time to make MELT usable with the GCC
4.7 released by Debian/Sid (exactly it), not with the GCC 4.7 I used to build myself
(which e.g. was compiled in C mode only).

Plugins don't have much sense if they don't work with widely distributed, but recent,
releases of GCC. So plugins people won't propose, as plugins developers, any API
interface to GCC; they will stick to whatever is available to them.

[in other words, I may propose some patch or enhancement to GCC, but then I am acting as
a GCC contributor, not as a plugin developer]

Cheers.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: gcc extensibility

[Gabriel Dos Reis]

On Thu, Mar 29, 2012 at 11:27 AM, Basile Starynkevitch
<basile@starynkevitch.net> wrote:
> On Thu, 29 Mar 2012 11:06:11 -0500
> Gabriel Dos Reis <gdr@integrable-solutions.net> wrote:
>
>> On Thu, Mar 29, 2012 at 10:34 AM, Romain Geissler
>> <romain.geissler@gmail.com> wrote:
>> > Hi
>> >
>> > Le 29 mars 2012 à 14:34, Niels Möller a écrit :
>> >
>> >> 1. I imagine the plugin API ought to stay in plain C, right?
>> >
>> > I don't know if this was already discussed and if the community
>> > ended up with a clear answer for this question. If it's not the case
>> > i would prefer a plugin interface in C++, for the same reasons it
>> > was decided to slowly move the internals to C++.
>> >
>>
>> I do not think people working on plugins have come up with a
>> specification and an API they agree on.
>
>
> I believe plugin makers (if you can count me amongst them) don't have at all this
> approach. By definition, a plugin should work with whatever interface a given GCC release
> makes available. Plugins people by definition cannot alter or improve the interface that
> GCC is giving to them (otherwise, they are no more plugins people, but GCC contributors).

I suspect that if plugins people want to make progress on this
recurring theme, they
will have to come up with a specification and an API.  Otherwise, they have only
themselves to blame if their plugins break from release to release.

-- Gaby

Re: gcc extensibility

[Basile Starynkevitch]

On Thu, 29 Mar 2012 11:42:30 -0500
Gabriel Dos Reis <gdr@integrable-solutions.net> wrote:
> I suspect that if plugins people want to make progress on this
> recurring theme, they
> will have to come up with a specification and an API.  Otherwise, they have only
> themselves to blame if their plugins break from release to release.


They blame nobody if their plugins break from one release to the next. They take this
incompatibility of GCC as part of their plugins developer's work.

Again, a plugin writer by definition uses whatever interface is given to him.

Cheers.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: gcc extensibility

[Gabriel Dos Reis]

On Thu, Mar 29, 2012 at 11:52 AM, Basile Starynkevitch
<basile@starynkevitch.net> wrote:

> They blame nobody if their plugins break from one release to the next. They take this
> incompatibility of GCC as part of their plugins developer's work.

Great!  Problem solved.

-- Gaby

Re: gcc extensibility

[Andrew MacLeod]

On 03/29/2012 12:52 PM, Basile Starynkevitch wrote:
>
> They blame nobody if their plugins break from one release to the next. They take this
> incompatibility of GCC as part of their plugins developer's work.
>
> Again, a plugin writer by definition uses whatever interface is given to him.
>
> Cheers.
>
>
That doesn't mean a plugin developer can't use their experience and 
offer opinions on what would be a good/better plugin interface.

If the plugin developers could agree on what a better interface would 
be, it would go a long way toward encouraging GCC to provide that interface.

Without such feedback, the status quo is unlikely to change since GCC 
developers don't know what a better interface would be...

Andrew

Re: gcc extensibility

[Romain Geissler]

Le 29 mars 2012 à 18:06, Gabriel Dos Reis a écrit :

> On Thu, Mar 29, 2012 at 10:34 AM, Romain Geissler
> <romain.geissler@gmail.com> wrote:
>> Hi
>> 
>> Le 29 mars 2012 à 14:34, Niels Möller a écrit :
>> 
>>> 1. I imagine the plugin API ought to stay in plain C, right?
>> 
>> I don't know if this was already discussed and if the community
>> ended up with a clear answer for this question. If it's not the case
>> i would prefer a plugin interface in C++, for the same reasons it
>> was decided to slowly move the internals to C++.
>> 
> 
> I do not think people working on plugins have come up with a
> specification and an API they agree on.  Which makes any talk
> of restricting GCC's own evolution premature if not pointless.

I didn't mean the choice of C or C++ for the future plugin API may
in any way alter the own GCC evolution. The API only consists in
a bunch of stable wrappers to the unstable internals. Once such
an API exists, that won't be hard to update the impacted wrappers
to follow that changes, and thus it would have only minor impact on
the internals evolution.

Re: gcc extensibility

[Romain Geissler]

Le 29 mars 2012 à 15:14, Richard Guenther a écrit :

> On Thu, Mar 29, 2012 at 2:34 PM, Niels Möller <nisse@lysator.liu.se> wrote:
>> I originally wrote this email as a reply to Ian Lance Taylor on a
>> different list, and he suggested that I send it also to the gcc list.
>> Please cc me on replies, since I'm not subscribed to the list. I hope
>> I'm not being too off-topic or off-the-mark.
>> 
>> Let me write down some reflections on gcc extensibility, even if I'm not
>> familiar at all with gcc internals.
>> 
>> 1. I imagine the plugin API ought to stay in plain C, right?
>> 
>> 2. Then there are at least two ways to think about the plugin API to,
>>   e.g., the gcc tree abstraction.
>> 
>>   Either one can define a C API one think the plugins will like, and
>>   then implement that on top of the internal C++ interfaces. These will
>>   be small wrapper functions, which lets the internal interfaces evolve
>>   without affecting the plugins.
>> 
>>   Or one sticks to a single "unified" tree API, to be used *both*
>>   internally and by plugins.
>> 
>>   I suspect the second option is the right one, because it brings some
>>   equality between plugin authors and gcc developers. It should make it
>>   easier to adopt a plugin into gcc proper. Together with (1), this
>>   forces the internal interface to be C rather than C++, which I guess
>>   you may see as a serious disadvantage.
> 
> On the contrary - I think the first option is the right one.  Only that way
> we can provide a stable ABI towards plugins.

I also think that the plugin layer should be as much as possible distinct from
the internals, may some patch break the plugin API for a short period of time
in the trunk, as soon as official releases keep it working well
(as i think plugins must not slow down GCC own evolution).

As plugins are only meant to perform introspection and minor tree transformations
they do not need the whole internal API. Such an API also require stability
contrary to internals which may change at any time. 

As a plugin developer, i don't think plugin developers should have
that equality with GCC developers you are worried about.

> 
>>   Going for a unified API, one gets less independence between plugins
>>   and gcc internals, but in return, one gets less clutter, and I think
>>   it may improve quality. Otherwise, it seems likely that one ends up
>>   with an internal interface which is powerful but somewhat ugly
>>   (internal use only, right?) and an external interface which may be
>>   beautiful on the surface, but in practice it's a second class citizen
>>   and awkward for doing interesting things with.
> 
> It really depends on what plugins should be doing.  Or rather what
> most plugins will end up doing.  In the end we probably will end up
> with a stable plugin API/ABI for that common case (introspection
> and simple instrumentation) and the awkward current one exporting
> every GCC internal.
> 
>> 3. What is the purpose of the plugin API? I can see that it should make
>>   it easier to prototype new optimization passes. Both for educational
>>   purposes, and research, as well as by the gcc developers themselves.
> 
> No.  That's way easier to do if you are _not_ a plugin.
> 
>>   One of the goals you stated was "I think parts of GCC needs to move
>>   toward being a component of tools other than pure compilation, such
>>   as refactoring, profile analysis, debugging."
> 
> Right, and I see primarily those uses.
> 
> Richard.

Well currently GCC deeply lacks the structure to break it into distinct
component. I roughly see plugins as a solution in the middle : you keep
the whole gcc in a single block, but still you can perform minor task
on your own by dlopening a plugin.so of your own rather than breaking
gcc in a proper structued libgcc.

Of course, a beautiful defined and structed libgcc would be welcome,
but this would take much more time refactoring the existing code base,
and see, plugins which are only a small part of that work, are already
tough to implement.

Romain Geissler

> 
> 
>>   I think we can all agree that's highly desirable. To be more
>>   concrete, I think it would be useful have access to information from
>>   the parse tree, from the symbol table (both for compiler and linker),
>>   dataflow analysis, etc, when editing C code in emacs. Is a plugin API
>>   the right tool for that type of integration? Or should one also have
>>   a gcc library, and have various other specialized tools link to that
>>   library?
>> 
>>   Maybe the organization of valgrind could provide some inspiration,
>>   with a couple of different tools on top of the same machinery.
>> 
>> Happy hacking,
>> /Niels
>> 
>> --
>> Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
>> Internet email is subject to wholesale government surveillance.
>> 

Re: gcc extensibility

[Niels Möller]

Gabriel Dos Reis <gdr@integrable-solutions.net> writes:

> It is a false equality.  The needs of plugins authors are not necessarily
> the same as the need of GCC development itself.

I'm not so sure of that. To be concrete, can you give some examples of
things that a plugin might want to do with an interface to the
tree-abstraction in gcc, which gcc itself would never do? And vice
versa?

The typical needs may be different, but I don't think it's wise to a
priori rule out the possibility of designing, e.g, a tree API, which is
both powerful enough and clean enough to serve well both plugins and the
rest of gcc.

> You really do not want the existence of plugins to hinder (internal)
> evolution of GCC itself.

Agreed. Question is how much interface stability is needed by plugins,
and how often gcc internals must change important interfaces. I would
expect that binary compatibility is not terribly important, i.e., it's
acceptable if plugins sometimes have to be recompiled even for a new
minor release (and we don't want to allow proprietary binary-only
plugins anyway, right?).

Breaking source level compatibility ought to be avoided for minor
releases. But the current situation, where, due to the name mangling, it
seems difficult for a plugin to be compatible even with different
configurations of the *same* minor release of gcc, seems a bit too
inconvenient.

Then I'd also expect that certain interfaces, e.g., the tree interface,
can be a lot more stable than others.

Bottom line: When interface changes in gcc really are needed, then just
do them (preferably in connection with a major release), and plugins
will have to follow.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

Re: gcc extensibility

[Jonathan Wakely]

On 29 March 2012 17:52, Basile Starynkevitch wrote:
> On Thu, 29 Mar 2012 11:42:30 -0500
> Gabriel Dos Reis <gdr@integrable-solutions.net> wrote:
>> I suspect that if plugins people want to make progress on this
>> recurring theme, they
>> will have to come up with a specification and an API.  Otherwise, they have only
>> themselves to blame if their plugins break from release to release.
>
>
> They blame nobody if their plugins break from one release to the next. They take this
> incompatibility of GCC as part of their plugins developer's work.
>
> Again, a plugin writer by definition uses whatever interface is given to him.

For a mature plugin API, maybe.

But plugins in GCC are still new and still evolving, if plugin authors
won't help shape the API and offer their advice now then they the API
will never be useful.

Don't expect people who don't care about plugins to do all the work of
making writing plugins easier.

Re: gcc extensibility

[Gabriel Dos Reis]

On Thu, Mar 29, 2012 at 12:39 PM, Romain Geissler
<romain.geissler@gmail.com> wrote:
> Le 29 mars 2012 à 18:06, Gabriel Dos Reis a écrit :
>
>> On Thu, Mar 29, 2012 at 10:34 AM, Romain Geissler
>> <romain.geissler@gmail.com> wrote:
>>> Hi
>>>
>>> Le 29 mars 2012 à 14:34, Niels Möller a écrit :
>>>
>>>> 1. I imagine the plugin API ought to stay in plain C, right?
>>>
>>> I don't know if this was already discussed and if the community
>>> ended up with a clear answer for this question. If it's not the case
>>> i would prefer a plugin interface in C++, for the same reasons it
>>> was decided to slowly move the internals to C++.
>>>
>>
>> I do not think people working on plugins have come up with a
>> specification and an API they agree on.  Which makes any talk
>> of restricting GCC's own evolution premature if not pointless.
>
> I didn't mean the choice of C or C++ for the future plugin API may
> in any way alter the own GCC evolution. The API only consists in
> a bunch of stable wrappers to the unstable internals. Once such
> an API exists, that won't be hard to update the impacted wrappers
> to follow that changes, and thus it would have only minor impact on
> the internals evolution.
>

From past discussions,  I gather that the plugins people
want an uncompromising access to every bits of GCC internals (hence
resist any notion of specification and API) and don't want to see evolution
of GCC that might break their working plugins, for example using C++ because
 their own plugins are written in C.  Yet, we also receive lectures on modules
and what they should look like in GCC.  I have concluded that unless they sort
out their internal inconsistencies, there is no hope of seeing progress
any time son.

-- Gaby

Re: gcc extensibility

[Basile Starynkevitch]

On Thu, 29 Mar 2012 19:41:07 +0100
Jonathan Wakely <jwakely.gcc@gmail.com> wrote:

> But plugins in GCC are still new and still evolving, if plugin authors
> won't help shape the API and offer their advice now then they the API
> will never be useful.
> 
> Don't expect people who don't care about plugins to do all the work of
> making writing plugins easier.

I actually agree with that. for the record, several PLUGIN events have been added at my
suggestion (notably those related to Ggc).

But I feel I don't wear the same hat as a plugin developer and as a GCC contributor.

Cheers.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: gcc extensibility

[Gabriel Dos Reis]

On Thu, Mar 29, 2012 at 1:31 PM, Niels Möller <nisse@lysator.liu.se> wrote:
> Gabriel Dos Reis <gdr@integrable-solutions.net> writes:
>
>> It is a false equality.  The needs of plugins authors are not necessarily
>> the same as the need of GCC development itself.
>
> I'm not so sure of that. To be concrete, can you give some examples of
> things that a plugin might want to do with an interface to the
> tree-abstraction in gcc, which gcc itself would never do? And vice
> versa?

The question isn't what GCC developers -can- do and plugins can't do
or vice-versa.  The question is *why* GCC -ought- to be doing it the plugins
way.

[...]
> Breaking source level compatibility ought to be avoided for minor
> releases. But the current situation, where, due to the name mangling, it
> seems difficult for a plugin to be compatible even with different
> configurations of the *same* minor release of gcc, seems a bit too
> inconvenient.

I have read assertions from plugins people  that is part of their work as plugin
developers.  Consequently, on this end I don't think it is problem.  If
plugin people think it is a problem my suggestion is: sort it out!

-- Gaby

Re: gcc extensibility

[Pedro Alves]

On 03/29/2012 05:52 PM, Basile Starynkevitch wrote:

> On Thu, 29 Mar 2012 11:42:30 -0500
> Gabriel Dos Reis <gdr@integrable-solutions.net> wrote:
>> I suspect that if plugins people want to make progress on this
>> recurring theme, they
>> will have to come up with a specification and an API.  Otherwise, they have only
>> themselves to blame if their plugins break from release to release.
> 
> 
> They blame nobody if their plugins break from one release to the next. They take this
> incompatibility of GCC as part of their plugins developer's work.
> 
> Again, a plugin writer by definition uses whatever interface is given to him.

IMO, the right way to approach this is instead:

#1 - I, a so called "plugin writer", have this use I could give to GCC,
 but it wouldn't make sense to include that code in the GCC sources/executable
 itself.  In fact, the maintainers would reject it, rightly.

#2 - However, if I could just add a little bit of glue interface to GCC
 that exposes just enough GCC internal bits that I could write my plugin
 against, in a way that is not invasive to the rest of the compiler,
 I know that would be accepted by the maintainers.  It's a compromise the
 GCC maintainers are willing to make.  They are aware that there's potential
 for other people to come up with other uses for the same minimal interfaces,
 so they accept this.  In the future, it's likely that other plugin authors
 will be satisfied by the interfaces I and other previous plugin authors have
 already added to GCC by then.

But, note it's clearly the plugin author that needs to write #2.  #1 too, obviously.  :-)

-- 
Pedro Alves

Re: gcc extensibility

[Diego Novillo]

On 3/29/12 3:01 PM, Basile Starynkevitch wrote:

> But I feel I don't wear the same hat as a plugin developer and as a GCC contributor.

Yes, you do.  You are both a GCC contributor and a plugin developer.  As 
such, you are in a unique position to know the needs of both sides.

Cleaning up the internal interfaces in the compiler may help plugin 
developers.  Perhaps one thing you could help design and implement is a 
plugin-specific API that can evolve independently of the internal APIs 
in the compiler.

Plugins that need pervasive access to compiler internals will need to 
evolve with it, and expect to need changes at every release of the 
compiler.  Others will be protected by the plugin-specific interfaces.

If we ever convert the compiler into a collection of self-contained 
libraries, then some of the things that today require plugins will be 
able to use these libraries directly.


Diego.

Re: gcc extensibility

[Romain Geissler]

Le 29 mars 2012 à 21:01, Gabriel Dos Reis a écrit :

> On Thu, Mar 29, 2012 at 12:39 PM, Romain Geissler
> <romain.geissler@gmail.com> wrote:
>> Le 29 mars 2012 à 18:06, Gabriel Dos Reis a écrit :
>> 
>>> On Thu, Mar 29, 2012 at 10:34 AM, Romain Geissler
>>> <romain.geissler@gmail.com> wrote:
>>>> Hi
>>>> 
>>>> Le 29 mars 2012 à 14:34, Niels Möller a écrit :
>>>> 
>>>>> 1. I imagine the plugin API ought to stay in plain C, right?
>>>> 
>>>> I don't know if this was already discussed and if the community
>>>> ended up with a clear answer for this question. If it's not the case
>>>> i would prefer a plugin interface in C++, for the same reasons it
>>>> was decided to slowly move the internals to C++.
>>>> 
>>> 
>>> I do not think people working on plugins have come up with a
>>> specification and an API they agree on.  Which makes any talk
>>> of restricting GCC's own evolution premature if not pointless.
>> 
>> I didn't mean the choice of C or C++ for the future plugin API may
>> in any way alter the own GCC evolution. The API only consists in
>> a bunch of stable wrappers to the unstable internals. Once such
>> an API exists, that won't be hard to update the impacted wrappers
>> to follow that changes, and thus it would have only minor impact on
>> the internals evolution.
>> 
> 
> From past discussions,  I gather that the plugins people
> want an uncompromising access to every bits of GCC internals (hence
> resist any notion of specification and API) and don't want to see evolution
> of GCC that might break their working plugins, for example using C++ because
> their own plugins are written in C.  Yet, we also receive lectures on modules
> and what they should look like in GCC.  I have concluded that unless they sort
> out their internal inconsistencies, there is no hope of seeing progress
> any time son.

I completely agree (for my own, I don't ask for full featured API, and
prioritize any internal enhancement over plugin API enhancement)

Re: gcc extensibility

[Basile Starynkevitch]

On Thu, 29 Mar 2012 15:22:28 -0400
Diego Novillo <dnovillo@google.com> wrote:

> On 3/29/12 3:01 PM, Basile Starynkevitch wrote:
> 
> > But I feel I don't wear the same hat as a plugin developer and as a GCC contributor.
> 
> Yes, you do.  You are both a GCC contributor and a plugin developer.  As 
> such, you are in a unique position to know the needs of both sides.

Well, I sort of agree, but I can't wear both hats *at the same time*. Either I feel as a
plugin developer, and then I struggle to make my plugin work with whatever versions of
GCC I have in mind, or I am a GCC contributor, and then I have to "forget" a bit my
plugin hat (of course keep "thinking" of my plugin needs). It is a bit like a kernel
developer who also develop a low level user-land application using his driver (imagine a
3D printer, or a computer-driven machining tool): when he develop the user-land thing
(imagine the 3D equivalent of CUPS, or a low-level CAD software for machining), he has to
aim some *existing* kernel version. Writing a plugin which would work only with an
experimental branch -and not with some GCC release, even a future one- is IMHO a complete
loss of time (and a huge but useless effort).

> Cleaning up the internal interfaces in the compiler may help plugin 
> developers.  Perhaps one thing you could help design and implement is a 
> plugin-specific API that can evolve independently of the internal APIs 
> in the compiler.

I am not sure to understand what you mean exactly. What is the plugin-specific API
(today, it is mostly the set of PLUGIN_* events, etc..). Or do you mean something else
which does not exist yet?

> 
> Plugins that need pervasive access to compiler internals will need to 
> evolve with it, and expect to need changes at every release of the 
> compiler. 
My feeling is that these "compiler internals" are actually the non-documented part of the
$(gcc -print-file-name=plugin/include) which today is the majority of the header files
there. Of course, I don't expect them to be stable. I do know they are not.
(And yes, sometimes these things bite and hurt).

> Others will be protected by the plugin-specific interfaces.
My feeling is that you think of the *documented* part of $(gcc
-print-file-name=plugin/include), that is the few things *documented* in 
http://gcc.gnu.org/onlinedocs/gccint/Plugins.html

Or perhaps I (Basile) am misunderstanding you (Diego) entirely. Sorry for that if it is
the case!


> 
> If we ever convert the compiler into a collection of self-contained 
> libraries, then some of the things that today require plugins will be 
> able to use these libraries directly.

I feel it almost the opposite way. When GCC becomes modular, it will be made of a
defined (and easily countable, i.e. a dozen or two) collection of libraries (each have to
be named! Names of modules [or libraries if you like this word] are important both to the
linker and to the developers (in particular the newbies). And a good picture of GCC would
be important too (recall the Gnome/GTK picture I mentioned many times).
The point is that I Basile sadly do not have the broad view and culture about GCC than
you (Diego, and other global reviewers) have. So I cannot even propose a set of module
names (or library names, if that hurts you less). Only people like you (Diego, and other
global reviewers) could propose -for discussion- a set of module (that is library) names.
I don't know GCC enough to think of one, sorry about that (I really mean it: I have
almost no idea of what the backends are made of, or of what the C++ or Ada or Go
frontends are made of).

Once GCC is defined by a set of modules (or libraries, for me a top-level module Foo is
implemented as a libfoo.so; exactly as in the Gnome world the Pango module is implemented
as libpango.so; so top-level modules are exactly implemented as shared-libraries), plugins
would have a well defined interface to it. Even, a plugin might replace exactly, or
supplement, an entire library (that would enable, for instance, providing a front-end or
a back-end as a plugin).

My feeling about a GCC modularity is that it would help *tremendously* if we have some
mechanical mean to describe -in details- that set of module. My feeling is that GTK could
teach us a lot about that (this does not mean that we should adopt GTK technology, just
that we should be inspired by GTK "philosophy" - but calling that a philosophy is an
insult towards the great philosophers like Socrate, Kant, etc...).

My dream -in particular because it would help MELT, and any language binding [Python,
Ruby, Ocaml, ...] to GCC- would be that GCC would have some meta-API, à la
http://live.gnome.org/GObjectIntrospection to query the set of modules and the large set
of API inside).

My other dream would be to have some plugins to help us (but sadly, I am not funded to
work on that; I would be delighted to be funded to use MELT for developping such MELT
extensions to help GCC) on that goal. It is quite sad that as a compiler community we use
so little compiler technology to help us work. There is a French wording about that: "les
coordonniers sont les plus mal chaussés", literally "shoe-repairers (or shoemakers) have
the worst shoes".


We (Diego & me) very probably agree a lot, but we very probably have linguistic or
cultural issues to understand each other. Sorry for my lack of fluency in English; it is
always hurting me.


Cheers.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: gcc extensibility

[Ludovic Courtès]

Hi,

Gabriel Dos Reis <gdr@integrable-solutions.net> skribis:

> I do not think people working on plugins have come up with a
> specification and an API they agree on.

I think it’s wrong to consider plug-ins as second-class citizens.

The plug-in mechanism is just a technical means to allow people to
extend the compiler without having to have their code checked in the
main tree.

Thanks,
Ludo’.

Re: gcc extensibility

[Richard Guenther]

On Fri, Mar 30, 2012 at 9:14 AM, Ludovic Courtès
<ludovic.courtes@inria.fr> wrote:
> Hi,
>
> Gabriel Dos Reis <gdr@integrable-solutions.net> skribis:
>
>> I do not think people working on plugins have come up with a
>> specification and an API they agree on.
>
> I think it’s wrong to consider plug-ins as second-class citizens.
>
> The plug-in mechanism is just a technical means to allow people to
> extend the compiler without having to have their code checked in the
> main tree.

No, that means would be using the old tool named 'patch' before building
GCC.

Richard.

> Thanks,
> Ludo’.
>

Re: gcc extensibility

[Bernd Schmidt]

On 03/30/2012 10:37 AM, Richard Guenther wrote:
> On Fri, Mar 30, 2012 at 9:14 AM, Ludovic Courtès
> <ludovic.courtes@inria.fr> wrote:
>> Hi,
>>
>> Gabriel Dos Reis <gdr@integrable-solutions.net> skribis:
>>
>>> I do not think people working on plugins have come up with a
>>> specification and an API they agree on.
>>
>> I think itÂ’s wrong to consider plug-ins as second-class citizens.
>>
>> The plug-in mechanism is just a technical means to allow people to
>> extend the compiler without having to have their code checked in the
>> main tree.
> 
> No, that means would be using the old tool named 'patch' before building
> GCC.

Or even the new tool named 'git'.


Bernd

Re: gcc extensibility

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 2:14 AM, Ludovic Courtès
<ludovic.courtes@inria.fr> wrote:
> Hi,
>
> Gabriel Dos Reis <gdr@integrable-solutions.net> skribis:
>
>> I do not think people working on plugins have come up with a
>> specification and an API they agree on.
>
> I think it’s wrong to consider plug-ins as second-class citizens.

Nobody has made that assertion that you are finding wrong so far.

> The plug-in mechanism is just a technical means to allow people to
> extend the compiler without having to have their code checked in the
> main tree.

That was understood.

-- Gaby

Re: gcc extensibility

[Miles Bader]

Bernd Schmidt <bernds@codesourcery.com> writes:
>> No, that means would be using the old tool named 'patch' before building
>> GCC.
>
> Or even the new tool named 'git'.

[... and note that "git apply" doesn't actually require a git repo,
and is a great (generally rather better behaved) replacement for the
"patch" command.]

-miles

-- 
Youth, n. The Period of Possibility, when Archimedes finds a fulcrum,
Cassandra has a following and seven cities compete for the honor of endowing a
living Homer.