Proposed plugin API for GCC

Proposed plugin API for GCC

[David Malcolm]

I had a go at writing a possible plugin API for GCC, and porting parts
of my python plugin to it:
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=36a0d6a45473c39db550915f8419a794f2f5653e

It's very much at the "crude early prototype" stage - all I've wrapped
is part of CFG-handling - but the important thing is that python plugin
*does* actually compile against this, and many of its selftests still
pass (though I'm breaking the self-imposed encapsulation in quite a few
places in order to get it to compile).

The code is currently jointly owned by me and Red Hat; I'm sure we can
do copyright assignment if/when it comes to that.

You can see the work-in-progress in the "proposed-plugin-api" branch of
gcc-python-plugin.

For example, the proposed public header file looks like this:
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=proposed-plugin-api/gcc-cfg.h;h=8dbeed0a6c5eb14b0336e89493746887c3bec20a;hb=36a0d6a45473c39db550915f8419a794f2f5653e

For example, some design notes can be seen at:
http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=proposed-plugin-api/design.rst;h=31b960ccac2dcf4d007701b5e9c6556e68e0d107;hb=36a0d6a45473c39db550915f8419a794f2f5653e

How do other plugin authors feel about the API?
How do GCC subsystem maintainers feel?

I initially attempted an underscore_based_naming_convention but quickly
found it difficult to get concise function names, so I switched to a
CamelCaseBased_NamingConvention with an underscore separating a notional
namespace element from a secondary element, which saved plenty of space.
The different naming convention also serves to highlight that this is
*not* part of GCC's internals.

Hope this is constructive
Dave

Re: Proposed plugin API for GCC

[Miles Bader]

David Malcolm <dmalcolm@redhat.com> writes:
> I initially attempted an underscore_based_naming_convention but quickly
> found it difficult to get concise function names, so I switched to a
> CamelCaseBased_NamingConvention with an underscore separating a notional
> namespace element from a secondary element, which saved plenty of space.

Just use the same names, but with underscore separated (lowercase) words
instead of StuDLyCapS; obviously they won't be significantly longer, but
will maintain gcc naming conventions, and will more readable.

-Miles

-- 
"Suppose we've chosen the wrong god. Every time we go to church we're
just making him madder and madder." -- Homer Simpson

Re: Proposed plugin API for GCC

[David Malcolm]

On Fri, 2012-03-30 at 09:05 +0900, Miles Bader wrote:
> David Malcolm <dmalcolm@redhat.com> writes:
> > I initially attempted an underscore_based_naming_convention but quickly
> > found it difficult to get concise function names, so I switched to a
> > CamelCaseBased_NamingConvention with an underscore separating a notional
> > namespace element from a secondary element, which saved plenty of space.
> 
> Just use the same names, but with underscore separated (lowercase) words
> instead of StuDLyCapS; obviously they won't be significantly longer, but
> will maintain gcc naming conventions, and will more readable.
Beauty is in the eye of the beholder, I guess.  I greatly prefer
CamelCase, but it's clear that I'm in the minority here.

I'll assume that it can be fixed using sed at some point; how we spell
the API is a minor surface detail compared to much deeper issues like
lifetimes/ownership of objects, the scope of the API, stability
guarantees or lack thereof etc

Re: Proposed plugin API for GCC

[Ian Lance Taylor]

David Malcolm <dmalcolm@redhat.com> writes:

> I had a go at writing a possible plugin API for GCC, and porting parts
> of my python plugin to it:
> http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=36a0d6a45473c39db550915f8419a794f2f5653e

Seems like a good start.

> I initially attempted an underscore_based_naming_convention but quickly
> found it difficult to get concise function names, so I switched to a
> CamelCaseBased_NamingConvention with an underscore separating a notional
> namespace element from a secondary element, which saved plenty of space.
> The different naming convention also serves to highlight that this is
> *not* part of GCC's internals.

Predictably, I don't care for the names.

I would recommend grouping functions by category, and making each
category be a struct with a set of function pointers.  That will give
you a namespace, and will greatly reduce the number of external names in
the API.

Ian

Re: Proposed plugin API for GCC

[Romain Geissler]

Hi

Le 30 mars 2012 à 06:18, Ian Lance Taylor a écrit :

> I would recommend grouping functions by category, and making each
> category be a struct with a set of function pointers.  That will give
> you a namespace, and will greatly reduce the number of external names in
> the API.
> 
> Ian

Using structs with some sets of function pointers may break compatibility
between minor release.

Imagine i've got the following struct publicly exported in 4.7.1 :

struct GCC_plugin_tree_functions{
	GCC_plugin_tree_code (*get_code)(GCC_plugin_tree tree);
	bool (*is_used)(GCC_plugin_tree tree);
}

Now some plugin writer needs to know if a tree is a constant. We add it in 4.7.2 :

struct GCC_plugin_tree_functions{
	GCC_plugin_tree_code (*get_code)(GCC_plugin_tree tree);
	bool (*is_constant)(GCC_plugin_tree tree);
	bool (*is_used)(GCC_plugin_tree tree);
}

We insert is_constant between get_code and is_used to reflect the actual
flag order defined in tree_base. But if we proceed that way, a plugin will
have to be rebuild with every gcc release, even if the plugin API is fully
backward compatible (ie we only added new feature without changing the
old ones).

Anyway, you're suggestion to group functions in common names, that's just
C++ motto. May the eventual plugin API in C++ (independently from internals
being C++ or not) ?

Romain Geissler

Re: Proposed plugin API for GCC

[Ian Lance Taylor]

Romain Geissler <romain.geissler@gmail.com> writes:

> Using structs with some sets of function pointers may break compatibility
> between minor release.

Yes, but fortunately we have a good understanding of how not to do that.

We could also go the even safer route used for linker plugins, in which
the plugin is invoked with a list of functions, where each function is
tagged with a code.  See include/plugin-api.h for the interface and
lto-plugin for an implementation.  The approach there is very clean and
permits forward and backward binary compatibility.  I don't know if we
want to go that far for compiler plugins.


> Anyway, you're suggestion to group functions in common names, that's just
> C++ motto. May the eventual plugin API in C++ (independently from internals
> being C++ or not) ?

I think we have a clear understanding of how to maintain compatibility
across releases in C.  I do not think we have that understanding in C++.

Ian

Re: Proposed plugin API for GCC

[Romain Geissler]

Le 30 mars 2012 à 15:48, Ian Lance Taylor a écrit :

> Romain Geissler <romain.geissler@gmail.com> writes:
> 
>> Using structs with some sets of function pointers may break compatibility
>> between minor release.
> 
> Yes, but fortunately we have a good understanding of how not to do that.
> 
> We could also go the even safer route used for linker plugins, in which
> the plugin is invoked with a list of functions, where each function is
> tagged with a code.  See include/plugin-api.h for the interface and
> lto-plugin for an implementation.  The approach there is very clean and
> permits forward and backward binary compatibility.  I don't know if we
> want to go that far for compiler plugins.
> 
> 
>> Anyway, you're suggestion to group functions in common names, that's just
>> C++ motto. May the eventual plugin API in C++ (independently from internals
>> being C++ or not) ?
> 
> I think we have a clear understanding of how to maintain compatibility
> across releases in C.  I do not think we have that understanding in C++.
> 
> Ian

Ok thank you, i didn't know about that. I'll take a look to the lto-plugin.

Re: Proposed plugin API for GCC

[Ludovic Courtès]

Hi,

David Malcolm <dmalcolm@redhat.com> skribis:

> How do other plugin authors feel about the API?

I think this approach would lead to a duplication of each GCC API.

The needs of plug-ins cannot be anticipated; artificially restricting
what plug-ins can do is likely to hinder wider extension of GCC.

As an example, when Emacs was written, probably nobody expected that
email and IRC clients would be written in it; likewise, probably most of
the projects at <http://llvm.org/ProjectsWithLLVM/> were not anticipated.


What about sticking to the current “API” instead, and explicitly marking
as internal those parts that core developers know are still in flux?

For instance, I would expect a large subset of <tree.h> and <cgraph.h>
to be stable (it’s been the case in my experience between 4.5 and 4.7.)
The rest can be tagged with a special convention (for instance, an ‘i_’
prefix), to make it clear that it’s only meant for internal consumption.

WDYT?

Thanks,
Ludo’.

Re: Proposed plugin API for GCC

[Richard Guenther]

On Fri, Mar 30, 2012 at 10:23 AM, Ludovic Courtès
<ludovic.courtes@inria.fr> wrote:
> Hi,
>
> David Malcolm <dmalcolm@redhat.com> skribis:
>
>> How do other plugin authors feel about the API?
>
> I think this approach would lead to a duplication of each GCC API.

I would call it an abstraction of GCC internals (disclaimer: I did not look
at the proposed API).  This abstraction should be easier to learn and
easier to use for 99% of the plugin users.  And it should offer a stable
abstraction ABI that makes plugins interoperate with different GCC versions
without recompiling.

> The needs of plug-ins cannot be anticipated; artificially restricting
> what plug-ins can do is likely to hinder wider extension of GCC.

Extension of GCC should happen within the GCC codebase.  Plugins
are not a replacement of improving GCC!

> As an example, when Emacs was written, probably nobody expected that
> email and IRC clients would be written in it; likewise, probably most of
> the projects at <http://llvm.org/ProjectsWithLLVM/> were not anticipated.
>
>
> What about sticking to the current “API” instead, and explicitly marking
> as internal those parts that core developers know are still in flux?

That's not possible.  And it is not wanted.

> For instance, I would expect a large subset of <tree.h> and <cgraph.h>
> to be stable (it’s been the case in my experience between 4.5 and 4.7.)
> The rest can be tagged with a special convention (for instance, an ‘i_’
> prefix), to make it clear that it’s only meant for internal consumption.
>
> WDYT?

Sounds like a stupid idea that does not work (yes, plugins that want to
do everything an embedded part of GCC can do are stupid).

Richard.

> Thanks,
> Ludo’.
>

Re: Proposed plugin API for GCC

[Ludovic Courtès]

Hi Richard,

Richard Guenther <richard.guenther@gmail.com> skribis:

> On Fri, Mar 30, 2012 at 10:23 AM, Ludovic Courtès

[...]

>> The needs of plug-ins cannot be anticipated; artificially restricting
>> what plug-ins can do is likely to hinder wider extension of GCC.
>
> Extension of GCC should happen within the GCC codebase.  Plugins
> are not a replacement of improving GCC!

Yes, I agree.

However, I’m sure that GCC can be extended in ways that are very
valuable, yet that would not fit in GCC itself for various
administrative or technical reasons.

I find it important to help such unanticipated uses of GCC spread.

>> For instance, I would expect a large subset of <tree.h> and <cgraph.h>
>> to be stable (it’s been the case in my experience between 4.5 and 4.7.)
>> The rest can be tagged with a special convention (for instance, an ‘i_’
>> prefix), to make it clear that it’s only meant for internal consumption.
>>
>> WDYT?
>
> Sounds like a stupid idea that does not work

The current situation is that nothing, or everything, is considered
internal, depending on who you ask.  ;-)

The above suggestion would be a recognition that yes, plug-ins /do/ need
<tree.h> & co. to do anything meaningful, but at the same time that
parts of it are internal and /will/ break eventually.

Thanks,
Ludo’.

Re: Proposed plugin API for GCC

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 4:45 AM, Ludovic Courtès
<ludovic.courtes@inria.fr> wrote:

> I find it important to help such unanticipated uses of GCC spread.

It is hard to design for things you do not know.  How about starting
with things you do know?  A possible path to progress on this issue, no?

-- Gaby

Re: Proposed plugin API for GCC

[Ludovic Courtès]

Gabriel Dos Reis <gdr@integrable-solutions.net> skribis:

> On Fri, Mar 30, 2012 at 4:45 AM, Ludovic Courtès
> <ludovic.courtes@inria.fr> wrote:
>
>> I find it important to help such unanticipated uses of GCC spread.
>
> It is hard to design for things you do not know.

Indeed, that’s the whole point.  Offer a ten-function stable API and
watch “plug-in people”, as you put it, do compiler work on top of LLVM.
As simple as this.

Ludo’.

Re: Proposed plugin API for GCC

[Basile Starynkevitch]

On Fri, Mar 30, 2012 at 10:23:16AM +0200, Ludovic Courtès wrote:
> Hi,
> 
> David Malcolm <dmalcolm@redhat.com> skribis:
> 
> > How do other plugin authors feel about the API?
> 
> I think this approach would lead to a duplication of each GCC API.
> 
> The needs of plug-ins cannot be anticipated; artificially restricting
> what plug-ins can do is likely to hinder wider extension of GCC.
> 
> As an example, when Emacs was written, probably nobody expected that
> email and IRC clients would be written in it; likewise, probably most of
> the projects at <http://llvm.org/ProjectsWithLLVM/> were not anticipated.


I entirely agree with the above statements. More generally, we cannot
predict what plugins will be written for GCC, and what part of the API they
will use, and for what purposes they will be written. I guess than most
plugins would provide something which would not have its place inside GCC.
(I also guess that some plugins might be written to experiment passes which
could go inside GCC, but which won't be proposed quickly into GCC, in
particular because having a patch being reviewed and accepted, when that
patch comes from a newbie, is very time consuming).


> 
> What about sticking to the current ???API??? instead, and explicitly marking
> as internal those parts that core developers know are still in flux?

I tend to like that, but my feeling is that there might not be a consensus
on what the current API is, and on what the API used by plugins should be.

(I have the impression that people discussing that on gcc@ list tend to
attach slightly different meanings to these terms)

 
> For instance, I would expect a large subset of <tree.h> and <cgraph.h>
> to be stable (it???s been the case in my experience between 4.5 and 4.7.)
> The rest can be tagged with a special convention (for instance, an ???i_???
> prefix), to make it clear that it???s only meant for internal consumption.

I like that approach. I also feel that the headers should contain structured
comments from which some (imperfect) documentation of (an imperfect API)
could be produced.

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Proposed plugin API for GCC

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 4:33 AM, Basile Starynkevitch
<basile@starynkevitch.net> wrote:

> I entirely agree with the above statements. More generally, we cannot
> predict what plugins will be written for GCC, and what part of the API they
> will use, and for what purposes they will be written.

I believe somethings is being lost here:

   plugins people: we don't want to define an API.
   GCC maintainers: you have to define an API or you're doomed to
                                be stuck forever.

Loop forever.

-- Gaby

Re: Proposed plugin API for GCC

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 3:23 AM, Ludovic Courtès
<ludovic.courtes@inria.fr> wrote:

> What about sticking to the current “API” instead, and explicitly marking
> as internal those parts that core developers know are still in flux?

A guarantee of perpetual discussions of the same topic, over and over.
(meh, GCC-x.y.z just broke my plugins; this is unacceptable, blah blah bah.)

-- Gaby

Re: Proposed plugin API for GCC

[Richard Guenther]

On Fri, Mar 30, 2012 at 1:54 PM, Gabriel Dos Reis
<gdr@integrable-solutions.net> wrote:
> On Fri, Mar 30, 2012 at 3:23 AM, Ludovic Courtès
> <ludovic.courtes@inria.fr> wrote:
>
>> What about sticking to the current “API” instead, and explicitly marking
>> as internal those parts that core developers know are still in flux?
>
> A guarantee of perpetual discussions of the same topic, over and over.
> (meh, GCC-x.y.z just broke my plugins; this is unacceptable, blah blah bah.)

Yeah.  Btw, the alternative to a stable plugin API is embedded support of
a scripting language (python, guile, etc.).  That's of course just another
supposed-to-be stable "plugin API".  Both are sufficient for introspection
and instrumentation tasks (a good test if the API is powerful enough is
to implement mudflap with it).

Richard.

Re: Proposed plugin API for GCC

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 6:59 AM, Richard Guenther
<richard.guenther@gmail.com> wrote:
> On Fri, Mar 30, 2012 at 1:54 PM, Gabriel Dos Reis
> <gdr@integrable-solutions.net> wrote:
>> On Fri, Mar 30, 2012 at 3:23 AM, Ludovic Courtès
>> <ludovic.courtes@inria.fr> wrote:
>>
>>> What about sticking to the current “API” instead, and explicitly marking
>>> as internal those parts that core developers know are still in flux?
>>
>> A guarantee of perpetual discussions of the same topic, over and over.
>> (meh, GCC-x.y.z just broke my plugins; this is unacceptable, blah blah bah.)
>
> Yeah.  Btw, the alternative to a stable plugin API is embedded support of
> a scripting language (python, guile, etc.).  That's of course just another
> supposed-to-be stable "plugin API".  Both are sufficient for introspection
> and instrumentation tasks (a good test if the API is powerful enough is
> to implement mudflap with it).

Exactly right.  I had some hope that the plugins people would coalesce behind
David M.'s proposal as a starting point, but now I don't know if the plugin
case has any merit anymore :-/

-- Gaby

Re: Proposed plugin API for GCC

[Ian Lance Taylor]

ludovic.courtes@inria.fr (Ludovic Courtès) writes:

> What about sticking to the current “API” instead, and explicitly marking
> as internal those parts that core developers know are still in flux?
>
> For instance, I would expect a large subset of <tree.h> and <cgraph.h>
> to be stable (it’s been the case in my experience between 4.5 and 4.7.)
> The rest can be tagged with a special convention (for instance, an ‘i_’
> prefix), to make it clear that it’s only meant for internal consumption.

This is an interesting goal but it is essentially impossible to
implement.  In both theory and practice, it's all in flux.  GCC has been
around for 25 years.  Everything has changed in that time, and most
things have changed multiple times.

We do not want plugins to constrain internal development.  So if we
adopted your approach, the only honest option would be to mark
everything as internal.  And that would leave us where we are today.

Ian

Re: Proposed plugin API for GCC

[Ludovic Courtès]

Ian Lance Taylor <iant@google.com> skribis:

> We do not want plugins to constrain internal development.  So if we
> adopted your approach, the only honest option would be to mark
> everything as internal.  And that would leave us where we are today.

My (limited) experience suggests that things aren’t that bad between 4.5
and 4.7, at least for the tree, cgraph, and gimple subsets that my
plug-in uses.

My feeling is that there aren’t so many ways to design the core of these
three APIs, anyway.  But surely your hindsight could help understand and
anticipate API changes.

Thanks,
Ludo’.

Re: Proposed plugin API for GCC

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 9:48 AM, Ludovic Courtès
<ludovic.courtes@inria.fr> wrote:
> Ian Lance Taylor <iant@google.com> skribis:
>
>> We do not want plugins to constrain internal development.  So if we
>> adopted your approach, the only honest option would be to mark
>> everything as internal.  And that would leave us where we are today.
>
> My (limited) experience suggests that things aren’t that bad between 4.5
> and 4.7, at least for the tree, cgraph, and gimple subsets that my
> plug-in uses.

Yes, but the time span between 4.5 and 4.7 is relatively small compared
to, say the EGCS days, or even the 3.x.y days.  A lot has changed.  I do
actually expect a lot to change in the 4.8-4.10 time frame once we
agreed on rules governing C++ usage in GCC -- I can tell you that we
have several projects in the pipeline.

-- Gaby

Re: Proposed plugin API for GCC

[Richard Guenther]

On Thu, Mar 29, 2012 at 10:58 PM, David Malcolm <dmalcolm@redhat.com> wrote:
> I had a go at writing a possible plugin API for GCC, and porting parts
> of my python plugin to it:
> http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=36a0d6a45473c39db550915f8419a794f2f5653e
>
> It's very much at the "crude early prototype" stage - all I've wrapped
> is part of CFG-handling - but the important thing is that python plugin
> *does* actually compile against this, and many of its selftests still
> pass (though I'm breaking the self-imposed encapsulation in quite a few
> places in order to get it to compile).
>
> The code is currently jointly owned by me and Red Hat; I'm sure we can
> do copyright assignment if/when it comes to that.
>
> You can see the work-in-progress in the "proposed-plugin-api" branch of
> gcc-python-plugin.
>
> For example, the proposed public header file looks like this:
> http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=proposed-plugin-api/gcc-cfg.h;h=8dbeed0a6c5eb14b0336e89493746887c3bec20a;hb=36a0d6a45473c39db550915f8419a794f2f5653e
>
> For example, some design notes can be seen at:
> http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=proposed-plugin-api/design.rst;h=31b960ccac2dcf4d007701b5e9c6556e68e0d107;hb=36a0d6a45473c39db550915f8419a794f2f5653e
>
> How do other plugin authors feel about the API?

Of course I don't like the CamelCasing ;)

The important part of the API is that it exposes no data structures and
all object references we expose are opaque.

You have mostly wrapped in an iterator-style way, I suppose that's fine.
That (and the CamelCasing) exposes that using a different language for
this API would be desired; use namespaces to avoid CamelCasing and
STL style iterators / functors for the rest.  If you stay with C rather
than using callbacks I would use an explicit iterator representation,
similar to how we have that now with gimple_stmt_iterator.  Of course
that's personal preference, subject to bikeshedding.

Would the python plugin have any issue with using C++?

> How do GCC subsystem maintainers feel?

Well, positive!  Thanks for starting.  CFG introspection is indeed one
of the most important parts, then callgraph and statement introspection.

> I initially attempted an underscore_based_naming_convention but quickly
> found it difficult to get concise function names, so I switched to a
> CamelCaseBased_NamingConvention with an underscore separating a notional
> namespace element from a secondary element, which saved plenty of space.
> The different naming convention also serves to highlight that this is
> *not* part of GCC's internals.
>
> Hope this is constructive

Indeed, and thank you for that.

Btw, how ugly is it to make this API grokable by swig?  Would that serve
the python plugin?

Thanks,
Richard.

Re: Proposed plugin API for GCC

[Basile Starynkevitch]

On Fri, Mar 30, 2012 at 02:14:31PM +0200, Richard Guenther wrote:
> 
> Btw, how ugly is it to make this API grokable by swig?  Would that serve
> the python plugin?


An alternative would be to have either some easily parsable API definition
(wwhich might be sort-of offered by Swig, but I'll bet that not in practice:
we'll need to use weird Swig tricks), or some way of querying at runtime that API.

The important part in my view is that such an API should not be targeted to
Python only. It should be usable by plugins coded in C++ (or in MELT), or in
other languages.

Again, the GTK guys did a good work with their Gobject introspection layer,
which is a meta-API providing that.

The point is that GCC will stay complex, and any API will by necessity be
huge. We have to know that and to ease it uses (e.g. to facilitate the
embedding of several scripting languages, not only Python).

So it really would help if the API is documented and mechanically queryable.

A traditional manual glue code is not enough.

(And there might be memory management issue; we have to specify very well
when a GCC data should be released, and by whom. I feel that Ggc is part of
the solution).

Cheers.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Proposed plugin API for GCC

[David Malcolm]

On Fri, 2012-03-30 at 15:08 +0200, Basile Starynkevitch wrote:
> On Fri, Mar 30, 2012 at 02:14:31PM +0200, Richard Guenther wrote:
> > 
> > Btw, how ugly is it to make this API grokable by swig?  Would that serve
> > the python plugin?
> 
> 
> An alternative would be to have either some easily parsable API definition
> (wwhich might be sort-of offered by Swig, but I'll bet that not in practice:
> we'll need to use weird Swig tricks), or some way of querying at runtime that API.
> 
> The important part in my view is that such an API should not be targeted to
> Python only. It should be usable by plugins coded in C++ (or in MELT), or in
> other languages.
> 
> Again, the GTK guys did a good work with their Gobject introspection layer,
> which is a meta-API providing that.

[I briefly worked on the Python 3 backend for GObject introspection,
fwiw]

Here's another proposal then: actually use GObject introspection -
provide a GObject-based API to GCC.

In this approach, GCC would gain a dependency on glib and gobject, and
expose its API via a .gir file.

Thanks to the existing backends, the API would then be immediately
available to Guile, Python 2 and 3, JavaScript, Ruby, etc (and my plugin
either needs a total rewrite, or becomes redundant, disappearing in a
libffi haze...).

For more information, see:
  https://live.gnome.org/GObjectIntrospection

This may be overkill, and the above has the warning "Note: GObject
Introspection is still in development" (I've spent far too long
debugging libffi crashes - due to bogus type signatures - in other
projects to be keen on it, fwiw).

> The point is that GCC will stay complex, and any API will by necessity be
> huge. We have to know that and to ease it uses (e.g. to facilitate the
> embedding of several scripting languages, not only Python).
> 
> So it really would help if the API is documented and mechanically queryable.
> 
> A traditional manual glue code is not enough.
> 
> (And there might be memory management issue; we have to specify very well
> when a GCC data should be released, and by whom. I feel that Ggc is part of
> the solution).
Object lifetimes could be a nuisance: GObjects are reference-counted
iirc.  I don't know if that would be a problem for MELT.

Dave

Re: Proposed plugin API for GCC

[Gabriel Dos Reis]

On Fri, Mar 30, 2012 at 7:45 PM, David Malcolm <dmalcolm@redhat.com> wrote:

> Here's another proposal then: actually use GObject introspection -
> provide a GObject-based API to GCC.
>
> In this approach, GCC would gain a dependency on glib and gobject, and
> expose its API via a .gir file.

I greatly prefer the other alternative, by far.

-- Gaby

Re: Proposed plugin API for GCC

[Michael Matz]

Hi,

On Fri, 30 Mar 2012, Gabriel Dos Reis wrote:

> On Fri, Mar 30, 2012 at 7:45 PM, David Malcolm <dmalcolm@redhat.com> wrote:
> 
> > Here's another proposal then: actually use GObject introspection -
> > provide a GObject-based API to GCC.
> >
> > In this approach, GCC would gain a dependency on glib and gobject, and
> > expose its API via a .gir file.
> 
> I greatly prefer the other alternative, by far.

By far, by far, by far!  I can't repeat it enough :)  glibc/gobject in 
cc1, *shudder*.

I think a clean C only API would be the best, and if only because a 
multitude of wrapper generators exist that work with that.  If you do away 
with the strange Gcc prefix and the 'I' suffix (Interface?) and use some 
common abbreviations (cfgblock -> bb, cfgedge -> edge, cgraph -> cg) it 
wouldn't be even too unwieldy.


Ciao,
Michael.

Re: Proposed plugin API for GCC

[Romain Geissler]

Hi

Le 31 mars 2012 à 02:45, David Malcolm a écrit :

> Here's another proposal then: actually use GObject introspection -
> provide a GObject-based API to GCC.
> 
> In this approach, GCC would gain a dependency on glib and gobject, and
> expose its API via a .gir file.

I don't think adding huge dependencies only for plugins is welcomed.
A C/C++ API is far enough and quite simple to use. People that are willing
to use Gobject or anyhting else are free to wrap the wrappers.

By the way, your proposed API is promising (through i also love
CamelCasing and i'd prefered a C++ API).

Cheers

Romain Geissler

Re: Proposed plugin API for GCC

[David Malcolm]

On Fri, 2012-03-30 at 14:14 +0200, Richard Guenther wrote:
> On Thu, Mar 29, 2012 at 10:58 PM, David Malcolm <dmalcolm@redhat.com> wrote:
> > I had a go at writing a possible plugin API for GCC, and porting parts
> > of my python plugin to it:
> > http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=36a0d6a45473c39db550915f8419a794f2f5653e
> >
> > It's very much at the "crude early prototype" stage - all I've wrapped
> > is part of CFG-handling - but the important thing is that python plugin
> > *does* actually compile against this, and many of its selftests still
> > pass (though I'm breaking the self-imposed encapsulation in quite a few
> > places in order to get it to compile).
> >
> > The code is currently jointly owned by me and Red Hat; I'm sure we can
> > do copyright assignment if/when it comes to that.
> >
> > You can see the work-in-progress in the "proposed-plugin-api" branch of
> > gcc-python-plugin.
> >
> > For example, the proposed public header file looks like this:
> > http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=proposed-plugin-api/gcc-cfg.h;h=8dbeed0a6c5eb14b0336e89493746887c3bec20a;hb=36a0d6a45473c39db550915f8419a794f2f5653e
> >
> > For example, some design notes can be seen at:
> > http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=proposed-plugin-api/design.rst;h=31b960ccac2dcf4d007701b5e9c6556e68e0d107;hb=36a0d6a45473c39db550915f8419a794f2f5653e
> >
> > How do other plugin authors feel about the API?
> 
> Of course I don't like the CamelCasing ;)
Seems like I'm the only person who does around here :)

> The important part of the API is that it exposes no data structures and
> all object references we expose are opaque.
Right.  There's a slight leak in gcc-semiprivate-types.h, but it's all
clearly marked as private (I've seen information hiding where the inner
pointers are all (void*) which sucks when you have to try and debug the
resulting code: it's bad if the information-hiding approach manages to
also hide things from the debugger!)

> You have mostly wrapped in an iterator-style way, I suppose that's fine.
> That (and the CamelCasing) exposes that using a different language for
> this API would be desired; use namespaces to avoid CamelCasing and
> STL style iterators / functors for the rest.  If you stay with C rather
> than using callbacks I would use an explicit iterator representation,
> similar to how we have that now with gimple_stmt_iterator.  Of course
> that's personal preference, subject to bikeshedding.
That raises fun issues like: can the thing be changed during an
iteration?

> Would the python plugin have any issue with using C++?
I have a dislike for it, and will involve a little extra coding, but if
C++ is what it will take to get me a stable API that I can code to
(ideally with some kind of ABI guarantee), that seems a price worth
paying.

I don't know how other plugin authors feel (and I'm keen on hearing if
they think my proposed API could work for them - modulo CamelCase, of
course!).

It's also possible to create a C wrapping around a C++ API and all kinds
of other shenanigans (my Python code is already a wrapper around a
wrapper, so yay, another level of indirection...)

> > How do GCC subsystem maintainers feel?
> 
> Well, positive!  Thanks for starting.  CFG introspection is indeed one
> of the most important parts, then callgraph and statement introspection.
> 
> > I initially attempted an underscore_based_naming_convention but quickly
> > found it difficult to get concise function names, so I switched to a
> > CamelCaseBased_NamingConvention with an underscore separating a notional
> > namespace element from a secondary element, which saved plenty of space.
> > The different naming convention also serves to highlight that this is
> > *not* part of GCC's internals.
> >
> > Hope this is constructive
> 
> Indeed, and thank you for that.
> 
> Btw, how ugly is it to make this API grokable by swig?  Would that serve
> the python plugin?
Why SWIG btw?   I happen to have an intense loathing of SWIG (sorry SWIG
maintainers): in my experience it gives ugly APIs that suffer from
memory leaks, and you end up having to do pointer management in Python,
without the typesafety that a C compiler would give you.  FWIW, Cython
is my wrapper-generator tool of choice, though that's for Python only.
I may be biased, of course!.

Would it be better to work from a higher-level representation of the API
(e.g. XML, or some microformat?) and autogenerate the headers and source
(and docs)?  For the first iteration I wanted to keep things simple,
hence I directly wrote code, rather than code-that-writes-code.

Dave