new plugin events

new plugin events

[Basile STARYNKEVITCH]

I am changing the subject to new plugin events. It used to be: PLUGIN_PRAGMAS event

The discussion has nothing to do with pragma inside plugins anymore.

Diego Novillo wrote:
> On Fri, Nov 6, 2009 at 12:56, Basile STARYNKEVITCH
> <basile@starynkevitch.net> wrote:
> 
>> What is the criteria for adding them or for rejecting them as theoretical?
> 
> That you have a specific plugin making concrete use of them.  A
> theoretical event would be one that is only added just in case a
> future plugin may have a use for it.
> 
> I don't have an internal plugin branch.  I am working off of a 4.4
> compiler with backported plugin support.  The events I need to add are
> all for dehydra's use.  I need to add an event for when the parser
> finishes recognizing a DECL (the PR I pointed to earlier) and also I
> need some pre-processor events for allowing dehydra to expose things
> like #define and #include to user javascript code.

I fully agree and I probably will use myself these events within 2 or 3 years,
but then the ICI effort seems to be exactly in the same position.
[They have a branch somewhere demonstration all their needs - this is code, not theory]

But there is a chichen & egg issue still here. So, to add a new event, people (outside of the first circle of GCC like 
Diego Novillo & Richard Guenther & perhaps Rafael Espindola, that is outside of the "reviewer" set) need to first code 
their plugin and patch their GCC trunk, and then submit a patch.

I repeat that adding several plugin events (I mean extending the enum plugin_event in gcc-plugins.h) is very inocolous, 
provided the event is not appearing in some call to invoke_plugin_callbacks which is executed many times.

I also am convinced that GCC need more plugins, it should not be afraid of them (GCC should be afraid of competitor 
compilers, not of plugins). Hence the GCC community should help the ICI community to make ICI a big plugin itself for 
the next gcc-4.5. And that help means accepting new events, if the ICI people need them. Of course, ICI should quickly 
define the set of events they need. But they should first push their new events into the GCC trunk, and then patch their 
existing ICI code to use them.

I really insist that the position "events are only accepted if there exist a mature & running plugin needing them" is 
not sustainable.

And I also insist that having 4.5 with a bigger set of plugins is important. I mean 4.5, not 4.6. The 4.5 compiler will 
still be used a lot in 2013!

I don't understand why adding new plugin events is so difficult for people outside the first circle.
The first circle should trust a bit more outsiders wanting to code plugins.

If some ICI guy says I need this and this event, we should listen & discuss his patch!
[Same goes for me]


Regards.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: new plugin events

[Diego Novillo]

On Fri, Nov 6, 2009 at 14:00, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:

> I really insist that the position "events are only accepted if there exist a
> mature & running plugin needing them" is not sustainable.

No.  We are not communicating here.  The relative maturity of the
plugin that needs it is not relevant.  Just that there is some plugin
that could take advantage of it.

If ICI and/or MELT and/or Dehydra and/or whatever other plugin people
are developing need new plugin events, then by all means lets add
them!

Let's not make a mountain out of a molehill.


Diego.

Re: new plugin events

[Basile STARYNKEVITCH]

Diego Novillo wrote:
> On Fri, Nov 6, 2009 at 14:00, Basile STARYNKEVITCH
> <basile@starynkevitch.net> wrote:
> 
>> I really insist that the position "events are only accepted if there exist a
>> mature & running plugin needing them" is not sustainable.
> 
> No.  We are not communicating here.  The relative maturity of the
> plugin that needs it is not relevant.  Just that there is some plugin
> that could take advantage of it.
> 
> If ICI and/or MELT and/or Dehydra and/or whatever other plugin people
> are developing need new plugin events, then by all means lets add
> them!


Then I believe that me Basile & Diego have the same position, while Richard Guenther did express an incompatibly 
different position in the past (or perhaps I Basile misunderstood what he Richard meant by theory).

So I understood that people could propose new plugin events, provided that they are not too costly to implement (mostly 
meaning that the invoke_register_plugins calling them does not slow down compilations not using plugins), and provided 
people can explain clearly why they need it. It is not necessary to have a useful plugin coded & runnable to show the 
need of the event.

Thanks for the clarification.

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: new plugin events

[Richard Guenther]

On Fri, Nov 6, 2009 at 8:28 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Diego Novillo wrote:
>>
>> On Fri, Nov 6, 2009 at 14:00, Basile STARYNKEVITCH
>> <basile@starynkevitch.net> wrote:
>>
>>> I really insist that the position "events are only accepted if there
>>> exist a
>>> mature & running plugin needing them" is not sustainable.
>>
>> No.  We are not communicating here.  The relative maturity of the
>> plugin that needs it is not relevant.  Just that there is some plugin
>> that could take advantage of it.
>>
>> If ICI and/or MELT and/or Dehydra and/or whatever other plugin people
>> are developing need new plugin events, then by all means lets add
>> them!
>
>
> Then I believe that me Basile & Diego have the same position, while Richard
> Guenther did express an incompatibly different position in the past (or
> perhaps I Basile misunderstood what he Richard meant by theory).

No, read what Diego said (emphasis mine): "Just that there _is_ some
plugin that could take advantage of it."

As opposed to "There _might_ be some plugin that could take advantage
of it."

I want to avoid the situation where we prematurely add a plugin hook
and later recognize it should have been different in a sublte way once
the first user appears.  Thus, I want to avoid 1) changing existing
plugin callbacks, 2) removing existing plugin callbacks, or even worst
3) changing its semantics.  This all applies to between major and
even more so minor GCC releases.

If we don't have at least plugin API stability we could throw plugins
down the toilet.

I especially want to avoid adding plugin callbacks in code that is
known to be not the way it should be.  Instead GCC internals should
be cleaned up and adjusted to be plugin-friendly.  This probably
will apply to some of the pass-manager hooks people are going to
propose.

> So I understood that people could propose new plugin events, provided that
> they are not too costly to implement (mostly meaning that the
> invoke_register_plugins calling them does not slow down compilations not
> using plugins), and provided people can explain clearly why they need it. It
> is not necessary to have a useful plugin coded & runnable to show the need
> of the event.

Yes it is.  And I am going to voice this opinion whenever I see people
are adding hooks because it is fun to do so.

Richard.

Re: new plugin events

[Basile STARYNKEVITCH]

Richard Guenther wrote:
> 
> No, read what Diego said (emphasis mine): "Just that there _is_ some
> plugin that could take advantage of it."
> 

Diego also said "could", not "can" (emphasis mine). I understood that the plugin is not supposed to be mature. It could 
even be in its design phase...

> As opposed to "There _might_ be some plugin that could take advantage
> of it."

The intermediate phrase is "There _will_ be some plugin...". I am mostly thinking of the brave guys wanting to make a 
plugin for GCC, and missing some particular event.

> 
> I want to avoid the situation where we prematurely add a plugin hook
> and later recognize it should have been different in a sublte way once
> the first user appears.  Thus, I want to avoid 1) changing existing
> plugin callbacks, 2) removing existing plugin callbacks, or even worst
> 3) changing its semantics.  This all applies to between major and
> even more so minor GCC releases.

I agree with the list to avoid, but it is really premature today for 4.5. It will be a concern for 4.6 or 5.0!! And it 
probably won't be that much a concern. I would imagine that removing useless plugin events in 4.6 or 5.0 would be 
possible...

> 
> If we don't have at least plugin API stability we could throw plugins
> down the toilet.

Could you define plugin API stability? For me it is premature to think of it.
First, the Plugin API is so new that it cannot be stable for 4.5. It could be stable if it existed before 4.5 which it 
didn't.

Now, if you think of stability of a given plugin -call it foo here, that is a foo1.c foo2.c foo.h compiled into foo.so 
and passed to gcc-4.5 -fplugin=foo.so- betweem eg 4.5 & 4.6 do you mean that

1.   the binary foo.so will work for both gcc-4.5 & gcc-4.6 ?
I strongly believe that is impossible, and should not be dreamed of.

2.   the binary foo.so will work for both gcc-4.5.1 & gcc-4.5.2 ?
I tend to believe that is not necessary. People (or more probably Linux distribution makers) will recompile foo for 
4.5.2. Plugins are free software, so users have the plugin sources (and distribution makers would recompile them).

3.  the sources foo*.[ch] can be recompiled without patches between gcc-4.5 & gcc-4.6?
I believe it is not realistic, but not because of any plugin API - whatever its stability. Looking into the past, I 
don't believe that a random (rarely used) pass which has been developped as a patch to 4.2 can work without changes on 
4.3. I admit I have no direct example. But look at today's passes. Many of them has been patched between 4.4 & 4.5... 
This problem has nothing to do with plugins API (I mean with gcc/gcc-plugin.h to be specific). It is related to the fact 
that GCC is not organised as a strict set of well defined interfaces. This is a social property of the GCC community, 
and it worked quite well in practice (GCC is not in a very bad shape today!). Of course, some files are more important 
than others. Very probably, adding fields inside gimple.h is much less common than adding some extra passes. And 
obviously, adding an important file like gimple.h is for elite GCC developers.

4. The sources foo*.[ch] can *usually* be recompiled without patches between gcc-4.5.1 & gcc-4.5.2 ?
This is my current intuition. I really believe that even for minor version changes like 4.5.1 -> 4.5.2, plugins will 
need to be at least recompiled. And if I understand correctly, the

> 
> I especially want to avoid adding plugin callbacks in code that is
> known to be not the way it should be.  Instead GCC internals should
> be cleaned up and adjusted to be plugin-friendly.  This probably
> will apply to some of the pass-manager hooks people are going to
> propose.
> 
>> So I understood that people could propose new plugin events, provided that
>> they are not too costly to implement (mostly meaning that the
>> invoke_register_plugins calling them does not slow down compilations not
>> using plugins), and provided people can explain clearly why they need it. It
>> is not necessary to have a useful plugin coded & runnable to show the need
>> of the event.
> 
> Yes it is.  And I am going to voice this opinion whenever I see people
> are adding hooks because it is fun to do so.

Nobody is adding plugin hooks because it is fun. Mind you, I find pushing minuscule hooks [like the PLUGIN_PRAGMAS] a 
very stressing & boring task (and I warmly thank Rafael Espindola & Diego Novillo for their help). It is really not fun 
for me. However, if I really miss a hook, I won't be able to design a future plugin the way I am thinking of it today. 
My fun is not in adding a new PLUGIN_* identifier in the plugin_event enum of gcc-plugin.h. My fun is more in coding the 
plugin itself (or at least, it should be).

Actually, I tend to believe that the plugin infrastructure should ideally permit to replace almost every single file in 
GCC by an equivalent file in the plugin. Take for example the pass manager (file gcc/passes.c) I understood that ICI 
have their own pass manager (By the way, I Basile don't care about the pass manager). So ICI guys would probably be 
happy to bypass all the functions in the gcc-4.5/gcc/passes.c and provide their own variant of these. My intuition is 
that this should be doable.

I probably am happy enough with the current set of hooks. I might wish in a year some hooks for CPP, but I understood 
Diego need them, so I believe they will appear.

But I am not sure that every GCC experiment is pluginable today (I really don't know). Take for instance all the papers 
in the recent GCC summits (let's say after 2007). All of them describe code related to GCC. I am not sure that every 
such code could fit as a plugin. I believe it should.

Perhaps all of GCC code is bypassable by a plugin which would handle PLUGIN_START_UNIT, copy or call all of the existing 
code of GCC inside it, and exit without returning... So probably, plugin could do anything.

But I believe we should provide a lot more hooks for them.

Plugins are important for a very social, non-technical reason: nobody today is compiling GCC -except the GCC community & 
  the linux distribution makers. The art of compiling monster software like GCC is almost lost. People don't compile 
free software anymore. (this has changed since the 1980s. At the time of SunOS3.5, people did compile gcc & emacs). And 
I recently discovered that compiling GCC is too difficult a task for most GCC users (at least outside academia). GCC 
plugin will help experimental extensions towards this population of GCC users (not able or not wanting to compile GCC).
Another important motivation for plugins is the licence (more exactly the legal requirement of FSF ownership). Coding 
GPLv3 code is accepted by many managers & organizations (because the GPLv3 is a mainstream licence), but transfering 
copyright to FSF is painful to most big organizations - because it is an unusual procedure -.

This is why defining a large enough set of hooks is important *today* [especially for hooks or events easy to implement 
today]. I am pretty sure that gcc-4.5 will still be a lot used in 2013 or 2017, when the GCC community will be working 
on GCC-5.5 or even 6.5 (I know about some small SMEs in the embedded software economy which are using today a 2.95 
version of GCC, and not really able to compile a 4.3 version of GCC, even if produces better code for their embedded ARM 
processor). So the set of hooks today is important for many years!


Notice also another thing. there have been a lot of plugin experimentation (at least in GCC summit papers). And every 
experiment was about code which the developer thought would never be accepted into the trunk (and that intuition is 
right: nothing of all the plugin experiments landed into the trunk [for example, TreeHydra did not land inside GCC, 
etc.], except the idea of plugins and the small plugin API).

This means that plugins would probably appear mostly for things that have no chance to land into GCC core. Plugin will 
more proably do everything but the kitchen sink. I would expect AMD & Intel to continue contributing to the GCC core to 
target their newest processors - I cannot imagine why AMD or Intel would target their 2011 processor with only a plugin.
[there is a simple reason for that: plugin are a bit more expensive to use that straight GCC - you need extra options & 
extra installations-, and human nature is lazy; the interest of constructors is to sell chips, not GCC plugins!]. 
Toplevel processor makers will continue to work inside GCC.

What could happen if some plugin events are not added quickly? People interested in them will simply lose interest in 
GCC (they could for instance go towards LLVM if they feel it is easier for them). People waited much too long for 
plugins to appear... they won't wait till 4.6 for an event essential for them which is basically a dozen-line patch; 
they will just stop looking at (or inside) GCC.

If the GCC community cared about experimental languages front-end (stuff like Modula3, or Mercury, ...), we should also 
have hooks for other front-ends [I personally have no real need for that]. As a concrete example, GNU CLISP could use 
GCC instead of GNU lightning (to translate its custom bytecode into binary). No, I am not volunteering -and when needing 
a Common Lisp, I'll use SBCL.

So in  contrast to Richard Guenther, I don't percieve any danger today in adding several new plugin events (even if they 
later happen to be nearly useless). Precisely because the plugin API is newborn.

We should not be afraid of new plugin events. We could be afraid of people wanting to start coding plugins going 
elsewhere (to some other compiler technology).

I am not afraid of ICI or TreeHydra guys adding useless plugin events. I am afraid of ICI or TreeHydra guys losing 
interest in GCC (and perhaps going to other compilers). If that happens, it will be a loss for the GCC community.

Regards.

PS. It could also be the case that plugin hooks could be bypassed by sophisticated ld.so tricks like LD_PRELOAD.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: new plugin events

[Richard Guenther]

On Fri, Nov 6, 2009 at 10:41 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Richard Guenther wrote:
>>
>> No, read what Diego said (emphasis mine): "Just that there _is_ some
>> plugin that could take advantage of it."
>>
>
> Diego also said "could", not "can" (emphasis mine). I understood that the
> plugin is not supposed to be mature. It could even be in its design phase...
>
>> As opposed to "There _might_ be some plugin that could take advantage
>> of it."
>
> The intermediate phrase is "There _will_ be some plugin...". I am mostly
> thinking of the brave guys wanting to make a plugin for GCC, and missing
> some particular event.
>
>>
>> I want to avoid the situation where we prematurely add a plugin hook
>> and later recognize it should have been different in a sublte way once
>> the first user appears.  Thus, I want to avoid 1) changing existing
>> plugin callbacks, 2) removing existing plugin callbacks, or even worst
>> 3) changing its semantics.  This all applies to between major and
>> even more so minor GCC releases.
>
> I agree with the list to avoid, but it is really premature today for 4.5. It
> will be a concern for 4.6 or 5.0!! And it probably won't be that much a
> concern. I would imagine that removing useless plugin events in 4.6 or 5.0
> would be possible...

What?  Removing useless plugin events?  How can a plugin event be
useless if you added it with the premise that it might have a valid good use?!
But I guess you agree with me here - you do _not_ know if the events
you want to add are useful!

>> If we don't have at least plugin API stability we could throw plugins
>> down the toilet.
>
> Could you define plugin API stability? For me it is premature to think of
> it.
> First, the Plugin API is so new that it cannot be stable for 4.5. It could
> be stable if it existed before 4.5 which it didn't.
>
> Now, if you think of stability of a given plugin -call it foo here, that is
> a foo1.c foo2.c foo.h compiled into foo.so and passed to gcc-4.5
> -fplugin=foo.so- betweem eg 4.5 & 4.6 do you mean that
>
> 1.   the binary foo.so will work for both gcc-4.5 & gcc-4.6 ?
> I strongly believe that is impossible, and should not be dreamed of.

Of course not.

> 2.   the binary foo.so will work for both gcc-4.5.1 & gcc-4.5.2 ?
> I tend to believe that is not necessary. People (or more probably Linux
> distribution makers) will recompile foo for 4.5.2. Plugins are free
> software, so users have the plugin sources (and distribution makers would
> recompile them).

Likewise.

> 3.  the sources foo*.[ch] can be recompiled without patches between gcc-4.5
> & gcc-4.6?

I expect so in most of the cases.  Well, with 1 and 2 it is already hard
to argue plugins are useful - look at the set of headers you need to install
to build a plugin!  It is as easy to just build a patched GCC.

What I expect will happen is:

People using multiple plugins will have multiple versions of GCC installed,
as the plugin developers are very likely not providing plugin sources for
all available GCC versions around in the wild - like vendor compilers
that are often far away from the branch they are packaged from.  Thus,
to these people plugins will be as useful^H^H^Hless as patches to GCC.

I can see that some very popular plugins be built and distributed alongside
vendor compilers.  But then it would be as much work for the vendors
to integrate GCC patches, which they of course do today!  Instead of
plugins being a clear advantage here they slow down GCC (because
its internal symbols are to be exported), they reduce the value of
the FSF GCC version because it lacks features only available as plugin.

So - I think the whole plugin story is somewhat poorly constructed.  Not
without 1 and 2 being part of the reason for this.  Adding another
churn to this, 4. the plugin api and hooks will change willy-nilly as we like,
isn't going to make that story any better.

Over-exaggerating as always on this topic, but I think I found the
shed I like to paint.  At least you can count me to the non-believers here.

To add something positive at the end - I do see plugin as a useful way
to integrate something into GCC that the GCC developer community
does not like to see integrated.  Like the static analysis plugins available
today, or ICI as available today.  But I do believe that both of them
do have a place inside GCC itself if they are well designed and integrated
within GCC!  Here often GCC and its grown architectural limitations
make it difficult to provide a convincing integration - a chance to fix
GCC!  Which unfortunately plugins will hinder rather than foster.

Thanks for still listening,
Richard.

Re: new plugin events

[Richard Guenther]

On Fri, Nov 6, 2009 at 10:41 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Richard Guenther wrote:

(stop reading now, what follows is pure cynicism - I have to do it, because
the following can't be let un-responded, rather than un-answered)

> Actually, I tend to believe that the plugin infrastructure should ideally
> permit to replace almost every single file in GCC by an equivalent file in
> the plugin. Take for example the pass manager (file gcc/passes.c) I
> understood that ICI have their own pass manager (By the way, I Basile don't
> care about the pass manager). So ICI guys would probably be happy to bypass
> all the functions in the gcc-4.5/gcc/passes.c and provide their own variant
> of these. My intuition is that this should be doable.

It should be possible already if you build GCC with -fPIC.  You then
can LD_PRELOAD a plugin replacing all functions you want to replace.
Thanks to the wonders of ELF.

> I probably am happy enough with the current set of hooks. I might wish in a
> year some hooks for CPP, but I understood Diego need them, so I believe they
> will appear.
>
> But I am not sure that every GCC experiment is pluginable today (I really
> don't know). Take for instance all the papers in the recent GCC summits
> (let's say after 2007). All of them describe code related to GCC. I am not
> sure that every such code could fit as a plugin. I believe it should.

Ah, you mean like doing the tuples conversion as plugin?  Or to
build the cgraph infrastructure and IPA optimization infrastructure
as plugin?  I guess what you say is - "stop developing gcc!  develop plugins!"?

> Perhaps all of GCC code is bypassable by a plugin which would handle
> PLUGIN_START_UNIT, copy or call all of the existing code of GCC inside it,
> and exit without returning... So probably, plugin could do anything.

What's the point?  We have plugins for the kernel for this - it's called
programs, the plugin architecture is processes.

> But I believe we should provide a lot more hooks for them.

Then add a GCC option you build GCC with, -finstrument-functions.
Wait - that already exists!  Or do you maybe mean a more structured
way to expose GCC internals?

> Plugins are important for a very social, non-technical reason: nobody today
> is compiling GCC -except the GCC community &  the linux distribution makers.
> The art of compiling monster software like GCC is almost lost. People don't
> compile free software anymore. (this has changed since the 1980s. At the
> time of SunOS3.5, people did compile gcc & emacs). And I recently discovered
> that compiling GCC is too difficult a task for most GCC users (at least
> outside academia). GCC plugin will help experimental extensions towards this
> population of GCC users (not able or not wanting to compile GCC).
> Another important motivation for plugins is the licence (more exactly the
> legal requirement of FSF ownership). Coding GPLv3 code is accepted by many
> managers & organizations (because the GPLv3 is a mainstream licence), but
> transfering copyright to FSF is painful to most big organizations - because
> it is an unusual procedure -.

Bullshit.  Replacing "monster software like GCC" with "moster plugins
like melt" isn't going to help.  It will just fragment the developer
communities.

> This is why defining a large enough set of hooks is important *today*
> [especially for hooks or events easy to implement today]. I am pretty sure
> that gcc-4.5 will still be a lot used in 2013 or 2017, when the GCC

I am pretty sure it will not.  I am pretty sure no single plugin developer
will keep its plugin compilable for gcc 4.5 in 2013.

> community will be working on GCC-5.5 or even 6.5 (I know about some small
> SMEs in the embedded software economy which are using today a 2.95 version
> of GCC, and not really able to compile a 4.3 version of GCC, even if
> produces better code for their embedded ARM processor). So the set of hooks
> today is important for many years!

They can patch in hooks as they like.  Their plugins will only work
against a patched GCC anyway.

> Notice also another thing. there have been a lot of plugin experimentation
> (at least in GCC summit papers). And every experiment was about code which
> the developer thought would never be accepted into the trunk (and that
> intuition is right: nothing of all the plugin experiments landed into the
> trunk [for example, TreeHydra did not land inside GCC, etc.], except the
> idea of plugins and the small plugin API).

Ah, I responded to that already.

> This means that plugins would probably appear mostly for things that have no
> chance to land into GCC core. Plugin will more proably do everything but the
> kitchen sink. I would expect AMD & Intel to continue contributing to the GCC
> core to target their newest processors - I cannot imagine why AMD or Intel
> would target their 2011 processor with only a plugin.
> [there is a simple reason for that: plugin are a bit more expensive to use
> that straight GCC - you need extra options & extra installations-, and human
> nature is lazy; the interest of constructors is to sell chips, not GCC
> plugins!]. Toplevel processor makers will continue to work inside GCC.

So you believe the only changes in GCC that will happen is newer
processor support?  That's sad.

> What could happen if some plugin events are not added quickly? People
> interested in them will simply lose interest in GCC (they could for instance
> go towards LLVM if they feel it is easier for them). People waited much too
> long for plugins to appear... they won't wait till 4.6 for an event
> essential for them which is basically a dozen-line patch; they will just
> stop looking at (or inside) GCC

Good.  People not wanting or being able to interact with the GCC
developer community neither do have a chance to successfully
develop a plugin nor are they of interest to GCC.

> If the GCC community cared about experimental languages front-end (stuff
> like Modula3, or Mercury, ...), we should also have hooks for other
> front-ends [I personally have no real need for that]. As a concrete example,
> GNU CLISP could use GCC instead of GNU lightning (to translate its custom
> bytecode into binary). No, I am not volunteering -and when needing a Common
> Lisp, I'll use SBCL.

The frontend interface is very well separated nowadays.  Pluginifizing
it should be possible - but please - this needs at least _some_ work
inside GCC.  Plugins never should be an excuse to not improve or
fix GCC, instead they should offer an opportunity to do so.

> So in  contrast to Richard Guenther, I don't percieve any danger today in
> adding several new plugin events (even if they later happen to be nearly
> useless). Precisely because the plugin API is newborn.

We don't add GCC internal APIs like that, so why should plugin APIs
any different here?  On the contrary - the hurdle to add external
interfaces should be much higher than adding internal ones.

> We should not be afraid of new plugin events. We could be afraid of people
> wanting to start coding plugins going elsewhere (to some other compiler
> technology).

I'm not afraid of the second.

> I am not afraid of ICI or TreeHydra guys adding useless plugin events. I am
> afraid of ICI or TreeHydra guys losing interest in GCC (and perhaps going to
> other compilers). If that happens, it will be a loss for the GCC community.

I do not see them as part of the GCC community (sorry, no offense intended,
exaggerating again here).

> Regards.
>
> PS. It could also be the case that plugin hooks could be bypassed by
> sophisticated ld.so tricks like LD_PRELOAD.

Oh, so you noticed!  Plugins on steroids.  Not even sophisticated, but
simpler.

Richard.

Re: new plugin events

[Joseph S. Myers]

On Fri, 6 Nov 2009, Richard Guenther wrote:

> its internal symbols are to be exported), they reduce the value of
> the FSF GCC version because it lacks features only available as plugin.

That's why I think it's important for people to consider carefully in each 
case whether a plugin or integration in GCC is the right way to implement 
a given feature.  I think of plugins as being for GCC features (warnings, 
optimizations, etc.) only relevant for compiling some piece of software or 
using some library, for example, and not generally relevant to standard 
code in the relevant source language.  They are not generally appropriate 
for optimizations relying only on standard language or library features, 
for example.

There are cases where the expected instability of the plugin API can be a 
*feature*.  For example, people have wanted ways for -Wformat to check 
custom types of format strings, but there's never been a design I've been 
happy with in terms of both sufficient flexibility and not exporting too 
much implementation internals that we might wish to change in future.  I'd 
be much happier there with a few plugin hooks that expose all the format 
checking datastructures, and an understanding that these will change or 
break when those structures change, than with directly exposing all those 
structures to user source code as part of the GNU C and C++ languages.  
(Making the checking for GCC-internal formats into such a plugin has been 
suggested in the past, and is not completely unreasonable.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: new plugin events

[Basile STARYNKEVITCH]

Richard Guenther wrote:
> On Fri, Nov 6, 2009 at 10:41 PM, Basile STARYNKEVITCH
> <basile@starynkevitch.net> wrote:
>> Richard Guenther wrote:
> 
> (stop reading now, what follows is pure cynicism - I have to do it, because
> the following can't be let un-responded, rather than un-answered)
> 
>> Actually, I tend to believe that the plugin infrastructure should ideally
>> permit to replace almost every single file in GCC by an equivalent file in
>> the plugin. Take for example the pass manager (file gcc/passes.c) I
>> understood that ICI have their own pass manager (By the way, I Basile don't
>> care about the pass manager). So ICI guys would probably be happy to bypass
>> all the functions in the gcc-4.5/gcc/passes.c and provide their own variant
>> of these. My intuition is that this should be doable.


I meant that some developers (not me) might dream of replace one single file X.c with their XX.c.
For ICI, I understood that X.c is gcc/passes.c (and I don't know what is their XX.c replacing it)
I don't understand the cynicism adjective (which I feel too aggressive).
  In other words, I believe that the plugin functionality should permit experimentation on most features of GCC. Of 
course I do not mean all of them at once, I just mean making an experiment where one single feature or functionality X 
[to simplify, an important GCC function or source file) of GCC is replaced by something provided by the plugin.



> 
> It should be possible already if you build GCC with -fPIC.  You then
> can LD_PRELOAD a plugin replacing all functions you want to replace.
> Thanks to the wonders of ELF.
> 
>> I probably am happy enough with the current set of hooks. I might wish in a
>> year some hooks for CPP, but I understood Diego need them, so I believe they
>> will appear.
>>
>> But I am not sure that every GCC experiment is pluginable today (I really
>> don't know). Take for instance all the papers in the recent GCC summits
>> (let's say after 2007). All of them describe code related to GCC. I am not
>> sure that every such code could fit as a plugin. I believe it should.
> 
> Ah, you mean like doing the tuples conversion as plugin?  Or to
> build the cgraph infrastructure and IPA optimization infrastructure
> as plugin?  I guess what you say is - "stop developing gcc!  develop plugins!"?

To be more precise, I believe that some few plugins (not the ones I Basile will write) might perhaps replace the branch 
role as  experimental ground for GCC. And I don't see any issue here, on the contrary. Plugins are for their users 
perhaps easier to try than a whole GCC branch. What's the difference between writing a plugin and writing a branch 
nobody will use? So perhaps people would start plugins instead of starting their branch. Why should that be bad? I feel 
it is very positive!

>> Perhaps all of GCC code is bypassable by a plugin which would handle
>> PLUGIN_START_UNIT, copy or call all of the existing code of GCC inside it,
>> and exit without returning... So probably, plugin could do anything.
> 
> What's the point?  We have plugins for the kernel for this - it's called
> programs, the plugin architecture is processes.

No plugins for the kernel are called modules. The command to list the set of loaded kernel modules is lsmod (not ps). 
The debian kernel I am using now have 93 modules loaded.

> 
>> But I believe we should provide a lot more hooks for them.
> 
> Then add a GCC option you build GCC with, -finstrument-functions.
> Wait - that already exists!  Or do you maybe mean a more structured
> way to expose GCC internals?
> 
>> Plugins are important for a very social, non-technical reason: nobody today
>> is compiling GCC -except the GCC community &  the linux distribution makers.
>> The art of compiling monster software like GCC is almost lost. People don't
>> compile free software anymore. (this has changed since the 1980s. At the
>> time of SunOS3.5, people did compile gcc & emacs). And I recently discovered
>> that compiling GCC is too difficult a task for most GCC users (at least
>> outside academia). GCC plugin will help experimental extensions towards this
>> population of GCC users (not able or not wanting to compile GCC).
>> Another important motivation for plugins is the licence (more exactly the
>> legal requirement of FSF ownership). Coding GPLv3 code is accepted by many
>> managers & organizations (because the GPLv3 is a mainstream licence), but
>> transfering copyright to FSF is painful to most big organizations - because
>> it is an unusual procedure -.


I insist that the copyright to FSF transfer is a big brake to enter GCC development. And this brake is not effective 
inside plugins. Plugins can be simply GPLv3, but not FSF copyrighted. I am sure it makes an important difference for 
putative developpers (more precisely for their bosses).


> Bullshit.  Replacing "monster software like GCC" with "moster plugins
> like melt" isn't going to help.  It will just fragment the developer
> communities.

You might tell everything about MELT, but not that is is monstrous. It is a small one-man work. To get the figures 
right, MELT code (the one written by a human, me [I am excluding the generated code]) is about 38KLOC and the GCC trunk 
is more than 3900KLOC - this is more than a hundred times more. [measured by sloccount, by copying all MELT specific 
files somewhere and renaming *.melt as *.lisp]. Of course MELT generated C code is big. That is true of most code 
generators!

And the GCC community is large enough to be fragmented today. People working on C++ or Ada front-ends are not very close 
of people working on AMD processor back-ends, or on register allocators. A lose community of hundreds of developers is 
necessarily fragmented. I would even wish it to be more fragmented: I would prefer a set of several gcc-middle-end@ 
gcc-front-ends@ gcc-back-ends@ mailing lists instead of the single gcc@ one.

> 
>> This is why defining a large enough set of hooks is important *today*
>> [especially for hooks or events easy to implement today]. I am pretty sure
>> that gcc-4.5 will still be a lot used in 2013 or 2017, when the GCC
> 
> I am pretty sure it will not.  I am pretty sure no single plugin developer
> will keep its plugin compilable for gcc 4.5 in 2013.

We don't know. Maybe you are right. But I would believe that a plugin written in 2010 for 4.5.0 could be used in 2013 
for the same 4.5.0, or perhaps if it was written for 4.5.0 could be recompiled for a gcc-4.5.4

> 
>> community will be working on GCC-5.5 or even 6.5 (I know about some small
>> SMEs in the embedded software economy which are using today a 2.95 version
>> of GCC, and not really able to compile a 4.3 version of GCC, even if
>> produces better code for their embedded ARM processor). So the set of hooks
>> today is important for many years!
> 
> They can patch in hooks as they like.  Their plugins will only work
> against a patched GCC anyway.

I don't understand that. My claim was the opposite. If we add the several hooks needed for ICI or another plugins, these 
plugins should work for an unpatched GCC.

> 
>> Notice also another thing. there have been a lot of plugin experimentation
>> (at least in GCC summit papers). And every experiment was about code which
>> the developer thought would never be accepted into the trunk (and that
>> intuition is right: nothing of all the plugin experiments landed into the
>> trunk [for example, TreeHydra did not land inside GCC, etc.], except the
>> idea of plugins and the small plugin API).
> 
> Ah, I responded to that already.
> 
>> This means that plugins would probably appear mostly for things that have no
>> chance to land into GCC core. Plugin will more proably do everything but the
>> kitchen sink. I would expect AMD & Intel to continue contributing to the GCC
>> core to target their newest processors - I cannot imagine why AMD or Intel
>> would target their 2011 processor with only a plugin.
>> [there is a simple reason for that: plugin are a bit more expensive to use
>> that straight GCC - you need extra options & extra installations-, and human
>> nature is lazy; the interest of constructors is to sell chips, not GCC
>> plugins!]. Toplevel processor makers will continue to work inside GCC.
> 
> So you believe the only changes in GCC that will happen is newer
> processor support?  That's sad.

No I did not tell that. I have no idea of where GCC will go in the future (I am not even sure GCC will remain dominant 
in 2015 - nobody knows!). I just wrote that the interest of AMD & Intel is to get code supporting their new processors 
inside the GCC trunk. They have no reason to put that code in a plugin, and they have strong and obvious reasons to put 
that code inside the GCC trunk. And there are large enough corporations to afford a few extra developers to push code 
into the trunk (instead of leaving it as a plugin).

> 
>> What could happen if some plugin events are not added quickly? People
>> interested in them will simply lose interest in GCC (they could for instance
>> go towards LLVM if they feel it is easier for them). People waited much too
>> long for plugins to appear... they won't wait till 4.6 for an event
>> essential for them which is basically a dozen-line patch; they will just
>> stop looking at (or inside) GCC
> 
> Good.  People not wanting or being able to interact with the GCC
> developer community neither do have a chance to successfully
> develop a plugin nor are they of interest to GCC.
> 
>> If the GCC community cared about experimental languages front-end (stuff
>> like Modula3, or Mercury, ...), we should also have hooks for other
>> front-ends [I personally have no real need for that]. As a concrete example,
>> GNU CLISP could use GCC instead of GNU lightning (to translate its custom
>> bytecode into binary). No, I am not volunteering -and when needing a Common
>> Lisp, I'll use SBCL.
> 
> The frontend interface is very well separated nowadays.  Pluginifizing
> it should be possible - but please - this needs at least _some_ work
> inside GCC.  Plugins never should be an excuse to not improve or
> fix GCC, instead they should offer an opportunity to do so.

I begin to understand a bit more Richard's position. Let me try to rephrase it.
"There are some current weaknesses in today's GCC architecture, and and it is more important to fix them first than to 
permit plugins - in particular plugins experimenting some workaround or improvements of these weaknesses."


I see these two issues as completely independent. There are indeed weaknesses in the GCC architecture. I perceive that 
as a normal state for an old -more than twenty years old- software like GCC (and given its age, GCC is in good shape). 
In addition, it is related to the way the GCC community works - lack of a benevolent dictator (like my understanding of 
the role of Linus Torvalds in the Linux kernel - but I admit I don't understand the details of Linus' work today). Also, 
I am not sure that any corporation has strong enough interest in improving the GCC architecture to pay a dozen of world 
class developers for that sole purpose of improving the GCC architecture. At least, I never saw that...
(the only possible exception is perhaps Google, but I am not naive enough to believe that Google would tell really why 
they invest in GCC [we can only guess that]. But I am still perceiving that even Google has not enough interest in 
improving the GCC architecture - it is only a good side effect of their work; Google does not seem to pay a dozen of 
talented individuals with the improvement of GCC architecture as their *only* goal!).

So I agree that GCC architecture has shortcomings. I am not quite sure that we (not only Richard Guenther & me Basile, 
but the whole community) agree on the exact list of GCC architectural weaknesses (as usual, the evil is in the details). 
And correcting GCC weaknesses is always much more complex than expected (look at the various efforts on register 
allocators - I know nothing of them, except that there have been many register allocator tentatives in the past, and 
that these efforts have been painful).

But why should the GCC architecture weakness prohibit experimental plugins? Why should we refuse the few events needed 
to ICI or others? Some ICI people might perhaps think that their alternative pass manager is better than the current 
one.. [I have no idea if this is true or not, and most importantly, there are a lot of different but valid points of 
view.]. Why could we let them experiment inside a plugin if that required only a few more PLUGIN_* events? Very 
probably, nobody looked precisely inside their branch!

> 
>> So in  contrast to Richard Guenther, I don't percieve any danger today in
>> adding several new plugin events (even if they later happen to be nearly
>> useless). Precisely because the plugin API is newborn.
> 
> We don't add GCC internal APIs like that, so why should plugin APIs
> any different here?  On the contrary - the hurdle to add external
> interfaces should be much higher than adding internal ones.

Well, the internal headers of GCC [files gcc/*.h] are still significantly evolving [more than 20 *.h files appear in 
gcc/ChangeLog just for the current week of November 2009] I perceive them as the GCC internal API and it did evolve this 
week.

So adding a PLUGIN_ event in 4.5 and removing another one in 4.7 is not so important. I guess that a few additional 
PLUGIN_* events are enough to permit experimenting their pass manager instead of the current one. Sorry, I cannot list 
these additional hooks (because I don't know the details of ICI, and I don't master entirely the current pass manager. I 
was dreaming of hooks inside execute_ipa_pass_list of passes.c or elsewhere, perhaps toplevel.c). I leave the details 
and effort of submitting such patches to other people (including ICI people).

The point is that push a few PLUGIN_* events into the trunk is a few small patches (a hundred lines at most), while 
pushing a whole new pass manager is a lot of big patches (several thousand lines), and pushing such big patches is 
extremely difficult and take years, and probably can only be done by old GCC top level contributors (like Diego Novillo 
or Richard Ghenther, not like me or ICI people).

I do agree that there are shortcomings in the GCC architecture. But I won't even dare proposing a patch on them [because 
I am 100% sure it won't even be reviewed, so that I am sure to lose my time.] This is a task for respected GCC 
developers (like Diego & Richard), not for a single Write-After-Approval random hacker like me.

My hypothesis is that trying a plugin is cheaper (from the GCC user's point of view) than trying a whole GCC branch. 
Compiling a whole branch is so hard in practice that nobody cares about branches (because almost nobody takes the time 
to compile GCC). As a case in point, I don't know of any GCC branch packaged in Debian (even Debian/Unstable!) or 
Mandriva. And I believe perhaps naively that in 2011 some Debian or Mandriva package will provide some plugin to gcc-4.5 
(or gcc-4.6?) there.

I definitely agree with Richard that the current pass management is not entirely satisfactory (however, the pass 
organizations of GCC is a strength of GCC!). But providing a few hooks inside it is much easier than replace the pass 
manager entirely. Adding a few hooks inside is quite simple, replacing it entirely by something better is very 
difficult, and very probably impossible for most GCC contributors. Even if someone -outside of the core GCC team, I mean 
outside of the set of reviewers or eminent GCC people like Richard Guenther, Diego Novillo and a small dozen of others- 
proposed a big patch about pass management, nobody would review or even read it. And the point is that some 
architectural changes in GCC cannot be split in small patches!

I believe that GCC architectural deficiencies will stay for a long time, because the set of people able to improve them 
is small (it is a select club), and because most of these people are not paid for improving GCC architecture as a whole, 
but for pushing some subset of features inside GCC. I can imagine that LTO & WHOPR are more important to Diego (and to 
his boss) than improving GCC architecture. And the set of GCC reviewers is also small.

So improving GCC architecture will continue to happen in small steps, each of them being quite big for the elite people 
involved. For instance, in the past, I can imagine that the tree -> gimple transition took several person*years of 
effort (on a small set of people).

> 
>> We should not be afraid of new plugin events. We could be afraid of people
>> wanting to start coding plugins going elsewhere (to some other compiler
>> technology).
> 
> I'm not afraid of the second.
> 
>> I am not afraid of ICI or TreeHydra guys adding useless plugin events. I am
>> afraid of ICI or TreeHydra guys losing interest in GCC (and perhaps going to
>> other compilers). If that happens, it will be a loss for the GCC community.
> 
> I do not see them as part of the GCC community (sorry, no offense intended,
> exaggerating again here).

We strongly disagree on that too. I really feel that ICI or TreeHydra people posting on gcc@ or gcc-patches@ lists are 
members of the GCC community as [un-]important as me.

I also understand that the GCC community is like Orwell's animal farm (this is very probably true of other big free 
software communities). Everyone is equal but some are more equal than others. That fact is normal in every human 
society. It is really normal that people care more about what Diego Novillo or Richard Guenther propose than about 
random other patches -including those I Basile submit.


Also, I don't view any precise PLUGIN_* event as a long term commitment to stay. Only time will tell which events are 
important, and which events are unneeded. This is exactly the same with passes inside GCC. Some of them are important, 
other are less used (we cannot tell in november 2009 what is the essential set of PLUGIN_* events. For example, I don't 
understand precisely the role of PLUGIN_START_UNIT, but I do believe someone else is needing that and that does not 
bothers me. In contrast, the PLUGIN_GGC_MARKING event is absolutely essential to MELT, but I don't know of any other 
plugin needing it, and I believe almost no other plugin will need it, even if it can use it. Any I am sure that 
PLUGIN_GGC_MARKING or PLUGIN_START_UNIT don't bother much GCC.). So some PLUGIN_* events could perhaps disappear later. 
What is the problem? What is the difference between dropping a particular builtin or pragma or target architecture?

I really believe it is acceptable to admit (before 4.5 release) a half-dozen or a dozen more events in enum plugin_event 
[but I Basile don't need any now]. This won't break any code today (because plugins are newborns). This will help the 
people proposing them for the plugins they *want* to develop - I don't claim that people develop plugins for their 
private branch of GCC. Why should we reject a priori any such additions?

And it is not related to the deficiencies of the GCC architecture. On the contrary, perhaps additional plugin hooks 
could permit experimentation - on a larger scale than GCC branches nobody cares about - of new features...

I really think that some plugins will appear and be widely distributed (in particular packaged in Debian or Redhat or 
SuSE or Mandriva or other Linux distributions.). This is not the case for GCC experimental branches. From the random GCC 
user point of view, branches are totally inexistent, while plugins are merely superfluous (like is -O3 today, or even 
more the optimisations not even triggered by -O3; almost nobody uses them; the average GCC user is using -g when 
developping and -O2 at most when deploying; he is not using -O3 and won't use plugins; both -O3 and GCC plugins have few 
users..).

Is there any experimental branch of GCC packaged in any Linux distribution? Of course, I am not thinking of vendor GCC 
branches (like suse ubuntu redhat...).

Suppose that the Mozilla people make the effort to make TreeHydra a real gcc-4.5 plugin when GCC 4.5 is released. Then I 
would imagine that this TreeHydra plugin could be packaged in Debian, much like there is a xulrunner-dev package in 
Debian today (I have no idea if the treeehydra.so GCC-4.5 plugin binary would go in xulrunner-dev or in 
gcc-4.5-treehydra Debian package; this is the responsability of Debian). But I just checked with "apt-cache search 
hydra" - there is no treehydra-gcc package in Debian/Unstable!


So I still hope ICI people and others will submit short patches for some few plugin events (for GCC 4.5). This is more 
likely to be accepted than a big thousand-line patch (like e.g. a patch pushing the ICI pass manager as a whole; nobody 
amongst the GCC reviewers cares...).

And I do agree with Richard that there are some GCC architectural deficiencies. In an ideal world, I would like to have 
them improved [I don't have the authority, wisdom, or time to improve them myself; I am sadly sure that even if I or 
some other random GCC hacker worked a whole year on them, nobody would care : any big patch won't even be reviewed, 
unless it has been proposed by an important person in the GCC community..; and architectural improvements are disruptive 
and require big patches; they cannot be proposed as a sequence of small patches.]. In the real world, I prefer that a 
few more plugin hooks would appear to permit experimenting new ways of improving these deficiencies inside plugins. I 
understood that Richard don't want that. [I guess Richard want such improvements to go straight into GCC code; I Basile 
think that won't happen, unless it is originated by the elite GCC developers.].

Personally, I Basile is now happy enough with the current set of plugin events, assuming that Diego will push a patch 
for extending the preprocessor (that is for offering from a plugin new preprocessor tricks like the current __COUNTER__ 
which is a GCC specific extension, not an ISO C standard if I understood it well). And probably, even without 
preprocessor plugins (pushed by Diego), I Basile can work for two years inside GCC (that is inside MELT).

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: new plugin events

[Steven Bosscher]

On Sat, Nov 7, 2009 at 11:49 AM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Richard Guenther wrote:
>> Ah, you mean like doing the tuples conversion as plugin?  Or to
>> build the cgraph infrastructure and IPA optimization infrastructure
>> as plugin?  I guess what you say is - "stop developing gcc!  develop
>> plugins!"?
>
> To be more precise, I believe that some few plugins (not the ones I Basile
> will write) might perhaps replace the branch role as  experimental ground
> for GCC. And I don't see any issue here, on the contrary. Plugins are for
> their users perhaps easier to try than a whole GCC branch. What's the
> difference between writing a plugin and writing a branch nobody will use? So
> perhaps people would start plugins instead of starting their branch. Why
> should that be bad? I feel it is very positive!

Because it encourages forking, especially because of...


> I insist that the copyright to FSF transfer is a big brake to enter GCC
> development. And this brake is not effective inside plugins. Plugins can be
> simply GPLv3, but not FSF copyrighted. I am sure it makes an important
> difference for putative developpers (more precisely for their bosses).

...this.  If you don't like to transfer copyright to the FSF, fine.
But don't expect the FSF GCC program (i.e. us) to assist you in
subverting itself.

Plugins should add special functionality that is needed for some niche
application of GCC, but not replace internals of GCC itself.

Ciao!
Steven

Re: new plugin events

[Basile STARYNKEVITCH]

Steven Bosscher wrote:
> On Sat, Nov 7, 2009 at 11:49 AM, Basile STARYNKEVITCH
> <basile@starynkevitch.net> wrote:
>> Richard Guenther wrote:
>>> Ah, you mean like doing the tuples conversion as plugin?  Or to
>>> build the cgraph infrastructure and IPA optimization infrastructure
>>> as plugin?  I guess what you say is - "stop developing gcc!  develop
>>> plugins!"?
>> To be more precise, I believe that some few plugins (not the ones I Basile
>> will write) might perhaps replace the branch role as  experimental ground
>> for GCC. And I don't see any issue here, on the contrary. Plugins are for
>> their users perhaps easier to try than a whole GCC branch. What's the
>> difference between writing a plugin and writing a branch nobody will use? So
>> perhaps people would start plugins instead of starting their branch. Why
>> should that be bad? I feel it is very positive!
> 

> Because it encourages forking, especially because of...

Not really.

(Historically, GCC had few forks, EGCS mostly. And forks are possible because of the GPL. The FSF copyright don't 
disallow them. Look at Xemacs vs Emacs, or SXemacs vs Xemacs. GPL-ed forks of GCC are inpractical, because GCC is too 
big.. They are not prohibited by legal means. Of course I am not a laywer and may be wrong.)

My view is more pragmatical. Getting the FSF copyright is really a hard effort (nothing to do with coding; a lot to do 
with lawyers and other people with which communication is difficult for a hacker.). So it is a *huge* barrier to enter 
the GCC community. It is so big a barrier that I know several persons who gave up. So I am convinced it is a barrier 
which restrict the arrival of new GCC hackers. I believe that if some Joe Hacker did develop a GPlv3 GCC plugin, then if 
the plugin has some success -this is two big ifs so is not probable but it could happen- he would later want to push 
that plugin into GCC core [or at least parts of it] and to transfer copyright to FSF. But getting that it that order 
-Joe Hacker do have a concrete, running, useful, GPLv3 plugin already [this is more than a year of interesting work], 
and Joe Hacker want to push it into GCC, and so Joe Hacker need to transfer copyright to FSF [this is 3 months of boring 
work] - is in my guess less hard than getting a paper signed before coding the very first line of code. It is simpler 
for a lawyer to understand that some particular existing GPLv3 piece code should be transfered to FSF that to sign a 
contract on unidentified future source code.


> 
>> I insist that the copyright to FSF transfer is a big brake to enter GCC
>> development. And this brake is not effective inside plugins. Plugins can be
>> simply GPLv3, but not FSF copyrighted. I am sure it makes an important
>> difference for putative developpers (more precisely for their bosses).
> 
> ...this.  If you don't like to transfer copyright to the FSF, fine.
> But don't expect the FSF GCC program (i.e. us) to assist you in
> subverting itself.
> 
> Plugins should add special functionality that is needed for some niche
> application of GCC, but not replace internals of GCC itself.

I entirely agree, and all the MILEPOST & ICI effort are a niche application (and also MELT).

Besides, both MILEPOST & MELT are FSF owned branches of GCC. [Nobody cares].

I can phrase my views in another way. I don't know much about ICI, but the few things I understood about ICI make me 
believe (half blinded believe, I don't understand the details) that ICI could be useful for GCC. There is a copyright 
transfer between INRIA (I mean the people behind ICI) and FSF. So no legal issues here.

Assuming that some parts of ICI could be useful to GCC, how can we help them in that process?
If they try to send a big patch, I am certain nobody will review it. So ICI people could send big patches -they already 
did that- and nobody is listening.

My suggestion to ICI friends is : just propose quickly your needed plugin events, and make your ICI a GPLv3 plugin.
When you can show that your ICI plugin to an *unmodified* gcc-4.5 brings some value, GCC people will perhaps start to 
listen and look inside.

The points are:

* nobody cares about experimental GCC branches (FSF copyrighted), and nobody looks inside, unless the branch happens to 
be created by some major GCC guru guy like Diego Novillo or Richard Guenther or Steven Bossher or Joseph Myers (or a 
dozen of important GCC guys). But branches created by some random hacker like me Basile or ICI people are virtually 
non-existent. Nobody cares about them, nobody retrieves them, nobody builds them, nobody looks inside them. I am 
entirely sure that if I put a naughty joke in some comment of my MELT branch, nobody would notice. But I won't do that 
anyway... Also, even if there is some brillant idea in some experimental GCC branch, nobody will be aware of it. So in 
my view most branches are only random bytes on disks. Likewise, nobody care about a thousand line patch sent by ICI.

* plugins are hopefully much easier to build than branches (because they are much smaller). So I hope that plugins will 
be a bit more used than experimental branches (which in my view are never used). They wont be a world-wide success 
neither. Let's hope a dozen users for a successful plugin. An experimental branch has 0 users, a successful GCC guru 
created breanch don't have many users neither.

As a case in point, look into Graphite: it took many years to people like Sebastian Pop to go from a prototype branch to 
  inside the trunk (and Graphite won't be used much, if it is in -O3 only). My guess is that the plugin facility will 
make that transition a bit less painful. Because using a plugin is less hard than using a branch.

Regards.





-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

RE: new plugin events

[Grigori Fursin]

Hi Basile et al,

> My suggestion to ICI friends is : just propose quickly your needed plugin events, and make
> your ICI a GPLv3 plugin.
> When you can show that your ICI plugin to an *unmodified* gcc-4.5 brings some value, GCC
> people will perhaps start to
> listen and look inside.

Just to mention that I am a bit confused because I actually don't expect to have problems moving 
ICI to the mainline unless we find some big bugs that can change GCC behavior (but I really don't
think so).
We had many online and offline discussions to move ICI to the mainline GCC in the last few years
with GCC 
colleagues/maintainers. We just sadly got delayed at INRIA this summer due to different reasons but
Joern 
is now working with us for 2 months fully time to clean and test ICI and submit patches as soon as
they are ready. 

It's true that we actually need a few hooks and Joern will communicate about that shortly BUT these
hooks are
already used in real plugins for real performance tuning (in a way as current hooks are used in
Dehydra 
for real program analysis in several companies). Our performance results are gradually added to the
online performance database at http://cTuning.org/cdatabase for EEMBC, SPEC and other programs
across multiple architectures which real users and companies are using to optimize
their real code... A few days ago I got a feedback from Loongson group that they considerably
speeded up EEMBC on their latest processor using GCC 4.4.x and they should upload the results
to the database shortly ... They have been actively working with us using and extending ICI ...
 
That's why only after we showed real results, we would now like to have MINIMAL ICI in mainline GCC
but patches for other extensions including GSOC'09 projects will be submitted to GCC only after
testing ...

We will keep in touch about that,
Grigori
 

Re: new plugin events

[Richard Guenther]

On Sat, Nov 7, 2009 at 1:24 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
> Hi Basile et al,
>
>> My suggestion to ICI friends is : just propose quickly your needed plugin events, and make
>> your ICI a GPLv3 plugin.
>> When you can show that your ICI plugin to an *unmodified* gcc-4.5 brings some value, GCC
>> people will perhaps start to
>> listen and look inside.
>
> Just to mention that I am a bit confused because I actually don't expect to have problems moving
> ICI to the mainline unless we find some big bugs that can change GCC behavior (but I really don't
> think so).
> We had many online and offline discussions to move ICI to the mainline GCC in the last few years
> with GCC
> colleagues/maintainers. We just sadly got delayed at INRIA this summer due to different reasons but
> Joern
> is now working with us for 2 months fully time to clean and test ICI and submit patches as soon as
> they are ready.
>
> It's true that we actually need a few hooks and Joern will communicate about that shortly BUT these
> hooks are
> already used in real plugins for real performance tuning (in a way as current hooks are used in
> Dehydra
> for real program analysis in several companies).

And I don't expect problems in adding hooks that ICI needs.  I expect
that ICI is a reason to improve GCCs pass manager - and I expect that
we will improve GCCs pass manager not by simply adding hooks to it
to "fix" it from inside plugins, but I expect that we'll get a more powerful
pass manager _inside_ GCC.  I also expect or at least hope that more
parts of the compilation process get driven by the pass manager rather
than by ad-hoc code gluing several phases of pass manager driven
stages.

Richard.

RE: new plugin events

[Grigori Fursin]

That's very reasonable and it's our eventual goal too so we will start discussions 
about that in detail whenever ICI is clean. 

By the way, just to mention that I am working with a student (Yuri) to
provide/understand/describe/characterize 
performance dependencies and interaction between passes using ICI to be able to make selection 
and ordering of passes more systematic and avoid having the same multiple passes in the compilation
sequence 
just because they may potentially improve the code. We use ICI with GSOC'09 extensions and is it so
far very 
simple to manipulate passes through plugins (we use XML outside to save all IP passes and passes per
function). 
This is a long term project, but as soon as we have some promising results, we will tell you ;) ...

Cheers,
Grigori

> -----Original Message-----
> From: Richard Guenther [mailto:richard.guenther@gmail.com]
> Sent: Saturday, November 07, 2009 3:06 PM
> To: Grigori Fursin
> Cc: Basile STARYNKEVITCH; Steven Bosscher; Diego Novillo; Rafael Espindola; gcc; Joern
> Rennecke; Zbigniew Chamski
> Subject: Re: new plugin events
> 
> On Sat, Nov 7, 2009 at 1:24 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
> > Hi Basile et al,
> >
> >> My suggestion to ICI friends is : just propose quickly your needed plugin events, and make
> >> your ICI a GPLv3 plugin.
> >> When you can show that your ICI plugin to an *unmodified* gcc-4.5 brings some value, GCC
> >> people will perhaps start to
> >> listen and look inside.
> >
> > Just to mention that I am a bit confused because I actually don't expect to have problems
> moving
> > ICI to the mainline unless we find some big bugs that can change GCC behavior (but I really
> don't
> > think so).
> > We had many online and offline discussions to move ICI to the mainline GCC in the last few
> years
> > with GCC
> > colleagues/maintainers. We just sadly got delayed at INRIA this summer due to different
> reasons but
> > Joern
> > is now working with us for 2 months fully time to clean and test ICI and submit patches as
> soon as
> > they are ready.
> >
> > It's true that we actually need a few hooks and Joern will communicate about that shortly
> BUT these
> > hooks are
> > already used in real plugins for real performance tuning (in a way as current hooks are used
> in
> > Dehydra
> > for real program analysis in several companies).
> 
> And I don't expect problems in adding hooks that ICI needs.  I expect
> that ICI is a reason to improve GCCs pass manager - and I expect that
> we will improve GCCs pass manager not by simply adding hooks to it
> to "fix" it from inside plugins, but I expect that we'll get a more powerful
> pass manager _inside_ GCC.  I also expect or at least hope that more
> parts of the compilation process get driven by the pass manager rather
> than by ad-hoc code gluing several phases of pass manager driven
> stages.
> 
> Richard.

Re: new plugin events

[Basile STARYNKEVITCH]

Richard Guenther wrote:
> On Sat, Nov 7, 2009 at 1:24 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
>> Hi Basile et al,
>>
>>> My suggestion to ICI friends is : just propose quickly your needed plugin events, and make
>>> your ICI a GPLv3 plugin.
>>> When you can show that your ICI plugin to an *unmodified* gcc-4.5 brings some value, GCC
>>> people will perhaps start to
>>> listen and look inside.

>> Just to mention that I am a bit confused because I actually don't expect to have problems moving
>> ICI to the mainline unless we find some big bugs that can change GCC behavior (but I really don't
>> think so).
>> We had many online and offline discussions to move ICI to the mainline GCC in the last few years
>> with GCC
>> colleagues/maintainers. We just sadly got delayed at INRIA this summer due to different reasons but
>> Joern
>> is now working with us for 2 months fully time to clean and test ICI and submit patches as soon as
>> they are ready.
>>
>> It's true that we actually need a few hooks and Joern will communicate about that shortly BUT these
>> hooks are
>> already used in real plugins for real performance tuning (in a way as current hooks are used in
>> Dehydra
>> for real program analysis in several companies).
> 
> And I don't expect problems in adding hooks that ICI needs.  I expect
> that ICI is a reason to improve GCCs pass manager - and I expect that
> we will improve GCCs pass manager not by simply adding hooks to it
> to "fix" it from inside plugins, but I expect that we'll get a more powerful
> pass manager _inside_ GCC.  I also expect or at least hope that more
> parts of the compilation process get driven by the pass manager rather
> than by ad-hoc code gluing several phases of pass manager driven
> stages.

We probably all agree on goals, but perhaps less on timeline.

My feeling (but I admit I don't understand well what stage 3 means precisely for gcc 4.5.0, in particular w.r.t. plugins 
& pass management, and why exactly stage 2 was skipped in 4.5) was up to now:

1. Only very small patches can go into 4.5. So ICI pass manager won't go into 4.5.0, and any improved pass manager won't 
go into 4.5.0, only in 4.6.0. This probably means in last quarter of 2010 or first quarter of 2011, since 4.4.0 was 
released in april 2009, and 4.3.0 was released in march 2008 and 4.2.0 was released in may 2007 and 4.1.0 at end of 
february 2006. I am guessing 4.5.0 would be released for christmas 2009 at the earliest, so 4.6.0 would go out at end of 
2010 in the best case.

2. I was hoping that the few PLUGIN_* hooks absolutely needed by ICI could go into 4.5. My intuition is that it really 
means small unobtrusive patches which might be accepted before Christmas 2009.

In other words, are there hope to delay 4.5.0 release for a month to get the ICI-improved pass manager inside it? In 
case the answer is yes, will we keep the same interface, so that the interface for  PLUGIN_PASS_MANAGER_SETUP won't 
change with the improved pass manager?
     register_callback (plugin_info->base_name, PLUGIN_PASS_MANAGER_SETUP, NULL, &pass_info);
with the same pass_info as documented today in http://gcc.gnu.org/onlinedocs/gccint/Plugins.html#Plugins ?


So perhaps I am simply not understanding what means the current stage 3 of trunk (ie future 4.5). I admit I did not 
understood exactly why stage 2 disappeared (or perhaps we call today stage 3 what was called stage 2 or stage 3 a year 
ago?). And perhaps I am wrongly guessing the release date of 4.5.0. Of course, if 4.5.0 is released in june 2010 things 
are different!

I suppose that ICI friends will be happy if they can hope to push a new pass manager inside 4.5. My personal guess was 
that this is hopeless - they need to wait for 4.6. I would be delighted to be wrong, and as Richard and others I would 
be extremely happy with a new pass manager.

I think MELT can cope with any reasonable evolution of the pass manager.

I believe (only my opinion) that ICI people care a lot about being able or not to push an improved pass manager inside 
4.5 or only inside 4.6. That makes a one year difference, and I guess one year is not insignificant to them.

Regards.

PS. BTW, I find that speaking of stages with number is confusing (especially since now stage 3 goes right after stage 
1). I would much prefer named stages, like "major change stage", "minor improvement stage", "bug fix stage".
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

RE: new plugin events

[Grigori Fursin]

Basile,

I understand your constraints and concerns.

I personally would also be happy to see ICI and pass manager in GCC soon,
BUT it was delay on our side that prevented submission/checking of the patch,
so I am just taking a pragmatic approach of preparing an ICI patch first 
(well, actually not me but Joern who is now working full time on that), 
testing it and then submitting it and discussing with everyone and you 
if it's reasonable. ONLY THEN, depending if the changes are small and if GCC 4.5.0 
is not yet closed, we will negotiate to move it to the mainline. But at the moment, 
before submitting it, it's just a gamble if it can go through or not and I personally 
don't want to do that because we will annoy all other GCC people who are working 
hard to make current GCC stable ...

So, let's continue ICI discussions as soon as the ICI patch is ready ;) ...

Cheers,
Grigori
 

> -----Original Message-----
> From: Basile STARYNKEVITCH [mailto:basile@starynkevitch.net]
> Sent: Saturday, November 07, 2009 3:45 PM
> To: Richard Guenther
> Cc: Grigori Fursin; Steven Bosscher; Diego Novillo; Rafael Espindola; gcc; Joern Rennecke;
> Zbigniew Chamski
> Subject: Re: new plugin events
> 
> Richard Guenther wrote:
> > On Sat, Nov 7, 2009 at 1:24 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
> >> Hi Basile et al,
> >>
> >>> My suggestion to ICI friends is : just propose quickly your needed plugin events, and make
> >>> your ICI a GPLv3 plugin.
> >>> When you can show that your ICI plugin to an *unmodified* gcc-4.5 brings some value, GCC
> >>> people will perhaps start to
> >>> listen and look inside.
> 
> >> Just to mention that I am a bit confused because I actually don't expect to have problems
> moving
> >> ICI to the mainline unless we find some big bugs that can change GCC behavior (but I really
> don't
> >> think so).
> >> We had many online and offline discussions to move ICI to the mainline GCC in the last few
> years
> >> with GCC
> >> colleagues/maintainers. We just sadly got delayed at INRIA this summer due to different
> reasons but
> >> Joern
> >> is now working with us for 2 months fully time to clean and test ICI and submit patches as
> soon as
> >> they are ready.
> >>
> >> It's true that we actually need a few hooks and Joern will communicate about that shortly
> BUT these
> >> hooks are
> >> already used in real plugins for real performance tuning (in a way as current hooks are
> used in
> >> Dehydra
> >> for real program analysis in several companies).
> >
> > And I don't expect problems in adding hooks that ICI needs.  I expect
> > that ICI is a reason to improve GCCs pass manager - and I expect that
> > we will improve GCCs pass manager not by simply adding hooks to it
> > to "fix" it from inside plugins, but I expect that we'll get a more powerful
> > pass manager _inside_ GCC.  I also expect or at least hope that more
> > parts of the compilation process get driven by the pass manager rather
> > than by ad-hoc code gluing several phases of pass manager driven
> > stages.
> 
> We probably all agree on goals, but perhaps less on timeline.
> 
> My feeling (but I admit I don't understand well what stage 3 means precisely for gcc 4.5.0, in
> particular w.r.t. plugins
> & pass management, and why exactly stage 2 was skipped in 4.5) was up to now:
> 
> 1. Only very small patches can go into 4.5. So ICI pass manager won't go into 4.5.0, and any
> improved pass manager won't
> go into 4.5.0, only in 4.6.0. This probably means in last quarter of 2010 or first quarter of
> 2011, since 4.4.0 was
> released in april 2009, and 4.3.0 was released in march 2008 and 4.2.0 was released in may
> 2007 and 4.1.0 at end of
> february 2006. I am guessing 4.5.0 would be released for christmas 2009 at the earliest, so
> 4.6.0 would go out at end of
> 2010 in the best case.
> 
> 2. I was hoping that the few PLUGIN_* hooks absolutely needed by ICI could go into 4.5. My
> intuition is that it really
> means small unobtrusive patches which might be accepted before Christmas 2009.
> 
> In other words, are there hope to delay 4.5.0 release for a month to get the ICI-improved pass
> manager inside it? In
> case the answer is yes, will we keep the same interface, so that the interface for
> PLUGIN_PASS_MANAGER_SETUP won't
> change with the improved pass manager?
>      register_callback (plugin_info->base_name, PLUGIN_PASS_MANAGER_SETUP, NULL, &pass_info);
> with the same pass_info as documented today in
> http://gcc.gnu.org/onlinedocs/gccint/Plugins.html#Plugins ?
> 
> 
> So perhaps I am simply not understanding what means the current stage 3 of trunk (ie future
> 4.5). I admit I did not
> understood exactly why stage 2 disappeared (or perhaps we call today stage 3 what was called
> stage 2 or stage 3 a year
> ago?). And perhaps I am wrongly guessing the release date of 4.5.0. Of course, if 4.5.0 is
> released in june 2010 things
> are different!
> 
> I suppose that ICI friends will be happy if they can hope to push a new pass manager inside
> 4.5. My personal guess was
> that this is hopeless - they need to wait for 4.6. I would be delighted to be wrong, and as
> Richard and others I would
> be extremely happy with a new pass manager.
> 
> I think MELT can cope with any reasonable evolution of the pass manager.
> 
> I believe (only my opinion) that ICI people care a lot about being able or not to push an
> improved pass manager inside
> 4.5 or only inside 4.6. That makes a one year difference, and I guess one year is not
> insignificant to them.
> 
> Regards.
> 
> PS. BTW, I find that speaking of stages with number is confusing (especially since now stage 3
> goes right after stage
> 1). I would much prefer named stages, like "major change stage", "minor improvement stage",
> "bug fix stage".
> --
> Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
> email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
> 8, rue de la Faiencerie, 92340 Bourg La Reine, France
> *** opinions {are only mines, sont seulement les miennes} ***