plugin hooks

plugin hooks

[Basile STARYNKEVITCH]

Hello All,

I feel that the current plugin hooks, that is the set of plugin events 
enumerated in the enum plugin_event of gcc/gcc-plugin.h and the 
associated API in gcc/plugin.h (e.g. register_attribute) is perhaps 
still incomplete.

My feeling is that adding plugin events (at least those for which 
invoke_plugin_callbacks is not called a lot of times) is a negligible 
addition that fits well in the current stage 3 of the trunk.
What do you think about that?

The point is that some plugins will need a lot more hooks, and that 
waiting for 4.6 or 5.0 (i.e. the next release after 4.5 from current 
trunk) will delay them possibly for at least a year. And adding these 
additional hooks seems easy and reasonable now. And perhaps adding new 
functions (I am thinking of register_plugin_builtin) to facilitate 
plugin extensions.

I started adding a new wiki page http://gcc.gnu.org/wiki/plugin%20hooks 
to list the possible addition that should go into trunk

Comments are welcome.

Should I try to propose some patches to gcc-patches@ about that, or is 
it impossible in the current stage3 of the trunk?

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Richard Guenther]

On Tue, Oct 27, 2009 at 1:15 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Hello All,
>
> I feel that the current plugin hooks, that is the set of plugin events
> enumerated in the enum plugin_event of gcc/gcc-plugin.h and the associated
> API in gcc/plugin.h (e.g. register_attribute) is perhaps still incomplete.
>
> My feeling is that adding plugin events (at least those for which
> invoke_plugin_callbacks is not called a lot of times) is a negligible
> addition that fits well in the current stage 3 of the trunk.
> What do you think about that?
>
> The point is that some plugins will need a lot more hooks, and that waiting
> for 4.6 or 5.0 (i.e. the next release after 4.5 from current trunk) will
> delay them possibly for at least a year. And adding these additional hooks
> seems easy and reasonable now. And perhaps adding new functions (I am
> thinking of register_plugin_builtin) to facilitate plugin extensions.
>
> I started adding a new wiki page http://gcc.gnu.org/wiki/plugin%20hooks to
> list the possible addition that should go into trunk
>
> Comments are welcome.
>
> Should I try to propose some patches to gcc-patches@ about that, or is it
> impossible in the current stage3 of the trunk?

Adding hooks just because you think they might be useful
isn't the way to go.

Richard.

Re: plugin hooks

[Basile STARYNKEVITCH]

Richard Guenther wrote:
> On Tue, Oct 27, 2009 at 1:15 PM, Basile STARYNKEVITCH
> 
> Adding hooks just because you think they might be useful
> isn't the way to go.

Then what is the correct way to enhance the current plugin API. There 
are a lot of stuff missing there.

I am not at all sure that all the experimental plugin work done so far 
(in particular the ICI work inside MILEPOST and http://ctuning.org) 
could fit into the existing plugin API.

So if you think that adding hooks which can be useful and are not costly 
(you forgot that second part of my proposal) is the wrong way to work, 
what do you propose?

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Ian Lance Taylor]

Basile STARYNKEVITCH <basile@starynkevitch.net> writes:

> Richard Guenther wrote:
>> On Tue, Oct 27, 2009 at 1:15 PM, Basile STARYNKEVITCH
>>
>> Adding hooks just because you think they might be useful
>> isn't the way to go.
>
> Then what is the correct way to enhance the current plugin API. There
> are a lot of stuff missing there.
>
> I am not at all sure that all the experimental plugin work done so far
> (in particular the ICI work inside MILEPOST and http://ctuning.org)
> could fit into the existing plugin API.
>
> So if you think that adding hooks which can be useful and are not
> costly (you forgot that second part of my proposal) is the wrong way
> to work, what do you propose?

We should add hooks as we find plugins that need them.  Simply adding
a laundry list of hooks that we think might be needed will most likely
cause us to overdesign.  We know that we can write interesting plugins
today, so we're not missing anything critical.

I agree that it should be OK to add hooks in stage 3, if they are
clearly safe and clearly needed.

Ian

Re: plugin hooks

[Basile STARYNKEVITCH]

Hello All,

>> Richard Guenther wrote:
>>> Adding hooks just because you think they might be useful
>>> isn't the way to go.


> Basile STARYNKEVITCH <basile@starynkevitch.net> writes:
>> Then what is the correct way to enhance the current plugin API. There
>> are a lot of stuff missing there.
> 

Ian Lance Taylor wrote:
> We should add hooks as we find plugins that need them.  Simply adding
> a laundry list of hooks that we think might be needed will most likely
> cause us to overdesign.  We know that we can write interesting plugins
> today, so we're not missing anything critical.

I believe there is a strong chicken & egg issue here. Ian is suggesting 
that hooks should be added only when an existing plugin would need them, 
but I believe that on the contrary plugins won't appear if they don't 
have a sufficient set of hooks. People won't even bother to make plugins 
if they feel the set of hooks is blantly unsufficient. [They will either
experiment on their own GCC branch, or avoid GCC entirely, for instance
using LLVM instead]. But working on one's branch is much more painful
than coding a plugin!

> I agree that it should be OK to add hooks in stage 3, if they are
> clearly safe and clearly needed.

I have a concrete example here: plugin-specific pragmas (see
PLUGIN_REGISTER_PRAGMA on http://gcc.gnu.org/wiki/plugin%20hook for
details)

I have two imaginary use cases here.

First, some sophisticated static analysers [like Frama-C on
http://frama-c.com which is LGPL licence and happens to be developped by
collegues] extend the C syntax to add some code annotations to
declarations and to blocks and perhaps to C statements. The GCC
equivalent would be to have attributes (for annotation of declarations)
and pragmas (for annotation of blocks), because there is no way from
Gimple to parse a comment near a declaration or a block. So if someone 
wanted to plug such kind of analyzer into GCC thru a plugin, he would 
need additional pragmas, just because the current GCC archtecture does 
not permit any kind of "language extension" from plugins. As another 
case, remember the old lint from the 1980s. It did parse comments like 
/* NOTREACHED */ and the equivalent functionality in GCC can only be 
provided by builtin-s (like our builtin_unreachable today) or by 
pragmas. If we want a superlint-like GCC plugin, we need such hooks like 
plugin specific pragmas & builtins.

Second, some optimisation pass which would perhaps detect some loops to 
be eligible for running on a GPU thru OpenCL and which would clone some 
simple functions into a stub calling OpenCL code and generate that 
OpenCL code -that is source to source transformation of some C or gimple 
into OpenCL- for some external OpenCL compiler [no free OpenCL 
implementation seems to exist today; if you know about one, tell me 
please!]. I have not the details in mind, because I did not work yet on 
that subject. But this is not a week of work, but probably two years of work
[I probably will work on such a GPLv3 plugin using my MELT technology in 
2010-2011, with funding from OpenGPU french project. I already feel that 
2 years is not enough for that goal.]

Both use cases above are very ambitious, and I definitely cannot make a 
plugin prototype in a week or two (or even in two months of work) to 
show that a PLUGIN_REGISTER_PRAGMA is useful. But I am able to propose a 
patch for PLUGIN_REGISTER_PRAGMA in a day or two of work.

So what do you (Ian, Richard and other) suggest:

* not bothering at all, and do all experience inside MELT branch only

* propose a simple patch to add the PLUGIN_REGISTER_PRAGMA event now?

I definitely am not able to propose a usable realistic plugin prototype 
for that PLUGIN_REGISTER_PRAGMA patch now, but I am able to propose a 
patch (and perhaps a testcase that is a plugin which invokes the 
system(3) function; of course such a plugin is extremely unsafe and 
useless, but fellow, it is the simplest example I can think of) now.

I also tend to believe that the discussion above applies to the 
PLUGIN_ADD_CPP_MACROS proposed in http://gcc.gnu.org/wiki/plugin%20hooks 
but I do admit that it is more tricky to implement (it means adding 
plugins inside libcpp).

I am waiting for a concrete reply to my following question:

should I propose a patch for registering pragmas for (and from inside) a 
plugin, given the motivations discussed above? Is that acceptable in the 
current stage3 state of the trunk?

I forgot to say that I believe plugins are a wonderful feature of GCC 
4.5 (and perhaps the condition of its success). I also believe that, if 
the plugin machinery is well designed (so provide enough hooks) it will 
be used by a lot of plugins, and so plugins will appear which will be 
everything but the kitchen sink, and that is a very good thing. We 
should not be afraid of plugin providing unexpected uses to GCC. We 
hould be afraid of lack of plugins (because eg the API is incomplete)!

Another plugin use case is frontend for exotic languages (like Mercury, 
Erlang, Pascal, ...). But this requires a lot more hooks.

And I don't think at all that plugins will open GCC to proprietary 
software invasion. GCC is too complex a beast for that!

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Ian Lance Taylor]

Basile STARYNKEVITCH <basile@starynkevitch.net> writes:

> Ian Lance Taylor wrote:
>> We should add hooks as we find plugins that need them.  Simply adding
>> a laundry list of hooks that we think might be needed will most likely
>> cause us to overdesign.  We know that we can write interesting plugins
>> today, so we're not missing anything critical.
>
> I believe there is a strong chicken & egg issue here. Ian is
> suggesting that hooks should be added only when an existing plugin
> would need them, but I believe that on the contrary plugins won't
> appear if they don't have a sufficient set of hooks.

I didn't mean that the plugin had to exist, I meant that there had to
be a real plugin in view which needed them, not a purely theoretical
need.

> I have a concrete example here: plugin-specific pragmas (see
> PLUGIN_REGISTER_PRAGMA on http://gcc.gnu.org/wiki/plugin%20hook for
> details)

Sure, we clearly need some way for plugins to manage pragmas and
attributes.  This doesn't require much support.  I would first ask
whether attributes are enough, but it wouldn't bother me to have
support for pragmas.

> * propose a simple patch to add the PLUGIN_REGISTER_PRAGMA event now?

Do this.

> should I propose a patch for registering pragmas for (and from inside)
> a plugin, given the motivations discussed above? Is that acceptable in
> the current stage3 state of the trunk?

I believe that such a patch is acceptable in stage 3.


On the other hand, skimming your list of hooks, I really question the
need for, e.g., REPLACE_PASS_MANAGER.  I would want to see a real use
case there before adding such a hook.

For something like PLUGIN_ADD_CPP_MACROS, the need is clear, but I
think it's an open question we should add a hook or whether the plugin
should just call cpp_define.  A hook means that we promise to always
have a way to do this in case cpp_define changes.  How do we decide
when to add that kind of thing to the plugin interface and when to
simply require the plugin to invoke gcc functions directly?

Ian

Re: plugin hooks

[Basile STARYNKEVITCH]

Ian Lance Taylor wrote:
> Basile STARYNKEVITCH <basile@starynkevitch.net> writes:
> 
>> * propose a simple patch to add the PLUGIN_REGISTER_PRAGMA event now?
> 
> Do this.

Will do probably tommorow or this evening!
> 
> On the other hand, skimming your list of hooks, I really question the
> need for, e.g., REPLACE_PASS_MANAGER.  I would want to see a real use
> case there before adding such a hook.

 From several face to face discussions with ICI CTuning people (that it 
Grigori Fursin, Albert Cohen, Zbignew Chamski; all 3 are in CC of this 
reply) I understood that GCC ICI is exactly doing that: it has replaced 
GCC pass manager by its own, which invokes all the existing GCC passes 
in some better order (or better number of invocation) and produces 
better generated code (the tuning is done by machine learning 
techniques; they did run their modified GCC a large number of times and 
did build a database of measures.).

I cannot explain more in details what GCC ICI is actually doing. I hope 
my ICI friends will answer here. Perhaps even Zbignew's latest patches 
are related to the answer.

Regards.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Basile STARYNKEVITCH]

Ian Lance Taylor wrote:
> 
> For something like PLUGIN_ADD_CPP_MACROS, the need is clear, but I
> think it's an open question we should add a hook or whether the plugin
> should just call cpp_define.  A hook means that we promise to always
> have a way to do this in case cpp_define changes.  How do we decide
> when to add that kind of thing to the plugin interface and when to
> simply require the plugin to invoke gcc functions directly?

I (Basile) don't know much of libcpp. If a plugin can just call 
cpp_define, this is ok for me. But then, we perhaps should document how 
can that be called, and when (at plugin initialization, at 
PLUGIN_START_UNIT? I don't know!)

I don't claim that a PLUGIN_ADD_CPP_MACROS is needed, I just think that 
we should be able to declare additional builtin CPP macros [like 
__DATE__ is, or perhaps like defined inside CPP is] from plugins.

I am not familiar enough with libcpp to understand if calling cpp_define 
from a plugin is enough.

As a matter of fact, there are several *.def files which seems to be 
#include-d to fill some constant arrays. For these kind of things, I 
would suppose that perhaps some registration facility is needed, 
probably as an addition register_plugin_XXXX function. Of course, not 
every *.def fits into that scheme: I don't claim that plugin should 
register a new kind of GIMPLE or TREE node (even if a file like 
gcc/gimple.def or gcc/tree.def exists). [*] But the files 
gcc/builtins.def & gcc/builtin-types.def seems to suggest a vector of 
extra plugin-registered builtins & builtin-types and perhaps a 
register_plugin_builtin function etc...

Regards.

(*) Half-joking note about adding GIMPLE node in plugins. Of course, if 
all GCC was in C++, plugins could register extra GIMPLE nodes :-) 
provided the GIMPLE node would be implemented as an abstract class (with 
many virtual method slots).
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Richard Guenther]

On Tue, Oct 27, 2009 at 4:06 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Ian Lance Taylor wrote:
>>
>> Basile STARYNKEVITCH <basile@starynkevitch.net> writes:
>>
>>> * propose a simple patch to add the PLUGIN_REGISTER_PRAGMA event now?
>>
>> Do this.
>
> Will do probably tommorow or this evening!
>>
>> On the other hand, skimming your list of hooks, I really question the
>> need for, e.g., REPLACE_PASS_MANAGER.  I would want to see a real use
>> case there before adding such a hook.
>
> From several face to face discussions with ICI CTuning people (that it
> Grigori Fursin, Albert Cohen, Zbignew Chamski; all 3 are in CC of this
> reply) I understood that GCC ICI is exactly doing that: it has replaced GCC
> pass manager by its own, which invokes all the existing GCC passes in some
> better order (or better number of invocation) and produces better generated
> code (the tuning is done by machine learning techniques; they did run their
> modified GCC a large number of times and did build a database of measures.).
>
> I cannot explain more in details what GCC ICI is actually doing. I hope my
> ICI friends will answer here. Perhaps even Zbignew's latest patches are
> related to the answer.

The fix is not to add a hook to replace the pass manager but instead
to make the GCC pass manager more flexible itself.

Richard.

Re: plugin hooks

[Basile STARYNKEVITCH]

Richard Guenther wrote:
> 
> The fix is not to add a hook to replace the pass manager but instead
> to make the GCC pass manager more flexible itself.

I leave that task to my ICI friends (in CC). They have already an 
implementation, and I believe they probably did propose some patches, or 
are working on it.

However, it is not a small patch.

Also, I still believe that a hook for tha pass manager may make sense to 
those brave souls wanting to experiment new pass managers in plugins, 
etc. Having a hook for that is a way to encourage such crazy 
experimentations.

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Joseph S. Myers]

On Tue, 27 Oct 2009, Basile STARYNKEVITCH wrote:

> I (Basile) don't know much of libcpp. If a plugin can just call cpp_define,
> this is ok for me. But then, we perhaps should document how can that be
> called, and when (at plugin initialization, at PLUGIN_START_UNIT? I don't
> know!)

A plugin can call any non-static function in GCC, with the conditions for 
its inputs and when it can be called being the same as for a direct call 
from GCC; this is a consequence of the design to use -rdynamic.  These may 
variously be documented in comments, the internals manual, list archives 
and collective wisdom.  If you want better documentation for a particular 
function, find where the existing documentation is (comments or internals 
manual) and propose a patch.  If you think a static function should be 
made non-static so plugins can call it, propose a patch with appropriate 
rationale.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: plugin hooks

[Richard Guenther]

On Tue, Oct 27, 2009 at 4:48 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Richard Guenther wrote:
>>
>> The fix is not to add a hook to replace the pass manager but instead
>> to make the GCC pass manager more flexible itself.
>
> I leave that task to my ICI friends (in CC). They have already an
> implementation, and I believe they probably did propose some patches, or are
> working on it.
>
> However, it is not a small patch.
>
> Also, I still believe that a hook for tha pass manager may make sense to
> those brave souls wanting to experiment new pass managers in plugins, etc.
> Having a hook for that is a way to encourage such crazy experimentations.

The point is that compilation is not driven by the pass manager
but by a bunch of twisted code througout the compiler.  Better
fix that first before adding more twists.

Richard.

Re: plugin hooks

[Basile STARYNKEVITCH]

Hello All

Basile STARYNKEVITCH wrote:
> Ian Lance Taylor wrote:
>> Basile STARYNKEVITCH <basile@starynkevitch.net> writes:
>>
>>> * propose a simple patch to add the PLUGIN_REGISTER_PRAGMA event now?
>>
>> Do this.
> 
> Will do probably tommorow or this evening!


I already did sent an initial proposal for the patch.
http://gcc.gnu.org/ml/gcc-patches/2009-10/msg01623.html

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Rafael Espindola]

> We should add hooks as we find plugins that need them.  Simply adding
> a laundry list of hooks that we think might be needed will most likely
> cause us to overdesign.  We know that we can write interesting plugins
> today, so we're not missing anything critical.
>
> I agree that it should be OK to add hooks in stage 3, if they are
> clearly safe and clearly needed.

Sorry for being late in joining the thread. I fully agree with this.
We should add hooks as someone finds a need for it.

Changes like this are likely to be safe, but I am not sure if they are
stage3 OK. We should probably look at it in a case by case basis.

> Ian
>

Cheers,
-- 
Rafael Ávila de Espíndola

Re: plugin hooks

[Rafael Espindola]

> I believe there is a strong chicken & egg issue here. Ian is suggesting that
> hooks should be added only when an existing plugin would need them, but I
> believe that on the contrary plugins won't appear if they don't have a
> sufficient set of hooks. People won't even bother to make plugins if they
> feel the set of hooks is blantly unsufficient. [They will either
> experiment on their own GCC branch, or avoid GCC entirely, for instance
> using LLVM instead]. But working on one's branch is much more painful
> than coding a plugin!

I don't see a chicken and egg problem. Your work on making the GC
accessible to plugins is a good example why it is not :-)


> I have a concrete example here: plugin-specific pragmas (see
> PLUGIN_REGISTER_PRAGMA on http://gcc.gnu.org/wiki/plugin%20hook for
> details)
>
> I have two imaginary use cases here.

Sorry, the point is precisely that they should be concrete :-)

If you are actually coding a plugin and get to the stage "ok, now I
need a new hook", please send a patch adding the hook and a
description of what the plugin is trying to do.

Cheers,
-- 
Rafael Ávila de Espíndola

Re: plugin hooks

[Rafael Espindola]

> From several face to face discussions with ICI CTuning people (that it
> Grigori Fursin, Albert Cohen, Zbignew Chamski; all 3 are in CC of this
> reply) I understood that GCC ICI is exactly doing that: it has replaced GCC
> pass manager by its own, which invokes all the existing GCC passes in some
> better order (or better number of invocation) and produces better generated
> code (the tuning is done by machine learning techniques; they did run their
> modified GCC a large number of times and did build a database of measures.).
>
> I cannot explain more in details what GCC ICI is actually doing. I hope my
> ICI friends will answer here. Perhaps even Zbignew's latest patches are
> related to the answer.

We talked a bit in the summit about it. I think that the correct
solution is to make the gcc pass manager have the flexibility they
need so that their plugin can just call into the regular pass manager.

> Regards.
>
>
> --
> Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
> email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
> 8, rue de la Faiencerie, 92340 Bourg La Reine, France
> *** opinions {are only mines, sont seulement les miennes} ***
>

Cheers,
-- 
Rafael Ávila de Espíndola

Re: plugin hooks

[Basile STARYNKEVITCH]

Hello All

Rafael Espindola wrote:
>> I have a concrete example here: plugin-specific pragmas (see
>> PLUGIN_REGISTER_PRAGMA on http://gcc.gnu.org/wiki/plugin%20hook for
>> details)
>>
>> I have two imaginary use cases here.
> 

There are not so imaginary. I am sure I will need plugin specific 
pragmas in 2010. I explained why. But I also told that it is more than 
one man-year effort to exhibit a realistic plugin needing 
PLUGIN_REGISTER_PRAGMA


>> People won't even bother to make plugins if they
>> > feel the set of hooks is blantly unsufficient. [They will either
>> > experiment on their own GCC branch, or avoid GCC entirely, for instance
>> > using LLVM instead]. But working on one's branch is much more painful
>> > than coding a plugin!
> 
> I don't see a chicken and egg problem. Your work on making the GC
> accessible to plugins is a good example why it is not  :-) 

Honestly, the [enormous in my view] effort I did make to push that GGC 
related features into the trunk have been disproportionately important 
w.r.t. the result. My work on GGC plugin events is precisely a good 
example of why I need to anticipate my plugin needs.


> Sorry, the point is precisely that they should be concrete :-)
> 
> If you are actually coding a plugin and get to the stage "ok, now I
> need a new hook", please send a patch adding the hook and a
> description of what the plugin is trying to do.


You are forgetting an important point in GCC.

The majority of GCC users won't compile GCC. Actually, I have concrete 
experience that most GCC users are not even able (that is willing to 
make the effort) to compile it. They will use a GCC in some distributed 
*binary* form (e.g. the gcc-4.5 package of some future Ubuntu, Debian, 
... distribution, released in mid 2010).

So in christmas 2010, a recently released Ubuntu contains the gcc-4.5 
package (the compiler) and the gcc-dev-4.5 package (enough to compile a 
gcc 4.5 plugin).

Now I (Basile) have developed a plugin by that time which will use a 
PLUGIN_REGISTER_PRAGMA. If I requires its users to compile a gcc-trunk 
(since only after mid 2010 will I be able to really demonstrate a valid 
use of PLUGIN_REGISTER_PRAGMA), nobody -I really think not a single 
person- will use my plugin. If my plugin runs on a standard 4.5 binary 
distribution of gcc, I might have a few users!


I really think we should add more plugin hooks *now* for gcc-4.5. They 
really cannot wait the future 4.6 release (in 2011, 2012?), especially 
since the patches adding them are so short. And by definition, these 
patches can't break anything [provided they don't hurt performance 
measurably on compilation not using plugins, this is why I wrote about 
plugin events whose invoke_plugin_callbacks is not called a million 
times but only a few dozen times... or perhaps a few hundred times on a 
big compilation. I mean that one additional invoke_plugin_callbacks at 
each pass execution is probably acceptable]. So adding new events should 
be done now. The PLUGIN_REGISTER_PRAGMA should not hurt performance of 
GCC, since its invoke_plugin_callbacks is called once for a whole cc1 
process!

And in a couple of years, indeed we might want to obsolete some plugin 
events (like we do obsolete some other features of GCC). Only time can 
tell which events are really useful.

Mind you, the buying point for plugins is to avoid asking users to 
compile the entire GCC compiler but still have some extension ability. 
Nobody likes to do that, except we GCC hackers, and Linux distribution 
makers.

Sometimes I have the intuition that several GCC people are afraid of 
plugins. I cannot understand why. When the Adobe Flash plugin crash in 
my Mozilla, I don't curse the Mozilla team, but Adobe (and of course the 
many sites using too much that proprietary technology). The main concern 
of the GCC community should not be abuse of plugins, it should be the 
survival of GCC in the long term.

I really think we should help as much as possible plugins to be a 
success! And in my opinion, that means provide the necessary hooks in 
4.5 [especially those for which the patch is tiny, like 
PLUGIN_REGISTER_PRAGMA] not in the trunk in a couple of years. This 
takes into account that GCC releases are quite a rare event (there is no 
major GCC release every 3 months).


Regards


PS. Are we absolutely sure that GCC will still be dominant in the 
compiler landscape five or ten years from now? I am not!

PPS. I think that most north-american GCC hackers grossly underestimate 
the effort needed to push even a minor patch for e.g. european hackers. 
There are lot of issues (English language, timezone, lack of near 
collegues working on GCC and which I could meet in a few minutes, major 
cultural differences, market differences [most processor makers are not 
European]...) to consider

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin hooks

[Richard Guenther]

On Wed, Oct 28, 2009 at 8:34 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
> Hello All
>
> Rafael Espindola wrote:
>>>
>>> I have a concrete example here: plugin-specific pragmas (see
>>> PLUGIN_REGISTER_PRAGMA on http://gcc.gnu.org/wiki/plugin%20hook for
>>> details)
>>>
>>> I have two imaginary use cases here.
>>
>
> There are not so imaginary. I am sure I will need plugin specific pragmas in
> 2010. I explained why. But I also told that it is more than one man-year
> effort to exhibit a realistic plugin needing PLUGIN_REGISTER_PRAGMA
>
>
>>> People won't even bother to make plugins if they
>>> > feel the set of hooks is blantly unsufficient. [They will either
>>> > experiment on their own GCC branch, or avoid GCC entirely, for instance
>>> > using LLVM instead]. But working on one's branch is much more painful
>>> > than coding a plugin!
>>
>> I don't see a chicken and egg problem. Your work on making the GC
>> accessible to plugins is a good example why it is not  :-)
>
> Honestly, the [enormous in my view] effort I did make to push that GGC
> related features into the trunk have been disproportionately important
> w.r.t. the result. My work on GGC plugin events is precisely a good example
> of why I need to anticipate my plugin needs.
>
>
>> Sorry, the point is precisely that they should be concrete :-)
>>
>> If you are actually coding a plugin and get to the stage "ok, now I
>> need a new hook", please send a patch adding the hook and a
>> description of what the plugin is trying to do.
>
>
> You are forgetting an important point in GCC.
>
> The majority of GCC users won't compile GCC. Actually, I have concrete
> experience that most GCC users are not even able (that is willing to make
> the effort) to compile it. They will use a GCC in some distributed *binary*
> form (e.g. the gcc-4.5 package of some future Ubuntu, Debian, ...
> distribution, released in mid 2010).
>
> So in christmas 2010, a recently released Ubuntu contains the gcc-4.5
> package (the compiler) and the gcc-dev-4.5 package (enough to compile a gcc
> 4.5 plugin).
>
> Now I (Basile) have developed a plugin by that time which will use a
> PLUGIN_REGISTER_PRAGMA. If I requires its users to compile a gcc-trunk
> (since only after mid 2010 will I be able to really demonstrate a valid use
> of PLUGIN_REGISTER_PRAGMA), nobody -I really think not a single person- will
> use my plugin. If my plugin runs on a standard 4.5 binary distribution of
> gcc, I might have a few users!
>
>
> I really think we should add more plugin hooks *now* for gcc-4.5. They
> really cannot wait the future 4.6 release (in 2011, 2012?), especially since

I don't agree at all.  Also I have the feeling that plugins will in
the end result
in less developers on the core GCC infrastructure and thus plugins are bad
for GCC.

Richard.

Re: plugin hooks

[Basile STARYNKEVITCH]

Richard Guenther wrote:
> On Wed, Oct 28, 2009 at 8:34 PM, Basile STARYNKEVITCH
> <basile@starynkevitch.net> wrote:
>>
>>
>> I really think we should add more plugin hooks *now* for gcc-4.5. They
>> really cannot wait the future 4.6 release (in 2011, 2012?), especially since
> 
> I don't agree at all.  Also I have the feeling that plugins will in
> the end result
> in less developers on the core GCC infrastructure and thus plugins are bad
> for GCC.


I respectfully disagree with Richard, and have the following opinion.

Because of plugins ability, some *more* people will learn a bit the 
internals of GCC (enough to code their plugin) and amongst them, a few 
people will eventually -later- join the core GCC developers (e.g. 
because they need a hook). So the effect of plugins on the size of the 
GCC community should be a gradual increase, not a regression!

The point is that plugins will enable unexpected usage of GCC, and this 
is positive (and will bring gradually more people inside the community).

Regards.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

RE: plugin hooks

[Joern Rennecke]

Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> Also, I hope that we will start collaborating with Joern Rennecke in  
>  a few weeks to update the ICI
> and GSOC'09
>
> developments based on the recent feedback to see if we can move it   
> to the mainline ...

We still need a branch name for that.  Since GCC is currently in phase 3,
I suppose we should plan on re-baselining at least once - re-baselining
from 4.6 experimental mainline after 4.5 is branched - so instead of
a gcc version number a branch date seems more suitable.

I.e. if the branch were to be created today, I propose to use the name
branches/ici-20091102-branch for the branch.

RE: plugin hooks

[Grigori Fursin]

Hi Joern,

I think, that's very reasonable.

Just as we agreed, wait a few days until you are covered by the INRIA agreement
to transfer copyright to FSF ... I should be able to tell you more within a few
days ...

Also, I added Yuanjie, Liang (they did GSOC developments) and Yuri in CC since
they will help with testing and extending the ICI and GSOC developments...

Cheers,
Grigori

> -----Original Message-----
> From: Joern Rennecke [mailto:amylaar@spamcop.net]
> Sent: Monday, November 02, 2009 1:50 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; Yuri Kashnikoff
> Subject: RE: plugin hooks
> 
> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> > Also, I hope that we will start collaborating with Joern Rennecke in
> >  a few weeks to update the ICI
> > and GSOC'09
> >
> > developments based on the recent feedback to see if we can move it
> > to the mainline ...
> 
> We still need a branch name for that.  Since GCC is currently in phase 3,
> I suppose we should plan on re-baselining at least once - re-baselining
> from 4.6 experimental mainline after 4.5 is branched - so instead of
> a gcc version number a branch date seems more suitable.
> 
> I.e. if the branch were to be created today, I propose to use the name
> branches/ici-20091102-branch for the branch.

[plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi all,

Just wanted to announce that we will now collaborative with Joern until the end of 
December to sync the GCC GSOC'09 developments and ICI with the mainline GCC. Joern 
can transfer all the code to FSF since he is covered by INRIA agreement 
until the end of December at least.

As a reminder, the goal of our developments is to sync our Interactive Compilation
Interface with the current plugin framework to enable already established research community
continue working with ICI directly in the official GCC without any changes. We would
also like to continue attracting more researchers to GCC and enable quick prototyping 
of research ideas in a real production quality compiler while avoiding a long process
of hacking it ;) ... This in turn should help us move useful research ideas back to
GCC much faster to improve program performance, compilation time, code size, etc ...

Furthermore, this can eventually help us clean and modularize compiler, and hopefully
eventually provide a clean interface and full documentation of all modules (I think we
all agreed on that in the last few years and there are already multiple on-going projects 
in that direction ...)

Our current plan of work:

* Create ICI branch, create related Wiki page on GCC website (or reuse/update the current one), 
agree on testing methodology, agree on plugin location

* Sync current ICI and pass manager hooks with the branch (Zbigniew can advise on that).
This will allow users select/deselect/reorder current passes to continue research on adaptive compilation 
technology to improve performance, code size, compilation time or add new passes (generic function cloning,
feature extraction from the MILEPOST project, analysis and instrumentation routines, passes written in MELT, etc).
Our latest plugins also use XML to dump current info about executed passes, change them and put them back to the compiler...

* Check that ICI uses the same calling convention as current plugins (compiler flags, etc)

* Sync function cloning and basic program instrumentation (GSOC'09 extensions by Liang and Yuanjie - they can help with that).
This will allow users create multiversioning applications to adapt to different architectures or program behavior at run-time,
or even create multi-target applications such as GPU/CELL/accelerators, etc. It is implemented as a generic IP pass that can
be called on demand through ICI and the clones can be optimized differently selecting/deselecting/reordering passes through ICI...

All the above is already in the reasonably stable stage, requires minimal modifications to GCC and mainly requires testing and 
moving to the mainline. Me and Yuri (in CC) are testing pass selection and reordering mechanism in GCC...

* Check fine-grain optimization selection in GCC through ICI (another GCC GSOC'09 development by Yunajie and Liang). The idea
is to enable selection and tuning of individual transformations such as unrolling, vectorization, tiling, inlining, etc through 
plugins. The prototype is available, however it seems that it slows down the compiler so we have to check what to do if we have
too many hooks in the compiler - we need some simple testing mechanism that will turn them off if not used ...
So, before moving those developments, we have to thoroughly investigate this matter since we can't afford to slow down GCC ;) ..

* Check/add MILEPOST feature extractor - it just a pass that is not invoked unless called by ICI so it is simple to add but 
we need to show the useful usage cases ...

The latest version of ICI and GSOC developments for GCC 4.4.0 is available at SVN:
http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
(gcc-4.4.0-ici-2.0-adapt)

The stable version of MILEPOST GCC with ICI and feature extractor from IBM is available at:
http://sourceforge.net/projects/gcc-ici/files/

The latest documentation for ICI and GSOC developments is available here:
http://ctuning.org/wiki/index.php/CTools:ICI:Documentation
http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Function_cloning_and_program_instrumentation
http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Fine_grain_tuning

The idea is to move all the these developments and the documentation to the GCC branch and GCC site now

By the way, just a small reminder that the submission deadline for the GROW'10 workshop
is coming soon: http://ctuning.org/workshop-grow10

We hope to have a 1-hour open discussion at the end of the GROW workshop about plugins, LTO and
further R&D opportunities in GCC ...

Also, until the end of December, the main contact point about these developments will be Joern ...

Will keep in touch,
Grigori
 

> -----Original Message-----
> From: Joern Rennecke [mailto:amylaar@spamcop.net]
> Sent: Monday, November 02, 2009 1:50 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; Yuri Kashnikoff
> Subject: RE: plugin hooks
> 
> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> > Also, I hope that we will start collaborating with Joern Rennecke in
> >  a few weeks to update the ICI
> > and GSOC'09
> >
> > developments based on the recent feedback to see if we can move it
> > to the mainline ...
> 
> We still need a branch name for that.  Since GCC is currently in phase 3,
> I suppose we should plan on re-baselining at least once - re-baselining
> from 4.6 experimental mainline after 4.5 is branched - so instead of
> a gcc version number a branch date seems more suitable.
> 
> I.e. if the branch were to be created today, I propose to use the name
> branches/ici-20091102-branch for the branch.

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> The latest version of ICI and GSOC developments for GCC 4.4.0 is   
> available at SVN:
> http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
> (gcc-4.4.0-ici-2.0-adapt)

Can I start with gcc-4.4.0-ici-2.0 or is this now outdated relative
to gcc-4.4.0-ici-2.0-adapt ?

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Joern,

Normally, there should be no new changes in the ICI2.0 API during GSOC'09 developments
(i.e. in adapt branch), just new hooks. 
Yuanjie and Liang, can you please confirm that ASAP?..

Also, I am trying to figure out if GCC 4.5 branch is still open or not?..

The thing is that I think Zbigniew sent a first patch of the ICI2.0 for GCC 4.5
but I don't remember now what happened with that since I was not available for a few months ...

Since the changes are really minimal on top of plugin branch on GCC 4.5 and we had been discussing
that with Diego for a while (in CC), maybe we can still resubmit the ICI2.0 to the GCC 4.5 or check
what was the status of the previous patch and do all the rest (function cloning/program instrumentation) 
for GCC 4.6?..

If the GCC 4.5 is officially closed, we can start working on the next version 
and as soon as Yuanjie and Liang reply we take either gcc-4.4.0-ici-2.0 or gcc-4.4.0-ici-2.0-adapt ...

Cheers,
Grigori



> -----Original Message-----
> From: Joern Rennecke [mailto:amylaar@spamcop.net]
> Sent: Thursday, November 05, 2009 2:26 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff';
> Yuanjie Huang; Liang Peng; announce@hipeac.net; ctuning-announce@googlegroups.com;
> dorit@il.ibm.com; Mircea Namolaru; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> > The latest version of ICI and GSOC developments for GCC 4.4.0 is
> > available at SVN:
> > http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
> > (gcc-4.4.0-ici-2.0-adapt)
> 
> Can I start with gcc-4.4.0-ici-2.0 or is this now outdated relative
> to gcc-4.4.0-ici-2.0-adapt ?

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Yuanjie Huang]

Hi all,

There is a change in the parameter API, which is used to add a type
information to the parameters.
I've document the changes here:
http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Fine_grain_tuning#ICI_Parameter_Handling_APIs

Cheers,
Yuanjie

On Thu, Nov 5, 2009 at 9:54 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
> Hi Joern,
>
> Normally, there should be no new changes in the ICI2.0 API during GSOC'09 developments
> (i.e. in adapt branch), just new hooks.
> Yuanjie and Liang, can you please confirm that ASAP?..
>
> Also, I am trying to figure out if GCC 4.5 branch is still open or not?..
>
> The thing is that I think Zbigniew sent a first patch of the ICI2.0 for GCC 4.5
> but I don't remember now what happened with that since I was not available for a few months ...
>
> Since the changes are really minimal on top of plugin branch on GCC 4.5 and we had been discussing
> that with Diego for a while (in CC), maybe we can still resubmit the ICI2.0 to the GCC 4.5 or check
> what was the status of the previous patch and do all the rest (function cloning/program instrumentation)
> for GCC 4.6?..
>
> If the GCC 4.5 is officially closed, we can start working on the next version
> and as soon as Yuanjie and Liang reply we take either gcc-4.4.0-ici-2.0 or gcc-4.4.0-ici-2.0-adapt ...
>
> Cheers,
> Grigori
>
>
>
>> -----Original Message-----
>> From: Joern Rennecke [mailto:amylaar@spamcop.net]
>> Sent: Thursday, November 05, 2009 2:26 PM
>> To: Grigori Fursin
>> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
>> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff';
>> Yuanjie Huang; Liang Peng; announce@hipeac.net; ctuning-announce@googlegroups.com;
>> dorit@il.ibm.com; Mircea Namolaru; 'Diego Novillo'
>> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>>
>> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
>> > The latest version of ICI and GSOC developments for GCC 4.4.0 is
>> > available at SVN:
>> > http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
>> > (gcc-4.4.0-ici-2.0-adapt)
>>
>> Can I start with gcc-4.4.0-ici-2.0 or is this now outdated relative
>> to gcc-4.4.0-ici-2.0-adapt ?
>
>
>
>



-- 
Yuanjie HUANG

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Thanks, Yuanjie.
By the way, I think you still didn't reply which fine-grain
optimizations did you manage to add, i.e. you could see
that they properly transform the code ...
Thanks again,
Grigori

> -----Original Message-----
> From: huangyuanjie@gmail.com [mailto:huangyuanjie@gmail.com] On Behalf Of Yuanjie Huang
> Sent: Thursday, November 05, 2009 3:34 PM
> To: Grigori Fursin
> Cc: Joern Rennecke; Liang Peng; Zbigniew Chamski; GCC Mailing List; ctuning-
> discussions@googlegroups.com; Yuri Kashnikoff; Diego Novillo
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> Hi all,
> 
> There is a change in the parameter API, which is used to add a type
> information to the parameters.
> I've document the changes here:
> http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Fine_grain_tuning#ICI_Parameter_H
> andling_APIs
> 
> Cheers,
> Yuanjie
> 
> On Thu, Nov 5, 2009 at 9:54 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
> > Hi Joern,
> >
> > Normally, there should be no new changes in the ICI2.0 API during GSOC'09 developments
> > (i.e. in adapt branch), just new hooks.
> > Yuanjie and Liang, can you please confirm that ASAP?..
> >
> > Also, I am trying to figure out if GCC 4.5 branch is still open or not?..
> >
> > The thing is that I think Zbigniew sent a first patch of the ICI2.0 for GCC 4.5
> > but I don't remember now what happened with that since I was not available for a few months
> ...
> >
> > Since the changes are really minimal on top of plugin branch on GCC 4.5 and we had been
> discussing
> > that with Diego for a while (in CC), maybe we can still resubmit the ICI2.0 to the GCC 4.5
> or check
> > what was the status of the previous patch and do all the rest (function cloning/program
> instrumentation)
> > for GCC 4.6?..
> >
> > If the GCC 4.5 is officially closed, we can start working on the next version
> > and as soon as Yuanjie and Liang reply we take either gcc-4.4.0-ici-2.0 or gcc-4.4.0-ici-
> 2.0-adapt ...
> >
> > Cheers,
> > Grigori
> >
> >
> >
> >> -----Original Message-----
> >> From: Joern Rennecke [mailto:amylaar@spamcop.net]
> >> Sent: Thursday, November 05, 2009 2:26 PM
> >> To: Grigori Fursin
> >> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor';
> 'GCC
> >> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff';
> >> Yuanjie Huang; Liang Peng; announce@hipeac.net; ctuning-announce@googlegroups.com;
> >> dorit@il.ibm.com; Mircea Namolaru; 'Diego Novillo'
> >> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> >>
> >> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> >> > The latest version of ICI and GSOC developments for GCC 4.4.0 is
> >> > available at SVN:
> >> > http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
> >> > (gcc-4.4.0-ici-2.0-adapt)
> >>
> >> Can I start with gcc-4.4.0-ici-2.0 or is this now outdated relative
> >> to gcc-4.4.0-ici-2.0-adapt ?
> >
> >
> >
> >
> 
> 
> 
> --
> Yuanjie HUANG

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Ian Lance Taylor]

"Grigori Fursin" <grigori.fursin@inria.fr> writes:

> Also, I am trying to figure out if GCC 4.5 branch is still open or not?..

gcc 4.5 is in stage 3, which normally means that no new features are
permitted.  That said, plugins are a new feature in gcc 4.5, so the
release managers are likely to grant some leeway for changing them.
But major rearrangements or modifications of existing code are not
appropriate at this time.

Ian

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Yuanjie HUANG]

Sorry, I missed the question. I have enabeld  fine-grain tuning in
loop unrolling, strip mine and blocking in the GSoC project.

Cheers,
Yuanjie

On Fri, Nov 6, 2009 at 12:27 AM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
> Thanks, Yuanjie.
> By the way, I think you still didn't reply which fine-grain
> optimizations did you manage to add, i.e. you could see
> that they properly transform the code ...
> Thanks again,
> Grigori
>
>> -----Original Message-----
>> From: huangyuanjie@gmail.com [mailto:huangyuanjie@gmail.com] On Behalf Of Yuanjie Huang
>> Sent: Thursday, November 05, 2009 3:34 PM
>> To: Grigori Fursin
>> Cc: Joern Rennecke; Liang Peng; Zbigniew Chamski; GCC Mailing List; ctuning-
>> discussions@googlegroups.com; Yuri Kashnikoff; Diego Novillo
>> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>>
>> Hi all,
>>
>> There is a change in the parameter API, which is used to add a type
>> information to the parameters.
>> I've document the changes here:
>> http://ctuning.org/wiki/index.php/CTools:ICI:Projects:GSOC09:Fine_grain_tuning#ICI_Parameter_H
>> andling_APIs
>>
>> Cheers,
>> Yuanjie
>>
>> On Thu, Nov 5, 2009 at 9:54 PM, Grigori Fursin <grigori.fursin@inria.fr> wrote:
>> > Hi Joern,
>> >
>> > Normally, there should be no new changes in the ICI2.0 API during GSOC'09 developments
>> > (i.e. in adapt branch), just new hooks.
>> > Yuanjie and Liang, can you please confirm that ASAP?..
>> >
>> > Also, I am trying to figure out if GCC 4.5 branch is still open or not?..
>> >
>> > The thing is that I think Zbigniew sent a first patch of the ICI2.0 for GCC 4.5
>> > but I don't remember now what happened with that since I was not available for a few months
>> ...
>> >
>> > Since the changes are really minimal on top of plugin branch on GCC 4.5 and we had been
>> discussing
>> > that with Diego for a while (in CC), maybe we can still resubmit the ICI2.0 to the GCC 4.5
>> or check
>> > what was the status of the previous patch and do all the rest (function cloning/program
>> instrumentation)
>> > for GCC 4.6?..
>> >
>> > If the GCC 4.5 is officially closed, we can start working on the next version
>> > and as soon as Yuanjie and Liang reply we take either gcc-4.4.0-ici-2.0 or gcc-4.4.0-ici-
>> 2.0-adapt ...
>> >
>> > Cheers,
>> > Grigori
>> >
>> >
>> >
>> >> -----Original Message-----
>> >> From: Joern Rennecke [mailto:amylaar@spamcop.net]
>> >> Sent: Thursday, November 05, 2009 2:26 PM
>> >> To: Grigori Fursin
>> >> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor';
>> 'GCC
>> >> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff';
>> >> Yuanjie Huang; Liang Peng; announce@hipeac.net; ctuning-announce@googlegroups.com;
>> >> dorit@il.ibm.com; Mircea Namolaru; 'Diego Novillo'
>> >> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
>> >>
>> >> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
>> >> > The latest version of ICI and GSOC developments for GCC 4.4.0 is
>> >> > available at SVN:
>> >> > http://gcc-ici.svn.sourceforge.net/viewvc/gcc-ici/branches/
>> >> > (gcc-4.4.0-ici-2.0-adapt)
>> >>
>> >> Can I start with gcc-4.4.0-ici-2.0 or is this now outdated relative
>> >> to gcc-4.4.0-ici-2.0-adapt ?
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Yuanjie HUANG
>
>
>
>



-- 
Yuanjie HUANG

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Sure, thanks for the info, Ian.

The ICI changes are very small and seems that they already had been pre-reviewed.
Joern is taking care of that now.

As for GSOC'09 developments, it seems that they will require more changes 
(though improvement to generic function cloning may be fairly small) so will
keep them for the next version ...

Cheers,
Grigori

> -----Original Message-----
> From: Ian Lance Taylor [mailto:iant@google.com]
> Sent: Friday, November 06, 2009 12:34 AM
> To: Grigori Fursin
> Cc: 'Joern Rennecke'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List';
> ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> "Grigori Fursin" <grigori.fursin@inria.fr> writes:
> 
> > Also, I am trying to figure out if GCC 4.5 branch is still open or not?..
> 
> gcc 4.5 is in stage 3, which normally means that no new features are
> permitted.  That said, plugins are a new feature in gcc 4.5, so the
> release managers are likely to grant some leeway for changing them.
> But major rearrangements or modifications of existing code are not
> appropriate at this time.
> 
> Ian

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

Quoting Grigori Fursin <grigori.fursin@inria.fr>:

> Sure, thanks for the info, Ian.
>
> The ICI changes are very small and seems that they already had been   
> pre-reviewed.
> Joern is taking care of that now.

After the pass-name changes are taken out (this is the pre-reviewed bit
I'm currently trying to contribute to mainline), there is about 65 KB of
unidiff left from the ICI 2.0 branch.  That code needs a bit of GNU
reformatting, so I doubt it has been reviewed in its entirety.
As it looks like the pass-name changes can go into mainline soon, I want
to hold off creating a branch for the rest of the code till the former
changes are in mainline.

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Ok, that's fine ...

By the way, the ICI itself should not touch too much of the GCC code - 
it's an API on top of current plugin API. After resolving the issue
with the pass names that is necessary for ICI to work properly,
could you actually please check what will be all the necessary current 
GCC modifications to add ICI ... If it really modifies some critical
parts, we can leave it for the next version then ...

Grigori

> -----Original Message-----
> From: ctuning-discussions@googlegroups.com [mailto:ctuning-discussions@googlegroups.com] On
> Behalf Of Joern Rennecke
> Sent: Friday, November 06, 2009 3:54 PM
> To: Grigori Fursin
> Cc: 'Ian Lance Taylor'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List';
> ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> 
> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> 
> > Sure, thanks for the info, Ian.
> >
> > The ICI changes are very small and seems that they already had been
> > pre-reviewed.
> > Joern is taking care of that now.
> 
> After the pass-name changes are taken out (this is the pre-reviewed bit
> I'm currently trying to contribute to mainline), there is about 65 KB of
> unidiff left from the ICI 2.0 branch.  That code needs a bit of GNU
> reformatting, so I doubt it has been reviewed in its entirety.
> As it looks like the pass-name changes can go into mainline soon, I want
> to hold off creating a branch for the rest of the code till the former
> changes are in mainline.
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "ctuning-
> discussions" group.
> To post to this group, send email to ctuning-discussions@googlegroups.com
> To unsubscribe from this group, send email to ctuning-discussions+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=en
> -~----------~----~----~----~------~----~------~--~---

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

Quoting Grigori Fursin <grigori.fursin@inria.fr>:

>
> Ok, that's fine ...
>
> By the way, the ICI itself should not touch too much of the GCC code -
> it's an API on top of current plugin API. After resolving the issue
> with the pass names that is necessary for ICI to work properly,
> could you actually please check what will be all the necessary current
> GCC modifications to add ICI ... If it really modifies some critical
> parts, we can leave it for the next version then ...

I think things like sticking -export-dynamic into make files rules are
a portability showstopper.
There should be an autoconf test to check if the option is available.

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

True, but I guess it should have been resolved with the current plugin mechanism in GCC 4.5
so maybe we do not need this option anymore ?..

Cheers,
Grigori

> -----Original Message-----
> From: ctuning-discussions@googlegroups.com [mailto:ctuning-discussions@googlegroups.com] On
> Behalf Of Joern Rennecke
> Sent: Friday, November 06, 2009 4:28 PM
> To: ctuning-discussions@googlegroups.com; Grigori Fursin
> Cc: 'Ian Lance Taylor'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List';
> 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> 
> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> 
> >
> > Ok, that's fine ...
> >
> > By the way, the ICI itself should not touch too much of the GCC code -
> > it's an API on top of current plugin API. After resolving the issue
> > with the pass names that is necessary for ICI to work properly,
> > could you actually please check what will be all the necessary current
> > GCC modifications to add ICI ... If it really modifies some critical
> > parts, we can leave it for the next version then ...
> 
> I think things like sticking -export-dynamic into make files rules are
> a portability showstopper.
> There should be an autoconf test to check if the option is available.
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "ctuning-
> discussions" group.
> To post to this group, send email to ctuning-discussions@googlegroups.com
> To unsubscribe from this group, send email to ctuning-discussions+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=en
> -~----------~----~----~----~------~----~------~--~---

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Basile STARYNKEVITCH]

Joern Rennecke wrote:
> Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> 
>> Sure, thanks for the info, Ian.
>>
>> The ICI changes are very small and seems that they already had been  
>> pre-reviewed.
>> Joern is taking care of that now.
> 
> After the pass-name changes are taken out (this is the pre-reviewed bit
> I'm currently trying to contribute to mainline), there is about 65 KB of
> unidiff left from the ICI 2.0 branch.  That code needs a bit of GNU
> reformatting, so I doubt it has been reviewed in its entirety.
> As it looks like the pass-name changes can go into mainline soon, I want
> to hold off creating a branch for the rest of the code till the former
> changes are in mainline.


My suggestion would be to try pushing the minimal missing plugin hooks needed for ICI into the trunk (that could perhaps 
happen in gcc 4.5 timeframe), and adjust your ICI 2.x branch to make it a plugin (perhaps dlopen-ing itself other code).
What I am thinking of is just adding a few more PLUGIN_* events. Even adding a single event take much time in practice. 
http://gcc.gnu.org/ml/gcc/2009-10/msg00547.html http://gcc.gnu.org/ml/gcc/2009-10/msg00548.html

You'll need to

1.) convince the GCC community that your event is useful. This is sometimes not as easy as it seems.
(and I Basile did try a bit to explain what possible hooks could be useful to *ICI* & also MELT). 
http://gcc.gnu.org/ml/gcc/2009-10/msg00549.html

2.) propose a small patch to add it
    2.1) add your PLUGIN_* event in gcc/gcc-plugin.h & gcc/plugin.c
    2.2) just a call to invoke_plugin_callbacks (at the place you need).
    2.3) document it in gcc/doc/plugins.texi
    2.4) make a testcase [this means understanding dejagnu]

3.) wait for your patch to be accepted.

My feeling is that ICI should have a small steps strategy to add the minimal hooks required, and be [like MELT] mostly a 
plugin and/or an experimental branch nobody really cares about.
I am not sure it is realistic to try to push (at least for gcc 4.5, and perhaps even gcc 4.6) a multithousand lines ICI 
patch.... Better push several smaller patches, each adding a PLUGIN_* event, and make most of ICI a plugin itself.

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Basile,

Well, the point of ICI is actually not to hardwire specific events at the beginning 
but allow users/researchers easily add their own hooks referenced by any string 
to simplify prototyping of new ideas. Then if they see that it is useful and they
can persuade the community about that, those hooks go to the mainline and are hardwired
in the PLUGIN_* event ...

We found such usage very convenient in the last few years and during that
time got a considerable feedback about ICI - it is also being used at the moment
in multiple projects that's why we are now trying to add it to the mainline
to help our colleagues use it directly in the mainline GCC and continue research...

Again, several thousands of lines of code in ICI have been related just to manage
event referencing by the string (with hashing, etc) + clean API but it doesn't really
change GCC too much. Actually, I am thinking that maybe we should actually
make ICI as a library and provide only a few necessary hooks/changes to GCC 
to support it. 

Joern and Zbigniew, what do you think about that?..

Cheers,
Grigori


> -----Original Message-----
> From: ctuning-discussions@googlegroups.com [mailto:ctuning-discussions@googlegroups.com] On
> Behalf Of Basile STARYNKEVITCH
> Sent: Friday, November 06, 2009 4:55 PM
> To: Joern Rennecke
> Cc: Grigori Fursin; 'Ian Lance Taylor'; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski';
> 'GCC Mailing List'; ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> 
> Joern Rennecke wrote:
> > Quoting Grigori Fursin <grigori.fursin@inria.fr>:
> >
> >> Sure, thanks for the info, Ian.
> >>
> >> The ICI changes are very small and seems that they already had been
> >> pre-reviewed.
> >> Joern is taking care of that now.
> >
> > After the pass-name changes are taken out (this is the pre-reviewed bit
> > I'm currently trying to contribute to mainline), there is about 65 KB of
> > unidiff left from the ICI 2.0 branch.  That code needs a bit of GNU
> > reformatting, so I doubt it has been reviewed in its entirety.
> > As it looks like the pass-name changes can go into mainline soon, I want
> > to hold off creating a branch for the rest of the code till the former
> > changes are in mainline.
> 
> 
> My suggestion would be to try pushing the minimal missing plugin hooks needed for ICI into the
> trunk (that could perhaps
> happen in gcc 4.5 timeframe), and adjust your ICI 2.x branch to make it a plugin (perhaps
> dlopen-ing itself other code).
> What I am thinking of is just adding a few more PLUGIN_* events. Even adding a single event
> take much time in practice.
> http://gcc.gnu.org/ml/gcc/2009-10/msg00547.html http://gcc.gnu.org/ml/gcc/2009-
> 10/msg00548.html
> 
> You'll need to
> 
> 1.) convince the GCC community that your event is useful. This is sometimes not as easy as it
> seems.
> (and I Basile did try a bit to explain what possible hooks could be useful to *ICI* & also
> MELT).
> http://gcc.gnu.org/ml/gcc/2009-10/msg00549.html
> 
> 2.) propose a small patch to add it
>     2.1) add your PLUGIN_* event in gcc/gcc-plugin.h & gcc/plugin.c
>     2.2) just a call to invoke_plugin_callbacks (at the place you need).
>     2.3) document it in gcc/doc/plugins.texi
>     2.4) make a testcase [this means understanding dejagnu]
> 
> 3.) wait for your patch to be accepted.
> 
> My feeling is that ICI should have a small steps strategy to add the minimal hooks required,
> and be [like MELT] mostly a
> plugin and/or an experimental branch nobody really cares about.
> I am not sure it is realistic to try to push (at least for gcc 4.5, and perhaps even gcc 4.6)
> a multithousand lines ICI
> patch.... Better push several smaller patches, each adding a PLUGIN_* event, and make most of
> ICI a plugin itself.
> 
> Regards.
> 
> --
> Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
> email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
> 8, rue de la Faiencerie, 92340 Bourg La Reine, France
> *** opinions {are only mines, sont seulement les miennes} ***
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "ctuning-
> discussions" group.
> To post to this group, send email to ctuning-discussions@googlegroups.com
> To unsubscribe from this group, send email to ctuning-discussions+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=en
> -~----------~----~----~----~------~----~------~--~---

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Basile STARYNKEVITCH]

Grigori Fursin wrote:
> Hi Basile,
> 
> Well, the point of ICI is actually not to hardwire specific events at the beginning 
> but allow users/researchers easily add their own hooks referenced by any string 
> to simplify prototyping of new ideas. Then if they see that it is useful and they
> can persuade the community about that, those hooks go to the mainline and are hardwired
> in the PLUGIN_* event ...


My intuition was that ICI need a single (or probably a few) hook to replace the entire pass manager....
Then the ICI plugin would call the existing GCC passes in the order it wants....

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Yes, that is true and that is one of the most usable features of ICI so far
since we use it to manipulate passes, add new analysis passes of generic function cloning
(during GSOC'09), etc. So we will need a hard-wired hook to substitute pass manager.

Other hooks such as fine-grain tuning for another GSOC'09 project
are still experimental and are not hardwired, but just referenced by some strings ...

Cheers,
Grigori


> -----Original Message-----
> From: Basile STARYNKEVITCH [mailto:basile@starynkevitch.net]
> Sent: Friday, November 06, 2009 6:00 PM
> To: Grigori Fursin
> Cc: ctuning-discussions@googlegroups.com; 'Joern Rennecke'; 'Ian Lance Taylor'; 'Yuanjie
> Huang'; 'Liang Peng'; 'Zbigniew Chamski'; 'GCC Mailing List'; 'Yuri Kashnikoff'; 'Diego
> Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> Grigori Fursin wrote:
> > Hi Basile,
> >
> > Well, the point of ICI is actually not to hardwire specific events at the beginning
> > but allow users/researchers easily add their own hooks referenced by any string
> > to simplify prototyping of new ideas. Then if they see that it is useful and they
> > can persuade the community about that, those hooks go to the mainline and are hardwired
> > in the PLUGIN_* event ...
> 
> 
> My intuition was that ICI need a single (or probably a few) hook to replace the entire pass
> manager....
> Then the ICI plugin would call the existing GCC passes in the order it wants....
> 
> Regards.
> --
> Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
> email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
> 8, rue de la Faiencerie, 92340 Bourg La Reine, France
> *** opinions {are only mines, sont seulement les miennes} ***

Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

We got a name clash.
In gcc 4.5, passes.c:register_pass is used to register a pass in a
specific place in the pass tree.
In gcc-4.4.0-ici, pass-manager.c:register_pass is used to register a  
pass by name.

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Joern,

Actually, I think I realized why you still see many differences in ICI 2.0 and GCC 4.5 -
my memory is not working well after being away for 2 months and I am trying to remember
what happened with ICI in summer - sorry about that.

So, if I remember correctly, when we saw the new GCC plugin branch open and that it overlaps
considerably with ICI we decided to change all our function names that had the same functionality 
as plugin branch to sync it later with plugin branch goes to the mainline (and that is happening
now in GCC 4.5) while we continued using baseline GCC 4.4.0 for all our further experiments.

In parallel Zbigniew prepared a patch that synchronized ICI with the plugin branch
and if I am correct, he removed all the duplicating functionality so the changes became
quite minimal. BUT we did not use that branch for GSOC projects, so what you see
in the gcc-4.4.0-ici2.0-adapt is the ICI with duplicate functionality of the old plugin branch
and normally the current mainline GCC 4.5. That's I think you start having conflict names.

I sent another private email to Zbigniew and you to check if I am right and if so,
I think you should look at the Zbigniew patch from June and try to update it according
to the current GCC 4.5 mainline plus add small extension from GSOC'09 about types of
the plugin parameters. After that we will synchronize GSOC'09 branches with this new
ICI and with the mainline GCC 4.5 but not versa!!!

Please, tell me if it makes sense. If someone thinks that I am still wrong (particularly
Zbigniew, Yuanjie and Liang) please respond to that so that we could decide on
our work plan ...

By the way, by accident I added 2 emails to the CC due to auto completing of emails:
announce@hipeac.net and ctuning-announce@googlegroups.com - I removed them
since they are not related to this discussion. Replying to this message is now fine
but if you reply to other messages, please remove those emails ...
I am sorry about that...

Cheers,
Grigori



> -----Original Message-----
> From: ctuning-discussions@googlegroups.com [mailto:ctuning-discussions@googlegroups.com] On
> Behalf Of Joern Rennecke
> Sent: Friday, November 06, 2009 6:14 PM
> To: Grigori Fursin
> Cc: 'Zbigniew Chamski'; 'Richard Guenther'; 'Basile STARYNKEVITCH'; 'Ian Lance Taylor'; 'GCC
> Mailing List'; 'Albert Cohen'; ctuning-discussions@googlegroups.com; 'Yuri Kashnikoff';
> Yuanjie Huang; Liang Peng; announce@hipeac.net; ctuning-announce@googlegroups.com;
> dorit@il.ibm.com; Mircea Namolaru; 'Diego Novillo'
> Subject: Re: [plugins-ici-cloning-instrumentation] new GCC plugin developements
> 
> 
> We got a name clash.
> In gcc 4.5, passes.c:register_pass is used to register a pass in a
> specific place in the pass tree.
> In gcc-4.4.0-ici, pass-manager.c:register_pass is used to register a
> pass by name.
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "ctuning-
> discussions" group.
> To post to this group, send email to ctuning-discussions@googlegroups.com
> To unsubscribe from this group, send email to ctuning-discussions+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/ctuning-discussions?hl=en
> -~----------~----~----~----~------~----~------~--~---

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

Quoting Grigori Fursin <grigori.fursin@inria.fr>:
...
> In parallel Zbigniew prepared a patch that synchronized ICI with the  
>  plugin branch
> and if I am correct, he removed all the duplicating functionality so  
>  the changes became
> quite minimal.

Where is that patch?

> BUT we did not use that branch for GSOC projects, so  what you see
> in the gcc-4.4.0-ici2.0-adapt is the ICI with duplicate   
> functionality of the old plugin branch
> and normally the current mainline GCC 4.5. That's I think you start   
> having conflict names.

The two register_pass functions are quite distinct in their functionality
(except that logically the passes.c one should call - directly or
indirectly - the pass_manager.c one).  Both are an interface exposed to
plugins, so we can't simply rename an internal function.  One of the
interfaces has to change.

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Joern,

> > In parallel Zbigniew prepared a patch that synchronized ICI with the
> >  plugin branch
> > and if I am correct, he removed all the duplicating functionality so
> >  the changes became
> > quite minimal.
> 
> Where is that patch?

After looking at all my email I think we only had internal copies of the patch
so it has never been sent out. It seems like marriage can break memory completely -
so I am really sorry for those memory blackouts :( ...

This means that we are getting back to the same task of syncing the ICI2.0 from adapt
SVN to the current GCC. Actually, could you please check what is duplicate functionality
between current GCC 4.5 in the ICI and if we can remove it from ICI ...
It should naturally relate mainly to the plugin functions ...
Only when it's done we can see if ICI will be intrusive or not ...

> > BUT we did not use that branch for GSOC projects, so  what you see
> > in the gcc-4.4.0-ici2.0-adapt is the ICI with duplicate
> > functionality of the old plugin branch
> > and normally the current mainline GCC 4.5. That's I think you start
> > having conflict names.
> 
> The two register_pass functions are quite distinct in their functionality
> (except that logically the passes.c one should call - directly or
> indirectly - the pass_manager.c one).  Both are an interface exposed to
> plugins, so we can't simply rename an internal function.  One of the
> interfaces has to change.

Well, ICI is not in the mainline so it has to change and adapt to the
current mainline... Can you please check if there are other similar cases apart
from this to decide on our changes in one go since we will need to alter
all our plugins due to that (unless we provide some legacy wrapper but
I think at this stage I would prefer to have a clean ICI in GCC)...

Thanks a lot,
Grigori

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

After checking in the patch to provide unique pass names for all passes,
I created

svn://gcc.gnu.org/svn/gcc/branches/ici-20091108-branch

and merged in the patches from:

http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/patch-gcc-4.4.0-ici-2.0

Could you please check that this contains the functionality that we want to
integrate in the first step.

FWIW I know that the code does not conform to the GNU coding standard yet.

I've changed register_pass to register_pass_name to resolve the name clash.
I'm not sure if it should be called register_pass_by_name or something else,
opinions welcome.

Both the gcc 4.5 code and the ICI patches have the concept of events, but
the implementations are so different that the functionality is pretty much
orthogonal.

4.5 has a real C-style interface with an enum to identify the event and
a single pointer for the data.  I.e. low overhead, but rigid typing,
and the different parts of the program presumably find their data by virtue
of using the same header files.
Multiple plugins can register a callback for any event, and all will get
called.  However, since the set of events is hard-coded by an enum
declaration, you can't have two plugins communicating using events that
the gcc binary doesn't know about.

The ICI events feel much more like TCL variables and procedure calls.
Events are identified by strings, and parameters are effectively global
variables found in a hash table.  This is very flexible and can allow
a good deal of ABI stability, but costs a lot of processor cycles as
before an event call the parameters are looked up to be entered in the hash
table, and afterwards they are looked up to be removed, even if no callback
is registered for the event.
Also, when a plugin registers a callback for an ICI event, it overrides any
previous callback registered by another (or even the same) plugin.

I think we could have the ICI event flexibility/stability with lower
overhead if the event sender requests an event identifier number (which
can be allocated after the numbers of the gcc 4.5 static event enum values)
for an event name at or before the first event with that name, and then
sends this identifier number with one or more pointers, which might point
to internal gcc data structures, and a pointer to a function to look up
the address of a named parameter.  The event sender site source code can
then provide information to build the lookup functions at build time,
e.g. using gperf.

I.e.:
/* Call an event with number ID, which is either a value of enum plugin_event,
    or a number allocated for a named event.  If the event named parameters,
    the first parameter after id should be as if declared
    void * (*lookup_func) (const char *, va_list) .
    LOOKUP_FUNC can be passed the name of a parameter as its first argument,
    and a va_list as its second argument, which will be the list of parameters
    after LOOKUP_FUNC, to find the named parameter.  */
void
call_plugin_event (int id, ...)
{
   struct callback_info *callback;
   va_list ap;

   gcc_assert (id < event_id_max);
   callback = plugin_callbacks[id];
   va_start (ap, id);
   for (callback = plugin_callbacks[id]; callback; callback = callback->next)
     (*callback->func) ((void *) ap, callback->user_data);
   va_end (ap);
}

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Joern,

> After checking in the patch to provide unique pass names for all passes,
> I created
> 
> svn://gcc.gnu.org/svn/gcc/branches/ici-20091108-branch
> 
> and merged in the patches from:
> 
> http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/patch-gcc-4.4.0-ici-2.0
> 
> Could you please check that this contains the functionality that we want to
> integrate in the first step.

Thanks a lot, Joern!
I downloaded it and will be gradually checking it.

In the mean time, Yuanjie, Liang and Yuri - could you check this version ASAP
and check that the functionality provided during GSOC'09 developments/pass reordering
work is correct in this version!.. The reason is that since there will be some small
changes, our plugins will have to slightly change as well (see register_pass change)...

By the way, we should keep the track of such changes on the GCC Wiki for ICI ...

> FWIW I know that the code does not conform to the GNU coding standard yet.
> 
> I've changed register_pass to register_pass_name to resolve the name clash.
> I'm not sure if it should be called register_pass_by_name or something else,
> opinions welcome.

I think register_pass_by_name will be better to show what it does now ;) ...

> Both the gcc 4.5 code and the ICI patches have the concept of events, but
> the implementations are so different that the functionality is pretty much
> orthogonal.
> 
> 4.5 has a real C-style interface with an enum to identify the event and
> a single pointer for the data.  I.e. low overhead, but rigid typing,
> and the different parts of the program presumably find their data by virtue
> of using the same header files.
> Multiple plugins can register a callback for any event, and all will get
> called.  However, since the set of events is hard-coded by an enum
> declaration, you can't have two plugins communicating using events that
> the gcc binary doesn't know about.
> 
> The ICI events feel much more like TCL variables and procedure calls.
> Events are identified by strings, and parameters are effectively global
> variables found in a hash table.  This is very flexible and can allow
> a good deal of ABI stability, but costs a lot of processor cycles as
> before an event call the parameters are looked up to be entered in the hash
> table, and afterwards they are looked up to be removed, even if no callback
> is registered for the event.
> Also, when a plugin registers a callback for an ICI event, it overrides any
> previous callback registered by another (or even the same) plugin.

That's very true. Our current idea is that for prototyping of ideas, it is
often fine to slow down the compiler slightly but as soon as development
matures and there are some interesting results, the developers will try
to persuade the GCC community to add the event permanently...

> I think we could have the ICI event flexibility/stability with lower
> overhead if the event sender requests an event identifier number (which
> can be allocated after the numbers of the gcc 4.5 static event enum values)
> for an event name at or before the first event with that name, and then
> sends this identifier number with one or more pointers, which might point
> to internal gcc data structures, and a pointer to a function to look up
> the address of a named parameter.  The event sender site source code can
> then provide information to build the lookup functions at build time,
> e.g. using gperf.
> 
> I.e.:
> /* Call an event with number ID, which is either a value of enum plugin_event,
>     or a number allocated for a named event.  If the event named parameters,
>     the first parameter after id should be as if declared
>     void * (*lookup_func) (const char *, va_list) .
>     LOOKUP_FUNC can be passed the name of a parameter as its first argument,
>     and a va_list as its second argument, which will be the list of parameters
>     after LOOKUP_FUNC, to find the named parameter.  */
> void
> call_plugin_event (int id, ...)
> {
>    struct callback_info *callback;
>    va_list ap;
> 
>    gcc_assert (id < event_id_max);
>    callback = plugin_callbacks[id];
>    va_start (ap, id);
>    for (callback = plugin_callbacks[id]; callback; callback = callback->next)
>      (*callback->func) ((void *) ap, callback->user_data);
>    va_end (ap);
> }

I have been discussing that with Zbigniew some months ago and I think it can be reasonable
to add such functionality on top of current ICI. ICI users will still prototype their ideas
using events referenced by name, however if it works fine and before potential lengthy approval
to get such an event, they can speed up their plugins using this extended functionality... 
The good thing is that their plugins will still be compatible if we will decide to keep associated 
names with the hardwired event number ...

Cheers,
Grigori

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

> After checking in the patch to provide unique pass names for all passes,
> I created
> 
> svn://gcc.gnu.org/svn/gcc/branches/ici-20091108-branch
> 
> and merged in the patches from:
> 
> http://gcc-ici.svn.sourceforge.net/svnroot/gcc-ici/branches/patch-gcc-4.4.0-ici-2.0

By the way, not to forget - we should compile/test GCC with ICI with the following libraries
(orthogonal to ICI but we need them for our experiments):

* gmp & mpfr (for fortran)
* ppl & cloog (for GRAPHITE)

i.e. I configure GCC with the following flags:
configure --enable-languages=c,c++,fortran --with-mpfr=$BUILD_DIR --with-gmp=$BUILD_DIR --with-ppl=$BUILD_DIR --with-cloog=$BUILD_DIR

I used it for the GCC 4.4.0 - maybe some GRAPHITE related flags changed ...
The idea is to have the same setup that we used for our local developments ...

Also, we have been using ICI with C and Fortran a lot, but never checked C++ -
it will be important to check it too ...

Cheers,
Grigori

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

Quoting Joern Rennecke <amylaar@spamcop.net>:
> I think we could have the ICI event flexibility/stability with lower
> overhead if the event sender requests an event identifier number (which
> can be allocated after the numbers of the gcc 4.5 static event enum values)
> for an event name at or before the first event with that name, and then
> sends this identifier number with one or more pointers, which might point
> to internal gcc data structures, and a pointer to a function to look up
> the address of a named parameter.  The event sender site source code can
> then provide information to build the lookup functions at build time,
> e.g. using gperf.

I thought a bit more about this, and decided that using gperf-generated hash
tables is probably overkill.

It is useful to have provisions for the event generator and the event
callback being somewhat out of step, but do we really have to cater
for multiple sources of the same event providing their parameters in
a different way?
If there is only one way to find a parameter with a particular name for
a particular event (for a given combination of compiler binary and plugins),
that this can be defined with an accessor function, which would  
generally be defined in the same module which raises the event.
Actually, if we passed around the dso which raises the event, we could
dynamically look up the accessor function to allow co-existence of different
accessor functions for the same event::parameter tuple, but I don't think
there is a need for that.

Here is an example of how I think we can reduce the overhead while keeping
a good amount of flexibility; in loop-unroll.c we currently have:
         /* Code for loop-unrolling ICI decision enabling.  */
         register_event_parameter ("loop->num", &(loop->num));
         register_event_parameter ("loop->ninsns", &(loop->ninsns));
         register_event_parameter ("loop->av_ninsns", &(loop->av_ninsns));

         register_event_parameter ("loop->lpt_decision.times",  
&(loop->lpt_decision.times));
         register_event_parameter ("loop->lpt_decision.decision",  
&(loop->lpt_decision.decision));
         register_event_parameter ("loop->lpt_decision.unroll_runtime",
               loop->lpt_decision.decision == LPT_UNROLL_RUNTIME ?  
(void *) 1 : (void *) 0);
         register_event_parameter ("loop->lpt_decision.unroll_constant",
               loop->lpt_decision.decision == LPT_UNROLL_CONSTANT ?  
(void *) 1 : (void *) 0);

         call_plugin_event("unroll_feature_change");

         unregister_event_parameter ("loop->num");
         unregister_event_parameter ("loop->ninsns");

         unregister_event_parameter ("loop->av_ninsns");
         unregister_event_parameter ("loop->lpt_decision.times");
         unregister_event_parameter ("loop->lpt_decision.decision");

Instead we could have:

         invoke_plugin_va_callbacks (PLUGIN_UNROLL_FEATURE_CHANGE, loop);
and then accessor functions:
int
plugin_unroll_feature_change_param_loop_num (va_list va)
{
   struct loop *loop = va_arg (va, struct loop *);
   return loop->num;
}

unsigned
plugin_unroll_feature_change_param_loop_ninsns (va_list va)
{
   struct loop *loop = va_arg (va, struct loop *);
   return loop->ninsns;
}

unsigned
plugin_unroll_feature_change_param_loop_av_ninsns (va_list va)
{
   struct loop *loop = va_arg (va, struct loop *);
   return loop->av_ninsns;
}
...
bool
plugin_unroll_feature_change_param_loop_lpt_decision_unroll_runtime  
(va_list va)
{
   struct loop *loop = va_arg (va, struct loop *);
   return loop->lpt_decision.decision == LPT_UNROLL_RUNTIME;
}
...


There is still another practical issue: as I change the ICI infrastructure
to fit better with the existing gcc 4.5 plugin infrastructure,
the ICI plugins must be adapted to keep working.
As we are trying to have something working in a short time frame, I think
I should pick one plugin and modify it in lock-step with the infrastructure
to demonstrate how it is supposed to work.

Do you think the adapt.c plugin is suitable for that purpose?

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Joern,

> > I think we could have the ICI event flexibility/stability with lower
> > overhead if the event sender requests an event identifier number (which
> > can be allocated after the numbers of the gcc 4.5 static event enum values)
> > for an event name at or before the first event with that name, and then
> > sends this identifier number with one or more pointers, which might point
> > to internal gcc data structures, and a pointer to a function to look up
> > the address of a named parameter.  The event sender site source code can
> > then provide information to build the lookup functions at build time,
> > e.g. using gperf.
> 
> I thought a bit more about this, and decided that using gperf-generated hash
> tables is probably overkill.
> 
> It is useful to have provisions for the event generator and the event
> callback being somewhat out of step, but do we really have to cater
> for multiple sources of the same event providing their parameters in
> a different way?
> If there is only one way to find a parameter with a particular name for
> a particular event (for a given combination of compiler binary and plugins),
> that this can be defined with an accessor function, which would
> generally be defined in the same module which raises the event.
> Actually, if we passed around the dso which raises the event, we could
> dynamically look up the accessor function to allow co-existence of different
> accessor functions for the same event::parameter tuple, but I don't think
> there is a need for that.
> 
> Here is an example of how I think we can reduce the overhead while keeping
> a good amount of flexibility; in loop-unroll.c we currently have:
>          /* Code for loop-unrolling ICI decision enabling.  */
>          register_event_parameter ("loop->num", &(loop->num));
>          register_event_parameter ("loop->ninsns", &(loop->ninsns));
>          register_event_parameter ("loop->av_ninsns", &(loop->av_ninsns));
> 
>          register_event_parameter ("loop->lpt_decision.times",
> &(loop->lpt_decision.times));
>          register_event_parameter ("loop->lpt_decision.decision",
> &(loop->lpt_decision.decision));
>          register_event_parameter ("loop->lpt_decision.unroll_runtime",
>                loop->lpt_decision.decision == LPT_UNROLL_RUNTIME ?
> (void *) 1 : (void *) 0);
>          register_event_parameter ("loop->lpt_decision.unroll_constant",
>                loop->lpt_decision.decision == LPT_UNROLL_CONSTANT ?
> (void *) 1 : (void *) 0);
> 
>          call_plugin_event("unroll_feature_change");
> 
>          unregister_event_parameter ("loop->num");
>          unregister_event_parameter ("loop->ninsns");
> 
>          unregister_event_parameter ("loop->av_ninsns");
>          unregister_event_parameter ("loop->lpt_decision.times");
>          unregister_event_parameter ("loop->lpt_decision.decision");
> 
> Instead we could have:
> 
>          invoke_plugin_va_callbacks (PLUGIN_UNROLL_FEATURE_CHANGE, loop);
> and then accessor functions:
> int
> plugin_unroll_feature_change_param_loop_num (va_list va)
> {
>    struct loop *loop = va_arg (va, struct loop *);
>    return loop->num;
> }
> 
> unsigned
> plugin_unroll_feature_change_param_loop_ninsns (va_list va)
> {
>    struct loop *loop = va_arg (va, struct loop *);
>    return loop->ninsns;
> }
> 
> unsigned
> plugin_unroll_feature_change_param_loop_av_ninsns (va_list va)
> {
>    struct loop *loop = va_arg (va, struct loop *);
>    return loop->av_ninsns;
> }
> ...
> bool
> plugin_unroll_feature_change_param_loop_lpt_decision_unroll_runtime
> (va_list va)
> {
>    struct loop *loop = va_arg (va, struct loop *);
>    return loop->lpt_decision.decision == LPT_UNROLL_RUNTIME;
> }
> ...

I am a bit confused about your above example - you suggest to add 
this functionality on top of current ICI or substitute it? 

If you want to substitute it, I personally disagree. We spent a very long time 
with many colleagues and real ICI users discussing how to simplify the usage of 
events for people who are not programming professionals, 
so I really prefer to keep the current structure as it is ... 

However, if it is the way to speedup slow prototype plugins and addition to ICI,
it may be fine but I need to think about it more. 

In both cases, I think it is not critical for now and should be the second step 
after current ICI is synced with the mainline. However, suggestions from others 
are very welcome ;) !..

> There is still another practical issue: as I change the ICI infrastructure
> to fit better with the existing gcc 4.5 plugin infrastructure,
> the ICI plugins must be adapted to keep working.
> As we are trying to have something working in a short time frame, I think
> I should pick one plugin and modify it in lock-step with the infrastructure
> to demonstrate how it is supposed to work.
> 
> Do you think the adapt.c plugin is suitable for that purpose?

Yes, adapt.c is the latest plugin that we use for our experiments ...

Cheers,
Grigori

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Joern Rennecke]

Quoting Grigori Fursin <gfursin@gmail.com>:
  > I am a bit confused about your above example - you suggest to add
> this functionality on top of current ICI or substitute it?

The idea was to replace it.  The current event implementation has two
issues:
- It is very different from the existing GCC 4.5 events which makes it
   confusing to have it side-by-side.  I think if we can make the ICI code
   more of a natural extension of the current event code, it would be more
   likely to be acceptable.
- The parameter registering / unregistering and the event call itself
   introduce a number of hashing operations which are performed irrespective
   of the presence of a plugin.  This makes these events intrusive if placed
   in places with a high execution frequency.

> If you want to substitute it, I personally disagree. We spent a very  
>  long time
> with many colleagues and real ICI users discussing how to simplify   
> the usage of
> events for people who are not programming professionals,
> so I really prefer to keep the current structure as it is ...

Were these discussions done face-to-face or on a news group/mailing list?
If the latter, could you point me where I can read up on the discussion
so that I better understand the issues involved.
Would these people who are not programming professionals both insert
event raising points as well as write event handling code?

If we would use C++, some things would get easier, i.e. we could have an
event class with subclasses for the separate event types, and then have
the parameter accessors be member functions.  This would remove the need
to repeatedly type the event name when accessing the parameters.
However, it would require to build GCC with C++, so I'd say this
significantly reduces the chances of having ICI integrated into the
gcc mainline and having release GCC binaries shipped with the
functionality enabled.

If plugin-side run time is not important, we can have register_plugin_event
as a wrapper for register_callback and use user_data to squirrel away the
event name and the actual callback function; then we can have a wrapper
callback function which sets a thread-local variable (can be non-exported
global inside the plugin as long as no threads are used) to the plugin name,
and make get_event_parameter use a dynamic function name lookup by stringing
together the event name with the parameter name.
This way, we could look the ICI interface on the plugin side pretty much
what it looks now, except that we should probably restrict the event and
parameter names to identifier characters, lest we need name mangling  
to translate them to function names.

I had a look at the adapt branch to gauge if the there is really a point for
having the varargs stuff, i.e. events with parameters that are not readily
available in a single struct.
The unroll_parameter_handler / graphite_parameter_handler events are somewhat
odd because they have a varying set of parameters.  So they really have a list
of parameters.  We could do this with somehing like:
          invoke_plugin_va_callbacks (PLUGIN_UNROLL_PARAMETER_HANDLER,
                                      "_loop.id", EP_INT, loop->num,
                                      "_loop.can_duplicate_loop_p",
                                      EP_UNSIGNED_CHAR,
                                      ici_can_duplicate_loop_p);
And then have the plugin pick through the va_list.  Or, if preferred,
have a helper function - possibly invoked implicitly by the ICI
infrastructure - go through the va_list and build the hash table of arguments
from it so that the current parameter query interface can be used.
In fact, we could use such a mechanism in general, so if we go back to  
passing pointers to parameters instead of parameters, you'd have  
backward
compatibility on the plugin side.
OTOH, does using pointers to parameters really make this interface easy to
use - it seems you had to use this to avoid type compatibility issues, which
are sidestepped by using a va_list.  If we pass parameter values instead, you
could have a query function that tells you if a parameter is set.  Possibly
also with a pointer argument to assign the value if the parameter is set.
tests if a parameter exists - if you like

> However, if it is the way to speedup slow prototype plugins and   
> addition to ICI,
> it may be fine but I need to think about it more.

Having a single call to invoke an event by number (even if it is a
varargs call) is certainly much faster than having all these
register_parameter / unregister_parameter calls, particularly in the
case that no plugin has registered a callback for the event.

> In both cases, I think it is not critical for now and should be the   
> second step
> after current ICI is synced with the mainline. However, suggestions   
> from others
> are very welcome ;) !..

I thought you were hoping to push some ICI infrastructure into mainline
in time for 4.5 .  In order to do that
- Time is of the essence.
- We should reduce the impact that ICI has on the rest of GCC.

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi Joern,

>   > I am a bit confused about your above example - you suggest to add
> > this functionality on top of current ICI or substitute it?
> 
> The idea was to replace it.  The current event implementation has two
> issues:
> - It is very different from the existing GCC 4.5 events which makes it
>    confusing to have it side-by-side.  I think if we can make the ICI code
>    more of a natural extension of the current event code, it would be more
>    likely to be acceptable.
> - The parameter registering / unregistering and the event call itself
>    introduce a number of hashing operations which are performed irrespective
>    of the presence of a plugin.  This makes these events intrusive if placed
>    in places with a high execution frequency.

Ok, thanks for the update. I will send you another private email asking
for more tech. details to be sure that we are in sync before doing
further changes in ICI ...

> > If you want to substitute it, I personally disagree. We spent a very
> >  long time
> > with many colleagues and real ICI users discussing how to simplify
> > the usage of
> > events for people who are not programming professionals,
> > so I really prefer to keep the current structure as it is ...
> 
> Were these discussions done face-to-face or on a news group/mailing list?
> If the latter, could you point me where I can read up on the discussion
> so that I better understand the issues involved.
> Would these people who are not programming professionals both insert
> event raising points as well as write event handling code?

Well, it's a long story. I moved first prototype of ICI from Open64
to GCC in 2006 and then had discussions with Albert and Diego at
one of the HiPEAC tutorials about having plugins in GCC. However,
since it was a taboo for GCC at that time, I continued extending
it in the MILEPOST project and within the HiPEAC network of excellence.
We had multiple discussions during MILEPOST and HiPEAC face-to-face
meetings and private mailing lists. Recent emails at GCC mailing lists
and comparison of plugin systems at GCC Wiki are here:
http://www.mail-archive.com/gcc@gcc.gnu.org/msg41368.html
http://gcc.gnu.org/wiki/GCC_PluginComparison

> If we would use C++, some things would get easier, i.e. we could have an
> event class with subclasses for the separate event types, and then have
> the parameter accessors be member functions.  This would remove the need
> to repeatedly type the event name when accessing the parameters.
> However, it would require to build GCC with C++, so I'd say this
> significantly reduces the chances of having ICI integrated into the
> gcc mainline and having release GCC binaries shipped with the
> functionality enabled.

I didn't mean rewriting ICI in C++. I meant that we need to check that
the functionality works (i.e. pass selection and reordering + fine-grain
selection of optimization and function cloning for c++ programs) when using 
g++ ...

> If plugin-side run time is not important, we can have register_plugin_event
> as a wrapper for register_callback and use user_data to squirrel away the
> event name and the actual callback function; then we can have a wrapper
> callback function which sets a thread-local variable (can be non-exported
> global inside the plugin as long as no threads are used) to the plugin name,
> and make get_event_parameter use a dynamic function name lookup by stringing
> together the event name with the parameter name.
> This way, we could look the ICI interface on the plugin side pretty much
> what it looks now, except that we should probably restrict the event and
> parameter names to identifier characters, lest we need name mangling
> to translate them to function names.
> 
> I had a look at the adapt branch to gauge if the there is really a point for
> having the varargs stuff, i.e. events with parameters that are not readily
> available in a single struct.
> The unroll_parameter_handler / graphite_parameter_handler events are somewhat
> odd because they have a varying set of parameters.  So they really have a list
> of parameters.  We could do this with somehing like:
>           invoke_plugin_va_callbacks (PLUGIN_UNROLL_PARAMETER_HANDLER,
>                                       "_loop.id", EP_INT, loop->num,
>                                       "_loop.can_duplicate_loop_p",
>                                       EP_UNSIGNED_CHAR,
>                                       ici_can_duplicate_loop_p);
> And then have the plugin pick through the va_list.  Or, if preferred,
> have a helper function - possibly invoked implicitly by the ICI
> infrastructure - go through the va_list and build the hash table of arguments
> from it so that the current parameter query interface can be used.
> In fact, we could use such a mechanism in general, so if we go back to
> passing pointers to parameters instead of parameters, you'd have
> backward
> compatibility on the plugin side.
> OTOH, does using pointers to parameters really make this interface easy to
> use - it seems you had to use this to avoid type compatibility issues, which
> are sidestepped by using a va_list.  If we pass parameter values instead, you
> could have a query function that tells you if a parameter is set.  Possibly
> also with a pointer argument to assign the value if the parameter is set.
> tests if a parameter exists - if you like

I need to think about that. Let's sync offline first since I didn't look
at the ICI code for some time and have a few deadlines coming this and
next week...

> 
> > However, if it is the way to speedup slow prototype plugins and
> > addition to ICI,
> > it may be fine but I need to think about it more.
> 
> Having a single call to invoke an event by number (even if it is a
> varargs call) is certainly much faster than having all these
> register_parameter / unregister_parameter calls, particularly in the
> case that no plugin has registered a callback for the event.

Sure. 

> 
> > In both cases, I think it is not critical for now and should be the
> > second step
> > after current ICI is synced with the mainline. However, suggestions
> > from others
> > are very welcome ;) !..
> 
> I thought you were hoping to push some ICI infrastructure into mainline
> in time for 4.5 .  In order to do that
> - Time is of the essence.
> - We should reduce the impact that ICI has on the rest of GCC.

Agree. However, again, I don't see the impact of ICI on the rest of GCC now.
I think oppositely is barely touches the internals, except the pass manager ...
Again, let's sync offline ...

Cheers,
Grigori

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Hi all,

Just a small update, that after some discussions with Joern we think that based on our 
time constraints and the current state of GCC, instead of trying to push full ICI into GCC
we start from the opposite approach:
We take all our plugins (support pass selection and reordering from MILEPOST;
generic function cloning and fine-grain optimizations from GSOC'09) and trying
to see which low-level GCC functionality is missing to support them.
Then we provide a few hooks to support them, provide a few small updates to GCC and rewrite 
our plugins to support low-level plugin system. Joern will continue communicating
about a few extensions to the plugin system we need to make it happen.

This is a pragmatic step and should require minimal changes in GCC and will help us already 
use current plugin system for our work. However, I think there is still a benefit
of ICI in separating GCC and plugins when using internal data structures, i.e. currently
the referencing of data structures in GCC is hardwired in plugins. If one day these
data structures change, we will need to rewrite all plugins. Using referencing mechanism
in ICI (data is used in plugins indirectly through parameter registering) allows us
to insure plugins compatibility but with the performance degradation. We can discuss
that later after GCC 4.5 release and when we get some more experience from the users
about plugins ...

By the way, due to that, I think that maybe besides documenting all the data structures
we should maybe also start providing info if they are used in some plugins. This maybe will 
help clearning up the internals of the compiler and will prevent careless changes of the data 
structures in GCC to keep plugins compatible?..

Anyway, Joern will continue communicating about the progress and extensions to the plugin
system ...

Take care and have a good weekend,
Grigori

RE: [plugins-ici-cloning-instrumentation] new GCC plugin developements

[Grigori Fursin]

Just one more issue to mention (particularly for those who have been writing ICI plugins).
ICI sometime has been using environment variables inside GCC with its own invocation flags
(-fici) and dynamic library loading.

Naturally, Joern will remove duplicate dynamic library handling and invocation flags from ICI to
use the plugin functionality from GCC 4.5.

As for environment variables inside GCC - we needed it for transparent program analysis
and optimizations but I remember that there were several concerns about that (I think 
Richard mentioned several times that it complicated debugging if there are problems),
so we will remove them too. We will use current plugin flags to pass parameters to
the plugins (we can pass a configuration file I guess that will be parsed by our plugins
if there is lots of information to pass) and if we need to do transparent program
analysis and optimization we will use a script and a wrapper around GCC that translated
our environment variables into flags. I already did that recently for MILEPOST GCC so it 
should be easy. 

But for now we assume that it's fine to use environment variables (such as to control
verbose output or pass some parameters) in plugins themselves ... However, eventually
we should also use some configuration files for plugins that can be easily shared with the community
if there is a bug in plugins ...

Cheers,
Grigori

[plugins-ici-cloning-instrumentation] install-plugin Makefile target

[Joern Rennecke]

What do people think about making install-plugin not only install
headers to build new plugins, but also install all plugins that
have been contributed up to the code freeze for the release.

First, it would make testing the plugin interface and the plugins easier.

Second, if the version of a plugin available with the release is good enough
for the needs of a user, (s)he can try out without having to install -  
or having to have installed - the plugins separately.

Third, when someone uses an experimental plugin for a while and then upgrades
some time later to a newer gcc release, it is useful when an updated plugin
version that works with the new gcc release comes with the new gcc
distribution.

Re: [plugins-ici-cloning-instrumentation] install-plugin Makefile target

[Rafael Espindola]

2009/11/18 Joern Rennecke <amylaar@spamcop.net>:
> What do people think about making install-plugin not only install
> headers to build new plugins, but also install all plugins that
> have been contributed up to the code freeze for the release.
>
> First, it would make testing the plugin interface and the plugins easier.
>
> Second, if the version of a plugin available with the release is good enough
> for the needs of a user, (s)he can try out without having to install - or
> having to have installed - the plugins separately.
>
> Third, when someone uses an experimental plugin for a while and then
> upgrades
> some time later to a newer gcc release, it is useful when an updated plugin
> version that works with the new gcc release comes with the new gcc
> distribution.
>

Sound like a good idea, but I don't think we have agreed about
including plugins with gcc. I would like it, but it is not the
consensus.

Cheers,
-- 
Rafael Ávila de Espíndola

Re: [plugins-ici-cloning-instrumentation] install-plugin Makefile target

[Diego Novillo]

On Wed, Nov 18, 2009 at 09:05, Joern Rennecke <amylaar@spamcop.net> wrote:
> What do people think about making install-plugin not only install
> headers to build new plugins, but also install all plugins that
> have been contributed up to the code freeze for the release.

I agree, but we have no plugins included with the release.  I think it
would be beneficial to include a tutorial plugin somewhere that shows
the basics.  I have no opinion on where in the tree.


Diego.

Re: [plugins-ici-cloning-instrumentation] install-plugin Makefile target

[Basile STARYNKEVITCH]

Diego Novillo wrote:
> On Wed, Nov 18, 2009 at 09:05, Joern Rennecke <amylaar@spamcop.net> wrote:
>> What do people think about making install-plugin not only install
>> headers to build new plugins, but also install all plugins that
>> have been contributed up to the code freeze for the release.
> 
> I agree, but we have no plugins included with the release.  I think it
> would be beneficial to include a tutorial plugin somewhere that shows
> the basics.  I have no opinion on where in the tree.

Diego, are you speaking of the GCC source tree, the GCC build tree, the GCC installation tree?

We could simply adapt a plugin from our testsuite, and install it...

The interesting question is: do we have an installed plugins directory? (We might have already discussed that, I forgot 
the details and the context, probably more than a year ago). I wish we had one:

We might even consider that invoking [sorry to be so selfish and take MELT as a plugin example]
    gcc-4.5 -fplugin=melt ....
doing the same as
    gcc-4.5 -fplugin=$(gcc-trunk --print-file-name=plugin)/libexec/melt.so ....

That is, a plugin only specified with XXX [only letters or digits or underscores, no dots, so no .so extensions, no 
slashes] being searched in some well known directory like 
/usr/local/lib/gcc-trunk/gcc/x86_64-unknown-linux-gnu/4.5.0/plugin/libexec/ . If I recall correctly, most other 
pluginable software have their "standard plugin directory".

I would be delighted by having in practice short -fplugin=XXX program argument to GCC. In my understanding, the current 
one is almost always long in practice [because in practice the plugin directory should depend of the GCC version].

The issues are:

   * do we agree that having a well defined directory to contain some installed plugins is desirable?

   * what is that standard plugins directory?

And then, we have to implement & document that.

In the current state of the trunk, it seems that plugins are the only point where environment variables matter, and that 
environment variable is LD_LIBRARY_PATH ... (because dlopen mandates that behavior).

BTW, I think that both MELT & ICI are dlopen-ing their own shared objects (in MELT I call these modules), and that MELT 
(& probably ICI) have some mechanism for some standard paths for these.

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: [plugins-ici-cloning-instrumentation] install-plugin Makefile target

[Joern Rennecke]

Quoting Basile STARYNKEVITCH <basile@starynkevitch.net>:
> The interesting question is: do we have an installed plugins   
> directory? (We might have already discussed that, I forgot
> the details and the context, probably more than a year ago). I wish   
> we had one:

At the moment we have a directory for plugin header files, which is a  
subdirectory of an otherwise empty plugin directory:

[amylaar@laria gcc]$ /user/inria/bin/gcc --print-file-name=libgcc.a
/user/inria/lib/gcc/i686-pc-linux-gnu/4.5.0/libgcc.a
[amylaar@laria gcc]$ /user/inria/bin/gcc --print-file-name=plugin
/user/inria/lib/gcc/i686-pc-linux-gnu/4.5.0/plugin
[amylaar@laria gcc]$ ls -l `!!`
ls -l `/user/inria/bin/gcc --print-file-name=plugin`
total 4
drwxrwxr-x. 7 amylaar amylaar 4096 2009-11-18 17:48 include
[amylaar@laria gcc]$ /user/inria/bin/gcc --version
gcc (GCC) 4.5.0 20091108 (experimental)
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

target hooks / plugins

[Joern Rennecke]

Target hooks would often be interesting for plugins to modify.  And
some proposed new plugin callbacks would also be interesting to have as
target hooks.  Therefore, I would like target hooks to become writeable
by plugins, and make it much easier to add new target hooks in the GCC
sources.

Right now, to make a new target hook, you have to add a new field in
target.h, define a new default in target-def.h, place the new macro
in exactly the right position there of the right initializer macro,
describe the new hook in tm.texi, and if you need a new function with
a bunch of parameters returning a constant, you have to add this to
hooks.c .

I would like to be able to do all this by adding a single entry in a
new definition file; and the information should also be usable by
plugin sources so that they can automatically make wrappers for all
function-type hooks.

Most of the ICI unroll_parameter_handler / graphite_parameter_handler
callbacks can the be made into target hooks.

Re: target hooks / plugins

[Joern Rennecke]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

Quoting Joern Rennecke <amylaar@spamcop.net>:
> Right now, to make a new target hook, you have to add a new field in
> target.h, define a new default in target-def.h, place the new macro
> in exactly the right position there of the right initializer macro,
> describe the new hook in tm.texi, and if you need a new function with
> a bunch of parameters returning a constant, you have to add this to
> hooks.c .
>
> I would like to be able to do all this by adding a single entry in a
> new definition file; and the information should also be usable by
> plugin sources so that they can automatically make wrappers for all
> function-type hooks.

I've attached what I have so far.
There is an issue that the struct gcc_target member names don't always agree
with the TARGET_* macro names.

Should I rather change one or the other to make them agree, or add an extra
parameter in the definition file macros to specify these names independently?



[-- Attachment #2: thook-diff-20091223 --]
[-- Type: text/plain, Size: 17043 bytes --]

Index: target.def
===================================================================
--- target.def	(revision 0)
+++ target.def	(revision 0)
@@ -0,0 +1,263 @@
+/* Target hook definitions.
+   Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+   Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by the
+   Free Software Foundation; either version 3, or (at your option) any
+   later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; see the file COPYING3.  If not see
+   <http://www.gnu.org/licenses/>.
+
+   In other words, you are welcome to use, share and improve this program.
+   You are forbidden to forbid anyone else to use, share and improve
+   what you give them.   Help stamp out software-hoarding!  */
+
+/* The following macros should be provided by the including file:
+
+   DEFHOOK(NAME, DOC, TYPE, PARAMS, INIT): Define a function-valued hook.
+   DEFHOOKPOD(DOC, TYPE, NAME, INIT): Define a piece-of-data 'hook'.  */
+
+/* Defaults for optional macros:
+   DEFHOOKPODX(NAME, NAME, INIT): Like DEFHOOKPOD, but share documentation
+   with the previous 'hook'.  */
+#ifndef DEFHOOKPODX
+#define DEFHOOKPODX(NAME, TYPE, INIT) DEFHOOKPOD (NAME, 0, TYPE, INIT)
+#endif
+   
+/* HOOKSTRUCT(FRAGMENT): Declarator fragments to encapsulate all the
+   members into a struct gcc_target, which in turn contains several
+   sub-structs.  */
+#ifndef HOOKSTRUCT
+#define HOOKSTRUCT(FRAGMENT)
+#endif
+
+HOOKSTRUCT (struct gcc_target {)
+
+/* Functions that output assembler for the target.  */
+#define HOOK_PREFIX "TARGET_ASM_"
+HOOKSTRUCT (struct asm_out {)
+
+DEFHOOKPOD
+(open_paren,
+"These target hooks are C string constants, describing the syntax in the\
+ assembler for grouping arithmetic expressions.  If not overridden, they\
+ default to normal parentheses, which is correct for most assemblers.",
+ const char *, "(")
+DEFHOOKPODX (close_paren, const char *, ")")
+
+DEFHOOKPOD
+(byte_op,
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_ALIGNED_HI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_ALIGNED_SI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_ALIGNED_DI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_ALIGNED_TI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_UNALIGNED_HI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_UNALIGNED_SI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_UNALIGNED_DI_OP\n"
+"@deftypevrx {Target Hook} {const char *} TARGET_ASM_UNALIGNED_TI_OP\n"
+"These hooks specify assembly directives for creating certain kinds\
+ of integer object.  The @code{TARGET_ASM_BYTE_OP} directive creates a\
+ byte-sized object, the @code{TARGET_ASM_ALIGNED_HI_OP} one creates an\
+ aligned two-byte object, and so on.  Any of the hooks may be\
+ @code{NULL}, indicating that no suitable directive is available.\n\n"
+
+"The compiler will print these strings at the start of a new line,\
+ followed immediately by the object's initial value.  In most cases,\
+ the string should contain a tab, a pseudo-op, and then another tab.",
+ const char *, "\t.byte\t")
+DEFHOOKPOD
+(aligned_op, "*", struct asm_int_op,
+ ({ TARGET_ASM_ALIGNED_HI_OP, TARGET_ASM_ALIGNED_SI_OP,
+    TARGET_ASM_ALIGNED_DI_OP, TARGET_ASM_ALIGNED_TI_OP }) )
+DEFHOOKPOD
+(unaligned_op, "*", struct asm_int_op,
+ ({ TARGET_ASM_UNALIGNED_HI_OP, TARGET_ASM_UNALIGNED_SI_OP,
+    TARGET_ASM_UNALIGNED_DI_OP, TARGET_ASM_UNALIGNED_TI_OP }) )
+    /* Assembler instructions for creating various kinds of integer object.  */
+
+DEFHOOK
+(integer,
+"The @code{assemble_integer} function uses this hook to output an\
+ integer object.  @var{x} is the object's value, @var{size} is its size\
+ in bytes and @var{aligned_p} indicates whether it is aligned.  The\
+ function should return @code{true} if it was able to output the\
+ object.  If it returns false, @code{assemble_integer} will try to\
+ split the object into smaller parts.\n\n"
+
+"The default implementation of this hook will use the\
+ @code{TARGET_ASM_BYTE_OP} family of strings, returning @code{false}\
+ when the relevant string is @code{NULL}.",
+ /* Only handles cases for which BYTE_OP, ALIGNED_OP and UNALIGNED_OP are
+    NULL.  */
+ bool, (rtx x, unsigned int size, int aligned_p),
+ default_assemble_integer)
+
+DEFHOOK
+(globalize_label,
+"This target hook is a function to output to the stdio stream\
+ @var{stream} some commands that will make the label @var{name} global;\
+ that is, available for reference from other files.\n\n"
+
+"The default implementation relies on a proper definition of\
+ @code{GLOBAL_ASM_OP}.",
+ void, (FILE *stream, const char *name),
+ default_globalize_label)
+
+DEFHOOK
+(globalize_decl_name,
+"This target hook is a function to output to the stdio stream\
+ @var{stream} some commands that will make the name associated with @var{decl}\
+ global; that is, available for reference from other files.\n\n"
+
+"The default implementation uses the TARGET_ASM_GLOBALIZE_LABEL target hook.",
+ void, (FILE *stream, tree decl), default_globalize_decl_name)
+
+DEFHOOK
+(emit_unwind_label,
+"This target hook emits a label at the beginning of each FDE@.  It \
+ should be defined on targets where FDEs need special labels, and it \
+ should write the appropriate label, for the FDE associated with the \
+ function declaration @var{decl}, to the stdio stream @var{stream}. \
+ The third argument, @var{for_eh}, is a boolean: true if this is for an \
+ exception table.  The fourth argument, @var{empty}, is a boolean: \
+ true if this is a placeholder label for an omitted FDE@.\n\n"
+
+"The default is that FDEs are not given nonlocal labels.",
+ void, (FILE *stream, tree decl, int for_eh, int empty),
+ default_emit_unwind_label)
+
+#if 0
+    /* Output code that will emit a label to divide up the exception
+       table.  */
+    void (* except_table_label) (FILE *);
+
+    /* Emit any directives required to unwind this instruction.  */
+    void (* unwind_emit) (FILE *, rtx);
+
+    /* Output an internal label.  */
+    void (* internal_label) (FILE *, const char *, unsigned long);
+
+    /* Emit a ttype table reference to a typeinfo object.  */
+    bool (* ttype) (rtx);
+
+    /* Emit an assembler directive to set visibility for the symbol
+       associated with the tree decl.  */
+    void (* visibility) (tree, int);
+
+    /* Output the assembler code for entry to a function.  */
+    void (* function_prologue) (FILE *, HOST_WIDE_INT);
+
+    /* Output the assembler code for end of prologue.  */
+    void (* function_end_prologue) (FILE *);
+
+    /* Output the assembler code for start of epilogue.  */
+    void (* function_begin_epilogue) (FILE *);
+
+    /* Output the assembler code for function exit.  */
+    void (* function_epilogue) (FILE *, HOST_WIDE_INT);
+
+    /* Initialize target-specific sections.  */
+    void (* init_sections) (void);
+
+    /* Tell assembler to change to section NAME with attributes FLAGS.
+       If DECL is non-NULL, it is the VAR_DECL or FUNCTION_DECL with
+       which this section is associated.  */
+    void (* named_section) (const char *name, unsigned int flags, tree decl);
+
+    /* Return a mask describing how relocations should be treated when
+       selecting sections.  Bit 1 should be set if global relocations
+       should be placed in a read-write section; bit 0 should be set if
+       local relocations should be placed in a read-write section.  */
+    int (*reloc_rw_mask) (void);
+
+    /* Return a section for EXP.  It may be a DECL or a constant.  RELOC
+       is nonzero if runtime relocations must be applied; bit 1 will be
+       set if the runtime relocations require non-local name resolution.
+       ALIGN is the required alignment of the data.  */
+    section *(* select_section) (tree, int, unsigned HOST_WIDE_INT);
+
+    /* Return a section for X.  MODE is X's mode and ALIGN is its
+       alignment in bits.  */
+    section *(* select_rtx_section) (enum machine_mode, rtx,
+                                     unsigned HOST_WIDE_INT);
+
+    /* Select a unique section name for DECL.  RELOC is the same as
+       for SELECT_SECTION.  */
+    void (* unique_section) (tree, int);
+
+    /* Return the readonly data section associated with function DECL.  */
+    section *(* function_rodata_section) (tree);
+
+    /* Output a constructor for a symbol with a given priority.  */
+    void (* constructor) (rtx, int);
+
+    /* Output a destructor for a symbol with a given priority.  */
+    void (* destructor) (rtx, int);
+
+    /* Output the assembler code for a thunk function.  THUNK_DECL is the
+       declaration for the thunk function itself, FUNCTION is the decl for
+       the target function.  DELTA is an immediate constant offset to be
+       added to THIS.  If VCALL_OFFSET is nonzero, the word at
+       *(*this + vcall_offset) should be added to THIS.  */
+    void (* output_mi_thunk) (FILE *file, tree thunk_decl,
+                              HOST_WIDE_INT delta, HOST_WIDE_INT vcall_offset,
+                              tree function_decl);
+
+    /* Determine whether output_mi_thunk would succeed.  */
+    /* ??? Ideally, this hook would not exist, and success or failure
+       would be returned from output_mi_thunk directly.  But there's
+       too much undo-able setup involved in invoking output_mi_thunk.
+       Could be fixed by making output_mi_thunk emit rtl instead of
+       text to the output file.  */
+    bool (* can_output_mi_thunk) (const_tree thunk_decl, HOST_WIDE_INT delta,
+                                  HOST_WIDE_INT vcall_offset,
+                                  const_tree function_decl);
+
+    /* Output any boilerplate text needed at the beginning of a
+       translation unit.  */
+    void (*file_start) (void);
+
+    /* Output any boilerplate text needed at the end of a
+       translation unit.  */
+    void (*file_end) (void);
+
+    /* Output an assembler pseudo-op to declare a library function name
+       external.  */
+    void (*external_libcall) (rtx);
+
+    /* Output an assembler directive to mark decl live. This instructs
+        linker to not dead code strip this symbol.  */
+    void (*mark_decl_preserved) (const char *);
+
+    /* Output a record of the command line switches that have been passed.  */
+    print_switch_fn_type record_gcc_switches;
+    /* The name of the section that the example ELF implementation of
+       record_gcc_switches will use to store the information.  Target
+       specific versions of record_gcc_switches may or may not use
+       this information.  */
+    const char * record_gcc_switches_section;
+
+    /* Output the definition of a section anchor.  */
+    void (*output_anchor) (rtx);
+
+    /* Output a DTP-relative reference to a TLS symbol.  */
+    void (*output_dwarf_dtprel) (FILE *file, int size, rtx x);
+
+    /* Some target machines need to postscan each insn after it is output.  */
+    void (*final_postscan_insn) (FILE *, rtx, rtx *, int);
+
+    /* Emit the trampoline template.  This hook may be NULL.  */
+    void (*trampoline_template) (FILE *);
+
+HOOKSTRUCT (} asm_out;)
+
+HOOKSTRUCT (struct sched {)
+#endif /* 0 */
Index: target.h
===================================================================
--- target.h	(revision 155367)
+++ target.h	(working copy)
@@ -108,7 +108,13 @@ struct asm_int_op
 };
 
 /* The target structure.  This holds all the backend hooks.  */
+#define DEFHOOKPOD(NAME, DOC, TYPE, INIT) TYPE NAME;
+#define DEFHOOK(NAME, DOC, TYPE, PARAMS, INIT) TYPE (* NAME) PARAMS;
+#define HOOKSTRUCT(FRAGMENT) FRAGMENT
 
+#include "target.def"
+
+#if 0
 struct gcc_target
 {
   /* Functions that output assembler for the target.  */
@@ -141,6 +147,7 @@ struct gcc_target
        this is only a placeholder for an omitted FDE.  */
     void (* unwind_label) (FILE *, tree, int, int);
 
+#endif /* 0 */
     /* Output code that will emit a label to divide up the exception
        table.  */
     void (* except_table_label) (FILE *);
Index: dwarf2out.c
===================================================================
--- dwarf2out.c	(revision 155367)
+++ dwarf2out.c	(working copy)
@@ -3457,8 +3457,8 @@ output_fde (dw_fde_ref fde, bool for_eh,
   char l1[20], l2[20];
   dw_cfi_ref cfi;
 
-  targetm.asm_out.unwind_label (asm_out_file, fde->decl, for_eh,
-				/* empty */ 0);
+  targetm.asm_out.emit_unwind_label (asm_out_file, fde->decl, for_eh,
+				     /* empty */ 0);
   targetm.asm_out.internal_label (asm_out_file, FDE_LABEL,
 				  for_eh + j);
   ASM_GENERATE_INTERNAL_LABEL (l1, FDE_AFTER_SIZE_LABEL, for_eh + j);
@@ -3651,8 +3651,8 @@ output_call_frame_info (int for_eh)
       if ((fde_table[i].nothrow || fde_table[i].all_throwers_are_sibcalls)
 	  && !fde_table[i].uses_eh_lsda
 	  && ! DECL_WEAK (fde_table[i].decl))
-	targetm.asm_out.unwind_label (asm_out_file, fde_table[i].decl,
-				      for_eh, /* empty */ 1);
+	targetm.asm_out.emit_unwind_label (asm_out_file, fde_table[i].decl,
+					   for_eh, /* empty */ 1);
 
   /* If we don't have any functions we'll want to unwind out of, don't
      emit any EH unwind information.  Note that if exceptions aren't
Index: gentarget.c
===================================================================
--- gentarget.c	(revision 0)
+++ gentarget.c	(revision 0)
@@ -0,0 +1,70 @@
+#include "bconfig.h"
+#include "system.h"
+#include <string.h>
+
+struct hook_desc { const char *doc, *type, *name, *param, *init; };
+static struct hook_desc hook_array[] = {
+#define DEFHOOKPOD(NAME, DOC, TYPE, INIT) \
+  { DOC, #TYPE, HOOK_PREFIX #NAME, 0, #INIT },
+#define DEFHOOK(NAME, DOC, TYPE, PARAMS, INIT) \
+  { DOC, #TYPE, HOOK_PREFIX #NAME, #PARAMS, #INIT },
+#include "target.def"
+#undef DEFHOOK
+};
+
+int
+main (int argc, char **argv)
+{
+  bool doc_p = false;
+  int i, j;
+
+  if (argc >= 2 && strcmp (argv[1], "-doc") == 0)
+    doc_p = true;
+  for (i = 0; i < (int) (sizeof hook_array / sizeof hook_array[0]); i++)
+    {
+      char *name = xstrdup (hook_array[i].name);
+      char *p;
+      const char *q, *e;
+
+      for (p = name; *p; p++)
+	*p = TOUPPER (*p);
+      if (doc_p)
+	{
+	  const char *deftype;
+
+	  if (!hook_array[i].doc)
+	    continue;
+	  if (*hook_array[i].doc == '*')
+	    {
+	      printf ("%s", hook_array[i].doc + 1);
+	      continue;
+	    }
+	  deftype = hook_array[i].param ? "deftypefn" : "deftypevr";
+	  printf ("@%s {Target Hook} {%s} %s",
+		  deftype, hook_array[i].type, name);
+	  if (hook_array[i].param)
+	    {
+	      /* Print the parameter list, with the parameter names
+		 enclosed in @var{}.  */
+	      printf (" ");
+	      for (q = hook_array[i].param; (e = strpbrk (q, " ,)"));
+		   q = e + 1)
+		if (*e == ' ')
+		  printf ("%.*s", e - q + 1, q);
+		else
+		  printf ("@var{%.*s}%c", e - q, q, *e);
+	    }
+	  printf ("\n");
+	  for (j = i + 1;
+	       j < (int) (sizeof hook_array / sizeof hook_array[0])
+	       && hook_array[j].doc == 0; j++)
+	    printf ("@%sx {Target Hook} {%s} %s\n",
+		    deftype, hook_array[i].type, name);
+	  printf ("%s\n@end %s\n\n", hook_array[i].doc, deftype);
+	}
+      else
+	printf ("#ifndef %s\n#define %s %s\n#endif\n",
+		name, name, hook_array[i].init);
+    }
+  return 0;
+}
Index: Makefile.in
===================================================================
--- Makefile.in	(revision 155367)
+++ Makefile.in	(working copy)
@@ -3714,6 +3714,7 @@ build/genpreds.o : genpreds.c $(RTL_BASE
   coretypes.h $(GTM_H) errors.h gensupport.h $(OBSTACK_H)
 build/genrecog.o : genrecog.c $(RTL_BASE_H) $(BCONFIG_H) $(SYSTEM_H)	\
   coretypes.h $(GTM_H) errors.h gensupport.h
+build/gentarget.o : gentarget.c target.def $(BCONFIG_H) $(SYSTEM_H)
 
 # Compile the programs that generate insn-* from the machine description.
 # They are compiled with $(COMPILER_FOR_BUILD), and associated libraries,
@@ -3738,6 +3739,7 @@ build/gengenrtl$(build_exeext) : $(BUILD
 build/genmodes$(build_exeext) : $(BUILD_ERRORS)
 build/gengtype$(build_exeext) : build/gengtype-lex.o build/gengtype-parse.o \
 				$(BUILD_ERRORS)
+build/gentarget$(build_exeext) : $(BUILD_ERRORS)
 
 # Generated source files for gengtype.
 gengtype-lex.c : gengtype-lex.l

Re: target hooks / plugins

[Joseph S. Myers]

On Wed, 23 Dec 2009, Joern Rennecke wrote:

> I've attached what I have so far.

If you want to have documentation extracted from source files, you need to 
engage with the SC and FSF at an early stage to get suitable license 
exception wording to permit the relevant text to be used in the manuals as 
well as the main (GPL) sources.  Of course, the ordering and (especially) 
section divisions in the internals manual are human-designed not arbitrary 
so you need to retain the ability to put the documentation of each hook in 
an appropriate position in an appropriate section of the manual.

If you can make this work it should reduce the risk of people not 
documenting new hooks, but for it to work the FSF agreement is critical as 
is ensuring it genuinely works as well as the existing system without 
regressions (including working in update_web_docs).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: target hooks / plugins

[Joseph S. Myers]

It appears you CC:ed your message to a closed mailing list 
ctuning-discussions@googlegroups.com that bounces messages from 
non-subscribers.  Please avoid doing this; messages to public lists should 
not be CC:ed to any list that will send bounces or other automatic replies 
to non-subscribers.  You are free to forward GCC list messages manually to 
such a list, of course.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: target hooks / plugins

[Richard Guenther]

On Thu, Dec 24, 2009 at 1:32 AM, Joern Rennecke <amylaar@spamcop.net> wrote:
> Quoting Joern Rennecke <amylaar@spamcop.net>:
>>
>> Right now, to make a new target hook, you have to add a new field in
>> target.h, define a new default in target-def.h, place the new macro
>> in exactly the right position there of the right initializer macro,
>> describe the new hook in tm.texi, and if you need a new function with
>> a bunch of parameters returning a constant, you have to add this to
>> hooks.c .
>>
>> I would like to be able to do all this by adding a single entry in a
>> new definition file; and the information should also be usable by
>> plugin sources so that they can automatically make wrappers for all
>> function-type hooks.
>
> I've attached what I have so far.
> There is an issue that the struct gcc_target member names don't always agree
> with the TARGET_* macro names.
>
> Should I rather change one or the other to make them agree, or add an extra
> parameter in the definition file macros to specify these names
> independently?

I didn't look at the patch yet but I'd simply change one or the other to
make the names consistent.

Richard.

>
>

RE: target hooks / plugins

[Grigori Fursin]

Hi Joseph,
cTuning-discussions is an open public mailing list -
I have been moderating lots of spam there recently and
mixed up some settings but it is fixed now ...
Cheers,
Grigori

> -----Original Message-----
> From: Joseph Myers [mailto:joseph@codesourcery.com]
> Sent: Thursday, December 24, 2009 1:26 PM
> To: Joern Rennecke
> Cc: 'GCC Mailing List'; Grigori Fursin; 'Yuanjie Huang'; 'Liang Peng'; 'Zbigniew Chamski';
> 'Yuri Kashnikoff'; 'Diego Novillo'
> Subject: Re: target hooks / plugins
> 
> It appears you CC:ed your message to a closed mailing list
> ctuning-discussions@googlegroups.com that bounces messages from
> non-subscribers.  Please avoid doing this; messages to public lists should
> not be CC:ed to any list that will send bounces or other automatic replies
> to non-subscribers.  You are free to forward GCC list messages manually to
> such a list, of course.
> 
> --
> Joseph S. Myers
> joseph@codesourcery.com

Re: target hooks / plugins

[Joern Rennecke]

Quoting "Joseph S. Myers" <joseph@codesourcery.com>:

> On Wed, 23 Dec 2009, Joern Rennecke wrote:
>
>> I've attached what I have so far.
>
> If you want to have documentation extracted from source files, you need to
> engage with the SC and FSF at an early stage to get suitable license
> exception wording to permit the relevant text to be used in the manuals as
> well as the main (GPL) sources.

I suppose I'm reaching the steering commitee by posting to gcc@gcc.gnu.org.
What email address should I use to engage the FSF?

I want to move the hook descriptions from tm.texi into target.def .
gentarget.c includes target.def, and gentarget (re-)generates
$(srcdir)/doc/targhooks.texi (the makefile dependency would be keyed on
--enable-maintainer-mode) ; targhooks.texi would then include the hook
descriptions from target.def .

As the person who regenerates targhooks.texi is in possession of target.def ,
I think the (s)he can use the license on target.def to generate a GFDL
(terms according to gcc.texi) targhooks.texi which includes pieces of  
target.def, without requiring the gentarget executable to have a special
license; the executable is only a means to create a (documentation) work
based on target.def .

I've looked at the bison skeleton for inspiration on the license blurb
for target.def .
This is what I have come up with:

/* Target hook definitions.
    Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010
    Free Software Foundation, Inc.

    This program is free software; you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by the
    Free Software Foundation; either version 3, or (at your option) any
    later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; see the file COPYING3.  If not see
    <http://www.gnu.org/licenses/>.

    In other words, you are welcome to use, share and improve this program.
    You are forbidden to forbid anyone else to use, share and improve
    what you give them.   Help stamp out software-hoarding!  */

/* As a special exception, you may create documentation that contains
    part or all of this target hooks definitions file and distribute that
    work under the copying terms outlined in doc/gcc.texi .
    Alternatively, if you modify or redistribute this target hooks
    definitions file itself, you may (at your option) remove this special
    exception, which will cause the target hooks definitions file (and any
    program output which incorporates parts of this file) to be licensed
    under the GNU General Public License without this special exception.  */


> Of course, the ordering and (especially)
> section divisions in the internals manual are human-designed not arbitrary
> so you need to retain the ability to put the documentation of each hook in
> an appropriate position in an appropriate section of the manual.

Yes, I plan to allow any number of explicit placements of hooks descriptions
in the resulting documentation, and any hooks that are not explicitly placed
will appear as successor of their predecessor in target.def .

Re: target hooks / plugins

[Joern Rennecke]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

Quoting "Joseph S. Myers" <joseph@codesourcery.com>:
> If you want to have documentation extracted from source files, you need to
> engage with the SC and FSF at an early stage to get suitable license
> exception wording to permit the relevant text to be used in the manuals as
> well as the main (GPL) sources.

I haven't gotten any reply to my proposal from the 5th of January
http://gcc.gnu.org/ml/gcc/2010-01/msg00082.html

Is the GCC mailing list no longer the right place to reach the  
steering committee?

>  Of course, the ordering and (especially)
> section divisions in the internals manual are human-designed not arbitrary
> so you need to retain the ability to put the documentation of each hook in
> an appropriate position in an appropriate section of the manual.

I have posted my current patch here:
http://gcc.gnu.org/ml/gcc-patches/2010-01/msg00559.html

> If you can make this work it should reduce the risk of people not
> documenting new hooks

We actually have a number of instances where the target hook documentation is
out of sync with the sources.  I have attached a file with my notes on this.

> , but for it to work the FSF agreement is critical

So who do I have to talk to for this?

> as
> is ensuring it genuinely works as well as the existing system without
> regressions (including working in update_web_docs).

tm.texi is half a megabyte, so it would be nice not to have it as a generated
file in the repository.  Also, it'd be annoying to have to regenerate & check
it in for every target.def change even though the build works fine with
the generated file in the build directory.
Will it be acceptable to have update_web_docs build a generator program
(written in C) to rebuild tm.texi?

[-- Attachment #2: FIX_TYPE --]
[-- Type: text/plain, Size: 3585 bytes --]

various hooks: why are we referring to 'the diagnostic message'?
the context does not suggest a particular one, and I thought we had more
than one in gcc.

various documentation used rtx / tree instead of const_rtx / const_tree.

TARGET_ASM_ASSEMBLE_VISIBILITY: const char *@var{visibility} -> int
clarified current_function_pops_args description and removed comment on
unclear "its arguments" reference.
TARGET_ASM_NAMED_SECTION unsigned int @var{align} -> tree decl
TARGET_ASM_MARK_DECL_PRESERVED tree @var{decl} -> const char *symbol
TARGET_SCHED_ADJUST_COST_2 was undocumented.
TARGET_SCHED_SPECULATE_INSN: Fixed return value description.
TARGET_SCHED_NEEDS_BLOCK_P: int -> bool; rtx @var{insn} -> int dep_status (text is still bogus)
TARGET_SCHED_GEN_CHECK -> TARGET_SCHED_GEN_SPEC_CHECK
TARGET_SCHED_SET_SCHED_FLAGS unsigned int *@var{flags} -> /dev/null
TARGET_VECTORIZE_BUILTIN_VECTORIZED_FUNCTION enum built_in_function @var{code} -> unsigned
TARGET_VECTORIZE_BUILTIN_VECTORIZATION_COST was undocumented.
TARGET_[VECTORIZE_]VECTOR_ALIGNMENT_REACHABLE was undocumented.
TARGET_VECTORIZE_BUILTIN_VEC_PERM was undocumented.
TARGET_VECTORIZE_BUILTIN_VEC_PERM_OK was undocumented.
TARGET_EH_RETURN_FILTER_MODE was undocumented.
TARGET_UNWIND_WORD_MODE was undocumented.
TARGET_BUILTIN_FUNCTION -> TARGET_BUILTIN_DECL
TARGET_RESOLVE_OVERLOADED_BUILTIN addded unsigned int /*location_t*/ loc; tree @var{arglist} -> void *arglist
TARGET_BUILTIN_RECIPROCAL enum tree_code @var{fn} -> unsigned fn
TARGET_CANNOT_COPY_INSN_P was undocumented.
TARGET_EXPAND_BUILTIN_VA_START was undocumented.
TARGET_GIMPLIFY_VA_ARG_EXPR tree *@var{pre_p}, tree *@var{post_p} -> gimple_seq *pre_p, gimple_seq *post_p
TARGET_STDARG_OPTIMIZE_HOOK was undocumented.
TARGET_INTERNAL_ARG_POINTER was undocumented.
TARGET_GET_DRAP_RTX: text was garbled, e.g. confusion macro vs. hook.
TARGET_C_MODE_FOR_SUFFIX was undocumented.
TARGET_VALID_OPTION_ATTRIBUTE_P -> TARGET_OPTION_VALID_ATTRIBUTE_P
TARGET_OPTION_PRINT: new parameters FILE *file, int indent
TARGET_OPTION_PRAGMA_PARSE: (target @var{args}) -> tree, tree
TARGET_EXTRA_LIVE_ON_ENTRY: (bitmap *@var{regs}) -> bitmap
TARGET_HANDLE_PRAGMA_EXTERN_PREFIX was undocumented.
TARGET_HELP bool -> void
TARGET_BUILTIN_SETJMP_FRAME_VALUE bool -> rtx
TARGET_PRETEND_OUTGOING_VARARGS_NAMED: add (CUMULATIVE_ARGS *@var{ca})
TARGET_LEGITIMATE_ADDRESS_P: returns bool
TARGET_SCHED_REORDER2 @var{clock} -> int clock
TARGET_SCHED_DFA_PRE_CYCLE_INSN int -> rtx
TARGET_SCHED_DFA_POST_CYCLE_INSN int -> rtx
TARGET_SCHED_DFA_PRE_CYCLE_ADVANCE -> TARGET_SCHED_DFA_PRE_ADVANCE_CYCLE
TARGET_SCHED_DFA_POST_CYCLE_ADVANCE -> TARGET_SCHED_DFA_POST_ADVANCE_CYCLE
TARGET_SCHED_ALLOC_SCHED_CONTEXT, TARGET_SCHED_INIT_SCHED_CONTEXT, TARGET_SCHED_SET_SCHED_CONTEXT, TARGET_SCHED_CLEAR_SCHED_CONTEXT, TARGET_SCHED_FREE_SCHED_CONTEXT were duplicated.
TARGET_SCHED_FIRST_CYCLE_MULTIPASS_DFA_LOOKAHEAD_GUARD_SPEC int -> bool
TARGET_SCHED_GET_INSN_SPEC_DS was undocumented.
TARGET_SCHED_GET_INSN_CHECKED_DS was undocumented.
TARGET_SCHED_SKIP_RTX_P was undocumented.
TARGET_ASM_RELOC_RW_MASK: 'returns' void.
TARGET_MANGLE_DECL_ASSEMBLER_NAME: void -> tree
TARGET_ASM_EMIT_UNWIND_LABEL was missing types.
Why is the TARGET_DWARF_CALLING_CONVENTION documentation nested in the
DWARF2_DEBUGGING_INFO documentation?
TARGET_MIN_DIVISIONS_FOR_RECIP_MUL int -> unsigned int
TARGET_SHIFT_TRUNCATION_MASK int -> unsigned HOST_WIDE_INT
TARGET_BRANCH_TARGET_REGISTER_CLASS int -> enum reg_class
TARGET_USE_LOCAL_THUNK_ALIAS_P: Not a hook.

Re: target hooks / plugins

[Joseph S. Myers]

On Wed, 13 Jan 2010, Joern Rennecke wrote:

> Quoting "Joseph S. Myers" <joseph@codesourcery.com>:
> > If you want to have documentation extracted from source files, you need to
> > engage with the SC and FSF at an early stage to get suitable license
> > exception wording to permit the relevant text to be used in the manuals as
> > well as the main (GPL) sources.
> 
> I haven't gotten any reply to my proposal from the 5th of January
> http://gcc.gnu.org/ml/gcc/2010-01/msg00082.html
> 
> Is the GCC mailing list no longer the right place to reach the steering
> committee?

It is the right place, but you should start a new thread with just the 
relevant issue and an appropriate subject line marked for the attention of 
the SC.  You should expect it to take at least several months for the FSF 
to prepare an exception (worded as additional permissions under Section 7 
of GPL version 3, like COPYING.RUNTIME, *not* as an old-style "As a 
special exception"), maybe years, if the SC persuades them that there 
should be such an exception.

> > Of course, the ordering and (especially)
> > section divisions in the internals manual are human-designed not arbitrary
> > so you need to retain the ability to put the documentation of each hook in
> > an appropriate position in an appropriate section of the manual.
> 
> I have posted my current patch here:
> http://gcc.gnu.org/ml/gcc-patches/2010-01/msg00559.html
> 
> > If you can make this work it should reduce the risk of people not
> > documenting new hooks
> 
> We actually have a number of instances where the target hook documentation is
> out of sync with the sources.  I have attached a file with my notes on this.

Please note that your initial change to implement automatic doc extraction 
should not result in any changes to the Texinfo content of the manual.  
Such fixes should all go in either before or after the automatic doc 
extraction change, but not at the same time; the doc extraction change 
should result in identical text in the manual, but with the Texinfo files 
produced in a different way.  I recommend sending such fixes before the 
automatic doc extraction change, since they do not depend on the FSF doing 
anything.

> > , but for it to work the FSF agreement is critical
> 
> So who do I have to talk to for this?

The SC.  Not in the middle of a technical thread, but a thread on its own 
marked for the SC and indicating exactly what you want them to raise with 
the FSF.

> tm.texi is half a megabyte, so it would be nice not to have it as a generated
> file in the repository.  Also, it'd be annoying to have to regenerate & check
> it in for every target.def change even though the build works fine with
> the generated file in the build directory.
> Will it be acceptable to have update_web_docs build a generator program
> (written in C) to rebuild tm.texi?

It already builds a generator program in Ada for the Ada manual, so yes.

-- 
Joseph S. Myers
joseph@codesourcery.com

Target hook definition licensing problems (GPL vs GFDL)

[Joern Rennecke]

Adding and maintaining target hooks is unnecessarily hard at the moment,
because the definition is spread across three places, and these are supposed
to be kept in sync.  The code is necessarily kept more or less in sync because
it generally fails to compile or work when it isn't - and if someone can't
get it working, a patch will most likely be left unposted; but the
documentation is silently accumulating inconsistencies with the code,
to the point that is useless or worse than useless for some hooks.

Ideally, target hooks should be easy to maintain in a consistent manner,
and also easy to add to experimental GCC versions, so that people who are
new to GCC or who patch GCC only as a side aspect
of their work can work with them in their contributions.

I have a prototype patch that allows to define target hooks in
a single file - there called target.def - to define the structure member,
its initializer, and its documentation:

http://gcc.gnu.org/ml/gcc-patches/2010-01/msg00559.html

Name and types of the hook is shared between code and documentation, and
the member initializer is automatically placed in the right place of the
right initializer, so it takes the worry out getting and keeping this data
consistent.

However, there is a licensing problem.  The code is GPLed, and the
documentation is GFDLed.  The entire file is included for the struct
gcc_target by a number of GCC file, thus a GPL license is required.
For the initializer macro definitions and the documentation, the file
is included by a generator file, which processes the various fields of
each hook to generate the output files:
The initializer macros end up in a header file which is needed with GPL
license.
For the documentation, the generator file reads the GFDLed file tm.texi.in
which contains all the non-hook documentation, documentation about hooks
that does not pertain to any particular hook, and placement information
to arrange the hook documentation from the input file; the output is
written to tm.texi in the build directory.

As far as I can tell, there is no need for the generator program binary
to have a fancy license; GPL should be just fine, as long as the target.def
source file allows use of the pieces that appear in the output under the GFDL.
The generator program is supposed to be used by people who are in possession
of a copy of target.def, so they should be able to rely on the license terms
of that file.
Here is my proposal for the license header of that file:

/* Target hook definitions.
    Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010
    Free Software Foundation, Inc.

    This program is free software; you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by the
    Free Software Foundation; either version 3, or (at your option) any
    later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; see the file COPYING3.  If not see
    <http://www.gnu.org/licenses/>.

    In other words, you are welcome to use, share and improve this program.
    You are forbidden to forbid anyone else to use, share and improve
    what you give them.   Help stamp out software-hoarding!  */

/* As a special exception, you may create documentation that contains
    part or all of this target hooks definitions file and distribute that
    work under the copying terms outlined in doc/gcc.texi .
    Alternatively, if you modify or redistribute this target hooks
    definitions file itself, you may (at your option) remove this special
    exception, which will cause the target hooks definitions file (and any
    program output which incorporates parts of this file) to be licensed
    under the GNU General Public License without this special exception.  */

We'd then need the FSF to agree that we can use the existing target hook
structure member definitions, default initializers, and documentation under
this dual license.

Quoting "Joseph S. Myers" <joseph@codesourcery.com>:
> You should expect it to take at least several months for the FSF
> to prepare an exception (worded as additional permissions under Section 7
> of GPL version 3, like COPYING.RUNTIME, *not* as an old-style "As a
> special exception"), maybe years, if the SC persuades them that there
> should be such an exception.

I don't see why something so elaborate is needed.  The file in question is
about 150 KB of structure definition, interface definition, and documentation.
The aim is that this file can be used in a GPLed program (GCC) and that parts
of it (emitted by a generator program) can be used in GFDLed documentation.
So dual-licensing of this file should work just fine.
I find it quite challenging to come up with scenarios where GFDLing some
target hook structure, or GPLing some of the target.def target hook
documentation (which is incomplete anyways without the pure-GFDL tm.texi.in),
would impede the FSFs aim to keep GCC Free.

Re: target hooks / plugins

[Joern Rennecke]

Quoting "Joseph S. Myers" <joseph@codesourcery.com>:

> Please note that your initial change to implement automatic doc extraction
> should not result in any changes to the Texinfo content of the manual.
> Such fixes should all go in either before or after the automatic doc
> extraction change, but not at the same time; the doc extraction change
> should result in identical text in the manual, but with the Texinfo files
> produced in a different way.  I recommend sending such fixes before the
> automatic doc extraction change, since they do not depend on the FSF doing
> anything.

Duplicating all these changes separately by hand seems nigh impossible.
I think the best approach is then to take the auto-generated tm.texi as
the new tm.texi, and packages it up as a patch together with the
struct member / hook name changes that I made for consistency.

There is only one issue with using the current auto-generated tm.texi:
Unless special formatting was in force (e.g. @smallexample), I've removed
intra-paragraph newlines.  This should work in principle just as will
as with these newlines for producing output, but it looks somewhat daft
in tm.texi when you consider it as a source file.
Putting newlines in the input file would make it harder to read & edit,
since the documentation comes as C strings - and the GNU multiline string
extension has been deprecated some time ago.
I could pipe the documentation through fold -s, but that would also fold
extra-long lines outside the hook documentation, e.g.:
@@ -28,7 +28,8 @@
  @menu
  * Target Structure::    The @code{targetm} variable.
  * Driver::              Controlling how the driver runs the  
compilation passes.
-* Run-time Target::     Defining @samp{-m} options like  
@option{-m68000} and @option{-m68020}.
+* Run-time Target::     Defining @samp{-m} options like @option{-m68000} and
+@option{-m68020}.
  * Per-Function Data::   Defining data structures for per-function  
information.
  * Storage Layout::      Defining sizes and alignments of data.
  * Type Layout::         Defining sizes and properties of basic user  
data types.

So I suppose I'll have to add some fold mechanism into the documentation
output code.

Re: target hooks / plugins

[Joseph S. Myers]

On Wed, 13 Jan 2010, Joern Rennecke wrote:

> Duplicating all these changes separately by hand seems nigh impossible.
> I think the best approach is then to take the auto-generated tm.texi as
> the new tm.texi, and packages it up as a patch together with the
> struct member / hook name changes that I made for consistency.
> 
> There is only one issue with using the current auto-generated tm.texi:
> Unless special formatting was in force (e.g. @smallexample), I've removed
> intra-paragraph newlines.  This should work in principle just as will
> as with these newlines for producing output, but it looks somewhat daft
> in tm.texi when you consider it as a source file.

I am not particularly concerned about newlines, if the file is identical 
apart from whitespace.  But your text file lists things such as "was 
undocumented" and "Fixed return value description".  Each such change 
needs its own review, by someone familiar with the relevant part of the 
compiler, and needs its own explanation of the problem posted.  Remember, 
a patch should not contain multiple changes that can logically be 
considered separately, and in this case I expect many different people to 
be appropriate reviewers for changes relating to different hooks, so it's 
important not to mix changes relating to hooks in different areas of the 
compiler.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Target hook definition licensing problems (GPL vs GFDL)

[Dave Korn]

Joern Rennecke wrote:
> References: <4AE6E471.4020200@starynkevitch.net>
> <mcr3a54lwb4.fsf@dhcp-172-17-9-151.mtv.corp.google.com>
> <4AE70C5E.4050005@starynkevitch.net>
> <84fc9c000910270839v2d9efe0dw829c8647f361c96f@mail.gmail.com>
> <4AE7164D.9010200@starynkevitch.net>
> <84fc9c000910270855w736df367qe511d8db280aaeb4@mail.gmail.com>
> <2dc303d60910271056h17038110ib63c53cfa374f5c7@mail.gmail.com>
> <20091102074959.p8410ulv28sg0w44-nzlynne@webmail.spamcop.net>
> <20091105082557.75c2estyoog8ss0c-nzlynne@webmail.spamcop.net>
> <-2186575642631489790@unknownmsgid>

[  Never seen that before, looks as if someone used "%i" when they probably
meant "%u" or "%x"! ]

> <55692dc10911050634y54a5fea7jd2ba773086cda60b@mail.gmail.com>
> <00d801ca5e34$e5384160$afa8c420$@fursin@inria.fr>
> <20091223101256.z6a8ug32o8k84o4o-nzlynne@webmail.spamcop.net>
> <20091223193244.hqaet9zf488gw844-nzlynne@webmail.spamcop.net>
> <Pine.LNX.4.64.0912241209420.32398@digraph.polyomino.org.uk>
> <20100113032220.pzerwhbtog0w4gsk-nzlynne@webmail.spamcop.net>
> <Pine.LNX.4.64.1001131343010.2183@digraph.polyomino.org.uk>
> <20100113110917.vqaoqpyyc0ck8gso-nzlynne@webmail.spamcop.net>

  That didn't "start a new thread"!

    cheers,
      DaveK

Re: Target hook definition licensing problems (GPL vs GFDL)

[Gerald Pfeifer]

This has been forwarded to the steering committee (thanks, David)
and the FSF and we are pushing to get it addressed.

Gerald

Re: Target hook definition licensing problems (GPL vs GFDL)

[Joern Rennecke]

Quoting Gerald Pfeifer <gerald@pfeifer.com>:

> This has been forwarded to the steering committee (thanks, David)
> and the FSF and we are pushing to get it addressed.

What is the current status on this?

As we go into phase 1 of gcc 4.6, we are likely to fall back on
code/documentation consistency unless/until we have the mechanism in
place to have a system in place to use a single definition for the
information that is redundant between code and documentation.