Plugins & GGC ie GTY

Plugins & GGC ie GTY

[Basile STARYNKEVITCH]

Hello All,

[I don't know if this discussion belongs to gcc@ or gcc-patches@ so I'm 
sending it on gcc@ since I don't propose or discuss any code yet]

My understanding was that most plugins people are aware that somehow 
some plugins would need to have static GTY-ed roots for the GGC machinery.

So it seems to me that we'll need :

1. The ability to run gengtype in a special mode for plugins, on the 
source code of the plugin. to generate the gt-*.h files included by the 
source code of the plugin. This would require a patch to gengtype.

2. The ability to handle, very probably using dlsym, or perhaps by 
having a gcc_plugin_register_gty_table function called from the plugin 
initialization routine, to plug the generated root tables (an array of 
struct ggc_root_tab, which is defined in the generated gt-*.h files) of 
the plugin into the GGC machinery. This would require a short patch to 
ggc-common.c or similar ggc-*.c files.

Did I miss anything?

Regards.



PS. This is not an April Fool joke! I am quit serious about extra GTY-ed 
roots. It is for me an essential & required condition to perhaps 
transform the MELT branch into a mega-plugin.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Plugins & GGC ie GTY

[Richard Guenther]

On Wed, Apr 1, 2009 at 7:22 PM, Basile STARYNKEVITCH
<basile@starynkevitch.net> wrote:
>
> Hello All,
>
> [I don't know if this discussion belongs to gcc@ or gcc-patches@ so I'm
> sending it on gcc@ since I don't propose or discuss any code yet]
>
> My understanding was that most plugins people are aware that somehow some
> plugins would need to have static GTY-ed roots for the GGC machinery.
>
> So it seems to me that we'll need :
>
> 1. The ability to run gengtype in a special mode for plugins, on the source
> code of the plugin. to generate the gt-*.h files included by the source code
> of the plugin. This would require a patch to gengtype.
>
> 2. The ability to handle, very probably using dlsym, or perhaps by having a
> gcc_plugin_register_gty_table function called from the plugin initialization
> routine, to plug the generated root tables (an array of struct ggc_root_tab,
> which is defined in the generated gt-*.h files) of the plugin into the GGC
> machinery. This would require a short patch to ggc-common.c or similar
> ggc-*.c files.
>
> Did I miss anything?

Plugins shouldn't keep permanent references to GCed memory.  At least
that would make it unnecessary to do what you suggest.

Richard.

Re: Plugins & GGC ie GTY

[Basile STARYNKEVITCH]

Richard Guenther wrote:
>
>
> Plugins shouldn't keep permanent references to GCed memory.  At least
> that would make it unnecessary to do what you suggest.


I strongly disagree with that, and I simply do not understand your 
position. In my perception, plugins are essentially loaded (dlopen-ed) 
but never unloaded (dlclose-ed; perhaps they would be dlclose-ed when 
every thing is done). Why should they not use any GTY stuff? What makes 
you think that some features of GCC (like GTY ...) are invalid inside 
plugins?

I find your position as extreme as saying: plugins should never call 
malloc or xcalloc, or should never call warning_at, or should never call 
XXX! Why?

So what are plugins for?

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Plugins & GGC ie GTY

[Basile STARYNKEVITCH]

Basile STARYNKEVITCH wrote:
> Richard Guenther wrote:
>>
>>
>> Plugins shouldn't keep permanent references to GCed memory.  At least
>> that would make it unnecessary to do what you suggest.
>
>
> I strongly disagree with that, and I simply do not understand your 
> position. In my perception, plugins are essentially loaded (dlopen-ed) 
> but never unloaded (dlclose-ed; perhaps they would be dlclose-ed when 
> every thing is done). Why should they not use any GTY stuff? What 
> makes you think that some features of GCC (like GTY ...) are invalid 
> inside plugins?
>
> I find your position as extreme as saying: plugins should never call 
> malloc or xcalloc, or should never call warning_at, or should never 
> call XXX! Why?
>
> So what are plugins for?

I really think we should provide some usage scenarios for plugins.

For argumenting GTY-ed static data in plugins, I would just mention some of the reasons GTY-ed data are already used in GCC.

First, most of us do know that a real garbage collector is the only way to manage complex data structures (with circularities, many kind of cross references, complex reference graph, ...). There is a reason why Lisp, Java, ML, ... have garbage collectors for several dozens of year, and GCC is becoming such a complex beast that GC is unavoidable.
[BTW, I am surprised that while some people are advocating recoding GCC in C++, nobody suggested to recode it in Java or C#]


And if garbage collection is avoidable in GCC, given the strong opposition it has, all the GTY & gengtype stuff would have been removed by now. The mere fact it is staying here is in my opinion very significant. If GC was not relevant in GCC, GGC & GTY would have gone long time ago. They didn't!

I find the mere fact that nobody bothered to remove GTY & gengtype from GCC significant, even if many people (me including) don't like very much the GGC implementation (I would prefer a systematic way of using a GC, and a rule as strong as "nobody delete data" except the garbage collector. I do admit that I am quite extremist on memory management. But memory management are non-modular issues: they are whole program features!).

To be more specific and concrete, many passes use GTY-ed data. It seems that the common scenario is to have some data built in one pass and reused in some other passes. Then there is no easy way to find out who would delete the data, 

Now, I don't see why a plugin won't fall in that "pattern". Definitely, some guy would want to code a plugin which provides eg two or three (not only one) passes in GCC, all related in the sense that they would operate on some common data. Here a garbage collector makes sense, and having some GTY-ed data (or a vector) inside a plugin is definitely relevant.

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Plugins & GGC ie GTY

[Joern Rennecke]

> And if garbage collection is avoidable in GCC, given the strong opposition it has, all the GTY & gengtype stuff would have been removed by now. The mere fact it is staying here is in my opinion very significant. If GC was not relevant in GCC, GGC & GTY would have gone long time ago. They didn't!

I believe it is avoidable, but:
- There is no consensus that it should be avoided, and there are more
  important issues to discuss.
- Even if there was a consensus to avoid it, nobody is willing to but in
  the time to change gcc and keep it leak-free (or merely keep memory leaks
  at a level comparable to GGC memory overhead).

> To be more specific and concrete, many passes use GTY-ed data. It seems that the common scenario is to have some data built in one pass and reused in some other passes. Then there is no easy way to find out who would delete the data,
> 
> Now, I don't see why a plugin won't fall in that "pattern". Definitely, some guy would want to code a plugin which provides eg two or three (not only one) passes in GCC, all related in the sense that they would operate on some common data. Here a garbage collector makes sense, and having some GTY-ed data (or a vector) inside a plugin is definitely relevant.

As long as you only need to GTY known types, you can avoid having extra GTY
roots by having all plugins share one GTY root in the plugin infrastructure;
this root can point to a list to which each plugin can add at will.

If you want new types, it gets ugly, because how can you independently
develop plugins and avoid collisions?  An enum to describe a type
won't be enough anymore, unless you want to go te a really silly width and
statart doling out namespaces.

You could approach it a bit more object-oriented and have a pointer in
each object that points to a descriptor for the type.
If you define such a type in the plugin infrastructure, it again becomes
a special kind of pre-defined type.
I suppose the main work will be to get the GTY parser to create the
type descriptors for custom types in modules. 
This e-mail was sent from a group e-mail system of ARC International Plc. Full details of the registered names and addresses of companies within the ARC group can be found on the ARC website.ARC International plc, Registered Office: Verulam Point, Station WaySt. Albans AL1 5HE United Kingdom Registered in England and Wales No.  3592130savm-exch03 
 
 
 
 
 
 
 

Re: Plugins & GGC ie GTY

[Steven Bosscher]

On Thu, Apr 2, 2009 at 12:48 AM, Joern Rennecke <joernr@arc.com> wrote:
>> And if garbage collection is avoidable in GCC, given the
>> strong opposition it has, all the GTY & gengtype stuff would
>> have been removed by now.

This looks like a rather uninformed opinion...


>> The mere fact it is staying here is
>> in my opinion very significant. If GC was not relevant in GCC,
>> GGC & GTY would have gone long time ago. They didn't!

Welcome... to the desert.. of the Real.

Once something is in GCC, it is near impossible to get it out again.
In this case, the "something" is PCH, which still heavily depends on
GGC.  That does *not* mean everyone (or even a majority) likes it.

And more importantly for this discussion: It also doesn't mean that
other, new infrastructure should increases GCC's dependency on the GGC
stuff.


> I believe it is avoidable, but:
> - There is no consensus that it should be avoided, and there are more
>  important issues to discuss.

Exactly!

Actually, I think there is more-or-less consensus that it should be
avoided if possible.  The current GTY machinery forces *everything*
into GC memory, mostly just to avoid broken PCH.  But that is a
problem with the design/implementation of the GGC stuff, not
necessarily with the concept (obstacks and pools are also not all
that...).

Some examples: RTL *must* live in GGC memory only for PCH.  Likewise
with alias set, and most static global variables decorated with a GTY
marker.  I don't think anyone really likes that.  I also don't think
anyone is really sure what should be decorated with a GTY marker and
what can safely be left undecorated (nobody ever un-decorated anything
after the tree-ssa merge, though I would expect some markers are
redundant now).


> - Even if there was a consensus to avoid it, nobody is willing to but in
>  the time to change gcc and keep it leak-free (or merely keep memory leaks
>  at a level comparable to GGC memory overhead).

This is easier nowadays than it used to be, when GCC switched from
obstacks to garbage collection.  Tools like valgrind make life a tad
less unfriendly for the debugger.  But yes, +1.


>> To be more specific and concrete, many passes use GTY-ed
>> data. It seems that the common scenario is to have some
>> data built in one pass and reused in some other passes.

I don't see this scenario really.  There is (should be?) almost no
data transfer between passes (except the intermediate representation,
which obviously has a special position in this discussion).


>> Then there is no easy way to find out who would delete
>> the data,

So Don't Do That (tm).


>> Now, I don't see why a plugin won't fall in that "pattern".
>> Definitely, some guy would want to code a plugin which
>> provides eg two or three (not only one) passes in GCC,

And use other types than the types already in GCC itself, and hand
that data over across a gcc_collect call?  Sounds broken by design to
me.  At least, I don't think that works for any pass even within GCC
itself, so why should it work for plugins?

You should not pass data from one pass to another via GC memory,
except the call graph and everything that hangs from there (cgraph ->
functions -> cfg -> stmts -> operands).  As long as you only modify
the existing intermediate representation, you don't have to worry
about GGC at all!

To me, it seems that plugins should be sufficiently self-contained to
take care of their own memory management, *without* GGC, except for
data that is shared between the plugin and the compiler proper (i.e.
the cgraph, etc.).

But you seem to imply that your plugin needs to do more than that...?


> As long as you only need to GTY known types, you can avoid having extra GTY
> roots by having all plugins share one GTY root in the plugin infrastructure;
> this root can point to a list to which each plugin can add at will.

I don't think this would work.  The marker routines wouldn't know what
would be in this one root with new types.  But, as you said...

> If you want new types, it gets ugly,

...there's an understatement!


>because how can you independently
> develop plugins and avoid collisions?  An enum to describe a type
> won't be enough anymore, unless you want to go te a really silly width and
> statart doling out namespaces.

Exactly.

How would you write the markers and how would you let the GTY
mechanism know which markers to call??  This is all automatically
generated from the GCC source now, but I don't see how that woudl work
with plugins unless you build the plugin along with GCC (and thereby
defeat the purpose of plugins).  Write them by hand?  More power to
you if you even think you *can* do that.

Ciao!
Steven

Re: Plugins & GGC ie GTY

[Basile STARYNKEVITCH]

Hello All


Joern Rennecke wrote:
>
>
> As long as you only need to GTY known types, you can avoid having extra GTY
> roots by having all plugins share one GTY root in the plugin infrastructure;
> this root can point to a list to which each plugin can add at will.
>
> If you want new types, it gets ugly, because how can you independently
> develop plugins and avoid collisions?  An enum to describe a type
> won't be enough anymore, unless you want to go te a really silly width and
> statart doling out namespaces.
>
>   
We don't need any enum to describe the type, unless the new type is 
added into existing ones.

To be more concrete, let's suppose at first that a plugin FOO, which is 
loaded by dlopen-ing FOO.so, uses GTY for some static roots. For 
simplication assume FOO is only one C source file FOO.c which contains 
therefore something like

struct someFOOstruct_st GTY(()) {
};

static GTY(()) struct someFOOstruct_st *someFOOptr;

...
#include "gt-FOO.h"

Now, I assume gengtype has been hacked to generate gt-FOO.h correctly 
from FOO.c. I hope that is not an impossible hypothesis [1].

The generated file gt-FOO.h contains something like

const struct ggc_root_tab gt_ggc_r_gt_FOO_h[] = {...}

We just have to add some small code [2] into the ggc_mark_roots routine 
of gcc/ggc-common.c to add the scanning using this gt_ggc_r_gt_FOO_h. 
This is not a big deal. We should simply add a routine 
ggc_register_plugin_root_tab(const struct ggc_root_tab*) that adds its 
argument to some (static variable in gcc/ggc-common.c which is a) linked 
list or vector, and have ggc_mark_root scan that list or vector by 
adding a few lines there. We require plugin initializers to call 
ggc_register_plugin_root_tab appropriatly, exactly like they do already 
call register_callback etc...

Now there is the precompiled header issue. As an important remark, PCH 
are not stable from one minor version of GCC to another (eg 4.3.1 to 
4.3.2). As another remark, PCH are not very used. So my first attitude 
would be simply that: any compilation involving a plugin cannot use or 
generate any precompiled header. And PCH and plugins are already messy, 
even without any additional GTY-ed roots, for the simple reason that 
plugins are adding new code (the dlopen-ed one) into GCC whose behavior 
is altered by the plugin.

If we really care about PCH with plugins (I really think we should not 
care that much initially and leave the subject to future work), we 
should write in the PCH file the full list of all the plugins, probably 
their crypto-checksum (like md5 or sha1) to be sure that the plugin 
loaded again is exactly the same as the plugin which was active when 
generating the PCH, etc...

I believe we should not care about PCH and plugins at first, 
independently of any GTY in plugins.


==============

Note [1]: the generated gt-FOO.h depends mostly of FOO.c, not of much 
else. There is indeed some generated code (specific to each GTY-ed 
struct) in gtype-desc.c, and we have to figure out how to generate it 
inside gt-FOO.h (or maybe another plugin specific generated file like 
gt2-FOO.h) and use it in plugins context. In gtype-desc.c there is 
another  struct ggc_root_tab gt_ggc_r_gtype_desc_c which we have to 
somehow make more dynamic (I don't know how exactly yet), and also the 
rtx_next array. Maybe we have to change the RTX_NEXT macro in rtl.h (but 
I don't know what this macro is for).

Note [2]: the code to add into ggc_mark_roots is quite similar to 
http://gcc.gnu.org/ml/gcc-patches/2009-01/msg00431.html; not a big deal, 
very probably less than 50 lines.

=========================


I don't claim to have figured out all the details, but I don't see what is very difficult in adding GTY-ed stuff in plugins (this is a feature
I really need to make MELT becoming a plugin) provided of course we
don't care yet about PCH & plugins, which is a difficult subject by
itself (even without any GTY inside plugins).




Comments (especially constructive comments) are welcome

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: Plugins & GGC ie GTY

[Justin Seyster]

Sorry for dragging this discussion out from the distant past.  I'm in
the process of porting some plug-ins from the old plugin SVN branch to
the new plug-in system, and this is one of the issues blocking me.  My
plug-ins maintain some tree nodes that I want to stay alive from
function to function.  Managing these nodes manually isn't an option,
since they get allocated (with garbage collection) by internal GCC
functions.

Basically, my plug-in wants to sprinkle logging function calls
throughout compiled code.  I generate the types and decls for these
with build_function_type_list() and build_fn_decl(), and then use them
when I need them.  Now, I don't _need_ to hold on to these references,
but regenerating them every time I use them seems inefficient to me.

> We just have to add some small code [2] into the ggc_mark_roots routine of
> gcc/ggc-common.c to add the scanning using this gt_ggc_r_gt_FOO_h. This is
> not a big deal. We should simply add a routine
> ggc_register_plugin_root_tab(const struct ggc_root_tab*) that adds its
> argument to some (static variable in gcc/ggc-common.c which is a) linked
> list or vector, and have ggc_mark_root scan that list or vector by adding a
> few lines there. We require plugin initializers to call
> ggc_register_plugin_root_tab appropriatly, exactly like they do already call
> register_callback etc...

This is in fact exactly the solution I was using in the old plugin
branch (though I never submitted the code once it became clear that
the newer plug-in system was going to take over).  I can send out the
patch if anyone is interested; it's as simple as it sounds.

With that extra bit of garbage collector support in place, I hacked up
a simple header file that provides macros to manually generate
ggc_add_plugin_root struct entries just like the ones gengtype would
generate (for a small set of types).  So a plug-in author can declare
a vector of tree nodes like so:

static GC_ROOT_TREE_VEC (my_tree_vector);

Another macro (to be called from plugin_init()) lets the author
register that root with the garbage collector.

register_root (my_tree_vector);

This isn't an end-all solution, but it actually provides all the
support that I need for my plug-ins, and it is perhaps a useful
stepping stone.

The main reason I'm bringing this up is because I've got a handful of
plug-ins I'd like to make public.  Without any support from the
garbage collector, I'll have to do some rewriting to get around the
issue.  If other plug-in authors thinks this is a useful idea (would
it be sufficient to port MELT, for instance?), I'd be willing to
prepare some patches to add it in.

        --Justin

Re: Plugins & GGC ie GTY

[Basile STARYNKEVITCH]

Justin Seyster wrote:
>
>   
>> We just have to add some small code [2] into the ggc_mark_roots routine of
>> gcc/ggc-common.c to add the scanning using this gt_ggc_r_gt_FOO_h. This is
>> not a big deal. We should simply add a routine
>> ggc_register_plugin_root_tab(const struct ggc_root_tab*) that adds its
>> argument to some (static variable in gcc/ggc-common.c which is a) linked
>> list or vector, and have ggc_mark_root scan that list or vector by adding a
>> few lines there. We require plugin initializers to call
>> ggc_register_plugin_root_tab appropriatly, exactly like they do already call
>> register_callback etc...
>>     
>
> This is in fact exactly the solution I was using in the old plugin
> branch (though I never submitted the code once it became clear that
> the newer plug-in system was going to take over).  I can send out the
> patch if anyone is interested; it's as simple as it sounds.
>
> With that extra bit of garbage collector support in place, I hacked up
> a simple header file that provides macros to manually generate
> ggc_add_plugin_root struct entries just like the ones gengtype would
> generate (for a small set of types).  So a plug-in author can declare
> a vector of tree nodes like so:
>
> static GC_ROOT_TREE_VEC (my_tree_vector);
>
> Another macro (to be called from plugin_init()) lets the author
> register that root with the garbage collector.
>
> register_root (my_tree_vector);
>
> This isn't an end-all solution, but it actually provides all the
> support that I need for my plug-ins, and it is perhaps a useful
> stepping stone.
>
> The main reason I'm bringing this up is because I've got a handful of
> plug-ins I'd like to make public.  Without any support from the
> garbage collector, I'll have to do some rewriting to get around the
> issue.  If other plug-in authors thinks this is a useful idea (would
> it be sufficient to port MELT, for instance?), I'd be willing to
> prepare some patches to add it in.
>
>   


I think the major issue is not to code the patch. It is to have it 
accepted in the trunk. My perception is that many people dislike GGC and 
reject the whole idea of garbage collection inside GCC [1] For example, 
http://gcc.gnu.org/ml/gcc-patches/2009-01/msg00431.html was a small 
[rejected] patch which would have provide the equivalent functionality.

I definitely dream of making MELT a plugin. What I need for this is

a. The ability to add a few (perhaps only 2 or 4) static or global GGC 
roots inside a plugin. One of these roots has a very special scanning 
routine.

b. The ability to add a few GTY-ed types. This would require essentially 
a patch to gengtype to have it run in a plugin mode where it take one 
single foo.c source files with GTY as input and output one single 
gt-foo.h file with the scanning/marking routine.

c. The ability to have a custom scanning/marking routine. This I could 
perhaps do with a #define gt_ggc_mx_FOOTYPE(Root) my_marking_rootine(Root)

The main reason of all this is that MELT does have its own GC backed up 
by GGC, and that MELT handle all its local pointers appropriately (in 
generated C code, this is quite easy: just hack the C generator). So in 
the infrequent case (only full MELT garbage collection, since the minor 
MELT GC is handled by MELT using a copying strategy) when GGC is called, 
a special routine has to mark all the local pointers. Remember that MELT 
is a Lisp dialect translated to C: you don't handle explicitly any GC 
local roots in MELT code, they appear only in the generated C code.

Also, GGC is perhaps more associated -in the mind of most people here- 
with precompiled headers than with memory management. MELT is not 
compatible with precompiled headers (in the sense that they could not be 
generated by MELT stuff; however, one could imagine loading some 
precompiled headers before using some MELT pass). My belief is that 
precompiled headers and plugins [2] are not really compatible (I am 
thinking of generating a *.gch file using plugins, I am not thinking of 
reading a *.gch file using a plugin; that should be already ok), and 
both are rarely used (so will probably be even less often used 
together). The insight is that there is no way to guarantee that a 
plugin would restore its entire state (at time of *.gch writing) when 
reading again the *.gch

Regards

Note [1]: my opposite view is that garbage collection is a 
whole-program-wide feature which is required in any sufficiently complex 
compiler (so I believe it is here to stay) but that GGC is not the 
garbage collector I'm dreaming (mostly because it does not handle local 
roots), burt barely enough for me. I am aware that my position is in the 
minority.

Note [2]: See the 
http://gcc.gnu.org/ml/gcc-patches/2009-04/msg02233.html thread

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***