plugin issues to fix (or document) before 4.5 release

plugin issues to fix (or document) before 4.5 release

[Basile STARYNKEVITCH]

Hello All,

I believe there are several plugin issues to fix before 4.5 releases:

1. use of libiberty from plugins.

   As several patches recently sent demonstrated, the current state of the trunk does not work with plugins calling some 
of the libiberty functions is IMHO not acceptable.

   we could take some of the following solutions

   a) explicitly document in plugins.texi that libiberty is not callable from plugins (e.g. functions like pex_execute 
or make_temp_file, which are not currently linked into cc1). This is the easiest to do. My feeling is that it would be 
very unfortunate. Libiberty is documented as a portability layer; if plugins cannot use it, that means that plugins will 
never be a way of experimenting code which might (much latter) be (partly) proposed into the trunk. So giving up 
libiberty in plugins is an important social decision; it discourage people coding plugins to try to propose (much 
latter, when their plugin has a big enough audience) their code into the trunk latter.

   b) use some tricks to ensure that libiberty remains usable from plugins (I proposed a very crude one in
http://gcc.gnu.org/ml/gcc-patches/2009-11/msg01413.html - it is very ugly, but it should be highly portable, which is 
the aim of libiberty).

   c) when plugins are enabled, link libiberty as a dynamic libiberty.so (this has been sadly rejected some time ago). 
My feeling is that when cc1 depends upon dlopen, we might reconsider that decision, and have the dynamic library 
libiberty.so linked into cc1 when gcc is configured with plugins enabled  (leaving cc1 linking statically libiberty.a 
when plugins are disabled at configure time). I do confess that not being a configure.ac guru, I am not able to propose 
a good patch for that, and I don't know anymore well enough non-linux systems offering dlopen (I used to know quite well 
old SunOS5.2 about dlopen stuff) to understand all the issues involved.


2. organizing the plugin/ directory (and hinting how plugins could be distributed)
AFAIK this directory is only mentionned in http://gcc.gnu.org/onlinedocs/gccint/Plugins.html#Plugins §23.7 Building GCC 
plugins as containing the include/ directory required to compile plugins. We don't tell anything about what other stuff 
it could (or not contain), and if we really think it should only contain header files, why do we call it this way (it 
should better be called plugins-include), and why do we put an include/ sub-directory inside?
My personal feeling is that some few plugins -probably the most complex ones- could need to have some additional files 
for them (default configuration files, extra parameters, ...), in particular plugins providing extra domain or 
application specific diagnostics [like those wished inside big graphical toolkits like Gtk or Qt, or inside the linux 
kernel, etc..] will probably want to add extra information about those functions they know about outside of system 
header files (so perhaps in some other files). Likewise, plugin providing application or library specific optimisations 
(a trivial simplistic example is transforming fprintf(stdout, ...) into printf(...), but I am sure most people could 
imagine better examples, e.g. specializing qsort calls depending upon the size of the sorted array, and the compare 
functions, ...) probably might be conplex enough to want to have some files for them accessible by plugins.

We could even define a convention for system installed plugins, which would help the average user. Instead of typing 
-fplugin=/a/long/complex/path/to/a/common/plugin/like/foo.so he would just type -fplugin=foo. This is quite easy to 
implement.

And "complex" plugins needing to access their files have so far no convention or function to assist them. We could

a. minimally document that plugins need to organize entirely themselves any additional resources (notably files) that 
they require, and that it is not the business of GCC to help organize that. This would lead to huge heterogenity, even 
between different but similar Linux distributions. In particular, a same common plugin provided by two different linux 
distributions would have to be invoked differently....

b. make the plugin directory accessible from plugins and provide a function (usable from plugins) to query that.
I proposed a buggy patch in http://gcc.gnu.org/ml/gcc-patches/2009-11/msg01403.html. By thinking more about it, I would 
believe that the gcc driver should actually explicitly pass to cc1 (and other) programs the path to the plugin 
directory, so that querying it from plugins would be much easier. So that apparently means mostly a gcc *.spec file 
patch (I am not very familiar with spec files).

All the above issues are motivated by my MELT experience. In the current state, I am able to overcome them, in 
particular by duplicating withing MELT as plugin some functionality already provided by gcc (but not accessible to 
plugins), and/or by having my own conventions. But all the issues involved are not specific to MELT, and any other big 
or complex enough plugin will encounter them. Again, I can live without, but painfully.


3. Internationalization of plugins. I don't know much about gettext & friends (and most importantly, I don't understand 
all of the localization issues inside GCC) but I am not sure we have enough features to localize the warnings emitted by 
plugins. Leaving that entirely to plugins might be a mistake (again, because different Linux distributions might provide 
different answers). I agree it is not that important today: 4.5 plugins could be restricted to the C locale in practice!


Maybe what is just missing are a few sentences in plugin.texi about the shortcomings identified in this message (and a 
mention that most issues would be fixed in 4.6).

As a whole, plugins are a very nice feature of 4.5 (the other nice feature of 4.5 is LTO), and in my view they are very 
positive and will bring more people interested in GCC internals, ultimately enforcing the GCC community. But I do know 
that some people disagree with that view, and are scared, or not interested in, plugins becoming a success and widely 
used. My feeling & hope is that maybe within a year, some few plugins might be successful enough to be commonly used.
And certainly, plugins will permit some specialized processing (library or domain specific diagnostic or optimisation) 
which probably will never go into the trunk - and this is a good point.

All the issues involved in this email are not theoretical in any sense: the MELT plugin do encounter them, and have 
naughty tricks to avoid (ie circumvent) them. It would be much better if it didn't had to avoid these issues.

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: plugin issues to fix (or document) before 4.5 release

[Rafael Espindola]

2009/11/29 Basile STARYNKEVITCH <basile@starynkevitch.net>:
> Hello All,
>
> I believe there are several plugin issues to fix before 4.5 releases:
>
> 1. use of libiberty from plugins.
>
>  As several patches recently sent demonstrated, the current state of the
> trunk does not work with plugins calling some of the libiberty functions is
> IMHO not acceptable.
>
>  we could take some of the following solutions
>
>  a) explicitly document in plugins.texi that libiberty is not callable from
> plugins (e.g. functions like pex_execute or make_temp_file, which are not
> currently linked into cc1). This is the easiest to do. My feeling is that it
> would be very unfortunate. Libiberty is documented as a portability layer;
> if plugins cannot use it, that means that plugins will never be a way of
> experimenting code which might (much latter) be (partly) proposed into the
> trunk. So giving up libiberty in plugins is an important social decision; it
> discourage people coding plugins to try to propose (much latter, when their
> plugin has a big enough audience) their code into the trunk latter.

So, for 4.5 I think that documenting the issue might be the right thing.

For 4.6 I am not sure. Do the functions you want to use keep internal
state? If they don't, it should be safe to link a static (but PIC)
libiberty in the plugin. The objection for a libiberty.so was that we
would have to start versioning it and making sure the ABI was stable.
Was there something else?

If the only objection to using a libiberty.so is the ABI, maybe we
could install it in a non standard place and use rpaths. That way gcc
4.4 and 4.5 can use completely different and incompatible versions of
libiberty. The plugin would have to make sure that it is linked with
the correct one, but that is in line with we not making any promises
for backward compatibility.

Are rpaths as portable as shared libraries or do we support a host
architecture that has shared libraries but no equivalent to rpath?


That failing, IMHO, the best proposal is  to have the build system do
its best at passing --whole-archive or equivalent when linking
libiberty in cc1.

>
> Regards.
>
> --
> Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
> email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
> 8, rue de la Faiencerie, 92340 Bourg La Reine, France
> *** opinions {are only mines, sont seulement les miennes} ***
>


Cheers,
-- 
Rafael Ávila de Espíndola

Re: plugin issues to fix (or document) before 4.5 release

[Arnaud Charlet]

> Are rpaths as portable as shared libraries or do we support a host
> architecture that has shared libraries but no equivalent to rpath?

Windows (mingw) comes to mind at least.

Arno

Re: plugin issues to fix (or document) before 4.5 release

[Dave Korn]

Arnaud Charlet wrote:
>> Are rpaths as portable as shared libraries or do we support a host
>> architecture that has shared libraries but no equivalent to rpath?
> 
> Windows (mingw) comes to mind at least.

  Cygwin too, of course, and I think Darwin may have some quirks in that area
as well.

    cheers,
      DaveK

Re: plugin issues to fix (or document) before 4.5 release

[Joseph S. Myers]

On Thu, 3 Dec 2009, Arnaud Charlet wrote:

> > Are rpaths as portable as shared libraries or do we support a host
> > architecture that has shared libraries but no equivalent to rpath?
> 
> Windows (mingw) comes to mind at least.

If the hypothetical libiberty.dll were only used by cc1 etc. (not by the 
driver), putting it in the same directory as cc1, libexecsubdir, where 
Windows would expect to find it, should work to allow separate copies for 
each GCC install.  Though I prefer the --whole-archive approach (with a 
fallback of adding references to libiberty symbols or extracting the .o 
files if we wish to support plugins on hosts without --whole-archive or 
equivalent).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: plugin issues to fix (or document) before 4.5 release

[Dave Korn]

Joseph S. Myers wrote:
> On Thu, 3 Dec 2009, Arnaud Charlet wrote:
> 
>>> Are rpaths as portable as shared libraries or do we support a host
>>> architecture that has shared libraries but no equivalent to rpath?
>> Windows (mingw) comes to mind at least.
> 
> If the hypothetical libiberty.dll were only used by cc1 etc. (not by the 
> driver), putting it in the same directory as cc1, libexecsubdir, where 
> Windows would expect to find it, should work to allow separate copies for 
> each GCC install.  Though I prefer the --whole-archive approach (with a 
> fallback of adding references to libiberty symbols or extracting the .o 
> files if we wish to support plugins on hosts without --whole-archive or 
> equivalent).

  Has anyone tested plugins on windows yet?  (I haven't, but I'll find some
time to give it a bit of basic testing during stage 4.)

    cheers,
      DaveK

Re: plugin issues to fix (or document) before 4.5 release

[Kai Tietz]

2009/12/3 Dave Korn <dave.korn.cygwin@googlemail.com>:
> Joseph S. Myers wrote:
>> On Thu, 3 Dec 2009, Arnaud Charlet wrote:
>>
>>>> Are rpaths as portable as shared libraries or do we support a host
>>>> architecture that has shared libraries but no equivalent to rpath?
>>> Windows (mingw) comes to mind at least.
>>
>> If the hypothetical libiberty.dll were only used by cc1 etc. (not by the
>> driver), putting it in the same directory as cc1, libexecsubdir, where
>> Windows would expect to find it, should work to allow separate copies for
>> each GCC install.  Though I prefer the --whole-archive approach (with a
>> fallback of adding references to libiberty symbols or extracting the .o
>> files if we wish to support plugins on hosts without --whole-archive or
>> equivalent).
>
>  Has anyone tested plugins on windows yet?  (I haven't, but I'll find some
> time to give it a bit of basic testing during stage 4.)
>
>    cheers,
>      DaveK
>
>

No, we are testing at the moment the different new optimizations of
gcc. The plugin feature is on our todo list.

Cheers,
Kai

-- 
|  (\_/) This is Bunny. Copy and paste
| (='.'=) Bunny into your signature to help
| (")_(") him gain world domination