libiberty should be a shared library when cc1 has plugin enabled.

libiberty should be a shared library when cc1 has plugin enabled.

[Basile Starynkevitch]

Hello All,

Perhaps libiberty should be a shared library, not a static one, linked from
cc1, when GCC has plugin enabled.

I noticed the following in the MELT branch (while trying to build MELT as a
GCC-Trunk plugin).

Some functions of libiberty.h are not linked in cc1 (because cc1 don't call
them, and libliberty.a is a static library, not a shared one). A concrete
example is make_temp_file. It is used in gcc.c but not in the entire cc1,
and one could imagine a plugin might want it. But if a plugin calls it, the
dlopen of that plugin fails with undefined symbol: make_temp_file.

Of course, for that simple case a workaround is possible (at least on
Linux): use a standard temporary file function like tmpfile() instead of
make_temp_file.

But it seems to me that a plugin can call a libliberty function only if that
function is already referenced (e.g. called) from cc1. This is not the case
of all libiberty functions.

We might also artificially add a reference to each libiberty function from
cc1. Or we should at least document (perhaps as a comment in libiberty.h)
that all these functions are not available from plugins.

Linking statically libiberty.a into a plugin is not recommended (because
*.so should not contain non PIC code on many platforms).

If we did link dynamically libiberty.so:

* execution time (of cc1) might suffer a bit, because perhaps calling a
function in a dynamic library could be slower than calling it in a static
library.

* every plugin could use all of libliberty.so

* the Makefile.in files should be changed.


Comments are welcome.

Regards.
-- 
Basile STARYNKEVITCH http://starynkevitch.net/Basile/ 
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359 
8, rue de la Faiencerie, 92340 Bourg La Reine, France 
*** opinions {are only mines, sont seulement les miennes} ***

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Dave Korn]

Basile Starynkevitch wrote:
> Hello All,
> 
> Perhaps libiberty should be a shared library, not a static one, linked from
> cc1, when GCC has plugin enabled.

> We might also artificially add a reference to each libiberty function from
> cc1. 

  Or link it into cc1 et al. using "--whole-archive".

> If we did link dynamically libiberty.so:

  We would also have to install it, and start worrying about library API
versioning and backward compatibility, or at any rate I think that's the main
reason why this has not been done in the pasy (cf. libbfd).

    cheers,
      DaveK

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Basile STARYNKEVITCH]

Dave Korn wrote:
>
>> We might also artificially add a reference to each libiberty function from
>> cc1. 
>>     
>
>   Or link it into cc1 et al. using "--whole-archive".
>
>   
Sorry, I am not aware of this option. And are we sure it works on all 
the systems GCC is supposed to run on?
>> If we did link dynamically libiberty.so:
>>     
>
>   We would also have to install it, ..

Agreed.

Anyway, we really should have all of libiberty inside cc1. What is the 
best way to achieve that so that it works on all the systems GCC wants?

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Dave Korn]

Basile STARYNKEVITCH wrote:
> Dave Korn wrote:
>>
>>> We might also artificially add a reference to each libiberty function
>>> from
>>> cc1.     
>>
>>   Or link it into cc1 et al. using "--whole-archive".
>>
>>   
> Sorry, I am not aware of this option. And are we sure it works on all
> the systems GCC is supposed to run on?

  Ach.  It requires GNU ld, of course.  We could just link the objects from
the libiberty/ dir into cc1 (etc.) instead of the lib .a archive.

    cheers,
      DaveK

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Basile STARYNKEVITCH]

Dave Korn wrote:
> Basile STARYNKEVITCH wrote:
>   
>> Dave Korn wrote:
>>     
>>>> We might also artificially add a reference to each libiberty function
>>>> from
>>>> cc1.     
>>>>         
>>>   Or link it into cc1 et al. using "--whole-archive".
>>>
>>>   
>>>       
>> Sorry, I am not aware of this option. And are we sure it works on all
>> the systems GCC is supposed to run on?
>>     
>
>   Ach.  It requires GNU ld, of course.  We could just link the objects from
> the libiberty/ dir into cc1 (etc.) instead of the lib .a archive.

Anyway, I am not able to do that well. The point is that libiberty is 
designed to provide a common interface to all the many systems GCC is 
supposed to run on, and even if restricting ourselves to those having 
ELF & a working dlopen, that makes a lot. In other words, I am 
unfamiliar with libiberty (and I am not sure to understand why in 2009, 
with the existing Posix & OpenGroup standards, it is still required. I 
suppose it is a legacy).
Also, I don't know if we should patch libiberty or gcc/ directory to 
solve that (I am not sure if libiberty is legally a part of GCC; ie does 
the legal right to patch GCC is enough for libliberty, which is perhaps 
shared with other stuff like binutils).

The quick & dirty fix would be to add inside eg gcc-plugin.c a table of 
function pointers, or perhaps some code, referencing most (or preferably 
all) of libiberty functions.

While pluginifying MELT I have to add a lot of dirty patches to avoid 
(i.e. circumvent them) libiberty functions. This is a pity!

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Daniel Jacobowitz]

On Thu, Jul 09, 2009 at 10:49:49AM +0200, Basile Starynkevitch wrote:
> But it seems to me that a plugin can call a libliberty function only if that
> function is already referenced (e.g. called) from cc1. This is not the case
> of all libiberty functions.

So... link libiberty into your plugin and get make_temp_file that way.

-- 
Daniel Jacobowitz
CodeSourcery

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Basile STARYNKEVITCH]

Daniel Jacobowitz wrote:
> On Thu, Jul 09, 2009 at 10:49:49AM +0200, Basile Starynkevitch wrote:
>   
>> But it seems to me that a plugin can call a libliberty function only if that
>> function is already referenced (e.g. called) from cc1. This is not the case
>> of all libiberty functions.
>>     
>
> So... link libiberty into your plugin and get make_temp_file that way.
>   

As far as I understand ELF shared objects, it is not recommended to link 
a static library into a dynamic plugin (i.e. an ELF shared object), 
because the static library is not compiled with -fPIC.
(if you do that, the resulting plugin is getting a huge lot of 
relocation information).
In simpler words, *.so have to be compiled with -fPIC, and libiberty is 
not compiled with -fPIC.

I am clear enough or should I explain more?

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Ralf Wildenhues]

* Daniel Jacobowitz wrote on Thu, Jul 09, 2009 at 02:50:04PM CEST:
> On Thu, Jul 09, 2009 at 10:49:49AM +0200, Basile Starynkevitch wrote:
> > But it seems to me that a plugin can call a libliberty function only if that
> > function is already referenced (e.g. called) from cc1. This is not the case
> > of all libiberty functions.
> 
> So... link libiberty into your plugin and get make_temp_file that way.

Ahh, another opportunity for subtle bugs stemming from multiple entities
of static data, such as random numbers used in a plugin being
suspiciously correlated to random numbers used in cc1 proper,
setenv in cc1 segfaulting because a plugin used it too, invalidating
cc1's copy of the last_environ pointer ...

Cheers,
Ralf

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Daniel Jacobowitz]

On Thu, Jul 09, 2009 at 03:01:01PM +0200, Basile STARYNKEVITCH wrote:
> In simpler words, *.so have to be compiled with -fPIC, and libiberty
> is not compiled with -fPIC.

We build a PIC libiberty already.

While Ralf's point about static data is valid, the functions likely to
be in libiberty on any platform supporting plugins should not suffer
from this problem.

If you're concerned about it, then build a subset.  I've considered a
separation of libiberty into replacements and utilities, anyway.

-- 
Daniel Jacobowitz
CodeSourcery

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Basile STARYNKEVITCH]

Daniel Jacobowitz wrote:
> On Thu, Jul 09, 2009 at 03:01:01PM +0200, Basile STARYNKEVITCH wrote:
>   
>> In simpler words, *.so have to be compiled with -fPIC, and libiberty
>> is not compiled with -fPIC.
>>     
>
> We build a PIC libiberty already.
>
>   

Thanks!

Where and how is it built? I cannot find any *iberty*.so in my build tree!

Regards.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Daniel Jacobowitz]

On Thu, Jul 09, 2009 at 03:45:52PM +0200, Basile STARYNKEVITCH wrote:
> Daniel Jacobowitz wrote:
> >On Thu, Jul 09, 2009 at 03:01:01PM +0200, Basile STARYNKEVITCH wrote:
> >>In simpler words, *.so have to be compiled with -fPIC, and libiberty
> >>is not compiled with -fPIC.
> >
> >We build a PIC libiberty already.
> >
> 
> Thanks!
> 
> Where and how is it built? I cannot find any *iberty*.so in my build tree!

It's an archive, not a shared library.  It's probably in
libiberty/pic/ but there's configury involved.

-- 
Daniel Jacobowitz
CodeSourcery

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Basile STARYNKEVITCH]

Daniel Jacobowitz wrote:
> On Thu, Jul 09, 2009 at 03:45:52PM +0200, Basile STARYNKEVITCH wrote:
>   
>> Daniel Jacobowitz wrote:
>>     
>>> On Thu, Jul 09, 2009 at 03:01:01PM +0200, Basile STARYNKEVITCH wrote:
>>>       
>>>> In simpler words, *.so have to be compiled with -fPIC, and libiberty
>>>> is not compiled with -fPIC.
>>>>         
>>> We build a PIC libiberty already.
>>>
>>>       
>> Thanks!
>>
>> Where and how is it built? I cannot find any *iberty*.so in my build tree!
>>     
>
> It's an archive, not a shared library.  It's probably in
> libiberty/pic/ but there's configury involved.
>   

Perhaps we should document that in gcc/docs/plugin.texi?

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Ralf Wildenhues]

* Daniel Jacobowitz wrote on Thu, Jul 09, 2009 at 03:43:18PM CEST:
> While Ralf's point about static data is valid, the functions likely to
> be in libiberty on any platform supporting plugins should not suffer
> from this problem.

Solaris 9 and IRIX 6.5 support dlopen; gnulib documents them as missing
setenv, and of course they are tertiary platforms only.  However, I
wouldn't be surprised if plugins such as melt used setenv, esp. if they
spawn other processes, e.g., to compile code.

> If you're concerned about it, then build a subset.  I've considered a
> separation of libiberty into replacements and utilities, anyway.

Why would that help in this case?  Even if the static data issue
concerns one set of these functions only now (does it?), it won't
prevent anyone from adding problems to the other set tomorrow, unless
you also introduce a policy that libiberty functions be safe against
multiple entities.

Cheers,
Ralf

Re: libiberty should be a shared library when cc1 has plugin enabled.

[Basile STARYNKEVITCH]

Ralf Wildenhues wrote:
> * Daniel Jacobowitz wrote on Thu, Jul 09, 2009 at 03:43:18PM CEST:
>   
>> While Ralf's point about static data is valid, the functions likely to
>> be in libiberty on any platform supporting plugins should not suffer
>> from this problem.
>>     
>
> Solaris 9 and IRIX 6.5 support dlopen; gnulib documents them as missing
> setenv, and of course they are tertiary platforms only.  However, I
> wouldn't be surprised if plugins such as melt used setenv, esp. if they
> spawn other processes, e.g., to compile code.
>
>   
MELT is not currently using setenv yet, but may very easily need to use 
it. MELT does spawn processes already and uses getenv.
>> If you're concerned about it, then build a subset.  I've considered a
>> separation of libiberty into replacements and utilities, anyway.
>>     
>
> Why would that help in this case?  Even if the static data issue
> concerns one set of these functions only now (does it?), it won't
> prevent anyone from adding problems to the other set tomorrow, unless
> you also introduce a policy that libiberty functions be safe against
> multiple entities.

I do agree with this point.


Currently, in plugin mode, MELT does not use libiberty any more (which 
is a shame). So far, the only functions annoying me are pex_execute & 
friends (-in plugin mode MELT is simply calling system-) and 
choose_tmpdir (-in plugin mode MELT just calls tmpnam, which is pityful).


But we really need to decide if all of libiberty is usable in plugins. I 
believe it should be usable in plugins, because that is the easiest way 
to prototype future GCC core code: make it a plugin, and when it is 
ready & mature, propose it as a patch. I think that plugins are a new 
possibility in GCC, and FSF owned plugins should be welcomed (sadly I 
know that most people disagree with that).

We could also state that libiberty is becoming deprecated (in favor of 
recent standards like some flavor of Posix, ...). [I am half joking; I 
understand that they are many platforms requiring libiberty]

But we need to state a clear policy about libiberty & plugins. Can 
plugins use all of libiberty or should they use only functions already 
called by cc1?

[I am not able to propose a patch about this libiberty point because I 
do not understand all the portability issues]

Regards.

PS. I hope that my RTLD_GLOBAL patch 
http://gcc.gnu.org/ml/gcc-patches/2009-07/msg00497.html will be reviewed 
& accepted. It is essential (& tiny).

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***