[RFC] gcc lto&binutils: Load different gcc's bfd-plugin automatically

[RFC] gcc lto&binutils: Load different gcc's bfd-plugin automatically

[JunMa]

Hi All

We are now optimizing some projects with lto enabled, however,
there are some issues.
First, lto_plugin.so needs to be passed to ar/nm/ranlib.
For example, build static library with lto:

gcc -flto -O2 a.c -c -o a.o
gcc -flto -O2 b.c -c -o b.o
ar rcs --plugin=/path/to/lto_plugin.so   libx.a  a.o b.o

This is a little bit anoying. Also, it is not easy and convincible to use
gcc-ar/gcc-nm/gcc-ranlib on those projects. Luckily, binutils offers a
default plugin searching path(/lib/bfd-plugins), it can load plugin
automatically from that directory.

However, this brings up the second issue: binutils doesn't support multiple
version plugins of gccs which have the same plugin name:"lto_plugin.so"
int the same directory. Since these projects require different versions of
gccs, multiple gccs co-exist in our build system, we cannot put lto_plugin
in /lib/bfd-plugins. Although plugins may be compatible with each other,
we do want to decouple it among different versions of GCC.

I also have seen some discussions in
https://bugzilla.redhat.com/show_bug.cgi?id=1467409
where I don't seeany clear solutions.

I thought about this and had some ideas. I want ask for some feedback here.
The idea is to use versioned plugin searching paths
(/lib/bfd-plugins/$cc/$target/$version)
The $cc and $version information can be found in .comment section
of elf  file, as for file which does not have comment section, just keep
original searching path(/lib/bfd-plugins).

Here are few steps:
1) ar/nm/ranlib keep same behavior when '--plugin' is passed.
2) If --plugin is missed, find whether there are at least one .comment
section in object file/archive file if not goto 5).
3) for elf object file, get $cc and $version information. for elf 
archive file,
iterate comment section of all object files in archive, make sure $cc and
$version are same. if not, goto 5).
4) Get $target from target_alias variable in configure.
5) Find plugin from /lib/bfd-plugins/$cc/$target/$version or
/lib/bfd-plugins directory.

Looking forward to your replies!

Regards
Jun

Re: [RFC] gcc lto&binutils: Load different gcc's bfd-plugin automatically

[Richard Biener]

On Tue, Mar 5, 2019 at 10:41 AM JunMa <JunMa@linux.alibaba.com> wrote:
>
> Hi All
>
> We are now optimizing some projects with lto enabled, however,
> there are some issues.
> First, lto_plugin.so needs to be passed to ar/nm/ranlib.
> For example, build static library with lto:
>
> gcc -flto -O2 a.c -c -o a.o
> gcc -flto -O2 b.c -c -o b.o
> ar rcs --plugin=/path/to/lto_plugin.so   libx.a  a.o b.o
>
> This is a little bit anoying. Also, it is not easy and convincible to use
> gcc-ar/gcc-nm/gcc-ranlib on those projects. Luckily, binutils offers a
> default plugin searching path(/lib/bfd-plugins), it can load plugin
> automatically from that directory.
>
> However, this brings up the second issue: binutils doesn't support multiple
> version plugins of gccs which have the same plugin name:"lto_plugin.so"
> int the same directory. Since these projects require different versions of
> gccs, multiple gccs co-exist in our build system, we cannot put lto_plugin
> in /lib/bfd-plugins. Although plugins may be compatible with each other,
> we do want to decouple it among different versions of GCC.
>
> I also have seen some discussions in
> https://bugzilla.redhat.com/show_bug.cgi?id=1467409
> where I don't seeany clear solutions.
>
> I thought about this and had some ideas. I want ask for some feedback here.
> The idea is to use versioned plugin searching paths
> (/lib/bfd-plugins/$cc/$target/$version)
> The $cc and $version information can be found in .comment section
> of elf  file, as for file which does not have comment section, just keep
> original searching path(/lib/bfd-plugins).

Well, why not go a step further and add a bfd-plugin note that suggests
the plugin to be used if it is installed?  That could contain for example
lto_plugin_gcc8.so (to be installed in /lib/bfd-plugins/).  Alternatively
a full path could be specified (though the files wouldn't then necessarily
work when moving between different compiler installs).

The install location could be modified to a location BFD searches just
for this note/comment but not for other auto-loading tries.

Or, since mostly archive related stuff is the issue, we could bundle
the plugin as a special archieve member... (ok, that creates a chicken
and egg issue at archieve creation time).

Richard.

> Here are few steps:
> 1) ar/nm/ranlib keep same behavior when '--plugin' is passed.
> 2) If --plugin is missed, find whether there are at least one .comment
> section in object file/archive file if not goto 5).
> 3) for elf object file, get $cc and $version information. for elf
> archive file,
> iterate comment section of all object files in archive, make sure $cc and
> $version are same. if not, goto 5).
> 4) Get $target from target_alias variable in configure.
> 5) Find plugin from /lib/bfd-plugins/$cc/$target/$version or
> /lib/bfd-plugins directory.
>
> Looking forward to your replies!
>
> Regards
> Jun

Re: [RFC] gcc lto&binutils: Load different gcc's bfd-plugin automatically

[JunMa]

在 2019/3/5 下午5:55, Richard Biener 写道:
> On Tue, Mar 5, 2019 at 10:41 AM JunMa <JunMa@linux.alibaba.com> wrote:
>> Hi All
>>
>> We are now optimizing some projects with lto enabled, however,
>> there are some issues.
>> First, lto_plugin.so needs to be passed to ar/nm/ranlib.
>> For example, build static library with lto:
>>
>> gcc -flto -O2 a.c -c -o a.o
>> gcc -flto -O2 b.c -c -o b.o
>> ar rcs --plugin=/path/to/lto_plugin.so   libx.a  a.o b.o
>>
>> This is a little bit anoying. Also, it is not easy and convincible to use
>> gcc-ar/gcc-nm/gcc-ranlib on those projects. Luckily, binutils offers a
>> default plugin searching path(/lib/bfd-plugins), it can load plugin
>> automatically from that directory.
>>
>> However, this brings up the second issue: binutils doesn't support multiple
>> version plugins of gccs which have the same plugin name:"lto_plugin.so"
>> int the same directory. Since these projects require different versions of
>> gccs, multiple gccs co-exist in our build system, we cannot put lto_plugin
>> in /lib/bfd-plugins. Although plugins may be compatible with each other,
>> we do want to decouple it among different versions of GCC.
>>
>> I also have seen some discussions in
>> https://bugzilla.redhat.com/show_bug.cgi?id=1467409
>> where I don't seeany clear solutions.
>>
>> I thought about this and had some ideas. I want ask for some feedback here.
>> The idea is to use versioned plugin searching paths
>> (/lib/bfd-plugins/$cc/$target/$version)
>> The $cc and $version information can be found in .comment section
>> of elf  file, as for file which does not have comment section, just keep
>> original searching path(/lib/bfd-plugins).
> Well, why not go a step further and add a bfd-plugin note that suggests
> the plugin to be used if it is installed?  That could contain for example
> lto_plugin_gcc8.so (to be installed in /lib/bfd-plugins/).  Alternatively
> a full path could be specified (though the files wouldn't then necessarily
> work when moving between different compiler installs).
Well, I also thought about add new section to contain lto_plugin info, 
however,
in order to parse lto related sections in binutils, you need load lto_plugin
first. I'm not sure whether this idea can be accepted.(it's sort of another
chicken and egg issue).
> The install location could be modified to a location BFD searches just
> for this note/comment but not for other auto-loading tries.
Of course, I prefer this general idea than only support elf.
> Or, since mostly archive related stuff is the issue, we could bundle
> the plugin as a special archieve member... (ok, that creates a chicken
> and egg issue at archieve creation time).
>
> Richard.
>> Here are few steps:
>> 1) ar/nm/ranlib keep same behavior when '--plugin' is passed.
>> 2) If --plugin is missed, find whether there are at least one .comment
>> section in object file/archive file if not goto 5).
>> 3) for elf object file, get $cc and $version information. for elf
>> archive file,
>> iterate comment section of all object files in archive, make sure $cc and
>> $version are same. if not, goto 5).
>> 4) Get $target from target_alias variable in configure.
>> 5) Find plugin from /lib/bfd-plugins/$cc/$target/$version or
>> /lib/bfd-plugins directory.
>>
>> Looking forward to your replies!
>>
>> Regards
>> Jun
Regards
Jun

Re: [RFC] gcc lto&binutils: Load different gcc's bfd-plugin automatically

[Joseph Myers]

On Tue, 5 Mar 2019, Richard Biener wrote:

> Well, why not go a step further and add a bfd-plugin note that suggests
> the plugin to be used if it is installed?  That could contain for example
> lto_plugin_gcc8.so (to be installed in /lib/bfd-plugins/).  Alternatively
> a full path could be specified (though the files wouldn't then necessarily
> work when moving between different compiler installs).

I think specifying full paths to code to execute in object files if very 
dubious on security grounds.

-- 
Joseph S. Myers
joseph@codesourcery.com