Update LTO plugin interface

Update LTO plugin interface

[H.J. Lu]

Hi,

Here is a proposal to update LTO plugin interface.  Any comments?

Thanks.

-- 
H.J.
---
Goal:  We should preserve the same linker command line order as if
there are no IR.
Problem:
	a. LTO may generate extra symbol references which aren't in IR.
	b. It was worked around with -pass-through hack.  But it doesn't
preserve the link command line order.

Proposal:
	a. Remove -pass-through hack in GCC.
	b. Compiler plugin controls what linker uses to generate the final executable:
		i. The linker command line order should be the same, with or without LTO.
	c. Add a cmdline bit field to
	struct ld_plugin_input_file
	{
	   const char *name;
	   int fd;
	   off_t offset;
	   off_t filesize;
	   void *handle;
	   unsigned int cmdline : 1;
	};
	It is used by linker to tell plugin that the input file comes from
linker command line.
	d. 2 stage linker:
		i. Stage 1: Normal symbol resolution with plugin.
		ii. Stage 2:
			1) Call the "all symbols read" handler to get the final linker inputs.
			2) Discard all previous inputs.
			3) Generate the final executable with inputs from plugin.
	e. Compiler plugin:
		i. For a file, which comes from the linker command line and isn't
claimed by plugin, save it in the linker pass-through list in the same
order as it comes in.
		ii. For the first file claimed by plugin,  remember the last
pass-through linker input.
		iii. The "all symbols read" handler adds input files to the linker
in the order:
			1) All linker input files on the linker pass-through list up to the
first file claimed by plugin.
			2) All linker input files generated by plugin.
			3) The rest of linker input files on the linker pass-through list.
	f. Limitation:
		i. All files claimed by plugin are grouped together.  Any archives
between files claimed by plugin are placed after all linker input
files generated by plugin when passed to linker.

Re: Update LTO plugin interface

[Basile Starynkevitch]

On Wed, 1 Dec 2010 10:18:58 -0800
"H.J. Lu" <hjl.tools@gmail.com> wrote:

> Here is a proposal to update LTO plugin interface.  

How should we parse the above sentence?

Is it about an interface to plugin inside binutils to support LTO?

Is it about an interface for GCC plugins to help them be more LTO friendly?

Is it about an interface inside the GOLD linker to dlopen the LTO plugin provided with GCC sources?

Cheers.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 10:28 AM, Basile Starynkevitch
<basile@starynkevitch.net> wrote:
> On Wed, 1 Dec 2010 10:18:58 -0800
> "H.J. Lu" <hjl.tools@gmail.com> wrote:
>
>> Here is a proposal to update LTO plugin interface.
>
> How should we parse the above sentence?
>
> Is it about an interface to plugin inside binutils to support LTO?
>
> Is it about an interface for GCC plugins to help them be more LTO friendly?
>
> Is it about an interface inside the GOLD linker to dlopen the LTO plugin provided with GCC sources?
>

It is about external linker plugin API as specified at

http://gcc.gnu.org/wiki/whopr/driver

-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> 	b. Compiler plugin controls what linker uses to generate the final executable:
> 		i. The linker command line order should be the same, with or without LTO.
> 	c. Add a cmdline bit field to
> 	struct ld_plugin_input_file
> 	{
> 	   const char *name;
> 	   int fd;
> 	   off_t offset;
> 	   off_t filesize;
> 	   void *handle;
> 	   unsigned int cmdline : 1;
> 	};

Just make it an int.  But I don't see why this is needed.  The plugin
already knows the files that it passed to add_input_file and
add_input_library.  Why does it need to linker to report back where the
file came from?  Why doesn't the plugin just keep track?

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 10:54 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>>       b. Compiler plugin controls what linker uses to generate the final executable:
>>               i. The linker command line order should be the same, with or without LTO.
>>       c. Add a cmdline bit field to
>>       struct ld_plugin_input_file
>>       {
>>          const char *name;
>>          int fd;
>>          off_t offset;
>>          off_t filesize;
>>          void *handle;
>>          unsigned int cmdline : 1;
>>       };
>
> Just make it an int.  But I don't see why this is needed.  The plugin
> already knows the files that it passed to add_input_file and
> add_input_library.  Why does it need to linker to report back where the
> file came from?  Why doesn't the plugin just keep track?
>

It is used to keep the same linker command line order. With LTO,
linker should use

crtX.o *trans*.o -lbar -lgcc -lc ... crtX.o

instead of

crtX.o -lbar -lgcc -lc ... crtX.o  *trans*.o

to generate final executable.  2 orders may generate different
executables.

-- 
H.J.

Re: Update LTO plugin interface

[Jan Hubicka]

> On Wed, Dec 1, 2010 at 10:54 AM, Ian Lance Taylor <iant@google.com> wrote:
> > "H.J. Lu" <hjl.tools@gmail.com> writes:
> >
> >>       b. Compiler plugin controls what linker uses to generate the final executable:
> >>               i. The linker command line order should be the same, with or without LTO.
> >>       c. Add a cmdline bit field to
> >>       struct ld_plugin_input_file
> >>       {
> >>          const char *name;
> >>          int fd;
> >>          off_t offset;
> >>          off_t filesize;
> >>          void *handle;
> >>          unsigned int cmdline : 1;
> >>       };
> >
> > Just make it an int.  But I don't see why this is needed.  The plugin
> > already knows the files that it passed to add_input_file and
> > add_input_library.  Why does it need to linker to report back where the
> > file came from?  Why doesn't the plugin just keep track?
> >
> 
> It is used to keep the same linker command line order. With LTO,
> linker should use
> 
> crtX.o *trans*.o -lbar -lgcc -lc ... crtX.o
> 
> instead of
> 
> crtX.o -lbar -lgcc -lc ... crtX.o  *trans*.o
> 
> to generate final executable.  2 orders may generate different
> executables.

Hmm and when I have something like

ctrX.o non-lto1.o lto1.o non-lto2.o lto2.o .... crtX.o
and then linker plugin produce ltrans0.o combining both lto1.o and lto2.o, ho
we will deal with non-lto2.o?

If we get into extending linker plugin interface, it would be great if we would
do somehting about COMDAT.  We now have RESOLVED and RESOLVED_IRONLY, while the
problem is that all non-hidden COMDAT symbols get RESOLVED that pretty much
fixes them in the output library.

I would propose adding RESOLVED_IRDYNAMIC for cases where symbol was resolved
IRONLY except that it is externally visible to dynamic linker.  We can then allow
compiler to optimize this symbol out (same way as IRONLY) if it knows it may or
may not be exported - i.e. from COMDAT flag or via -fwhole-program.

Honza
> 
> -- 
> H.J.

Re: Update LTO plugin interface

[H.J. Lu]

2010/12/1 Jan Hubicka <hubicka@ucw.cz>:
>> On Wed, Dec 1, 2010 at 10:54 AM, Ian Lance Taylor <iant@google.com> wrote:
>> > "H.J. Lu" <hjl.tools@gmail.com> writes:
>> >
>> >>       b. Compiler plugin controls what linker uses to generate the final executable:
>> >>               i. The linker command line order should be the same, with or without LTO.
>> >>       c. Add a cmdline bit field to
>> >>       struct ld_plugin_input_file
>> >>       {
>> >>          const char *name;
>> >>          int fd;
>> >>          off_t offset;
>> >>          off_t filesize;
>> >>          void *handle;
>> >>          unsigned int cmdline : 1;
>> >>       };
>> >
>> > Just make it an int.  But I don't see why this is needed.  The plugin
>> > already knows the files that it passed to add_input_file and
>> > add_input_library.  Why does it need to linker to report back where the
>> > file came from?  Why doesn't the plugin just keep track?
>> >
>>
>> It is used to keep the same linker command line order. With LTO,
>> linker should use
>>
>> crtX.o *trans*.o -lbar -lgcc -lc ... crtX.o
>>
>> instead of
>>
>> crtX.o -lbar -lgcc -lc ... crtX.o  *trans*.o
>>
>> to generate final executable.  2 orders may generate different
>> executables.
>
> Hmm and when I have something like
>
> ctrX.o non-lto1.o lto1.o non-lto2.o lto2.o .... crtX.o
> and then linker plugin produce ltrans0.o combining both lto1.o and lto2.o, ho
> we will deal with non-lto2.o?
>

My current implementation groups all LTO files together and linker will see

ctrX.o non-lto1.o ltrans0.o non-lto2.o .... crtX.o

-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Wed, Dec 1, 2010 at 10:54 AM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>>       b. Compiler plugin controls what linker uses to generate the final executable:
>>>               i. The linker command line order should be the same, with or without LTO.
>>>       c. Add a cmdline bit field to
>>>       struct ld_plugin_input_file
>>>       {
>>>          const char *name;
>>>          int fd;
>>>          off_t offset;
>>>          off_t filesize;
>>>          void *handle;
>>>          unsigned int cmdline : 1;
>>>       };
>>
>> Just make it an int.  But I don't see why this is needed.  The plugin
>> already knows the files that it passed to add_input_file and
>> add_input_library.  Why does it need to linker to report back where the
>> file came from?  Why doesn't the plugin just keep track?
>>
>
> It is used to keep the same linker command line order. With LTO,
> linker should use
>
> crtX.o *trans*.o -lbar -lgcc -lc ... crtX.o
>
> instead of
>
> crtX.o -lbar -lgcc -lc ... crtX.o  *trans*.o
>
> to generate final executable.  2 orders may generate different
> executables.

I'm sorry, I'm missing something.  What does adding that bit have to do
with keeping the same linker command line order?

Is your concern that when the plugin adds a new input file to the link,
that new input file does not cause additional objects to be pulled out
of archives later in the link?  At least in gold, what matters for that
is when the plugin calls the add_input_file or add_input_library
callback.  In gold it would be fairly difficult to have that work any
other way.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 11:12 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 10:54 AM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>>       b. Compiler plugin controls what linker uses to generate the final executable:
>>>>               i. The linker command line order should be the same, with or without LTO.
>>>>       c. Add a cmdline bit field to
>>>>       struct ld_plugin_input_file
>>>>       {
>>>>          const char *name;
>>>>          int fd;
>>>>          off_t offset;
>>>>          off_t filesize;
>>>>          void *handle;
>>>>          unsigned int cmdline : 1;
>>>>       };
>>>
>>> Just make it an int.  But I don't see why this is needed.  The plugin
>>> already knows the files that it passed to add_input_file and
>>> add_input_library.  Why does it need to linker to report back where the
>>> file came from?  Why doesn't the plugin just keep track?
>>>
>>
>> It is used to keep the same linker command line order. With LTO,
>> linker should use
>>
>> crtX.o *trans*.o -lbar -lgcc -lc ... crtX.o
>>
>> instead of
>>
>> crtX.o -lbar -lgcc -lc ... crtX.o  *trans*.o
>>
>> to generate final executable.  2 orders may generate different
>> executables.
>
> I'm sorry, I'm missing something.  What does adding that bit have to do
> with keeping the same linker command line order?

We don't want to put all unclaimed files passed to plugin back to linker.
On Linux,

[hjl@gnu-6 gcc-lto]$ cat /usr/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED (
/lib/ld-linux.so.2 ) )
[hjl@gnu-6 gcc-lto]$

Linker should use /usr/lib/libc.so, not /lib/libc.so.6,
/usr/lib/libc_nonshared.a,
/lib/ld-linux.so.2,  for final linker.  With the new cmdline field,
plugin can only pass
those unclaimed files from linker command line back to linker for the
final link.

> Is your concern that when the plugin adds a new input file to the link,
> that new input file does not cause additional objects to be pulled out
> of archives later in the link?  At least in gold, what matters for that
> is when the plugin calls the add_input_file or add_input_library
> callback.  In gold it would be fairly difficult to have that work any
> other way.
>

Please try the testcase in

http://sourceware.org/bugzilla/show_bug.cgi?id=12248#c5

with gold.

-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> We don't want to put all unclaimed files passed to plugin back to linker.
> On Linux,
>
> [hjl@gnu-6 gcc-lto]$ cat /usr/lib/libc.so
> /* GNU ld script
>    Use the shared library, but some functions are only in
>    the static library, so try that secondarily.  */
> OUTPUT_FORMAT(elf32-i386)
> GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED (
> /lib/ld-linux.so.2 ) )
> [hjl@gnu-6 gcc-lto]$
>
> Linker should use /usr/lib/libc.so, not /lib/libc.so.6,
> /usr/lib/libc_nonshared.a,
> /lib/ld-linux.so.2,  for final linker.  With the new cmdline field,
> plugin can only pass
> those unclaimed files from linker command line back to linker for the
> final link.

Thanks, at least now I understand what the new field means: it is true
for a file explicitly named on the command line, false for a file named
in a linker script.

Are you planning to have the plugin claim all files, even linker
scripts, and then pass only the command line files back to the linker?

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> We don't want to put all unclaimed files passed to plugin back to linker.
>> On Linux,
>>
>> [hjl@gnu-6 gcc-lto]$ cat /usr/lib/libc.so
>> /* GNU ld script
>>    Use the shared library, but some functions are only in
>>    the static library, so try that secondarily.  */
>> OUTPUT_FORMAT(elf32-i386)
>> GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED (
>> /lib/ld-linux.so.2 ) )
>> [hjl@gnu-6 gcc-lto]$
>>
>> Linker should use /usr/lib/libc.so, not /lib/libc.so.6,
>> /usr/lib/libc_nonshared.a,
>> /lib/ld-linux.so.2,  for final linker.  With the new cmdline field,
>> plugin can only pass
>> those unclaimed files from linker command line back to linker for the
>> final link.
>
> Thanks, at least now I understand what the new field means: it is true
> for a file explicitly named on the command line, false for a file named
> in a linker script.
>
> Are you planning to have the plugin claim all files, even linker
> scripts, and then pass only the command line files back to the linker?
>

Plugin will keep the same claim strategy.  For those aren't claimed by
plugin, plugin will save and pass them back to linker only if they are
specified at command line.


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
>
>> Are you planning to have the plugin claim all files, even linker
>> scripts, and then pass only the command line files back to the linker?
>>
>
> Plugin will keep the same claim strategy.  For those aren't claimed by
> plugin, plugin will save and pass them back to linker only if they are
> specified at command line.

Just to be clear, that does not make sense as written.  If the plugin
does not claim a file, it should not then pass it back to the linker.

In fact, if the plugin claims all files, then as far as I can see your
new ld_plugin_input_file field is not required.  And if the plugin does
not claim all files, I don't see how this can work.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 12:55 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
>>
>>> Are you planning to have the plugin claim all files, even linker
>>> scripts, and then pass only the command line files back to the linker?
>>>
>>
>> Plugin will keep the same claim strategy.  For those aren't claimed by
>> plugin, plugin will save and pass them back to linker only if they are
>> specified at command line.
>
> Just to be clear, that does not make sense as written.  If the plugin
> does not claim a file, it should not then pass it back to the linker.

API has

typedef
enum ld_plugin_status
(*ld_plugin_claim_file_handler) (
  const struct ld_plugin_input_file *file, int *claimed);

For linker script, archive, DSO and object file without IR,
*claimed will return 0 and plugin will save and pass it back to
linker later in  if it is specified at command line.

> In fact, if the plugin claims all files, then as far as I can see your
> new ld_plugin_input_file field is not required.  And if the plugin does
> not claim all files, I don't see how this can work.

Stage 2 linker should:

1. Discard all previous inputs.
2. Generate the final executable with inputs from plugin, which include
linker script, archive, DSO and object file without IR specified at
command line as well as trans files from LTO.

My implementation is available on hjl/lto branch at

http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
http://git.kernel.org/?p=devel/gcc/hjl/x86.git;a=summary


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Wed, Dec 1, 2010 at 12:55 PM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
>>>
>>>> Are you planning to have the plugin claim all files, even linker
>>>> scripts, and then pass only the command line files back to the linker?
>>>>
>>>
>>> Plugin will keep the same claim strategy.  For those aren't claimed by
>>> plugin, plugin will save and pass them back to linker only if they are
>>> specified at command line.
>>
>> Just to be clear, that does not make sense as written.  If the plugin
>> does not claim a file, it should not then pass it back to the linker.
>
> API has
>
> typedef
> enum ld_plugin_status
> (*ld_plugin_claim_file_handler) (
>   const struct ld_plugin_input_file *file, int *claimed);
>
> For linker script, archive, DSO and object file without IR,
> *claimed will return 0 and plugin will save and pass it back to
> linker later in  if it is specified at command line.

I don't understand what you wrote, so I am going to write what I think
happens.

The claim_file handler is an interface provided by the plugin itself.
The plugin will register it via LDPT_REGISTER_CLAIM_FILE_HOOK.  The
linker proper will call it for each input file.

In the case of the LTO plugin, this is the static function
claim_file_handler in lto-plugin.c.

If the plugin registers a claim_file handler, and, when the linker calls
it, it returns with *claimed == 0, then the linker will process the file
as it normally does.  Since the file will already have been processed,
it does not make sense for the plugin to then pass it back to the
linker.  The effect would be similar to listing the file twice on the
command line.


>> In fact, if the plugin claims all files, then as far as I can see your
>> new ld_plugin_input_file field is not required.  And if the plugin does
>> not claim all files, I don't see how this can work.
>
> Stage 2 linker should:
>
> 1. Discard all previous inputs.

How is this step done?


> My implementation is available on hjl/lto branch at

Thanks, but I don't see any changes to gold there, so I don't see what
you have done to change the plugin interface.

Ian

Re: Update LTO plugin interface

[Richard Guenther]

On Wed, Dec 1, 2010 at 10:28 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 12:55 PM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
>>>>
>>>>> Are you planning to have the plugin claim all files, even linker
>>>>> scripts, and then pass only the command line files back to the linker?
>>>>>
>>>>
>>>> Plugin will keep the same claim strategy.  For those aren't claimed by
>>>> plugin, plugin will save and pass them back to linker only if they are
>>>> specified at command line.
>>>
>>> Just to be clear, that does not make sense as written.  If the plugin
>>> does not claim a file, it should not then pass it back to the linker.
>>
>> API has
>>
>> typedef
>> enum ld_plugin_status
>> (*ld_plugin_claim_file_handler) (
>>   const struct ld_plugin_input_file *file, int *claimed);
>>
>> For linker script, archive, DSO and object file without IR,
>> *claimed will return 0 and plugin will save and pass it back to
>> linker later in  if it is specified at command line.
>
> I don't understand what you wrote, so I am going to write what I think
> happens.
>
> The claim_file handler is an interface provided by the plugin itself.
> The plugin will register it via LDPT_REGISTER_CLAIM_FILE_HOOK.  The
> linker proper will call it for each input file.
>
> In the case of the LTO plugin, this is the static function
> claim_file_handler in lto-plugin.c.
>
> If the plugin registers a claim_file handler, and, when the linker calls
> it, it returns with *claimed == 0, then the linker will process the file
> as it normally does.  Since the file will already have been processed,
> it does not make sense for the plugin to then pass it back to the
> linker.  The effect would be similar to listing the file twice on the
> command line.

The basic problem is that if lto-plugin claims a file and provides a symtab
to the linker the link-time optimization might change that, including
adding new undefined symbols (think of libcalls).  The linker needs
to re-process even not-claimed static archives (such as libgcc) to
resolve those new undefs.  We hack around this by adding another
-lgcc at the end of the command-line, but that does change linker
resolution as the link order does matter.

Basically we need to trigger a complete re-link with the claimed
object files substituted for the link-time optimized ones.

Richard.

>
>>> In fact, if the plugin claims all files, then as far as I can see your
>>> new ld_plugin_input_file field is not required.  And if the plugin does
>>> not claim all files, I don't see how this can work.
>>
>> Stage 2 linker should:
>>
>> 1. Discard all previous inputs.
>
> How is this step done?
>
>
>> My implementation is available on hjl/lto branch at
>
> Thanks, but I don't see any changes to gold there, so I don't see what
> you have done to change the plugin interface.
>
> Ian
>

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 1:28 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 12:55 PM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
>>>>
>>>>> Are you planning to have the plugin claim all files, even linker
>>>>> scripts, and then pass only the command line files back to the linker?
>>>>>
>>>>
>>>> Plugin will keep the same claim strategy.  For those aren't claimed by
>>>> plugin, plugin will save and pass them back to linker only if they are
>>>> specified at command line.
>>>
>>> Just to be clear, that does not make sense as written.  If the plugin
>>> does not claim a file, it should not then pass it back to the linker.
>>
>> API has
>>
>> typedef
>> enum ld_plugin_status
>> (*ld_plugin_claim_file_handler) (
>>   const struct ld_plugin_input_file *file, int *claimed);
>>
>> For linker script, archive, DSO and object file without IR,
>> *claimed will return 0 and plugin will save and pass it back to
>> linker later in  if it is specified at command line.
>
> I don't understand what you wrote, so I am going to write what I think
> happens.
>
> The claim_file handler is an interface provided by the plugin itself.
> The plugin will register it via LDPT_REGISTER_CLAIM_FILE_HOOK.  The
> linker proper will call it for each input file.
>
> In the case of the LTO plugin, this is the static function
> claim_file_handler in lto-plugin.c.
>
> If the plugin registers a claim_file handler, and, when the linker calls
> it, it returns with *claimed == 0, then the linker will process the file
> as it normally does.  Since the file will already have been processed,
> it does not make sense for the plugin to then pass it back to the
> linker.  The effect would be similar to listing the file twice on the
> command line.

That is what "Discard all previous inputs" in stage 2 linking is for.

>
>>> In fact, if the plugin claims all files, then as far as I can see your
>>> new ld_plugin_input_file field is not required.  And if the plugin does
>>> not claim all files, I don't see how this can work.
>>
>> Stage 2 linker should:
>>
>> 1. Discard all previous inputs.
>
> How is this step done?

For GNU linker, I mark all sections in a bfd file, which
will be sent back from plugin, with SEC_EXCLUDE. I also
free and recreate the output hash table.

>
>> My implementation is available on hjl/lto branch at
>
> Thanks, but I don't see any changes to gold there, so I don't see what
> you have done to change the plugin interface.
>

My changes should be visible now.


-- 
H.J.

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 1:33 PM, Richard Guenther
<richard.guenther@gmail.com> wrote:
> On Wed, Dec 1, 2010 at 10:28 PM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> On Wed, Dec 1, 2010 at 12:55 PM, Ian Lance Taylor <iant@google.com> wrote:
>>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>>
>>>>> On Wed, Dec 1, 2010 at 12:37 PM, Ian Lance Taylor <iant@google.com> wrote:
>>>>>
>>>>>> Are you planning to have the plugin claim all files, even linker
>>>>>> scripts, and then pass only the command line files back to the linker?
>>>>>>
>>>>>
>>>>> Plugin will keep the same claim strategy.  For those aren't claimed by
>>>>> plugin, plugin will save and pass them back to linker only if they are
>>>>> specified at command line.
>>>>
>>>> Just to be clear, that does not make sense as written.  If the plugin
>>>> does not claim a file, it should not then pass it back to the linker.
>>>
>>> API has
>>>
>>> typedef
>>> enum ld_plugin_status
>>> (*ld_plugin_claim_file_handler) (
>>>   const struct ld_plugin_input_file *file, int *claimed);
>>>
>>> For linker script, archive, DSO and object file without IR,
>>> *claimed will return 0 and plugin will save and pass it back to
>>> linker later in  if it is specified at command line.
>>
>> I don't understand what you wrote, so I am going to write what I think
>> happens.
>>
>> The claim_file handler is an interface provided by the plugin itself.
>> The plugin will register it via LDPT_REGISTER_CLAIM_FILE_HOOK.  The
>> linker proper will call it for each input file.
>>
>> In the case of the LTO plugin, this is the static function
>> claim_file_handler in lto-plugin.c.
>>
>> If the plugin registers a claim_file handler, and, when the linker calls
>> it, it returns with *claimed == 0, then the linker will process the file
>> as it normally does.  Since the file will already have been processed,
>> it does not make sense for the plugin to then pass it back to the
>> linker.  The effect would be similar to listing the file twice on the
>> command line.
>
> The basic problem is that if lto-plugin claims a file and provides a symtab
> to the linker the link-time optimization might change that, including
> adding new undefined symbols (think of libcalls).  The linker needs
> to re-process even not-claimed static archives (such as libgcc) to
> resolve those new undefs.  We hack around this by adding another
> -lgcc at the end of the command-line, but that does change linker
> resolution as the link order does matter.
>
> Basically we need to trigger a complete re-link with the claimed
> object files substituted for the link-time optimized ones.
>

That is what my implementation does.


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Wed, Dec 1, 2010 at 1:28 PM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>> For linker script, archive, DSO and object file without IR,
>>> *claimed will return 0 and plugin will save and pass it back to
>>> linker later in  if it is specified at command line.
>>
>> I don't understand what you wrote, so I am going to write what I think
>> happens.
>>
>> The claim_file handler is an interface provided by the plugin itself.
>> The plugin will register it via LDPT_REGISTER_CLAIM_FILE_HOOK.  The
>> linker proper will call it for each input file.
>>
>> In the case of the LTO plugin, this is the static function
>> claim_file_handler in lto-plugin.c.
>>
>> If the plugin registers a claim_file handler, and, when the linker calls
>> it, it returns with *claimed == 0, then the linker will process the file
>> as it normally does.  Since the file will already have been processed,
>> it does not make sense for the plugin to then pass it back to the
>> linker.  The effect would be similar to listing the file twice on the
>> command line.
>
> That is what "Discard all previous inputs" in stage 2 linking is for.

But what does that mean?  Are you saying that the linker interface to
the plugin should change to work that way?  If we do that, then we
should change other aspects of the plugin interface as well.  It could
probably become quite a bit simpler.


The only reason we would ever need to do a complete relink is if the LTO
plugin can introduce arbitrary new symbol references.  Is that ever
possible?  If it is, we need to rethink the whole approach.  If the LTO
plugin can introduce arbitrary new symbol references, that means that
LTO plugin can cause arbitrary objects to be pulled in from archives.
And that means that if we only run the plugin once, we are losing
possible optimizations, because the plugin will never those new objects.


My suspicion is that the LTO plugin can only introduce a small bounded
set of new symbol references, namely those which we assume can be
satisified from -lc or -lgcc.  Is that true?

Ian

Re: Update LTO plugin interface

[Cary Coutant]

> If we get into extending linker plugin interface, it would be great if we would
> do somehting about COMDAT.  We now have RESOLVED and RESOLVED_IRONLY, while the
> problem is that all non-hidden COMDAT symbols get RESOLVED that pretty much
> fixes them in the output library.
>
> I would propose adding RESOLVED_IRDYNAMIC for cases where symbol was resolved
> IRONLY except that it is externally visible to dynamic linker.  We can then allow
> compiler to optimize this symbol out (same way as IRONLY) if it knows it may or
> may not be exported - i.e. from COMDAT flag or via -fwhole-program.

(This is off the main topic...)

Actually, we have PREVAILING_DEF and PREVAILING_DEF_IRONLY, plus
RESOLVED_IR, RESOLVED_EXEC, and RESOLVED_DYN. If the symbol was
resolved elsewhere, we don't have any way to say whether it was IRONLY
or not, and that's a problem for common symbols, because there really
is no prevailing def -- the linker just allocates the space itself.
Currently, gold picks one of the common symbols and calls it the
prevailing def, but the one it picks might not actually be the largest
one. I'd prefer to add something like COMMON and COMMON_IRONLY as
possible resolutions.

I'm not sure if you're talking about that, or about real COMDAT
groups. As far as gold is concerned, it picks one COMDAT group and
throws the rest of them away, but for the one it picks, you'll get
either PREVAILING_DEF or PREVAILING_DEF_IRONLY. That should tell the
compiler what it needs to know.

I'm also not sure what you mean by "resolved IRONLY except that it is
externally visible to the dynamic linker." If we're building a shared
library, and the symbol is exported, it's not going to be IRONLY, and
I don't see how it would be valid to optimize it out. If we're
building an executable with --export-dynamic, same thing.

-cary

Re: Update LTO plugin interface

[Cary Coutant]

>> That is what "Discard all previous inputs" in stage 2 linking is for.
>
> But what does that mean?  Are you saying that the linker interface to
> the plugin should change to work that way?  If we do that, then we
> should change other aspects of the plugin interface as well.  It could
> probably become quite a bit simpler.
>
> The only reason we would ever need to do a complete relink is if the LTO
> plugin can introduce arbitrary new symbol references.  Is that ever
> possible?  If it is, we need to rethink the whole approach.  If the LTO
> plugin can introduce arbitrary new symbol references, that means that
> LTO plugin can cause arbitrary objects to be pulled in from archives.
> And that means that if we only run the plugin once, we are losing
> possible optimizations, because the plugin will never those new objects.
>
> My suspicion is that the LTO plugin can only introduce a small bounded
> set of new symbol references, namely those which we assume can be
> satisified from -lc or -lgcc.  Is that true?

Exactly. The plugin API was designed for this model -- if you want to
start the link all over again, you may as well stick with the collect2
approach and enhance it to deal with archives of IR files.

The plugin API, as implemented in gold (not sure about gnu ld), does
maintain the original order of input files as far as symbol binding is
concerned. When IR files are claimed, the plugin provides the list of
symbols defined and referenced, and the linker builds the symbol table
as if those files were linked in at that particular spot in the
command line. When the compiler provides real definitions of those
symbols later, the real definitions simply replace the "placeholders"
that were left in the linker's symbol table. The only aspect of link
order that isn't maintained is the physical order of the sections in
memory.

As Ian noted, if the compiler introduces new references that weren't
there before, the new references must be from a limited set of
libcalls that the backend can introduce, and those should all be
resolved with an extra pass through -lc or -lgcc. That's not exactly
pretty, but I don't see how it destroys the notion of link order --
the only way those new symbols could have been resolved differently is
if a user library interposed definitions for the libcall, and those
certainly can't be what the compiler intended to bind to. In PR 12248,
I think it's questionable to claim that the compiler-introduced call
to __udivdi3 should not resolve to the version in libgcc. Sure, I
understand it's useful for library developers while debugging and
testing, but an ordinary user certainly can't count on his own
definition of that routine to get called -- the compiler might
generate the division inline, or call a different specialized version.
All of these routines are outside the user's namespace, and we should
be able to optimize without regard for what the user's libraries might
contain.

An improvement could be for the claim file handler to determine what
libcalls might be introduced and add them to the list of referenced
symbols so that the linker can bring in the definitions in the
original pass through the input files -- any that end up not being
referenced can be garbage collected. Alternatively, we could do a
whole-archive link of the library that contains the libcalls, again
discarding unreferenced routines via garbage collection. Neither of
these require a change to the API.

-cary

Re: Update LTO plugin interface

[H.J. Lu]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

On Wed, Dec 1, 2010 at 3:06 PM, Cary Coutant <ccoutant@google.com> wrote:
>>> That is what "Discard all previous inputs" in stage 2 linking is for.
>>
>> But what does that mean?  Are you saying that the linker interface to
>> the plugin should change to work that way?  If we do that, then we
>> should change other aspects of the plugin interface as well.  It could
>> probably become quite a bit simpler.
>>
>> The only reason we would ever need to do a complete relink is if the LTO
>> plugin can introduce arbitrary new symbol references.  Is that ever
>> possible?  If it is, we need to rethink the whole approach.  If the LTO
>> plugin can introduce arbitrary new symbol references, that means that
>> LTO plugin can cause arbitrary objects to be pulled in from archives.
>> And that means that if we only run the plugin once, we are losing
>> possible optimizations, because the plugin will never those new objects.
>>
>> My suspicion is that the LTO plugin can only introduce a small bounded
>> set of new symbol references, namely those which we assume can be
>> satisified from -lc or -lgcc.  Is that true?
>
> Exactly. The plugin API was designed for this model -- if you want to
> start the link all over again, you may as well stick with the collect2
> approach and enhance it to deal with archives of IR files.

Some compilers duplicates the whole linker symbol resolution in their
"collect2" program to get it right.

> The plugin API, as implemented in gold (not sure about gnu ld), does
> maintain the original order of input files as far as symbol binding is
> concerned. When IR files are claimed, the plugin provides the list of
> symbols defined and referenced, and the linker builds the symbol table
> as if those files were linked in at that particular spot in the
> command line. When the compiler provides real definitions of those
> symbols later, the real definitions simply replace the "placeholders"
> that were left in the linker's symbol table. The only aspect of link
> order that isn't maintained is the physical order of the sections in
> memory.

That is exactly the problem my proposal tries to address.

> As Ian noted, if the compiler introduces new references that weren't
> there before, the new references must be from a limited set of
> libcalls that the backend can introduce, and those should all be
> resolved with an extra pass through -lc or -lgcc. That's not exactly
> pretty, but I don't see how it destroys the notion of link order --
> the only way those new symbols could have been resolved differently is
> if a user library interposed definitions for the libcall, and those
> certainly can't be what the compiler intended to bind to. In PR 12248,
> I think it's questionable to claim that the compiler-introduced call
> to __udivdi3 should not resolve to the version in libgcc. Sure, I
> understand it's useful for library developers while debugging and
> testing, but an ordinary user certainly can't count on his own
> definition of that routine to get called -- the compiler might
> generate the division inline, or call a different specialized version.
> All of these routines are outside the user's namespace, and we should
> be able to optimize without regard for what the user's libraries might
> contain.
>

__udivdi3 is just an example.  It can also happen to memcpy, or
any library calls generated by GCC. I am enclosing a testcase for memcpy.


-- 
H.J.

[-- Attachment #2: bug-2.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 851 bytes --]

Re: Update LTO plugin interface

[Cary Coutant]

>> The only aspect of link
>> order that isn't maintained is the physical order of the sections in
>> memory.
>
> That is exactly the problem my proposal tries to address.

Really? That's not at all what PR 12248 is about. The physical order
of the sections (meaning the order of contributions within each output
section) -- in the absence of any linker scripts -- should be
irrelevant. With linker scripts, or any other form of layout control,
the link order is decoupled from the layout anyway.

> __udivdi3 is just an example.  It can also happen to memcpy, or
> any library calls generated by GCC. I am enclosing a testcase for memcpy.

Regardless, if the compiler backend introduces a call to a runtime
support routine, it's expecting to bind to a specific routine in its
runtime support library. Anything else is unsupported. For gcc or
libgcc hackers, if you *really* need the interposed routine, it's
simple enough to link the .o instead of the .a, or use
--whole-archive.

Think about it -- any failure to bind to an interposed copy of memcpy
(or any other library call generated by gcc) is indistinguishable from
the compiler choosing to generate the code inline.

-cary

Re: Update LTO plugin interface

[Jan Hubicka]

> I'm also not sure what you mean by "resolved IRONLY except that it is
> externally visible to the dynamic linker." If we're building a shared
> library, and the symbol is exported, it's not going to be IRONLY, and
> I don't see how it would be valid to optimize it out. If we're

Well, the typical COMDAT symbols (ignoring side cases) needs to be put into
binary/library only if they are actually used, as all the other DSOs will define
them too if they are used there.
So it is valid to optimize out COMDAT after you optimized out all its uses. This
commonly happens at linktime.

Honza

> building an executable with --export-dynamic, same thing.
> 
> -cary

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> __udivdi3 is just an example.  It can also happen to memcpy, or
> any library calls generated by GCC. I am enclosing a testcase for memcpy.

I believe we can solve that specific problem much more efficiently than
requiring a complete link of all the input files.  We currently solve it
using the -pass-through option which is passed to the linker plugin.
Are there any cases for which using -pass-through=-lc
-pass-through=-lgcc would not be a complete solution?

Ian

Re: Update LTO plugin interface

[Cary Coutant]

>> I'm also not sure what you mean by "resolved IRONLY except that it is
>> externally visible to the dynamic linker." If we're building a shared
>> library, and the symbol is exported, it's not going to be IRONLY, and
>> I don't see how it would be valid to optimize it out. If we're
>
> Well, the typical COMDAT symbols (ignoring side cases) needs to be put into
> binary/library only if they are actually used, as all the other DSOs will define
> them too if they are used there.
> So it is valid to optimize out COMDAT after you optimized out all its uses. This
> commonly happens at linktime.

Ahh, OK. I was worried about those side cases where sometimes a pure
reference is emitted. From a linker point of view, that's something
that theoretically could happen, although it may be the case that we
don't actually have to support it. If we had a resolution like
PREVAILING_DEF_IRONLY_BUT_EXPORTED (preferably something shorter than
that), I think that would give the compiler the information it needs.
Is that pretty much what your RESOLVED_IRDYNAMIC was intended to mean?

Another thing that I don't remember offhand whether I got right or not
in gold is that if a COMDAT group is defined in IR and non-IR files,
we want to choose one of the IR files as the instance to keep. I'll
have to check.

-cary

Re: Update LTO plugin interface

[Jan Hubicka]

> >> I'm also not sure what you mean by "resolved IRONLY except that it is
> >> externally visible to the dynamic linker." If we're building a shared
> >> library, and the symbol is exported, it's not going to be IRONLY, and
> >> I don't see how it would be valid to optimize it out. If we're
> >
> > Well, the typical COMDAT symbols (ignoring side cases) needs to be put into
> > binary/library only if they are actually used, as all the other DSOs will define
> > them too if they are used there.
> > So it is valid to optimize out COMDAT after you optimized out all its uses. This
> > commonly happens at linktime.
> 
> Ahh, OK. I was worried about those side cases where sometimes a pure
> reference is emitted. From a linker point of view, that's something
> that theoretically could happen, although it may be the case that we
> don't actually have to support it. If we had a resolution like
> PREVAILING_DEF_IRONLY_BUT_EXPORTED (preferably something shorter than
> that), I think that would give the compiler the information it needs.
> Is that pretty much what your RESOLVED_IRDYNAMIC was intended to mean?

Ah, yes.  My first attempt for name was same as yours ;)
GCC knows what COMDATs have to be output even if unused.

> 
> Another thing that I don't remember offhand whether I got right or not
> in gold is that if a COMDAT group is defined in IR and non-IR files,
> we want to choose one of the IR files as the instance to keep. I'll
> have to check.

At GCC side I am trying to keep those grouped. So as soon as you ask for
one symbol from the group, you will get all of them, even if you possibly
decide to resolve others from other group (that indeed should not happen).

Also from QOI point of view, it is better if linker was choosing variant
with IR definitions over variant without if given multiple variants.

Honza
> 
> -cary

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 4:48 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> __udivdi3 is just an example.  It can also happen to memcpy, or
>> any library calls generated by GCC. I am enclosing a testcase for memcpy.
>
> I believe we can solve that specific problem much more efficiently than
> requiring a complete link of all the input files.  We currently solve it
> using the -pass-through option which is passed to the linker plugin.
> Are there any cases for which using -pass-through=-lc
> -pass-through=-lgcc would not be a complete solution?
>

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46760


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Wed, Dec 1, 2010 at 4:48 PM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> __udivdi3 is just an example.  It can also happen to memcpy, or
>>> any library calls generated by GCC. I am enclosing a testcase for memcpy.
>>
>> I believe we can solve that specific problem much more efficiently than
>> requiring a complete link of all the input files.  We currently solve it
>> using the -pass-through option which is passed to the linker plugin.
>> Are there any cases for which using -pass-through=-lc
>> -pass-through=-lgcc would not be a complete solution?
>>
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46760

Sigh, OK, so I'll just add -pass-through=-lgcov when appropriate.

Or I'll rephrase: are there any cases for which using -pass-through for
the set of libraries that the gcc driver automatically adds to the end
of the link line would not be a complete solution?

I'm not saying -pass-through is the solution we should be using.  It's
clearly a bit of a hack.  However, I am asking the question seriously,
because if we have to do a complete relink, then let's do a complete
relink, but if we don't have to do one, let's definitely not.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Wed, Dec 1, 2010 at 5:53 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 4:48 PM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> __udivdi3 is just an example.  It can also happen to memcpy, or
>>>> any library calls generated by GCC. I am enclosing a testcase for memcpy.
>>>
>>> I believe we can solve that specific problem much more efficiently than
>>> requiring a complete link of all the input files.  We currently solve it
>>> using the -pass-through option which is passed to the linker plugin.
>>> Are there any cases for which using -pass-through=-lc
>>> -pass-through=-lgcc would not be a complete solution?
>>>
>>
>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46760
>
> Sigh, OK, so I'll just add -pass-through=-lgcov when appropriate.
>
> Or I'll rephrase: are there any cases for which using -pass-through for
> the set of libraries that the gcc driver automatically adds to the end
> of the link line would not be a complete solution?
>
> I'm not saying -pass-through is the solution we should be using.  It's
> clearly a bit of a hack.  However, I am asking the question seriously,
> because if we have to do a complete relink, then let's do a complete
> relink, but if we don't have to do one, let's definitely not.

Nothing can't be solved with a hack :-).


-- 
H.J.

Re: Update LTO plugin interface

[Dave Korn]

[-- Attachment #1: Type: text/plain, Size: 2979 bytes --]

On 01/12/2010 23:06, Cary Coutant wrote:

>> My suspicion is that the LTO plugin can only introduce a small bounded
>> set of new symbol references, namely those which we assume can be
>> satisified from -lc or -lgcc.  Is that true?
> 
> Exactly. 

  Potentially also gcov, ssp, mudflap?

> The plugin API, as implemented in gold (not sure about gnu ld), does
> maintain the original order of input files as far as symbol binding is
> concerned. When IR files are claimed, the plugin provides the list of
> symbols defined and referenced, and the linker builds the symbol table
> as if those files were linked in at that particular spot in the
> command line. When the compiler provides real definitions of those
> symbols later, the real definitions simply replace the "placeholders"
> that were left in the linker's symbol table. 

  We just ran into a new problem with that:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375#c12

  (Brief summary: no symbol type info is conveyed in the LTO symtab, and in
the ELF linker, _bfd_elf_merge_symbol errors out when the real symbol is TLS
and the LTO symtab one didn't have the same type, and it does so too early for
the plugin api's multiple_definition linker callback hook to do anything about
it.)

> As Ian noted, if the compiler introduces new references that weren't
> there before, the new references must be from a limited set of
> libcalls that the backend can introduce, and those should all be
> resolved with an extra pass through -lc or -lgcc. That's not exactly
> pretty, but I don't see how it destroys the notion of link order --
> the only way those new symbols could have been resolved differently is
> if a user library interposed definitions for the libcall, and those
> certainly can't be what the compiler intended to bind to. In PR 12248,
> I think it's questionable to claim that the compiler-introduced call
> to __udivdi3 should not resolve to the version in libgcc. Sure, I
> understand it's useful for library developers while debugging and
> testing, but an ordinary user certainly can't count on his own
> definition of that routine to get called -- the compiler might
> generate the division inline, or call a different specialized version.
> All of these routines are outside the user's namespace, and we should
> be able to optimize without regard for what the user's libraries might
> contain.

  I tend to follow this theory.  I think that the current approach should be
sufficient, but I think we probably need to arrange a few more pass-throughs:
for (some/all of?) the libraries I mentioned above, and HJ pointed out another
relevant issue: we should pass-through the crt endfiles as well, because the
new object files introduced by LTO might pull in new dynamic references from
libc et. al., and apparently you mustn't do that after crtn.o has been linked.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42690#c27

  I was going to test something like the attached.


    cheers,
      DaveK


[-- Attachment #2: pr42690-part2.diff --]
[-- Type: text/x-c, Size: 3888 bytes --]

Index: gcc/gcc.c
===================================================================
--- gcc/gcc.c	(revision 167334)
+++ gcc/gcc.c	(working copy)
@@ -264,7 +264,9 @@ static const char *print_asm_header_spec_function
 static const char *compare_debug_dump_opt_spec_function (int, const char **);
 static const char *compare_debug_self_opt_spec_function (int, const char **);
 static const char *compare_debug_auxbase_opt_spec_function (int, const char **);
+static const char *gen_pass_through_spec (int, const char **, bool, bool);
 static const char *pass_through_libs_spec_func (int, const char **);
+static const char *pass_through_objs_spec_func (int, const char **);
 \f
 /* The Specs Language
 
@@ -637,7 +639,9 @@ proper position among the other output files.  */
     -plugin %(linker_plugin_file) \
     -plugin-opt=%(lto_wrapper) \
     -plugin-opt=-fresolution=%u.res \
+    %{fprofile-arcs|fprofile-generate*|coverage:-plugin-opt=-pass-through=-lgcov} \
     %{!nostdlib:%{!nodefaultlibs:%:pass-through-libs(%(link_gcc_c_sequence))}} \
+    %{!A:%{!nostdlib:%{!nostartfiles:%:pass-through-objs(%E)}}}
     } \
     %{flto*:%<fcompare-debug*} \
     %{flto*} %l " LINK_PIE_SPEC \
@@ -648,7 +652,7 @@ proper position among the other output files.  */
     %(mflib) " STACK_SPLIT_SPEC "\
     %{fprofile-arcs|fprofile-generate*|coverage:-lgcov}\
     %{!nostdlib:%{!nodefaultlibs:%(link_ssp) %(link_gcc_c_sequence)}}\
-    %{!A:%{!nostdlib:%{!nostartfiles:%E}}} %{T*} }}}}}}"
+    %{!fuse-linker-plugin:%{!A:%{!nostdlib:%{!nostartfiles:%E}}}} %{T*} }}}}}}"
 #endif
 
 #ifndef LINK_LIBGCC_SPEC
@@ -1221,6 +1225,7 @@ static const struct spec_function static_spec_func
   { "compare-debug-self-opt",	compare_debug_self_opt_spec_function },
   { "compare-debug-auxbase-opt", compare_debug_auxbase_opt_spec_function },
   { "pass-through-libs",	pass_through_libs_spec_func },
+  { "pass-through-objs",	pass_through_objs_spec_func },
 #ifdef EXTRA_SPEC_FUNCTIONS
   EXTRA_SPEC_FUNCTIONS
 #endif
@@ -8171,8 +8176,8 @@ compare_debug_auxbase_opt_spec_function (int arg,
    prepended with the plugin option to cause them to be passed through
    to the final link after all the new object files have been added.  */
 
-const char *
-pass_through_libs_spec_func (int argc, const char **argv)
+static const char *
+gen_pass_through_spec (int argc, const char **argv, bool do_libs, bool do_objs)
 {
   char *prepended = xstrdup (" ");
   int n;
@@ -8185,7 +8190,7 @@ compare_debug_auxbase_opt_spec_function (int arg,
       /* Anything that isn't an option is a full path to an output
          file; pass it through if it ends in '.a'.  Among options,
 	 pass only -l.  */
-      if (argv[n][0] == '-' && argv[n][1] == 'l')
+      if (do_libs && argv[n][0] == '-' && argv[n][1] == 'l')
 	{
 	  const char *lopt = argv[n] + 2;
 	  /* Handle both joined and non-joined -l options.  If for any
@@ -8198,13 +8203,32 @@ compare_debug_auxbase_opt_spec_function (int arg,
 	  prepended = concat (prepended, "-plugin-opt=-pass-through=-l",
 		lopt, " ", NULL);
 	}
-      else if (!strcmp (".a", argv[n] + strlen (argv[n]) - 2))
+      else if (do_libs && !strcmp (".a", argv[n] + strlen (argv[n]) - 2))
 	{
 	  prepended = concat (prepended, "-plugin-opt=-pass-through=",
 		argv[n], " ", NULL);
 	}
+      else if (do_objs && !strcmp (TARGET_OBJECT_SUFFIX,
+				  argv[n] + strlen (argv[n])
+				  - strlen (TARGET_OBJECT_SUFFIX)))
+	{
+	  prepended = concat (prepended, "-plugin-opt=-pass-through=",
+		argv[n], " ", NULL);
+	}
       if (prepended != old)
 	free (old);
     }
   return prepended;
 }
+
+static const char *
+pass_through_libs_spec_func (int argc, const char **argv)
+{
+  return gen_pass_through_spec (argc, argv, true, false);
+}
+
+static const char *
+pass_through_objs_spec_func (int argc, const char **argv)
+{
+  return gen_pass_through_spec (argc, argv, false, true);
+}

Re: Update LTO plugin interface

[Dave Korn]

On 02/12/2010 04:28, Dave Korn wrote:

>   I was going to test something like the attached.

  Oops, there was a typo in that:

> @@ -637,7 +639,9 @@ proper position among the other output files.  */
>      -plugin %(linker_plugin_file) \
>      -plugin-opt=%(lto_wrapper) \
>      -plugin-opt=-fresolution=%u.res \
> +    %{fprofile-arcs|fprofile-generate*|coverage:-plugin-opt=-pass-through=-lgcov} \
>      %{!nostdlib:%{!nodefaultlibs:%:pass-through-libs(%(link_gcc_c_sequence))}} \
> +    %{!A:%{!nostdlib:%{!nostartfiles:%:pass-through-objs(%E)}}}
>      } \
>      %{flto*:%<fcompare-debug*} \
>      %{flto*} %l " LINK_PIE_SPEC \


  The second added line in that hunk is missing a continuation char.  JFTR, I
won't post a respin of the patch until it's been bootstrapped and tested a bit.

    cheers,
      DaveK

Re: Update LTO plugin interface

[Dave Korn]

On 02/12/2010 00:12, Cary Coutant wrote:

> Think about it -- any failure to bind to an interposed copy of memcpy
> (or any other library call generated by gcc) is indistinguishable from
> the compiler choosing to generate the code inline.

  Indeed, replacing library functions is a tricky business in the presence of
optimisations:

> $ cat main.c
> #include <stdio.h>
> 
> int main (int argc, const char **argv)
> {
>   printf ("hello world\n");
>   return 0;
> }
> 
> $ cat myprintf.c
> #include <stdlib.h>
> 
> int printf (const char *fmt, ...)
> {
>   abort ();
> }
> 
> $ gcc -O3 main.c myprintf.c -o test1
> 
> $ ./test1.exe
> hello world
> 
> $ cat main2.c
> #include <stdio.h>
> 
> int main (int argc, const char **argv)
> {
>   printf ("<%s>", "hello world\n");
>   return 0;
> }
> 
> $ gcc -O3 main2.c myprintf.c -o test2
> 
> $ ./test2.exe
> Aborted (core dumped)
> 
> $

  I think the answer to this is that you have to use -fno-builtin if you want
to interpose a library function, regardless of LTO or not.

    cheers,
      DaveK

Re: Update LTO plugin interface

[Richard Guenther]

On Wed, Dec 1, 2010 at 11:44 PM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 1:28 PM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>> For linker script, archive, DSO and object file without IR,
>>>> *claimed will return 0 and plugin will save and pass it back to
>>>> linker later in  if it is specified at command line.
>>>
>>> I don't understand what you wrote, so I am going to write what I think
>>> happens.
>>>
>>> The claim_file handler is an interface provided by the plugin itself.
>>> The plugin will register it via LDPT_REGISTER_CLAIM_FILE_HOOK.  The
>>> linker proper will call it for each input file.
>>>
>>> In the case of the LTO plugin, this is the static function
>>> claim_file_handler in lto-plugin.c.
>>>
>>> If the plugin registers a claim_file handler, and, when the linker calls
>>> it, it returns with *claimed == 0, then the linker will process the file
>>> as it normally does.  Since the file will already have been processed,
>>> it does not make sense for the plugin to then pass it back to the
>>> linker.  The effect would be similar to listing the file twice on the
>>> command line.
>>
>> That is what "Discard all previous inputs" in stage 2 linking is for.
>
> But what does that mean?  Are you saying that the linker interface to
> the plugin should change to work that way?  If we do that, then we
> should change other aspects of the plugin interface as well.  It could
> probably become quite a bit simpler.
>
>
> The only reason we would ever need to do a complete relink is if the LTO
> plugin can introduce arbitrary new symbol references.  Is that ever
> possible?  If it is, we need to rethink the whole approach.  If the LTO
> plugin can introduce arbitrary new symbol references, that means that
> LTO plugin can cause arbitrary objects to be pulled in from archives.
> And that means that if we only run the plugin once, we are losing
> possible optimizations, because the plugin will never those new objects.
>
>
> My suspicion is that the LTO plugin can only introduce a small bounded
> set of new symbol references, namely those which we assume can be
> satisified from -lc or -lgcc.  Is that true?

At the moment it would be -lc -lgcc -lm -lpthreads I guess (and maybe
some others, dependent on the target), maybe also our own runtimes
like libssp, libgomp, etc..

The biggest issue is that we can introduce new uses of symbols that the user
is allowed to provide.  For example the linux kernel has a local implementation
of divdi3 because it doesn't want to link libgcc.  If that implementation
is in a static library without LTO bytecode we have to re-consider
that static archive after link-time optimization.

Richard.

> Ian
>

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Wed, Dec 1, 2010 at 5:53 PM, Ian Lance Taylor <iant@google.com> wrote:
>
>> Or I'll rephrase: are there any cases for which using -pass-through for
>> the set of libraries that the gcc driver automatically adds to the end
>> of the link line would not be a complete solution?
>>
>> I'm not saying -pass-through is the solution we should be using.  It's
>> clearly a bit of a hack.  However, I am asking the question seriously,
>> because if we have to do a complete relink, then let's do a complete
>> relink, but if we don't have to do one, let's definitely not.
>
> Nothing can't be solved with a hack :-).

I propose that we add a new linker option: --plugin-callback.  At each
point where this option appears on the command line, the linker will
call a new plugin callback entry point.  The LTO plugin will replace the
all_symbols_read callback with this one.  We will have the gcc driver
run the linker more or less like this:

object-files -lc -lgcc --plugin-callback -lc -lgcc crtend.o crtn.o

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 6:52 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Wed, Dec 1, 2010 at 5:53 PM, Ian Lance Taylor <iant@google.com> wrote:
>>
>>> Or I'll rephrase: are there any cases for which using -pass-through for
>>> the set of libraries that the gcc driver automatically adds to the end
>>> of the link line would not be a complete solution?
>>>
>>> I'm not saying -pass-through is the solution we should be using.  It's
>>> clearly a bit of a hack.  However, I am asking the question seriously,
>>> because if we have to do a complete relink, then let's do a complete
>>> relink, but if we don't have to do one, let's definitely not.
>>
>> Nothing can't be solved with a hack :-).
>
> I propose that we add a new linker option: --plugin-callback.  At each
> point where this option appears on the command line, the linker will
> call a new plugin callback entry point.  The LTO plugin will replace the
> all_symbols_read callback with this one.  We will have the gcc driver
> run the linker more or less like this:
>
> object-files -lc -lgcc --plugin-callback -lc -lgcc crtend.o crtn.o
>

I am not saying we should go with 2 stage linking. Just for clarification.
In 2 stage linking, the first stage only does symbol resolution. The complete
linking starts at the second stage.  So we don't do really complete relink
in 2 stage linking.


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> I am not saying we should go with 2 stage linking. Just for clarification.
> In 2 stage linking, the first stage only does symbol resolution. The complete
> linking starts at the second stage.  So we don't do really complete relink
> in 2 stage linking.

Understood.  But you still read and process all the symbols twice.  It
can be done if it must be done, but it would be significantly better to
do something else if possible.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 9:00 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> I am not saying we should go with 2 stage linking. Just for clarification.
>> In 2 stage linking, the first stage only does symbol resolution. The complete
>> linking starts at the second stage.  So we don't do really complete relink
>> in 2 stage linking.
>
> Understood.  But you still read and process all the symbols twice.  It
> can be done if it must be done, but it would be significantly better to
> do something else if possible.
>

Without 2 stage linking, which many compilers do for LTO, we will add hack
on top of hack.  As Richard pointed out, we need at least

-lc -lgcc -lm -lpthreads -lgcov  and maybe libssp, libgomp, ...

-lm is currently added by user, not GCC driver

Pass-through is just a hack, not a solution.


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Thu, Dec 2, 2010 at 9:00 AM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> I am not saying we should go with 2 stage linking. Just for clarification.
>>> In 2 stage linking, the first stage only does symbol resolution. The complete
>>> linking starts at the second stage.  So we don't do really complete relink
>>> in 2 stage linking.
>>
>> Understood.  But you still read and process all the symbols twice.  It
>> can be done if it must be done, but it would be significantly better to
>> do something else if possible.
>>
>
> Without 2 stage linking, which many compilers do for LTO, we will add hack
> on top of hack.  As Richard pointed out, we need at least
>
> -lc -lgcc -lm -lpthreads -lgcov  and maybe libssp, libgomp, ...
>
> -lm is currently added by user, not GCC driver
>
> Pass-through is just a hack, not a solution.

But I just suggested a different approach, which as far as I can see
solves the entire problem.  Why not criticize that approach, rather than
continusing to discuss -pass-through, which we agree is a hack?

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 9:20 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Thu, Dec 2, 2010 at 9:00 AM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> I am not saying we should go with 2 stage linking. Just for clarification.
>>>> In 2 stage linking, the first stage only does symbol resolution. The complete
>>>> linking starts at the second stage.  So we don't do really complete relink
>>>> in 2 stage linking.
>>>
>>> Understood.  But you still read and process all the symbols twice.  It
>>> can be done if it must be done, but it would be significantly better to
>>> do something else if possible.
>>>
>>
>> Without 2 stage linking, which many compilers do for LTO, we will add hack
>> on top of hack.  As Richard pointed out, we need at least
>>
>> -lc -lgcc -lm -lpthreads -lgcov  and maybe libssp, libgomp, ...
>>
>> -lm is currently added by user, not GCC driver
>>
>> Pass-through is just a hack, not a solution.
>
> But I just suggested a different approach, which as far as I can see
> solves the entire problem.  Why not criticize that approach, rather than
> continusing to discuss -pass-through, which we agree is a hack?

How do you deal with -lm:

[hjl@gnu-6 pr12245-6]$ cat y.c
#include <stdio.h>
#include <stdlib.h>
#include <math.h>

int
main (int argc, char **argv)
{
  int d = atoi (argv[1]);
  printf ("%f\n", sin (d));
  return 0;
}
[hjl@gnu-6 pr12245-6]$ make
/usr/gcc-4.6/bin/gcc -O2 -fwhole-program -flto=jobserver -fuse-linker-plugin
-c -o y.o y.c
/usr/gcc-4.6/bin/gcc -static -o foo -O2 -fwhole-program -flto=jobserver
-fuse-linker-plugin y.o -lm
/tmp/ccrHdm25.ltrans0.ltrans.o: In function `main':
ccrHdm25.ltrans0.o:(.text.startup+0x19): undefined reference to `sin'
collect2: ld returned 1 exit status
make: *** [foo] Error 1
[hjl@gnu-6 pr12245-6]$

	
-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> How do you deal with -lm:

I believe we have agreed that LTO can only introduce new symbol
references that are satisfied by -lc and -lgcc.  Under those conditions,
I believe my proposal will work just fine.

Again, my proposal is this:

    I propose that we add a new linker option: --plugin-callback.  At each
    point where this option appears on the command line, the linker will
    call a new plugin callback entry point.  The LTO plugin will replace the
    all_symbols_read callback with this one.  We will have the gcc driver
    run the linker more or less like this:

    object-files -lc -lgcc --plugin-callback -lc -lgcc crtend.o crtn.o

So if the user runs something like

gcc -o foo foo.o -lm -flto

gcc will execute something like

ld crt1.o crti.o crtbegin.o foo.o -lm
   -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc
   --plugin-callback
   -lc -lgcc --as-needed -lgcc_s --no-as-needed crtend.o crtn.o

Why would that not work?

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 9:41 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> How do you deal with -lm:
>
> I believe we have agreed that LTO can only introduce new symbol
> references that are satisfied by -lc and -lgcc.  Under those conditions,

Have you looked my testcase? The assumption of "LTO can only
introduce new symbol references that are satisfied by -lc and -lgcc."
is wrong.  My testcase shows LTO may introduce new symbol references
to libm.


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Thu, Dec 2, 2010 at 9:41 AM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> How do you deal with -lm:
>>
>> I believe we have agreed that LTO can only introduce new symbol
>> references that are satisfied by -lc and -lgcc.  Under those conditions,
>
> Have you looked my testcase? The assumption of "LTO can only
> introduce new symbol references that are satisfied by -lc and -lgcc."
> is wrong.  My testcase shows LTO may introduce new symbol references
> to libm.

You're right, I didn't fully grasp that the reference to sin in the
source code was somehow being removed and then re-added.

However, I really don't see why this is a serious flaw in my proposal.
You have shown a case in which LTO can introduce a new symbol reference
to -lm.  So we just treat -lm as we do -lc and -lgcc.  This is similar
to how the g++ driver already treats -lm.  It's a detail, not a major
problem.

A major problem would be if LTO could introduce a new symbol reference
which required changing the way we search user defined archives.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 10:37 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Thu, Dec 2, 2010 at 9:41 AM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> How do you deal with -lm:
>>>
>>> I believe we have agreed that LTO can only introduce new symbol
>>> references that are satisfied by -lc and -lgcc.  Under those conditions,
>>
>> Have you looked my testcase? The assumption of "LTO can only
>> introduce new symbol references that are satisfied by -lc and -lgcc."
>> is wrong.  My testcase shows LTO may introduce new symbol references
>> to libm.
>
> You're right, I didn't fully grasp that the reference to sin in the
> source code was somehow being removed and then re-added.
>
> However, I really don't see why this is a serious flaw in my proposal.
> You have shown a case in which LTO can introduce a new symbol reference
> to -lm.  So we just treat -lm as we do -lc and -lgcc.  This is similar
> to how the g++ driver already treats -lm.  It's a detail, not a major
> problem.

g++ adds -lm since libstdc++ uses libm.  If we do it in gcc, we may
add run-time dependency on libm.so to all C programs even if they
don't use libm at all.

> A major problem would be if LTO could introduce a new symbol reference
> which required changing the way we search user defined archives.
>

Your proposal means we have to add lots of library references to
GCC driver. I am not sure if it is a good idea.



-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> g++ adds -lm since libstdc++ uses libm.  If we do it in gcc, we may
> add run-time dependency on libm.so to all C programs even if they
> don't use libm at all.

No.  All the gcc driver has to do is: *if* -lm is used on the link line,
also add it after the use of --plugin-callback.


> Your proposal means we have to add lots of library references to
> GCC driver. I am not sure if it is a good idea.

No.  No library references have to be added.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 11:37 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> g++ adds -lm since libstdc++ uses libm.  If we do it in gcc, we may
>> add run-time dependency on libm.so to all C programs even if they
>> don't use libm at all.
>
> No.  All the gcc driver has to do is: *if* -lm is used on the link line,
> also add it after the use of --plugin-callback.

How many libraries specified at command line do we have to check?
How do we handle -Wl, -Wl,-Bstatic/-Wl,-Bdynamic,-Wl,--start-group, ....
and other linker switches?


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> On Thu, Dec 2, 2010 at 11:37 AM, Ian Lance Taylor <iant@google.com> wrote:
>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>
>>> g++ adds -lm since libstdc++ uses libm.  If we do it in gcc, we may
>>> add run-time dependency on libm.so to all C programs even if they
>>> don't use libm at all.
>>
>> No.  All the gcc driver has to do is: *if* -lm is used on the link line,
>> also add it after the use of --plugin-callback.
>
> How many libraries specified at command line do we have to check?

We only have to check the ones for which LTO can potentially generate a
reference which did not exist in the original object file.  Or, more
precisely, for which LTO can generate a reference which the plugin did
not report when the original object file was seen.

As far as I can tell at this point, the complete set of libraries we
need to check is: -lc -lm -lgcc -lgcc_s.

We could remove -lm from that list if we arranged for the plugin to
report a potential reference to sin even if the object file does not
list it for whatever reason.  I saw your earlier test case but I don't
actually know why the reference to sin in the source code disappeared
from the object file and was reinserted by LTO.


> How do we handle -Wl, -Wl,-Bstatic/-Wl,-Bdynamic,-Wl,--start-group, ....
> and other linker switches?

A fair point.  Ideally we would want to recreate the state of those
options for all libraries explicitly mentioned by the user.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 11:53 AM, Ian Lance Taylor <iant@google.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> On Thu, Dec 2, 2010 at 11:37 AM, Ian Lance Taylor <iant@google.com> wrote:
>>> "H.J. Lu" <hjl.tools@gmail.com> writes:
>>>
>>>> g++ adds -lm since libstdc++ uses libm.  If we do it in gcc, we may
>>>> add run-time dependency on libm.so to all C programs even if they
>>>> don't use libm at all.
>>>
>>> No.  All the gcc driver has to do is: *if* -lm is used on the link line,
>>> also add it after the use of --plugin-callback.
>>
>> How many libraries specified at command line do we have to check?
>
> We only have to check the ones for which LTO can potentially generate a
> reference which did not exist in the original object file.  Or, more
> precisely, for which LTO can generate a reference which the plugin did
> not report when the original object file was seen.
>
> As far as I can tell at this point, the complete set of libraries we
> need to check is: -lc -lm -lgcc -lgcc_s.

It isn't a complete list. At least, we need -lgcov and may be others.

>
> We could remove -lm from that list if we arranged for the plugin to
> report a potential reference to sin even if the object file does not
> list it for whatever reason.  I saw your earlier test case but I don't
> actually know why the reference to sin in the source code disappeared
> from the object file and was reinserted by LTO.
>
>
>> How do we handle -Wl, -Wl,-Bstatic/-Wl,-Bdynamic,-Wl,--start-group, ....
>> and other linker switches?
>
> A fair point.  Ideally we would want to recreate the state of those
> options for all libraries explicitly mentioned by the user.

It can get quite complex. You start turning GCC driver into a linker :-).
What do we do when a new switch is added to linker?

-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

"H.J. Lu" <hjl.tools@gmail.com> writes:

>> As far as I can tell at this point, the complete set of libraries we
>> need to check is: -lc -lm -lgcc -lgcc_s.
>
> It isn't a complete list. At least, we need -lgcov and may be others.

Only to the extent that we feel that it is important to permit the user
to specify -lgcov on the link line in an unusual way, and also to make
it work with LTO.  For complex cases I would recommend instead providing
an escape hatch.


>> We could remove -lm from that list if we arranged for the plugin to
>> report a potential reference to sin even if the object file does not
>> list it for whatever reason.  I saw your earlier test case but I don't
>> actually know why the reference to sin in the source code disappeared
>> from the object file and was reinserted by LTO.
>>
>>
>>> How do we handle -Wl, -Wl,-Bstatic/-Wl,-Bdynamic,-Wl,--start-group, ....
>>> and other linker switches?
>>
>> A fair point.  Ideally we would want to recreate the state of those
>> options for all libraries explicitly mentioned by the user.
>
> It can get quite complex. You start turning GCC driver into a linker :-).
> What do we do when a new switch is added to linker?

There is a big difference between automatically handling a few linker
options and turning the driver into a linker.  What we are talking about
here is quite similar to what the g++ driver already does.

And we really just have to make the simple cases work cleanly.  For
anything complicated we can quite reasonably say "this is what you have
to do to use LTO."

Anyhow, I have a proposal on the table which I think can do the job.  If
you want to instead advocate for a complete relink approach, then we
should significantly change and simplify the plugin interface.  The
plugin support in gold will need to be rewritten; not sure about GNU ld.
I think that will force the linker to do a fair bit of extra work to no
useful purpose.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 12:10 PM, Ian Lance Taylor <iant@google.com> wrote:
>
> Anyhow, I have a proposal on the table which I think can do the job.  If
> you want to instead advocate for a complete relink approach, then we
> should significantly change and simplify the plugin interface.  The
> plugin support in gold will need to be rewritten; not sure about GNU ld.
> I think that will force the linker to do a fair bit of extra work to no
> useful purpose.
>

There are 2 proposals:

1. 2 stage linking.
2.  --plugin-callback

2 stage linking in GNU linker isn't very hard. I just need to exclude
inputs from stage 1 and drop 2 hash tables. I can continue stage 2
linking to generate executable.

-- 
H.J.

Re: Update LTO plugin interface

[Dave Korn]

On 02/12/2010 20:10, Ian Lance Taylor wrote:

> Anyhow, I have a proposal on the table which I think can do the job.  If
> you want to instead advocate for a complete relink approach, then we
> should significantly change and simplify the plugin interface.  The
> plugin support in gold will need to be rewritten; not sure about GNU ld.
> I think that will force the linker to do a fair bit of extra work to no
> useful purpose.

  I'm wondering if the linker shouldn't just gather the plugin-contributed
object files, substitute them into appropriate places on the original
command-line, and re-exec itself.

    cheers,
      DaveK

Re: Update LTO plugin interface

[Ian Lance Taylor]

Dave Korn <dave.korn.cygwin@gmail.com> writes:

> On 02/12/2010 20:10, Ian Lance Taylor wrote:
>
>> Anyhow, I have a proposal on the table which I think can do the job.  If
>> you want to instead advocate for a complete relink approach, then we
>> should significantly change and simplify the plugin interface.  The
>> plugin support in gold will need to be rewritten; not sure about GNU ld.
>> I think that will force the linker to do a fair bit of extra work to no
>> useful purpose.
>
>   I'm wondering if the linker shouldn't just gather the plugin-contributed
> object files, substitute them into appropriate places on the original
> command-line, and re-exec itself.

We are we letting these small, solvable, problems head us toward a
solution which is much less efficient?  When we've already put a great
deal of work into making the more efficient solution work?

Ian

Re: Update LTO plugin interface

[Cary Coutant]

>  I'm wondering if the linker shouldn't just gather the plugin-contributed
> object files, substitute them into appropriate places on the original
> command-line, and re-exec itself.

That's not effectively different from the collect2 approach that we
had before the linker plugin interface. The only thing collect2
couldn't handle was IR files inside archives, and that wouldn't have
required much additional effort to add.

The benefit of the plugin approach is that you don't have to link twice.

-cary

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 1:01 PM, Cary Coutant <ccoutant@google.com> wrote:
>>  I'm wondering if the linker shouldn't just gather the plugin-contributed
>> object files, substitute them into appropriate places on the original
>> command-line, and re-exec itself.
>
> That's not effectively different from the collect2 approach that we
> had before the linker plugin interface. The only thing collect2
> couldn't handle was IR files inside archives, and that wouldn't have
> required much additional effort to add.

Does collect2 know linker scripts? Like

hjl@gnu-32 doc]$ cat /usr/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-i386)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED (
/lib/ld-linux.so.2 ) )
[hjl@gnu-32 doc]$

> The benefit of the plugin approach is that you don't have to link twice.
>

You can just do symbol resolution in stage 1.  No need for
complete linking in stage 1.


-- 
H.J.

Re: Update LTO plugin interface

[Cary Coutant]

> I propose that we add a new linker option: --plugin-callback.  At each
> point where this option appears on the command line, the linker will
> call a new plugin callback entry point.  The LTO plugin will replace the
> all_symbols_read callback with this one.  We will have the gcc driver
> run the linker more or less like this:
>
> object-files -lc -lgcc --plugin-callback -lc -lgcc crtend.o crtn.o

I'm not sure how this is any better than the pass-through option we're
already using. That just has the plugin re-inject those libraries that
you have placed after the --plugin-callback option. The crtend.o and
crtn.o files could be handled by having the plugin claim them and
re-inject them at the end.

For every new routine that the gcc backend generates a new call to, it
ought to know which library that routine is defined in, and should be
able to add that library after the generated object(s) during the
all-symbols-read callback. We really don't want to support arbitrary
interposition at that point, because a user-supplied replacement might
invalidate some assumptions that were made during optimization.

-cary

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 1:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>> I propose that we add a new linker option: --plugin-callback.  At each
>> point where this option appears on the command line, the linker will
>> call a new plugin callback entry point.  The LTO plugin will replace the
>> all_symbols_read callback with this one.  We will have the gcc driver
>> run the linker more or less like this:
>>
>> object-files -lc -lgcc --plugin-callback -lc -lgcc crtend.o crtn.o
>
> I'm not sure how this is any better than the pass-through option we're
> already using. That just has the plugin re-inject those libraries that
> you have placed after the --plugin-callback option. The crtend.o and
> crtn.o files could be handled by having the plugin claim them and
> re-inject them at the end.
>
> For every new routine that the gcc backend generates a new call to, it
> ought to know which library that routine is defined in, and should be
> able to add that library after the generated object(s) during the
> all-symbols-read callback. We really don't want to support arbitrary
> interposition at that point, because a user-supplied replacement might
> invalidate some assumptions that were made during optimization.
>

It sounds reasonable. We have to add those libraries very carefully.

For example, with "gcc -static" on Linux, the reference to glibc should
be passed to linker as "--start-group -lgcc -lgcc_eh -lc --end-group".


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

Cary Coutant <ccoutant@google.com> writes:

>> I propose that we add a new linker option: --plugin-callback.  At each
>> point where this option appears on the command line, the linker will
>> call a new plugin callback entry point.  The LTO plugin will replace the
>> all_symbols_read callback with this one.  We will have the gcc driver
>> run the linker more or less like this:
>>
>> object-files -lc -lgcc --plugin-callback -lc -lgcc crtend.o crtn.o
>
> I'm not sure how this is any better than the pass-through option we're
> already using. That just has the plugin re-inject those libraries that
> you have placed after the --plugin-callback option. The crtend.o and
> crtn.o files could be handled by having the plugin claim them and
> re-inject them at the end.

I think it's a little bit better because it specifically handles the
case of crtend.o and crtn.o.  You're right, we could have the plugin
know about those and claim them.  But now the knowledge is in two
places.  But I don't feel very strongly about it as long as we can make
it work.


> For every new routine that the gcc backend generates a new call to, it
> ought to know which library that routine is defined in, and should be
> able to add that library after the generated object(s) during the
> all-symbols-read callback. We really don't want to support arbitrary
> interposition at that point, because a user-supplied replacement might
> invalidate some assumptions that were made during optimization.

That is another good point.  I still don't understand why in HJ's
example a new not-previously-existing reference to -lm is introduced.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 1:28 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>> For every new routine that the gcc backend generates a new call to, it
>> ought to know which library that routine is defined in, and should be
>> able to add that library after the generated object(s) during the
>> all-symbols-read callback. We really don't want to support arbitrary
>> interposition at that point, because a user-supplied replacement might
>> invalidate some assumptions that were made during optimization.
>>
>
> It sounds reasonable. We have to add those libraries very carefully.
>
> For example, with "gcc -static" on Linux, the reference to glibc should
> be passed to linker as "--start-group -lgcc -lgcc_eh -lc --end-group".
>
>

One approach is to use

[hjl@gnu-6 pr12245-6]$ cat libc_static.a
GROUP (-lgcc -lgcc_eh -lc)
[hjl@gnu-6 pr12245-6]$

instead of  "--start-group -lgcc -lgcc_eh -lc --end-group".

-- 
H.J.

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 1:10 PM, Cary Coutant <ccoutant@google.com> wrote:
>
> For every new routine that the gcc backend generates a new call to, it
> ought to know which library that routine is defined in, and should be
> able to add that library after the generated object(s) during the
> all-symbols-read callback. We really don't want to support arbitrary
> interposition at that point, because a user-supplied replacement might
> invalidate some assumptions that were made during optimization.
>

For each libcall, we need to decorate

1. Which library it comes from. It is OS/target dependent.
2. The dynamic and static library names.  In most cases,  they
are the same.  For glibc, they are different.


-- 
H.J.

Re: Update LTO plugin interface

[Cary Coutant]

> For each libcall, we need to decorate
>
> 1. Which library it comes from. It is OS/target dependent.
> 2. The dynamic and static library names.  In most cases,  they
> are the same.  For glibc, they are different.

Is there a relatively painless way to enumerate all the possible
libcalls? We could add a new plugin api and have the LTO plugin
register those symbols up front. The linker would make a note of where
each symbol is defined, and then automatically go back and add any
objects needed to resolve any references introduced by the
optimization pass.

For the crtend files we could add a linker option that makes them
known as endcaps, and the linker could make sure they get laid out
last:

   ld ... -lc -lgcc ... --endcap crtend.o crtn.o

That puts the special knowledge about those files back in the gcc driver.

-cary

Re: Update LTO plugin interface

[Dave Korn]

On 02/12/2010 23:29, Cary Coutant wrote:

> For the crtend files we could add a linker option that makes them
> known as endcaps, and the linker could make sure they get laid out
> last:
> 
>    ld ... -lc -lgcc ... --endcap crtend.o crtn.o
> 
> That puts the special knowledge about those files back in the gcc driver.

  Hmm, yes.  It doesn't work to just pass-through crtn.o, because ...

> Executing on host: /home/davek/gcc/obj-gold2/gcc/xgcc -B/home/davek/gcc/obj-gold2/gcc/ c_lto_20100722-1_0.o  -O0 -flto -flto-partition=none  -fuse-linker-plugin      -o gcc-dg-lto-20100722-1-01    (timeout = 300)
> gold: /home/davek/gcc/obj-gold2/gcc/crtbegin.o:(.text+0x13): error: undefined reference to '__DTOR_END__'
> collect2: ld returned 1 exit status
> compiler exited with status 1
> output is:
> gold: /home/davek/gcc/obj-gold2/gcc/crtbegin.o:(.text+0x13): error: undefined reference to '__DTOR_END__'
> collect2: ld returned 1 exit status
> 

... it's needed by the first pass of symbol resolution.

    cheers,
      DaveK

Re: Update LTO plugin interface

[Ian Lance Taylor]

Cary Coutant <ccoutant@google.com> writes:

>> For each libcall, we need to decorate
>>
>> 1. Which library it comes from. It is OS/target dependent.
>> 2. The dynamic and static library names.  In most cases,  they
>> are the same.  For glibc, they are different.
>
> Is there a relatively painless way to enumerate all the possible
> libcalls? We could add a new plugin api and have the LTO plugin
> register those symbols up front. The linker would make a note of where
> each symbol is defined, and then automatically go back and add any
> objects needed to resolve any references introduced by the
> optimization pass.

This isn't quite what should happen, though.  If the user does not
specify -lm on the link line, then we should not add -lm, even if LTO
somehow introduces a function that is defined in libm.  Automatically
adding -lm would introduce a surprising dynamic dependency in some
cases.  The user should be explicit about that.

That is, it really doesn't matter which library libcalls come from.  All
we need to know is which libraries might satisfy new libcalls.

This is what leads me in the direction of copying certain libraries
mentioned on the command line to follow the LTO objects, much as
-pass-through does.  I think we just need to make that a little bit more
automatic.

Ian

Re: Update LTO plugin interface

[Cary Coutant]

>> For the crtend files we could add a linker option that makes them
>> known as endcaps, and the linker could make sure they get laid out
>> last:
>>
>>    ld ... -lc -lgcc ... --endcap crtend.o crtn.o
>>
>> That puts the special knowledge about those files back in the gcc driver.
>
>  Hmm, yes.  It doesn't work to just pass-through crtn.o, because ...
>
>> Executing on host: /home/davek/gcc/obj-gold2/gcc/xgcc -B/home/davek/gcc/obj-gold2/gcc/ c_lto_20100722-1_0.o  -O0 -flto -flto-partition=none  -fuse-linker-plugin      -o gcc-dg-lto-20100722-1-01    (timeout = 300)
>> gold: /home/davek/gcc/obj-gold2/gcc/crtbegin.o:(.text+0x13): error: undefined reference to '__DTOR_END__'
>> collect2: ld returned 1 exit status
>> compiler exited with status 1
>> output is:
>> gold: /home/davek/gcc/obj-gold2/gcc/crtbegin.o:(.text+0x13): error: undefined reference to '__DTOR_END__'
>> collect2: ld returned 1 exit status
>>
>
> ... it's needed by the first pass of symbol resolution.

Yeah, that's why I originally suggested having the plugin claim it, so
that the plugin could define the symbols when they need to be defined.

-cary

Re: Update LTO plugin interface

[Cary Coutant]

> This isn't quite what should happen, though.  If the user does not
> specify -lm on the link line, then we should not add -lm, even if LTO
> somehow introduces a function that is defined in libm.  Automatically
> adding -lm would introduce a surprising dynamic dependency in some
> cases.  The user should be explicit about that.

I wouldn't expect the compiler to introduce a call to anything in libm
when -lm isn't already specified on the command line (that would break
even without LTO).

With what I'm suggesting, we'll only resolve libcalls to libraries
that were originally specified on the command line.

> That is, it really doesn't matter which library libcalls come from.  All
> we need to know is which libraries might satisfy new libcalls.
>
> This is what leads me in the direction of copying certain libraries
> mentioned on the command line to follow the LTO objects, much as
> -pass-through does.  I think we just need to make that a little bit more
> automatic.

Another way to do this would be to put a marker in the command line
that identifies where those libraries begin, and the linker could just
go back and rescan those libraries if needed, before the final layout
of the endcaps.

-cary

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 4:04 PM, Cary Coutant <ccoutant@google.com> wrote:
>> This isn't quite what should happen, though.  If the user does not
>> specify -lm on the link line, then we should not add -lm, even if LTO
>> somehow introduces a function that is defined in libm.  Automatically
>> adding -lm would introduce a surprising dynamic dependency in some
>> cases.  The user should be explicit about that.
>
> I wouldn't expect the compiler to introduce a call to anything in libm
> when -lm isn't already specified on the command line (that would break
> even without LTO).
>
> With what I'm suggesting, we'll only resolve libcalls to libraries
> that were originally specified on the command line.
>
>> That is, it really doesn't matter which library libcalls come from.  All
>> we need to know is which libraries might satisfy new libcalls.
>>
>> This is what leads me in the direction of copying certain libraries
>> mentioned on the command line to follow the LTO objects, much as
>> -pass-through does.  I think we just need to make that a little bit more
>> automatic.
>
> Another way to do this would be to put a marker in the command line
> that identifies where those libraries begin, and the linker could just
> go back and rescan those libraries if needed, before the final layout
> of the endcaps.
>

We also need to handle .a vs .so.  We shouldn't add libm,so when user
specifies libm.a at command line.


-- 
H.J.

Re: Update LTO plugin interface

[Ian Lance Taylor]

Cary Coutant <ccoutant@google.com> writes:

>> This isn't quite what should happen, though.  If the user does not
>> specify -lm on the link line, then we should not add -lm, even if LTO
>> somehow introduces a function that is defined in libm.  Automatically
>> adding -lm would introduce a surprising dynamic dependency in some
>> cases.  The user should be explicit about that.
>
> I wouldn't expect the compiler to introduce a call to anything in libm
> when -lm isn't already specified on the command line (that would break
> even without LTO).

Right--that would break without LTO, but it doesn't follow that we
should make it work with LTO.

> With what I'm suggesting, we'll only resolve libcalls to libraries
> that were originally specified on the command line.

OK.  Another concern I have with this is that there is no canonical
location across all systems for some of these functions.  In general -lc
and -lm will suffice, but on some specific systems they may not.  And
there are functions which are in libc on some systems and in libm on
others, such as frexp.  So if we have a mapping from function to
library, I'm afraid we may get caught up in system dependencies which
are really kind of irrelevant.  What really matters, I think, is not the
specific library per function; it's the set of libraries we need to
rescan.


> Another way to do this would be to put a marker in the command line
> that identifies where those libraries begin, and the linker could just
> go back and rescan those libraries if needed, before the final layout
> of the endcaps.

I like that idea in general, but the difficulty is knowing where to put
the marker.  E.g., the user is going to specify -lm, and we are going to
need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
rescan -lm but we don't really need to rescan mylib1 and mylib2.

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Thu, Dec 2, 2010 at 5:39 PM, Ian Lance Taylor <iant@google.com> wrote:
> Cary Coutant <ccoutant@google.com> writes:
>
>>> This isn't quite what should happen, though.  If the user does not
>>> specify -lm on the link line, then we should not add -lm, even if LTO
>>> somehow introduces a function that is defined in libm.  Automatically
>>> adding -lm would introduce a surprising dynamic dependency in some
>>> cases.  The user should be explicit about that.
>>
>> I wouldn't expect the compiler to introduce a call to anything in libm
>> when -lm isn't already specified on the command line (that would break
>> even without LTO).
>
> Right--that would break without LTO, but it doesn't follow that we
> should make it work with LTO.
>
>> With what I'm suggesting, we'll only resolve libcalls to libraries
>> that were originally specified on the command line.
>
> OK.  Another concern I have with this is that there is no canonical
> location across all systems for some of these functions.  In general -lc
> and -lm will suffice, but on some specific systems they may not.  And
> there are functions which are in libc on some systems and in libm on
> others, such as frexp.  So if we have a mapping from function to
> library, I'm afraid we may get caught up in system dependencies which
> are really kind of irrelevant.  What really matters, I think, is not the
> specific library per function; it's the set of libraries we need to
> rescan.
>
>
>> Another way to do this would be to put a marker in the command line
>> that identifies where those libraries begin, and the linker could just
>> go back and rescan those libraries if needed, before the final layout
>> of the endcaps.
>
> I like that idea in general, but the difficulty is knowing where to put
> the marker.  E.g., the user is going to specify -lm, and we are going to
> need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
> rescan -lm but we don't really need to rescan mylib1 and mylib2.
>

All those complexities make 2 stage linking more attractive.  I
think I can implement it in GNU linker with the current plugin API.

Linker just needs to remember the command line options, including

--start-group/--end-group
-as-needed/--no-as-needed
--copy-dt-needed-entries/--no-copy-dt-needed-entries

in stage 1.

In stage 2, it will place LTO trans files before the first IR file
claimed by plugin and process the command line options.

--whole-archive may need some special handling.  Archives
after --whole-archive will be ignored in stage 2.


-- 
H.J.

Re: Update LTO plugin interface

[Cary Coutant]

>> I wouldn't expect the compiler to introduce a call to anything in libm
>> when -lm isn't already specified on the command line (that would break
>> even without LTO).
>
> Right--that would break without LTO, but it doesn't follow that we
> should make it work with LTO.

That's my point, too.

>> With what I'm suggesting, we'll only resolve libcalls to libraries
>> that were originally specified on the command line.
>
> OK.  Another concern I have with this is that there is no canonical
> location across all systems for some of these functions.  In general -lc
> and -lm will suffice, but on some specific systems they may not.  And
> there are functions which are in libc on some systems and in libm on
> others, such as frexp.  So if we have a mapping from function to
> library, I'm afraid we may get caught up in system dependencies which
> are really kind of irrelevant.  What really matters, I think, is not the
> specific library per function; it's the set of libraries we need to
> rescan.

My suggestion was to have the plugin give the linker a list of
libcalls, and the linker would note where it found each symbol during
its normal library scanning. If we need one of those symbols later and
haven't already linked it in, we'll know where to go find it (we won't
even have to rescan the library). We don't need to have a canonical
list of libraries or locations -- we just go with what was given on
the command line.

>> Another way to do this would be to put a marker in the command line
>> that identifies where those libraries begin, and the linker could just
>> go back and rescan those libraries if needed, before the final layout
>> of the endcaps.
>
> I like that idea in general, but the difficulty is knowing where to put
> the marker.  E.g., the user is going to specify -lm, and we are going to
> need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
> rescan -lm but we don't really need to rescan mylib1 and mylib2.

Right. The gcc driver would have to have some concept of which
libraries might contain libcalls, and add brackets around them on the
linker command line to mark them as runtime support libraries, whether
they're added by the driver itself, or passed on the gcc command line
by the user.

-cary

Re: Update LTO plugin interface

[Ian Lance Taylor]

Cary Coutant <ccoutant@google.com> writes:

> My suggestion was to have the plugin give the linker a list of
> libcalls, and the linker would note where it found each symbol during
> its normal library scanning.

Ah, I see.  OK, that makes sense.

> Right. The gcc driver would have to have some concept of which
> libraries might contain libcalls, and add brackets around them on the
> linker command line to mark them as runtime support libraries, whether
> they're added by the driver itself, or passed on the gcc command line
> by the user.

Sounds good.

Ian

Re: Update LTO plugin interface

[Cary Coutant]

> For the crtend files we could add a linker option that makes them
> known as endcaps, and the linker could make sure they get laid out
> last:
>
>   ld ... -lc -lgcc ... --endcap crtend.o crtn.o
>
> That puts the special knowledge about those files back in the gcc driver.

I should have remembered that I already dealt with this problem long
ago -- gold defers the layout of all real objects that follow the
first claimed IR file until after the replacement files have been laid
out. With respect to physical layout of the sections, this effectively
makes the link order equivalent to putting the replacement files in
the place of the first IR file. No "endcap" option is necessary.

-cary

Re: Update LTO plugin interface

[Cary Coutant]

>>> Another way to do this would be to put a marker in the command line
>>> that identifies where those libraries begin, and the linker could just
>>> go back and rescan those libraries if needed, before the final layout
>>> of the endcaps.
>>
>> I like that idea in general, but the difficulty is knowing where to put
>> the marker.  E.g., the user is going to specify -lm, and we are going to
>> need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
>> rescan -lm but we don't really need to rescan mylib1 and mylib2.
>
> All those complexities make 2 stage linking more attractive.  I
> think I can implement it in GNU linker with the current plugin API.
>
> Linker just needs to remember the command line options, including
>
> --start-group/--end-group
> -as-needed/--no-as-needed
> --copy-dt-needed-entries/--no-copy-dt-needed-entries
>
> in stage 1.
>
> In stage 2, it will place LTO trans files before the first IR file
> claimed by plugin and process the command line options.
>
> --whole-archive may need some special handling.  Archives
> after --whole-archive will be ignored in stage 2.

It seems to me that we just need to add a few more libraries as
pass-through libraries, being careful to add a pass-through option
only for libraries that are already on the command line. How does that
add up to "all those complexities"?

With what you've written here, you've just added to the complexity of
your proposed solution, which makes it a much bigger change --
especially since what you're proposing will require changes in both
linkers. Adding pass-through options is a gcc driver change only.

The pass-through option may be seen as a hack, but I don't think it's
that big of a hack, and it does work. I don't see it as fundamentally
different from adding an option to mark runtime support libraries --
the difference is really just syntax.

In the long term, I'd prefer to see improvements more along the lines
of what I've suggested earlier in this thread -- define a set of
runtime support routines that the backend can generate calls to, and
make those known to the linker so that they can be located during the
first pass. That's the best way to ensure that we have a complete
picture of the whole program for the optimizer.

For now, I think it's sufficient to fix the driver to add the
necessary pass-through options, and maybe gnu ld needs to be fixed to
handle the crtend files correctly. We also should address Jan's
concerns with COMDAT.

-cary

Re: Update LTO plugin interface

[H.J. Lu]

On Fri, Dec 3, 2010 at 11:10 AM, Cary Coutant <ccoutant@google.com> wrote:
>>>> Another way to do this would be to put a marker in the command line
>>>> that identifies where those libraries begin, and the linker could just
>>>> go back and rescan those libraries if needed, before the final layout
>>>> of the endcaps.
>>>
>>> I like that idea in general, but the difficulty is knowing where to put
>>> the marker.  E.g., the user is going to specify -lm, and we are going to
>>> need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
>>> rescan -lm but we don't really need to rescan mylib1 and mylib2.
>>
>> All those complexities make 2 stage linking more attractive.  I
>> think I can implement it in GNU linker with the current plugin API.
>>
>> Linker just needs to remember the command line options, including
>>
>> --start-group/--end-group
>> -as-needed/--no-as-needed
>> --copy-dt-needed-entries/--no-copy-dt-needed-entries
>>
>> in stage 1.
>>
>> In stage 2, it will place LTO trans files before the first IR file
>> claimed by plugin and process the command line options.
>>
>> --whole-archive may need some special handling.  Archives
>> after --whole-archive will be ignored in stage 2.
>
> It seems to me that we just need to add a few more libraries as
> pass-through libraries, being careful to add a pass-through option
> only for libraries that are already on the command line. How does that
> add up to "all those complexities"?
>
> With what you've written here, you've just added to the complexity of
> your proposed solution, which makes it a much bigger change --
> especially since what you're proposing will require changes in both
> linkers. Adding pass-through options is a gcc driver change only.
>

I will implement 2 stage linking in GNU linker with the current plugin API.
The change shouldn't be too big.  pass-through isn't needed.  If we
keep it in GCC driver, my linker will simply ignore it.

One benefit is everything will just work, with or without LTO.

-- 
H.J.

Re: Update LTO plugin interface

[H.J. Lu]

On Fri, Dec 3, 2010 at 11:16 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Dec 3, 2010 at 11:10 AM, Cary Coutant <ccoutant@google.com> wrote:
>>>>> Another way to do this would be to put a marker in the command line
>>>>> that identifies where those libraries begin, and the linker could just
>>>>> go back and rescan those libraries if needed, before the final layout
>>>>> of the endcaps.
>>>>
>>>> I like that idea in general, but the difficulty is knowing where to put
>>>> the marker.  E.g., the user is going to specify -lm, and we are going to
>>>> need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
>>>> rescan -lm but we don't really need to rescan mylib1 and mylib2.
>>>
>>> All those complexities make 2 stage linking more attractive.  I
>>> think I can implement it in GNU linker with the current plugin API.
>>>
>>> Linker just needs to remember the command line options, including
>>>
>>> --start-group/--end-group
>>> -as-needed/--no-as-needed
>>> --copy-dt-needed-entries/--no-copy-dt-needed-entries
>>>
>>> in stage 1.
>>>
>>> In stage 2, it will place LTO trans files before the first IR file
>>> claimed by plugin and process the command line options.
>>>
>>> --whole-archive may need some special handling.  Archives
>>> after --whole-archive will be ignored in stage 2.
>>
>> It seems to me that we just need to add a few more libraries as
>> pass-through libraries, being careful to add a pass-through option
>> only for libraries that are already on the command line. How does that
>> add up to "all those complexities"?
>>
>> With what you've written here, you've just added to the complexity of
>> your proposed solution, which makes it a much bigger change --
>> especially since what you're proposing will require changes in both
>> linkers. Adding pass-through options is a gcc driver change only.
>>
>
> I will implement 2 stage linking in GNU linker with the current plugin API.
> The change shouldn't be too big.  pass-through isn't needed.  If we
> keep it in GCC driver, my linker will simply ignore it.
>
> One benefit is everything will just work, with or without LTO.
>

I checked the first patch into hjl/lto branch at

http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary

Now I can collect stage 2 command line.

-- 
H.J.

Re: Update LTO plugin interface

[H.J. Lu]

On Fri, Dec 3, 2010 at 4:49 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Dec 3, 2010 at 11:16 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>> On Fri, Dec 3, 2010 at 11:10 AM, Cary Coutant <ccoutant@google.com> wrote:
>>>>>> Another way to do this would be to put a marker in the command line
>>>>>> that identifies where those libraries begin, and the linker could just
>>>>>> go back and rescan those libraries if needed, before the final layout
>>>>>> of the endcaps.
>>>>>
>>>>> I like that idea in general, but the difficulty is knowing where to put
>>>>> the marker.  E.g., the user is going to specify -lm, and we are going to
>>>>> need to rescan it.  If the user writes -lm -lmylib1 -lmylib2 we want to
>>>>> rescan -lm but we don't really need to rescan mylib1 and mylib2.
>>>>
>>>> All those complexities make 2 stage linking more attractive.  I
>>>> think I can implement it in GNU linker with the current plugin API.
>>>>
>>>> Linker just needs to remember the command line options, including
>>>>
>>>> --start-group/--end-group
>>>> -as-needed/--no-as-needed
>>>> --copy-dt-needed-entries/--no-copy-dt-needed-entries
>>>>
>>>> in stage 1.
>>>>
>>>> In stage 2, it will place LTO trans files before the first IR file
>>>> claimed by plugin and process the command line options.
>>>>
>>>> --whole-archive may need some special handling.  Archives
>>>> after --whole-archive will be ignored in stage 2.
>>>
>>> It seems to me that we just need to add a few more libraries as
>>> pass-through libraries, being careful to add a pass-through option
>>> only for libraries that are already on the command line. How does that
>>> add up to "all those complexities"?
>>>
>>> With what you've written here, you've just added to the complexity of
>>> your proposed solution, which makes it a much bigger change --
>>> especially since what you're proposing will require changes in both
>>> linkers. Adding pass-through options is a gcc driver change only.
>>>
>>
>> I will implement 2 stage linking in GNU linker with the current plugin API.
>> The change shouldn't be too big.  pass-through isn't needed.  If we
>> keep it in GCC driver, my linker will simply ignore it.
>>
>> One benefit is everything will just work, with or without LTO.
>>
>
> I checked the first patch into hjl/lto branch at
>
> http://git.kernel.org/?p=devel/binutils/hjl/x86.git;a=summary
>
> Now I can collect stage 2 command line.
>

I checked in a patch to implement stage 2 linking. Everything
seems to work, including "gcc -static ... -lm".


-- 
H.J.

Re: Update LTO plugin interface

[Dave Korn]

On 04/12/2010 01:24, H.J. Lu wrote:

> I checked in a patch to implement stage 2 linking. Everything
> seems to work, including "gcc -static ... -lm".

  Any chance you could send a complete diff?

    cheers,
      DaveK

Re: Update LTO plugin interface

[Ian Lance Taylor]

Dave Korn <dave.korn.cygwin@gmail.com> writes:

> On 04/12/2010 01:24, H.J. Lu wrote:
>
>> I checked in a patch to implement stage 2 linking. Everything
>> seems to work, including "gcc -static ... -lm".
>
>   Any chance you could send a complete diff?

I just want to note that I continue to think this is a really bad idea,
and that we can easily fix the problems without going down this route.

Also, if we do decide to go this route for some reason I can not
currently fathom, we should completely revamp the plugin interface to
make it much simpler.

Any timing comparisons for working cases?

Ian

Re: Update LTO plugin interface

[H.J. Lu]

On Fri, Dec 3, 2010 at 6:23 PM, Dave Korn <dave.korn.cygwin@gmail.com> wrote:
> On 04/12/2010 01:24, H.J. Lu wrote:
>
>> I checked in a patch to implement stage 2 linking. Everything
>> seems to work, including "gcc -static ... -lm".
>
>  Any chance you could send a complete diff?
>

I will submit a complete diff after I fix a few corner cases.
In the meantime, you can clone my git tree and do a "git diff".


-- 
H.J.

Re: Update LTO plugin interface

[H.J. Lu]

On Fri, Dec 3, 2010 at 6:34 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Dec 3, 2010 at 6:23 PM, Dave Korn <dave.korn.cygwin@gmail.com> wrote:
>> On 04/12/2010 01:24, H.J. Lu wrote:
>>
>>> I checked in a patch to implement stage 2 linking. Everything
>>> seems to work, including "gcc -static ... -lm".
>>
>>  Any chance you could send a complete diff?
>>
>
> I will submit a complete diff after I fix a few corner cases.
> In the meantime, you can clone my git tree and do a "git diff".
>

I checked a few more fixes into hjl/lto branch.

The known issue:  --whole-archive will call plugin on archives with IR
in stage 2
linking.



-- 
H.J.

Re: Update LTO plugin interface

[H.J. Lu]

On Fri, Dec 3, 2010 at 6:15 PM, Ian Lance Taylor <iant@google.com> wrote:
> Dave Korn <dave.korn.cygwin@gmail.com> writes:
>
>> On 04/12/2010 01:24, H.J. Lu wrote:
>>
>>> I checked in a patch to implement stage 2 linking. Everything
>>> seems to work, including "gcc -static ... -lm".
>>
>>   Any chance you could send a complete diff?
>
> I just want to note that I continue to think this is a really bad idea,
> and that we can easily fix the problems without going down this route.
>
> Also, if we do decide to go this route for some reason I can not
> currently fathom, we should completely revamp the plugin interface to
> make it much simpler.

The current API works OK with 2 stage BFD linker. My 2 stage BFD
linker works with unmodified GCC 4.6. I will try to get it into binutils.

Gold and ld can have different strategies for LTO plugin support as
long as the same GCC driver binary works with both gold and ld binaries.

> Any timing comparisons for working cases?
>

Here are timings to link GCC 4.6 cc1 on Core i7 870 2.93GHz:

a. 1 stage linking:

409.94s user 5.48s system 99% cpu 6:58.88 total

b. 2 stage linking:

410.08s user 5.53s system 98% cpu 7:00.00 total

It isn't a real surprise since most times are spent in LTO plugin.

Thanks.


-- 
H.J.