externally_visible and resoultion file

externally_visible and resoultion file

[Bingfeng Mei]

Hi, Richard,
With resolution file generated by GOLD (or I am going to hack gnu LD),  is 
externally_visible attribute still needed to annotate those symbols accessed
from non-LTO objects when compiling with -fwhole-program.

In theory, it shouldn't be needed since LTO has all information. But what 
about current implementation. I checked relevant source files and can't 
get immediate clue. 

Thanks,
Bingfeng

Re: externally_visible and resoultion file

[Richard Guenther]

On Wed, May 26, 2010 at 5:53 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> Hi, Richard,
> With resolution file generated by GOLD (or I am going to hack gnu LD),  is
> externally_visible attribute still needed to annotate those symbols accessed
> from non-LTO objects when compiling with -fwhole-program.

Yes it is.  We do not parse the complete resolution file but only the
entries we'll end up reading .o files with LTO information for.

> In theory, it shouldn't be needed since LTO has all information. But what
> about current implementation. I checked relevant source files and can't
> get immediate clue.

The current implementation has no idea of the external references
(though it's probably not too hard to implement).

Richard.

Re: externally_visible and resoultion file

[Jan Hubicka]

> On Wed, May 26, 2010 at 5:53 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> > Hi, Richard,
> > With resolution file generated by GOLD (or I am going to hack gnu LD),  is
> > externally_visible attribute still needed to annotate those symbols accessed
> > from non-LTO objects when compiling with -fwhole-program.
> 
> Yes it is.  We do not parse the complete resolution file but only the
> entries we'll end up reading .o files with LTO information for.
> 
> > In theory, it shouldn't be needed since LTO has all information. But what
> > about current implementation. I checked relevant source files and can't
> > get immediate clue.
> 
> The current implementation has no idea of the external references
> (though it's probably not too hard to implement).

Yep, this is also one of my TODO list items...

Honza
> 
> Richard.

RE: externally_visible and resoultion file

[Bingfeng Mei]

Hi, 
Sorry for coming back to this issue after a while. I am still puzzled
by this. The following are two test files:

a.c

#include <string.h>
#include <stdio.h>
extern int foo(int);
void bar() 
{
  printf("bar\n");
}  
extern int src[], dst[];
int vvvvvv;
int main()
{ 
  int ret;
  vvvvvv = 12; 
  ret = foo(20);
  memcpy(dst, src, 100);
  return ret + 3;
}  

b.c 
#include <stdio.h>
int src[100];
int dst[100];
extern int vvvvvv;
extern void bar();
int foo(int c)
{
  printf("Hello world: %d\n", c);
  bar();
  return 1000 + vvvvvv;
}

I compiled with following steps
~/work/install-x86/bin/gcc -flto a.c -O2 -c
~/work/install-x86/bin/gcc b.c -O2 -c
ar cru libb.a b.o
~/work/install-x86/bin/gcc -flto a.o -L. -lb -O2 -fuse-linker-plugin -o f -fwhole-program -save-temps

Since you both mentioned that resolution file is not used to replace externally_visible attribute yet, 
I expect there will be link errors regarding both vvvvvv and bar. Somehow, GCC doesn't complain at all.
However, looking at generated assembly code, GCC produces wrong code for calling bar, and vvvvvv is 
linked correctly though. 

0000000000400510 <foo>:
  400510:	48 83 ec 08          	sub    $0x8,%rsp
  400514:	89 fe                	mov    %edi,%esi
  400516:	31 c0                	xor    %eax,%eax
  400518:	bf 1c 06 40 00       	mov    $0x40061c,%edi
  40051d:	e8 56 01 00 00       	callq  400678 <printf@plt>
  400522:	31 c0                	xor    %eax,%eax
  400524:	e8 d7 fa bf ff       	callq  0 <__fini_array_end>    <--------  should call bar. 
  400529:	8b 05 a1 17 00 00    	mov    0x17a1(%rip),%eax        # 401cd0 <vvvvvv>
  40052f:	48 83 c4 08          	add    $0x8,%rsp
  400533:	05 e8 03 00 00       	add    $0x3e8,%eax
  400538:	c3                   	retq   
  400539:	0f 1f 80 00 00 00 00 	nopl   0x0(%rax)

Is this a bug? GCC should issue warning/error here. Why is vvvvvv linked correctly?
Shouldn't it be treated as static with -fwhile-program and without externally_visible? 

My trunk is 160104 by the way.

Thanks,
Bingfeng

> -----Original Message-----
> From: Jan Hubicka [mailto:hubicka@ucw.cz]
> Sent: 27 May 2010 09:04
> To: Richard Guenther
> Cc: Bingfeng Mei; gcc@gcc.gnu.org
> Subject: Re: externally_visible and resoultion file
> 
> > On Wed, May 26, 2010 at 5:53 PM, Bingfeng Mei <bmei@broadcom.com>
> wrote:
> > > Hi, Richard,
> > > With resolution file generated by GOLD (or I am going to hack gnu
> LD),  is
> > > externally_visible attribute still needed to annotate those symbols
> accessed
> > > from non-LTO objects when compiling with -fwhole-program.
> >
> > Yes it is.  We do not parse the complete resolution file but only the
> > entries we'll end up reading .o files with LTO information for.
> >
> > > In theory, it shouldn't be needed since LTO has all information.
> But what
> > > about current implementation. I checked relevant source files and
> can't
> > > get immediate clue.
> >
> > The current implementation has no idea of the external references
> > (though it's probably not too hard to implement).
> 
> Yep, this is also one of my TODO list items...
> 
> Honza
> >
> > Richard.

Re: externally_visible and resoultion file

[Richard Guenther]

On Tue, Jun 8, 2010 at 3:01 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> Hi,
> Sorry for coming back to this issue after a while. I am still puzzled
> by this. The following are two test files:
>
> a.c
>
> #include <string.h>
> #include <stdio.h>
> extern int foo(int);
> void bar()
> {
>  printf("bar\n");
> }
> extern int src[], dst[];
> int vvvvvv;
> int main()
> {
>  int ret;
>  vvvvvv = 12;
>  ret = foo(20);
>  memcpy(dst, src, 100);
>  return ret + 3;
> }
>
> b.c
> #include <stdio.h>
> int src[100];
> int dst[100];
> extern int vvvvvv;
> extern void bar();
> int foo(int c)
> {
>  printf("Hello world: %d\n", c);
>  bar();
>  return 1000 + vvvvvv;
> }
>
> I compiled with following steps
> ~/work/install-x86/bin/gcc -flto a.c -O2 -c
> ~/work/install-x86/bin/gcc b.c -O2 -c
> ar cru libb.a b.o
> ~/work/install-x86/bin/gcc -flto a.o -L. -lb -O2 -fuse-linker-plugin -o f -fwhole-program -save-temps
>
> Since you both mentioned that resolution file is not used to replace externally_visible attribute yet,
> I expect there will be link errors regarding both vvvvvv and bar. Somehow, GCC doesn't complain at all.
> However, looking at generated assembly code, GCC produces wrong code for calling bar, and vvvvvv is
> linked correctly though.
>
> 0000000000400510 <foo>:
>  400510:       48 83 ec 08             sub    $0x8,%rsp
>  400514:       89 fe                   mov    %edi,%esi
>  400516:       31 c0                   xor    %eax,%eax
>  400518:       bf 1c 06 40 00          mov    $0x40061c,%edi
>  40051d:       e8 56 01 00 00          callq  400678 <printf@plt>
>  400522:       31 c0                   xor    %eax,%eax
>  400524:       e8 d7 fa bf ff          callq  0 <__fini_array_end>    <--------  should call bar.
>  400529:       8b 05 a1 17 00 00       mov    0x17a1(%rip),%eax        # 401cd0 <vvvvvv>
>  40052f:       48 83 c4 08             add    $0x8,%rsp
>  400533:       05 e8 03 00 00          add    $0x3e8,%eax
>  400538:       c3                      retq
>  400539:       0f 1f 80 00 00 00 00    nopl   0x0(%rax)
>
> Is this a bug? GCC should issue warning/error here. Why is vvvvvv linked correctly?
> Shouldn't it be treated as static with -fwhile-program and without externally_visible?

bar should be indeed eliminated which is what I see with not
using the linker plugin.

If you use the linker-plugin I see in the .optimized dump that
the were eliminated.  But it seems that gold picks them up
from the original objects which seem to be retained somehow.

I guess this is a bug in the linker-plugin.

Richard.

> My trunk is 160104 by the way.

Re: externally_visible and resoultion file

[Richard Guenther]

On Tue, Jun 8, 2010 at 4:15 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> Thanks. But why "vvvvv" is linked correctly here? Shouldn't it
> be treated as static with -fwhole-program?

Works for me without the linker plugin as well:

> gcc-4.5 -O2 -o t t1.o t2.o -flto -fwhole-program -B /abuild/rguenther/trunk-g/gcc
t2.o: In function `foo':
t2.c:(.text+0x15): undefined reference to `bar'
t2.c:(.text+0x1b): undefined reference to `vvvvvv'
collect2: ld returned 1 exit status

Richard.

RE: externally_visible and resoultion file

[Bingfeng Mei]

Thanks. But why "vvvvv" is linked correctly here? Shouldn't it 
be treated as static with -fwhole-program? 

Bingfeng

> -----Original Message-----
> From: Richard Guenther [mailto:richard.guenther@gmail.com]
> Sent: 08 June 2010 14:10
> To: Bingfeng Mei
> Cc: Jan Hubicka; gcc@gcc.gnu.org; Cary Coutant
> Subject: Re: externally_visible and resoultion file
> 
> On Tue, Jun 8, 2010 at 3:01 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> > Hi,
> > Sorry for coming back to this issue after a while. I am still puzzled
> > by this. The following are two test files:
> >
> > a.c
> >
> > #include <string.h>
> > #include <stdio.h>
> > extern int foo(int);
> > void bar()
> > {
> >  printf("bar\n");
> > }
> > extern int src[], dst[];
> > int vvvvvv;
> > int main()
> > {
> >  int ret;
> >  vvvvvv = 12;
> >  ret = foo(20);
> >  memcpy(dst, src, 100);
> >  return ret + 3;
> > }
> >
> > b.c
> > #include <stdio.h>
> > int src[100];
> > int dst[100];
> > extern int vvvvvv;
> > extern void bar();
> > int foo(int c)
> > {
> >  printf("Hello world: %d\n", c);
> >  bar();
> >  return 1000 + vvvvvv;
> > }
> >
> > I compiled with following steps
> > ~/work/install-x86/bin/gcc -flto a.c -O2 -c
> > ~/work/install-x86/bin/gcc b.c -O2 -c
> > ar cru libb.a b.o
> > ~/work/install-x86/bin/gcc -flto a.o -L. -lb -O2 -fuse-linker-plugin
> -o f -fwhole-program -save-temps
> >
> > Since you both mentioned that resolution file is not used to replace
> externally_visible attribute yet,
> > I expect there will be link errors regarding both vvvvvv and bar.
> Somehow, GCC doesn't complain at all.
> > However, looking at generated assembly code, GCC produces wrong code
> for calling bar, and vvvvvv is
> > linked correctly though.
> >
> > 0000000000400510 <foo>:
> >  400510:       48 83 ec 08             sub    $0x8,%rsp
> >  400514:       89 fe                   mov    %edi,%esi
> >  400516:       31 c0                   xor    %eax,%eax
> >  400518:       bf 1c 06 40 00          mov    $0x40061c,%edi
> >  40051d:       e8 56 01 00 00          callq  400678 <printf@plt>
> >  400522:       31 c0                   xor    %eax,%eax
> >  400524:       e8 d7 fa bf ff          callq  0
> <__fini_array_end>    <--------  should call bar.
> >  400529:       8b 05 a1 17 00
> 00       mov    0x17a1(%rip),%eax        # 401cd0 <vvvvvv>
> >  40052f:       48 83 c4 08             add    $0x8,%rsp
> >  400533:       05 e8 03 00 00          add    $0x3e8,%eax
> >  400538:       c3                      retq
> >  400539:       0f 1f 80 00 00 00 00    nopl   0x0(%rax)
> >
> > Is this a bug? GCC should issue warning/error here. Why is vvvvvv
> linked correctly?
> > Shouldn't it be treated as static with -fwhile-program and without
> externally_visible?
> 
> bar should be indeed eliminated which is what I see with not
> using the linker plugin.
> 
> If you use the linker-plugin I see in the .optimized dump that
> the were eliminated.  But it seems that gold picks them up
> from the original objects which seem to be retained somehow.
> 
> I guess this is a bug in the linker-plugin.
> 
> Richard.
> 
> > My trunk is 160104 by the way.

RE: externally_visible and resoultion file

[Bingfeng Mei]

Yes, this is also what I saw without plugin. I just wonder why "vvvvv"
is linked with plugin if resolution file is not used to eliminate need
of externally_visible attribute here. 

Thanks,
Bingfeng

> -----Original Message-----
> From: Richard Guenther [mailto:richard.guenther@gmail.com]
> Sent: 08 June 2010 15:18
> To: Bingfeng Mei
> Cc: Jan Hubicka; gcc@gcc.gnu.org; Cary Coutant
> Subject: Re: externally_visible and resoultion file
> 
> On Tue, Jun 8, 2010 at 4:15 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> > Thanks. But why "vvvvv" is linked correctly here? Shouldn't it
> > be treated as static with -fwhole-program?
> 
> Works for me without the linker plugin as well:
> 
> > gcc-4.5 -O2 -o t t1.o t2.o -flto -fwhole-program -B
> /abuild/rguenther/trunk-g/gcc
> t2.o: In function `foo':
> t2.c:(.text+0x15): undefined reference to `bar'
> t2.c:(.text+0x1b): undefined reference to `vvvvvv'
> collect2: ld returned 1 exit status
> 
> Richard.

Re: externally_visible and resoultion file

[Richard Guenther]

On Tue, Jun 8, 2010 at 4:22 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
> Yes, this is also what I saw without plugin. I just wonder why "vvvvv"
> is linked with plugin if resolution file is not used to eliminate need
> of externally_visible attribute here.

Probably because of the same linker-plugin bug that causes bar
to be resolved.

Richard.

> Thanks,
> Bingfeng
>
>> -----Original Message-----
>> From: Richard Guenther [mailto:richard.guenther@gmail.com]
>> Sent: 08 June 2010 15:18
>> To: Bingfeng Mei
>> Cc: Jan Hubicka; gcc@gcc.gnu.org; Cary Coutant
>> Subject: Re: externally_visible and resoultion file
>>
>> On Tue, Jun 8, 2010 at 4:15 PM, Bingfeng Mei <bmei@broadcom.com> wrote:
>> > Thanks. But why "vvvvv" is linked correctly here? Shouldn't it
>> > be treated as static with -fwhole-program?
>>
>> Works for me without the linker plugin as well:
>>
>> > gcc-4.5 -O2 -o t t1.o t2.o -flto -fwhole-program -B
>> /abuild/rguenther/trunk-g/gcc
>> t2.o: In function `foo':
>> t2.c:(.text+0x15): undefined reference to `bar'
>> t2.c:(.text+0x1b): undefined reference to `vvvvvv'
>> collect2: ld returned 1 exit status
>>
>> Richard.
>
>
>

Re: externally_visible and resoultion file

[Cary Coutant]

>> Yes, this is also what I saw without plugin. I just wonder why "vvvvv"
>> is linked with plugin if resolution file is not used to eliminate need
>> of externally_visible attribute here.
>
> Probably because of the same linker-plugin bug that causes bar
> to be resolved.

Just to make sure I understand the problem:

- The IR file for a.c contains definitions for vvvvv and bar.
- The linker identifies that both symbols are referenced from outside
the LTO world (PREVAILING_DEF rather than PREVAILING_DEF_IRONLY), but
gcc isn't (yet) reading that info from the resolution file.
- WPA eliminates bar() and makes vvvvv static in the replacement object file.
- There are still references to those symbols in b.o, which was
compiled outside LTO.
- The linker should be complaining about undefined symbols in both
cases, but isn't (perhaps because it's still seeing defs left over
from the IR files). The symbol bar has a value of 0, while the
reference to vvvvv seems to have the right address.

Is that about right? What you're expecting is a link-time error
reporting both bar and vvvvv as undefined, right?

-cary

Re: externally_visible and resoultion file

[Richard Guenther]

On Wed, Jun 9, 2010 at 7:43 PM, Cary Coutant <ccoutant@google.com> wrote:
>>> Yes, this is also what I saw without plugin. I just wonder why "vvvvv"
>>> is linked with plugin if resolution file is not used to eliminate need
>>> of externally_visible attribute here.
>>
>> Probably because of the same linker-plugin bug that causes bar
>> to be resolved.
>
> Just to make sure I understand the problem:
>
> - The IR file for a.c contains definitions for vvvvv and bar.
> - The linker identifies that both symbols are referenced from outside
> the LTO world (PREVAILING_DEF rather than PREVAILING_DEF_IRONLY), but
> gcc isn't (yet) reading that info from the resolution file.
> - WPA eliminates bar() and makes vvvvv static in the replacement object file.
> - There are still references to those symbols in b.o, which was
> compiled outside LTO.
> - The linker should be complaining about undefined symbols in both
> cases, but isn't (perhaps because it's still seeing defs left over
> from the IR files). The symbol bar has a value of 0, while the
> reference to vvvvv seems to have the right address.
>
> Is that about right? What you're expecting is a link-time error
> reporting both bar and vvvvv as undefined, right?

That's correct.

Richard.

> -cary
>

RE: externally_visible and resoultion file

[Bingfeng Mei]

> -----Original Message-----
> From: Cary Coutant [mailto:ccoutant@google.com]
> Sent: 09 June 2010 18:43
> To: Richard Guenther
> Cc: Bingfeng Mei; Jan Hubicka; gcc@gcc.gnu.org
> Subject: Re: externally_visible and resoultion file
> 
> >> Yes, this is also what I saw without plugin. I just wonder why
> "vvvvv"
> >> is linked with plugin if resolution file is not used to eliminate
> need
> >> of externally_visible attribute here.
> >
> > Probably because of the same linker-plugin bug that causes bar
> > to be resolved.
> 
> Just to make sure I understand the problem:
> 
> - The IR file for a.c contains definitions for vvvvv and bar.
> - The linker identifies that both symbols are referenced from outside
> the LTO world (PREVAILING_DEF rather than PREVAILING_DEF_IRONLY), but
> gcc isn't (yet) reading that info from the resolution file.

> - WPA eliminates bar() and makes vvvvv static in the replacement object
> file.
> - There are still references to those symbols in b.o, which was
> compiled outside LTO.
> - The linker should be complaining about undefined symbols in both
> cases, but isn't (perhaps because it's still seeing defs left over
> from the IR files). The symbol bar has a value of 0, while the
> reference to vvvvv seems to have the right address.
> 
> Is that about right? What you're expecting is a link-time error
> reporting both bar and vvvvv as undefined, right?
> 
> -cary

Yes, I expect link-time errors of undefined reference for both
bar and vvvvvv. Instead, bar is linked to a bogus one (address 0)
and vvvvvv is linked correctly.