ld tests about protected visibility

ld tests about protected visibility

[CHIGOT, CLEMENT]

Hi everyone, 

I'm currently implementing visibility for XCOFF format and while doing 
so I'm running tests inside "ld-elfvsb" testsuite. 
However, I'm not sure all the tests are doing what they are supposed to do. 
Especially tests about the protected visibility, I have exactly the same result 
whether or not ".protected" instructions are being added or not. 
   
With ".protected" in main.c, 
$ readelf -s tmpdir/mainnp.o | grep shared_
    42: 0000000000000000     4 OBJECT  GLOBAL PROTECTED    2 shared_data
    43: 0000000000000020    10 FUNC    GLOBAL PROTECTED    1 shared_func
$ gcc ... -o tmpdir/vp ... tmpdir/mainnp.o tmpdir/vp.so
$ ./tmpdir/vp
...
$ echo $?
0


Without ".protected": 
$ readelf -s tmpdir/mainnp.o | grep shared_
    44: 0000000000000020    10 FUNC    GLOBAL DEFAULT    1 shared_func
    80: 0000000000000000     4 OBJECT  GLOBAL DEFAULT    2 shared_data
$ gcc ... -o tmpdir/vp ... tmpdir/mainnp.o tmpdir/vp.so
$ ./tmpdir/vp
...
$ echo $?
0


As I'm not fully aware of all the subtleties of ELF and visibility, does anyone know 
if this is intended and if it's not what it should be ? 

Thanks, 
Clément

Re: ld tests about protected visibility

[Michael Matz]

On Thu, 25 Nov 2021, CHIGOT, CLEMENT via Binutils wrote:

> Hi everyone, 
> 
> I'm currently implementing visibility for XCOFF format and while doing 
> so I'm running tests inside "ld-elfvsb" testsuite. 

As you probably noticed those tests test many different things with the
same sources, depending on how they are compiled.  So, that makes it all a 
bit non-obvious.  In particular you aren't providing enough information to 
know ...

> With ".protected" in main.c, 

... how main.c ...

> $ readelf -s tmpdir/mainnp.o | grep shared_
>     42: 0000000000000000     4 OBJECT  GLOBAL PROTECTED    2 shared_data
>     43: 0000000000000020    10 FUNC    GLOBAL PROTECTED    1 shared_func
> $ gcc ... -o tmpdir/vp ... tmpdir/mainnp.o tmpdir/vp.so

... and especially how vp.so was compiled for the cases you are interested 
in.  The protected tests about shared_func and shared_data are testing the 
following scenarios (I'll only state shared_data, shared_func is similar, 
except it can also be called):

(a) protected shared_data in executable, _no_ shared_data in shared lib
(b) protected shared_data in executable, global shared_data in shared lib

(In particular, the shared_data in executable is always protected in these 
tests, so your experiment (removing protected in main.c) is not one of the 
tests).  In both cases all references to shared_data should resolve to the 
one in the executable.  That is tested, and the return code is 0 if that 
works as expected.  In your cases it does work as expected, the definition 
in the executable prevails.

If you remove .protected from main.c then the shared_data in executable 
will be global and trivially be the one that everything is resolved to 
(because in ELF the executable is first in all symbol lookups).  It's only 
when the symbol in the exe is protected that something interesting happens 
(namely one could be of the opinion that the definition of shared_data in 
the shared lib in case (b) should prevail, but that is explicitely not so, 
which is what's tested).

> As I'm not fully aware of all the subtleties of ELF and visibility, does 
> anyone know if this is intended and if it's not what it should be ?

So, yes, removing .protected from main.c should have the effect you see; 
but as said, it's not an interesting test as no protected visibility would 
be involved anymore, just normal default ELF lookup rules.

In case it helps: protected symbols are conceptually fairly simple: a 
protected symbol in component (exe or shared lib) A is visible from other 
components B (i.e. it's exported).  References to that symbol from within 
component A are resolved to the symbol in component A, no matter if other 
components also define that symbol (this is in difference to normal ELF 
lookup rules, which state that the first symbol found in a global breadth 
first search of all components is the one that all references resolve to).


Ciao,
Michael.

Re: ld tests about protected visibility

[CHIGOT, CLEMENT]

>> Hi everyone,
>>
>> I'm currently implementing visibility for XCOFF format and while doing
>> so I'm running tests inside "ld-elfvsb" testsuite.
>
> As you probably noticed those tests test many different things with the
> same sources, depending on how they are compiled.  So, that makes it all a
> bit non-obvious.  In particular you aren't providing enough information to
> know ...

I didn't want to add too much details in order to be as concise as possible.
I said that I was speaking about the test called "protected". But it wasn't
that obvious actually.
Anyway, I should have explained a bit more sorry.

>> With ".protected" in main.c,

>... how main.c ...

>> $ readelf -s tmpdir/mainnp.o | grep shared_
>>     42: 0000000000000000     4 OBJECT  GLOBAL PROTECTED    2 shared_data
>>     43: 0000000000000020    10 FUNC    GLOBAL PROTECTED    1 shared_func
>> $ gcc ... -o tmpdir/vp ... tmpdir/mainnp.o tmpdir/vp.so

>... and especially how vp.so was compiled for the cases you are interested
>in.  The protected tests about shared_func and shared_data are testing the
>following scenarios (I'll only state shared_data, shared_func is similar,
>except it can also be called):
>
>(a) protected shared_data in executable, _no_ shared_data in shared lib
>(b) protected shared_data in executable, global shared_data in shared lib

As I said, I was following the test named "protected" so vp.so is simply
sh1.c with -DPROTECTED -DSHARED.
Thus, shared_data is defined in the shared lib.  But once again, I should
have mentioned it.

>So, yes, removing .protected from main.c should have the effect you see;
>but as said, it's not an interesting test as no protected visibility would
>be involved anymore, just normal default ELF lookup rules.

Ok, I understand a bit more. I just find it really weird that the test has
the same result whether or not ".protected" is there. Because, that means
that if protected for whatever reasons stops working and is kind of ignored
by the linker, it won't be detected right ?

>In case it helps: protected symbols are conceptually fairly simple: a
>protected symbol in component (exe or shared lib) A is visible from other
>components B (i.e. it's exported).  References to that symbol from within
>component A are resolved to the symbol in component A, no matter if other
>components also define that symbol (this is in difference to normal ELF
>lookup rules, which state that the first symbol found in a global breadth
>first search of all components is the one that all references resolve to).

So if I understand correctly, if ".protected" is removed from our testcase and
vp.so is still exporting shared_data, thus, linking with vp.so before mainnp.o
would mean that the data_shared referenced in mainnp.o would be the one
within the shared lib and not the one from the main (because vp.so is
included before mainnp.o) ? While with ".protected" whatever the order is,
"data_shared" references in mainnp.o would always target the one defined
in mainnp.o ?



Thanks,
Clément

Re: ld tests about protected visibility

[Michael Matz]

Hello,

On Thu, 25 Nov 2021, CHIGOT, CLEMENT wrote:

> >So, yes, removing .protected from main.c should have the effect you see;
> >but as said, it's not an interesting test as no protected visibility would
> >be involved anymore, just normal default ELF lookup rules.
> 
> Ok, I understand a bit more. I just find it really weird that the test has
> the same result whether or not ".protected" is there. Because, that means
> that if protected for whatever reasons stops working and is kind of ignored
> by the linker, it won't be detected right ?

Not by _this_ test, correct.  But that is not the point of this test.  
Simply testing that the .protected directive cause the symbols to be of 
PROTECTED visibility is the point of other tests (define.s and 
protected0.d in this directory).  _This_ test checks if references from 
the shared library are resolved to the protected symbol in the executable, 
no matter if the shared lib does or does not contain its own definition of 
the symbol.  I.e. it tests that the presence of protected in the 
executable doesn't accidentally lead to that symbol not being used by the 
shared lib anymore.  See below for the other direction.

> >In case it helps: protected symbols are conceptually fairly simple: a
> >protected symbol in component (exe or shared lib) A is visible from other
> >components B (i.e. it's exported).  References to that symbol from within
> >component A are resolved to the symbol in component A, no matter if other
> >components also define that symbol (this is in difference to normal ELF
> >lookup rules, which state that the first symbol found in a global breadth
> >first search of all components is the one that all references resolve to).
> 
> So if I understand correctly, if ".protected" is removed from our testcase and
> vp.so is still exporting shared_data, thus, linking with vp.so before mainnp.o
> would mean that the data_shared referenced in mainnp.o would be the one
> within the shared lib and not the one from the main (because vp.so is
> included before mainnp.o) ?

No.  There are runtime and link editing time resolving steps.  In this 
case we are interested in the effect of both, and for that the order of 
the cmdline arguments to ld don't matter here: shared libraries (like 
vp.so) implicitely loaded by executables (vp, link-edited from mainnp.o) 
always come after the executable in symbol lookups.  So, if both lib and 
exe contain global defines then the only one found by lookups from any 
component would be the one in the exe.

But, a similar case can be constructed where there's a difference in 
behaviour when protected is or isn't used: the executable defines a GLOBAL 
'foobar', the shared library defines a PROTECTED foobar (i.e. opposite of 
the tests related to "shared_data").  Now references from within the 
executable would be resolved to the exe-foobar (by normal lookup rules), 
and references from within the shared lib would resolve to lib-foobar (by 
protected lookup rules).

This latter scenario is tested by the 'visibility_var' variables with 
certain defines.

If the lib would define a GLOBAL foobar, then references from within the 
lib would also resolve to exe-foobar.  (In effect lib-foobar would be 
globally shadowed by exe-foobar).

> While with ".protected" whatever the order is, "data_shared" references 
> in mainnp.o would always target the one defined in mainnp.o ?

This is correct, though.  No matter into which component mainnp.o would be 
link-edited (shared lib or exe) references from it to "data_shared" would 
always resolve locally in that component.


Ciao,
Michael.

Re: ld tests about protected visibility

[CHIGOT, CLEMENT]

Hi Michael

Thanks a lot for all the explanations. 
I'm not yet sure if it can be applied to XCOFF, which 
tries to resolve everything at link time. 

Thanks anyway !  

Clément