Preventing preemption of 'protected' symbols in GNU ld 2.26

Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:

https://bugs.swift.org/browse/SR-1023

This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Wed, Mar 23, 2016 at 5:00 PM, Joe Groff <jgroff@apple.com> wrote:
> Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:
>
> https://bugs.swift.org/browse/SR-1023
>
> This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!
>

What you want is not protected symbol, which is expensive to lookup
at run-time.  You need:

  -z nocopyreloc              Don't create copy relocs

But it requires that everything must be compiled with -fPIC, not even
-fPIE, for both executable and shared library.  Otherwise, you will
get dynamic relocations in text section.

Swift should do something like -fPIC-data, to access external data via
PIC.  Then you can pass

 -z nocopyreloc              Don't create copy relocs

to linker.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 23, 2016, at 5:45 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Wed, Mar 23, 2016 at 5:00 PM, Joe Groff <jgroff@apple.com> wrote:
>> Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:
>> 
>> https://bugs.swift.org/browse/SR-1023
>> 
>> This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!
>> 
> 
> What you want is not protected symbol, which is expensive to lookup
> at run-time.  You need:
> 
>  -z nocopyreloc              Don't create copy relocs
> 
> But it requires that everything must be compiled with -fPIC, not even
> -fPIE, for both executable and shared library.  Otherwise, you will
> get dynamic relocations in text section.
> 
> Swift should do something like -fPIC-data, to access external data via
> PIC.  Then you can pass
> 
> -z nocopyreloc              Don't create copy relocs
> 
> to linker.

Thanks H.J. Is it possible to specify this in the object file, at the symbol level, or does "nocopyreloc" have to be specified by linker flags?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Wed, Mar 23, 2016 at 5:52 PM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 23, 2016, at 5:45 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Wed, Mar 23, 2016 at 5:00 PM, Joe Groff <jgroff@apple.com> wrote:
>>> Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:
>>>
>>> https://bugs.swift.org/browse/SR-1023
>>>
>>> This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!
>>>
>>
>> What you want is not protected symbol, which is expensive to lookup
>> at run-time.  You need:
>>
>>  -z nocopyreloc              Don't create copy relocs
>>
>> But it requires that everything must be compiled with -fPIC, not even
>> -fPIE, for both executable and shared library.  Otherwise, you will
>> get dynamic relocations in text section.
>>
>> Swift should do something like -fPIC-data, to access external data via
>> PIC.  Then you can pass
>>
>> -z nocopyreloc              Don't create copy relocs
>>
>> to linker.
>
> Thanks H.J. Is it possible to specify this in the object file, at the symbol level, or does "nocopyreloc" have to be specified by linker flags?
>

Not at this moment.  I do have a proposal to specify something like
that in object files.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> Hi everyone. On the Swift project, we're getting bug reports that
> people are no longer able to link Swift programs using binutils
> 2.26:
>
> https://bugs.swift.org/browse/SR-1023
>
> This appears to be due to an intentional behavior change in
> commit
> https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit
> ;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected
> symbols to be the targets of copy relocations. This breaks our
> intended use of "protected" in Swift, where we really want to be
> able to assume that the relative addresses of our symbols within
> a single executable or .so won't be preempted or moved. This lets
> us emit position-independent constant reflection metadata that
> doesn't need load-time relocations, saving us some startup time
> and dirty pages. I wanted to know if there's a supported way to
> express to the linker that "this symbol is protected, and also
> can never be copied". I'd like to avoid globally setting
> -Bsymbolic since I wouldn't want to impact non-Swift .o files in
> a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent
> object files from trying to use copy relocations to our symbols
> and breaking our assumptions about their address; I'd like to
> know if it's possible to prevent that too. Thanks for any help
> you can give!

Your intended use of "protected" actually matches the actual intended
use, and I believe the change you point to was a mistake. The gABI
says that STV_PROTECTED must resolve to a definition in the same
module:

"A symbol defined in the current component is protected if it is
visible in other components but not preemptable, meaning that any
reference to such a symbol from within the defining component must be
resolved to the definition in that component, even if there is a
definition in another component that would preempt by the default
rules."

That's pretty clear, and its intent was to allow compilers to generate
more efficient code to access protected variables, knowing that they
can't be pre-empted. Allowing COPY relocations for protected symbols
breaks this guarantee, and is a bug. In fact, I'm about to fix just
such a bug in gold:

   https://sourceware.org/bugzilla/show_bug.cgi?id=19823

Worse, GCC 5 and 6 no longer generate direct references to protected
symbols, so the whole point of STV_PROTECTED (at least as it applies
to data symbols) is currently broken in the Gnu toolchain.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 8:01 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> Hi everyone. On the Swift project, we're getting bug reports that
>> people are no longer able to link Swift programs using binutils
>> 2.26:
>>
>> https://bugs.swift.org/browse/SR-1023
>>
>> This appears to be due to an intentional behavior change in
>> commit
>> https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit
>> ;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected
>> symbols to be the targets of copy relocations. This breaks our
>> intended use of "protected" in Swift, where we really want to be
>> able to assume that the relative addresses of our symbols within
>> a single executable or .so won't be preempted or moved. This lets
>> us emit position-independent constant reflection metadata that
>> doesn't need load-time relocations, saving us some startup time
>> and dirty pages. I wanted to know if there's a supported way to
>> express to the linker that "this symbol is protected, and also
>> can never be copied". I'd like to avoid globally setting
>> -Bsymbolic since I wouldn't want to impact non-Swift .o files in
>> a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent
>> object files from trying to use copy relocations to our symbols
>> and breaking our assumptions about their address; I'd like to
>> know if it's possible to prevent that too. Thanks for any help
>> you can give!
>
> Your intended use of "protected" actually matches the actual intended
> use, and I believe the change you point to was a mistake. The gABI
> says that STV_PROTECTED must resolve to a definition in the same
> module:
>
> "A symbol defined in the current component is protected if it is
> visible in other components but not preemptable, meaning that any
> reference to such a symbol from within the defining component must be
> resolved to the definition in that component, even if there is a
> definition in another component that would preempt by the default
> rules."
>
> That's pretty clear, and its intent was to allow compilers to generate
> more efficient code to access protected variables, knowing that they
> can't be pre-empted. Allowing COPY relocations for protected symbols
> breaks this guarantee, and is a bug. In fact, I'm about to fix just
> such a bug in gold:
>
>    https://sourceware.org/bugzilla/show_bug.cgi?id=19823
>
> Worse, GCC 5 and 6 no longer generate direct references to protected
> symbols, so the whole point of STV_PROTECTED (at least as it applies
> to data symbols) is currently broken in the Gnu toolchain.

I disagree.  Protected means that it won't be preempted.  It
doesn't mean the address of protected symbol will be local.
With copy relocation, the run-time address of the protected
symbol can be in executable.  There are a couple run-time
tests in glibc to verify it.  See:

https://sourceware.org/bugzilla/show_bug.cgi?id=17711


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> I disagree.  Protected means that it won't be preempted.  It
> doesn't mean the address of protected symbol will be local.
> With copy relocation, the run-time address of the protected
> symbol can be in executable.  There are a couple run-time
> tests in glibc to verify it.  See:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=17711

No. You can't just redefine away the meaning of "must resolve to a
definition in that component." A COPY relocation makes a new
definition in the main program and that new definition pre-empts the
one in the shared library. That breaks everything that the rule in the
gABI was designed to allow.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 9:06 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> I disagree.  Protected means that it won't be preempted.  It
>> doesn't mean the address of protected symbol will be local.
>> With copy relocation, the run-time address of the protected
>> symbol can be in executable.  There are a couple run-time
>> tests in glibc to verify it.  See:
>>
>> https://sourceware.org/bugzilla/show_bug.cgi?id=17711
>
> No. You can't just redefine away the meaning of "must resolve to a
> definition in that component." A COPY relocation makes a new
> definition in the main program and that new definition pre-empts the
> one in the shared library. That breaks everything that the rule in the
> gABI was designed to allow.
>

What you are proposing is to disallow copy relocation against
protected symbol.  Am I correct?


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> What you are proposing is to disallow copy relocation against
> protected symbol.  Am I correct?

Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
should have been resolved as "working as intended", and that your
patches for GCC PR 65248 should never have been applied (curiously, I
can't find any actual approval for the GCC patch). I'm sorry I missed
those conversations -- I would have raised bloody hell at the time.
I'm surprised that no one else did.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 9:56 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> What you are proposing is to disallow copy relocation against
>> protected symbol.  Am I correct?
>
> Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
> should have been resolved as "working as intended", and that your
> patches for GCC PR 65248 should never have been applied (curiously, I
> can't find any actual approval for the GCC patch). I'm sorry I missed
> those conversations -- I would have raised bloody hell at the time.
> I'm surprised that no one else did.
>

What relocation do you propose to access external protected
symbol on x86 for non-PIC code?

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 24, 2016, at 10:04 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Thu, Mar 24, 2016 at 9:56 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>> What you are proposing is to disallow copy relocation against
>>> protected symbol.  Am I correct?
>> 
>> Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
>> should have been resolved as "working as intended", and that your
>> patches for GCC PR 65248 should never have been applied (curiously, I
>> can't find any actual approval for the GCC patch). I'm sorry I missed
>> those conversations -- I would have raised bloody hell at the time.
>> I'm surprised that no one else did.
>> 
> 
> What relocation do you propose to access external protected
> symbol on x86 for non-PIC code?

Non-PIC code can still use a GOT, can't it?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 10:06 AM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 24, 2016, at 10:04 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Thu, Mar 24, 2016 at 9:56 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> What you are proposing is to disallow copy relocation against
>>>> protected symbol.  Am I correct?
>>>
>>> Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
>>> should have been resolved as "working as intended", and that your
>>> patches for GCC PR 65248 should never have been applied (curiously, I
>>> can't find any actual approval for the GCC patch). I'm sorry I missed
>>> those conversations -- I would have raised bloody hell at the time.
>>> I'm surprised that no one else did.
>>>
>>
>> What relocation do you propose to access external protected
>> symbol on x86 for non-PIC code?
>
> Non-PIC code can still use a GOT, can't it?

Yes.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>>> What relocation do you propose to access external protected
>>> symbol on x86 for non-PIC code?
>>
>> Non-PIC code can still use a GOT, can't it?
>
> Yes.

Right.

Some additional thoughts:

- This has nothing to do with PIE. Non-PIC-non-PIE code has been
living with this restriction for at least 18 years (since visiblity
was introduced into the gABI).

- This has nothing to do with Ulrich's diatribe against protected
visibility, which applies only to function symbols (and really only to
one platform, due to how function pointer comparison works).

- You don't need to go full-on PIC to reference a protected data
symbol in a shared library. You can always statically initialize a
pointer variable, and go indirect through that ("poor-man's PIC").

- We could add support for __attribute__((dllimport)) or
__declspec(dllimport) when declaring an external variable that is
expected to be defined in a shared library (ideally, that attribute
ought to be placed in the library's public interface). The compiler
can generate a PIC-style reference for that variable without
penalizing all the other variables in the main program. This is how
HP-UX compilers work on PA and Itanium (using #pragma external).

- Compilers could also generate tentative PIC-style references, with
sufficient relocations to allow the linker to convert the indirect
reference to a direct reference when possible (changing the second
load into a nop or copy). HP-UX also does this.

- Indirect references from PIC code to a protected symbol penalize the
common case (referencing a symbol within its own module) to support
the uncommon case, while introducing the nasty side effects of a COPY
relocation.

- COPY relocations are evil. They bind an application to a specific
version of a shared library's ABI, and introduce a hidden startup
cost. If we're going to make any changes, we should be moving towards
elimination of COPY relocations, rather than disabling features that
were designed to improve performance.

- Arguing that protected means that the definition is in the same
module but its address might be external is absurd. The *only* reason
for the gABI to make that guarantee is so the compilers can optimize
the code based on the knowledge that the symbol can't be pre-empted.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Rafael Espíndola]

On 24 March 2016 at 14:31, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> What relocation do you propose to access external protected
>>>> symbol on x86 for non-PIC code?
>>>
>>> Non-PIC code can still use a GOT, can't it?
>>
>> Yes.
>
> Right.
>
> Some additional thoughts:
>
> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
> living with this restriction for at least 18 years (since visiblity
> was introduced into the gABI).
>
> - This has nothing to do with Ulrich's diatribe against protected
> visibility, which applies only to function symbols (and really only to
> one platform, due to how function pointer comparison works).


The old powerpc ABI?

> - You don't need to go full-on PIC to reference a protected data
> symbol in a shared library. You can always statically initialize a
> pointer variable, and go indirect through that ("poor-man's PIC").
>
> - We could add support for __attribute__((dllimport)) or
> __declspec(dllimport) when declaring an external variable that is
> expected to be defined in a shared library (ideally, that attribute
> ought to be placed in the library's public interface). The compiler
> can generate a PIC-style reference for that variable without
> penalizing all the other variables in the main program. This is how
> HP-UX compilers work on PA and Itanium (using #pragma external).
>
> - Compilers could also generate tentative PIC-style references, with
> sufficient relocations to allow the linker to convert the indirect
> reference to a direct reference when possible (changing the second
> load into a nop or copy). HP-UX also does this.
>
> - Indirect references from PIC code to a protected symbol penalize the
> common case (referencing a symbol within its own module) to support
> the uncommon case, while introducing the nasty side effects of a COPY
> relocation.
>
> - COPY relocations are evil. They bind an application to a specific
> version of a shared library's ABI, and introduce a hidden startup
> cost. If we're going to make any changes, we should be moving towards
> elimination of COPY relocations, rather than disabling features that
> were designed to improve performance.
>
> - Arguing that protected means that the definition is in the same
> module but its address might be external is absurd. The *only* reason
> for the gABI to make that guarantee is so the compilers can optimize
> the code based on the knowledge that the symbol can't be pre-empted.

For what it is worth I strongly agree with Cary on these points.

I never quite understood the warning against using protected. Without
copy relocations or the PLT hacks, they are a really handy way to
produce visible symbols from dsos that are efficient to use in the DSO
itself.

It is unfortunate that without copy reloc and PLT hack they are not
easy to use from executables, but that can be fixed. Having the
compiler be conservative and the linker relax access would do it and
it is something I think we should try.

I will try to audit the llvm side to make sure protected in there
works as expected.

Cheers,
Rafael

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 11:31 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> What relocation do you propose to access external protected
>>>> symbol on x86 for non-PIC code?
>>>
>>> Non-PIC code can still use a GOT, can't it?
>>
>> Yes.
>
> Right.
>
> Some additional thoughts:
>
> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
> living with this restriction for at least 18 years (since visiblity
> was introduced into the gABI).
>
> - This has nothing to do with Ulrich's diatribe against protected
> visibility, which applies only to function symbols (and really only to
> one platform, due to how function pointer comparison works).

Copy relocation and protected symbol are mutually exclusive.
Since copy relocation is the part of x86 psABIs, it limits
protected symbol effectiveness on x86.  In glibc, we add a
local alias to symbol we want to protect and access the local
alias within glibc without GOT.

> - You don't need to go full-on PIC to reference a protected data
> symbol in a shared library. You can always statically initialize a
> pointer variable, and go indirect through that ("poor-man's PIC").
>
> - We could add support for __attribute__((dllimport)) or
> __declspec(dllimport) when declaring an external variable that is
> expected to be defined in a shared library (ideally, that attribute
> ought to be placed in the library's public interface). The compiler
> can generate a PIC-style reference for that variable without
> penalizing all the other variables in the main program. This is how
> HP-UX compilers work on PA and Itanium (using #pragma external).

This is a useful feature, similar to -fno-plt, like -fgot, but only on
selective symbols.

> - Compilers could also generate tentative PIC-style references, with
> sufficient relocations to allow the linker to convert the indirect
> reference to a direct reference when possible (changing the second
> load into a nop or copy). HP-UX also does this.

I extended the x86 psABIs with relaxable GOT relocations and
implemented the similar linker optimization in ld in binutils 2.26
and gold also implemented the subset of the linker optimization.

> - Indirect references from PIC code to a protected symbol penalize the
> common case (referencing a symbol within its own module) to support
> the uncommon case, while introducing the nasty side effects of a COPY
> relocation.
>
> - COPY relocations are evil. They bind an application to a specific
> version of a shared library's ABI, and introduce a hidden startup
> cost. If we're going to make any changes, we should be moving towards
> elimination of COPY relocations, rather than disabling features that
> were designed to improve performance.

Copy relocation can improve performance.  Google enabled copy
relocations for PIE in ld/gold and GCC to improve prefomance by
up to 5%:

https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html

It is in GCC 5.

> - Arguing that protected means that the definition is in the same
> module but its address might be external is absurd. The *only* reason
> for the gABI to make that guarantee is so the compilers can optimize
> the code based on the knowledge that the symbol can't be pre-empted.

One possible solution:

1. Compiler accesses protected symbols without GOT in PIC mode
and marks object files no-copy-relocation-against-protected-symbol.
2. Compiler accesses external symbols with GOT in non-PIC mode.
3. Linker marks shared object with no-copy-relocation-against-protected-symbol
if any input files have no-copy-relocation-against-protected-symbol
marker.
4. Linker optimizes out GOT access if symbol is defined locally
in executable.
5. ld.so checks no-copy-relocation-against-protected-symbol marker
on shared object to avoid copy relocation against protected symbol
at run-time.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 28, 2016, at 5:11 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Thu, Mar 24, 2016 at 11:31 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>>> What relocation do you propose to access external protected
>>>>> symbol on x86 for non-PIC code?
>>>> 
>>>> Non-PIC code can still use a GOT, can't it?
>>> 
>>> Yes.
>> 
>> Right.
>> 
>> Some additional thoughts:
>> 
>> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
>> living with this restriction for at least 18 years (since visiblity
>> was introduced into the gABI).
>> 
>> - This has nothing to do with Ulrich's diatribe against protected
>> visibility, which applies only to function symbols (and really only to
>> one platform, due to how function pointer comparison works).
> 
> Copy relocation and protected symbol are mutually exclusive.
> Since copy relocation is the part of x86 psABIs, it limits
> protected symbol effectiveness on x86.  In glibc, we add a
> local alias to symbol we want to protect and access the local
> alias within glibc without GOT.

I had thought about doing this as a workaround for us, but if I understand correctly, this could result in two canonical copies of the object in the process's memory. For what we're using protected symbols for in Swift, we really want only one canonical object, and would prefer that protected references from the executable go through a GOT, similar to how references to external symbols always work in Mach-O.

>> - You don't need to go full-on PIC to reference a protected data
>> symbol in a shared library. You can always statically initialize a
>> pointer variable, and go indirect through that ("poor-man's PIC").
>> 
>> - We could add support for __attribute__((dllimport)) or
>> __declspec(dllimport) when declaring an external variable that is
>> expected to be defined in a shared library (ideally, that attribute
>> ought to be placed in the library's public interface). The compiler
>> can generate a PIC-style reference for that variable without
>> penalizing all the other variables in the main program. This is how
>> HP-UX compilers work on PA and Itanium (using #pragma external).
> 
> This is a useful feature, similar to -fno-plt, like -fgot, but only on
> selective symbols.

"protected" is already selective, though. The code generator had to intentionally set protected visibility on symbols.

> 
>> - Compilers could also generate tentative PIC-style references, with
>> sufficient relocations to allow the linker to convert the indirect
>> reference to a direct reference when possible (changing the second
>> load into a nop or copy). HP-UX also does this.
> 
> I extended the x86 psABIs with relaxable GOT relocations and
> implemented the similar linker optimization in ld in binutils 2.26
> and gold also implemented the subset of the linker optimization.
> 
>> - Indirect references from PIC code to a protected symbol penalize the
>> common case (referencing a symbol within its own module) to support
>> the uncommon case, while introducing the nasty side effects of a COPY
>> relocation.
>> 
>> - COPY relocations are evil. They bind an application to a specific
>> version of a shared library's ABI, and introduce a hidden startup
>> cost. If we're going to make any changes, we should be moving towards
>> elimination of COPY relocations, rather than disabling features that
>> were designed to improve performance.
> 
> Copy relocation can improve performance.  Google enabled copy
> relocations for PIE in ld/gold and GCC to improve prefomance by
> up to 5%:
> 
> https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html
> 
> It is in GCC 5.

Did you look at what the costs were in startup time and dirty pages by using copy relocations? What do you do if the size of the definition changes in a new version of the library?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Mon, Mar 28, 2016 at 10:01 AM, Joe Groff <jgroff@apple.com> wrote:
>
> On Mar 28, 2016, at 5:11 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>
> On Thu, Mar 24, 2016 at 11:31 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>
> What relocation do you propose to access external protected
> symbol on x86 for non-PIC code?
>
>
> Non-PIC code can still use a GOT, can't it?
>
>
> Yes.
>
>
> Right.
>
> Some additional thoughts:
>
> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
> living with this restriction for at least 18 years (since visiblity
> was introduced into the gABI).
>
> - This has nothing to do with Ulrich's diatribe against protected
> visibility, which applies only to function symbols (and really only to
> one platform, due to how function pointer comparison works).
>
>
> Copy relocation and protected symbol are mutually exclusive.
> Since copy relocation is the part of x86 psABIs, it limits
> protected symbol effectiveness on x86.  In glibc, we add a
> local alias to symbol we want to protect and access the local
> alias within glibc without GOT.
>
>
> I had thought about doing this as a workaround for us, but if I understand
> correctly, this would result in two canonical copies of the object in the

That is correct.

> process's memory. For what we're using protected symbols for in Swift, we
> really want only one canonical object, and would prefer that protected
> references from the executable go through a GOT, similar to how references
> to external symbols always work in Mach-O.

Understood.

> - You don't need to go full-on PIC to reference a protected data
> symbol in a shared library. You can always statically initialize a
> pointer variable, and go indirect through that ("poor-man's PIC").
>
> - We could add support for __attribute__((dllimport)) or
> __declspec(dllimport) when declaring an external variable that is
> expected to be defined in a shared library (ideally, that attribute
> ought to be placed in the library's public interface). The compiler
> can generate a PIC-style reference for that variable without
> penalizing all the other variables in the main program. This is how
> HP-UX compilers work on PA and Itanium (using #pragma external).
>
>
> This is a useful feature, similar to -fno-plt, like -fgot, but only on
> selective symbols.
>
>
> "protected" is already selective, though. The code generator had to
> intentionally set protected visibility on symbols.

Compiler needs a way to know if an external symbol is protected
or not if we don't want to force to use GOT to access ALL external
symbols, even if they turn out to be defined locally at link-time.

>
> - Compilers could also generate tentative PIC-style references, with
> sufficient relocations to allow the linker to convert the indirect
> reference to a direct reference when possible (changing the second
> load into a nop or copy). HP-UX also does this.
>
>
> I extended the x86 psABIs with relaxable GOT relocations and
> implemented the similar linker optimization in ld in binutils 2.26
> and gold also implemented the subset of the linker optimization.
>
> - Indirect references from PIC code to a protected symbol penalize the
> common case (referencing a symbol within its own module) to support
> the uncommon case, while introducing the nasty side effects of a COPY
> relocation.
>
> - COPY relocations are evil. They bind an application to a specific
> version of a shared library's ABI, and introduce a hidden startup
> cost. If we're going to make any changes, we should be moving towards
> elimination of COPY relocations, rather than disabling features that
> were designed to improve performance.
>
>
> Copy relocation can improve performance.  Google enabled copy
> relocations for PIE in ld/gold and GCC to improve prefomance by
> up to 5%:
>
> https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html
>
> It is in GCC 5.
>
>
> Did you look at what the costs were in startup time and dirty pages by using

No, I haven't.  It will be a very useful data.

> copy relocations? What do you do if the size of the definition changes in a
> new version of the library?
>

The size of sys_errlist data symbol in glibc changed over time.  We
use symbol versioning to provide backward binary compatibility:

 1326: 001c7a80   492 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.0
  1327: 001c7a80   500 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.1
  1324: 001c7a80   540 OBJECT  GLOBAL DEFAULT   29 sys_errlist@@GLIBC_2.12
  1328: 001c7a80   504 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.3
  1330: 001c7a80   528 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.4


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> Copy relocation and protected symbol are mutually exclusive.
> Since copy relocation is the part of x86 psABIs, it limits
> protected symbol effectiveness on x86.  In glibc, we add a
> local alias to symbol we want to protect and access the local
> alias within glibc without GOT.

That can be dangerous -- the copy you reference from within glibc may
not be the same copy that all other code uses. You don't want to do
that in general.

>> - Compilers could also generate tentative PIC-style references, with
>> sufficient relocations to allow the linker to convert the indirect
>> reference to a direct reference when possible (changing the second
>> load into a nop or copy). HP-UX also does this.
>
> I extended the x86 psABIs with relaxable GOT relocations and
> implemented the similar linker optimization in ld in binutils 2.26
> and gold also implemented the subset of the linker optimization.

Yes; this came in after Sri's changes to enable direct references to
data for -fPIE. It would be interesting to see what benefit remains
after doing the mov-to-lea optimizations.

> Copy relocation can improve performance.  Google enabled copy
> relocations for PIE in ld/gold and GCC to improve prefomance by
> up to 5%:
>
> https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html
>
> It is in GCC 5.

I think you're overlooking the key point. The performance improvement
wasn't from COPY relocations -- it was from eliminating the load from
the GOT. Being able to use an occasional COPY relocation in PIE code,
as was always supported for non-PIE code, was done to enable the code
generation change, and make PIE more like non-PIE in that respect.

Sri's measurements were for apps that had up-to-then been built as
non-PIC-non-PIE code. We wanted to convert apps to PIE as a security
improvement, but there was a loss of performance due to the indirect
references through the GOT. There really was no good reason for PIE
code to have to go through the GOT to get to data, so this was just a
way to recover some (or most) of the lost performance.

COPY relocations aren't that significant, but because they weren't
supported for PIE code (a totally arbitrary implementation choice),
they were an obstacle to making the code gen change.

> 1. Compiler accesses protected symbols without GOT in PIC mode
> and marks object files no-copy-relocation-against-protected-symbol.
> 2. Compiler accesses external symbols with GOT in non-PIC mode.
> 3. Linker marks shared object with no-copy-relocation-against-protected-symbol
> if any input files have no-copy-relocation-against-protected-symbol
> marker.
> 4. Linker optimizes out GOT access if symbol is defined locally
> in executable.
> 5. ld.so checks no-copy-relocation-against-protected-symbol marker
> on shared object to avoid copy relocation against protected symbol
> at run-time.

This looks like it would work, but it seems like over-engineering to
me. The problem you're trying to solve just isn't that significant (or
new).

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> Did you look at what the costs were in startup time and dirty pages by using
> copy relocations? What do you do if the size of the definition changes in a
> new version of the library?

There wouldn't be a measurable cost in dirty pages; the copied objects
are simply allocated in bss in the executable.

Startup time wasn't a concern either, simply because COPY relocations
weren't the real issue. We had to support an *occasional* COPY
relocation in order to enable -fPIE to access globals directly rather
than through the GOT. These would be the same COPY relocations that
the apps would have already been using in non-PIE mode (things like
stdin, stdout, and stderr, e.g.).

The size of an object is baked into the ABI when a COPY relocation is
used, so any change in size is an ABI change (often handled, as HJ
pointed out, with versioning). For a lot of common COPY relocations,
that's not really an issue, because both the size and the layout are
baked in through macros like putc() and getc().

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 28, 2016, at 3:21 PM, Cary Coutant <ccoutant@gmail.com> wrote:
> 
>> Did you look at what the costs were in startup time and dirty pages by using
>> copy relocations? What do you do if the size of the definition changes in a
>> new version of the library?
> 
> There wouldn't be a measurable cost in dirty pages; the copied objects
> are simply allocated in bss in the executable.

Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?

-Joe

> 
> Startup time wasn't a concern either, simply because COPY relocations
> weren't the real issue. We had to support an *occasional* COPY
> relocation in order to enable -fPIE to access globals directly rather
> than through the GOT. These would be the same COPY relocations that
> the apps would have already been using in non-PIE mode (things like
> stdin, stdout, and stderr, e.g.).
> 
> The size of an object is baked into the ABI when a COPY relocation is
> used, so any change in size is an ABI change (often handled, as HJ
> pointed out, with versioning). For a lot of common COPY relocations,
> that's not really an issue, because both the size and the layout are
> baked in through macros like putc() and getc().
> 
> -cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>>> Did you look at what the costs were in startup time and dirty pages by using
>>> copy relocations? What do you do if the size of the definition changes in a
>>> new version of the library?
>>
>> There wouldn't be a measurable cost in dirty pages; the copied objects
>> are simply allocated in bss in the executable.
>
> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?

No, references from within the .so would have always used the GOT.
Non-protected global symbols in a shared library are still
pre-emptible, so they are always indirect, and there's always a
dynamic relocation for the GOT entry. Whether the prevailing
definition winds up in the executable or the shared library, the
dynamic loader still has to bind the symbol and apply the relocation.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 28, 2016, at 3:38 PM, Cary Coutant <ccoutant@gmail.com> wrote:
> 
>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>> copy relocations? What do you do if the size of the definition changes in a
>>>> new version of the library?
>>> 
>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>> are simply allocated in bss in the executable.
>> 
>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
> 
> No, references from within the .so would have always used the GOT.
> Non-protected global symbols in a shared library are still
> pre-emptible, so they are always indirect, and there's always a
> dynamic relocation for the GOT entry. Whether the prevailing
> definition winds up in the executable or the shared library, the
> dynamic loader still has to bind the symbol and apply the relocation.

Ah, right, ELF symbols are preemptible by default, so you'd already be dirtying any pages containing data-to-data references to the symbol regardless.

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
> >>> Did you look at what the costs were in startup time and dirty pages by using
> >>> copy relocations? What do you do if the size of the definition changes in a
> >>> new version of the library?
> >>
> >> There wouldn't be a measurable cost in dirty pages; the copied objects
> >> are simply allocated in bss in the executable.
> >
> > Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
> 
> No, references from within the .so would have always used the GOT.
> Non-protected global symbols in a shared library are still
> pre-emptible, so they are always indirect, and there's always a
> dynamic relocation for the GOT entry. Whether the prevailing
> definition winds up in the executable or the shared library, the
> dynamic loader still has to bind the symbol and apply the relocation.

HJ's changes to protected visibility meant compiler changes so that
protected visibility in shared libraries is no longer seen as local.
So yes, protected visibility symbols in shared libraries now go
through the GOT.  Prior to his changes, they were optimized to a
pc-relative access.  Joe is correct in pointing out that shared
libraries needed a change.  Bad luck if you're using an older
compiler.  Also bad luck if you want to use protected visibility to
optimize your shared library.

HJ also made glibc ld.so changes to ensure the semantics of protected
visibility symbols remain unchanged when multiple shared libraries
define the same protected visibility symbol.

Apparently most people in the gcc and glibc communities saw these
toolchain modifications as fiendishly clever.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>> > Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>
>> No, references from within the .so would have always used the GOT.
>> Non-protected global symbols in a shared library are still
>> pre-emptible, so they are always indirect, and there's always a
>> dynamic relocation for the GOT entry. Whether the prevailing
>> definition winds up in the executable or the shared library, the
>> dynamic loader still has to bind the symbol and apply the relocation.
>
> HJ's changes to protected visibility meant compiler changes so that
> protected visibility in shared libraries is no longer seen as local.
> So yes, protected visibility symbols in shared libraries now go
> through the GOT.  Prior to his changes, they were optimized to a
> pc-relative access.  Joe is correct in pointing out that shared
> libraries needed a change.  Bad luck if you're using an older
> compiler.  Also bad luck if you want to use protected visibility to
> optimize your shared library.
>
> HJ also made glibc ld.so changes to ensure the semantics of protected
> visibility symbols remain unchanged when multiple shared libraries
> define the same protected visibility symbol.
>
> Apparently most people in the gcc and glibc communities saw these
> toolchain modifications as fiendishly clever.

I may have misunderstood the context of Joe's question, but I read it
as asking about the effect of Sri's changes to use direct references
to all symbols in PIE mode, rather than the effect of HJ's changes to
use indirect references to protected symbols in PIC mode.

In the former context, we're talking about code where protected
symbols are not often used, references from the executable to shared
library data is infrequent, and the intersection of the two is
exceedingly rare. There was a clear benefit to changing PIE code to
use GOT-relative, rather than GOT-indirect, references to data. It was
only necessary to tweak the linker to allow COPY relocations from a
PIE executable so that the infrequent references from the executable
to (non-protected) shared library data symbols would still work. Any
non-PIC references to protected symbols would have already failed
because the code had already successfully built as a non-PIE
application.

In the latter context, we're talking about penalizing access to
protected symbols within a shared library in order to support that
exceedingly rare case that already didn't work for non-PIE code. I
just don't see the value.

It's too bad that Sri's PIE changes pre-dated HJ's mov-to-lea
link-time optimization. I suspect that that optimization, all by
itself, would have reclaimed most of the lost performance that Sri
observed when turning on -fPIE, and we may never have descended into
this morass.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Maciej W. Rozycki]

On Thu, 24 Mar 2016, Cary Coutant wrote:

> - Arguing that protected means that the definition is in the same
> module but its address might be external is absurd. The *only* reason
> for the gABI to make that guarantee is so the compilers can optimize
> the code based on the knowledge that the symbol can't be pre-empted.

 For the record, SGI's original 64-bit MIPS psABI document[1], which is 
where the concept of symbol's export class aka visibility came from first, 
before having been adopted by the gABI, has these notes on the semantics 
of protected symbols:

"References to protected symbols (and hence to hidden or internal symbols) 
may be optimized by using absolute addresses in executables or by assuming 
addresses to be relatively nearby."

and:

"An R_MIPS_JALR relocation is intended for optimization of jumps to 
protected symbols, i.e. symbols which may not be preempted.  The word to 
be relocated is a jump (typically a JALR) to the indicated symbol.  If it 
is not a preemptible symbol (and therefore defined in the current 
executable/DSO) the relocation is a request to the linker to convert it to 
a direct branch (typically a JAL in the main executable, or a BGEZAL in 
DSOs if the target symbol is close enough).  The linker must check that 
the symbol is not preemptible before performing the relocation, but no 
action is required for correctness -- this is strictly an optimization 
hint."

which clearly indicate the intent for protected symbols to be defined 
locally, in the same way as hidden and internal symbols are, with the 
exception of additionally being exported.  You can't use a BGEZAL 
instruction (which is a direct branch with a limited signed 18-bit range)
if its target is moved outside the DSO it's in.

References:

[1] "64-bit ELF Object File Specification, Draft Version 2.5"
    <http://techpubs.sgi.com/library/manuals/4000/007-4658-001/pdf/007-4658-001.pdf>

  Maciej

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>> >>> Did you look at what the costs were in startup time and dirty pages by using
>> >>> copy relocations? What do you do if the size of the definition changes in a
>> >>> new version of the library?
>> >>
>> >> There wouldn't be a measurable cost in dirty pages; the copied objects
>> >> are simply allocated in bss in the executable.
>> >
>> > Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>
>> No, references from within the .so would have always used the GOT.
>> Non-protected global symbols in a shared library are still
>> pre-emptible, so they are always indirect, and there's always a
>> dynamic relocation for the GOT entry. Whether the prevailing
>> definition winds up in the executable or the shared library, the
>> dynamic loader still has to bind the symbol and apply the relocation.
>
> HJ's changes to protected visibility meant compiler changes so that
> protected visibility in shared libraries is no longer seen as local.
> So yes, protected visibility symbols in shared libraries now go
> through the GOT.  Prior to his changes, they were optimized to a
> pc-relative access.  Joe is correct in pointing out that shared
> libraries needed a change.  Bad luck if you're using an older
> compiler.  Also bad luck if you want to use protected visibility to
> optimize your shared library.
>
> HJ also made glibc ld.so changes to ensure the semantics of protected
> visibility symbols remain unchanged when multiple shared libraries
> define the same protected visibility symbol.
>
> Apparently most people in the gcc and glibc communities saw these
> toolchain modifications as fiendishly clever.
>

As I said before, copy relocation and protected symbol are fundamentally
incompatible.  Since copy relocation is the part of x86 psABIs, I updated
GCC, glibc and ld to make protected symbol to work with copy relocation.
That is protected symbol may be external, but won't be preempted.  The
price I paid is that protected symbol won't be accessed via PC-relative
relocation within the shared object.  To access protected symbol via
PC-relative relocation within the shared object, we need to disable copy
relocation in executable, which is a psABI change.  That is why I proposed
to mark the object as such so that we won't get surprise at run-time.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Szabolcs Nagy]

* Maciej W. Rozycki <macro@imgtec.com> [2016-04-16 00:59:17 +0100]:
> On Fri, 15 Apr 2016, Szabolcs Nagy wrote:
> 
> > > Copy relocation and protected visibility are fundamentally incompatible.
> > > On on hand, copy relocation is the part of the psABI and is used to
> > > access global data defined in a shared object from the executable.  It
> > > moves the definition of global data, which is defined in a share object,
> > > to the executable at run-time.  On the other hand, protected visibility
> > > indicates that a symbol is defined locally in the shared object at
> > > run-time.  Both can't be true at the same time.  The current solution
> > 
> > protected visibility indicates
> > 
> >   "that references within the defining module bind to
> >   the definition in that module. That is, the declared
> >   entity cannot be overridden by another module."
> > 
> > here "definition in that module" does not mean addresses
> > at runtime, but the c language level definition.
> > (c semantics can only indicate abstract machine behaviour,
> > not relocation types and in memory layout at runtime).
> 
>  Where did you get this definition from and how do you infer this is a "c 
> language level" rather than a binary level definition?  The SVR4/ELF gABI 
> says[1]:
> 
> "A symbol defined in the current component is protected if it is visible
> in other components but not preemptable, meaning that any reference to
> such a symbol from within the defining component must be resolved to the
> definition in that component, even if there is a definition in another
> component that would preempt by the default rules."
> 
> and:
> 
> "The presence of the STV_PROTECTED flag on a symbol in a given load module
> does not affect the symbol resolution rules for references to that symbol
> from outside the containing load module."
> 
> so it clearly indicates that it is dynamic load modules (i.e. either the 
> main executable or any of its referred DSOs) that are considered here, not 
> C source-level semantics.  Besides the ELF ABI is programming language 
> agnostic, using the C language as a reference only, so it cannot really 
> consider source-level semantics.
> 

sorry, i just looked at the c extension as documented
by gcc, but the elf gabi is the right source. however
it seems to verify my interpretation.

> > so there is no conflict between copy relocation and
> > protected visibility: the definition is not overridden,
> > it's an implementation detail that the address happens
> > to be in the data section of the main executable instead
> > of the defining module.
> 
>  Both the reference above and established practice assume a protected 
> symbol must have its address within the module that defines it.  See also:

copy relocation is not another definition for the symbol
(it's an implementation detail), and symbol resolution
rules for external references are not affected, so both
constraints you cited above hold now.

> <https://sourceware.org/ml/binutils/2016-03/msg00368.html> for my earlier 
> comment.

ok, so it was intended to be an optimization (for function
symbols), but that breaks correctness for object symbols
in the presence of copy relocations.. so the optimization
is not valid: the symbol must be visible to other modules
which only works for the non-pic main module via copy relocs.

> References:
> 
> [1] "System V Application Binary Interface - DRAFT - 10 June 2013",
>     The Santa Cruz Operation, Inc., "Symbol Table",
>     <http://www.sco.com/developers/gabi/latest/ch4.symtab.html>
> 
>   Maciej

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Maciej W. Rozycki]

On Fri, 15 Apr 2016, Szabolcs Nagy wrote:

> > Copy relocation and protected visibility are fundamentally incompatible.
> > On on hand, copy relocation is the part of the psABI and is used to
> > access global data defined in a shared object from the executable.  It
> > moves the definition of global data, which is defined in a share object,
> > to the executable at run-time.  On the other hand, protected visibility
> > indicates that a symbol is defined locally in the shared object at
> > run-time.  Both can't be true at the same time.  The current solution
> 
> protected visibility indicates
> 
>   "that references within the defining module bind to
>   the definition in that module. That is, the declared
>   entity cannot be overridden by another module."
> 
> here "definition in that module" does not mean addresses
> at runtime, but the c language level definition.
> (c semantics can only indicate abstract machine behaviour,
> not relocation types and in memory layout at runtime).

 Where did you get this definition from and how do you infer this is a "c 
language level" rather than a binary level definition?  The SVR4/ELF gABI 
says[1]:

"A symbol defined in the current component is protected if it is visible
in other components but not preemptable, meaning that any reference to
such a symbol from within the defining component must be resolved to the
definition in that component, even if there is a definition in another
component that would preempt by the default rules."

and:

"The presence of the STV_PROTECTED flag on a symbol in a given load module
does not affect the symbol resolution rules for references to that symbol
from outside the containing load module."

so it clearly indicates that it is dynamic load modules (i.e. either the 
main executable or any of its referred DSOs) that are considered here, not 
C source-level semantics.  Besides the ELF ABI is programming language 
agnostic, using the C language as a reference only, so it cannot really 
consider source-level semantics.

> so there is no conflict between copy relocation and
> protected visibility: the definition is not overridden,
> it's an implementation detail that the address happens
> to be in the data section of the main executable instead
> of the defining module.

 Both the reference above and established practice assume a protected 
symbol must have its address within the module that defines it.  See also:
<https://sourceware.org/ml/binutils/2016-03/msg00368.html> for my earlier 
comment.

References:

[1] "System V Application Binary Interface - DRAFT - 10 June 2013",
    The Santa Cruz Operation, Inc., "Symbol Table",
    <http://www.sco.com/developers/gabi/latest/ch4.symtab.html>

  Maciej

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Fri, Apr 15, 2016 at 9:36 AM, Jeff Law <law@redhat.com> wrote:
> On 04/15/2016 10:16 AM, H.J. Lu wrote:
>>
>> On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com>
>> wrote:
>>>
>>> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>>>>
>>>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>>>>
>>>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>>>
>>>>>>>
>>>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>>>> could
>>>>>>> easily summarize correct behavior and the impact if we were to just
>>>>>>> revert
>>>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>>>
>>>>>>
>>>>>>
>>>>>> It looks like it's not just the one change. There's this patch:
>>>>>>
>>>>>>      https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>>>
>>>>>> which took the idea that protected can still be pre-empted by a COPY
>>>>>> relocation and extended it to three more targets that use COPY
>>>>>> relocations.
>>>>>>
>>>>>> I wonder how many other patches have been based on the same
>>>>>> misunderstanding?
>>>
>>>
>>> (sorry i missed this thread)
>>>
>>> this was not a misunderstanding.
>>>
>>> that patch is necessary for correctness (odr) in
>>> the presence of copy relocations as described in
>>> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
>>> and
>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>>
>>> this was a long standing code gen bug in gcc and was
>>> about time to fix it (it was also broken in glibc's
>>> dynamic linker, but e.g. not in musl libc).
>>>
>>> (i don't see what is the issue with using the copy in
>>> the main executable from a shared library, performance
>>> is not a correctness issue, nor how it is possible to
>>> avoid the copy relocs.)
>>>
>>
>> Here is my understanding:
>>
>> Copy relocation and protected visibility are fundamentally incompatible.
>> On on hand, copy relocation is the part of the psABI and is used to
>> access global data defined in a shared object from the executable.  It
>> moves the definition of global data, which is defined in a share object,
>> to the executable at run-time.  On the other hand, protected visibility
>> indicates that a symbol is defined locally in the shared object at
>> run-time.  Both can't be true at the same time.  The current solution
>> is to make protected symbol more or less like normal symbol, which
>> prevents optimizing local access to protected symbol within the shared
>> object.
>>
>> I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:
>>
>> https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI
>>
>> GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
>> should be no copy relocations against protected data symbols. If a
>> relocat-
>> able object contains this property, linker should treat protected data
>> symbol
>> as defined locally at run-time and copy this property to the output share
>> object. Linker should add this property to the output share object if any
>> pro-
>> tected symbol is expected to be defined locally at run-time. Run-time
>> loader
>> should disallow copy relocations against protected data symbols defined in
>> share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
>> erty. Its PR_DATASZ should be 0.
>
> I'd strongly suggest discussing directly with Carlos, Cary and Alan.  My
> worry here is this just adding another layer of stuff to deal with a
> fundamentally broken concept -- protected visibility.
>

Adding glibc.

Protected symbol has been a tricky issue for glibc and binutils
from day 1.  We have special treatment for pointers of protected
functions in both ld and ld.so.  Protected symbol is a useful
feature.  It just doesn't work with copy relocation.  My proposal
will make it work for psABIs with copy relocation by disallowing
copy relocation on protected symbol.  Linker change is minimum
and it will bypass extra symbol lookups in ld.so.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Szabolcs Nagy]

On 15/04/16 17:16, H.J. Lu wrote:
> On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>>
>>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>>> could
>>>>>> easily summarize correct behavior and the impact if we were to just
>>>>>> revert
>>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>>
>>>>>
>>>>> It looks like it's not just the one change. There's this patch:
>>>>>
>>>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>>
>>>>> which took the idea that protected can still be pre-empted by a COPY
>>>>> relocation and extended it to three more targets that use COPY
>>>>> relocations.
>>>>>
>>>>> I wonder how many other patches have been based on the same
>>>>> misunderstanding?
>>
>> (sorry i missed this thread)
>>
>> this was not a misunderstanding.
>>
>> that patch is necessary for correctness (odr) in
>> the presence of copy relocations as described in
>> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
>> and
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>
>> this was a long standing code gen bug in gcc and was
>> about time to fix it (it was also broken in glibc's
>> dynamic linker, but e.g. not in musl libc).
>>
>> (i don't see what is the issue with using the copy in
>> the main executable from a shared library, performance
>> is not a correctness issue, nor how it is possible to
>> avoid the copy relocs.)
>>
> 
> Here is my understanding:
> 
> Copy relocation and protected visibility are fundamentally incompatible.
> On on hand, copy relocation is the part of the psABI and is used to
> access global data defined in a shared object from the executable.  It
> moves the definition of global data, which is defined in a share object,
> to the executable at run-time.  On the other hand, protected visibility
> indicates that a symbol is defined locally in the shared object at
> run-time.  Both can't be true at the same time.  The current solution

protected visibility indicates

  "that references within the defining module bind to
  the definition in that module. That is, the declared
  entity cannot be overridden by another module."

here "definition in that module" does not mean addresses
at runtime, but the c language level definition.
(c semantics can only indicate abstract machine behaviour,
not relocation types and in memory layout at runtime).

so there is no conflict between copy relocation and
protected visibility: the definition is not overridden,
it's an implementation detail that the address happens
to be in the data section of the main executable instead
of the defining module.

i think your fixes in gcc, etc are ok.
(i understand that ppl want to optimize this but that
should be a separate discussion once there is consensus
about the correct semantics).

> is to make protected symbol more or less like normal symbol, which
> prevents optimizing local access to protected symbol within the shared
> object.
> 
> I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:
> 
> https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI
> 
> GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
> should be no copy relocations against protected data symbols. If a relocat-
> able object contains this property, linker should treat protected data symbol
> as defined locally at run-time and copy this property to the output share
> object. Linker should add this property to the output share object if any pro-
> tected symbol is expected to be defined locally at run-time. Run-time loader
> should disallow copy relocations against protected data symbols defined in
> share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
> erty. Its PR_DATASZ should be 0.
> 
> 
> 

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 04/15/2016 10:16 AM, H.J. Lu wrote:
> On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>>
>>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>>> could
>>>>>> easily summarize correct behavior and the impact if we were to just
>>>>>> revert
>>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>>
>>>>>
>>>>> It looks like it's not just the one change. There's this patch:
>>>>>
>>>>>      https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>>
>>>>> which took the idea that protected can still be pre-empted by a COPY
>>>>> relocation and extended it to three more targets that use COPY
>>>>> relocations.
>>>>>
>>>>> I wonder how many other patches have been based on the same
>>>>> misunderstanding?
>>
>> (sorry i missed this thread)
>>
>> this was not a misunderstanding.
>>
>> that patch is necessary for correctness (odr) in
>> the presence of copy relocations as described in
>> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
>> and
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>
>> this was a long standing code gen bug in gcc and was
>> about time to fix it (it was also broken in glibc's
>> dynamic linker, but e.g. not in musl libc).
>>
>> (i don't see what is the issue with using the copy in
>> the main executable from a shared library, performance
>> is not a correctness issue, nor how it is possible to
>> avoid the copy relocs.)
>>
>
> Here is my understanding:
>
> Copy relocation and protected visibility are fundamentally incompatible.
> On on hand, copy relocation is the part of the psABI and is used to
> access global data defined in a shared object from the executable.  It
> moves the definition of global data, which is defined in a share object,
> to the executable at run-time.  On the other hand, protected visibility
> indicates that a symbol is defined locally in the shared object at
> run-time.  Both can't be true at the same time.  The current solution
> is to make protected symbol more or less like normal symbol, which
> prevents optimizing local access to protected symbol within the shared
> object.
>
> I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:
>
> https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI
>
> GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
> should be no copy relocations against protected data symbols. If a relocat-
> able object contains this property, linker should treat protected data symbol
> as defined locally at run-time and copy this property to the output share
> object. Linker should add this property to the output share object if any pro-
> tected symbol is expected to be defined locally at run-time. Run-time loader
> should disallow copy relocations against protected data symbols defined in
> share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
> erty. Its PR_DATASZ should be 0.
I'd strongly suggest discussing directly with Carlos, Cary and Alan.  My 
worry here is this just adding another layer of stuff to deal with a 
fundamentally broken concept -- protected visibility.

Jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>
>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>> could
>>>>> easily summarize correct behavior and the impact if we were to just
>>>>> revert
>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>
>>>>
>>>> It looks like it's not just the one change. There's this patch:
>>>>
>>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>
>>>> which took the idea that protected can still be pre-empted by a COPY
>>>> relocation and extended it to three more targets that use COPY
>>>> relocations.
>>>>
>>>> I wonder how many other patches have been based on the same
>>>> misunderstanding?
>
> (sorry i missed this thread)
>
> this was not a misunderstanding.
>
> that patch is necessary for correctness (odr) in
> the presence of copy relocations as described in
> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
> and
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>
> this was a long standing code gen bug in gcc and was
> about time to fix it (it was also broken in glibc's
> dynamic linker, but e.g. not in musl libc).
>
> (i don't see what is the issue with using the copy in
> the main executable from a shared library, performance
> is not a correctness issue, nor how it is possible to
> avoid the copy relocs.)
>

Here is my understanding:

Copy relocation and protected visibility are fundamentally incompatible.
On on hand, copy relocation is the part of the psABI and is used to
access global data defined in a shared object from the executable.  It
moves the definition of global data, which is defined in a share object,
to the executable at run-time.  On the other hand, protected visibility
indicates that a symbol is defined locally in the shared object at
run-time.  Both can't be true at the same time.  The current solution
is to make protected symbol more or less like normal symbol, which
prevents optimizing local access to protected symbol within the shared
object.

I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:

https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI

GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
should be no copy relocations against protected data symbols. If a relocat-
able object contains this property, linker should treat protected data symbol
as defined locally at run-time and copy this property to the output share
object. Linker should add this property to the output share object if any pro-
tected symbol is expected to be defined locally at run-time. Run-time loader
should disallow copy relocations against protected data symbols defined in
share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
erty. Its PR_DATASZ should be 0.



-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Szabolcs Nagy]

On 31/03/16 14:26, Ramana Radhakrishnan wrote:
> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>
>>>> It would help me immensely on the GCC side if things if you and Alan
>>>> could
>>>> easily summarize correct behavior and the impact if we were to just
>>>> revert
>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>
>>>
>>> It looks like it's not just the one change. There's this patch:
>>>
>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>
>>> which took the idea that protected can still be pre-empted by a COPY
>>> relocation and extended it to three more targets that use COPY
>>> relocations.
>>>
>>> I wonder how many other patches have been based on the same
>>> misunderstanding?

(sorry i missed this thread)

this was not a misunderstanding.

that patch is necessary for correctness (odr) in
the presence of copy relocations as described in
https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
and
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012

this was a long standing code gen bug in gcc and was
about time to fix it (it was also broken in glibc's
dynamic linker, but e.g. not in musl libc).

(i don't see what is the issue with using the copy in
the main executable from a shared library, performance
is not a correctness issue, nor how it is possible to
avoid the copy relocs.)

>>
>> I don't think it was many -- I certainly recall the arm/aarch64 variant.
>> There may have been one other varasm.c change in this space or I might be
>> conflating it with the arm/aarch64 change.  Tracking those down is naturally
>> part of this work.
> 
> The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
> the same time.  IIUC, the reason the patch above went in were to fix
> failures on arm / aarch64 with those tests. I haven't yet worked out
> whether all this is the same issue.
> 
> CC'ing Szabolcs.
> 
> Thanks,
> Ramana
> 
>>
>> jeff
> 

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

On Fri, Apr 01, 2016 at 01:51:33PM -0600, Jeff Law wrote:
> Essentially the question that needs to be answered is if we roll
> back 65280, do we need to also roll back 65780?

No, I don't think so.  As I wrote in the bugzilla (paraphrased)

What's going on here is that HJ has made changes to the linker for x86
to always copy variables defined in a shared library and referenced by
the executable, into the executable's .dynbss.  This is necessary to
support non-PIC without text relocations, but is optional for PIEs.

If you do this for PIEs and gain some performance without affecting
shared library performance, and that is the case for x86_64, then the
optimisation is quite reasonable.  I think powerpc64 would gain too,
but I haven't done anything about it yet (and for powerpc64 we should
be able to implement it entirely in the linker, no gcc changes
needed).

I believe common variables can be treated like any other variable so
far as this optimisation goes.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 03/30/2016 06:40 PM, Cary Coutant wrote:
>> It would help me immensely on the GCC side if things if you and Alan could
>> easily summarize correct behavior and the impact if we were to just revert
>> HJ's change.  A testcase would be amazingly helpful too.
>
> It looks like it's not just the one change. There's this patch:
>
>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>
> which took the idea that protected can still be pre-empted by a COPY
> relocation and extended it to three more targets that use COPY
> relocations.
>
> I wonder how many other patches have been based on the same misunderstanding?
The patches around BZ65780 may be tangled in this mess as well.    And 
it bled into the s390 & darwin ports as well.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65780

Alan, could you take a peek at 65780 -- you're better versed than I in 
this stuff.  Essentially the question that needs to be answered is if we 
roll back 65280, do we need to also roll back 65780?

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 31, 2016 at 6:26 AM, Ramana Radhakrishnan
<ramana.gcc@googlemail.com> wrote:
> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>
>>>> It would help me immensely on the GCC side if things if you and Alan
>>>> could
>>>> easily summarize correct behavior and the impact if we were to just
>>>> revert
>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>
>>>
>>> It looks like it's not just the one change. There's this patch:
>>>
>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>
>>> which took the idea that protected can still be pre-empted by a COPY
>>> relocation and extended it to three more targets that use COPY
>>> relocations.
>>>
>>> I wonder how many other patches have been based on the same
>>> misunderstanding?
>>
>> I don't think it was many -- I certainly recall the arm/aarch64 variant.
>> There may have been one other varasm.c change in this space or I might be
>> conflating it with the arm/aarch64 change.  Tracking those down is naturally
>> part of this work.
>
> The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
> the same time.  IIUC, the reason the patch above went in were to fix
> failures on arm / aarch64 with those tests. I haven't yet worked out
> whether all this is the same issue.
>

These testcases show the problem between copy relocation and
protected data symbol.  I updated my program property extension
to GNU ABI to prevent copy relocation against protected data symbol:

https://sourceware.org/ml/gnu-gabi/2016-q1/msg00041.html

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Ramana Radhakrishnan]

On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>
>>> It would help me immensely on the GCC side if things if you and Alan
>>> could
>>> easily summarize correct behavior and the impact if we were to just
>>> revert
>>> HJ's change.  A testcase would be amazingly helpful too.
>>
>>
>> It looks like it's not just the one change. There's this patch:
>>
>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>
>> which took the idea that protected can still be pre-empted by a COPY
>> relocation and extended it to three more targets that use COPY
>> relocations.
>>
>> I wonder how many other patches have been based on the same
>> misunderstanding?
>
> I don't think it was many -- I certainly recall the arm/aarch64 variant.
> There may have been one other varasm.c change in this space or I might be
> conflating it with the arm/aarch64 change.  Tracking those down is naturally
> part of this work.

The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
the same time.  IIUC, the reason the patch above went in were to fix
failures on arm / aarch64 with those tests. I haven't yet worked out
whether all this is the same issue.

CC'ing Szabolcs.

Thanks,
Ramana

>
> jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 03/30/2016 06:40 PM, Cary Coutant wrote:
>> It would help me immensely on the GCC side if things if you and Alan could
>> easily summarize correct behavior and the impact if we were to just revert
>> HJ's change.  A testcase would be amazingly helpful too.
>
> It looks like it's not just the one change. There's this patch:
>
>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>
> which took the idea that protected can still be pre-empted by a COPY
> relocation and extended it to three more targets that use COPY
> relocations.
>
> I wonder how many other patches have been based on the same misunderstanding?
I don't think it was many -- I certainly recall the arm/aarch64 variant. 
  There may have been one other varasm.c change in this space or I might 
be conflating it with the arm/aarch64 change.  Tracking those down is 
naturally part of this work.

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> 2) Emit dynamic text relocations for non-PIC executable access to
> protected visibility variables defined in shared libraries.  Makes
> executable pages with text relocations non-shared between processes.
> (But can be shared again after relocation if two processes have the
> same layout.)

Note that this is not always possible, depending on the addressing
range of the code emitted. For example, a pc-relative access may have
sufficient range to reach a variable copied into the executable's bss,
but not sufficient range to reach the original definition in a shared
library.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> It would help me immensely on the GCC side if things if you and Alan could
> easily summarize correct behavior and the impact if we were to just revert
> HJ's change.  A testcase would be amazingly helpful too.

It looks like it's not just the one change. There's this patch:

   https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html

which took the idea that protected can still be pre-empted by a COPY
relocation and extended it to three more targets that use COPY
relocations.

I wonder how many other patches have been based on the same misunderstanding?

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

OK, I'll try to list the technical issues re. pr65248 "Copy relocation
against protected symbol doesn't work".

Given a shared library that defines a variable, and a non-PIC
executable that references that variable, the linker makes a duplicate
of the variable in the executable .dynbss section and arranges to have
the copy initialized by the dynamic loader with a copy relocation.
.dynbss is a linker created section that becomes part of the
executable bss segment.  The idea is that at run-time both the
executable and the shared library will use the executable copy of the
variable.  It's a rather ancient linker hack to avoid dynamic text
relocations, invented well before symbol visibility.

The problem with protected visibility variables in shared libraries is
that the shared library continues to access its own copy.  So we have
two copies of a variable where the source only contained one variable
definition.  (It's not really the copy relocation as such that causes
a problem, but the fact that we have a duplicate in .dynbss.)

Solutions are:

1) Have the linker emit an error, suggesting that the executable needs
to be PIC to access a protected visibility variable in a shared
library.  Pass the pain on to users.

2) Emit dynamic text relocations for non-PIC executable access to
protected visibility variables defined in shared libraries.  Makes
executable pages with text relocations non-shared between processes.
(But can be shared again after relocation if two processes have the
same layout.)

3) Teach the linker to edit non-PIC to PIC.  I've done this for ppc32,
but it's not very easy for x86 due to the fact that you can't find the
start of an instruction from a relocated field.  It also results in
slow code, since PIC sequences are usually larger than non-PIC,
requiring a branch into a patch area and a branch back.

4) Have the compiler always emit PIC for external variable access.
The .dynbss hack could then die a natural death.  Coupled with linker
editing of PIC to non-PIC, you'd get correct and reasonably fast code
with perhaps a few nops.  Again, x86 has difficulty due to variable
length insns, but this could be solved with marker relocs.

5) HJ's solution.  Make the compiler emit the same code for protected
visibility variables in shared libraries as it does for default
visibility variables.  ie. shared libraries access their protected
visibility variables via the GOT, which allows the .dynbss hack to
work.  Also, modify the dynamic loader to ensure protected visibility
semantics are enforced.  (Consider the case of shared lib A that
defines protected visibility var V, and shared lib B that also defines
protected visibility var V, and an executable that references V.
Assuming A is before B in breadth-first order, then the executable and
A should use the .dynbss copy of V, but B should use its own V!)  I've
got to hand it to HJ, I think he has all this working properly to
satisfy the semantics of protected visibility.

However, shared library access of protected visibility variables is of
course no quicker than default visibility variables.  Which defeats
the main reason to use protected visibility variables in shared
libraries!  Note also that it unnecessarily slows down shared library
code using protected visibility variables that are not accessed by the
executable (but perhaps accessed from some other shared library).

Also, I don't believe there is anything to prevent a newly compiled
program from running with an old glibc, which would mean protected
visibility semantics are broken on such a system.  To be fair, they
were probably broken on such a system before the linker complained
about non-PIC protected visibility access, anyway.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> See PR ld/15228 and PR
> gold/19823 (the latter being the bug report that started this current
> thread).

Sorry, it was actually Joe's note about Swift that started this
thread, but that came in right after the gold PR, so I was already
primed.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> And FWIW, there are some folks on the GCC side of things that think that
> HJ's change for 65248 is broken and needs to be reverted before gcc-6
> releases.

Thanks; I was thinking the next step was to move this discussion over
to the GCC side to see if anyone there was in agreement.

> It would help me immensely on the GCC side if things if you and Alan could
> easily summarize correct behavior and the impact if we were to just revert
> HJ's change.  A testcase would be amazingly helpful too.

On the compiler side, the main thing to check is that any reference to
a protected data symbol, whether extern, common, weak, or initialized,
is accessed with a pc-relative or GOT-relative addressing mode (i.e.,
not indirect through a GOT entry). This is the fundamental reason to
declare a data symbol protected.

The testcases in gcc.target/i386/pr65248-[1234].c look suitable to me,
if we simply switch "scan-assembler-not" with "scan-assembler"
throughout. These basically take a source like this:

__attribute__((visibility("protected")))
int xxx;

int
foo ()
{
  return xxx;
}

and compile with -fpic. Prior to GCC 5, we got

   movl    xxx(%rip), %eax

but with GCC 5, we now get

   movq    xxx@GOTPCREL(%rip), %rax
   movl    (%rax), %eax

There was another change to GCC at r218397, in Dec. 2014, to make data
access in PIE mode work more like non-PIC mode. This required linker
changes to enable the use of COPY relocations when generating PIE
output. This patch was originally developed by Sriraman Tallam on the
Google branch, but was checked on trunk in by HJ after considerable
discussion. As far as I'm concerned (and I advised Sri during the
development of this patch), this change was fine -- it simply
recognizes that a position-independent executable does not need to use
PIC-style references to data when a pc-relative or got-relative access
is possible. As in non-PIC mode, these accesses work fine for data
defined in the main program, but result in occasional COPY relocations
if the data happens to be defined in a shared library. Unfortunately,
I think this was the patch that set the rest in motion -- when
developing test cases, HJ noted that COPY relocations to protected
symbols were not allowed, and set out to fix that.

Ultimately, I'd like to see GCC support the use of declspec(dllimport)
or __attribute__((dllimport)) to declare a global variable known to be
defined externally in a shared library. It should then generate a
PIC-style reference even in non-PIC mode. This could be used to
eliminate all COPY relocations.

On the linker side, the linkers need to check that a reference from a
main program to a protected symbol will result in neither a text
relocation nor a COPY relocation, but instead issue an error message
explaining fairly clearly that a protected symbol in a shared library
cannot be referenced from non-PIC code. See PR ld/15228 and PR
gold/19823 (the latter being the bug report that started this current
thread).

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 03/29/2016 07:46 PM, Cary Coutant wrote:
>>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>>
>>
>> Protected symbol never worked correctly on x86 before.  My
>> change closed a few long-standing bugs.  There is no going-back.
>
> You keep countering my arguments with assertions like, "it was a bug
> and I fixed it," but you present no arguments of your own to support
> your position. I'm not sure what long-standing bugs you're referring
> to -- the only one I can find, PR target/65248 [1], was filed by you
> yourself, so you can't really use that as support. In fact, PR
> ld/15228 [2], was filed against ld for *not* refusing to make a COPY
> relocation to a protected symbol, and Alan fixed that. Gold has the
> same bug, and I intend to fix it there, too.
And FWIW, there are some folks on the GCC side of things that think that 
HJ's change for 65248 is broken and needs to be reverted before gcc-6 
releases.

I'm not familiar enough with all the issues, but I am familiar enough 
with the work of HJ, Alan and yourself that if you & Alan say HJ's GCC 
change is wrong, then, well, it's wrong and needs to be reverted.

It would help me immensely on the GCC side if things if you and Alan 
could easily summarize correct behavior and the impact if we were to 
just revert HJ's change.  A testcase would be amazingly helpful too.

I'm sure I could extract the relevant info out of the thread, but I'm 
just buried right now.

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>
>
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

You keep countering my arguments with assertions like, "it was a bug
and I fixed it," but you present no arguments of your own to support
your position. I'm not sure what long-standing bugs you're referring
to -- the only one I can find, PR target/65248 [1], was filed by you
yourself, so you can't really use that as support. In fact, PR
ld/15228 [2], was filed against ld for *not* refusing to make a COPY
relocation to a protected symbol, and Alan fixed that. Gold has the
same bug, and I intend to fix it there, too.

You have effectively made protected symbols "work" by disabling the
very thing they were designed to do. I don't know how to point out the
absurdity any more clearly.

<reductio ad absurdum>
By your logic, you should make GCC generate PIC code by default and
turn -fno-pic into a no-op, because non-PIC code doesn't "work" in a
shared library. (Ironically, that would actually make protected
symbols work again without your changes.) Over the years, I've heard
many complaints from developers that they couldn't put code into a
shared library without compiling it with a special option. When you
first heard that complaint, why wasn't your response to change the
compiler? Isn't that the only way to make shared libraries "work"?
</reductio ad absurdum>

-cary

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65248

[2] https://sourceware.org/bugzilla/show_bug.cgi?id=15228

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

On Tue, Mar 29, 2016 at 12:54:36PM -0700, H.J. Lu wrote:
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

The swift bug report and this thread should be telling you that
protected visibility symbols don't work correctly *now* on x86.

They did in fact work correctly before your changes, but had many
annoying restrictions.  Those restrictions, all due to non-PIC, can be
overcome in a number of ways without breaking shared libraries.
eg. ppc32 is non-PIC by default just like x64, but see bfd/elf32-ppc.c
	  /* Convert lis;addi or lis;load/store accessing a protected
	     variable defined in a shared library to PIC.  */
Yes, this makes ppc32 executables slower.  I'd rather that than making
shared libraries slower.

FWIW, I agree with all the points Cary makes in
https://sourceware.org/ml/binutils/2016-03/msg00331.html.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:54 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Tue, Mar 29, 2016 at 12:51 PM, Joe Groff <jgroff@apple.com> wrote:
>>
>>> On Mar 29, 2016, at 12:43 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>>
>>> The link-time shared library may be very different from the run-time
>>> shared library.  Since copy relocation works correctly against protected
>>> symbol with the current gcc, binutils and glibc, we can't simply disallow
>>> copy relocation against protected symbol at link-time.
>>
>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>
>
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

If you want to avoid copy relocation, you can do

1. Pass -z noextern-protected-data to build the shared object.
2. Use GOT to access external data in executable.


H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:51 PM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 29, 2016, at 12:43 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> The link-time shared library may be very different from the run-time
>> shared library.  Since copy relocation works correctly against protected
>> symbol with the current gcc, binutils and glibc, we can't simply disallow
>> copy relocation against protected symbol at link-time.
>
> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>

Protected symbol never worked correctly on x86 before.  My
change closed a few long-standing bugs.  There is no going-back.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 29, 2016, at 12:43 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> The link-time shared library may be very different from the run-time
> shared library.  Since copy relocation works correctly against protected
> symbol with the current gcc, binutils and glibc, we can't simply disallow
> copy relocation against protected symbol at link-time.

However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:36 PM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 29, 2016, at 12:33 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <jgroff@apple.com> wrote:
>>> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>>>
>>>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>>>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>>>> new version of the library?
>>>>>>>>
>>>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>>>> are simply allocated in bss in the executable.
>>>>>>>
>>>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>>>
>>>>>> No, references from within the .so would have always used the GOT.
>>>>>> Non-protected global symbols in a shared library are still
>>>>>> pre-emptible, so they are always indirect, and there's always a
>>>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>>>> definition winds up in the executable or the shared library, the
>>>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>>>
>>>>> HJ's changes to protected visibility meant compiler changes so that
>>>>> protected visibility in shared libraries is no longer seen as local.
>>>>> So yes, protected visibility symbols in shared libraries now go
>>>>> through the GOT.  Prior to his changes, they were optimized to a
>>>>> pc-relative access.  Joe is correct in pointing out that shared
>>>>> libraries needed a change.  Bad luck if you're using an older
>>>>> compiler.  Also bad luck if you want to use protected visibility to
>>>>> optimize your shared library.
>>>>>
>>>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>>>> visibility symbols remain unchanged when multiple shared libraries
>>>>> define the same protected visibility symbol.
>>>>>
>>>>> Apparently most people in the gcc and glibc communities saw these
>>>>> toolchain modifications as fiendishly clever.
>>>>>
>>>>
>>>> As I said before, copy relocation and protected symbol are fundamentally
>>>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>>>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>>>> That is protected symbol may be external, but won't be preempted.  The
>>>> price I paid is that protected symbol won't be accessed via PC-relative
>>>> relocation within the shared object.  To access protected symbol via
>>>> PC-relative relocation within the shared object, we need to disable copy
>>>> relocation in executable, which is a psABI change.  That is why I proposed
>>>> to mark the object as such so that we won't get surprise at run-time.
>>>
>>> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.
>>
>> That is what I meant by "That is why I proposed to mark the object as such so
>> that we won't get surprise at run-time."
>
> I don't see why it would be a runtime failure. ld could refuse to resolve the relocation at static link time, couldn't it?
>

The link-time shared library may be very different from the run-time
shared library.  Since copy relocation works correctly against protected
symbol with the current gcc, binutils and glibc, we can't simply disallow
copy relocation against protected symbol at link-time.  It should be done
with a link command-line as well as markers on the input files.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 29, 2016, at 12:33 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <jgroff@apple.com> wrote:
>> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>> 
>>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>>> new version of the library?
>>>>>>> 
>>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>>> are simply allocated in bss in the executable.
>>>>>> 
>>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>> 
>>>>> No, references from within the .so would have always used the GOT.
>>>>> Non-protected global symbols in a shared library are still
>>>>> pre-emptible, so they are always indirect, and there's always a
>>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>>> definition winds up in the executable or the shared library, the
>>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>> 
>>>> HJ's changes to protected visibility meant compiler changes so that
>>>> protected visibility in shared libraries is no longer seen as local.
>>>> So yes, protected visibility symbols in shared libraries now go
>>>> through the GOT.  Prior to his changes, they were optimized to a
>>>> pc-relative access.  Joe is correct in pointing out that shared
>>>> libraries needed a change.  Bad luck if you're using an older
>>>> compiler.  Also bad luck if you want to use protected visibility to
>>>> optimize your shared library.
>>>> 
>>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>>> visibility symbols remain unchanged when multiple shared libraries
>>>> define the same protected visibility symbol.
>>>> 
>>>> Apparently most people in the gcc and glibc communities saw these
>>>> toolchain modifications as fiendishly clever.
>>>> 
>>> 
>>> As I said before, copy relocation and protected symbol are fundamentally
>>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>>> That is protected symbol may be external, but won't be preempted.  The
>>> price I paid is that protected symbol won't be accessed via PC-relative
>>> relocation within the shared object.  To access protected symbol via
>>> PC-relative relocation within the shared object, we need to disable copy
>>> relocation in executable, which is a psABI change.  That is why I proposed
>>> to mark the object as such so that we won't get surprise at run-time.
>> 
>> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.
> 
> That is what I meant by "That is why I proposed to mark the object as such so
> that we won't get surprise at run-time."

I don't see why it would be a runtime failure. ld could refuse to resolve the relocation at static link time, couldn't it?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <jgroff@apple.com> wrote:
> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>> new version of the library?
>>>>>>
>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>> are simply allocated in bss in the executable.
>>>>>
>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>
>>>> No, references from within the .so would have always used the GOT.
>>>> Non-protected global symbols in a shared library are still
>>>> pre-emptible, so they are always indirect, and there's always a
>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>> definition winds up in the executable or the shared library, the
>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>
>>> HJ's changes to protected visibility meant compiler changes so that
>>> protected visibility in shared libraries is no longer seen as local.
>>> So yes, protected visibility symbols in shared libraries now go
>>> through the GOT.  Prior to his changes, they were optimized to a
>>> pc-relative access.  Joe is correct in pointing out that shared
>>> libraries needed a change.  Bad luck if you're using an older
>>> compiler.  Also bad luck if you want to use protected visibility to
>>> optimize your shared library.
>>>
>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>> visibility symbols remain unchanged when multiple shared libraries
>>> define the same protected visibility symbol.
>>>
>>> Apparently most people in the gcc and glibc communities saw these
>>> toolchain modifications as fiendishly clever.
>>>
>>
>> As I said before, copy relocation and protected symbol are fundamentally
>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>> That is protected symbol may be external, but won't be preempted.  The
>> price I paid is that protected symbol won't be accessed via PC-relative
>> relocation within the shared object.  To access protected symbol via
>> PC-relative relocation within the shared object, we need to disable copy
>> relocation in executable, which is a psABI change.  That is why I proposed
>> to mark the object as such so that we won't get surprise at run-time.
>
> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.

That is what I meant by "That is why I proposed to mark the object as such so
that we won't get surprise at run-time."


-- 
H.J.