Fwd: Preventing preemption of 'protected' symbols in GNU ld 2.26

Fwd: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>> new version of the library?
>>>>> 
>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>> are simply allocated in bss in the executable.
>>>> 
>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>> 
>>> No, references from within the .so would have always used the GOT.
>>> Non-protected global symbols in a shared library are still
>>> pre-emptible, so they are always indirect, and there's always a
>>> dynamic relocation for the GOT entry. Whether the prevailing
>>> definition winds up in the executable or the shared library, the
>>> dynamic loader still has to bind the symbol and apply the relocation.
>> 
>> HJ's changes to protected visibility meant compiler changes so that
>> protected visibility in shared libraries is no longer seen as local.
>> So yes, protected visibility symbols in shared libraries now go
>> through the GOT.  Prior to his changes, they were optimized to a
>> pc-relative access.  Joe is correct in pointing out that shared
>> libraries needed a change.  Bad luck if you're using an older
>> compiler.  Also bad luck if you want to use protected visibility to
>> optimize your shared library.
>> 
>> HJ also made glibc ld.so changes to ensure the semantics of protected
>> visibility symbols remain unchanged when multiple shared libraries
>> define the same protected visibility symbol.
>> 
>> Apparently most people in the gcc and glibc communities saw these
>> toolchain modifications as fiendishly clever.
>> 
> 
> As I said before, copy relocation and protected symbol are fundamentally
> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
> GCC, glibc and ld to make protected symbol to work with copy relocation.
> That is protected symbol may be external, but won't be preempted.  The
> price I paid is that protected symbol won't be accessed via PC-relative
> relocation within the shared object.  To access protected symbol via
> PC-relative relocation within the shared object, we need to disable copy
> relocation in executable, which is a psABI change.  That is why I proposed
> to mark the object as such so that we won't get surprise at run-time.

I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <jgroff@apple.com> wrote:
> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>> new version of the library?
>>>>>>
>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>> are simply allocated in bss in the executable.
>>>>>
>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>
>>>> No, references from within the .so would have always used the GOT.
>>>> Non-protected global symbols in a shared library are still
>>>> pre-emptible, so they are always indirect, and there's always a
>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>> definition winds up in the executable or the shared library, the
>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>
>>> HJ's changes to protected visibility meant compiler changes so that
>>> protected visibility in shared libraries is no longer seen as local.
>>> So yes, protected visibility symbols in shared libraries now go
>>> through the GOT.  Prior to his changes, they were optimized to a
>>> pc-relative access.  Joe is correct in pointing out that shared
>>> libraries needed a change.  Bad luck if you're using an older
>>> compiler.  Also bad luck if you want to use protected visibility to
>>> optimize your shared library.
>>>
>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>> visibility symbols remain unchanged when multiple shared libraries
>>> define the same protected visibility symbol.
>>>
>>> Apparently most people in the gcc and glibc communities saw these
>>> toolchain modifications as fiendishly clever.
>>>
>>
>> As I said before, copy relocation and protected symbol are fundamentally
>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>> That is protected symbol may be external, but won't be preempted.  The
>> price I paid is that protected symbol won't be accessed via PC-relative
>> relocation within the shared object.  To access protected symbol via
>> PC-relative relocation within the shared object, we need to disable copy
>> relocation in executable, which is a psABI change.  That is why I proposed
>> to mark the object as such so that we won't get surprise at run-time.
>
> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.

That is what I meant by "That is why I proposed to mark the object as such so
that we won't get surprise at run-time."


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 29, 2016, at 12:33 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <jgroff@apple.com> wrote:
>> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>> 
>>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>>> new version of the library?
>>>>>>> 
>>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>>> are simply allocated in bss in the executable.
>>>>>> 
>>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>> 
>>>>> No, references from within the .so would have always used the GOT.
>>>>> Non-protected global symbols in a shared library are still
>>>>> pre-emptible, so they are always indirect, and there's always a
>>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>>> definition winds up in the executable or the shared library, the
>>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>> 
>>>> HJ's changes to protected visibility meant compiler changes so that
>>>> protected visibility in shared libraries is no longer seen as local.
>>>> So yes, protected visibility symbols in shared libraries now go
>>>> through the GOT.  Prior to his changes, they were optimized to a
>>>> pc-relative access.  Joe is correct in pointing out that shared
>>>> libraries needed a change.  Bad luck if you're using an older
>>>> compiler.  Also bad luck if you want to use protected visibility to
>>>> optimize your shared library.
>>>> 
>>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>>> visibility symbols remain unchanged when multiple shared libraries
>>>> define the same protected visibility symbol.
>>>> 
>>>> Apparently most people in the gcc and glibc communities saw these
>>>> toolchain modifications as fiendishly clever.
>>>> 
>>> 
>>> As I said before, copy relocation and protected symbol are fundamentally
>>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>>> That is protected symbol may be external, but won't be preempted.  The
>>> price I paid is that protected symbol won't be accessed via PC-relative
>>> relocation within the shared object.  To access protected symbol via
>>> PC-relative relocation within the shared object, we need to disable copy
>>> relocation in executable, which is a psABI change.  That is why I proposed
>>> to mark the object as such so that we won't get surprise at run-time.
>> 
>> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.
> 
> That is what I meant by "That is why I proposed to mark the object as such so
> that we won't get surprise at run-time."

I don't see why it would be a runtime failure. ld could refuse to resolve the relocation at static link time, couldn't it?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:36 PM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 29, 2016, at 12:33 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Tue, Mar 29, 2016 at 12:31 PM, Joe Groff <jgroff@apple.com> wrote:
>>> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>>>
>>>> On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
>>>>> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>>>>>>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>>>>>>> copy relocations? What do you do if the size of the definition changes in a
>>>>>>>>> new version of the library?
>>>>>>>>
>>>>>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>>>>>>> are simply allocated in bss in the executable.
>>>>>>>
>>>>>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>>>>>
>>>>>> No, references from within the .so would have always used the GOT.
>>>>>> Non-protected global symbols in a shared library are still
>>>>>> pre-emptible, so they are always indirect, and there's always a
>>>>>> dynamic relocation for the GOT entry. Whether the prevailing
>>>>>> definition winds up in the executable or the shared library, the
>>>>>> dynamic loader still has to bind the symbol and apply the relocation.
>>>>>
>>>>> HJ's changes to protected visibility meant compiler changes so that
>>>>> protected visibility in shared libraries is no longer seen as local.
>>>>> So yes, protected visibility symbols in shared libraries now go
>>>>> through the GOT.  Prior to his changes, they were optimized to a
>>>>> pc-relative access.  Joe is correct in pointing out that shared
>>>>> libraries needed a change.  Bad luck if you're using an older
>>>>> compiler.  Also bad luck if you want to use protected visibility to
>>>>> optimize your shared library.
>>>>>
>>>>> HJ also made glibc ld.so changes to ensure the semantics of protected
>>>>> visibility symbols remain unchanged when multiple shared libraries
>>>>> define the same protected visibility symbol.
>>>>>
>>>>> Apparently most people in the gcc and glibc communities saw these
>>>>> toolchain modifications as fiendishly clever.
>>>>>
>>>>
>>>> As I said before, copy relocation and protected symbol are fundamentally
>>>> incompatible.  Since copy relocation is the part of x86 psABIs, I updated
>>>> GCC, glibc and ld to make protected symbol to work with copy relocation.
>>>> That is protected symbol may be external, but won't be preempted.  The
>>>> price I paid is that protected symbol won't be accessed via PC-relative
>>>> relocation within the shared object.  To access protected symbol via
>>>> PC-relative relocation within the shared object, we need to disable copy
>>>> relocation in executable, which is a psABI change.  That is why I proposed
>>>> to mark the object as such so that we won't get surprise at run-time.
>>>
>>> I think what Cary's arguing (and I honestly would expect) is that copying the protected symbol *is* for all intents and purposes a preemption. I'd expect copy relocations against protected symbols to be linker errors. I guess what's missing for gcc's intended optimization is an indication to the compiler that a symbol is protected in its home library, to suppress emitting PC-relative references to a copy relocation.
>>
>> That is what I meant by "That is why I proposed to mark the object as such so
>> that we won't get surprise at run-time."
>
> I don't see why it would be a runtime failure. ld could refuse to resolve the relocation at static link time, couldn't it?
>

The link-time shared library may be very different from the run-time
shared library.  Since copy relocation works correctly against protected
symbol with the current gcc, binutils and glibc, we can't simply disallow
copy relocation against protected symbol at link-time.  It should be done
with a link command-line as well as markers on the input files.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 29, 2016, at 12:43 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> The link-time shared library may be very different from the run-time
> shared library.  Since copy relocation works correctly against protected
> symbol with the current gcc, binutils and glibc, we can't simply disallow
> copy relocation against protected symbol at link-time.

However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:51 PM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 29, 2016, at 12:43 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> The link-time shared library may be very different from the run-time
>> shared library.  Since copy relocation works correctly against protected
>> symbol with the current gcc, binutils and glibc, we can't simply disallow
>> copy relocation against protected symbol at link-time.
>
> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>

Protected symbol never worked correctly on x86 before.  My
change closed a few long-standing bugs.  There is no going-back.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Tue, Mar 29, 2016 at 12:54 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Tue, Mar 29, 2016 at 12:51 PM, Joe Groff <jgroff@apple.com> wrote:
>>
>>> On Mar 29, 2016, at 12:43 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>>
>>> The link-time shared library may be very different from the run-time
>>> shared library.  Since copy relocation works correctly against protected
>>> symbol with the current gcc, binutils and glibc, we can't simply disallow
>>> copy relocation against protected symbol at link-time.
>>
>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>
>
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

If you want to avoid copy relocation, you can do

1. Pass -z noextern-protected-data to build the shared object.
2. Use GOT to access external data in executable.


H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

On Tue, Mar 29, 2016 at 12:54:36PM -0700, H.J. Lu wrote:
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

The swift bug report and this thread should be telling you that
protected visibility symbols don't work correctly *now* on x86.

They did in fact work correctly before your changes, but had many
annoying restrictions.  Those restrictions, all due to non-PIC, can be
overcome in a number of ways without breaking shared libraries.
eg. ppc32 is non-PIC by default just like x64, but see bfd/elf32-ppc.c
	  /* Convert lis;addi or lis;load/store accessing a protected
	     variable defined in a shared library to PIC.  */
Yes, this makes ppc32 executables slower.  I'd rather that than making
shared libraries slower.

FWIW, I agree with all the points Cary makes in
https://sourceware.org/ml/binutils/2016-03/msg00331.html.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>
>
> Protected symbol never worked correctly on x86 before.  My
> change closed a few long-standing bugs.  There is no going-back.

You keep countering my arguments with assertions like, "it was a bug
and I fixed it," but you present no arguments of your own to support
your position. I'm not sure what long-standing bugs you're referring
to -- the only one I can find, PR target/65248 [1], was filed by you
yourself, so you can't really use that as support. In fact, PR
ld/15228 [2], was filed against ld for *not* refusing to make a COPY
relocation to a protected symbol, and Alan fixed that. Gold has the
same bug, and I intend to fix it there, too.

You have effectively made protected symbols "work" by disabling the
very thing they were designed to do. I don't know how to point out the
absurdity any more clearly.

<reductio ad absurdum>
By your logic, you should make GCC generate PIC code by default and
turn -fno-pic into a no-op, because non-PIC code doesn't "work" in a
shared library. (Ironically, that would actually make protected
symbols work again without your changes.) Over the years, I've heard
many complaints from developers that they couldn't put code into a
shared library without compiling it with a special option. When you
first heard that complaint, why wasn't your response to change the
compiler? Isn't that the only way to make shared libraries "work"?
</reductio ad absurdum>

-cary

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65248

[2] https://sourceware.org/bugzilla/show_bug.cgi?id=15228

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 03/29/2016 07:46 PM, Cary Coutant wrote:
>>> However, protected doesn't work this way in older binutils, or with alternative tools like llvm or gold, and it sounds like protected was never intended to work this way either. Rather If gcc is interested in pursuing this optimization, it seems more responsible to me they could investigate introducing language-level annotations that let libraries opt into the optimization, instead of unilaterally breaking things for other binutils clients and introducing new complexity to get back to the original behavior.
>>>
>>
>> Protected symbol never worked correctly on x86 before.  My
>> change closed a few long-standing bugs.  There is no going-back.
>
> You keep countering my arguments with assertions like, "it was a bug
> and I fixed it," but you present no arguments of your own to support
> your position. I'm not sure what long-standing bugs you're referring
> to -- the only one I can find, PR target/65248 [1], was filed by you
> yourself, so you can't really use that as support. In fact, PR
> ld/15228 [2], was filed against ld for *not* refusing to make a COPY
> relocation to a protected symbol, and Alan fixed that. Gold has the
> same bug, and I intend to fix it there, too.
And FWIW, there are some folks on the GCC side of things that think that 
HJ's change for 65248 is broken and needs to be reverted before gcc-6 
releases.

I'm not familiar enough with all the issues, but I am familiar enough 
with the work of HJ, Alan and yourself that if you & Alan say HJ's GCC 
change is wrong, then, well, it's wrong and needs to be reverted.

It would help me immensely on the GCC side if things if you and Alan 
could easily summarize correct behavior and the impact if we were to 
just revert HJ's change.  A testcase would be amazingly helpful too.

I'm sure I could extract the relevant info out of the thread, but I'm 
just buried right now.

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> And FWIW, there are some folks on the GCC side of things that think that
> HJ's change for 65248 is broken and needs to be reverted before gcc-6
> releases.

Thanks; I was thinking the next step was to move this discussion over
to the GCC side to see if anyone there was in agreement.

> It would help me immensely on the GCC side if things if you and Alan could
> easily summarize correct behavior and the impact if we were to just revert
> HJ's change.  A testcase would be amazingly helpful too.

On the compiler side, the main thing to check is that any reference to
a protected data symbol, whether extern, common, weak, or initialized,
is accessed with a pc-relative or GOT-relative addressing mode (i.e.,
not indirect through a GOT entry). This is the fundamental reason to
declare a data symbol protected.

The testcases in gcc.target/i386/pr65248-[1234].c look suitable to me,
if we simply switch "scan-assembler-not" with "scan-assembler"
throughout. These basically take a source like this:

__attribute__((visibility("protected")))
int xxx;

int
foo ()
{
  return xxx;
}

and compile with -fpic. Prior to GCC 5, we got

   movl    xxx(%rip), %eax

but with GCC 5, we now get

   movq    xxx@GOTPCREL(%rip), %rax
   movl    (%rax), %eax

There was another change to GCC at r218397, in Dec. 2014, to make data
access in PIE mode work more like non-PIC mode. This required linker
changes to enable the use of COPY relocations when generating PIE
output. This patch was originally developed by Sriraman Tallam on the
Google branch, but was checked on trunk in by HJ after considerable
discussion. As far as I'm concerned (and I advised Sri during the
development of this patch), this change was fine -- it simply
recognizes that a position-independent executable does not need to use
PIC-style references to data when a pc-relative or got-relative access
is possible. As in non-PIC mode, these accesses work fine for data
defined in the main program, but result in occasional COPY relocations
if the data happens to be defined in a shared library. Unfortunately,
I think this was the patch that set the rest in motion -- when
developing test cases, HJ noted that COPY relocations to protected
symbols were not allowed, and set out to fix that.

Ultimately, I'd like to see GCC support the use of declspec(dllimport)
or __attribute__((dllimport)) to declare a global variable known to be
defined externally in a shared library. It should then generate a
PIC-style reference even in non-PIC mode. This could be used to
eliminate all COPY relocations.

On the linker side, the linkers need to check that a reference from a
main program to a protected symbol will result in neither a text
relocation nor a COPY relocation, but instead issue an error message
explaining fairly clearly that a protected symbol in a shared library
cannot be referenced from non-PIC code. See PR ld/15228 and PR
gold/19823 (the latter being the bug report that started this current
thread).

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> See PR ld/15228 and PR
> gold/19823 (the latter being the bug report that started this current
> thread).

Sorry, it was actually Joe's note about Swift that started this
thread, but that came in right after the gold PR, so I was already
primed.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

OK, I'll try to list the technical issues re. pr65248 "Copy relocation
against protected symbol doesn't work".

Given a shared library that defines a variable, and a non-PIC
executable that references that variable, the linker makes a duplicate
of the variable in the executable .dynbss section and arranges to have
the copy initialized by the dynamic loader with a copy relocation.
.dynbss is a linker created section that becomes part of the
executable bss segment.  The idea is that at run-time both the
executable and the shared library will use the executable copy of the
variable.  It's a rather ancient linker hack to avoid dynamic text
relocations, invented well before symbol visibility.

The problem with protected visibility variables in shared libraries is
that the shared library continues to access its own copy.  So we have
two copies of a variable where the source only contained one variable
definition.  (It's not really the copy relocation as such that causes
a problem, but the fact that we have a duplicate in .dynbss.)

Solutions are:

1) Have the linker emit an error, suggesting that the executable needs
to be PIC to access a protected visibility variable in a shared
library.  Pass the pain on to users.

2) Emit dynamic text relocations for non-PIC executable access to
protected visibility variables defined in shared libraries.  Makes
executable pages with text relocations non-shared between processes.
(But can be shared again after relocation if two processes have the
same layout.)

3) Teach the linker to edit non-PIC to PIC.  I've done this for ppc32,
but it's not very easy for x86 due to the fact that you can't find the
start of an instruction from a relocated field.  It also results in
slow code, since PIC sequences are usually larger than non-PIC,
requiring a branch into a patch area and a branch back.

4) Have the compiler always emit PIC for external variable access.
The .dynbss hack could then die a natural death.  Coupled with linker
editing of PIC to non-PIC, you'd get correct and reasonably fast code
with perhaps a few nops.  Again, x86 has difficulty due to variable
length insns, but this could be solved with marker relocs.

5) HJ's solution.  Make the compiler emit the same code for protected
visibility variables in shared libraries as it does for default
visibility variables.  ie. shared libraries access their protected
visibility variables via the GOT, which allows the .dynbss hack to
work.  Also, modify the dynamic loader to ensure protected visibility
semantics are enforced.  (Consider the case of shared lib A that
defines protected visibility var V, and shared lib B that also defines
protected visibility var V, and an executable that references V.
Assuming A is before B in breadth-first order, then the executable and
A should use the .dynbss copy of V, but B should use its own V!)  I've
got to hand it to HJ, I think he has all this working properly to
satisfy the semantics of protected visibility.

However, shared library access of protected visibility variables is of
course no quicker than default visibility variables.  Which defeats
the main reason to use protected visibility variables in shared
libraries!  Note also that it unnecessarily slows down shared library
code using protected visibility variables that are not accessed by the
executable (but perhaps accessed from some other shared library).

Also, I don't believe there is anything to prevent a newly compiled
program from running with an old glibc, which would mean protected
visibility semantics are broken on such a system.  To be fair, they
were probably broken on such a system before the linker complained
about non-PIC protected visibility access, anyway.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> 2) Emit dynamic text relocations for non-PIC executable access to
> protected visibility variables defined in shared libraries.  Makes
> executable pages with text relocations non-shared between processes.
> (But can be shared again after relocation if two processes have the
> same layout.)

Note that this is not always possible, depending on the addressing
range of the code emitted. For example, a pc-relative access may have
sufficient range to reach a variable copied into the executable's bss,
but not sufficient range to reach the original definition in a shared
library.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Jeff Law]

On 03/30/2016 08:34 AM, Alan Modra wrote:
> OK, I'll try to list the technical issues re. pr65248 "Copy relocation
> against protected symbol doesn't work".
>
> Given a shared library that defines a variable, and a non-PIC
> executable that references that variable, the linker makes a duplicate
> of the variable in the executable .dynbss section and arranges to have
> the copy initialized by the dynamic loader with a copy relocation.
> .dynbss is a linker created section that becomes part of the
> executable bss segment.  The idea is that at run-time both the
> executable and the shared library will use the executable copy of the
> variable.  It's a rather ancient linker hack to avoid dynamic text
> relocations, invented well before symbol visibility.
Right.  Essentially we have to ensure there's one and only one copy of 
the variable and that if the shared library and executable take the 
address of the variable that the addresses compare equal.   Pretty 
standard stuff.


>
> The problem with protected visibility variables in shared libraries is
> that the shared library continues to access its own copy.  So we have
> two copies of a variable where the source only contained one variable
> definition.  (It's not really the copy relocation as such that causes
> a problem, but the fact that we have a duplicate in .dynbss.)
Which is clearly bad/wrong.  But isn't this the state we were in prior 
to HJ's change?

ie, the DSO would use non-GOT sequences to access its private copy and 
the main executable would reference the duplicate in .dynbss?  That led 
to two definitions.

After HJ's change, the DSO references via the GOT and we have a single 
instance?  Right?

The problem in that case is we've essentially allowed preemption of a 
symbol that was not supposed to be preemptable.  Worse yet, all accesses 
are going through the GOT in the DSO, which is almost certainly a 
performance penalty.

ISTM that there is a fundamental incompatibility between protected 
visibility and these copy relocations.

>
> 1) Have the linker emit an error, suggesting that the executable needs
> to be PIC to access a protected visibility variable in a shared
> library.  Pass the pain on to users.
Not ideal, but at least we'd get a nice compile/link time error rather 
than silently doing the wrong thing in one case or the other.


>
> 2) Emit dynamic text relocations for non-PIC executable access to
> protected visibility variables defined in shared libraries.  Makes
> executable pages with text relocations non-shared between processes.
> (But can be shared again after relocation if two processes have the
> same layout.)
Yea, but, ewww.


>
> 3) Teach the linker to edit non-PIC to PIC.  I've done this for ppc32,
> but it's not very easy for x86 due to the fact that you can't find the
> start of an instruction from a relocated field.  It also results in
> slow code, since PIC sequences are usually larger than non-PIC,
> requiring a branch into a patch area and a branch back.
I'm not a fan of all the rewriting we currently do in the linker; adding 
more to deal with this case doesn't seem wise.  And we'd have to 
replicate it in every target that supports copy relocations.  ugh.

>
> 4) Have the compiler always emit PIC for external variable access.
> The .dynbss hack could then die a natural death.  Coupled with linker
> editing of PIC to non-PIC, you'd get correct and reasonably fast code
> with perhaps a few nops.  Again, x86 has difficulty due to variable
> length insns, but this could be solved with marker relocs.
Maybe long term.  With the caveats about rewriting above, of course.

It seems like we'd want to stage in the linker changes first, then the 
compiler changes.


>
> 5) HJ's solution.  Make the compiler emit the same code for protected
> visibility variables in shared libraries as it does for default
> visibility variables.  ie. shared libraries access their protected
> visibility variables via the GOT, which allows the .dynbss hack to
> work.  Also, modify the dynamic loader to ensure protected visibility
> semantics are enforced.  (Consider the case of shared lib A that
> defines protected visibility var V, and shared lib B that also defines
> protected visibility var V, and an executable that references V.
> Assuming A is before B in breadth-first order, then the executable and
> A should use the .dynbss copy of V, but B should use its own V!)  I've
> got to hand it to HJ, I think he has all this working properly to
> satisfy the semantics of protected visibility.
I'm even less inclined to dump this responsibility onto the dynamic loader.

>
> However, shared library access of protected visibility variables is of
> course no quicker than default visibility variables.  Which defeats
> the main reason to use protected visibility variables in shared
> libraries!  Note also that it unnecessarily slows down shared library
> code using protected visibility variables that are not accessed by the
> executable (but perhaps accessed from some other shared library).
Right.

>
> Also, I don't believe there is anything to prevent a newly compiled
> program from running with an old glibc, which would mean protected
> visibility semantics are broken on such a system.  To be fair, they
> were probably broken on such a system before the linker complained
> about non-PIC protected visibility access, anyway.
RIght.


So in the immediate term, if we drop the problem 65248 patch, we're back 
in a state where the DSO and the executable can have two different views 
of certain objects.  In which case we really need a solution in place to 
flag that as an error, which is #1 in your list of mitigations.

Right?

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Fri, Apr 15, 2016 at 2:49 PM, Jeff Law <law@redhat.com> wrote:
>
> So in the immediate term, if we drop the problem 65248 patch, we're back in
> a state where the DSO and the executable can have two different views of
> certain objects.  In which case we really need a solution in place to flag
> that as an error, which is #1 in your list of mitigations.

From program behavior perspective, there is nothing wrong with
my changes to ld, ld.so and GCC.  The main drawback is my
changes make protected symbol pretty much useless in term
of faster access within the shared object.  All these are due to
incompatibility between protected symbol and copy relocation.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Richard Biener]

On Fri, Apr 15, 2016 at 11:56 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Fri, Apr 15, 2016 at 2:49 PM, Jeff Law <law@redhat.com> wrote:
>>
>> So in the immediate term, if we drop the problem 65248 patch, we're back in
>> a state where the DSO and the executable can have two different views of
>> certain objects.  In which case we really need a solution in place to flag
>> that as an error, which is #1 in your list of mitigations.
>
> From program behavior perspective, there is nothing wrong with
> my changes to ld, ld.so and GCC.  The main drawback is my
> changes make protected symbol pretty much useless in term
> of faster access within the shared object.  All these are due to
> incompatibility between protected symbol and copy relocation.

I agree.  There's nothing to do for GCC 6.

> Given a shared library that defines a variable, and a non-PIC
> executable that references that variable, the linker makes a duplicate
> of the variable in the executable .dynbss section and arranges to have
> the copy initialized by the dynamic loader with a copy relocation.
> .dynbss is a linker created section that becomes part of the
> executable bss segment.  The idea is that at run-time both the
> executable and the shared library will use the executable copy of the
> variable.  It's a rather ancient linker hack to avoid dynamic text
> relocations, invented well before symbol visibility.

So what other choice does the linker have here?  AFAICS it's wrong
to create the .dynbss copy for protected symbols.  So why not
simply create 'dynamic text relocations' then?  Is that possible with
a pure linker change?

That said, correctness trumps optimization.  A correctness fix that
works with old objects trumps one that requires a compiler change.
Requiring a compiler change to get back optimization while preserving
correctness is fine.

To summarize: there is currently no testcase for a wrong-code issue
because there is no wrong-code issue.  There's a missed-optimization
issue (the library is less optimized) but if we fix that by revering HJs
patch we _do_ have a testcase for a wrong-code issue that re-appears.

=> nothing to block GCC 6 which behaves strictly better than GCC 5 here.

Richard.

>
> --
> H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Alan Modra]

On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
> To summarize: there is currently no testcase for a wrong-code issue
> because there is no wrong-code issue.

That depends entirely on how far you are willing to bend the ELF gABI.

Any testcase the takes the address of a protected visibility variable
defined in a shared library now can get the wrong answer, since you
can argue that any address outside the shared library is wrong
according to the gABI.

I expect you can also write a testcase using a const protected var in
a shared library that ought to segfault on writing to the var from
code within the shared library, that now merrily writes to a .dynbss
copy.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Mon, Apr 18, 2016 at 7:49 AM, Alan Modra <amodra@gmail.com> wrote:
> On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
>> To summarize: there is currently no testcase for a wrong-code issue
>> because there is no wrong-code issue.
>
> That depends entirely on how far you are willing to bend the ELF gABI.
>
> Any testcase the takes the address of a protected visibility variable
> defined in a shared library now can get the wrong answer, since you
> can argue that any address outside the shared library is wrong
> according to the gABI.
>

What value should you get when taking an address of a function
with protected visibility inside the shared library?  Should it be
pointed to the function body or its PLT entry in the executable
which calls the function?


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Maciej W. Rozycki]

On Mon, 18 Apr 2016, H.J. Lu wrote:

> > Any testcase the takes the address of a protected visibility variable
> > defined in a shared library now can get the wrong answer, since you
> > can argue that any address outside the shared library is wrong
> > according to the gABI.
> 
> What value should you get when taking an address of a function
> with protected visibility inside the shared library?  Should it be
> pointed to the function body or its PLT entry in the executable
> which calls the function?

 I think it has to be an address within the shared library itself, that is 
-- depending on the particular psABI -- either the function body or its 
corresponding PLT entry in the DSO, but not in the main executable.  The 
reason is DSO code (also handcoded assembly) may reasonably expect to be 
able to load the address with a PC-relative load-address type instruction 
(ADDIUPC, LEA, MOVAB, etc.) and the target may not even have suitable 
dynamic relocations available to apply any load-time fixup if the symbol 
referred turns up outside of the DSO.  The instruction used may have a 
PC-relative range limit too.

  Maciej

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Mon, Apr 18, 2016 at 10:03 AM, Maciej W. Rozycki <macro@imgtec.com> wrote:
> On Mon, 18 Apr 2016, H.J. Lu wrote:
>
>> > Any testcase the takes the address of a protected visibility variable
>> > defined in a shared library now can get the wrong answer, since you
>> > can argue that any address outside the shared library is wrong
>> > according to the gABI.
>>
>> What value should you get when taking an address of a function
>> with protected visibility inside the shared library?  Should it be
>> pointed to the function body or its PLT entry in the executable
>> which calls the function?
>
>  I think it has to be an address within the shared library itself, that is
> -- depending on the particular psABI -- either the function body or its
> corresponding PLT entry in the DSO, but not in the main executable.  The

Sorry,  this isn't how function pointer works on x86.

> reason is DSO code (also handcoded assembly) may reasonably expect to be
> able to load the address with a PC-relative load-address type instruction
> (ADDIUPC, LEA, MOVAB, etc.) and the target may not even have suitable
> dynamic relocations available to apply any load-time fixup if the symbol
> referred turns up outside of the DSO.  The instruction used may have a
> PC-relative range limit too.
>

That is why protected visibility is such a mess.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Michael Matz]

Hi,

On Mon, 18 Apr 2016, H.J. Lu wrote:

> > reason is DSO code (also handcoded assembly) may reasonably expect to 
> > be able to load the address with a PC-relative load-address type 
> > instruction (ADDIUPC, LEA, MOVAB, etc.) and the target may not even 
> > have suitable dynamic relocations available to apply any load-time 
> > fixup if the symbol referred turns up outside of the DSO.  The 
> > instruction used may have a PC-relative range limit too.
> 
> That is why protected visibility is such a mess.

Not mess, but it comes with certain limitations.  And that's okay.  It's 
intended as an optimization, and it should do that optimization if 
requested, and error out if it can't be done for whatever reason.

E.g. one limitation might very well be that function pointer comparison 
for protected functions doesn't work (gives different outcomes if the 
pointer is built from inside the exe or from a shared lib).  (No matter 
how it's built, it will still _work_ when called).  Alternatively we can 
make comparison work (by using the exe PLT slot), in which case Alans 
testcase will need more complications to show that protected visibility 
currently is broken.  Alans testcase will work right now (as in showing 
protected being broken) on data symbols.


Ciao,
Michael.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Mon, Apr 18, 2016 at 10:23 AM, Michael Matz <matz@suse.de> wrote:
> Hi,
>
> On Mon, 18 Apr 2016, H.J. Lu wrote:
>
>> > reason is DSO code (also handcoded assembly) may reasonably expect to
>> > be able to load the address with a PC-relative load-address type
>> > instruction (ADDIUPC, LEA, MOVAB, etc.) and the target may not even
>> > have suitable dynamic relocations available to apply any load-time
>> > fixup if the symbol referred turns up outside of the DSO.  The
>> > instruction used may have a PC-relative range limit too.
>>
>> That is why protected visibility is such a mess.
>
> Not mess, but it comes with certain limitations.  And that's okay.  It's
> intended as an optimization, and it should do that optimization if
> requested, and error out if it can't be done for whatever reason.
>
> E.g. one limitation might very well be that function pointer comparison
> for protected functions doesn't work (gives different outcomes if the
> pointer is built from inside the exe or from a shared lib).  (No matter
> how it's built, it will still _work_ when called).  Alternatively we can
> make comparison work (by using the exe PLT slot), in which case Alans
> testcase will need more complications to show that protected visibility
> currently is broken.  Alans testcase will work right now (as in showing
> protected being broken) on data symbols.
>

We have special treatment for pointer of protected function symbol
in ld and ld.so from day one, which, BTW, disables optimization of
pointer of protected function symbol inside the shared library.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Jakub Jelinek]

On Mon, Apr 18, 2016 at 10:27:45AM -0700, H.J. Lu wrote:
> On Mon, Apr 18, 2016 at 10:23 AM, Michael Matz <matz@suse.de> wrote:
> > Hi,
> >
> > On Mon, 18 Apr 2016, H.J. Lu wrote:
> >
> >> > reason is DSO code (also handcoded assembly) may reasonably expect to
> >> > be able to load the address with a PC-relative load-address type
> >> > instruction (ADDIUPC, LEA, MOVAB, etc.) and the target may not even
> >> > have suitable dynamic relocations available to apply any load-time
> >> > fixup if the symbol referred turns up outside of the DSO.  The
> >> > instruction used may have a PC-relative range limit too.
> >>
> >> That is why protected visibility is such a mess.
> >
> > Not mess, but it comes with certain limitations.  And that's okay.  It's
> > intended as an optimization, and it should do that optimization if
> > requested, and error out if it can't be done for whatever reason.
> >
> > E.g. one limitation might very well be that function pointer comparison
> > for protected functions doesn't work (gives different outcomes if the
> > pointer is built from inside the exe or from a shared lib).  (No matter
> > how it's built, it will still _work_ when called).  Alternatively we can
> > make comparison work (by using the exe PLT slot), in which case Alans
> > testcase will need more complications to show that protected visibility
> > currently is broken.  Alans testcase will work right now (as in showing
> > protected being broken) on data symbols.
> >
> 
> We have special treatment for pointer of protected function symbol
> in ld and ld.so from day one, which, BTW, disables optimization of
> pointer of protected function symbol inside the shared library.

But maybe that is the mistake.  Doing this makes protected visibility
no longer a userful optimization for anything, it is usually more expensive
than normal visibility, so it is generally a pessimization nobody should
really use.
Compared to this, having a protected-like visibility which doesn't care
about function pointer comparisons would be generally useful to many
projects, and e.g. glibc is heavily using it itself (using hacked up
macros).  Generally it could be even implementable just on the compiler side
and leave the badly designed "protected" to keep what it used to do (i.e.
revert the change) and hope or actively suggest to users that if they ever
think of this "protected" visibility, they are always doing something wrong.

	Jakub

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Mon, Apr 18, 2016 at 11:51 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Mon, Apr 18, 2016 at 10:27:45AM -0700, H.J. Lu wrote:
>> On Mon, Apr 18, 2016 at 10:23 AM, Michael Matz <matz@suse.de> wrote:
>> > Hi,
>> >
>> > On Mon, 18 Apr 2016, H.J. Lu wrote:
>> >
>> >> > reason is DSO code (also handcoded assembly) may reasonably expect to
>> >> > be able to load the address with a PC-relative load-address type
>> >> > instruction (ADDIUPC, LEA, MOVAB, etc.) and the target may not even
>> >> > have suitable dynamic relocations available to apply any load-time
>> >> > fixup if the symbol referred turns up outside of the DSO.  The
>> >> > instruction used may have a PC-relative range limit too.
>> >>
>> >> That is why protected visibility is such a mess.
>> >
>> > Not mess, but it comes with certain limitations.  And that's okay.  It's
>> > intended as an optimization, and it should do that optimization if
>> > requested, and error out if it can't be done for whatever reason.
>> >
>> > E.g. one limitation might very well be that function pointer comparison
>> > for protected functions doesn't work (gives different outcomes if the
>> > pointer is built from inside the exe or from a shared lib).  (No matter
>> > how it's built, it will still _work_ when called).  Alternatively we can
>> > make comparison work (by using the exe PLT slot), in which case Alans
>> > testcase will need more complications to show that protected visibility
>> > currently is broken.  Alans testcase will work right now (as in showing
>> > protected being broken) on data symbols.
>> >
>>
>> We have special treatment for pointer of protected function symbol
>> in ld and ld.so from day one, which, BTW, disables optimization of
>> pointer of protected function symbol inside the shared library.
>
> But maybe that is the mistake.  Doing this makes protected visibility
> no longer a userful optimization for anything, it is usually more expensive
> than normal visibility, so it is generally a pessimization nobody should
> really use.
> Compared to this, having a protected-like visibility which doesn't care
> about function pointer comparisons would be generally useful to many
> projects, and e.g. glibc is heavily using it itself (using hacked up
> macros).  Generally it could be even implementable just on the compiler side
> and leave the badly designed "protected" to keep what it used to do (i.e.
> revert the change) and hope or actively suggest to users that if they ever
> think of this "protected" visibility, they are always doing something wrong.
>

It is a good idea to revisit the whole protected visibility issue on x86
in terms of C/C++ languages, x86 psABIs, compiler, ld and ld.so.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Cary Coutant]

>> That is why protected visibility is such a mess.
>
> Not mess, but it comes with certain limitations.  And that's okay.  It's
> intended as an optimization, and it should do that optimization if
> requested, and error out if it can't be done for whatever reason.

I completely agree.

> E.g. one limitation might very well be that function pointer comparison
> for protected functions doesn't work (gives different outcomes if the
> pointer is built from inside the exe or from a shared lib).  (No matter
> how it's built, it will still _work_ when called).  Alternatively we can
> make comparison work (by using the exe PLT slot), in which case Alans
> testcase will need more complications to show that protected visibility
> currently is broken.  Alans testcase will work right now (as in showing
> protected being broken) on data symbols.

Function pointer comparison is also a mess, and is the only reason why
the treatment for protected function symbols is so complicated. It all
boils down the the language guarantees that (a) the address of a
function must be unique, (b) that the address of a given function must
always be the same value, and (c) that these guarantees survive a
conversion to void*.

I'd argue that all of these language guarantees are poor choices. Just
like constant strings, which are allowed to be pooled, two identical
functions ought to be allowed to be pooled (folded). If function
pointer comparison were restricted to function pointer types, we could
allow the address of a function to yield the address of a PLT entry,
and use a deep comparison to decide whether two unequal function
pointers were in fact equivalent.

But that's another topic for another day.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Jeff Law]

On 04/18/2016 11:55 AM, Cary Coutant wrote:
>>> That is why protected visibility is such a mess.
>>
>> Not mess, but it comes with certain limitations.  And that's okay.  It's
>> intended as an optimization, and it should do that optimization if
>> requested, and error out if it can't be done for whatever reason.
>
> I completely agree.
ISTM this ought to be the guiding principle here, with the additional 
caveat that if one of the limitations is tickled that we issue a good 
diagnostic.

The current situation (gcc-5, gcc-6-rc) essentially de-optimizes 
protected systems in an attempt to work around the various limitations 
of protected symbols.  Reverting that change is, IMHO, what needs to 
happen.  My worry is that we're so damn late in the gcc-6 cycle that it 
may need to be deferred to 6.2 or beyond.

Jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Mon, Apr 25, 2016 at 10:24 AM, Jeff Law <law@redhat.com> wrote:
> On 04/18/2016 11:55 AM, Cary Coutant wrote:
>>>>
>>>> That is why protected visibility is such a mess.
>>>
>>>
>>> Not mess, but it comes with certain limitations.  And that's okay.  It's
>>> intended as an optimization, and it should do that optimization if
>>> requested, and error out if it can't be done for whatever reason.
>>
>>
>> I completely agree.
>
> ISTM this ought to be the guiding principle here, with the additional caveat
> that if one of the limitations is tickled that we issue a good diagnostic.
>
> The current situation (gcc-5, gcc-6-rc) essentially de-optimizes protected
> systems in an attempt to work around the various limitations of protected
> symbols.  Reverting that change is, IMHO, what needs to happen.  My worry is
> that we're so damn late in the gcc-6 cycle that it may need to be deferred
> to 6.2 or beyond.

Please keep in mind that many limitations can only be detected at
link-time or run-time, which are yet to be implemented,  not at
compile-time.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Maciej W. Rozycki]

On Mon, 18 Apr 2016, Michael Matz wrote:

> E.g. one limitation might very well be that function pointer comparison 
> for protected functions doesn't work (gives different outcomes if the 
> pointer is built from inside the exe or from a shared lib).  (No matter 
> how it's built, it will still _work_ when called).  Alternatively we can 
> make comparison work (by using the exe PLT slot), in which case Alans 
> testcase will need more complications to show that protected visibility 
> currently is broken.  Alans testcase will work right now (as in showing 
> protected being broken) on data symbols.

 The way it works in the original MIPS SVR4 psABI is by using the relevant 
GOT entry's contents as the pointer, disabling lazy binding for any 
function symbols whose value is used as data rather than to make a call 
(no lazy binding stub is simply produced).  It's easy in that psABI 
because all code is PIC, even in executables -- which fulfils Cary's 
earlier postulate for protected symbol accesses.

 For non-PIC code the necessary arrangement can be made by the compiler 
based on symbol annotation (also proposed by Cary), or failing that a 
link-time fixup can be made, possibly branching to a thunk generated out 
of line if the sequence required to load a GOT entry is longer than the 
original absolute sequence (proposed by Alan).  I think this approach 
should work with x86 even, as its branch instruction has a single-byte 
opcode and a signed 32-bit span, so it certainly does not require more 
code space than any relocated instruction using the absolute or 
PC-relative addressing mode.

  Maciej

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Alan Modra]

On Mon, Apr 18, 2016 at 07:59:50AM -0700, H.J. Lu wrote:
> On Mon, Apr 18, 2016 at 7:49 AM, Alan Modra <amodra@gmail.com> wrote:
> > On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
> >> To summarize: there is currently no testcase for a wrong-code issue
> >> because there is no wrong-code issue.

I've added a testcase at
https://sourceware.org/bugzilla/show_bug.cgi?id=19965#c3
that shows the address problem (&x != x) with older gcc *or* older
glibc, and shows the program behaviour problem with current
binutils+gcc+glibc.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Richard Biener]

On Tue, Apr 19, 2016 at 7:08 AM, Alan Modra <amodra@gmail.com> wrote:
> On Mon, Apr 18, 2016 at 07:59:50AM -0700, H.J. Lu wrote:
>> On Mon, Apr 18, 2016 at 7:49 AM, Alan Modra <amodra@gmail.com> wrote:
>> > On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
>> >> To summarize: there is currently no testcase for a wrong-code issue
>> >> because there is no wrong-code issue.
>
> I've added a testcase at
> https://sourceware.org/bugzilla/show_bug.cgi?id=19965#c3
> that shows the address problem (&x != x) with older gcc *or* older
> glibc, and shows the program behaviour problem with current
> binutils+gcc+glibc.

Thanks.

So with all this it sounds that current protected visibility is just broken
and we should forgo with it, making it equal to default visibility?

At least I couldn't decipher a solution that solves all of the issues
with protected visibility apart from trying to error at link-time
(or runtime?) for the cases that are tricky (impossible?) to solve.

glibc uses "protected visibility" via its using of local aliases, correct?
But it doesn't use anything like that for data symbols?

Richard.

> --
> Alan Modra
> Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Szabolcs Nagy]

On 19/04/16 09:20, Richard Biener wrote:
> On Tue, Apr 19, 2016 at 7:08 AM, Alan Modra <amodra@gmail.com> wrote:
>> On Mon, Apr 18, 2016 at 07:59:50AM -0700, H.J. Lu wrote:
>>> On Mon, Apr 18, 2016 at 7:49 AM, Alan Modra <amodra@gmail.com> wrote:
>>>> On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
>>>>> To summarize: there is currently no testcase for a wrong-code issue
>>>>> because there is no wrong-code issue.
>>
>> I've added a testcase at
>> https://sourceware.org/bugzilla/show_bug.cgi?id=19965#c3
>> that shows the address problem (&x != x) with older gcc *or* older
>> glibc, and shows the program behaviour problem with current
>> binutils+gcc+glibc.
> 
> Thanks.
> 
> So with all this it sounds that current protected visibility is just broken
> and we should forgo with it, making it equal to default visibility?
> 

the test cases pass for me on musl libc,
it's just a glibc dynamic linker bug
that it does not handle extern protected
visibility correctly.

> At least I couldn't decipher a solution that solves all of the issues
> with protected visibility apart from trying to error at link-time
> (or runtime?) for the cases that are tricky (impossible?) to solve.
> 
> glibc uses "protected visibility" via its using of local aliases, correct?
> But it doesn't use anything like that for data symbols?
> 
> Richard.
> 
>> --
>> Alan Modra
>> Australia Development Lab, IBM
> 

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Michael Matz]

Hi,

On Tue, 19 Apr 2016, Richard Biener wrote:

> So with all this it sounds that current protected visibility is just 
> broken and we should forgo with it, making it equal to default 
> visibility?

Like how?  You mean in GCC regarding protected as default visibility?  No, 
that's just throwing out the baby with the water.  We should make 
protected do what it was intended to do and accept that not all invariants 
that are true for default visible symbols are also true for protected 
symbols, possibly by ...

> At least I couldn't decipher a solution that solves all of the issues 
> with protected visibility apart from trying to error at link-time (or 
> runtime?) for the cases that are tricky (impossible?) to solve.

... this.


Ciao,
Michael.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Cary Coutant]

>> So with all this it sounds that current protected visibility is just
>> broken and we should forgo with it, making it equal to default
>> visibility?
>
> Like how?  You mean in GCC regarding protected as default visibility?  No,
> that's just throwing out the baby with the water.  We should make
> protected do what it was intended to do and accept that not all invariants
> that are true for default visible symbols are also true for protected
> symbols, possibly by ...
>
>> At least I couldn't decipher a solution that solves all of the issues
>> with protected visibility apart from trying to error at link-time (or
>> runtime?) for the cases that are tricky (impossible?) to solve.
>
> ... this.

Right. Protected visibility worked fine without copy relocations for
15 years until HJ's patch. I don't know of anyone with a legitimate
complaint about that until HJ filed a bug based on his artificial test
case.

Other compilers implement protected the way it was intended, so the
linkers must still disallow copy relocations against protected
symbols, or we get legitimate complaints like these:

   https://sourceware.org/bugzilla/show_bug.cgi?id=15228
   https://sourceware.org/ml/binutils/2016-03/msg00312.html

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Tue, Apr 19, 2016 at 8:37 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>> So with all this it sounds that current protected visibility is just
>>> broken and we should forgo with it, making it equal to default
>>> visibility?
>>
>> Like how?  You mean in GCC regarding protected as default visibility?  No,
>> that's just throwing out the baby with the water.  We should make
>> protected do what it was intended to do and accept that not all invariants
>> that are true for default visible symbols are also true for protected
>> symbols, possibly by ...
>>
>>> At least I couldn't decipher a solution that solves all of the issues
>>> with protected visibility apart from trying to error at link-time (or
>>> runtime?) for the cases that are tricky (impossible?) to solve.
>>
>> ... this.
>
> Right. Protected visibility worked fine without copy relocations for
> 15 years until HJ's patch. I don't know of anyone with a legitimate
> complaint about that until HJ filed a bug based on his artificial test
> case.

Cary, please stop spreading the incorrect information.   There is
at lease one GCC bug against protected symbol:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012

which was reported by other people.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Tue, Apr 19, 2016 at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Tue, Apr 19, 2016 at 8:37 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> So with all this it sounds that current protected visibility is just
>>>> broken and we should forgo with it, making it equal to default
>>>> visibility?
>>>
>>> Like how?  You mean in GCC regarding protected as default visibility?  No,
>>> that's just throwing out the baby with the water.  We should make
>>> protected do what it was intended to do and accept that not all invariants
>>> that are true for default visible symbols are also true for protected
>>> symbols, possibly by ...
>>>
>>>> At least I couldn't decipher a solution that solves all of the issues
>>>> with protected visibility apart from trying to error at link-time (or
>>>> runtime?) for the cases that are tricky (impossible?) to solve.
>>>
>>> ... this.
>>
>> Right. Protected visibility worked fine without copy relocations for
>> 15 years until HJ's patch. I don't know of anyone with a legitimate
>> complaint about that until HJ filed a bug based on his artificial test
>> case.
>
> Cary, please stop spreading the incorrect information.   There is
> at lease one GCC bug against protected symbol:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>
> which was reported by other people.

I opened this bug in 2005:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=19520

Richard opened:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51880

in 2012.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Tue, Apr 19, 2016 at 8:52 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Tue, Apr 19, 2016 at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>> On Tue, Apr 19, 2016 at 8:37 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>>> So with all this it sounds that current protected visibility is just
>>>>> broken and we should forgo with it, making it equal to default
>>>>> visibility?
>>>>
>>>> Like how?  You mean in GCC regarding protected as default visibility?  No,
>>>> that's just throwing out the baby with the water.  We should make
>>>> protected do what it was intended to do and accept that not all invariants
>>>> that are true for default visible symbols are also true for protected
>>>> symbols, possibly by ...
>>>>
>>>>> At least I couldn't decipher a solution that solves all of the issues
>>>>> with protected visibility apart from trying to error at link-time (or
>>>>> runtime?) for the cases that are tricky (impossible?) to solve.
>>>>
>>>> ... this.
>>>
>>> Right. Protected visibility worked fine without copy relocations for
>>> 15 years until HJ's patch. I don't know of anyone with a legitimate
>>> complaint about that until HJ filed a bug based on his artificial test
>>> case.
>>
>> Cary, please stop spreading the incorrect information.   There is
>> at lease one GCC bug against protected symbol:
>>
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>
>> which was reported by other people.
>
> I opened this bug in 2005:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=19520
>
> Richard opened:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51880
>
> in 2012.
>

Another old bug:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=10908

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Cary Coutant]

> Another old bug:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=10908

Filed by you, and resolved (correctly) as invalid. Again, the real
problem was the lack of a linker diagnostic.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Tue, Apr 19, 2016 at 8:58 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> Another old bug:
>>
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=10908
>
> Filed by you, and resolved (correctly) as invalid. Again, the real
> problem was the lack of a linker diagnostic.
>

It doesn't make what you said was true.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Cary Coutant]

> Cary, please stop spreading the incorrect information.   There is
> at lease one GCC bug against protected symbol:
>
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>
> which was reported by other people.

OK, so it got reported once by someone else. But that bug was based on
an incorrect understanding of protected visibility; the real problem
there was the lack of diagnostic from the linker.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[H.J. Lu]

On Tue, Apr 19, 2016 at 7:06 AM, Michael Matz <matz@suse.de> wrote:
> Hi,
>
> On Tue, 19 Apr 2016, Richard Biener wrote:
>
>> So with all this it sounds that current protected visibility is just
>> broken and we should forgo with it, making it equal to default
>> visibility?
>
> Like how?  You mean in GCC regarding protected as default visibility?  No,
> that's just throwing out the baby with the water.  We should make
> protected do what it was intended to do and accept that not all invariants
> that are true for default visible symbols are also true for protected
> symbols, possibly by ...
>
>> At least I couldn't decipher a solution that solves all of the issues
>> with protected visibility apart from trying to error at link-time (or
>> runtime?) for the cases that are tricky (impossible?) to solve.
>

Protected visibility is a useful feature.  But as it stands today,
it is pretty much useless on x86 as seen in ld and ld.so.  We
have known this defect for a long time, almost from day 1.  To
make it truly useful, we need to clearly spell out how and when
it can be used.  We should enforce its limitation in compiler,
ld and ld.so so that there is no surprise, either for correctness or
performance,  at run-time.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Rich Felker]

On Tue, Apr 19, 2016 at 12:10:58PM -0700, H.J. Lu wrote:
> On Tue, Apr 19, 2016 at 7:06 AM, Michael Matz <matz@suse.de> wrote:
> > Hi,
> >
> > On Tue, 19 Apr 2016, Richard Biener wrote:
> >
> >> So with all this it sounds that current protected visibility is just
> >> broken and we should forgo with it, making it equal to default
> >> visibility?
> >
> > Like how?  You mean in GCC regarding protected as default visibility?  No,
> > that's just throwing out the baby with the water.  We should make
> > protected do what it was intended to do and accept that not all invariants
> > that are true for default visible symbols are also true for protected
> > symbols, possibly by ...
> >
> >> At least I couldn't decipher a solution that solves all of the issues
> >> with protected visibility apart from trying to error at link-time (or
> >> runtime?) for the cases that are tricky (impossible?) to solve.
> 
> Protected visibility is a useful feature.  But as it stands today,
> it is pretty much useless on x86 as seen in ld and ld.so.  We
> have known this defect for a long time, almost from day 1.  To
> make it truly useful, we need to clearly spell out how and when
> it can be used.  We should enforce its limitation in compiler,
> ld and ld.so so that there is no surprise, either for correctness or
> performance,  at run-time.

I believe protected visibility is useful with the current/correct
semantics. At the request of several people who've seen it, I'm
reposting the following reply I just made to the start of this thread,
since my original CC list seems to have omitted a lot of people
currently involved in the discussion:


As one of the strong advocates for the fix that was made to make
protected visibility work correctly with data symbols, I'd like to
explain why it was the right decision and why it matters. This whole
process is really frustrating to me -- having invested a lot of effort
into getting something important fixed, only to have people come
trying to break it again -- but I'm going to try to be calm and not to
snap at anybody. I was only vaguely aware of this thread until a few
days ago (because I've been trying to make new progress on other
things rather than revisit issues I thought were closed), so I'm just
replying to the beginning of this thread. I'll also try to comment on
particularly important points elsewhere in the thread as I read it,
but I don't want to pile on a bunch of scattered replies that focus on
small details I think others have gotten wrong and ignore the big
picture.

From a programming standpoint, the semantics of protected visibility
need to be free of arch-specific implementation details. Otherwise
programmers can't use it without hard-coding arch-specific details,
which for practical purposes, means good software can't use it at all.

My original motivation for wanting protected visibility to "just work"
was to be able to use:

	#pragma GCC visibility push(protected)

around the inclusion of a library's public headers when including them
from the implementation files, This is far from being the only usage
case, and I'll expand on more important usage cases below, but it is
an important one because it allows you to eliminate all GOT/PLT cost
of intra-library function calls without any fine-grained maintenance
of which declarations to apply visibility too (and without any
GNUC-specific clutter in the header files themselves).

I understand that some people want protected visibility to avoid the
GOT for data symbols too for the sake of performance, but for my usage
case, the fact that the semantics were wrong for data symbols meant
that my configure check for "does protected visibility work" would
return "no", and the whole optimization would get turned off.

Anyway, let's move past optimization, because it's a distraction.
After all, with the old (broken) behavior of protected data, one
_could_ work around the above problem and still get the performance
benefits for functions without breaking data by explicitly declaring
all data with default visibility. In fact, this is how I solve the
problem in musl libc, where there are only a small number of data
symbols that should be externally accessible, and maintaining a list
of them is managable:

http://git.musl-libc.org/cgit/musl/tree/src/internal/vis.h?id=d1b29c2a54588401494c1a3ac7103c1e91c61fa1

This is done for the sake of compatibility with a wide range of
toolchains including ones with the old/broken behavior for protected
data.

The actual documented purpose of protected visibility is to prevent
other definitions of a symbol from taking precedence over the one in
the library itself. For example, suppose you have the following
situation: mainapp depends on libA which defines symbol foo with
normal visibility, and libA depends on libB, which also defines foo,
but intentionally with protected visibility so that libB always uses
its own definition, not the one from libA. There is no reasonable way
to obtain the desired semantics here without the current/correct
behavior for protected data. Any other approaches I'm aware of would
either allow libB to bind to the wrong definition of foo, or would
prevent another main app which links libB (but not libA) from being
able to use the symbol foo from libB.

On the other hand, there are plenty of other ways to get the
old/broken behavior if desired. The easiest is to simply use hidden
visibility when you don't want the symbol to be accessible outside the
library. If you _do_ want it to be visible to and usable by other
shared libraries, just not main-programs, this can be achieved using a
hidden alias for a default-visibility symbol:

	int foo;
	extern int fast_foo __attribute__((__alias__("foo"),
	                                   __visibility__("hidden")));

I expressed that here explicitly for the sake of clarity but of course
in practice people use things like the glibc macro-maze to do this
kind of binding to hidden aliases.

Rich

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Cary Coutant]

> As one of the strong advocates for the fix that was made to make
> protected visibility work correctly with data symbols, I'd like to
> explain why it was the right decision and why it matters. This whole
> process is really frustrating to me -- having invested a lot of effort
> into getting something important fixed, only to have people come
> trying to break it again -- but I'm going to try to be calm and not to
> snap at anybody.

Ironically, you've just described my feelings almost exactly, only I
come here finding that someone already broke it, and I'm trying to get
it fixed again.

With all due respect, I think you're misinterpreting what the
visibility feature was intended for, and you're projecting your own
needs onto everyone else that uses this feature. I can state with
first-hand knowledge that the intent was so that compilers could
optimize access to the protected data symbols based on an assumption
that they are "relatively nearby". Here's a quote from the gABI
proposal (the latest revision that I could find, dated April 16,
1999), which was submitted by Jim Dehnert, then at SGI:

"Optimization Note:

"The visibility semantics of these attributes allow various
optimizations. While care must be taken to maintain
position-independence and proper GOT usage for references to and
definitions of symbols which might be preempted by or referenced from
other components, these restrictions all allow references from the
same component to make stricter assumptions about the definitions.
References to protected symbols (and hence to hidden or internal
symbols) may be optimized by using absolute or PC-relative addresses
in executable files or by assuming addresses to be relatively nearby.
Internal functions (as defined in the MIPS ABI) do not normally
require gp establishment code even in psABIs requiring callee
establishment/restore of gp, because they will always be entered from
the same component with the correct gp already in place from the
caller."

Unfortunately, this optimization note didn't make it into the gABI,
probably because the editor felt it was unnecessary, and because it
contained some MIPS-specific details. Nevertheless, it clearly shows
the intent.

> From a programming standpoint, the semantics of protected visibility
> need to be free of arch-specific implementation details. Otherwise
> programmers can't use it without hard-coding arch-specific details,
> which for practical purposes, means good software can't use it at all.

It's unfortunate that copy relocations intrude on your stated goal,
but I'd prefer to fix that problem without breaking (and I do mean
"break") the original intent behind protected visibility.

> My original motivation for wanting protected visibility to "just work"
> was to be able to use:
>
>         #pragma GCC visibility push(protected)
>
> around the inclusion of a library's public headers when including them
> from the implementation files, This is far from being the only usage
> case, and I'll expand on more important usage cases below, but it is
> an important one because it allows you to eliminate all GOT/PLT cost
> of intra-library function calls without any fine-grained maintenance
> of which declarations to apply visibility too (and without any
> GNUC-specific clutter in the header files themselves).
>
> I understand that some people want protected visibility to avoid the
> GOT for data symbols too for the sake of performance, but for my usage
> case, the fact that the semantics were wrong for data symbols meant
> that my configure check for "does protected visibility work" would
> return "no", and the whole optimization would get turned off.

Yes, some people do want protected visibility to avoid the GOT for
data symbols, and it makes a significant difference in many cases. For
those people, the changes I'm objecting to cause a performance
regression.

> Anyway, let's move past optimization, because it's a distraction.

I disagree. Optimization isn't a distraction -- it's the whole
motivation for the feature.

> After all, with the old (broken) behavior of protected data, one
> _could_ work around the above problem and still get the performance
> benefits for functions without breaking data by explicitly declaring
> all data with default visibility. In fact, this is how I solve the
> problem in musl libc, where there are only a small number of data
> symbols that should be externally accessible, and maintaining a list
> of them is managable:
>
> http://git.musl-libc.org/cgit/musl/tree/src/internal/vis.h?id=d1b29c2a54588401494c1a3ac7103c1e91c61fa1
>
> This is done for the sake of compatibility with a wide range of
> toolchains including ones with the old/broken behavior for protected
> data.

s/broken/correct/ :-)

Given that, now you can have efficient access to the symbols that
aren't externally accessible, and the symbols that are accessible are
marked correctly with default visibility, right? So what's the
problem? Why would you want to give up the efficient direct access to
the symbols that can remain protected?

It looks to me like you have a solution, and it's compatible with the
intent behind the feature.

> The actual documented purpose of protected visibility is to prevent
> other definitions of a symbol from taking precedence over the one in
> the library itself.

Um, I'd say that's the documented *meaning* -- the *purpose* is to
enable compiler optimizations.

> For example, suppose you have the following
> situation: mainapp depends on libA which defines symbol foo with
> normal visibility, and libA depends on libB, which also defines foo,
> but intentionally with protected visibility so that libB always uses
> its own definition, not the one from libA. There is no reasonable way
> to obtain the desired semantics here without the current/correct
> behavior for protected data. Any other approaches I'm aware of would
> either allow libB to bind to the wrong definition of foo, or would
> prevent another main app which links libB (but not libA) from being
> able to use the symbol foo from libB.

If libA and libB both export symbol foo (protected or not), you're
playing games with the linker. This is not good programming practice.
It was never our intent to enable developers to play linker games.

At any rate, I fail to see how you get unexpected semantics either
way. With the old (correct to me) behavior, the compiler and linker
can bind the references from within libB to its own foo, and the
dynamic loader never has to get involved. With the new behavior, the
compiler and linker must leave the GOT-indirect accesses to foo, and
rely on the dynamic loader to properly resolve the reference (and
introducing quite a bit of additional complexity in the process).

In addition, libB probably ought to be making foo hidden, not protected.

> On the other hand, there are plenty of other ways to get the
> old/broken behavior if desired. The easiest is to simply use hidden
> visibility when you don't want the symbol to be accessible outside the
> library. If you _do_ want it to be visible to and usable by other
> shared libraries, just not main-programs, this can be achieved using a
> hidden alias for a default-visibility symbol:
>
>         int foo;
>         extern int fast_foo __attribute__((__alias__("foo"),
>                                            __visibility__("hidden")));

"Just not main-programs." If you're willing to ignore main programs,
the copy relocations don't matter either, and what was the issue
again? A hidden alias makes the problem even worse, because now the
linker can't even diagnose the error when it makes a copy relocation
-- you'll simply get one set of code using the copy, and another set
of code using the original (equally broken with old and new behavior).

> I expressed that here explicitly for the sake of clarity but of course
> in practice people use things like the glibc macro-maze to do this
> kind of binding to hidden aliases.

But why go through that, when protected visibility does (used to do)
exactly the right thing?

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[anonymous]

H.J. Lu wrote:
> On Tue, Apr 19, 2016 at 7:06 AM, Michael Matz <matz@suse.de> wrote:
>> Hi,
>>
>> On Tue, 19 Apr 2016, Richard Biener wrote:
>>
>>> So with all this it sounds that current protected visibility is just
>>> broken and we should forgo with it, making it equal to default
>>> visibility?
>> Like how?  You mean in GCC regarding protected as default visibility?  No,
>> that's just throwing out the baby with the water.  We should make
>> protected do what it was intended to do and accept that not all invariants
>> that are true for default visible symbols are also true for protected
>> symbols, possibly by ...
>>
>>> At least I couldn't decipher a solution that solves all of the issues
>>> with protected visibility apart from trying to error at link-time (or
>>> runtime?) for the cases that are tricky (impossible?) to solve.
> 
> Protected visibility is a useful feature.  But as it stands today,
> it is pretty much useless on x86 as seen in ld and ld.so.  We
> have known this defect for a long time, almost from day 1.  To
> make it truly useful, we need to clearly spell out how and when
> it can be used.  We should enforce its limitation in compiler,
> ld and ld.so so that there is no surprise, either for correctness or
> performance,  at run-time.
> 
> 

my prefference would be if you (re)add it: do so such that there is no 
portability issue with next or previous gcc version (or other cc if 
possible), and that it be left as optional with a quick note that it is, 
and insure both on and off both build w/o fail in doing so.  picky?

thank you, have a nice day!

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Alan Modra]

On Tue, Apr 19, 2016 at 10:20:23AM +0200, Richard Biener wrote:
> On Tue, Apr 19, 2016 at 7:08 AM, Alan Modra <amodra@gmail.com> wrote:
> > On Mon, Apr 18, 2016 at 07:59:50AM -0700, H.J. Lu wrote:
> >> On Mon, Apr 18, 2016 at 7:49 AM, Alan Modra <amodra@gmail.com> wrote:
> >> > On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
> >> >> To summarize: there is currently no testcase for a wrong-code issue
> >> >> because there is no wrong-code issue.
> >
> > I've added a testcase at
> > https://sourceware.org/bugzilla/show_bug.cgi?id=19965#c3
> > that shows the address problem (&x != x) with older gcc *or* older
> > glibc, and shows the program behaviour problem with current
> > binutils+gcc+glibc.
> 
> Thanks.
> 
> So with all this it sounds that current protected visibility is just broken
> and we should forgo with it, making it equal to default visibility?

Well, using protected visibility variables makes no sense in
executables.  They really are only useful in shared libraries, but
have been of limited use on architectures like x86 for a long time
due to non-PIC executable copying shared library variables into
.dynbss.  The concepts of copying variables into .dynbss, and
protected visibility, are fundamentally incompatible.

HJ's changes addressed the program level semantic issues, but in the
process lost the main reason to use protected visibility variables,
which is to tell a compiler that a global variable cannot be preempted
(and therefore can use faster code for access, typically pc or GOT
pointer relative rather than GOT indirect.)  So IMO, "of limited use"
has now become "not much use at all" on x86_64 and other architectures
that have blindly followed suit.

> At least I couldn't decipher a solution that solves all of the issues
> with protected visibility apart from trying to error at link-time
> (or runtime?) for the cases that are tricky (impossible?) to solve.

I described the problem and solutions in
https://sourceware.org/ml/binutils/2016-03/msg00431.html.  A followup
by Cary pointed out that one of the solutions, emitting text dynamic
relocations, won't work on some architectures (of which x86_64 is
one).

> glibc uses "protected visibility" via its using of local aliases, correct?

Yes, glibc defines a hidden visibility symbol for internal use, with
an exported alias.

> But it doesn't use anything like that for data symbols?

I believe it does.  See occurrences of libc_hidden_data_def.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Jeff Law]

On 04/19/2016 02:20 AM, Richard Biener wrote:
> On Tue, Apr 19, 2016 at 7:08 AM, Alan Modra <amodra@gmail.com> wrote:
>> On Mon, Apr 18, 2016 at 07:59:50AM -0700, H.J. Lu wrote:
>>> On Mon, Apr 18, 2016 at 7:49 AM, Alan Modra <amodra@gmail.com> wrote:
>>>> On Mon, Apr 18, 2016 at 11:01:48AM +0200, Richard Biener wrote:
>>>>> To summarize: there is currently no testcase for a wrong-code issue
>>>>> because there is no wrong-code issue.
>>
>> I've added a testcase at
>> https://sourceware.org/bugzilla/show_bug.cgi?id=19965#c3
>> that shows the address problem (&x != x) with older gcc *or* older
>> glibc, and shows the program behaviour problem with current
>> binutils+gcc+glibc.
>
> Thanks.
>
> So with all this it sounds that current protected visibility is just broken
> and we should forgo with it, making it equal to default visibility?
No, we revert to the gcc-4.9 behavior WRT protected visibility and 
ensure that we're getting a proper diagnostic from the linker.

That direction is consistent with the intent of protected visibility, 
fixes the problem with preemption of protected symbols and gives us a 
diagnostic for the case that can't be reasonably handled.

Jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Alan Modra]

On Mon, Apr 25, 2016 at 11:35:46AM -0600, Jeff Law wrote:
> No, we revert to the gcc-4.9 behavior WRT protected visibility and ensure
> that we're getting a proper diagnostic from the linker.
> 
> That direction is consistent with the intent of protected visibility, fixes
> the problem with preemption of protected symbols and gives us a diagnostic
> for the case that can't be reasonably handled.

I agree that this is the correct solution.  Unfortunately there is a
complication.  PIE + shared lib using protected visibility worked fine
with gcc-4.9, but since then code generated by gcc for PIEs on x86_64
has been optimized to rely on the horrible old hack of .dynbss and
copy relocations.  That means you'll have regressions from 4.9 if just
reverting the protected visibility change..

The PIE optimization will need reverting too, and I imagine you'll see
some resistance to that idea due to the fact that it delivers quite a
nice performance improvement for PIEs.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Jakub Jelinek]

On Tue, Apr 26, 2016 at 03:24:48PM +0930, Alan Modra wrote:
> On Mon, Apr 25, 2016 at 11:35:46AM -0600, Jeff Law wrote:
> > No, we revert to the gcc-4.9 behavior WRT protected visibility and ensure
> > that we're getting a proper diagnostic from the linker.
> > 
> > That direction is consistent with the intent of protected visibility, fixes
> > the problem with preemption of protected symbols and gives us a diagnostic
> > for the case that can't be reasonably handled.
> 
> I agree that this is the correct solution.  Unfortunately there is a
> complication.  PIE + shared lib using protected visibility worked fine
> with gcc-4.9, but since then code generated by gcc for PIEs on x86_64
> has been optimized to rely on the horrible old hack of .dynbss and
> copy relocations.  That means you'll have regressions from 4.9 if just
> reverting the protected visibility change..
> 
> The PIE optimization will need reverting too, and I imagine you'll see
> some resistance to that idea due to the fact that it delivers quite a
> nice performance improvement for PIEs.

Yes, that change is IMHO too important to revert, it basically made PIEs
usable without significant slowdown.

Regressing on protected visibility is fine, that is something that is only
rarely used and in an ideal world wouldn't be used at all, because the
current definition really is not an optimization.  If we want to use
protected for something, it should be declared that address comparisons are
undefined for them, and for references to protected variables from binaries
(PIEs or normal) or perhaps even other shared libraries (other than the one
with definition) we should just require some extra attribute on them to make
this clear and force using GOT there.

	Jakub

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26 [aka should we revert the fix for 65248]

[Cary Coutant]

>> Given a shared library that defines a variable, and a non-PIC
>> executable that references that variable, the linker makes a duplicate
>> of the variable in the executable .dynbss section and arranges to have
>> the copy initialized by the dynamic loader with a copy relocation.
>> .dynbss is a linker created section that becomes part of the
>> executable bss segment.  The idea is that at run-time both the
>> executable and the shared library will use the executable copy of the
>> variable.  It's a rather ancient linker hack to avoid dynamic text
>> relocations, invented well before symbol visibility.
>
> So what other choice does the linker have here?  AFAICS it's wrong
> to create the .dynbss copy for protected symbols.  So why not
> simply create 'dynamic text relocations' then?  Is that possible with
> a pure linker change?

Ugh. Besides being a bad idea from a performance point of view, it's
not even always possible to do. Depending on the architecture, a
direct reference from an executable to a variable in a shared library
may not have the necessary reach.

> That said, correctness trumps optimization.  A correctness fix that
> works with old objects trumps one that requires a compiler change.
> Requiring a compiler change to get back optimization while preserving
> correctness is fine.

When the whole point of a feature is to enable a particular
optimization, the missed optimization *is* a correctness issue.

Symbol visibility is not a standard language feature. It's an
extension that many compilers provide to give access to an ELF/gABI
feature, and it comes with limitations. When the only way to eliminate
those limitations is to disable the intended optimization, the only
real choices are to live with the limitations (i.e., issue an error
when we would need a COPY relocation for a protected symbol), or to
forgo the extension altogether.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> It would help me immensely on the GCC side if things if you and Alan could
> easily summarize correct behavior and the impact if we were to just revert
> HJ's change.  A testcase would be amazingly helpful too.

It looks like it's not just the one change. There's this patch:

   https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html

which took the idea that protected can still be pre-empted by a COPY
relocation and extended it to three more targets that use COPY
relocations.

I wonder how many other patches have been based on the same misunderstanding?

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 03/30/2016 06:40 PM, Cary Coutant wrote:
>> It would help me immensely on the GCC side if things if you and Alan could
>> easily summarize correct behavior and the impact if we were to just revert
>> HJ's change.  A testcase would be amazingly helpful too.
>
> It looks like it's not just the one change. There's this patch:
>
>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>
> which took the idea that protected can still be pre-empted by a COPY
> relocation and extended it to three more targets that use COPY
> relocations.
>
> I wonder how many other patches have been based on the same misunderstanding?
I don't think it was many -- I certainly recall the arm/aarch64 variant. 
  There may have been one other varasm.c change in this space or I might 
be conflating it with the arm/aarch64 change.  Tracking those down is 
naturally part of this work.

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Ramana Radhakrishnan]

On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>
>>> It would help me immensely on the GCC side if things if you and Alan
>>> could
>>> easily summarize correct behavior and the impact if we were to just
>>> revert
>>> HJ's change.  A testcase would be amazingly helpful too.
>>
>>
>> It looks like it's not just the one change. There's this patch:
>>
>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>
>> which took the idea that protected can still be pre-empted by a COPY
>> relocation and extended it to three more targets that use COPY
>> relocations.
>>
>> I wonder how many other patches have been based on the same
>> misunderstanding?
>
> I don't think it was many -- I certainly recall the arm/aarch64 variant.
> There may have been one other varasm.c change in this space or I might be
> conflating it with the arm/aarch64 change.  Tracking those down is naturally
> part of this work.

The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
the same time.  IIUC, the reason the patch above went in were to fix
failures on arm / aarch64 with those tests. I haven't yet worked out
whether all this is the same issue.

CC'ing Szabolcs.

Thanks,
Ramana

>
> jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 31, 2016 at 6:26 AM, Ramana Radhakrishnan
<ramana.gcc@googlemail.com> wrote:
> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>
>>>> It would help me immensely on the GCC side if things if you and Alan
>>>> could
>>>> easily summarize correct behavior and the impact if we were to just
>>>> revert
>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>
>>>
>>> It looks like it's not just the one change. There's this patch:
>>>
>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>
>>> which took the idea that protected can still be pre-empted by a COPY
>>> relocation and extended it to three more targets that use COPY
>>> relocations.
>>>
>>> I wonder how many other patches have been based on the same
>>> misunderstanding?
>>
>> I don't think it was many -- I certainly recall the arm/aarch64 variant.
>> There may have been one other varasm.c change in this space or I might be
>> conflating it with the arm/aarch64 change.  Tracking those down is naturally
>> part of this work.
>
> The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
> the same time.  IIUC, the reason the patch above went in were to fix
> failures on arm / aarch64 with those tests. I haven't yet worked out
> whether all this is the same issue.
>

These testcases show the problem between copy relocation and
protected data symbol.  I updated my program property extension
to GNU ABI to prevent copy relocation against protected data symbol:

https://sourceware.org/ml/gnu-gabi/2016-q1/msg00041.html

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Szabolcs Nagy]

On 31/03/16 14:26, Ramana Radhakrishnan wrote:
> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>
>>>> It would help me immensely on the GCC side if things if you and Alan
>>>> could
>>>> easily summarize correct behavior and the impact if we were to just
>>>> revert
>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>
>>>
>>> It looks like it's not just the one change. There's this patch:
>>>
>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>
>>> which took the idea that protected can still be pre-empted by a COPY
>>> relocation and extended it to three more targets that use COPY
>>> relocations.
>>>
>>> I wonder how many other patches have been based on the same
>>> misunderstanding?

(sorry i missed this thread)

this was not a misunderstanding.

that patch is necessary for correctness (odr) in
the presence of copy relocations as described in
https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
and
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012

this was a long standing code gen bug in gcc and was
about time to fix it (it was also broken in glibc's
dynamic linker, but e.g. not in musl libc).

(i don't see what is the issue with using the copy in
the main executable from a shared library, performance
is not a correctness issue, nor how it is possible to
avoid the copy relocs.)

>>
>> I don't think it was many -- I certainly recall the arm/aarch64 variant.
>> There may have been one other varasm.c change in this space or I might be
>> conflating it with the arm/aarch64 change.  Tracking those down is naturally
>> part of this work.
> 
> The glibc tests elf/tst-protected1{a,b}.c also need to be reviewed at
> the same time.  IIUC, the reason the patch above went in were to fix
> failures on arm / aarch64 with those tests. I haven't yet worked out
> whether all this is the same issue.
> 
> CC'ing Szabolcs.
> 
> Thanks,
> Ramana
> 
>>
>> jeff
> 

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 03/30/2016 06:40 PM, Cary Coutant wrote:
>> It would help me immensely on the GCC side if things if you and Alan could
>> easily summarize correct behavior and the impact if we were to just revert
>> HJ's change.  A testcase would be amazingly helpful too.
>
> It looks like it's not just the one change. There's this patch:
>
>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>
> which took the idea that protected can still be pre-empted by a COPY
> relocation and extended it to three more targets that use COPY
> relocations.
>
> I wonder how many other patches have been based on the same misunderstanding?
The patches around BZ65780 may be tangled in this mess as well.    And 
it bled into the s390 & darwin ports as well.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65780

Alan, could you take a peek at 65780 -- you're better versed than I in 
this stuff.  Essentially the question that needs to be answered is if we 
roll back 65280, do we need to also roll back 65780?

jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

On Fri, Apr 01, 2016 at 01:51:33PM -0600, Jeff Law wrote:
> Essentially the question that needs to be answered is if we roll
> back 65280, do we need to also roll back 65780?

No, I don't think so.  As I wrote in the bugzilla (paraphrased)

What's going on here is that HJ has made changes to the linker for x86
to always copy variables defined in a shared library and referenced by
the executable, into the executable's .dynbss.  This is necessary to
support non-PIC without text relocations, but is optional for PIEs.

If you do this for PIEs and gain some performance without affecting
shared library performance, and that is the case for x86_64, then the
optimisation is quite reasonable.  I think powerpc64 would gain too,
but I haven't done anything about it yet (and for powerpc64 we should
be able to implement it entirely in the linker, no gcc changes
needed).

I believe common variables can be treated like any other variable so
far as this optimisation goes.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Fwd: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Rich Felker]

On Tue, Mar 29, 2016 at 12:31:38PM -0700, Joe Groff wrote:
> On Mar 29, 2016, at 8:44 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> > 
> > On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
> >> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
> >>>>>> Did you look at what the costs were in startup time and dirty pages by using
> >>>>>> copy relocations? What do you do if the size of the definition changes in a
> >>>>>> new version of the library?
> >>>>> 
> >>>>> There wouldn't be a measurable cost in dirty pages; the copied objects
> >>>>> are simply allocated in bss in the executable.
> >>>> 
> >>>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
> >>> 
> >>> No, references from within the .so would have always used the GOT.
> >>> Non-protected global symbols in a shared library are still
> >>> pre-emptible, so they are always indirect, and there's always a
> >>> dynamic relocation for the GOT entry. Whether the prevailing
> >>> definition winds up in the executable or the shared library, the
> >>> dynamic loader still has to bind the symbol and apply the relocation.
> >> 
> >> HJ's changes to protected visibility meant compiler changes so that
> >> protected visibility in shared libraries is no longer seen as local.
> >> So yes, protected visibility symbols in shared libraries now go
> >> through the GOT.  Prior to his changes, they were optimized to a
> >> pc-relative access.  Joe is correct in pointing out that shared
> >> libraries needed a change.  Bad luck if you're using an older
> >> compiler.  Also bad luck if you want to use protected visibility to
> >> optimize your shared library.
> >> 
> >> HJ also made glibc ld.so changes to ensure the semantics of protected
> >> visibility symbols remain unchanged when multiple shared libraries
> >> define the same protected visibility symbol.
> >> 
> >> Apparently most people in the gcc and glibc communities saw these
> >> toolchain modifications as fiendishly clever.
> >> 
> > 
> > As I said before, copy relocation and protected symbol are fundamentally
> > incompatible.  Since copy relocation is the part of x86 psABIs, I updated
> > GCC, glibc and ld to make protected symbol to work with copy relocation.
> > That is protected symbol may be external, but won't be preempted.  The
> > price I paid is that protected symbol won't be accessed via PC-relative
> > relocation within the shared object.  To access protected symbol via
> > PC-relative relocation within the shared object, we need to disable copy
> > relocation in executable, which is a psABI change.  That is why I proposed
> > to mark the object as such so that we won't get surprise at run-time.
> 
> I think what Cary's arguing (and I honestly would expect) is that
> copying the protected symbol *is* for all intents and purposes a
> preemption. I'd expect copy relocations against protected symbols to
> be linker errors. I guess what's missing for gcc's intended
> optimization is an indication to the compiler that a symbol is
> protected in its home library, to suppress emitting PC-relative
> references to a copy relocation.

As one of the strong advocates for the fix that was made to make
protected visibility work correctly with data symbols, I'd like to
explain why it was the right decision and why it matters. This whole
process is really frustrating to me -- having invested a lot of effort
into getting something important fixed, only to have people come
trying to break it again -- but I'm going to try to be calm and not to
snap at anybody. I was only vaguely aware of this thread until a few
days ago (because I've been trying to make new progress on other
things rather than revisit issues I thought were closed), so I'm just
replying to the beginning of this thread. I'll also try to comment on
particularly important points elsewhere in the thread as I read it,
but I don't want to pile on a bunch of scattered replies that focus on
small details I think others have gotten wrong and ignore the big
picture.

From a programming standpoint, the semantics of protected visibility
need to be free of arch-specific implementation details. Otherwise
programmers can't use it without hard-coding arch-specific details,
which for practical purposes, means good software can't use it at all.

My original motivation for wanting protected visibility to "just work"
was to be able to use:

	#pragma GCC visibility push(protected)

around the inclusion of a library's public headers when including them
from the implementation files, This is far from being the only usage
case, and I'll expand on more important usage cases below, but it is
an important one because it allows you to eliminate all GOT/PLT cost
of intra-library function calls without any fine-grained maintenance
of which declarations to apply visibility too (and without any
GNUC-specific clutter in the header files themselves).

I understand that some people want protected visibility to avoid the
GOT for data symbols too for the sake of performance, but for my usage
case, the fact that the semantics were wrong for data symbols meant
that my configure check for "does protected visibility work" would
return "no", and the whole optimization would get turned off.

Anyway, let's move past optimization, because it's a distraction.
After all, with the old (broken) behavior of protected data, one
_could_ work around the above problem and still get the performance
benefits for functions without breaking data by explicitly declaring
all data with default visibility. In fact, this is how I solve the
problem in musl libc, where there are only a small number of data
symbols that should be externally accessible, and maintaining a list
of them is managable:

http://git.musl-libc.org/cgit/musl/tree/src/internal/vis.h?id=d1b29c2a54588401494c1a3ac7103c1e91c61fa1

This is done for the sake of compatibility with a wide range of
toolchains including ones with the old/broken behavior for protected
data.

The actual documented purpose of protected visibility is to prevent
other definitions of a symbol from taking precedence over the one in
the library itself. For example, suppose you have the following
situation: mainapp depends on libA which defines symbol foo with
normal visibility, and libA depends on libB, which also defines foo,
but intentionally with protected visibility so that libB always uses
its own definition, not the one from libA. There is no reasonable way
to obtain the desired semantics here without the current/correct
behavior for protected data. Any other approaches I'm aware of would
either allow libB to bind to the wrong definition of foo, or would
prevent another main app which links libB (but not libA) from being
able to use the symbol foo from libB.

On the other hand, there are plenty of other ways to get the
old/broken behavior if desired. The easiest is to simply use hidden
visibility when you don't want the symbol to be accessible outside the
library. If you _do_ want it to be visible to and usable by other
shared libraries, just not main-programs, this can be achieved using a
hidden alias for a default-visibility symbol:

	int foo;
	extern int fast_foo __attribute__((__alias__("foo"),
	                                   __visibility__("hidden")));

I expressed that here explicitly for the sake of clarity but of course
in practice people use things like the glibc macro-maze to do this
kind of binding to hidden aliases.

Rich

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>
>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>> could
>>>>> easily summarize correct behavior and the impact if we were to just
>>>>> revert
>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>
>>>>
>>>> It looks like it's not just the one change. There's this patch:
>>>>
>>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>
>>>> which took the idea that protected can still be pre-empted by a COPY
>>>> relocation and extended it to three more targets that use COPY
>>>> relocations.
>>>>
>>>> I wonder how many other patches have been based on the same
>>>> misunderstanding?
>
> (sorry i missed this thread)
>
> this was not a misunderstanding.
>
> that patch is necessary for correctness (odr) in
> the presence of copy relocations as described in
> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
> and
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>
> this was a long standing code gen bug in gcc and was
> about time to fix it (it was also broken in glibc's
> dynamic linker, but e.g. not in musl libc).
>
> (i don't see what is the issue with using the copy in
> the main executable from a shared library, performance
> is not a correctness issue, nor how it is possible to
> avoid the copy relocs.)
>

Here is my understanding:

Copy relocation and protected visibility are fundamentally incompatible.
On on hand, copy relocation is the part of the psABI and is used to
access global data defined in a shared object from the executable.  It
moves the definition of global data, which is defined in a share object,
to the executable at run-time.  On the other hand, protected visibility
indicates that a symbol is defined locally in the shared object at
run-time.  Both can't be true at the same time.  The current solution
is to make protected symbol more or less like normal symbol, which
prevents optimizing local access to protected symbol within the shared
object.

I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:

https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI

GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
should be no copy relocations against protected data symbols. If a relocat-
able object contains this property, linker should treat protected data symbol
as defined locally at run-time and copy this property to the output share
object. Linker should add this property to the output share object if any pro-
tected symbol is expected to be defined locally at run-time. Run-time loader
should disallow copy relocations against protected data symbols defined in
share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
erty. Its PR_DATASZ should be 0.



-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Jeff Law]

On 04/15/2016 10:16 AM, H.J. Lu wrote:
> On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>>
>>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>>> could
>>>>>> easily summarize correct behavior and the impact if we were to just
>>>>>> revert
>>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>>
>>>>>
>>>>> It looks like it's not just the one change. There's this patch:
>>>>>
>>>>>      https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>>
>>>>> which took the idea that protected can still be pre-empted by a COPY
>>>>> relocation and extended it to three more targets that use COPY
>>>>> relocations.
>>>>>
>>>>> I wonder how many other patches have been based on the same
>>>>> misunderstanding?
>>
>> (sorry i missed this thread)
>>
>> this was not a misunderstanding.
>>
>> that patch is necessary for correctness (odr) in
>> the presence of copy relocations as described in
>> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
>> and
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>
>> this was a long standing code gen bug in gcc and was
>> about time to fix it (it was also broken in glibc's
>> dynamic linker, but e.g. not in musl libc).
>>
>> (i don't see what is the issue with using the copy in
>> the main executable from a shared library, performance
>> is not a correctness issue, nor how it is possible to
>> avoid the copy relocs.)
>>
>
> Here is my understanding:
>
> Copy relocation and protected visibility are fundamentally incompatible.
> On on hand, copy relocation is the part of the psABI and is used to
> access global data defined in a shared object from the executable.  It
> moves the definition of global data, which is defined in a share object,
> to the executable at run-time.  On the other hand, protected visibility
> indicates that a symbol is defined locally in the shared object at
> run-time.  Both can't be true at the same time.  The current solution
> is to make protected symbol more or less like normal symbol, which
> prevents optimizing local access to protected symbol within the shared
> object.
>
> I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:
>
> https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI
>
> GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
> should be no copy relocations against protected data symbols. If a relocat-
> able object contains this property, linker should treat protected data symbol
> as defined locally at run-time and copy this property to the output share
> object. Linker should add this property to the output share object if any pro-
> tected symbol is expected to be defined locally at run-time. Run-time loader
> should disallow copy relocations against protected data symbols defined in
> share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
> erty. Its PR_DATASZ should be 0.
I'd strongly suggest discussing directly with Carlos, Cary and Alan.  My 
worry here is this just adding another layer of stuff to deal with a 
fundamentally broken concept -- protected visibility.

Jeff

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Fri, Apr 15, 2016 at 9:36 AM, Jeff Law <law@redhat.com> wrote:
> On 04/15/2016 10:16 AM, H.J. Lu wrote:
>>
>> On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com>
>> wrote:
>>>
>>> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>>>>
>>>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>>>>
>>>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>>>
>>>>>>>
>>>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>>>> could
>>>>>>> easily summarize correct behavior and the impact if we were to just
>>>>>>> revert
>>>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>>>
>>>>>>
>>>>>>
>>>>>> It looks like it's not just the one change. There's this patch:
>>>>>>
>>>>>>      https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>>>
>>>>>> which took the idea that protected can still be pre-empted by a COPY
>>>>>> relocation and extended it to three more targets that use COPY
>>>>>> relocations.
>>>>>>
>>>>>> I wonder how many other patches have been based on the same
>>>>>> misunderstanding?
>>>
>>>
>>> (sorry i missed this thread)
>>>
>>> this was not a misunderstanding.
>>>
>>> that patch is necessary for correctness (odr) in
>>> the presence of copy relocations as described in
>>> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
>>> and
>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>>
>>> this was a long standing code gen bug in gcc and was
>>> about time to fix it (it was also broken in glibc's
>>> dynamic linker, but e.g. not in musl libc).
>>>
>>> (i don't see what is the issue with using the copy in
>>> the main executable from a shared library, performance
>>> is not a correctness issue, nor how it is possible to
>>> avoid the copy relocs.)
>>>
>>
>> Here is my understanding:
>>
>> Copy relocation and protected visibility are fundamentally incompatible.
>> On on hand, copy relocation is the part of the psABI and is used to
>> access global data defined in a shared object from the executable.  It
>> moves the definition of global data, which is defined in a share object,
>> to the executable at run-time.  On the other hand, protected visibility
>> indicates that a symbol is defined locally in the shared object at
>> run-time.  Both can't be true at the same time.  The current solution
>> is to make protected symbol more or less like normal symbol, which
>> prevents optimizing local access to protected symbol within the shared
>> object.
>>
>> I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:
>>
>> https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI
>>
>> GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
>> should be no copy relocations against protected data symbols. If a
>> relocat-
>> able object contains this property, linker should treat protected data
>> symbol
>> as defined locally at run-time and copy this property to the output share
>> object. Linker should add this property to the output share object if any
>> pro-
>> tected symbol is expected to be defined locally at run-time. Run-time
>> loader
>> should disallow copy relocations against protected data symbols defined in
>> share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
>> erty. Its PR_DATASZ should be 0.
>
> I'd strongly suggest discussing directly with Carlos, Cary and Alan.  My
> worry here is this just adding another layer of stuff to deal with a
> fundamentally broken concept -- protected visibility.
>

Adding glibc.

Protected symbol has been a tricky issue for glibc and binutils
from day 1.  We have special treatment for pointers of protected
functions in both ld and ld.so.  Protected symbol is a useful
feature.  It just doesn't work with copy relocation.  My proposal
will make it work for psABIs with copy relocation by disallowing
copy relocation on protected symbol.  Linker change is minimum
and it will bypass extra symbol lookups in ld.so.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Szabolcs Nagy]

On 15/04/16 17:16, H.J. Lu wrote:
> On Fri, Apr 15, 2016 at 9:09 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> On 31/03/16 14:26, Ramana Radhakrishnan wrote:
>>> On Thu, Mar 31, 2016 at 1:52 AM, Jeff Law <law@redhat.com> wrote:
>>>> On 03/30/2016 06:40 PM, Cary Coutant wrote:
>>>>>>
>>>>>> It would help me immensely on the GCC side if things if you and Alan
>>>>>> could
>>>>>> easily summarize correct behavior and the impact if we were to just
>>>>>> revert
>>>>>> HJ's change.  A testcase would be amazingly helpful too.
>>>>>
>>>>>
>>>>> It looks like it's not just the one change. There's this patch:
>>>>>
>>>>>     https://gcc.gnu.org/ml/gcc-patches/2015-07/msg01871.html
>>>>>
>>>>> which took the idea that protected can still be pre-empted by a COPY
>>>>> relocation and extended it to three more targets that use COPY
>>>>> relocations.
>>>>>
>>>>> I wonder how many other patches have been based on the same
>>>>> misunderstanding?
>>
>> (sorry i missed this thread)
>>
>> this was not a misunderstanding.
>>
>> that patch is necessary for correctness (odr) in
>> the presence of copy relocations as described in
>> https://gcc.gnu.org/ml/gcc-patches/2015-09/msg02365.html
>> and
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55012
>>
>> this was a long standing code gen bug in gcc and was
>> about time to fix it (it was also broken in glibc's
>> dynamic linker, but e.g. not in musl libc).
>>
>> (i don't see what is the issue with using the copy in
>> the main executable from a shared library, performance
>> is not a correctness issue, nor how it is possible to
>> avoid the copy relocs.)
>>
> 
> Here is my understanding:
> 
> Copy relocation and protected visibility are fundamentally incompatible.
> On on hand, copy relocation is the part of the psABI and is used to
> access global data defined in a shared object from the executable.  It
> moves the definition of global data, which is defined in a share object,
> to the executable at run-time.  On the other hand, protected visibility
> indicates that a symbol is defined locally in the shared object at
> run-time.  Both can't be true at the same time.  The current solution

protected visibility indicates

  "that references within the defining module bind to
  the definition in that module. That is, the declared
  entity cannot be overridden by another module."

here "definition in that module" does not mean addresses
at runtime, but the c language level definition.
(c semantics can only indicate abstract machine behaviour,
not relocation types and in memory layout at runtime).

so there is no conflict between copy relocation and
protected visibility: the definition is not overridden,
it's an implementation detail that the address happens
to be in the data section of the main executable instead
of the defining module.

i think your fixes in gcc, etc are ok.
(i understand that ppl want to optimize this but that
should be a separate discussion once there is consensus
about the correct semantics).

> is to make protected symbol more or less like normal symbol, which
> prevents optimizing local access to protected symbol within the shared
> object.
> 
> I propose to add GNU_PROPERTY_NO_COPY_ON_PROTECTED:
> 
> https://github.com/hjl-tools/linux-abi/wiki/Linux-Extensions-to-gABI
> 
> GNU_PROPERTY_NO_COPY_ON_PROTECTED This indicates that there
> should be no copy relocations against protected data symbols. If a relocat-
> able object contains this property, linker should treat protected data symbol
> as defined locally at run-time and copy this property to the output share
> object. Linker should add this property to the output share object if any pro-
> tected symbol is expected to be defined locally at run-time. Run-time loader
> should disallow copy relocations against protected data symbols defined in
> share objects with GNU_PROPERTY_NO_COPY_ON_PROTECTED prop-
> erty. Its PR_DATASZ should be 0.
> 
> 
> 

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Maciej W. Rozycki]

On Fri, 15 Apr 2016, Szabolcs Nagy wrote:

> > Copy relocation and protected visibility are fundamentally incompatible.
> > On on hand, copy relocation is the part of the psABI and is used to
> > access global data defined in a shared object from the executable.  It
> > moves the definition of global data, which is defined in a share object,
> > to the executable at run-time.  On the other hand, protected visibility
> > indicates that a symbol is defined locally in the shared object at
> > run-time.  Both can't be true at the same time.  The current solution
> 
> protected visibility indicates
> 
>   "that references within the defining module bind to
>   the definition in that module. That is, the declared
>   entity cannot be overridden by another module."
> 
> here "definition in that module" does not mean addresses
> at runtime, but the c language level definition.
> (c semantics can only indicate abstract machine behaviour,
> not relocation types and in memory layout at runtime).

 Where did you get this definition from and how do you infer this is a "c 
language level" rather than a binary level definition?  The SVR4/ELF gABI 
says[1]:

"A symbol defined in the current component is protected if it is visible
in other components but not preemptable, meaning that any reference to
such a symbol from within the defining component must be resolved to the
definition in that component, even if there is a definition in another
component that would preempt by the default rules."

and:

"The presence of the STV_PROTECTED flag on a symbol in a given load module
does not affect the symbol resolution rules for references to that symbol
from outside the containing load module."

so it clearly indicates that it is dynamic load modules (i.e. either the 
main executable or any of its referred DSOs) that are considered here, not 
C source-level semantics.  Besides the ELF ABI is programming language 
agnostic, using the C language as a reference only, so it cannot really 
consider source-level semantics.

> so there is no conflict between copy relocation and
> protected visibility: the definition is not overridden,
> it's an implementation detail that the address happens
> to be in the data section of the main executable instead
> of the defining module.

 Both the reference above and established practice assume a protected 
symbol must have its address within the module that defines it.  See also:
<https://sourceware.org/ml/binutils/2016-03/msg00368.html> for my earlier 
comment.

References:

[1] "System V Application Binary Interface - DRAFT - 10 June 2013",
    The Santa Cruz Operation, Inc., "Symbol Table",
    <http://www.sco.com/developers/gabi/latest/ch4.symtab.html>

  Maciej

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Szabolcs Nagy]

* Maciej W. Rozycki <macro@imgtec.com> [2016-04-16 00:59:17 +0100]:
> On Fri, 15 Apr 2016, Szabolcs Nagy wrote:
> 
> > > Copy relocation and protected visibility are fundamentally incompatible.
> > > On on hand, copy relocation is the part of the psABI and is used to
> > > access global data defined in a shared object from the executable.  It
> > > moves the definition of global data, which is defined in a share object,
> > > to the executable at run-time.  On the other hand, protected visibility
> > > indicates that a symbol is defined locally in the shared object at
> > > run-time.  Both can't be true at the same time.  The current solution
> > 
> > protected visibility indicates
> > 
> >   "that references within the defining module bind to
> >   the definition in that module. That is, the declared
> >   entity cannot be overridden by another module."
> > 
> > here "definition in that module" does not mean addresses
> > at runtime, but the c language level definition.
> > (c semantics can only indicate abstract machine behaviour,
> > not relocation types and in memory layout at runtime).
> 
>  Where did you get this definition from and how do you infer this is a "c 
> language level" rather than a binary level definition?  The SVR4/ELF gABI 
> says[1]:
> 
> "A symbol defined in the current component is protected if it is visible
> in other components but not preemptable, meaning that any reference to
> such a symbol from within the defining component must be resolved to the
> definition in that component, even if there is a definition in another
> component that would preempt by the default rules."
> 
> and:
> 
> "The presence of the STV_PROTECTED flag on a symbol in a given load module
> does not affect the symbol resolution rules for references to that symbol
> from outside the containing load module."
> 
> so it clearly indicates that it is dynamic load modules (i.e. either the 
> main executable or any of its referred DSOs) that are considered here, not 
> C source-level semantics.  Besides the ELF ABI is programming language 
> agnostic, using the C language as a reference only, so it cannot really 
> consider source-level semantics.
> 

sorry, i just looked at the c extension as documented
by gcc, but the elf gabi is the right source. however
it seems to verify my interpretation.

> > so there is no conflict between copy relocation and
> > protected visibility: the definition is not overridden,
> > it's an implementation detail that the address happens
> > to be in the data section of the main executable instead
> > of the defining module.
> 
>  Both the reference above and established practice assume a protected 
> symbol must have its address within the module that defines it.  See also:

copy relocation is not another definition for the symbol
(it's an implementation detail), and symbol resolution
rules for external references are not affected, so both
constraints you cited above hold now.

> <https://sourceware.org/ml/binutils/2016-03/msg00368.html> for my earlier 
> comment.

ok, so it was intended to be an optimization (for function
symbols), but that breaks correctness for object symbols
in the presence of copy relocations.. so the optimization
is not valid: the symbol must be visible to other modules
which only works for the non-pic main module via copy relocs.

> References:
> 
> [1] "System V Application Binary Interface - DRAFT - 10 June 2013",
>     The Santa Cruz Operation, Inc., "Symbol Table",
>     <http://www.sco.com/developers/gabi/latest/ch4.symtab.html>
> 
>   Maciej

Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:

https://bugs.swift.org/browse/SR-1023

This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Wed, Mar 23, 2016 at 5:00 PM, Joe Groff <jgroff@apple.com> wrote:
> Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:
>
> https://bugs.swift.org/browse/SR-1023
>
> This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!
>

What you want is not protected symbol, which is expensive to lookup
at run-time.  You need:

  -z nocopyreloc              Don't create copy relocs

But it requires that everything must be compiled with -fPIC, not even
-fPIE, for both executable and shared library.  Otherwise, you will
get dynamic relocations in text section.

Swift should do something like -fPIC-data, to access external data via
PIC.  Then you can pass

 -z nocopyreloc              Don't create copy relocs

to linker.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 23, 2016, at 5:45 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Wed, Mar 23, 2016 at 5:00 PM, Joe Groff <jgroff@apple.com> wrote:
>> Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:
>> 
>> https://bugs.swift.org/browse/SR-1023
>> 
>> This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!
>> 
> 
> What you want is not protected symbol, which is expensive to lookup
> at run-time.  You need:
> 
>  -z nocopyreloc              Don't create copy relocs
> 
> But it requires that everything must be compiled with -fPIC, not even
> -fPIE, for both executable and shared library.  Otherwise, you will
> get dynamic relocations in text section.
> 
> Swift should do something like -fPIC-data, to access external data via
> PIC.  Then you can pass
> 
> -z nocopyreloc              Don't create copy relocs
> 
> to linker.

Thanks H.J. Is it possible to specify this in the object file, at the symbol level, or does "nocopyreloc" have to be specified by linker flags?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Wed, Mar 23, 2016 at 5:52 PM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 23, 2016, at 5:45 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Wed, Mar 23, 2016 at 5:00 PM, Joe Groff <jgroff@apple.com> wrote:
>>> Hi everyone. On the Swift project, we're getting bug reports that people are no longer able to link Swift programs using binutils 2.26:
>>>
>>> https://bugs.swift.org/browse/SR-1023
>>>
>>> This appears to be due to an intentional behavior change in commit https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected symbols to be the targets of copy relocations. This breaks our intended use of "protected" in Swift, where we really want to be able to assume that the relative addresses of our symbols within a single executable or .so won't be preempted or moved. This lets us emit position-independent constant reflection metadata that doesn't need load-time relocations, saving us some startup time and dirty pages. I wanted to know if there's a supported way to express to the linker that "this symbol is protected, and also can never be copied". I'd like to avoid globally setting -Bsymbolic since I wouldn't want to impact non-Swift .o files in a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent object files from trying to use copy relocations to our symbols and breaking our assumptions about their address; I'd like to know if it's possible to prevent that too. Thanks for any help you can give!
>>>
>>
>> What you want is not protected symbol, which is expensive to lookup
>> at run-time.  You need:
>>
>>  -z nocopyreloc              Don't create copy relocs
>>
>> But it requires that everything must be compiled with -fPIC, not even
>> -fPIE, for both executable and shared library.  Otherwise, you will
>> get dynamic relocations in text section.
>>
>> Swift should do something like -fPIC-data, to access external data via
>> PIC.  Then you can pass
>>
>> -z nocopyreloc              Don't create copy relocs
>>
>> to linker.
>
> Thanks H.J. Is it possible to specify this in the object file, at the symbol level, or does "nocopyreloc" have to be specified by linker flags?
>

Not at this moment.  I do have a proposal to specify something like
that in object files.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> Hi everyone. On the Swift project, we're getting bug reports that
> people are no longer able to link Swift programs using binutils
> 2.26:
>
> https://bugs.swift.org/browse/SR-1023
>
> This appears to be due to an intentional behavior change in
> commit
> https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit
> ;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected
> symbols to be the targets of copy relocations. This breaks our
> intended use of "protected" in Swift, where we really want to be
> able to assume that the relative addresses of our symbols within
> a single executable or .so won't be preempted or moved. This lets
> us emit position-independent constant reflection metadata that
> doesn't need load-time relocations, saving us some startup time
> and dirty pages. I wanted to know if there's a supported way to
> express to the linker that "this symbol is protected, and also
> can never be copied". I'd like to avoid globally setting
> -Bsymbolic since I wouldn't want to impact non-Swift .o files in
> a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent
> object files from trying to use copy relocations to our symbols
> and breaking our assumptions about their address; I'd like to
> know if it's possible to prevent that too. Thanks for any help
> you can give!

Your intended use of "protected" actually matches the actual intended
use, and I believe the change you point to was a mistake. The gABI
says that STV_PROTECTED must resolve to a definition in the same
module:

"A symbol defined in the current component is protected if it is
visible in other components but not preemptable, meaning that any
reference to such a symbol from within the defining component must be
resolved to the definition in that component, even if there is a
definition in another component that would preempt by the default
rules."

That's pretty clear, and its intent was to allow compilers to generate
more efficient code to access protected variables, knowing that they
can't be pre-empted. Allowing COPY relocations for protected symbols
breaks this guarantee, and is a bug. In fact, I'm about to fix just
such a bug in gold:

   https://sourceware.org/bugzilla/show_bug.cgi?id=19823

Worse, GCC 5 and 6 no longer generate direct references to protected
symbols, so the whole point of STV_PROTECTED (at least as it applies
to data symbols) is currently broken in the Gnu toolchain.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 8:01 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> Hi everyone. On the Swift project, we're getting bug reports that
>> people are no longer able to link Swift programs using binutils
>> 2.26:
>>
>> https://bugs.swift.org/browse/SR-1023
>>
>> This appears to be due to an intentional behavior change in
>> commit
>> https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit
>> ;h=ca3fe95e469b9daec153caa2c90665f5daaec2b5, to allow protected
>> symbols to be the targets of copy relocations. This breaks our
>> intended use of "protected" in Swift, where we really want to be
>> able to assume that the relative addresses of our symbols within
>> a single executable or .so won't be preempted or moved. This lets
>> us emit position-independent constant reflection metadata that
>> doesn't need load-time relocations, saving us some startup time
>> and dirty pages. I wanted to know if there's a supported way to
>> express to the linker that "this symbol is protected, and also
>> can never be copied". I'd like to avoid globally setting
>> -Bsymbolic since I wouldn't want to impact non-Swift .o files in
>> a mixed-language project. AIUI, -Bsymbolic also wouldn't prevent
>> object files from trying to use copy relocations to our symbols
>> and breaking our assumptions about their address; I'd like to
>> know if it's possible to prevent that too. Thanks for any help
>> you can give!
>
> Your intended use of "protected" actually matches the actual intended
> use, and I believe the change you point to was a mistake. The gABI
> says that STV_PROTECTED must resolve to a definition in the same
> module:
>
> "A symbol defined in the current component is protected if it is
> visible in other components but not preemptable, meaning that any
> reference to such a symbol from within the defining component must be
> resolved to the definition in that component, even if there is a
> definition in another component that would preempt by the default
> rules."
>
> That's pretty clear, and its intent was to allow compilers to generate
> more efficient code to access protected variables, knowing that they
> can't be pre-empted. Allowing COPY relocations for protected symbols
> breaks this guarantee, and is a bug. In fact, I'm about to fix just
> such a bug in gold:
>
>    https://sourceware.org/bugzilla/show_bug.cgi?id=19823
>
> Worse, GCC 5 and 6 no longer generate direct references to protected
> symbols, so the whole point of STV_PROTECTED (at least as it applies
> to data symbols) is currently broken in the Gnu toolchain.

I disagree.  Protected means that it won't be preempted.  It
doesn't mean the address of protected symbol will be local.
With copy relocation, the run-time address of the protected
symbol can be in executable.  There are a couple run-time
tests in glibc to verify it.  See:

https://sourceware.org/bugzilla/show_bug.cgi?id=17711


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> I disagree.  Protected means that it won't be preempted.  It
> doesn't mean the address of protected symbol will be local.
> With copy relocation, the run-time address of the protected
> symbol can be in executable.  There are a couple run-time
> tests in glibc to verify it.  See:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=17711

No. You can't just redefine away the meaning of "must resolve to a
definition in that component." A COPY relocation makes a new
definition in the main program and that new definition pre-empts the
one in the shared library. That breaks everything that the rule in the
gABI was designed to allow.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 9:06 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> I disagree.  Protected means that it won't be preempted.  It
>> doesn't mean the address of protected symbol will be local.
>> With copy relocation, the run-time address of the protected
>> symbol can be in executable.  There are a couple run-time
>> tests in glibc to verify it.  See:
>>
>> https://sourceware.org/bugzilla/show_bug.cgi?id=17711
>
> No. You can't just redefine away the meaning of "must resolve to a
> definition in that component." A COPY relocation makes a new
> definition in the main program and that new definition pre-empts the
> one in the shared library. That breaks everything that the rule in the
> gABI was designed to allow.
>

What you are proposing is to disallow copy relocation against
protected symbol.  Am I correct?


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> What you are proposing is to disallow copy relocation against
> protected symbol.  Am I correct?

Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
should have been resolved as "working as intended", and that your
patches for GCC PR 65248 should never have been applied (curiously, I
can't find any actual approval for the GCC patch). I'm sorry I missed
those conversations -- I would have raised bloody hell at the time.
I'm surprised that no one else did.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 9:56 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>> What you are proposing is to disallow copy relocation against
>> protected symbol.  Am I correct?
>
> Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
> should have been resolved as "working as intended", and that your
> patches for GCC PR 65248 should never have been applied (curiously, I
> can't find any actual approval for the GCC patch). I'm sorry I missed
> those conversations -- I would have raised bloody hell at the time.
> I'm surprised that no one else did.
>

What relocation do you propose to access external protected
symbol on x86 for non-PIC code?

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 24, 2016, at 10:04 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Thu, Mar 24, 2016 at 9:56 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>> What you are proposing is to disallow copy relocation against
>>> protected symbol.  Am I correct?
>> 
>> Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
>> should have been resolved as "working as intended", and that your
>> patches for GCC PR 65248 should never have been applied (curiously, I
>> can't find any actual approval for the GCC patch). I'm sorry I missed
>> those conversations -- I would have raised bloody hell at the time.
>> I'm surprised that no one else did.
>> 
> 
> What relocation do you propose to access external protected
> symbol on x86 for non-PIC code?

Non-PIC code can still use a GOT, can't it?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 10:06 AM, Joe Groff <jgroff@apple.com> wrote:
>
>> On Mar 24, 2016, at 10:04 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>
>> On Thu, Mar 24, 2016 at 9:56 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> What you are proposing is to disallow copy relocation against
>>>> protected symbol.  Am I correct?
>>>
>>> Yes. That's what PR gold/19823 is about. I claim that GCC PR 55012
>>> should have been resolved as "working as intended", and that your
>>> patches for GCC PR 65248 should never have been applied (curiously, I
>>> can't find any actual approval for the GCC patch). I'm sorry I missed
>>> those conversations -- I would have raised bloody hell at the time.
>>> I'm surprised that no one else did.
>>>
>>
>> What relocation do you propose to access external protected
>> symbol on x86 for non-PIC code?
>
> Non-PIC code can still use a GOT, can't it?

Yes.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>>> What relocation do you propose to access external protected
>>> symbol on x86 for non-PIC code?
>>
>> Non-PIC code can still use a GOT, can't it?
>
> Yes.

Right.

Some additional thoughts:

- This has nothing to do with PIE. Non-PIC-non-PIE code has been
living with this restriction for at least 18 years (since visiblity
was introduced into the gABI).

- This has nothing to do with Ulrich's diatribe against protected
visibility, which applies only to function symbols (and really only to
one platform, due to how function pointer comparison works).

- You don't need to go full-on PIC to reference a protected data
symbol in a shared library. You can always statically initialize a
pointer variable, and go indirect through that ("poor-man's PIC").

- We could add support for __attribute__((dllimport)) or
__declspec(dllimport) when declaring an external variable that is
expected to be defined in a shared library (ideally, that attribute
ought to be placed in the library's public interface). The compiler
can generate a PIC-style reference for that variable without
penalizing all the other variables in the main program. This is how
HP-UX compilers work on PA and Itanium (using #pragma external).

- Compilers could also generate tentative PIC-style references, with
sufficient relocations to allow the linker to convert the indirect
reference to a direct reference when possible (changing the second
load into a nop or copy). HP-UX also does this.

- Indirect references from PIC code to a protected symbol penalize the
common case (referencing a symbol within its own module) to support
the uncommon case, while introducing the nasty side effects of a COPY
relocation.

- COPY relocations are evil. They bind an application to a specific
version of a shared library's ABI, and introduce a hidden startup
cost. If we're going to make any changes, we should be moving towards
elimination of COPY relocations, rather than disabling features that
were designed to improve performance.

- Arguing that protected means that the definition is in the same
module but its address might be external is absurd. The *only* reason
for the gABI to make that guarantee is so the compilers can optimize
the code based on the knowledge that the symbol can't be pre-empted.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Rafael Espíndola]

On 24 March 2016 at 14:31, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> What relocation do you propose to access external protected
>>>> symbol on x86 for non-PIC code?
>>>
>>> Non-PIC code can still use a GOT, can't it?
>>
>> Yes.
>
> Right.
>
> Some additional thoughts:
>
> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
> living with this restriction for at least 18 years (since visiblity
> was introduced into the gABI).
>
> - This has nothing to do with Ulrich's diatribe against protected
> visibility, which applies only to function symbols (and really only to
> one platform, due to how function pointer comparison works).


The old powerpc ABI?

> - You don't need to go full-on PIC to reference a protected data
> symbol in a shared library. You can always statically initialize a
> pointer variable, and go indirect through that ("poor-man's PIC").
>
> - We could add support for __attribute__((dllimport)) or
> __declspec(dllimport) when declaring an external variable that is
> expected to be defined in a shared library (ideally, that attribute
> ought to be placed in the library's public interface). The compiler
> can generate a PIC-style reference for that variable without
> penalizing all the other variables in the main program. This is how
> HP-UX compilers work on PA and Itanium (using #pragma external).
>
> - Compilers could also generate tentative PIC-style references, with
> sufficient relocations to allow the linker to convert the indirect
> reference to a direct reference when possible (changing the second
> load into a nop or copy). HP-UX also does this.
>
> - Indirect references from PIC code to a protected symbol penalize the
> common case (referencing a symbol within its own module) to support
> the uncommon case, while introducing the nasty side effects of a COPY
> relocation.
>
> - COPY relocations are evil. They bind an application to a specific
> version of a shared library's ABI, and introduce a hidden startup
> cost. If we're going to make any changes, we should be moving towards
> elimination of COPY relocations, rather than disabling features that
> were designed to improve performance.
>
> - Arguing that protected means that the definition is in the same
> module but its address might be external is absurd. The *only* reason
> for the gABI to make that guarantee is so the compilers can optimize
> the code based on the knowledge that the symbol can't be pre-empted.

For what it is worth I strongly agree with Cary on these points.

I never quite understood the warning against using protected. Without
copy relocations or the PLT hacks, they are a really handy way to
produce visible symbols from dsos that are efficient to use in the DSO
itself.

It is unfortunate that without copy reloc and PLT hack they are not
easy to use from executables, but that can be fixed. Having the
compiler be conservative and the linker relax access would do it and
it is something I think we should try.

I will try to audit the llvm side to make sure protected in there
works as expected.

Cheers,
Rafael

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Thu, Mar 24, 2016 at 11:31 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>> What relocation do you propose to access external protected
>>>> symbol on x86 for non-PIC code?
>>>
>>> Non-PIC code can still use a GOT, can't it?
>>
>> Yes.
>
> Right.
>
> Some additional thoughts:
>
> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
> living with this restriction for at least 18 years (since visiblity
> was introduced into the gABI).
>
> - This has nothing to do with Ulrich's diatribe against protected
> visibility, which applies only to function symbols (and really only to
> one platform, due to how function pointer comparison works).

Copy relocation and protected symbol are mutually exclusive.
Since copy relocation is the part of x86 psABIs, it limits
protected symbol effectiveness on x86.  In glibc, we add a
local alias to symbol we want to protect and access the local
alias within glibc without GOT.

> - You don't need to go full-on PIC to reference a protected data
> symbol in a shared library. You can always statically initialize a
> pointer variable, and go indirect through that ("poor-man's PIC").
>
> - We could add support for __attribute__((dllimport)) or
> __declspec(dllimport) when declaring an external variable that is
> expected to be defined in a shared library (ideally, that attribute
> ought to be placed in the library's public interface). The compiler
> can generate a PIC-style reference for that variable without
> penalizing all the other variables in the main program. This is how
> HP-UX compilers work on PA and Itanium (using #pragma external).

This is a useful feature, similar to -fno-plt, like -fgot, but only on
selective symbols.

> - Compilers could also generate tentative PIC-style references, with
> sufficient relocations to allow the linker to convert the indirect
> reference to a direct reference when possible (changing the second
> load into a nop or copy). HP-UX also does this.

I extended the x86 psABIs with relaxable GOT relocations and
implemented the similar linker optimization in ld in binutils 2.26
and gold also implemented the subset of the linker optimization.

> - Indirect references from PIC code to a protected symbol penalize the
> common case (referencing a symbol within its own module) to support
> the uncommon case, while introducing the nasty side effects of a COPY
> relocation.
>
> - COPY relocations are evil. They bind an application to a specific
> version of a shared library's ABI, and introduce a hidden startup
> cost. If we're going to make any changes, we should be moving towards
> elimination of COPY relocations, rather than disabling features that
> were designed to improve performance.

Copy relocation can improve performance.  Google enabled copy
relocations for PIE in ld/gold and GCC to improve prefomance by
up to 5%:

https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html

It is in GCC 5.

> - Arguing that protected means that the definition is in the same
> module but its address might be external is absurd. The *only* reason
> for the gABI to make that guarantee is so the compilers can optimize
> the code based on the knowledge that the symbol can't be pre-empted.

One possible solution:

1. Compiler accesses protected symbols without GOT in PIC mode
and marks object files no-copy-relocation-against-protected-symbol.
2. Compiler accesses external symbols with GOT in non-PIC mode.
3. Linker marks shared object with no-copy-relocation-against-protected-symbol
if any input files have no-copy-relocation-against-protected-symbol
marker.
4. Linker optimizes out GOT access if symbol is defined locally
in executable.
5. ld.so checks no-copy-relocation-against-protected-symbol marker
on shared object to avoid copy relocation against protected symbol
at run-time.


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 28, 2016, at 5:11 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
> 
> On Thu, Mar 24, 2016 at 11:31 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>>>>> What relocation do you propose to access external protected
>>>>> symbol on x86 for non-PIC code?
>>>> 
>>>> Non-PIC code can still use a GOT, can't it?
>>> 
>>> Yes.
>> 
>> Right.
>> 
>> Some additional thoughts:
>> 
>> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
>> living with this restriction for at least 18 years (since visiblity
>> was introduced into the gABI).
>> 
>> - This has nothing to do with Ulrich's diatribe against protected
>> visibility, which applies only to function symbols (and really only to
>> one platform, due to how function pointer comparison works).
> 
> Copy relocation and protected symbol are mutually exclusive.
> Since copy relocation is the part of x86 psABIs, it limits
> protected symbol effectiveness on x86.  In glibc, we add a
> local alias to symbol we want to protect and access the local
> alias within glibc without GOT.

I had thought about doing this as a workaround for us, but if I understand correctly, this could result in two canonical copies of the object in the process's memory. For what we're using protected symbols for in Swift, we really want only one canonical object, and would prefer that protected references from the executable go through a GOT, similar to how references to external symbols always work in Mach-O.

>> - You don't need to go full-on PIC to reference a protected data
>> symbol in a shared library. You can always statically initialize a
>> pointer variable, and go indirect through that ("poor-man's PIC").
>> 
>> - We could add support for __attribute__((dllimport)) or
>> __declspec(dllimport) when declaring an external variable that is
>> expected to be defined in a shared library (ideally, that attribute
>> ought to be placed in the library's public interface). The compiler
>> can generate a PIC-style reference for that variable without
>> penalizing all the other variables in the main program. This is how
>> HP-UX compilers work on PA and Itanium (using #pragma external).
> 
> This is a useful feature, similar to -fno-plt, like -fgot, but only on
> selective symbols.

"protected" is already selective, though. The code generator had to intentionally set protected visibility on symbols.

> 
>> - Compilers could also generate tentative PIC-style references, with
>> sufficient relocations to allow the linker to convert the indirect
>> reference to a direct reference when possible (changing the second
>> load into a nop or copy). HP-UX also does this.
> 
> I extended the x86 psABIs with relaxable GOT relocations and
> implemented the similar linker optimization in ld in binutils 2.26
> and gold also implemented the subset of the linker optimization.
> 
>> - Indirect references from PIC code to a protected symbol penalize the
>> common case (referencing a symbol within its own module) to support
>> the uncommon case, while introducing the nasty side effects of a COPY
>> relocation.
>> 
>> - COPY relocations are evil. They bind an application to a specific
>> version of a shared library's ABI, and introduce a hidden startup
>> cost. If we're going to make any changes, we should be moving towards
>> elimination of COPY relocations, rather than disabling features that
>> were designed to improve performance.
> 
> Copy relocation can improve performance.  Google enabled copy
> relocations for PIE in ld/gold and GCC to improve prefomance by
> up to 5%:
> 
> https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html
> 
> It is in GCC 5.

Did you look at what the costs were in startup time and dirty pages by using copy relocations? What do you do if the size of the definition changes in a new version of the library?

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Mon, Mar 28, 2016 at 10:01 AM, Joe Groff <jgroff@apple.com> wrote:
>
> On Mar 28, 2016, at 5:11 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>
> On Thu, Mar 24, 2016 at 11:31 AM, Cary Coutant <ccoutant@gmail.com> wrote:
>
> What relocation do you propose to access external protected
> symbol on x86 for non-PIC code?
>
>
> Non-PIC code can still use a GOT, can't it?
>
>
> Yes.
>
>
> Right.
>
> Some additional thoughts:
>
> - This has nothing to do with PIE. Non-PIC-non-PIE code has been
> living with this restriction for at least 18 years (since visiblity
> was introduced into the gABI).
>
> - This has nothing to do with Ulrich's diatribe against protected
> visibility, which applies only to function symbols (and really only to
> one platform, due to how function pointer comparison works).
>
>
> Copy relocation and protected symbol are mutually exclusive.
> Since copy relocation is the part of x86 psABIs, it limits
> protected symbol effectiveness on x86.  In glibc, we add a
> local alias to symbol we want to protect and access the local
> alias within glibc without GOT.
>
>
> I had thought about doing this as a workaround for us, but if I understand
> correctly, this would result in two canonical copies of the object in the

That is correct.

> process's memory. For what we're using protected symbols for in Swift, we
> really want only one canonical object, and would prefer that protected
> references from the executable go through a GOT, similar to how references
> to external symbols always work in Mach-O.

Understood.

> - You don't need to go full-on PIC to reference a protected data
> symbol in a shared library. You can always statically initialize a
> pointer variable, and go indirect through that ("poor-man's PIC").
>
> - We could add support for __attribute__((dllimport)) or
> __declspec(dllimport) when declaring an external variable that is
> expected to be defined in a shared library (ideally, that attribute
> ought to be placed in the library's public interface). The compiler
> can generate a PIC-style reference for that variable without
> penalizing all the other variables in the main program. This is how
> HP-UX compilers work on PA and Itanium (using #pragma external).
>
>
> This is a useful feature, similar to -fno-plt, like -fgot, but only on
> selective symbols.
>
>
> "protected" is already selective, though. The code generator had to
> intentionally set protected visibility on symbols.

Compiler needs a way to know if an external symbol is protected
or not if we don't want to force to use GOT to access ALL external
symbols, even if they turn out to be defined locally at link-time.

>
> - Compilers could also generate tentative PIC-style references, with
> sufficient relocations to allow the linker to convert the indirect
> reference to a direct reference when possible (changing the second
> load into a nop or copy). HP-UX also does this.
>
>
> I extended the x86 psABIs with relaxable GOT relocations and
> implemented the similar linker optimization in ld in binutils 2.26
> and gold also implemented the subset of the linker optimization.
>
> - Indirect references from PIC code to a protected symbol penalize the
> common case (referencing a symbol within its own module) to support
> the uncommon case, while introducing the nasty side effects of a COPY
> relocation.
>
> - COPY relocations are evil. They bind an application to a specific
> version of a shared library's ABI, and introduce a hidden startup
> cost. If we're going to make any changes, we should be moving towards
> elimination of COPY relocations, rather than disabling features that
> were designed to improve performance.
>
>
> Copy relocation can improve performance.  Google enabled copy
> relocations for PIE in ld/gold and GCC to improve prefomance by
> up to 5%:
>
> https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html
>
> It is in GCC 5.
>
>
> Did you look at what the costs were in startup time and dirty pages by using

No, I haven't.  It will be a very useful data.

> copy relocations? What do you do if the size of the definition changes in a
> new version of the library?
>

The size of sys_errlist data symbol in glibc changed over time.  We
use symbol versioning to provide backward binary compatibility:

 1326: 001c7a80   492 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.0
  1327: 001c7a80   500 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.1
  1324: 001c7a80   540 OBJECT  GLOBAL DEFAULT   29 sys_errlist@@GLIBC_2.12
  1328: 001c7a80   504 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.3
  1330: 001c7a80   528 OBJECT  GLOBAL DEFAULT   29 sys_errlist@GLIBC_2.4


-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> Did you look at what the costs were in startup time and dirty pages by using
> copy relocations? What do you do if the size of the definition changes in a
> new version of the library?

There wouldn't be a measurable cost in dirty pages; the copied objects
are simply allocated in bss in the executable.

Startup time wasn't a concern either, simply because COPY relocations
weren't the real issue. We had to support an *occasional* COPY
relocation in order to enable -fPIE to access globals directly rather
than through the GOT. These would be the same COPY relocations that
the apps would have already been using in non-PIE mode (things like
stdin, stdout, and stderr, e.g.).

The size of an object is baked into the ABI when a COPY relocation is
used, so any change in size is an ABI change (often handled, as HJ
pointed out, with versioning). For a lot of common COPY relocations,
that's not really an issue, because both the size and the layout are
baked in through macros like putc() and getc().

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 28, 2016, at 3:21 PM, Cary Coutant <ccoutant@gmail.com> wrote:
> 
>> Did you look at what the costs were in startup time and dirty pages by using
>> copy relocations? What do you do if the size of the definition changes in a
>> new version of the library?
> 
> There wouldn't be a measurable cost in dirty pages; the copied objects
> are simply allocated in bss in the executable.

Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?

-Joe

> 
> Startup time wasn't a concern either, simply because COPY relocations
> weren't the real issue. We had to support an *occasional* COPY
> relocation in order to enable -fPIE to access globals directly rather
> than through the GOT. These would be the same COPY relocations that
> the apps would have already been using in non-PIE mode (things like
> stdin, stdout, and stderr, e.g.).
> 
> The size of an object is baked into the ABI when a COPY relocation is
> used, so any change in size is an ABI change (often handled, as HJ
> pointed out, with versioning). For a lot of common COPY relocations,
> that's not really an issue, because both the size and the layout are
> baked in through macros like putc() and getc().
> 
> -cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>>> Did you look at what the costs were in startup time and dirty pages by using
>>> copy relocations? What do you do if the size of the definition changes in a
>>> new version of the library?
>>
>> There wouldn't be a measurable cost in dirty pages; the copied objects
>> are simply allocated in bss in the executable.
>
> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?

No, references from within the .so would have always used the GOT.
Non-protected global symbols in a shared library are still
pre-emptible, so they are always indirect, and there's always a
dynamic relocation for the GOT entry. Whether the prevailing
definition winds up in the executable or the shared library, the
dynamic loader still has to bind the symbol and apply the relocation.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Joe Groff]

> On Mar 28, 2016, at 3:38 PM, Cary Coutant <ccoutant@gmail.com> wrote:
> 
>>>> Did you look at what the costs were in startup time and dirty pages by using
>>>> copy relocations? What do you do if the size of the definition changes in a
>>>> new version of the library?
>>> 
>>> There wouldn't be a measurable cost in dirty pages; the copied objects
>>> are simply allocated in bss in the executable.
>> 
>> Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
> 
> No, references from within the .so would have always used the GOT.
> Non-protected global symbols in a shared library are still
> pre-emptible, so they are always indirect, and there's always a
> dynamic relocation for the GOT entry. Whether the prevailing
> definition winds up in the executable or the shared library, the
> dynamic loader still has to bind the symbol and apply the relocation.

Ah, right, ELF symbols are preemptible by default, so you'd already be dirtying any pages containing data-to-data references to the symbol regardless.

-Joe

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Alan Modra]

On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
> >>> Did you look at what the costs were in startup time and dirty pages by using
> >>> copy relocations? What do you do if the size of the definition changes in a
> >>> new version of the library?
> >>
> >> There wouldn't be a measurable cost in dirty pages; the copied objects
> >> are simply allocated in bss in the executable.
> >
> > Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
> 
> No, references from within the .so would have always used the GOT.
> Non-protected global symbols in a shared library are still
> pre-emptible, so they are always indirect, and there's always a
> dynamic relocation for the GOT entry. Whether the prevailing
> definition winds up in the executable or the shared library, the
> dynamic loader still has to bind the symbol and apply the relocation.

HJ's changes to protected visibility meant compiler changes so that
protected visibility in shared libraries is no longer seen as local.
So yes, protected visibility symbols in shared libraries now go
through the GOT.  Prior to his changes, they were optimized to a
pc-relative access.  Joe is correct in pointing out that shared
libraries needed a change.  Bad luck if you're using an older
compiler.  Also bad luck if you want to use protected visibility to
optimize your shared library.

HJ also made glibc ld.so changes to ensure the semantics of protected
visibility symbols remain unchanged when multiple shared libraries
define the same protected visibility symbol.

Apparently most people in the gcc and glibc communities saw these
toolchain modifications as fiendishly clever.

-- 
Alan Modra
Australia Development Lab, IBM

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

>> > Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>
>> No, references from within the .so would have always used the GOT.
>> Non-protected global symbols in a shared library are still
>> pre-emptible, so they are always indirect, and there's always a
>> dynamic relocation for the GOT entry. Whether the prevailing
>> definition winds up in the executable or the shared library, the
>> dynamic loader still has to bind the symbol and apply the relocation.
>
> HJ's changes to protected visibility meant compiler changes so that
> protected visibility in shared libraries is no longer seen as local.
> So yes, protected visibility symbols in shared libraries now go
> through the GOT.  Prior to his changes, they were optimized to a
> pc-relative access.  Joe is correct in pointing out that shared
> libraries needed a change.  Bad luck if you're using an older
> compiler.  Also bad luck if you want to use protected visibility to
> optimize your shared library.
>
> HJ also made glibc ld.so changes to ensure the semantics of protected
> visibility symbols remain unchanged when multiple shared libraries
> define the same protected visibility symbol.
>
> Apparently most people in the gcc and glibc communities saw these
> toolchain modifications as fiendishly clever.

I may have misunderstood the context of Joe's question, but I read it
as asking about the effect of Sri's changes to use direct references
to all symbols in PIE mode, rather than the effect of HJ's changes to
use indirect references to protected symbols in PIC mode.

In the former context, we're talking about code where protected
symbols are not often used, references from the executable to shared
library data is infrequent, and the intersection of the two is
exceedingly rare. There was a clear benefit to changing PIE code to
use GOT-relative, rather than GOT-indirect, references to data. It was
only necessary to tweak the linker to allow COPY relocations from a
PIE executable so that the infrequent references from the executable
to (non-protected) shared library data symbols would still work. Any
non-PIC references to protected symbols would have already failed
because the code had already successfully built as a non-PIE
application.

In the latter context, we're talking about penalizing access to
protected symbols within a shared library in order to support that
exceedingly rare case that already didn't work for non-PIE code. I
just don't see the value.

It's too bad that Sri's PIE changes pre-dated HJ's mov-to-lea
link-time optimization. I suspect that that optimization, all by
itself, would have reclaimed most of the lost performance that Sri
observed when turning on -fPIE, and we may never have descended into
this morass.

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[H.J. Lu]

On Mon, Mar 28, 2016 at 4:21 PM, Alan Modra <amodra@gmail.com> wrote:
> On Mon, Mar 28, 2016 at 03:38:01PM -0700, Cary Coutant wrote:
>> >>> Did you look at what the costs were in startup time and dirty pages by using
>> >>> copy relocations? What do you do if the size of the definition changes in a
>> >>> new version of the library?
>> >>
>> >> There wouldn't be a measurable cost in dirty pages; the copied objects
>> >> are simply allocated in bss in the executable.
>> >
>> > Wouldn't references to the symbol from within the .so need to be relocated to reference the now-canonical copy in the executable?
>>
>> No, references from within the .so would have always used the GOT.
>> Non-protected global symbols in a shared library are still
>> pre-emptible, so they are always indirect, and there's always a
>> dynamic relocation for the GOT entry. Whether the prevailing
>> definition winds up in the executable or the shared library, the
>> dynamic loader still has to bind the symbol and apply the relocation.
>
> HJ's changes to protected visibility meant compiler changes so that
> protected visibility in shared libraries is no longer seen as local.
> So yes, protected visibility symbols in shared libraries now go
> through the GOT.  Prior to his changes, they were optimized to a
> pc-relative access.  Joe is correct in pointing out that shared
> libraries needed a change.  Bad luck if you're using an older
> compiler.  Also bad luck if you want to use protected visibility to
> optimize your shared library.
>
> HJ also made glibc ld.so changes to ensure the semantics of protected
> visibility symbols remain unchanged when multiple shared libraries
> define the same protected visibility symbol.
>
> Apparently most people in the gcc and glibc communities saw these
> toolchain modifications as fiendishly clever.
>

As I said before, copy relocation and protected symbol are fundamentally
incompatible.  Since copy relocation is the part of x86 psABIs, I updated
GCC, glibc and ld to make protected symbol to work with copy relocation.
That is protected symbol may be external, but won't be preempted.  The
price I paid is that protected symbol won't be accessed via PC-relative
relocation within the shared object.  To access protected symbol via
PC-relative relocation within the shared object, we need to disable copy
relocation in executable, which is a psABI change.  That is why I proposed
to mark the object as such so that we won't get surprise at run-time.

-- 
H.J.

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Cary Coutant]

> Copy relocation and protected symbol are mutually exclusive.
> Since copy relocation is the part of x86 psABIs, it limits
> protected symbol effectiveness on x86.  In glibc, we add a
> local alias to symbol we want to protect and access the local
> alias within glibc without GOT.

That can be dangerous -- the copy you reference from within glibc may
not be the same copy that all other code uses. You don't want to do
that in general.

>> - Compilers could also generate tentative PIC-style references, with
>> sufficient relocations to allow the linker to convert the indirect
>> reference to a direct reference when possible (changing the second
>> load into a nop or copy). HP-UX also does this.
>
> I extended the x86 psABIs with relaxable GOT relocations and
> implemented the similar linker optimization in ld in binutils 2.26
> and gold also implemented the subset of the linker optimization.

Yes; this came in after Sri's changes to enable direct references to
data for -fPIE. It would be interesting to see what benefit remains
after doing the mov-to-lea optimizations.

> Copy relocation can improve performance.  Google enabled copy
> relocations for PIE in ld/gold and GCC to improve prefomance by
> up to 5%:
>
> https://gcc.gnu.org/ml/gcc-patches/2014-05/msg01215.html
>
> It is in GCC 5.

I think you're overlooking the key point. The performance improvement
wasn't from COPY relocations -- it was from eliminating the load from
the GOT. Being able to use an occasional COPY relocation in PIE code,
as was always supported for non-PIE code, was done to enable the code
generation change, and make PIE more like non-PIE in that respect.

Sri's measurements were for apps that had up-to-then been built as
non-PIC-non-PIE code. We wanted to convert apps to PIE as a security
improvement, but there was a loss of performance due to the indirect
references through the GOT. There really was no good reason for PIE
code to have to go through the GOT to get to data, so this was just a
way to recover some (or most) of the lost performance.

COPY relocations aren't that significant, but because they weren't
supported for PIE code (a totally arbitrary implementation choice),
they were an obstacle to making the code gen change.

> 1. Compiler accesses protected symbols without GOT in PIC mode
> and marks object files no-copy-relocation-against-protected-symbol.
> 2. Compiler accesses external symbols with GOT in non-PIC mode.
> 3. Linker marks shared object with no-copy-relocation-against-protected-symbol
> if any input files have no-copy-relocation-against-protected-symbol
> marker.
> 4. Linker optimizes out GOT access if symbol is defined locally
> in executable.
> 5. ld.so checks no-copy-relocation-against-protected-symbol marker
> on shared object to avoid copy relocation against protected symbol
> at run-time.

This looks like it would work, but it seems like over-engineering to
me. The problem you're trying to solve just isn't that significant (or
new).

-cary

Re: Preventing preemption of 'protected' symbols in GNU ld 2.26

[Maciej W. Rozycki]

On Thu, 24 Mar 2016, Cary Coutant wrote:

> - Arguing that protected means that the definition is in the same
> module but its address might be external is absurd. The *only* reason
> for the gABI to make that guarantee is so the compilers can optimize
> the code based on the knowledge that the symbol can't be pre-empted.

 For the record, SGI's original 64-bit MIPS psABI document[1], which is 
where the concept of symbol's export class aka visibility came from first, 
before having been adopted by the gABI, has these notes on the semantics 
of protected symbols:

"References to protected symbols (and hence to hidden or internal symbols) 
may be optimized by using absolute addresses in executables or by assuming 
addresses to be relatively nearby."

and:

"An R_MIPS_JALR relocation is intended for optimization of jumps to 
protected symbols, i.e. symbols which may not be preempted.  The word to 
be relocated is a jump (typically a JALR) to the indicated symbol.  If it 
is not a preemptible symbol (and therefore defined in the current 
executable/DSO) the relocation is a request to the linker to convert it to 
a direct branch (typically a JAL in the main executable, or a BGEZAL in 
DSOs if the target symbol is close enough).  The linker must check that 
the symbol is not preemptible before performing the relocation, but no 
action is required for correctness -- this is strictly an optimization 
hint."

which clearly indicate the intent for protected symbols to be defined 
locally, in the same way as hidden and internal symbols are, with the 
exception of additionally being exported.  You can't use a BGEZAL 
instruction (which is a direct branch with a limited signed 18-bit range)
if its target is moved outside the DSO it's in.

References:

[1] "64-bit ELF Object File Specification, Draft Version 2.5"
    <http://techpubs.sgi.com/library/manuals/4000/007-4658-001/pdf/007-4658-001.pdf>

  Maciej