Re: global pointer gets overwritten with dlopen(3) on RISC-V

[Palmer Dabbelt <palmer@dabbelt.com>]

On Mon, 15 May 2023 20:56:47 PDT (-0700), maskray@google.com wrote:
> On 2023-05-12, Palmer Dabbelt wrote:
>>On Fri, 12 May 2023 17:05:02 PDT (-0700), maskray@google.com wrote:
>>>On 2023-05-12, Palmer Dabbelt wrote:
>>>>On Fri, 12 May 2023 15:34:21 PDT (-0700), maskray@google.com wrote:
>>>>>On 2023-05-12, Palmer Dabbelt wrote:
>>>>>>On Fri, 12 May 2023 14:09:08 PDT (-0700), maskray@google.com wrote:
>>>>>>>On 2023-05-12, Palmer Dabbelt wrote:
>>>>>>>>On Fri, 12 May 2023 13:11:43 PDT (-0700), fweimer@redhat.com wrote:
>>>>>>>>>* Fangrui Song:
>>>>>>>>>
>>>>>>>>>>>1. Make -mno-relax the default for ld(1) (on Linux?). We have no
>>>>>>>>>>>benchmarks whatsoever, but global variables aren't very popular in
>>>>>>>>>>>application code these days and the gp register allows access to a
>>>>>>>>>>>single memory page (4kB) only. No big deal really.
>>>>>>>>>>
>>>>>>>>>>I do agree that --no-relax-gp is a sensible default choice for GNU ld.
>>>>>>>>>>https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation#global-pointer-relaxation
>>>>>>>>>>
>>>>>>>>>>Perhaps you can start a separate topic on binutils? :)
>>>>>>>>>>
>>>>>>>>>>According to a doc from SiFive about -static -mcpu=sifive-u74 builds,
>>>>>>>>>>https://docs.google.com/spreadsheets/d/14V7cPbyc80AcGHzsMaw9hYb232dzRbGCmTApnxj-SpU/edit#gid=0
>>>>>>>>>>global pointer relaxation saves at best 0.5% size (I guess that refers
>>>>>>>>>>to .text. If we count all allocable sections, the percentage is likely
>>>>>>>>>>even smaller.)
>>>>>>>>>
>>>>>>>>>For a mature toolchain, 0.5% in code size reduction would be *a lot*,
>>>>>>>>>so I wouldn't dismiss that.
>>>>>>>>
>>>>>>>>That's broadly speaking why it sticks around.  We've got a bunch of
>>>>>>>>headaches related to relaxation, GP or otherwise, but they improve
>>>>>>>>performance and nobody's figured out how to replace that yet.
>>>>>>>>
>>>>>>>>>Do we have a reproducer?  Is the issue actually about gp relaxation for
>>>>>>>>>the main executable?
>>>>>>>>
>>>>>>>>In general we don't reference GP from shared libraries as we don't
>>>>>>>>have a GP save/restore scheme.  There may be a bug floating around
>>>>>>>>here somewhere, in which case we should fix it, but the original post
>>>>>>>>sounds like it wasn't a supported use case.
>>>>>>>>
>>>>>>>>>Thanks,
>>>>>>>>>Florian
>>>>>>>
>>>>>>>Global pointer relaxation only applies to +-2KiB data relative to __global_pointer$ (= .sdata + 0x800).
>>>>>>>The area that potentially benefits global pointer relaxation is very small.
>>>>>>>
>>>>>>>0.5% code size reduction (relative to .text?) is the best case. I
>>>>>>>suspect the program somehow has a lot global variable accesses and
>>>>>>>placing these variables in .sdata helps.
>>>>>>>
>>>>>>>I've got results from Yingwei Zheng at PLCT lab using many
>>>>>>>configurations. The saving is like 0.1%.
>>>>>>>https://docs.google.com/spreadsheets/d/1Gz0h-C4U0toa9qELFtRaEWT_CzauE5JD9xMsLR8RyK8/edit#gid=1721258109
>>>>>>>
>>>>>>>On the binutils side, we occasionally see patches to fix global pointer
>>>>>>>relaxation bugs, e.g. the patch just sent a few hours ago:
>>>>>>>https://sourceware.org/pipermail/binutils/2023-May/127413.html
>>>>>>>
>>>>>>>I do not know the embedded toolchain well, but for Linux desktop/server,
>>>>>>>disabling global pointer relaxation seems like a sensible choice. If we
>>>>>>>discover a better way to utilize GP (x3) in the future, disabling global
>>>>>>>pointer relaxation today will result in fewer compatibility issues.
>>>>>>
>>>>>>This comes up all the time, you're just pushing for a backdoor ABI
>>>>>>break.  I get the desire to remove GP, if we were to be able to redo
>>>>>>things I'd also not have it, but it's in the ABI and we can't change
>>>>>>the binaries that exist.
>>>>>>
>>>>>>If you want a GP-free ABI then you should just go write one up.  Then
>>>>>>it'll become a distro problem, and if it turns out that users also
>>>>>>don't want in then the GP ABI will rot and we can eventually deprecate
>>>>>>it.
>>>>>
>>>>>I am advocating for a change in GNU ld to make --no-relax-gp the default
>>>>>option, but I am not sure it can be considered an ABI break.
>>>>>
>>>>>When using ld --no-relax-gp, the conversion of code sequences to gp is
>>>>>disabled, thus eliminating an assumption related to global pointer relaxation.
>>>>>If an executable is relinked without global pointer relaxation, it should still
>>>>>function properly.
>>>>>
>>>>>To the best of my knowledge, there is no official documentation that designates
>>>>>linker relaxation as a mandatory feature. Relaxation schemes, including global
>>>>>pointer relaxation, are optional. Making an optional feature opt-in does not
>>>>>constitute an ABI break.
>>>>>
>>>>>glibc continues to initialize gp to __global_pointer$ to accommodate users who
>>>>>opt-in for global pointer relaxation. Removing the initialization of gp would
>>>>>indeed be an ABI break, and I have never proposed such a change.
>>>>
>>>>The ABI break is using GP for something else, but that always ends up
>>>>being part of the argument for disabling GP-based relaxation.  Without
>>>>that the argument ends up just being that the GP-relaxations are too
>>>>complicated for the benefit they provide.
>>>
>>>This has happened. Some platforms are investigating or using GP for
>>>software shadow call stack. I think they include Android and Fuchsia.  I
>>>am not affiliated with them.
>>
>>IIUC the Android shadow stack stuff isn't 100% decided, but it's the
>>way they're headed.  I'd argue that's a great reason to actually write
>>this down in an ABI spec: we've got users who either are or will soon
>>violate the spec, we should fix the spec rather than force those users
>>to fork it.
>>
>>>>I certainly understand that argument, these (and for the rest of the
>>>>relaxation stuff) is a huge amount of pain for a small amount of
>>>>benefit (which seems to be mostly based on benchmark dragracing, which
>>>>is never that strong of an argument).  That said, last time we had
>>>>this discussion was only a few months ago.  I don't think anything has
>>>>changed since then so I doubt things well go any differently.
>>>
>>>The rest of the relaxation mechanism does offer significant savings that
>>>should not be overlooked. I have heard claims of up to a 10% .text
>>>reduction in certain embedded systems.
>>>
>>>However, it is important to acknowledge the costs associated with
>>>increased toolchain complexity and bloating of debug information, LSDA ,
>>>and custom metadata sections. I discussed these concerns in detail at
>>>https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation
>>>(Apologies if the title came across as derogatory. It was never my
>>>intention, and I genuinely enjoyed delving into the intricacies of
>>>this toolchain technology.)
>>
>>I'd argue we should make this new ABI just forbid all relaxation.
>>It's a ton of headache for an unknown amount of benefit, we can't
>>easily measure it because we haven't chased down all the missed
>>optimizations due to relaxation.  Modern Linux distros are all PIE
>>anyway, so it's not like most of the relaxation does much.
>>
>>Pretty much everyone who looks at RISC-V says relaxation is bad, let's
>>go let folks prove that out -- and if it's right we can get ride of a
>>pile of complexity, which is great.
>
> It's too late to make linker relaxation opt-in. I'll do my part to fix
> some LLVM assembler deficiency, though.
>
>>>>Just turning GP-based relaxation off by default doesn't get rid of the
>>>>complexity, they're still in the ABI and people will still use them
>>>>(even if just for benchmark dragracing) so they'll still need to work.
>>>
>>>Regarding the ABI story, shared objects do not utilize global pointer
>>>relaxation. Therefore, if only an executable requires GP for a
>>>platform-specific purpose, I believe there is no ABI risk involved. The
>>>risk arises when shared objects rely on the executable's initialized GP
>>>for certain purposes.
>>>
>>>ABI breaks related to global pointer usage occur when a platform
>>>initially ships something using GP and then switches to a different GP
>>>usage.
>>
>>Using GP for anything that's not the global pointer violates the
>>psABI.
>
> I disagree.  The psABI makes it clear that linker relaxation is optional
> with wording like "With linker relaxation enabled" "the linker is
> permitted to use ... relaxation".
>
>>It's just a spec violation and I agree it's possible to
>>violate the specs while still producing working systems.  At that
>>point it's really up to the projects to ensure their spec-violating
>>systems work, there's not much we can do on the toolchain side of
>>things.  I'm guilty of doing that too (just look at a buncho of the
>>Linux port, for example), but that's always a complicated way to do
>>things.
>
> "If a platform requires use of a dedicated general-purpose register for a
> platform-specific purpose, it is recommended to use gp (x3). The
> platform ABI specification must document the use of this register. For
> such platforms, care must be taken to ensure all code (compiler
> generated or otherwise) avoids using gp in a way incompatible with the
> platform specific purpose, and that global
> pointer relaxation is disabled in the toolchain."
>
>>IMO it's still better to have a spec written down so the toolchain and
>>users can be on the same page.
>
> You can make a proposal that global pointer must not be used for other
> purposes and someone will object to it:)

https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/379

>>>As I mentioned earlier, many non-Linux/glibc platforms don't use global
>>>pointer relaxation from the beginning.  Even within Linux/glibc, a new
>>>distribution may choose not to use global pointer relaxation initially.
>>>For these platforms, there would be no ABI break.
>>>
>>>Changing the default behavior does indeed have an impact.
>>>--no-relax-gp is a binutils 2.41 feature, which means that projects that
>>>discover improved ways to utilize GP in the future need to avoid
>>>--no-relax-gp to maintain compatibility with older GNU ld versions.
>>>They could use --no-relax with GNU ld versions, but that's a big hammer.
>>
>>I would describe that as an ABI break.
>
> I provide evidence to back up my point.
> Can you give reasons that this is an ABI break? And break what?
>
>>>
>>>>>
>>>>>>>Haiku
>>>>>>>(https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/298#issuecomment-1344724796), Android, and Fuchsia
>>>>>>>have mentioned that they don't use global pointer relaxation.