Re: lld status with powerpc64

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

On 05/11/2021 04:23, Fangrui Song wrote:
> 
> On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>> Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:
>>
>>> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>
>>>> On 2021-10-26, Adhemerval Zanella wrote:
>>>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>>>     from
>>>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>>>  #3  0x0000000000000000 in ?? ()
>>>>>  (gdb) disas
>>>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>>>
>>> This is a pla, but this GDB isn't able to disassemble it.  This instruction
>>> shouldn't be used unless when configuring glibc using --with-cpu=power10.
>>
>> I can reproduce this issue even when configuring glibc with
>> --with-cpu=power9 --disable-multi-arch, which means the build should not have
>> any Power10 instructions.
>>
>> -- 
>> Tulio Magno
> 
> __gep_setup___vmx__sigjmp_save means a r12 setup stub in ld.lld and is
> used with R_PPC64_REL24_NOTOC for a non-PLT branch.
> 
> Clang only emits `bl foo@notoc` with -mcpu=power10.
> However, sysdeps/powerpc/powerpc64/configure.ac enables USE_PPC64_NOTOC
> when the assembler (gas) supports it.
> 
> 
> Adhemerval noticed that ld.lld has a behavior difference from GNU ld:
> ld.lld defaults to PC-relative paddi stub while GNU ld doesn't (like
> --power10-stubs=no).
> Is USE_PPC64_NOTOC supposed to be used when targeting POWER9 and below?
> If yes, the ld.lld default should be fixed.

I figure out the issue and both bfd, gold, and lld align themselves on the
power10-stubs options handling.  Using --power10-stubs without an arg is 
equivalent to --power10-stubs=yes, but not specifying --power10-stubs at
all should be equivalent to --power10-stubs=auto.  It somewhat confusing,
but I think it is to allow linker and compiler to be independently
regarding power10 stub generation.

The issue is bfd enables power10 relocs generation on stubs iff it sees
the new pc-relative relocations (for instance R_PPC64_D34), otherwise 
it generates default stubs (ppc64_elf_check_relocs:4700).

So we have two options here:

  1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
     aims to support such behavior).

  2. Define USE_PPC64_NOTOC iff linker supports such optimization.
     It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
     link a simple binary without any pcrel and check if the stub has 
     power10 instruction.

  3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
     and on the syscall definition to call the __syscall_error.

I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC 
manually on config.h when configuring with lld I can build glibc.

> 
> 
> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
> USE_PPC64_NOTOC is undefined.
> 
> % gcc --version
> gcc (Debian 8.3.0-6) 8.3.0
> ...
> % ld.bfd --version
> GNU ld (GNU Binutils for Debian) 2.31.1
> ...
> 
> I only see 8 more FAILs with ld.lld than ld.bfd

I am still having trouble to *finish* make check with lld release 13
my environment (some tests stuck on infinite loop).

> 
> 
> % diff -u 0 1
> --- 0   2021-11-05 00:11:43.218731302 -0700
> +++ 1   2021-11-05 00:11:37.659286448 -0700
> @@ -9,6 +9,14 @@
>  FAIL: debug/tst-lfschk6
>  FAIL: dlfcn/bug-atexit3
>  FAIL: elf/check-abi-libc
> +FAIL: elf/ifuncmain1pic
> +FAIL: elf/ifuncmain1pie
> +FAIL: elf/ifuncmain1vis
> +FAIL: elf/ifuncmain1vispic
> +FAIL: elf/ifuncmain1vispie
> +FAIL: elf/ifuncmain3
> +FAIL: elf/ifuncmain6pie
> +FAIL: elf/tst-tlsopt-powerpc
>  FAIL: nptl/tst-cancel24
>  FAIL: nptl/tst-minstack-throw
>  FAIL: nptl/tst-once5
> 
> 
> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
> But perhaps Adhemerval can look a look at it.

I still need to understand why armhf also fails with the ifunc using
protected symbols, although I am not if they are related to IRELATIVE
ordering.

> 
> 
> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
> Let me send a configure patch to disable it...
> 
> Actually I do not know how to disable tst-tlsopt-powerpc properly.
> 
> Perhaps add sysdeps/powerpc/configure.ac and move the
> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
> to sysdeps/powerpc/configure.ac?
> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
> 
> The patch requires some non-trivial configure.ac change, so I'd hope
> that an expert can do it 

Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
by the linker?


> 
> 
> Hey, so lld linked glibc powerpc64 is in a pretty good status!

I am still not sure about it, I did could run some tests but I am still
struggling to get a make check to finish.