Re: [PATCH 0/3] Improve lld support and current status

[Adhemerval Zanella <adhemerval.zanella@linaro.org>]

On 26/10/2021 17:33, Fangrui Song wrote:
>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>> arm the loader fails to build:
>>
>>  ld.so fails with
>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>  _dl_argv; recompile with -fPIC
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>  >>> referenced by rtld.c:164
>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
> 
> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
> Such a relocation referencing a preemptible symbol cannot be used.
> 
> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
> 
> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.

I think it is something else, the symbols is defined as hidden (output of
preprocessor):

  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));

However, hidden is set only the internal symbol:

  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv

And on rtld.c assembly we have the directive to use the internal definition:

    .set    _dl_argv,__GI__dl_argv

So why lld is not binding the usage to internal to the local hidden alias?

> 
>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>> an error instead of silent ignoring it:
>>
>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>  ld.lld: error: unknown argument '-relax'
> 
> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
> (see https://lld.llvm.org/ "production quality").
> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).

Right, but lld is still showing a different ld.bfd semantic that might trigger
other issues where on platforms where neither --relax nor -no-relax it supported
ld.bfd accepts both argument but ignore it [1].

At least on sparc, gcc does pass --relax and expects that static linker just
ignore if it is no supported.

[1] https://sourceware.org/binutils/docs-2.37/ld.html


> 
>> And even when -mno-relax is explicit add, lld does not support some relocations
>> generated by GCC:
>>
>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>  | head -n2
>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>  0000000000000058 _dl_skip_args + 0
>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>  0000000000000058 _dl_skip_args + 0
>>
>> The mips/mipsel also fails to build the loader:
>>
>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>
>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
> 
> lld.llvm.org says "MIPS seems decent too." and I personally know really
> little about MIPS.
> 
> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
> Relocation processing is rigid and consistent in LLD.
> GNU ld may miss some error checking for some relocation types.
> 
>> Similar to mips64/mips64el:
>>
>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>  against non-function symbol . This is invalid and most likely a compiler
>>  bug.
>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
> 
> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
> reporting a genuine issue.

I give you that loader code is tricky, but it does not really explain why
lld is failing to link the eh_frame.  It *might* due some internal assembly
routines, but at least from logs it does not seem so.

> 
>> The riscv32/riscv64 fails to the loader, but it is due missing support
>> to relaxation:
>>
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-relax
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-relax
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-rela
> 
> LLD does not implement RISC-V linker relaxation.
> R_RISCV_ALIGN has weird linker unfriendly semantics:
> simply ignoring the relocation type can break semantics
> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
> 
>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
> 
> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...

In fact it is, the problem was the autogenerated syscalls built from
the assembly.  I could avoid the R_RISCV_ALIGN with:

diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
index 20a9968106..28dbd89cb0 100644
--- a/sysdeps/riscv/Makefile
+++ b/sysdeps/riscv/Makefile
@@ -5,3 +5,9 @@ endif
 # RISC-V's assembler also needs to know about PIC as it changes the definition
 # of some assembler macros.
 ASFLAGS-.os += $(pic-ccflag)
+
+# lld does not implement R_RISCV_ALIGN relaxation optimization.
+ifeq (yes,$(with-lld))
+ASFLAGS-.os += -Wa,-mno-relax
+sysdep-CFLAGS += -mno-relax
+endif


However it does fail later for the libc.so with:

ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)

ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)
collect2: error: ld returned 1 exit status