Re: [PATCH 0/3] Improve lld support and current status

[Fangrui Song <maskray@google.com>]

On 2021-10-27, Adhemerval Zanella wrote:
>
>
>On 26/10/2021 17:33, Fangrui Song wrote:
>>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>>> arm the loader fails to build:
>>>
>>>  ld.so fails with
>>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>>  _dl_argv; recompile with -fPIC
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>>  >>> referenced by rtld.c:164
>>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
>>
>> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
>> Such a relocation referencing a preemptible symbol cannot be used.
>>
>> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
>>
>> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.
>
>I think it is something else, the symbols is defined as hidden (output of
>preprocessor):
>
>  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));
>
>However, hidden is set only the internal symbol:
>
>  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
>    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
>    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv
>
>And on rtld.c assembly we have the directive to use the internal definition:
>
>    .set    _dl_argv,__GI__dl_argv
>
>So why lld is not binding the usage to internal to the local hidden alias?

I use the Debian package g++-arm-linux-gnueabihf (nowadays glibc build needs a C++ compiler even if I don't build tests...).

mkdir -p out/arm; cd out/arm
../../configure --prefix=/tmp/glibc/arm --host=arm-linux-gnueabihf
make -j 50

Here is GNU ld produced elf/librtld.os:

% readelf -Ws elf/librtld.os | grep _dl_argv
   1582: 00000000     4 OBJECT  GLOBAL HIDDEN    13 __GI__dl_argv
   1603: 00000000     4 OBJECT  GLOBAL DEFAULT   13 _dl_argv

So LLD's librtld.os matches GNU ld for the two symbols.



The .set directive sets value/type but not binding/visibility.
Perhaps sysdeps/arm/dl-machine.h should use

.word   __GI__dl_argv(GOTOFF)

instead of 

.word   _dl_argv(GOTOFF)

>>
>>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>>> an error instead of silent ignoring it:
>>>
>>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>>  ld.lld: error: unknown argument '-relax'
>>
>> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
>> (see https://lld.llvm.org/ "production quality").
>> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).
>
>Right, but lld is still showing a different ld.bfd semantic that might trigger
>other issues where on platforms where neither --relax nor -no-relax it supported
>ld.bfd accepts both argument but ignore it [1].
>
>At least on sparc, gcc does pass --relax and expects that static linker just
>ignore if it is no supported.
>
>[1] https://sourceware.org/binutils/docs-2.37/ld.html

LLD's sparcv9 port is quite immature, so I cannot even advertise it.
A large portion should be rewritten but the sparc port is only cared by very few
retrocomputing fans now..

>>
>>> And even when -mno-relax is explicit add, lld does not support some relocations
>>> generated by GCC:
>>>
>>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>>  | head -n2
>>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>>  0000000000000058 _dl_skip_args + 0
>>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>>  0000000000000058 _dl_skip_args + 0
>>>
>>> The mips/mipsel also fails to build the loader:
>>>
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>>
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
>>
>> lld.llvm.org says "MIPS seems decent too." and I personally know really
>> little about MIPS.
>>
>> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
>> Relocation processing is rigid and consistent in LLD.
>> GNU ld may miss some error checking for some relocation types.
>>
>>> Similar to mips64/mips64el:
>>>
>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>  against non-function symbol . This is invalid and most likely a compiler
>>>  bug.
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>
>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>> reporting a genuine issue.
>
>I give you that loader code is tricky, but it does not really explain why
>lld is failing to link the eh_frame.  It *might* due some internal assembly
>routines, but at least from logs it does not seem so.

Hope Joseph can answer the question.

If we have easy-to-follow instructions building glibc mips with LLD, I can
forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
contributor of LLD's MIPS port)...

But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.

>>
>>> The riscv32/riscv64 fails to the loader, but it is due missing support
>>> to relaxation:
>>>
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-relax
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-relax
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-rela
>>
>> LLD does not implement RISC-V linker relaxation.
>> R_RISCV_ALIGN has weird linker unfriendly semantics:
>> simply ignoring the relocation type can break semantics
>> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
>>
>>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
>>
>> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...
>
>In fact it is, the problem was the autogenerated syscalls built from
>the assembly.  I could avoid the R_RISCV_ALIGN with:
>
>diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
>index 20a9968106..28dbd89cb0 100644
>--- a/sysdeps/riscv/Makefile
>+++ b/sysdeps/riscv/Makefile
>@@ -5,3 +5,9 @@ endif
> # RISC-V's assembler also needs to know about PIC as it changes the definition
> # of some assembler macros.
> ASFLAGS-.os += $(pic-ccflag)
>+
>+# lld does not implement R_RISCV_ALIGN relaxation optimization.
>+ifeq (yes,$(with-lld))
>+ASFLAGS-.os += -Wa,-mno-relax
>+sysdep-CFLAGS += -mno-relax
>+endif
>
>
>However it does fail later for the libc.so with:
>
>ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
>ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)
>
>ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)

There is a GNU ld issue: https://sourceware.org/bugzilla/show_bug.cgi?id=28509
It incorrectly allows non-PLT non-GOT relocation referencing a preemptible symbol.


LLD uses consistent categories (e.g. R_PC / R_PLT_PC) to describe relocation types.
It has consistent diagnostic across all ports it supports.
GNU ld uses dispatches for each relocation type and may miss some diagnostics.

>collect2: error: ld returned 1 exit status