[PATCH 0/3] Improve lld support and current status

[PATCH 0/3] Improve lld support and current status

[Adhemerval Zanella]

Ths patch allows x86_64, i686, aarch64, powerpc64le, powerpc64, and
powerpc to build, including the tests, with LLD packed in LLVM 13 [1].

For x86_64 and aarch64 there is no regression in testssuite.  On 
i686 there is only one:

  FAIL: elf/ifuncmain6pie

Which segfaults calling the 'foo' functions:

  Program received signal SIGSEGV, Segmentation fault.
  0xfb010101 in ?? ()
  (gdb) bt
  #0  0xfb010101 in ?? ()
  #1  0xf7fba620 in call_foo () at ifuncmod6.c:18
  #2  0xf7fc0857 in ?? ()
  #3  0xf7e1b195 in __libc_start_call_main (main=main@entry=0xf7fc0830,
  argc=argc@entry=2, argv=argv@entry=0xffffc610) at
  ../sysdeps/nptl/libc_start_call_main.h:58
  #4  0xf7e1b286 in __libc_start_main_impl (main=0xf7fc0830, argc=2,
  argv=0xffffc610, init=0x0, fini=0x0, rtld_fini=0xf7fe79b0 <_dl_fini>,
  stack_end=0xffffc60c) at ../csu/libc-start.c:409
  #5  0xf7fc06d7 in ?? ()

The powerpc64le fails at libc.so start:

  (gdb) bt
  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
    from
  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
     from
  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
     from
  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
  #3  0x0000000000000000 in ?? ()
  (gdb) disas
  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
     0x00007ffff7f2a984 <+4>:     li      r12,-1280
     0x00007ffff7f2a988 <+8>:     mtctr   r12
     0x00007ffff7f2a98c <+12>:    bctr

And powerpc64-linux-gnu fails on dynamic loaders start:

  (gdb) bt
  #0  0x00007ffff7ffa9a8 in _start ()
  (gdb) disas
  Dump of assembler code for function _start:
  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
     0x00007ffff7ffa9ac <+4>:     .long 0x0
     0x00007ffff7ffa9b0 <+8>:     .long 0x0
     0x00007ffff7ffa9b4 <+12>:    .long 0x0


The arm, sparcv9, mips, and riscv fail to build due different issues. On
arm the loader fails to build:

  ld.so fails with
  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
  _dl_argv; recompile with -fPIC
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
  >>> referenced by rtld.c:164
  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)

On sparcv9, lld handles --relax option different than ld.bfd: it throws
an error instead of silent ignoring it:

  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
  ld.lld: error: unknown argument '-relax'

And even when -mno-relax is explicit add, lld does not support some relocations
generated by GCC:

  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
  | head -n2
  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
  0000000000000058 _dl_skip_args + 0
  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
  0000000000000058 _dl_skip_args + 0

The mips/mipsel also fails to build the loader:

  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
  symbol in readonly segment; recompile object files with -fPIC or pass
  '-Wl,-z,notext' to allow text relocations in the output
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)

  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
  symbol in readonly segment; recompile object files with -fPIC or pass
  '-Wl,-z,notext' to allow text relocations in the output
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)

Similar to mips64/mips64el:

  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
  against non-function symbol . This is invalid and most likely a compiler
  bug.
  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
  symbol in readonly segment; recompile object files with -fPIC or pass
  '-Wl,-z,notext' to allow text relocations in the output
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)

The riscv32/riscv64 fails to the loader, but it is due missing support
to relaxation:

  ld.lld: error:
  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
  recompile with -mno-relax
  ld.lld: error:
  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
  recompile with -mno-relax
  ld.lld: error:
  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
  recompile with -mno-rela

Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.


[1] https://releases.llvm.org/download.html#13.0.0

Adhemerval Zanella (3):
  elf: Disable ifuncmain{1,5,5pic,5pie} when using LLD
  Fix LIBC_PROG_BINUTILS for -fuse-ld=lld
  Check if linker also support -mtls-dialect=gnu2

 aclocal.m4   |  6 +++++-
 configure    | 10 +++++++---
 configure.ac |  4 ++--
 elf/Makefile | 17 +++++++++++++----
 4 files changed, 27 insertions(+), 10 deletions(-)

-- 
2.32.0

[PATCH 1/3] elf: Disable ifuncmain{1,5,5pic,5pie} when using LLD

[Adhemerval Zanella]

These tests takes the address of a protected symbol (foo_protected)
and lld does not support copy relocations on protected data symbols.

Checked on x86_64-linux-gnu.
---
 elf/Makefile | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/elf/Makefile b/elf/Makefile
index bdcf4cb885..477207a297 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -427,11 +427,14 @@ ifeq (yes,$(build-shared))
 tests += tst-ifunc-fault-lazy tst-ifunc-fault-bindnow
 # Note: sysdeps/x86_64/ifuncmain8.c uses ifuncmain8.
 tests-internal += \
-	 ifuncmain1 ifuncmain1pic ifuncmain1vis ifuncmain1vispic \
+	 ifuncmain1pic ifuncmain1vis ifuncmain1vispic \
 	 ifuncmain1staticpic \
 	 ifuncmain2 ifuncmain2pic ifuncmain3 ifuncmain4 \
-	 ifuncmain5 ifuncmain5pic ifuncmain5staticpic \
+	 ifuncmain5staticpic \
 	 ifuncmain7 ifuncmain7pic
+ifeq (no,$(with-lld))
+tests-internal += ifuncmain1 ifuncmain5 ifuncmain5pic
+endif
 ifeq (yes,$(have-gcc-ifunc))
 tests-internal += ifuncmain9 ifuncmain9pic
 endif
@@ -441,17 +444,23 @@ extra-test-objs += $(ifunc-test-modules:=.o)
 test-internal-extras += $(ifunc-test-modules)
 ifeq (yes,$(have-fpie))
 ifunc-pie-tests = ifuncmain1pie ifuncmain1vispie ifuncmain1staticpie \
-		  ifuncmain5pie ifuncmain6pie ifuncmain7pie
+		  ifuncmain6pie ifuncmain7pie
 ifeq (yes,$(have-gcc-ifunc))
 ifunc-pie-tests += ifuncmain9pie
 endif
+ifeq (no,$(with-lld))
+ifunc-pie-tests += ifuncmain5pie
+endif
 ifeq (yes,$(have-textrel_ifunc))
 ifunc-pie-tests += tst-ifunc-textrel
 endif
 tests-internal += $(ifunc-pie-tests)
 tests-pie += $(ifunc-pie-tests)
 endif
-modules-names += ifuncmod1 ifuncmod3 ifuncmod5 ifuncmod6
+modules-names += ifuncmod1 ifuncmod3 ifuncmod6
+ifeq (no,$(with-lld))
+modules-names += ifuncmod5
+endif
 endif
 endif
 
-- 
2.32.0

[PATCH 2/3] Fix LIBC_PROG_BINUTILS for -fuse-ld=lld

[Adhemerval Zanella]

GCC does not print the correct linker when -fuse-ld=lld is used with
the -print-prog-name=ld:

  $ gcc -v 2>&1 | tail -n 1
  gcc version 11.2.0 (Ubuntu 11.2.0-7ubuntu2)
  $ gcc
  ld

This is different than for gold:

  $ gcc -fuse-ld=gold -print-prog-name=ld
  ld.gold

Using ld.lld as the static linker name prints the expected result.

This is only required when -fuse-ld=lld is used, if lld is used as
the 'ld' programs (through a symlink) LIBC_PROG_BINUTILS works
as expected.

Checked on x86_64-linux-gnu.
---
 aclocal.m4 | 6 +++++-
 configure  | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/aclocal.m4 b/aclocal.m4
index c195c4db56..83da631822 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -114,8 +114,12 @@ if test -n "$path_binutils"; then
     path_binutils=`(cd $path_binutils; pwd) | sed 's%/*$%/%'`
     CC="$CC -B$path_binutils"
 fi
+case "$CC" in
+    *fuse-ld=lld*) LDNAME=ld.lld;;
+    *)             LDNAME=ld;;
+esac
 AS=`$CC -print-prog-name=as`
-LD=`$CC -print-prog-name=ld`
+LD=`$CC -print-prog-name=$LDNAME`
 AR=`$CC -print-prog-name=ar`
 AC_SUBST(AR)
 OBJDUMP=`$CC -print-prog-name=objdump`
diff --git a/configure b/configure
index 39d75eb4ed..2a28c58cd6 100755
--- a/configure
+++ b/configure
@@ -4549,8 +4549,12 @@ if test -n "$path_binutils"; then
     path_binutils=`(cd $path_binutils; pwd) | sed 's%/*$%/%'`
     CC="$CC -B$path_binutils"
 fi
+case "$CC" in
+    *fuse-ld=lld*) LDNAME=ld.lld;;
+    *)             LDNAME=ld;;
+esac
 AS=`$CC -print-prog-name=as`
-LD=`$CC -print-prog-name=ld`
+LD=`$CC -print-prog-name=$LDNAME`
 AR=`$CC -print-prog-name=ar`
 
 OBJDUMP=`$CC -print-prog-name=objdump`
-- 
2.32.0

[PATCH 3/3] Check if linker also support -mtls-dialect=gnu2

[Adhemerval Zanella]

Since some linkers (for instance lld for i386) does not support it
for all architectures.

Checked on i686-linux-gnu.
---
 configure    | 4 ++--
 configure.ac | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/configure b/configure
index 2a28c58cd6..c66082232e 100755
--- a/configure
+++ b/configure
@@ -6324,8 +6324,8 @@ void foo (void)
   i = 10;
 }
 EOF
-if { ac_try='${CC-cc} $CFLAGS $CPPFLAGS -S -fPIC -mtls-dialect=gnu2
-		   conftest.c 1>&5'
+if { ac_try='${CC-cc} $CFLAGS $CPPFLAGS -fPIC -mtls-dialect=gnu2 -nostdlib -nostartfiles
+		   conftest.c -o conftest 1>&5'
   { { eval echo "\"\$as_me\":${as_lineno-$LINENO}: \"$ac_try\""; } >&5
   (eval $ac_try) 2>&5
   ac_status=$?
diff --git a/configure.ac b/configure.ac
index 00f49f09f7..7eb4239359 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1493,8 +1493,8 @@ void foo (void)
   i = 10;
 }
 EOF
-if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -S -fPIC -mtls-dialect=gnu2
-		   conftest.c 1>&AS_MESSAGE_LOG_FD])
+if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -fPIC -mtls-dialect=gnu2 -nostdlib -nostartfiles
+		   conftest.c -o conftest 1>&AS_MESSAGE_LOG_FD])
 then
   libc_cv_mtls_dialect_gnu2=yes
 else
-- 
2.32.0

Re: [PATCH 0/3] Improve lld support and current status

[Fangrui Song]

On 2021-10-26, Adhemerval Zanella wrote:
>Ths patch allows x86_64, i686, aarch64, powerpc64le, powerpc64, and
>powerpc to build, including the tests, with LLD packed in LLVM 13 [1].

Thanks for checking!

>For x86_64 and aarch64 there is no regression in testssuite.  On
>i686 there is only one:
>
>  FAIL: elf/ifuncmain6pie
>
>Which segfaults calling the 'foo' functions:
>
>  Program received signal SIGSEGV, Segmentation fault.
>  0xfb010101 in ?? ()
>  (gdb) bt
>  #0  0xfb010101 in ?? ()
>  #1  0xf7fba620 in call_foo () at ifuncmod6.c:18
>  #2  0xf7fc0857 in ?? ()
>  #3  0xf7e1b195 in __libc_start_call_main (main=main@entry=0xf7fc0830,
>  argc=argc@entry=2, argv=argv@entry=0xffffc610) at
>  ../sysdeps/nptl/libc_start_call_main.h:58
>  #4  0xf7e1b286 in __libc_start_main_impl (main=0xf7fc0830, argc=2,
>  argv=0xffffc610, init=0x0, fini=0x0, rtld_fini=0xf7fe79b0 <_dl_fini>,
>  stack_end=0xffffc60c) at ../csu/libc-start.c:409
>  #5  0xf7fc06d7 in ?? ()
>
>The powerpc64le fails at libc.so start:
>
>  (gdb) bt
>  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
>    from
>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
>     from
>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>     from
>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>  #3  0x0000000000000000 in ?? ()
>  (gdb) disas
>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>     0x00007ffff7f2a988 <+8>:     mtctr   r12
>     0x00007ffff7f2a98c <+12>:    bctr
>
>And powerpc64-linux-gnu fails on dynamic loaders start:
>
>  (gdb) bt
>  #0  0x00007ffff7ffa9a8 in _start ()
>  (gdb) disas
>  Dump of assembler code for function _start:
>  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
>     0x00007ffff7ffa9ac <+4>:     .long 0x0
>     0x00007ffff7ffa9b0 <+8>:     .long 0x0
>     0x00007ffff7ffa9b4 <+12>:    .long 0x0

CCed the powerpc maintainer.

I am not familiar with debugging rtld for powerpc64...

>
>The arm, sparcv9, mips, and riscv fail to build due different issues. On
>arm the loader fails to build:
>
>  ld.so fails with
>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>  _dl_argv; recompile with -fPIC
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>  >>> referenced by rtld.c:164
>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)

R_ARM_GOTOFF32 = S + A - GOT_ORG  .
Such a relocation referencing a preemptible symbol cannot be used.

Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501

_dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.

>On sparcv9, lld handles --relax option different than ld.bfd: it throws
>an error instead of silent ignoring it:
>
>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>  ld.lld: error: unknown argument '-relax'

LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
(see https://lld.llvm.org/ "production quality").
The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).

>And even when -mno-relax is explicit add, lld does not support some relocations
>generated by GCC:
>
>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>  | head -n2
>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>  0000000000000058 _dl_skip_args + 0
>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>  0000000000000058 _dl_skip_args + 0
>
>The mips/mipsel also fails to build the loader:
>
>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>  symbol in readonly segment; recompile object files with -fPIC or pass
>  '-Wl,-z,notext' to allow text relocations in the output
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>
>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>  symbol in readonly segment; recompile object files with -fPIC or pass
>  '-Wl,-z,notext' to allow text relocations in the output
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)

lld.llvm.org says "MIPS seems decent too." and I personally know really
little about MIPS.

However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
Relocation processing is rigid and consistent in LLD.
GNU ld may miss some error checking for some relocation types.

>Similar to mips64/mips64el:
>
>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>  against non-function symbol . This is invalid and most likely a compiler
>  bug.
>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>  symbol in readonly segment; recompile object files with -fPIC or pass
>  '-Wl,-z,notext' to allow text relocations in the output
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)

I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
reporting a genuine issue.

>The riscv32/riscv64 fails to the loader, but it is due missing support
>to relaxation:
>
>  ld.lld: error:
>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>  recompile with -mno-relax
>  ld.lld: error:
>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>  recompile with -mno-relax
>  ld.lld: error:
>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>  recompile with -mno-rela

LLD does not implement RISC-V linker relaxation.
R_RISCV_ALIGN has weird linker unfriendly semantics:
simply ignoring the relocation type can break semantics
(https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).

>Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.

So sysdep-CFLAGS is not a catch-all option affecting all C compiles...

>
>[1] https://releases.llvm.org/download.html#13.0.0
>
>Adhemerval Zanella (3):
>  elf: Disable ifuncmain{1,5,5pic,5pie} when using LLD
>  Fix LIBC_PROG_BINUTILS for -fuse-ld=lld
>  Check if linker also support -mtls-dialect=gnu2
>
> aclocal.m4   |  6 +++++-
> configure    | 10 +++++++---
> configure.ac |  4 ++--
> elf/Makefile | 17 +++++++++++++----
> 4 files changed, 27 insertions(+), 10 deletions(-)
>
>-- 
>2.32.0
>

Re: [PATCH 2/3] Fix LIBC_PROG_BINUTILS for -fuse-ld=lld

[Fangrui Song]

On 2021-10-26, Adhemerval Zanella wrote:
>GCC does not print the correct linker when -fuse-ld=lld is used with
>the -print-prog-name=ld:
>
>  $ gcc -v 2>&1 | tail -n 1
>  gcc version 11.2.0 (Ubuntu 11.2.0-7ubuntu2)
>  $ gcc
>  ld
>
>This is different than for gold:
>
>  $ gcc -fuse-ld=gold -print-prog-name=ld
>  ld.gold

I think it may be debatable whether -fuse-ld={bfd,gold,lld} should
affect -print-prog-name=ld output.

   # Should -fuse-ld= do anything with this?
   % aarch64-linux-gnu-gcc -print-prog-name=../bin/ld
   /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/../bin/ld

gcc/gcc.c has OPT_fuse_ld_bfd but not OPT_fuse_ld_lld, which may explain
the behavior difference.

Clang does not have the GCC -fuse-ld= --print-prog-name=ld behavior.




>Using ld.lld as the static linker name prints the expected result.
>
>This is only required when -fuse-ld=lld is used, if lld is used as
>the 'ld' programs (through a symlink) LIBC_PROG_BINUTILS works
>as expected.
>
>Checked on x86_64-linux-gnu.
>---
> aclocal.m4 | 6 +++++-
> configure  | 6 +++++-
> 2 files changed, 10 insertions(+), 2 deletions(-)
>
>diff --git a/aclocal.m4 b/aclocal.m4
>index c195c4db56..83da631822 100644
>--- a/aclocal.m4
>+++ b/aclocal.m4
>@@ -114,8 +114,12 @@ if test -n "$path_binutils"; then
>     path_binutils=`(cd $path_binutils; pwd) | sed 's%/*$%/%'`
>     CC="$CC -B$path_binutils"
> fi
>+case "$CC" in
>+    *fuse-ld=lld*) LDNAME=ld.lld;;
>+    *)             LDNAME=ld;;

Perhaps hard code -fuse-ld=bfd and -fuse-ld=gold as well?

I think gcc -print-prog-name=ld.bfd is more robust than
gcc -print-prog-name=ld -fuse-ld=bfd.

>+esac
> AS=`$CC -print-prog-name=as`
>-LD=`$CC -print-prog-name=ld`
>+LD=`$CC -print-prog-name=$LDNAME`
> AR=`$CC -print-prog-name=ar`
> AC_SUBST(AR)
> OBJDUMP=`$CC -print-prog-name=objdump`
>diff --git a/configure b/configure
>index 39d75eb4ed..2a28c58cd6 100755
>--- a/configure
>+++ b/configure
>@@ -4549,8 +4549,12 @@ if test -n "$path_binutils"; then
>     path_binutils=`(cd $path_binutils; pwd) | sed 's%/*$%/%'`
>     CC="$CC -B$path_binutils"
> fi
>+case "$CC" in
>+    *fuse-ld=lld*) LDNAME=ld.lld;;
>+    *)             LDNAME=ld;;
>+esac
> AS=`$CC -print-prog-name=as`
>-LD=`$CC -print-prog-name=ld`
>+LD=`$CC -print-prog-name=$LDNAME`
> AR=`$CC -print-prog-name=ar`
>
> OBJDUMP=`$CC -print-prog-name=objdump`
>-- 
>2.32.0
>

Re: [PATCH 3/3] Check if linker also support -mtls-dialect=gnu2

[Fāng-ruì Sòng]

On Tue, Oct 26, 2021 at 1:03 PM Adhemerval Zanella
<adhemerval.zanella@linaro.org> wrote:
>
> Since some linkers (for instance lld for i386) does not support it
> for all architectures.
>
> Checked on i686-linux-gnu.
> ---
>  configure    | 4 ++--
>  configure.ac | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/configure b/configure
> index 2a28c58cd6..c66082232e 100755
> --- a/configure
> +++ b/configure
> @@ -6324,8 +6324,8 @@ void foo (void)
>    i = 10;
>  }
>  EOF
> -if { ac_try='${CC-cc} $CFLAGS $CPPFLAGS -S -fPIC -mtls-dialect=gnu2
> -                  conftest.c 1>&5'
> +if { ac_try='${CC-cc} $CFLAGS $CPPFLAGS -fPIC -mtls-dialect=gnu2 -nostdlib -nostartfiles
> +                  conftest.c -o conftest 1>&5'
>    { { eval echo "\"\$as_me\":${as_lineno-$LINENO}: \"$ac_try\""; } >&5
>    (eval $ac_try) 2>&5
>    ac_status=$?
> diff --git a/configure.ac b/configure.ac
> index 00f49f09f7..7eb4239359 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -1493,8 +1493,8 @@ void foo (void)
>    i = 10;
>  }
>  EOF
> -if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -S -fPIC -mtls-dialect=gnu2
> -                  conftest.c 1>&AS_MESSAGE_LOG_FD])
> +if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -fPIC -mtls-dialect=gnu2 -nostdlib -nostartfiles
> +                  conftest.c -o conftest 1>&AS_MESSAGE_LOG_FD])
>  then
>    libc_cv_mtls_dialect_gnu2=yes
>  else
> --
> 2.32.0
>

Looks good to me.

I'll add R_386_TLS_DESC support to LLD with https://reviews.llvm.org/D112582

Re: [PATCH 2/3] Fix LIBC_PROG_BINUTILS for -fuse-ld=lld

[Adhemerval Zanella]

On 26/10/2021 17:48, Fangrui Song wrote:
> 
> On 2021-10-26, Adhemerval Zanella wrote:
>> GCC does not print the correct linker when -fuse-ld=lld is used with
>> the -print-prog-name=ld:
>>
>>  $ gcc -v 2>&1 | tail -n 1
>>  gcc version 11.2.0 (Ubuntu 11.2.0-7ubuntu2)
>>  $ gcc
>>  ld
>>
>> This is different than for gold:
>>
>>  $ gcc -fuse-ld=gold -print-prog-name=ld
>>  ld.gold
> 
> I think it may be debatable whether -fuse-ld={bfd,gold,lld} should
> affect -print-prog-name=ld output.
> 
>   # Should -fuse-ld= do anything with this?
>   % aarch64-linux-gnu-gcc -print-prog-name=../bin/ld
>   /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/../bin/ld

I think it would make sense, it allows to query the underlying static
linker used by the driver.  What does not make much sense to me is
allow arbitrary paths as input.

>  
> gcc/gcc.c has OPT_fuse_ld_bfd but not OPT_fuse_ld_lld, which may explain
> the behavior difference.

I think this is was oversight from lld gcc support and any case we need
to handle gcc that does not print the correct linker used if we want to
allow a way to set a non-default linker.

> 
> Clang does not have the GCC -fuse-ld= --print-prog-name=ld behavior.
> 
> 
> 
> 

This option is used solely to support --with-binutils.  Maybe we should
just remove it, since it is just simplify the use of a different binutils
and for clang it does not make sense (since it has different the integrated
assembly and different tool names).



>> Using ld.lld as the static linker name prints the expected result.
>>
>> This is only required when -fuse-ld=lld is used, if lld is used as
>> the 'ld' programs (through a symlink) LIBC_PROG_BINUTILS works
>> as expected.
>>
>> Checked on x86_64-linux-gnu.
>> ---
>> aclocal.m4 | 6 +++++-
>> configure  | 6 +++++-
>> 2 files changed, 10 insertions(+), 2 deletions(-)
>>
>> diff --git a/aclocal.m4 b/aclocal.m4
>> index c195c4db56..83da631822 100644
>> --- a/aclocal.m4
>> +++ b/aclocal.m4
>> @@ -114,8 +114,12 @@ if test -n "$path_binutils"; then
>>     path_binutils=`(cd $path_binutils; pwd) | sed 's%/*$%/%'`
>>     CC="$CC -B$path_binutils"
>> fi
>> +case "$CC" in
>> +    *fuse-ld=lld*) LDNAME=ld.lld;;
>> +    *)             LDNAME=ld;;
> 
> Perhaps hard code -fuse-ld=bfd and -fuse-ld=gold as well?
> 
> I think gcc -print-prog-name=ld.bfd is more robust than
> gcc -print-prog-name=ld -fuse-ld=bfd.

It outlines the missing gcc support, since with ld.bfd and ld.gold
does work with gcc.

I will maybe send a patch to remove LIBC_PROG_BINUTILS.

> 
>> +esac
>> AS=`$CC -print-prog-name=as`
>> -LD=`$CC -print-prog-name=ld`
>> +LD=`$CC -print-prog-name=$LDNAME`
>> AR=`$CC -print-prog-name=ar`
>> AC_SUBST(AR)
>> OBJDUMP=`$CC -print-prog-name=objdump`
>> diff --git a/configure b/configure
>> index 39d75eb4ed..2a28c58cd6 100755
>> --- a/configure
>> +++ b/configure
>> @@ -4549,8 +4549,12 @@ if test -n "$path_binutils"; then
>>     path_binutils=`(cd $path_binutils; pwd) | sed 's%/*$%/%'`
>>     CC="$CC -B$path_binutils"
>> fi
>> +case "$CC" in
>> +    *fuse-ld=lld*) LDNAME=ld.lld;;
>> +    *)             LDNAME=ld;;
>> +esac
>> AS=`$CC -print-prog-name=as`
>> -LD=`$CC -print-prog-name=ld`
>> +LD=`$CC -print-prog-name=$LDNAME`
>> AR=`$CC -print-prog-name=ar`
>>
>> OBJDUMP=`$CC -print-prog-name=objdump`
>> -- 
>> 2.32.0
>>

Re: [PATCH 0/3] Improve lld support and current status

[Adhemerval Zanella]

On 26/10/2021 17:33, Fangrui Song wrote:
>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>> arm the loader fails to build:
>>
>>  ld.so fails with
>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>  _dl_argv; recompile with -fPIC
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>  >>> referenced by rtld.c:164
>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
> 
> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
> Such a relocation referencing a preemptible symbol cannot be used.
> 
> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
> 
> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.

I think it is something else, the symbols is defined as hidden (output of
preprocessor):

  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));

However, hidden is set only the internal symbol:

  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv

And on rtld.c assembly we have the directive to use the internal definition:

    .set    _dl_argv,__GI__dl_argv

So why lld is not binding the usage to internal to the local hidden alias?

> 
>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>> an error instead of silent ignoring it:
>>
>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>  ld.lld: error: unknown argument '-relax'
> 
> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
> (see https://lld.llvm.org/ "production quality").
> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).

Right, but lld is still showing a different ld.bfd semantic that might trigger
other issues where on platforms where neither --relax nor -no-relax it supported
ld.bfd accepts both argument but ignore it [1].

At least on sparc, gcc does pass --relax and expects that static linker just
ignore if it is no supported.

[1] https://sourceware.org/binutils/docs-2.37/ld.html


> 
>> And even when -mno-relax is explicit add, lld does not support some relocations
>> generated by GCC:
>>
>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>  | head -n2
>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>  0000000000000058 _dl_skip_args + 0
>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>  0000000000000058 _dl_skip_args + 0
>>
>> The mips/mipsel also fails to build the loader:
>>
>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>
>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
> 
> lld.llvm.org says "MIPS seems decent too." and I personally know really
> little about MIPS.
> 
> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
> Relocation processing is rigid and consistent in LLD.
> GNU ld may miss some error checking for some relocation types.
> 
>> Similar to mips64/mips64el:
>>
>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>  against non-function symbol . This is invalid and most likely a compiler
>>  bug.
>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
> 
> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
> reporting a genuine issue.

I give you that loader code is tricky, but it does not really explain why
lld is failing to link the eh_frame.  It *might* due some internal assembly
routines, but at least from logs it does not seem so.

> 
>> The riscv32/riscv64 fails to the loader, but it is due missing support
>> to relaxation:
>>
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-relax
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-relax
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-rela
> 
> LLD does not implement RISC-V linker relaxation.
> R_RISCV_ALIGN has weird linker unfriendly semantics:
> simply ignoring the relocation type can break semantics
> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
> 
>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
> 
> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...

In fact it is, the problem was the autogenerated syscalls built from
the assembly.  I could avoid the R_RISCV_ALIGN with:

diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
index 20a9968106..28dbd89cb0 100644
--- a/sysdeps/riscv/Makefile
+++ b/sysdeps/riscv/Makefile
@@ -5,3 +5,9 @@ endif
 # RISC-V's assembler also needs to know about PIC as it changes the definition
 # of some assembler macros.
 ASFLAGS-.os += $(pic-ccflag)
+
+# lld does not implement R_RISCV_ALIGN relaxation optimization.
+ifeq (yes,$(with-lld))
+ASFLAGS-.os += -Wa,-mno-relax
+sysdep-CFLAGS += -mno-relax
+endif


However it does fail later for the libc.so with:

ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)

ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)
collect2: error: ld returned 1 exit status

Re: [PATCH 0/3] Improve lld support and current status

[Tulio Magno Quites Machado Filho]

Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:

> On 2021-10-26, Adhemerval Zanella wrote:
>>The powerpc64le fails at libc.so start:
>>
>>  (gdb) bt
>>  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
>>    from
>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
>>     from
>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6

This is a local call from libc.so to libc.so.
It should be using the local entry point for __vmx__sigjmp_save.

>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>     from
>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>  #3  0x0000000000000000 in ?? ()
>>  (gdb) disas
>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280

This is a pla, but this GDB isn't able to disassemble it.  This instruction
shouldn't be used unless when configuring glibc using --with-cpu=power10.

>>And powerpc64-linux-gnu fails on dynamic loaders start:
>>
>>  (gdb) bt
>>  #0  0x00007ffff7ffa9a8 in _start ()
>>  (gdb) disas
>>  Dump of assembler code for function _start:
>>  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
>>     0x00007ffff7ffa9ac <+4>:     .long 0x0
>>     0x00007ffff7ffa9b0 <+8>:     .long 0x0
>>     0x00007ffff7ffa9b4 <+12>:    .long 0x0

I've been told that LLD doesn't support the powerpc ELFv1 yet.

> CCed the powerpc maintainer.

Thanks!

-- 
Tulio Magno

Re: [PATCH 0/3] Improve lld support and current status

[Tulio Magno Quites Machado Filho]

Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:

> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>
>> On 2021-10-26, Adhemerval Zanella wrote:
>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>     from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>  #3  0x0000000000000000 in ?? ()
>>>  (gdb) disas
>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>
> This is a pla, but this GDB isn't able to disassemble it.  This instruction
> shouldn't be used unless when configuring glibc using --with-cpu=power10.

I can reproduce this issue even when configuring glibc with
--with-cpu=power9 --disable-multi-arch, which means the build should not have
any Power10 instructions.

-- 
Tulio Magno

Re: [PATCH 0/3] Improve lld support and current status

[Fangrui Song]

On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>
>> On 2021-10-26, Adhemerval Zanella wrote:
>>>The powerpc64le fails at libc.so start:
>>>
>>>  (gdb) bt
>>>  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
>>>    from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
>>>     from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>
>This is a local call from libc.so to libc.so.
>It should be using the local entry point for __vmx__sigjmp_save.
>
>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>     from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>  #3  0x0000000000000000 in ?? ()
>>>  (gdb) disas
>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>
>This is a pla, but this GDB isn't able to disassemble it.  This instruction
>shouldn't be used unless when configuring glibc using --with-cpu=power10.
>
>>>And powerpc64-linux-gnu fails on dynamic loaders start:
>>>
>>>  (gdb) bt
>>>  #0  0x00007ffff7ffa9a8 in _start ()
>>>  (gdb) disas
>>>  Dump of assembler code for function _start:
>>>  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
>>>     0x00007ffff7ffa9ac <+4>:     .long 0x0
>>>     0x00007ffff7ffa9b0 <+8>:     .long 0x0
>>>     0x00007ffff7ffa9b4 <+12>:    .long 0x0
>
>I've been told that LLD doesn't support the powerpc ELFv1 yet.

You are right:) The incomplete PowerPC64 ELF v1 support was removed by
https://reviews.llvm.org/D46316 (2018-05).  Is there remaining ELF v1
usage for powerpc64le in glibc?

(I only know some 64-bit ELF v2 and some 32-bit, but know really little about 64-bit ELF v1.)

Legacy ELF v1 is also the issue that powerpc64 big-endian Linux kernel
cannot be linked with LLD (https://github.com/ClangBuiltLinux/linux/issues/602).

There is little benefit retroactively adding ELF v1 support to LLD, so
migration to ELF v2 will be nice :)

>> CCed the powerpc maintainer.
>
>Thanks!
>
>-- 
>Tulio Magno

Re: [PATCH 0/3] Improve lld support and current status

[Fangrui Song]

On 2021-10-27, Adhemerval Zanella wrote:
>
>
>On 26/10/2021 17:33, Fangrui Song wrote:
>>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>>> arm the loader fails to build:
>>>
>>>  ld.so fails with
>>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>>  _dl_argv; recompile with -fPIC
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>>  >>> referenced by rtld.c:164
>>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
>>
>> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
>> Such a relocation referencing a preemptible symbol cannot be used.
>>
>> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
>>
>> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.
>
>I think it is something else, the symbols is defined as hidden (output of
>preprocessor):
>
>  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));
>
>However, hidden is set only the internal symbol:
>
>  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
>    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
>    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv
>
>And on rtld.c assembly we have the directive to use the internal definition:
>
>    .set    _dl_argv,__GI__dl_argv
>
>So why lld is not binding the usage to internal to the local hidden alias?

I use the Debian package g++-arm-linux-gnueabihf (nowadays glibc build needs a C++ compiler even if I don't build tests...).

mkdir -p out/arm; cd out/arm
../../configure --prefix=/tmp/glibc/arm --host=arm-linux-gnueabihf
make -j 50

Here is GNU ld produced elf/librtld.os:

% readelf -Ws elf/librtld.os | grep _dl_argv
   1582: 00000000     4 OBJECT  GLOBAL HIDDEN    13 __GI__dl_argv
   1603: 00000000     4 OBJECT  GLOBAL DEFAULT   13 _dl_argv

So LLD's librtld.os matches GNU ld for the two symbols.



The .set directive sets value/type but not binding/visibility.
Perhaps sysdeps/arm/dl-machine.h should use

.word   __GI__dl_argv(GOTOFF)

instead of 

.word   _dl_argv(GOTOFF)

>>
>>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>>> an error instead of silent ignoring it:
>>>
>>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>>  ld.lld: error: unknown argument '-relax'
>>
>> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
>> (see https://lld.llvm.org/ "production quality").
>> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).
>
>Right, but lld is still showing a different ld.bfd semantic that might trigger
>other issues where on platforms where neither --relax nor -no-relax it supported
>ld.bfd accepts both argument but ignore it [1].
>
>At least on sparc, gcc does pass --relax and expects that static linker just
>ignore if it is no supported.
>
>[1] https://sourceware.org/binutils/docs-2.37/ld.html

LLD's sparcv9 port is quite immature, so I cannot even advertise it.
A large portion should be rewritten but the sparc port is only cared by very few
retrocomputing fans now..

>>
>>> And even when -mno-relax is explicit add, lld does not support some relocations
>>> generated by GCC:
>>>
>>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>>  | head -n2
>>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>>  0000000000000058 _dl_skip_args + 0
>>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>>  0000000000000058 _dl_skip_args + 0
>>>
>>> The mips/mipsel also fails to build the loader:
>>>
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>>
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
>>
>> lld.llvm.org says "MIPS seems decent too." and I personally know really
>> little about MIPS.
>>
>> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
>> Relocation processing is rigid and consistent in LLD.
>> GNU ld may miss some error checking for some relocation types.
>>
>>> Similar to mips64/mips64el:
>>>
>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>  against non-function symbol . This is invalid and most likely a compiler
>>>  bug.
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>
>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>> reporting a genuine issue.
>
>I give you that loader code is tricky, but it does not really explain why
>lld is failing to link the eh_frame.  It *might* due some internal assembly
>routines, but at least from logs it does not seem so.

Hope Joseph can answer the question.

If we have easy-to-follow instructions building glibc mips with LLD, I can
forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
contributor of LLD's MIPS port)...

But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.

>>
>>> The riscv32/riscv64 fails to the loader, but it is due missing support
>>> to relaxation:
>>>
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-relax
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-relax
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-rela
>>
>> LLD does not implement RISC-V linker relaxation.
>> R_RISCV_ALIGN has weird linker unfriendly semantics:
>> simply ignoring the relocation type can break semantics
>> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
>>
>>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
>>
>> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...
>
>In fact it is, the problem was the autogenerated syscalls built from
>the assembly.  I could avoid the R_RISCV_ALIGN with:
>
>diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
>index 20a9968106..28dbd89cb0 100644
>--- a/sysdeps/riscv/Makefile
>+++ b/sysdeps/riscv/Makefile
>@@ -5,3 +5,9 @@ endif
> # RISC-V's assembler also needs to know about PIC as it changes the definition
> # of some assembler macros.
> ASFLAGS-.os += $(pic-ccflag)
>+
>+# lld does not implement R_RISCV_ALIGN relaxation optimization.
>+ifeq (yes,$(with-lld))
>+ASFLAGS-.os += -Wa,-mno-relax
>+sysdep-CFLAGS += -mno-relax
>+endif
>
>
>However it does fail later for the libc.so with:
>
>ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
>ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)
>
>ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)

There is a GNU ld issue: https://sourceware.org/bugzilla/show_bug.cgi?id=28509
It incorrectly allows non-PLT non-GOT relocation referencing a preemptible symbol.


LLD uses consistent categories (e.g. R_PC / R_PLT_PC) to describe relocation types.
It has consistent diagnostic across all ports it supports.
GNU ld uses dispatches for each relocation type and may miss some diagnostics.

>collect2: error: ld returned 1 exit status

Re: [PATCH 0/3] Improve lld support and current status

[Fangrui Song]

On 2021-10-27, Fangrui Song wrote:
>
>On 2021-10-27, Adhemerval Zanella wrote:
>>>>Similar to mips64/mips64el:
>>>>
>>>> ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>> _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>> against non-function symbol . This is invalid and most likely a compiler
>>>> bug.
>>>> ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>> symbol in readonly segment; recompile object files with -fPIC or pass
>>>> '-Wl,-z,notext' to allow text relocations in the output
>>>> >>> defined in
>>>> >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>> >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>> >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>>
>>>I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>>>reporting a genuine issue.
>>
>>I give you that loader code is tricky, but it does not really explain why
>>lld is failing to link the eh_frame.  It *might* due some internal assembly
>>routines, but at least from logs it does not seem so.
>
>Hope Joseph can answer the question.
>
>If we have easy-to-follow instructions building glibc mips with LLD, I can
>forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
>contributor of LLD's MIPS port)...
>
>But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.

jrtc27 told me that BFD mips has a hack to rewrite R_MIPS_64 in
.eh_frame to avoid such a R_MIPS_64 diagnostic. LLD doesn't implement it.

Clang's mips port emits relative relocations in .eh_frame (DW_EH_PE_pcrel | dwarf::DW_EH_PE_sdata4)
to avoid the linker issue: https://reviews.llvm.org/D72228

Lack of 64-bit PC-relative relocation is a longstanding problem for mips.
LLVM integrated assembler and linker actually support `.quad foo - .`
via composed relocations, but it is still unsupported in GNU as
(feature request: https://sourceware.org/bugzilla/show_bug.cgi?id=26222)

I know really little about MIPS but I'd wish that GNU as implements
.quad foo-.  and GCC migrates to PC-relative relocations like Clang.

Re: [PATCH 0/3] Improve lld support and current status

[Adhemerval Zanella]

On 27/10/2021 22:06, Fangrui Song wrote:
> 
> On 2021-10-27, Adhemerval Zanella wrote:
>>
>>
>> On 26/10/2021 17:33, Fangrui Song wrote:
>>>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>>>> arm the loader fails to build:
>>>>
>>>>  ld.so fails with
>>>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>>>  _dl_argv; recompile with -fPIC
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>>>  >>> referenced by rtld.c:164
>>>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
>>>
>>> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
>>> Such a relocation referencing a preemptible symbol cannot be used.
>>>
>>> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
>>>
>>> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.
>>
>> I think it is something else, the symbols is defined as hidden (output of
>> preprocessor):
>>
>>  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));
>>
>> However, hidden is set only the internal symbol:
>>
>>  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
>>    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
>>    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv
>>
>> And on rtld.c assembly we have the directive to use the internal definition:
>>
>>    .set    _dl_argv,__GI__dl_argv
>>
>> So why lld is not binding the usage to internal to the local hidden alias?
> 
> I use the Debian package g++-arm-linux-gnueabihf (nowadays glibc build needs a C++ compiler even if I don't build tests...).

You can set an empty CXX to avoid it ("CXX=") at configure time.

> 
> mkdir -p out/arm; cd out/arm
> ../../configure --prefix=/tmp/glibc/arm --host=arm-linux-gnueabihf
> make -j 50
> 
> Here is GNU ld produced elf/librtld.os:
> 
> % readelf -Ws elf/librtld.os | grep _dl_argv
>   1582: 00000000     4 OBJECT  GLOBAL HIDDEN    13 __GI__dl_argv
>   1603: 00000000     4 OBJECT  GLOBAL DEFAULT   13 _dl_argv
> 
> So LLD's librtld.os matches GNU ld for the two symbols.
> 
> 
> 
> The .set directive sets value/type but not binding/visibility.
> Perhaps sysdeps/arm/dl-machine.h should use
> 
> .word   __GI__dl_argv(GOTOFF)
> 
> instead of
> .word   _dl_argv(GOTOFF)
> 

It think it should be ok. So it seems that ld.bfd is being forgiving here
and binding to the internal alias. 

>>>
>>>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>>>> an error instead of silent ignoring it:
>>>>
>>>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>>>  ld.lld: error: unknown argument '-relax'
>>>
>>> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
>>> (see https://lld.llvm.org/ "production quality").
>>> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).
>>
>> Right, but lld is still showing a different ld.bfd semantic that might trigger
>> other issues where on platforms where neither --relax nor -no-relax it supported
>> ld.bfd accepts both argument but ignore it [1].
>>
>> At least on sparc, gcc does pass --relax and expects that static linker just
>> ignore if it is no supported.
>>
>> [1] https://sourceware.org/binutils/docs-2.37/ld.html
> 
> LLD's sparcv9 port is quite immature, so I cannot even advertise it.
> A large portion should be rewritten but the sparc port is only cared by very few
> retrocomputing fans now..

It would be good if we could detect it at configure time and just warn that linker
is not sufficient to build glibc.

> 
>>>
>>>> And even when -mno-relax is explicit add, lld does not support some relocations
>>>> generated by GCC:
>>>>
>>>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>>>  | head -n2
>>>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>>>  0000000000000058 _dl_skip_args + 0
>>>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>>>  0000000000000058 _dl_skip_args + 0
>>>>
>>>> The mips/mipsel also fails to build the loader:
>>>>
>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>>>
>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
>>>
>>> lld.llvm.org says "MIPS seems decent too." and I personally know really
>>> little about MIPS.
>>>
>>> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
>>> Relocation processing is rigid and consistent in LLD.
>>> GNU ld may miss some error checking for some relocation types.
>>>
>>>> Similar to mips64/mips64el:
>>>>
>>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>>  against non-function symbol . This is invalid and most likely a compiler
>>>>  bug.
>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>>
>>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>>> reporting a genuine issue.
>>
>> I give you that loader code is tricky, but it does not really explain why
>> lld is failing to link the eh_frame.  It *might* due some internal assembly
>> routines, but at least from logs it does not seem so.
> 
> Hope Joseph can answer the question.
> 
> If we have easy-to-follow instructions building glibc mips with LLD, I can
> forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
> contributor of LLD's MIPS port)...
> 
> But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.
> 
>>>
>>>> The riscv32/riscv64 fails to the loader, but it is due missing support
>>>> to relaxation:
>>>>
>>>>  ld.lld: error:
>>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>>  recompile with -mno-relax
>>>>  ld.lld: error:
>>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>>  recompile with -mno-relax
>>>>  ld.lld: error:
>>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>>  recompile with -mno-rela
>>>
>>> LLD does not implement RISC-V linker relaxation.
>>> R_RISCV_ALIGN has weird linker unfriendly semantics:
>>> simply ignoring the relocation type can break semantics
>>> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
>>>
>>>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
>>>
>>> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...
>>
>> In fact it is, the problem was the autogenerated syscalls built from
>> the assembly.  I could avoid the R_RISCV_ALIGN with:
>>
>> diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
>> index 20a9968106..28dbd89cb0 100644
>> --- a/sysdeps/riscv/Makefile
>> +++ b/sysdeps/riscv/Makefile
>> @@ -5,3 +5,9 @@ endif
>> # RISC-V's assembler also needs to know about PIC as it changes the definition
>> # of some assembler macros.
>> ASFLAGS-.os += $(pic-ccflag)
>> +
>> +# lld does not implement R_RISCV_ALIGN relaxation optimization.
>> +ifeq (yes,$(with-lld))
>> +ASFLAGS-.os += -Wa,-mno-relax
>> +sysdep-CFLAGS += -mno-relax
>> +endif
>>
>>
>> However it does fail later for the libc.so with:
>>
>> ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
>> ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>> ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>> ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)
>>
>> ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)
> 
> There is a GNU ld issue: https://sourceware.org/bugzilla/show_bug.cgi?id=28509
> It incorrectly allows non-PLT non-GOT relocation referencing a preemptible symbol.
> 
> 
> LLD uses consistent categories (e.g. R_PC / R_PLT_PC) to describe relocation types.
> It has consistent diagnostic across all ports it supports.
> GNU ld uses dispatches for each relocation type and may miss some diagnostics.
> 
>> collect2: error: ld returned 1 exit status

Indeed these are the same ARM issues, we will need to use internal symbols.

Re: [PATCH 0/3] Improve lld support and current status

[Tulio Magno Quites Machado Filho]

Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:

> You are right:) The incomplete PowerPC64 ELF v1 support was removed by
> https://reviews.llvm.org/D46316 (2018-05).  Is there remaining ELF v1
> usage for powerpc64le in glibc?

No. We reuse code whenever possible, but that's all.
However, without this support we can't build glibc for powerpc64 big-endian
(aka. powerpc64-linux).

-- 
Tulio Magno

Re: [PATCH 0/3] Improve lld support and current status

[Adhemerval Zanella]

On 27/10/2021 22:18, Fangrui Song wrote:
> On 2021-10-27, Fangrui Song wrote:
>>
>> On 2021-10-27, Adhemerval Zanella wrote:
>>>>> Similar to mips64/mips64el:
>>>>>
>>>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>>>  against non-function symbol . This is invalid and most likely a compiler
>>>>>  bug.
>>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>>  >>> defined in
>>>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>>>
>>>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>>>> reporting a genuine issue.
>>>
>>> I give you that loader code is tricky, but it does not really explain why
>>> lld is failing to link the eh_frame.  It *might* due some internal assembly
>>> routines, but at least from logs it does not seem so.
>>
>> Hope Joseph can answer the question.
>>
>> If we have easy-to-follow instructions building glibc mips with LLD, I can
>> forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
>> contributor of LLD's MIPS port)...
>>
>> But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.
> 
> jrtc27 told me that BFD mips has a hack to rewrite R_MIPS_64 in
> .eh_frame to avoid such a R_MIPS_64 diagnostic. LLD doesn't implement it.
> 
> Clang's mips port emits relative relocations in .eh_frame (DW_EH_PE_pcrel | dwarf::DW_EH_PE_sdata4)
> to avoid the linker issue: https://reviews.llvm.org/D72228
> 
> Lack of 64-bit PC-relative relocation is a longstanding problem for mips.
> LLVM integrated assembler and linker actually support `.quad foo - .`
> via composed relocations, but it is still unsupported in GNU as
> (feature request: https://sourceware.org/bugzilla/show_bug.cgi?id=26222)
> 
> I know really little about MIPS but I'd wish that GNU as implements
> .quad foo-.  and GCC migrates to PC-relative relocations like Clang.

In any case, it would be good to a configure check to warn uses that
R_MIPS_64 on .eh_frame might no be supported and bail out if lld
idea is to not implement any workaround.

Re: [PATCH 3/3] Check if linker also support -mtls-dialect=gnu2

[Fāng-ruì Sòng]

On Tue, Oct 26, 2021 at 7:04 PM Fāng-ruì Sòng <maskray@google.com> wrote:
>
> On Tue, Oct 26, 2021 at 1:03 PM Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
> >
> > Since some linkers (for instance lld for i386) does not support it
> > for all architectures.
> >
> > Checked on i686-linux-gnu.
> > ---
> >  configure    | 4 ++--
> >  configure.ac | 4 ++--
> >  2 files changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/configure b/configure
> > index 2a28c58cd6..c66082232e 100755
> > --- a/configure
> > +++ b/configure
> > @@ -6324,8 +6324,8 @@ void foo (void)
> >    i = 10;
> >  }
> >  EOF
> > -if { ac_try='${CC-cc} $CFLAGS $CPPFLAGS -S -fPIC -mtls-dialect=gnu2
> > -                  conftest.c 1>&5'
> > +if { ac_try='${CC-cc} $CFLAGS $CPPFLAGS -fPIC -mtls-dialect=gnu2 -nostdlib -nostartfiles
> > +                  conftest.c -o conftest 1>&5'
> >    { { eval echo "\"\$as_me\":${as_lineno-$LINENO}: \"$ac_try\""; } >&5
> >    (eval $ac_try) 2>&5
> >    ac_status=$?
> > diff --git a/configure.ac b/configure.ac
> > index 00f49f09f7..7eb4239359 100644
> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -1493,8 +1493,8 @@ void foo (void)
> >    i = 10;
> >  }
> >  EOF
> > -if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -S -fPIC -mtls-dialect=gnu2
> > -                  conftest.c 1>&AS_MESSAGE_LOG_FD])
> > +if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -fPIC -mtls-dialect=gnu2 -nostdlib -nostartfiles
> > +                  conftest.c -o conftest 1>&AS_MESSAGE_LOG_FD])
> >  then
> >    libc_cv_mtls_dialect_gnu2=yes
> >  else
> > --
> > 2.32.0
> >
>
> Looks good to me.
>
> I'll add R_386_TLS_DESC support to LLD with https://reviews.llvm.org/D112582

Reviewed-by: Fangrui Song <maskray@google.com>

I pushed https://reviews.llvm.org/D112582 as a maintainer of lld/ELF.

You can say that LLD before 14 did not support x86-32 TLSDESC.

Re: [PATCH 1/3] elf: Disable ifuncmain{1,5,5pic,5pie} when using LLD

[Fangrui Song]

On 2021-10-26, Adhemerval Zanella wrote:
>These tests takes the address of a protected symbol (foo_protected)
>and lld does not support copy relocations on protected data symbols.
>
>Checked on x86_64-linux-gnu.

You may use some information from my previous reply
https://sourceware.org/pipermail/libc-alpha/2021-October/132292.html
to make a better commit message.

ifuncmain1.c and ifuncmain5.c take the address of the function `foo_protected`.
If compiled with -fno-pic, the produced relocation is R_X86_64_PC32.
Nowadays most GCC releases are configured with enable-default-pie and
therefore ifuncmain5.c is compiled with the implicit -fpie.
So I did not catch this issue.


Reviewed-by: Fangrui Song <maskray@google.com>

lld status with powerpc64

[Fangrui Song]

On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:
>
>> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>>
>>> On 2021-10-26, Adhemerval Zanella wrote:
>>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>>     from
>>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>>  #3  0x0000000000000000 in ?? ()
>>>>  (gdb) disas
>>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>>
>> This is a pla, but this GDB isn't able to disassemble it.  This instruction
>> shouldn't be used unless when configuring glibc using --with-cpu=power10.
>
>I can reproduce this issue even when configuring glibc with
>--with-cpu=power9 --disable-multi-arch, which means the build should not have
>any Power10 instructions.
>
>-- 
>Tulio Magno

__gep_setup___vmx__sigjmp_save means a r12 setup stub in ld.lld and is
used with R_PPC64_REL24_NOTOC for a non-PLT branch.

Clang only emits `bl foo@notoc` with -mcpu=power10.
However, sysdeps/powerpc/powerpc64/configure.ac enables USE_PPC64_NOTOC
when the assembler (gas) supports it.


Adhemerval noticed that ld.lld has a behavior difference from GNU ld:
ld.lld defaults to PC-relative paddi stub while GNU ld doesn't (like
--power10-stubs=no).
Is USE_PPC64_NOTOC supposed to be used when targeting POWER9 and below?
If yes, the ld.lld default should be fixed.


In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
USE_PPC64_NOTOC is undefined.

% gcc --version
gcc (Debian 8.3.0-6) 8.3.0
...
% ld.bfd --version
GNU ld (GNU Binutils for Debian) 2.31.1
...

I only see 8 more FAILs with ld.lld than ld.bfd


% diff -u 0 1
--- 0   2021-11-05 00:11:43.218731302 -0700
+++ 1   2021-11-05 00:11:37.659286448 -0700
@@ -9,6 +9,14 @@
  FAIL: debug/tst-lfschk6
  FAIL: dlfcn/bug-atexit3
  FAIL: elf/check-abi-libc
+FAIL: elf/ifuncmain1pic
+FAIL: elf/ifuncmain1pie
+FAIL: elf/ifuncmain1vis
+FAIL: elf/ifuncmain1vispic
+FAIL: elf/ifuncmain1vispie
+FAIL: elf/ifuncmain3
+FAIL: elf/ifuncmain6pie
+FAIL: elf/tst-tlsopt-powerpc
  FAIL: nptl/tst-cancel24
  FAIL: nptl/tst-minstack-throw
  FAIL: nptl/tst-once5


I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
(something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
But perhaps Adhemerval can look a look at it.


For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
__tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
Let me send a configure patch to disable it...



Hey, so lld linked glibc powerpc64 is in a pretty good status!

Re: lld status with powerpc64

[Fangrui Song]

On 2021-11-05, Fangrui Song wrote:
>
>On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>>Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:
>>
>>>Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>
>>>>On 2021-10-26, Adhemerval Zanella wrote:
>>>>> #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>>>    from
>>>>> /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>>> #3  0x0000000000000000 in ?? ()
>>>>> (gdb) disas
>>>>> Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>>> => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>>>    0x00007ffff7f2a984 <+4>:     li      r12,-1280
>>>
>>>This is a pla, but this GDB isn't able to disassemble it.  This instruction
>>>shouldn't be used unless when configuring glibc using --with-cpu=power10.
>>
>>I can reproduce this issue even when configuring glibc with
>>--with-cpu=power9 --disable-multi-arch, which means the build should not have
>>any Power10 instructions.
>>
>>-- 
>>Tulio Magno
>
>__gep_setup___vmx__sigjmp_save means a r12 setup stub in ld.lld and is
>used with R_PPC64_REL24_NOTOC for a non-PLT branch.
>
>Clang only emits `bl foo@notoc` with -mcpu=power10.
>However, sysdeps/powerpc/powerpc64/configure.ac enables USE_PPC64_NOTOC
>when the assembler (gas) supports it.
>
>
>Adhemerval noticed that ld.lld has a behavior difference from GNU ld:
>ld.lld defaults to PC-relative paddi stub while GNU ld doesn't (like
>--power10-stubs=no).
>Is USE_PPC64_NOTOC supposed to be used when targeting POWER9 and below?
>If yes, the ld.lld default should be fixed.
>
>
>In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
>USE_PPC64_NOTOC is undefined.
>
>% gcc --version
>gcc (Debian 8.3.0-6) 8.3.0
>...
>% ld.bfd --version
>GNU ld (GNU Binutils for Debian) 2.31.1
>...
>
>I only see 8 more FAILs with ld.lld than ld.bfd
>
>
>% diff -u 0 1
>--- 0   2021-11-05 00:11:43.218731302 -0700
>+++ 1   2021-11-05 00:11:37.659286448 -0700
>@@ -9,6 +9,14 @@
> FAIL: debug/tst-lfschk6
> FAIL: dlfcn/bug-atexit3
> FAIL: elf/check-abi-libc
>+FAIL: elf/ifuncmain1pic
>+FAIL: elf/ifuncmain1pie
>+FAIL: elf/ifuncmain1vis
>+FAIL: elf/ifuncmain1vispic
>+FAIL: elf/ifuncmain1vispie
>+FAIL: elf/ifuncmain3
>+FAIL: elf/ifuncmain6pie
>+FAIL: elf/tst-tlsopt-powerpc
> FAIL: nptl/tst-cancel24
> FAIL: nptl/tst-minstack-throw
> FAIL: nptl/tst-once5
>
>
>I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
>regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
>(something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
>But perhaps Adhemerval can look a look at it.
>
>
>For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
>__tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
>Let me send a configure patch to disable it...

Actually I do not know how to disable tst-tlsopt-powerpc properly.

Perhaps add sysdeps/powerpc/configure.ac and move the
sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
to sysdeps/powerpc/configure.ac?
sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).

The patch requires some non-trivial configure.ac change, so I'd hope
that an expert can do it :)

>
>
>Hey, so lld linked glibc powerpc64 is in a pretty good status!

Re: lld status with powerpc64

[Adhemerval Zanella]

On 05/11/2021 04:23, Fangrui Song wrote:
> 
> On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>> Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:
>>
>>> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>
>>>> On 2021-10-26, Adhemerval Zanella wrote:
>>>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>>>     from
>>>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>>>  #3  0x0000000000000000 in ?? ()
>>>>>  (gdb) disas
>>>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>>>
>>> This is a pla, but this GDB isn't able to disassemble it.  This instruction
>>> shouldn't be used unless when configuring glibc using --with-cpu=power10.
>>
>> I can reproduce this issue even when configuring glibc with
>> --with-cpu=power9 --disable-multi-arch, which means the build should not have
>> any Power10 instructions.
>>
>> -- 
>> Tulio Magno
> 
> __gep_setup___vmx__sigjmp_save means a r12 setup stub in ld.lld and is
> used with R_PPC64_REL24_NOTOC for a non-PLT branch.
> 
> Clang only emits `bl foo@notoc` with -mcpu=power10.
> However, sysdeps/powerpc/powerpc64/configure.ac enables USE_PPC64_NOTOC
> when the assembler (gas) supports it.
> 
> 
> Adhemerval noticed that ld.lld has a behavior difference from GNU ld:
> ld.lld defaults to PC-relative paddi stub while GNU ld doesn't (like
> --power10-stubs=no).
> Is USE_PPC64_NOTOC supposed to be used when targeting POWER9 and below?
> If yes, the ld.lld default should be fixed.

I figure out the issue and both bfd, gold, and lld align themselves on the
power10-stubs options handling.  Using --power10-stubs without an arg is 
equivalent to --power10-stubs=yes, but not specifying --power10-stubs at
all should be equivalent to --power10-stubs=auto.  It somewhat confusing,
but I think it is to allow linker and compiler to be independently
regarding power10 stub generation.

The issue is bfd enables power10 relocs generation on stubs iff it sees
the new pc-relative relocations (for instance R_PPC64_D34), otherwise 
it generates default stubs (ppc64_elf_check_relocs:4700).

So we have two options here:

  1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
     aims to support such behavior).

  2. Define USE_PPC64_NOTOC iff linker supports such optimization.
     It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
     link a simple binary without any pcrel and check if the stub has 
     power10 instruction.

  3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
     and on the syscall definition to call the __syscall_error.

I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC 
manually on config.h when configuring with lld I can build glibc.

> 
> 
> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
> USE_PPC64_NOTOC is undefined.
> 
> % gcc --version
> gcc (Debian 8.3.0-6) 8.3.0
> ...
> % ld.bfd --version
> GNU ld (GNU Binutils for Debian) 2.31.1
> ...
> 
> I only see 8 more FAILs with ld.lld than ld.bfd

I am still having trouble to *finish* make check with lld release 13
my environment (some tests stuck on infinite loop).

> 
> 
> % diff -u 0 1
> --- 0   2021-11-05 00:11:43.218731302 -0700
> +++ 1   2021-11-05 00:11:37.659286448 -0700
> @@ -9,6 +9,14 @@
>  FAIL: debug/tst-lfschk6
>  FAIL: dlfcn/bug-atexit3
>  FAIL: elf/check-abi-libc
> +FAIL: elf/ifuncmain1pic
> +FAIL: elf/ifuncmain1pie
> +FAIL: elf/ifuncmain1vis
> +FAIL: elf/ifuncmain1vispic
> +FAIL: elf/ifuncmain1vispie
> +FAIL: elf/ifuncmain3
> +FAIL: elf/ifuncmain6pie
> +FAIL: elf/tst-tlsopt-powerpc
>  FAIL: nptl/tst-cancel24
>  FAIL: nptl/tst-minstack-throw
>  FAIL: nptl/tst-once5
> 
> 
> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
> But perhaps Adhemerval can look a look at it.

I still need to understand why armhf also fails with the ifunc using
protected symbols, although I am not if they are related to IRELATIVE
ordering.

> 
> 
> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
> Let me send a configure patch to disable it...
> 
> Actually I do not know how to disable tst-tlsopt-powerpc properly.
> 
> Perhaps add sysdeps/powerpc/configure.ac and move the
> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
> to sysdeps/powerpc/configure.ac?
> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
> 
> The patch requires some non-trivial configure.ac change, so I'd hope
> that an expert can do it 

Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
by the linker?


> 
> 
> Hey, so lld linked glibc powerpc64 is in a pretty good status!

I am still not sure about it, I did could run some tests but I am still
struggling to get a make check to finish.

Re: lld status with powerpc64

[Adhemerval Zanella]

On 05/11/2021 10:58, Adhemerval Zanella wrote:

> So we have two options here:
> 
>   1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
>      aims to support such behavior).
> 
>   2. Define USE_PPC64_NOTOC iff linker supports such optimization.
>      It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
>      link a simple binary without any pcrel and check if the stub has 
>      power10 instruction.
> 
>   3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
>      and on the syscall definition to call the __syscall_error.
> 
> I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC 
> manually on config.h when configuring with lld I can build glibc.
> 
>>
>>
>> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
>> USE_PPC64_NOTOC is undefined.
>>
>> % gcc --version
>> gcc (Debian 8.3.0-6) 8.3.0
>> ...
>> % ld.bfd --version
>> GNU ld (GNU Binutils for Debian) 2.31.1
>> ...
>>
>> I only see 8 more FAILs with ld.lld than ld.bfd
> 
> I am still having trouble to *finish* make check with lld release 13
> my environment (some tests stuck on infinite loop).
> 
>>
>>
>> % diff -u 0 1
>> --- 0   2021-11-05 00:11:43.218731302 -0700
>> +++ 1   2021-11-05 00:11:37.659286448 -0700
>> @@ -9,6 +9,14 @@
>>  FAIL: debug/tst-lfschk6
>>  FAIL: dlfcn/bug-atexit3
>>  FAIL: elf/check-abi-libc
>> +FAIL: elf/ifuncmain1pic
>> +FAIL: elf/ifuncmain1pie
>> +FAIL: elf/ifuncmain1vis
>> +FAIL: elf/ifuncmain1vispic
>> +FAIL: elf/ifuncmain1vispie
>> +FAIL: elf/ifuncmain3
>> +FAIL: elf/ifuncmain6pie
>> +FAIL: elf/tst-tlsopt-powerpc
>>  FAIL: nptl/tst-cancel24
>>  FAIL: nptl/tst-minstack-throw
>>  FAIL: nptl/tst-once5
>>
>>
>> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
>> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
>> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
>> But perhaps Adhemerval can look a look at it.
> 
> I still need to understand why armhf also fails with the ifunc using
> protected symbols, although I am not if they are related to IRELATIVE
> ordering.
> 
>>
>>
>> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
>> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
>> Let me send a configure patch to disable it...
>>
>> Actually I do not know how to disable tst-tlsopt-powerpc properly.
>>
>> Perhaps add sysdeps/powerpc/configure.ac and move the
>> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
>> to sysdeps/powerpc/configure.ac?
>> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
>>
>> The patch requires some non-trivial configure.ac change, so I'd hope
>> that an expert can do it 
> 
> Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
> by the linker?
> 
> 
>>
>>
>> Hey, so lld linked glibc powerpc64 is in a pretty good status!
> 
> I am still not sure about it, I did could run some tests but I am still
> struggling to get a make check to finish.
> 

So it turned out the another issues was my gcc toolchain did not have
initfini array set which lets to compiler generate .ctors instead of
.init_array and lld does not support .ctors (it just ignores instead of
converting to .init_array entries as bfd).

With NOTOC fix and a proper toolchain I am seeing the same regression
you are seeing:

FAIL: elf/ifuncmain1pic
FAIL: elf/ifuncmain1pie
FAIL: elf/ifuncmain1vis
FAIL: elf/ifuncmain1vispic
FAIL: elf/ifuncmain1vispie
FAIL: elf/ifuncmain3
FAIL: elf/ifuncmain6pie
FAIL: elf/tst-tlsopt-powerpc

I think it would be good use --enable-initfini-array on build-many-glibc.py
until we set gcc-12 as default.

Re: lld status with powerpc64

[H.J. Lu]

On Fri, Nov 5, 2021 at 12:33 PM Adhemerval Zanella via Libc-alpha
<libc-alpha@sourceware.org> wrote:
>
>
>
> On 05/11/2021 10:58, Adhemerval Zanella wrote:
>
> > So we have two options here:
> >
> >   1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
> >      aims to support such behavior).
> >
> >   2. Define USE_PPC64_NOTOC iff linker supports such optimization.
> >      It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
> >      link a simple binary without any pcrel and check if the stub has
> >      power10 instruction.
> >
> >   3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
> >      and on the syscall definition to call the __syscall_error.
> >
> > I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC
> > manually on config.h when configuring with lld I can build glibc.
> >
> >>
> >>
> >> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
> >> USE_PPC64_NOTOC is undefined.
> >>
> >> % gcc --version
> >> gcc (Debian 8.3.0-6) 8.3.0
> >> ...
> >> % ld.bfd --version
> >> GNU ld (GNU Binutils for Debian) 2.31.1
> >> ...
> >>
> >> I only see 8 more FAILs with ld.lld than ld.bfd
> >
> > I am still having trouble to *finish* make check with lld release 13
> > my environment (some tests stuck on infinite loop).
> >
> >>
> >>
> >> % diff -u 0 1
> >> --- 0   2021-11-05 00:11:43.218731302 -0700
> >> +++ 1   2021-11-05 00:11:37.659286448 -0700
> >> @@ -9,6 +9,14 @@
> >>  FAIL: debug/tst-lfschk6
> >>  FAIL: dlfcn/bug-atexit3
> >>  FAIL: elf/check-abi-libc
> >> +FAIL: elf/ifuncmain1pic
> >> +FAIL: elf/ifuncmain1pie
> >> +FAIL: elf/ifuncmain1vis
> >> +FAIL: elf/ifuncmain1vispic
> >> +FAIL: elf/ifuncmain1vispie
> >> +FAIL: elf/ifuncmain3
> >> +FAIL: elf/ifuncmain6pie
> >> +FAIL: elf/tst-tlsopt-powerpc
> >>  FAIL: nptl/tst-cancel24
> >>  FAIL: nptl/tst-minstack-throw
> >>  FAIL: nptl/tst-once5
> >>
> >>
> >> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
> >> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
> >> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
> >> But perhaps Adhemerval can look a look at it.
> >
> > I still need to understand why armhf also fails with the ifunc using
> > protected symbols, although I am not if they are related to IRELATIVE
> > ordering.
> >
> >>
> >>
> >> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
> >> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
> >> Let me send a configure patch to disable it...
> >>
> >> Actually I do not know how to disable tst-tlsopt-powerpc properly.
> >>
> >> Perhaps add sysdeps/powerpc/configure.ac and move the
> >> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
> >> to sysdeps/powerpc/configure.ac?
> >> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
> >>
> >> The patch requires some non-trivial configure.ac change, so I'd hope
> >> that an expert can do it
> >
> > Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
> > by the linker?
> >
> >
> >>
> >>
> >> Hey, so lld linked glibc powerpc64 is in a pretty good status!
> >
> > I am still not sure about it, I did could run some tests but I am still
> > struggling to get a make check to finish.
> >
>
> So it turned out the another issues was my gcc toolchain did not have
> initfini array set which lets to compiler generate .ctors instead of
> .init_array and lld does not support .ctors (it just ignores instead of
> converting to .init_array entries as bfd).
>
> With NOTOC fix and a proper toolchain I am seeing the same regression
> you are seeing:
>
> FAIL: elf/ifuncmain1pic
> FAIL: elf/ifuncmain1pie
> FAIL: elf/ifuncmain1vis
> FAIL: elf/ifuncmain1vispic
> FAIL: elf/ifuncmain1vispie
> FAIL: elf/ifuncmain3
> FAIL: elf/ifuncmain6pie
> FAIL: elf/tst-tlsopt-powerpc
>
> I think it would be good use --enable-initfini-array on build-many-glibc.py
> until we set gcc-12 as default.

https://sourceware.org/pipermail/libc-alpha/2021-June/127413.html

-- 
H.J.

Re: lld status with powerpc64

[H.J. Lu]

On Fri, Nov 5, 2021 at 12:38 PM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> On Fri, Nov 5, 2021 at 12:33 PM Adhemerval Zanella via Libc-alpha
> <libc-alpha@sourceware.org> wrote:
> >
> >
> >
> > On 05/11/2021 10:58, Adhemerval Zanella wrote:
> >
> > > So we have two options here:
> > >
> > >   1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
> > >      aims to support such behavior).
> > >
> > >   2. Define USE_PPC64_NOTOC iff linker supports such optimization.
> > >      It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
> > >      link a simple binary without any pcrel and check if the stub has
> > >      power10 instruction.
> > >
> > >   3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
> > >      and on the syscall definition to call the __syscall_error.
> > >
> > > I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC
> > > manually on config.h when configuring with lld I can build glibc.
> > >
> > >>
> > >>
> > >> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
> > >> USE_PPC64_NOTOC is undefined.
> > >>
> > >> % gcc --version
> > >> gcc (Debian 8.3.0-6) 8.3.0
> > >> ...
> > >> % ld.bfd --version
> > >> GNU ld (GNU Binutils for Debian) 2.31.1
> > >> ...
> > >>
> > >> I only see 8 more FAILs with ld.lld than ld.bfd
> > >
> > > I am still having trouble to *finish* make check with lld release 13
> > > my environment (some tests stuck on infinite loop).
> > >
> > >>
> > >>
> > >> % diff -u 0 1
> > >> --- 0   2021-11-05 00:11:43.218731302 -0700
> > >> +++ 1   2021-11-05 00:11:37.659286448 -0700
> > >> @@ -9,6 +9,14 @@
> > >>  FAIL: debug/tst-lfschk6
> > >>  FAIL: dlfcn/bug-atexit3
> > >>  FAIL: elf/check-abi-libc
> > >> +FAIL: elf/ifuncmain1pic
> > >> +FAIL: elf/ifuncmain1pie
> > >> +FAIL: elf/ifuncmain1vis
> > >> +FAIL: elf/ifuncmain1vispic
> > >> +FAIL: elf/ifuncmain1vispie
> > >> +FAIL: elf/ifuncmain3
> > >> +FAIL: elf/ifuncmain6pie
> > >> +FAIL: elf/tst-tlsopt-powerpc
> > >>  FAIL: nptl/tst-cancel24
> > >>  FAIL: nptl/tst-minstack-throw
> > >>  FAIL: nptl/tst-once5
> > >>
> > >>
> > >> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
> > >> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
> > >> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
> > >> But perhaps Adhemerval can look a look at it.
> > >
> > > I still need to understand why armhf also fails with the ifunc using
> > > protected symbols, although I am not if they are related to IRELATIVE
> > > ordering.
> > >
> > >>
> > >>
> > >> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
> > >> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
> > >> Let me send a configure patch to disable it...
> > >>
> > >> Actually I do not know how to disable tst-tlsopt-powerpc properly.
> > >>
> > >> Perhaps add sysdeps/powerpc/configure.ac and move the
> > >> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
> > >> to sysdeps/powerpc/configure.ac?
> > >> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
> > >>
> > >> The patch requires some non-trivial configure.ac change, so I'd hope
> > >> that an expert can do it
> > >
> > > Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
> > > by the linker?
> > >
> > >
> > >>
> > >>
> > >> Hey, so lld linked glibc powerpc64 is in a pretty good status!
> > >
> > > I am still not sure about it, I did could run some tests but I am still
> > > struggling to get a make check to finish.
> > >
> >
> > So it turned out the another issues was my gcc toolchain did not have
> > initfini array set which lets to compiler generate .ctors instead of
> > .init_array and lld does not support .ctors (it just ignores instead of
> > converting to .init_array entries as bfd).
> >
> > With NOTOC fix and a proper toolchain I am seeing the same regression
> > you are seeing:
> >
> > FAIL: elf/ifuncmain1pic
> > FAIL: elf/ifuncmain1pie
> > FAIL: elf/ifuncmain1vis
> > FAIL: elf/ifuncmain1vispic
> > FAIL: elf/ifuncmain1vispie
> > FAIL: elf/ifuncmain3
> > FAIL: elf/ifuncmain6pie
> > FAIL: elf/tst-tlsopt-powerpc
> >
> > I think it would be good use --enable-initfini-array on build-many-glibc.py
> > until we set gcc-12 as default.
>
> https://sourceware.org/pipermail/libc-alpha/2021-June/127413.html
>
> --
> H.J.

https://patchwork.sourceware.org/project/glibc/patch/20210609122916.3884385-1-hjl.tools@gmail.com/

-- 
H.J.

Re: lld status with powerpc64

[Fāng-ruì Sòng]

On Fri, Nov 5, 2021 at 12:39 PM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> On Fri, Nov 5, 2021 at 12:33 PM Adhemerval Zanella via Libc-alpha
> <libc-alpha@sourceware.org> wrote:
> >
> >
> >
> > On 05/11/2021 10:58, Adhemerval Zanella wrote:
> >
> > > So we have two options here:
> > >
> > >   1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
> > >      aims to support such behavior).
> > >
> > >   2. Define USE_PPC64_NOTOC iff linker supports such optimization.
> > >      It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
> > >      link a simple binary without any pcrel and check if the stub has
> > >      power10 instruction.
> > >
> > >   3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
> > >      and on the syscall definition to call the __syscall_error.
> > >
> > > I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC
> > > manually on config.h when configuring with lld I can build glibc.
> > >
> > >>
> > >>
> > >> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
> > >> USE_PPC64_NOTOC is undefined.
> > >>
> > >> % gcc --version
> > >> gcc (Debian 8.3.0-6) 8.3.0
> > >> ...
> > >> % ld.bfd --version
> > >> GNU ld (GNU Binutils for Debian) 2.31.1
> > >> ...
> > >>
> > >> I only see 8 more FAILs with ld.lld than ld.bfd
> > >
> > > I am still having trouble to *finish* make check with lld release 13
> > > my environment (some tests stuck on infinite loop).
> > >
> > >>
> > >>
> > >> % diff -u 0 1
> > >> --- 0   2021-11-05 00:11:43.218731302 -0700
> > >> +++ 1   2021-11-05 00:11:37.659286448 -0700
> > >> @@ -9,6 +9,14 @@
> > >>  FAIL: debug/tst-lfschk6
> > >>  FAIL: dlfcn/bug-atexit3
> > >>  FAIL: elf/check-abi-libc
> > >> +FAIL: elf/ifuncmain1pic
> > >> +FAIL: elf/ifuncmain1pie
> > >> +FAIL: elf/ifuncmain1vis
> > >> +FAIL: elf/ifuncmain1vispic
> > >> +FAIL: elf/ifuncmain1vispie
> > >> +FAIL: elf/ifuncmain3
> > >> +FAIL: elf/ifuncmain6pie
> > >> +FAIL: elf/tst-tlsopt-powerpc
> > >>  FAIL: nptl/tst-cancel24
> > >>  FAIL: nptl/tst-minstack-throw
> > >>  FAIL: nptl/tst-once5
> > >>
> > >>
> > >> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
> > >> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
> > >> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
> > >> But perhaps Adhemerval can look a look at it.
> > >
> > > I still need to understand why armhf also fails with the ifunc using
> > > protected symbols, although I am not if they are related to IRELATIVE
> > > ordering.
> > >
> > >>
> > >>
> > >> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
> > >> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
> > >> Let me send a configure patch to disable it...
> > >>
> > >> Actually I do not know how to disable tst-tlsopt-powerpc properly.
> > >>
> > >> Perhaps add sysdeps/powerpc/configure.ac and move the
> > >> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
> > >> to sysdeps/powerpc/configure.ac?
> > >> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
> > >>
> > >> The patch requires some non-trivial configure.ac change, so I'd hope
> > >> that an expert can do it
> > >
> > > Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
> > > by the linker?
> > >
> > >
> > >>
> > >>
> > >> Hey, so lld linked glibc powerpc64 is in a pretty good status!
> > >
> > > I am still not sure about it, I did could run some tests but I am still
> > > struggling to get a make check to finish.
> > >
> >
> > So it turned out the another issues was my gcc toolchain did not have
> > initfini array set which lets to compiler generate .ctors instead of
> > .init_array and lld does not support .ctors (it just ignores instead of
> > converting to .init_array entries as bfd).
> >
> > With NOTOC fix and a proper toolchain I am seeing the same regression
> > you are seeing:
> >
> > FAIL: elf/ifuncmain1pic
> > FAIL: elf/ifuncmain1pie
> > FAIL: elf/ifuncmain1vis
> > FAIL: elf/ifuncmain1vispic
> > FAIL: elf/ifuncmain1vispie
> > FAIL: elf/ifuncmain3
> > FAIL: elf/ifuncmain6pie
> > FAIL: elf/tst-tlsopt-powerpc
> >
> > I think it would be good use --enable-initfini-array on build-many-glibc.py
> > until we set gcc-12 as default.
>
> https://sourceware.org/pipermail/libc-alpha/2021-June/127413.html

I wasn't subscribed to libc-alpha, so it's not easy for me to reply
there. The patch looks good to me :)

Reviewed-by: Fangrui Song <maskray@google.com>

GCC side may use more clean-up
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100896#c4

Re: lld status with powerpc64

[Bill Schmidt]

Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.

Thanks!
Bill

On 11/5/21 8:58 AM, Adhemerval Zanella via Libc-alpha wrote:
>
> On 05/11/2021 04:23, Fangrui Song wrote:
>> On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>>> Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>
>>>> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>>>>
>>>>> On 2021-10-26, Adhemerval Zanella wrote:
>>>>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>>>>     from
>>>>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>>>>  #3  0x0000000000000000 in ?? ()
>>>>>>  (gdb) disas
>>>>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>>>> This is a pla, but this GDB isn't able to disassemble it.  This instruction
>>>> shouldn't be used unless when configuring glibc using --with-cpu=power10.
>>> I can reproduce this issue even when configuring glibc with
>>> --with-cpu=power9 --disable-multi-arch, which means the build should not have
>>> any Power10 instructions.
>>>
>>> -- 
>>> Tulio Magno
>> __gep_setup___vmx__sigjmp_save means a r12 setup stub in ld.lld and is
>> used with R_PPC64_REL24_NOTOC for a non-PLT branch.
>>
>> Clang only emits `bl foo@notoc` with -mcpu=power10.
>> However, sysdeps/powerpc/powerpc64/configure.ac enables USE_PPC64_NOTOC
>> when the assembler (gas) supports it.
>>
>>
>> Adhemerval noticed that ld.lld has a behavior difference from GNU ld:
>> ld.lld defaults to PC-relative paddi stub while GNU ld doesn't (like
>> --power10-stubs=no).
>> Is USE_PPC64_NOTOC supposed to be used when targeting POWER9 and below?
>> If yes, the ld.lld default should be fixed.
> I figure out the issue and both bfd, gold, and lld align themselves on the
> power10-stubs options handling.  Using --power10-stubs without an arg is 
> equivalent to --power10-stubs=yes, but not specifying --power10-stubs at
> all should be equivalent to --power10-stubs=auto.  It somewhat confusing,
> but I think it is to allow linker and compiler to be independently
> regarding power10 stub generation.
>
> The issue is bfd enables power10 relocs generation on stubs iff it sees
> the new pc-relative relocations (for instance R_PPC64_D34), otherwise 
> it generates default stubs (ppc64_elf_check_relocs:4700).
>
> So we have two options here:
>
>   1. Do not define USE_PPC64_NOTOC if with-lld (not optimal if ldd
>      aims to support such behavior).
>
>   2. Define USE_PPC64_NOTOC iff linker supports such optimization.
>      It means to emit a NOTOC relocation (R_PPC64_REL24_NOTOC),
>      link a simple binary without any pcrel and check if the stub has 
>      power10 instruction.
>
>   3. Remove the USE_PPC64_NOTOC usage.  It is used on setjmp routines
>      and on the syscall definition to call the __syscall_error.
>
> I am aiming to implement 2. since at least by disabling USE_PPC64_NOTOC 
> manually on config.h when configuring with lld I can build glibc.
>
>>
>> In my testing environment (POWER9), ld.bfd doesn't support @notoc, so
>> USE_PPC64_NOTOC is undefined.
>>
>> % gcc --version
>> gcc (Debian 8.3.0-6) 8.3.0
>> ...
>> % ld.bfd --version
>> GNU ld (GNU Binutils for Debian) 2.31.1
>> ...
>>
>> I only see 8 more FAILs with ld.lld than ld.bfd
> I am still having trouble to *finish* make check with lld release 13
> my environment (some tests stuck on infinite loop).
>
>>
>> % diff -u 0 1
>> --- 0   2021-11-05 00:11:43.218731302 -0700
>> +++ 1   2021-11-05 00:11:37.659286448 -0700
>> @@ -9,6 +9,14 @@
>>  FAIL: debug/tst-lfschk6
>>  FAIL: dlfcn/bug-atexit3
>>  FAIL: elf/check-abi-libc
>> +FAIL: elf/ifuncmain1pic
>> +FAIL: elf/ifuncmain1pie
>> +FAIL: elf/ifuncmain1vis
>> +FAIL: elf/ifuncmain1vispic
>> +FAIL: elf/ifuncmain1vispie
>> +FAIL: elf/ifuncmain3
>> +FAIL: elf/ifuncmain6pie
>> +FAIL: elf/tst-tlsopt-powerpc
>>  FAIL: nptl/tst-cancel24
>>  FAIL: nptl/tst-minstack-throw
>>  FAIL: nptl/tst-once5
>>
>>
>> I suspect ifuncmain1* is again related to the order of R_*_IRELATIVE with
>> regard to R_*_JUMP_SLOT referencing a STT_GNU_IFUNC symbol.
>> (something like https://maskray.me/blog/2021-01-18-gnu-indirect-function#relocation-resolving-order)
>> But perhaps Adhemerval can look a look at it.
> I still need to understand why armhf also fails with the ifunc using
> protected symbols, although I am not if they are related to IRELATIVE
> ordering.
>
>>
>> For elf/tst-tlsopt-powerpc, it is simply because ld.lld doesn't implement the GNU ld powerpc64's
>> __tls_get_addr_opt (pseudo-TLSDESC): https://maskray.me/blog/2021-02-14-all-about-thread-local-storage#powerpc-__tls_get_addr_opt
>> Let me send a configure patch to disable it...
>>
>> Actually I do not know how to disable tst-tlsopt-powerpc properly.
>>
>> Perhaps add sysdeps/powerpc/configure.ac and move the
>> sysdeps/unix/sysv/linux/powerpc/configure.ac --no-tls-get-addr-optimize
>> to sysdeps/powerpc/configure.ac?
>> sysdeps/powerpc/preconfigure.ac exists (I don't know how it is used).
>>
>> The patch requires some non-trivial configure.ac change, so I'd hope
>> that an expert can do it 
> Maybe just disable the test if __tls_get_addr_opt (pseudo-TLSDESC) is not supported
> by the linker?
>
>
>>
>> Hey, so lld linked glibc powerpc64 is in a pretty good status!
> I am still not sure about it, I did could run some tests but I am still
> struggling to get a make check to finish.

Re: lld status with powerpc64

[Adhemerval Zanella]

On 07/11/2021 11:24, Bill Schmidt wrote:
> Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.

For this specific issue I just sent a patch to fix it on glibc side [1].
However I think it would be good if lld also implements the ld.bfd
optimization to fallback to older stub generation if no pcrel relocation
is found (although it is debatable that @notoc should implicit generate
older ISA code depending of the resulting objects being linked against).

[1] https://patchwork.sourceware.org/project/glibc/patch/20211108113316.8867-1-adhemerval.zanella@linaro.org/

Re: lld status with powerpc64

[Bill Schmidt]

On 11/8/21 5:37 AM, Adhemerval Zanella wrote:
>
> On 07/11/2021 11:24, Bill Schmidt wrote:
>> Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.
> For this specific issue I just sent a patch to fix it on glibc side [1].
> However I think it would be good if lld also implements the ld.bfd
> optimization to fallback to older stub generation if no pcrel relocation
> is found (although it is debatable that @notoc should implicit generate
> older ISA code depending of the resulting objects being linked against).
>
> [1] https://patchwork.sourceware.org/project/glibc/patch/20211108113316.8867-1-adhemerval.zanella@linaro.org/

Hi Adhemerval,

Current course and speed is that @notoc will imply pcrel stubs on P10 and later,
but will not do so on P9 and earlier.  The relocation currently associated with
@notoc actually came with ELFv2 on P8 and, although no compilers ever generated 
@notoc, assembly routines using it prior to P10 are a valid case and should not be
punished.  This can be handled by generating a different reloc for @notoc in the
two cases.

If this solution holds up, then changes to glibc should be unnecessary.

Thanks,
Bill

Re: lld status with powerpc64

[Adhemerval Zanella]

On 08/11/2021 10:26, Bill Schmidt wrote:
> On 11/8/21 5:37 AM, Adhemerval Zanella wrote:
>>
>> On 07/11/2021 11:24, Bill Schmidt wrote:
>>> Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.
>> For this specific issue I just sent a patch to fix it on glibc side [1].
>> However I think it would be good if lld also implements the ld.bfd
>> optimization to fallback to older stub generation if no pcrel relocation
>> is found (although it is debatable that @notoc should implicit generate
>> older ISA code depending of the resulting objects being linked against).
>>
>> [1] https://patchwork.sourceware.org/project/glibc/patch/20211108113316.8867-1-adhemerval.zanella@linaro.org/
> 
> Hi Adhemerval,
> 
> Current course and speed is that @notoc will imply pcrel stubs on P10 and later,
> but will not do so on P9 and earlier.  The relocation currently associated with
> @notoc actually came with ELFv2 on P8 and, although no compilers ever generated 
> @notoc, assembly routines using it prior to P10 are a valid case and should not be
> punished.  This can be handled by generating a different reloc for @notoc in the
> two cases.
> 
> If this solution holds up, then changes to glibc should be unnecessary.

The main problem is this imposes an extra burden for the linker, where it 
need to implement the ld.bfd optimization to not generate the power10 stubs 
if no pcrel is found.  And it seems that lld does not yet support this
yes and I guess it has not been an issue because @notoc in assembly
routines should be rare. 

It also means that stubs generation are subject to a combination of
relocation on different objects (@notoc on assembly does not necessary
generate power10 stub with default linker option).  I think it should
be ok, although I see this as really confusing since it took some time
to figure out what ld.lfd was doing.

Re: lld status with powerpc64

[Bill Schmidt]

On 11/8/21 7:54 AM, Adhemerval Zanella wrote:
>
> On 08/11/2021 10:26, Bill Schmidt wrote:
>> On 11/8/21 5:37 AM, Adhemerval Zanella wrote:
>>> On 07/11/2021 11:24, Bill Schmidt wrote:
>>>> Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.
>>> For this specific issue I just sent a patch to fix it on glibc side [1].
>>> However I think it would be good if lld also implements the ld.bfd
>>> optimization to fallback to older stub generation if no pcrel relocation
>>> is found (although it is debatable that @notoc should implicit generate
>>> older ISA code depending of the resulting objects being linked against).
>>>
>>> [1] https://patchwork.sourceware.org/project/glibc/patch/20211108113316.8867-1-adhemerval.zanella@linaro.org/
>> Hi Adhemerval,
>>
>> Current course and speed is that @notoc will imply pcrel stubs on P10 and later,
>> but will not do so on P9 and earlier.  The relocation currently associated with
>> @notoc actually came with ELFv2 on P8 and, although no compilers ever generated 
>> @notoc, assembly routines using it prior to P10 are a valid case and should not be
>> punished.  This can be handled by generating a different reloc for @notoc in the
>> two cases.
>>
>> If this solution holds up, then changes to glibc should be unnecessary.
> The main problem is this imposes an extra burden for the linker, where it 
> need to implement the ld.bfd optimization to not generate the power10 stubs 
> if no pcrel is found.  And it seems that lld does not yet support this
> yes and I guess it has not been an issue because @notoc in assembly
> routines should be rare. 
>
> It also means that stubs generation are subject to a combination of
> relocation on different objects (@notoc on assembly does not necessary
> generate power10 stub with default linker option).  I think it should
> be ok, although I see this as really confusing since it took some time
> to figure out what ld.lfd was doing.

My only point is that the linker teams from both groups have been huddling about
this, so I wanted you to be aware the issue goes a little deeper than it appears
on the surface.  How glibc deals with this is up to glibc, of course. :)

Re: lld status with powerpc64

[Adhemerval Zanella]

On 08/11/2021 10:59, Bill Schmidt wrote:
> On 11/8/21 7:54 AM, Adhemerval Zanella wrote:
>>
>> On 08/11/2021 10:26, Bill Schmidt wrote:
>>> On 11/8/21 5:37 AM, Adhemerval Zanella wrote:
>>>> On 07/11/2021 11:24, Bill Schmidt wrote:
>>>>> Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.
>>>> For this specific issue I just sent a patch to fix it on glibc side [1].
>>>> However I think it would be good if lld also implements the ld.bfd
>>>> optimization to fallback to older stub generation if no pcrel relocation
>>>> is found (although it is debatable that @notoc should implicit generate
>>>> older ISA code depending of the resulting objects being linked against).
>>>>
>>>> [1] https://patchwork.sourceware.org/project/glibc/patch/20211108113316.8867-1-adhemerval.zanella@linaro.org/
>>> Hi Adhemerval,
>>>
>>> Current course and speed is that @notoc will imply pcrel stubs on P10 and later,
>>> but will not do so on P9 and earlier.  The relocation currently associated with
>>> @notoc actually came with ELFv2 on P8 and, although no compilers ever generated 
>>> @notoc, assembly routines using it prior to P10 are a valid case and should not be
>>> punished.  This can be handled by generating a different reloc for @notoc in the
>>> two cases.
>>>
>>> If this solution holds up, then changes to glibc should be unnecessary.
>> The main problem is this imposes an extra burden for the linker, where it 
>> need to implement the ld.bfd optimization to not generate the power10 stubs 
>> if no pcrel is found.  And it seems that lld does not yet support this
>> yes and I guess it has not been an issue because @notoc in assembly
>> routines should be rare. 
>>
>> It also means that stubs generation are subject to a combination of
>> relocation on different objects (@notoc on assembly does not necessary
>> generate power10 stub with default linker option).  I think it should
>> be ok, although I see this as really confusing since it took some time
>> to figure out what ld.lfd was doing.
> 
> My only point is that the linker teams from both groups have been huddling about
> this, so I wanted you to be aware the issue goes a little deeper than it appears
> on the surface.  How glibc deals with this is up to glibc, of course. :)
> 

Well, my patch had to disable notoc exactly because lld version does not support
such optimization (so default builds always use power10 instruction).  It means
that lld powerpc64le support is either incomplete or @notoc handling need to be
clarified.

Re: lld status with powerpc64

[Bill Schmidt]

On 11/8/21 8:11 AM, Adhemerval Zanella wrote:
>
> On 08/11/2021 10:59, Bill Schmidt wrote:
>> On 11/8/21 7:54 AM, Adhemerval Zanella wrote:
>>> On 08/11/2021 10:26, Bill Schmidt wrote:
>>>> On 11/8/21 5:37 AM, Adhemerval Zanella wrote:
>>>>> On 07/11/2021 11:24, Bill Schmidt wrote:
>>>>>> Please coordinate with Alan Modra and Nemanja Ivanovic on this topic.  There are ongoing discussions about bfd and lld linker support around @notoc that will be resolved soon, and Tulio is on vacation, so I don't want the community to make steps they'll have to undo later, or for people to engage in duplicate work.
>>>>> For this specific issue I just sent a patch to fix it on glibc side [1].
>>>>> However I think it would be good if lld also implements the ld.bfd
>>>>> optimization to fallback to older stub generation if no pcrel relocation
>>>>> is found (although it is debatable that @notoc should implicit generate
>>>>> older ISA code depending of the resulting objects being linked against).
>>>>>
>>>>> [1] https://patchwork.sourceware.org/project/glibc/patch/20211108113316.8867-1-adhemerval.zanella@linaro.org/
>>>> Hi Adhemerval,
>>>>
>>>> Current course and speed is that @notoc will imply pcrel stubs on P10 and later,
>>>> but will not do so on P9 and earlier.  The relocation currently associated with
>>>> @notoc actually came with ELFv2 on P8 and, although no compilers ever generated 
>>>> @notoc, assembly routines using it prior to P10 are a valid case and should not be
>>>> punished.  This can be handled by generating a different reloc for @notoc in the
>>>> two cases.
>>>>
>>>> If this solution holds up, then changes to glibc should be unnecessary.
>>> The main problem is this imposes an extra burden for the linker, where it 
>>> need to implement the ld.bfd optimization to not generate the power10 stubs 
>>> if no pcrel is found.  And it seems that lld does not yet support this
>>> yes and I guess it has not been an issue because @notoc in assembly
>>> routines should be rare. 
>>>
>>> It also means that stubs generation are subject to a combination of
>>> relocation on different objects (@notoc on assembly does not necessary
>>> generate power10 stub with default linker option).  I think it should
>>> be ok, although I see this as really confusing since it took some time
>>> to figure out what ld.lfd was doing.
>> My only point is that the linker teams from both groups have been huddling about
>> this, so I wanted you to be aware the issue goes a little deeper than it appears
>> on the surface.  How glibc deals with this is up to glibc, of course. :)
>>
> Well, my patch had to disable notoc exactly because lld version does not support
> such optimization (so default builds always use power10 instruction).  It means
> that lld powerpc64le support is either incomplete or @notoc handling need to be
> clarified.
Precisely.

Re: lld status with powerpc64

[Fangrui Song]

I agree with Adhemerval's point that "The main problem is this imposes
an extra burden for the linker" is unfortunate. (Magic behavior of
--power10-stubs=auto).

On 2021-11-08, Nemanja Ivanovic wrote:
>Just to chime in here...
> 
>Adhemerval,
>just a couple of notes about what you are referring to as an "optimization".
>The reason that LLD emits P10 PC-Relative prefixed instructions in stubs is
>because of an implicit assumption that the *_NOTOC relocation is restricted for
>P10 PC-Relative builds as it was never emitted by any compiler prior. However,
>I don't think it is accurate to refer to no P10 stubs with *_NOTOC as an
>"optimization". The two linkers have slightly different logic for deciding when
>it is OK to use P10 stubs. GLIBC is developed using the GNU toolchain so
>naturally relies on behaviour of GNU tools. In this case, the GNU linkers do
>not emit P10 stubs so GLIBC decided to use the @notoc. However, in a slightly
>different situation, the GNU linkers would generate P10 stubs as well. I
>imagine if GLIBC development added code that causes that to happen, that code
>would never be upstreamed as it causes broken builds.
> 
>Ultimately, what is needed is a clear indication in each relocation that using
>P10 stubs is OK for that call. Relying on presence of P10 relocations elsewhere
>in the compilation unit is problematic for at least two reasons (less efficient
>for the linker to look over all relocations twice, can lead to incorrect
>choices). This is why the proposal is to have different relocations for P10 and
>pre-P10 code. My preference would be to make this clear in the textual
>relocations as well (rather than the assembler emitting two different
>relocations for @notoc depending on -mpower10 vs -m<something_older>).

Thanks for investigating the long-term solution.

Nowadays among popular architectures, all of AArch64/RISC-V/x86 have
open places discussing ABI and toolchain implementations. It'd be nice
if ppc64 had a similar place so that we could detect problems earlier.
If more eyes had stared at the @notoc usage, we probably could have
avoided some friction in glibc and ld.lld...

---

> Current course and speed is that @notoc will imply pcrel stubs on P10
> and later, but will not do so on P9 and earlier.  The relocation
> currently associated with @notoc actually came with ELFv2 on P8 and,
> although no compilers ever generated @notoc, assembly routines using it
> prior to P10 are a valid case and should not be punished.  This can be
> handled by generating a different reloc for @notoc in the two cases.

In the context of glibc's @notoc assembly usage, I don't see why
suppresing @notoc for pre-POWER10 builds  would be a punishment:)

The branch targets are non-preemptible symbols.
In the absence of a long branch, @notoc to toc code needs a thunk (in
both GNU ld and ld.lld) while toc to toc doesn't.

In the case of a long branch, @notoc with PC-relative instructions is
slightly more efficient because it uses PC-relative instead of TOC's loading an address from .branch_lt
but the very minor improvement doesn't seem to justify the complexity :)

I say "minor" because I have checked the glibc build on a POWER9 machine
(GCC 8.3.0, binutils 2.31.1) and an x86-64 using cross compilatioin (GCC
10.2.1, binutils 2.37).
On the POWER9 machine, I don't find any thunk for __syscall_error and a
few other symbols using the glibc NOTOC macro.
On x86-64, I saw a long branch call site but I am not sure it can ever
be hot :)

For
https://sourceware.org/pipermail/libc-alpha/2021-November/132767.html ,
I'd like to support it. It will address the immediate problem linking
glibc with LLD 13.0.0 since glibc 2.35 is not too faraway (scheduled on
2022-02-01).

But the patch seems to complicate things.
I think checking whether with the supplied CFLAGS, CC generates @notoc
is better than checking assembler support.

>Nemanja Ivanovic
>LLVM PPC Backend Development
>IBM Toronto Lab
>Email: nemanjai@ca.ibm.com
>Phone: 905-413-3388
> 
>
>    ----- Original message -----
>    From: "Bill Schmidt" <wschmidt@linux.ibm.com>
>    To: "Adhemerval Zanella" <adhemerval.zanella@linaro.org>, "Fangrui Song"
>    <maskray@google.com>, "Tulio Magno Quites Machado Filho"
>    <tuliom@linux.ibm.com>, "Alan Modra" <amodra@gmail.com>, "Nemanja Ivanovic"
>    <nemanjai@ca.ibm.com>
>    Cc: libc-alpha@sourceware.org
>    Subject: Re: lld status with powerpc64
>    Date: Mon, Nov 8, 2021 9:12 AM
>     
>    On 11/8/21 8:11 AM, Adhemerval Zanella wrote:
>    >
>    > On 08/11/2021 10:59, Bill Schmidt wrote:
>    >> On 11/8/21 7:54 AM, Adhemerval Zanella wrote:
>    >>> On 08/11/2021 10:26, Bill Schmidt wrote:
>    >>>> On 11/8/21 5:37 AM, Adhemerval Zanella wrote:
>    >>>>> On 07/11/2021 11:24, Bill Schmidt wrote:
>    >>>>>> Please coordinate with Alan Modra and Nemanja Ivanovic on this
>    topic.  There are ongoing discussions about bfd and lld linker support
>    around @notoc that will be resolved soon, and Tulio is on vacation, so I
>    don't want the community to make steps they'll have to undo later, or for
>    people to engage in duplicate work.
>    >>>>> For this specific issue I just sent a patch to fix it on glibc side
>    [1].
>    >>>>> However I think it would be good if lld also implements the ld.bfd
>    >>>>> optimization to fallback to older stub generation if no pcrel
>    relocation
>    >>>>> is found (although it is debatable that @notoc should implicit
>    generate
>    >>>>> older ISA code depending of the resulting objects being linked
>    against).
>    >>>>>
>    >>>>> [1] https://patchwork.sourceware.org/project/glibc/patch/
>    20211108113316.8867-1-adhemerval.zanella@linaro.org/
>    >>>> Hi Adhemerval,
>    >>>>
>    >>>> Current course and speed is that @notoc will imply pcrel stubs on P10
>    and later,
>    >>>> but will not do so on P9 and earlier.  The relocation currently
>    associated with
>    >>>> @notoc actually came with ELFv2 on P8 and, although no compilers ever
>    generated
>    >>>> @notoc, assembly routines using it prior to P10 are a valid case and
>    should not be
>    >>>> punished.  This can be handled by generating a different reloc for
>    @notoc in the
>    >>>> two cases.
>    >>>>
>    >>>> If this solution holds up, then changes to glibc should be
>    unnecessary.
>    >>> The main problem is this imposes an extra burden for the linker, where
>    it
>    >>> need to implement the ld.bfd optimization to not generate the power10
>    stubs
>    >>> if no pcrel is found.  And it seems that lld does not yet support this
>    >>> yes and I guess it has not been an issue because @notoc in assembly
>    >>> routines should be rare.
>    >>>
>    >>> It also means that stubs generation are subject to a combination of
>    >>> relocation on different objects (@notoc on assembly does not necessary
>    >>> generate power10 stub with default linker option).  I think it should
>    >>> be ok, although I see this as really confusing since it took some time
>    >>> to figure out what ld.lfd was doing.
>    >> My only point is that the linker teams from both groups have been
>    huddling about
>    >> this, so I wanted you to be aware the issue goes a little deeper than it
>    appears
>    >> on the surface.  How glibc deals with this is up to glibc, of course. :)
>    >>
>    > Well, my patch had to disable notoc exactly because lld version does not
>    support
>    > such optimization (so default builds always use power10 instruction).  It
>    means
>    > that lld powerpc64le support is either incomplete or @notoc handling need
>    to be
>    > clarified.
>    Precisely.
>
> 
>

Re: lld status with powerpc64

[Adhemerval Zanella]

On 08/11/2021 19:38, Fangrui Song wrote:
> I agree with Adhemerval's point that "The main problem is this imposes
> an extra burden for the linker" is unfortunate. (Magic behavior of
> --power10-stubs=auto).

That the main problem in fact, where ld.bfd and lld have different semantics
for --power10-stubs=auto and it is the default option. 

> 
> On 2021-11-08, Nemanja Ivanovic wrote:
>> Just to chime in here...
>>
>> Adhemerval,
>> just a couple of notes about what you are referring to as an "optimization".
>> The reason that LLD emits P10 PC-Relative prefixed instructions in stubs is
>> because of an implicit assumption that the *_NOTOC relocation is restricted for
>> P10 PC-Relative builds as it was never emitted by any compiler prior. However,
>> I don't think it is accurate to refer to no P10 stubs with *_NOTOC as an
>> "optimization". The two linkers have slightly different logic for deciding when
>> it is OK to use P10 stubs. GLIBC is developed using the GNU toolchain so
>> naturally relies on behaviour of GNU tools. In this case, the GNU linkers do
>> not emit P10 stubs so GLIBC decided to use the @notoc. However, in a slightly
>> different situation, the GNU linkers would generate P10 stubs as well. I
>> imagine if GLIBC development added code that causes that to happen, that code
>> would never be upstreamed as it causes broken builds.

I agree that glibc is tied to GNU toolchain (we aim to use 'GNU C' along with
all its extension as the base language) and I also agree that such optimization
might be costly for lld.

But from an user perspective I would expect that if ldd is trying to mimic 
ld.bfd behavior regarding power10 stub creation, the semantic should be at least 
the same or error if something is not supported. The issue is exactly that subtle
issues that arise when something is either underspecified or not properly documented.

>>
>> Ultimately, what is needed is a clear indication in each relocation that using
>> P10 stubs is OK for that call. Relying on presence of P10 relocations elsewhere
>> in the compilation unit is problematic for at least two reasons (less efficient
>> for the linker to look over all relocations twice, can lead to incorrect
>> choices). This is why the proposal is to have different relocations for P10 and
>> pre-P10 code. My preference would be to make this clear in the textual
>> relocations as well (rather than the assembler emitting two different
>> relocations for @notoc depending on -mpower10 vs -m<something_older>).
> 
> Thanks for investigating the long-term solution.
> 
> Nowadays among popular architectures, all of AArch64/RISC-V/x86 have
> open places discussing ABI and toolchain implementations. It'd be nice
> if ppc64 had a similar place so that we could detect problems earlier.
> If more eyes had stared at the @notoc usage, we probably could have
> avoided some friction in glibc and ld.lld...
> 
> ---
> 
>> Current course and speed is that @notoc will imply pcrel stubs on P10
>> and later, but will not do so on P9 and earlier.  The relocation
>> currently associated with @notoc actually came with ELFv2 on P8 and,
>> although no compilers ever generated @notoc, assembly routines using it
>> prior to P10 are a valid case and should not be punished.  This can be
>> handled by generating a different reloc for @notoc in the two cases.
> 
> In the context of glibc's @notoc assembly usage, I don't see why
> suppresing @notoc for pre-POWER10 builds  would be a punishment:)
> 
> The branch targets are non-preemptible symbols.
> In the absence of a long branch, @notoc to toc code needs a thunk (in
> both GNU ld and ld.lld) while toc to toc doesn't.
> 
> In the case of a long branch, @notoc with PC-relative instructions is
> slightly more efficient because it uses PC-relative instead of TOC's loading an address from .branch_lt
> but the very minor improvement doesn't seem to justify the complexity :)
> 
> I say "minor" because I have checked the glibc build on a POWER9 machine
> (GCC 8.3.0, binutils 2.31.1) and an x86-64 using cross compilatioin (GCC
> 10.2.1, binutils 2.37).
> On the POWER9 machine, I don't find any thunk for __syscall_error and a
> few other symbols using the glibc NOTOC macro.
> On x86-64, I saw a long branch call site but I am not sure it can ever
> be hot :)

I won't indeed, since @notoc only is helpful when power8 objects are being
created.

> 
> For
> https://sourceware.org/pipermail/libc-alpha/2021-November/132767.html ,
> I'd like to support it. It will address the immediate problem linking
> glibc with LLD 13.0.0 since glibc 2.35 is not too faraway (scheduled on
> 2022-02-01).
> 
> But the patch seems to complicate things.
> I think checking whether with the supplied CFLAGS, CC generates @notoc
> is better than checking assembler support.

Well, that is exactly of the patch does:

  libc_cv_ppc64_notoc=no
  if AC_TRY_COMMAND([${CC-cc} $CFLAGS $CPPFLAGS -S -o conftest.s conftest.c]) \
     && AC_TRY_COMMAND([grep -q -E 'bar@notoc' conftest.s])
  then
    libc_cv_ppc64_notoc=yes
  fi

I think the title is misleading since it tests only compiler, not linker.