Dynamic loader path on x86_64

Dynamic loader path on x86_64

[Ilya Trukhanov]

Hello!

I'm building glibc-2.34 on x86_64 for a no-multilib prefix like so:

$ mkdir build && cd build
$ ../configure --prefix=/pfx
$ make -j16
$ make install

After looking at sysdeps/unix/sysv/linux/x86_64/ldconfig.h:

#define SYSDEP_KNOWN_INTERPRETER_NAMES \
  { "/lib/ld-linux.so.2", FLAG_ELF_LIBC6 }, \
  { "/libx32/ld-linux-x32.so.2", FLAG_ELF_LIBC6 }, \
  { "/lib64/ld-linux-x86-64.so.2", FLAG_ELF_LIBC6 },

I'd expect the loader to be installed to
/pfx/lib64/ld-linux-x86-64.so.2, but it is instead installed to
/pfx/lib/ld-linux-x86-64.so.2, breaking (at least) /pfx/bin/ldd, which
gives me "not a dynamic executable" for all files. Strace hints at the
problem:

...
faccessat2(AT_FDCWD, "/pfx/lib/ld-linux.so.2", X_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
faccessat2(AT_FDCWD, "/pfx/lib64/ld-linux-x86-64.so.2", X_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
faccessat2(AT_FDCWD, "/pfx/libx32/ld-linux-x32.so.2", X_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
...

Indeed, symlinking /pfx/lib64 to /pfx/lib makes it work.

Is this some sort of multilib quirk? Aarch64 and riscv ldconfig.h seem
to have theirs at /lib just fine.

Is the lib64->lib symlink the intended way to solve this?

Would things break if I simply patch ldconfig.h to add
{ "/lib/ld-linux-x86-64.so.2", FLAG_ELF_LIBC6 }?

Is there better way to do this that I'm not aware of, maybe?

Best,
Ilya Trukhanov.

Re: Dynamic loader path on x86_64

[Adhemerval Zanella]

On 02/01/2022 14:29, Ilya Trukhanov via Libc-help wrote:
> Hello!
> 
> I'm building glibc-2.34 on x86_64 for a no-multilib prefix like so:
> 
> $ mkdir build && cd build
> $ ../configure --prefix=/pfx
> $ make -j16
> $ make install
> 
> After looking at sysdeps/unix/sysv/linux/x86_64/ldconfig.h:
> 
> #define SYSDEP_KNOWN_INTERPRETER_NAMES \
>   { "/lib/ld-linux.so.2", FLAG_ELF_LIBC6 }, \
>   { "/libx32/ld-linux-x32.so.2", FLAG_ELF_LIBC6 }, \
>   { "/lib64/ld-linux-x86-64.so.2", FLAG_ELF_LIBC6 },
> 
> I'd expect the loader to be installed to
> /pfx/lib64/ld-linux-x86-64.so.2, but it is instead installed to
> /pfx/lib/ld-linux-x86-64.so.2, breaking (at least) /pfx/bin/ldd, which
> gives me "not a dynamic executable" for all files. Strace hints at the
> problem:

This is expected behavior and the idea is to allow 32-bit and 64-bit
binaries to work on same installation.  You can check the aclocal.m4
rule that selects the folder that install the loader:

aclocal.m4:

296 dnl Default to slibdir named SLIBDIR instead of "lib", and rtlddir
297 dnl named RTLDDIR instead of "lib".  This is used to put 64-bit
298 dnl libraries in /lib64.
299 dnl LIBC_SLIBDIR_RTLDDIR([slibdir], [rtlddir])
300 AC_DEFUN([LIBC_SLIBDIR_RTLDDIR],
301 [test -n "$libc_cv_slibdir" ||
302 case "$prefix" in
303 /usr | /usr/)
304   libc_cv_slibdir='/$1'
305   libc_cv_rtlddir='/$2'
306   if test "$libdir" = '${exec_prefix}/lib'; then
307     libdir='${exec_prefix}/$1';
308     # Locale data can be shared between 32-bit and 64-bit libraries.
309     libc_cv_complocaledir='${exec_prefix}/lib/locale'
310   fi
311   ;;
312 esac])
 
As you can see, you can override by defining libc_cv_slibdir at configure
time:

$ libc_cv_slibdir=/lib64 ../../glibc-git/configure --prefix=/usr
$ make -j24
$ make install DESTDIR=$PWD/install
$ find install/ -iname ld-linux-x86-64.so.2
install/lib64/ld-linux-x86-64.so.2

However the ldd iss is unexpected, since it should handle it by rewriting
the rtld path:

elf/Makefile

756 ldd-rewrite = -e 's%@RTLD@%$(rtlddir)/$(rtld-installed-name)%g' \
757               -e 's%@VERSION@%$(version)%g' \
758               -e 's|@PKGVERSION@|$(PKGVERSION)|g' \
759               -e 's|@REPORT_BUGS_TO@|$(REPORT_BUGS_TO)|g' \
760               -e 's%@TEXTDOMAINDIR@%$(localedir)%g'
761 
762 ifeq ($(ldd-rewrite-script),no)
763 define gen-ldd
764 LC_ALL=C sed $(ldd-rewrite) < $< > $@.new
765 endef
766 else
767 define gen-ldd
768 LC_ALL=C sed $(ldd-rewrite) < $< \
769 | LC_ALL=C sed -f $(patsubst $(..)/%,/%,$(..)$(ldd-rewrite-script)) > $@.new
770 endef
771 endif

And x86_64 does instruct make to rewrite it:

$ git grep -w ldd_rewrite_script | grep x86_64
sysdeps/unix/sysv/linux/x86_64/configure:ldd_rewrite_script=sysdeps/unix/sysv/linux/x86_64/ldd-rewrite.sed
sysdeps/unix/sysv/linux/x86_64/configure.ac:ldd_rewrite_script=sysdeps/unix/sysv/linux/x86_64/ldd-rewrite.sed

You can see that a normal installation will indeed set all the possible
loaders on the RTDLIST:

$ cat install/usr/bin/ldd
[...]
29 RTLDLIST="/lib/ld-linux.so.2 /lib64/ld-linux-x86-64.so.2 /libx32/ld-linux-x32.so.2"
[...]

> 
> ...
> faccessat2(AT_FDCWD, "/pfx/lib/ld-linux.so.2", X_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
> faccessat2(AT_FDCWD, "/pfx/lib64/ld-linux-x86-64.so.2", X_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
> faccessat2(AT_FDCWD, "/pfx/libx32/ld-linux-x32.so.2", X_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
> ...
> 
> Indeed, symlinking /pfx/lib64 to /pfx/lib makes it work.
> 
> Is this some sort of multilib quirk? Aarch64 and riscv ldconfig.h seem
> to have theirs at /lib just fine.

aarch64, I am not sure about riscv, does not really have the concept of
multilib since A32 supports is not strictly required by ISA. 

> 
> Is the lib64->lib symlink the intended way to solve this?

It is a way to solve afaiu yes.

> 
> Would things break if I simply patch ldconfig.h to add
> { "/lib/ld-linux-x86-64.so.2", FLAG_ELF_LIBC6 }?
> 
> Is there better way to do this that I'm not aware of, maybe?

I think it is better to just use libc_cv_slibdir if you not intend to
use 32-bit libc along with the 64-bit install.  I would also check on
how each distro does that, I know debian based system does patch glibc
to support their own internal triple organization.

> 
> Best,
> Ilya Trukhanov.