Re: glibc-hwcaps for armv7 (neon-vfpv4)

[Mathieu Malaterre <malat@debian.org>]

Hi !

On Thu, Aug 31, 2023 at 5:30 PM Adhemerval Zanella Netto
<adhemerval.zanella@linaro.org> wrote:
>
>
>
> On 31/08/23 09:08, Mathieu Malaterre wrote:
> > Hi Florian !
> >
> > On Thu, Aug 31, 2023 at 1:24 PM Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >> * Mathieu Malaterre:
> >>
> >>> Dear glibc maintainer,
> >>>
> >>> I fail to understand the ld.so man page (Debian/sid version: man-pages
> >>> 6.03) for hwcaps support. Specifically I'd like to install a shared
> >>> lib on a Debian/armhf system (baseline is neon-less) which was build
> >>> with gcc option:
> >>>
> >>>   -march=armv7-a -mfpu=neon-vfpv4
> >>>
> >>> What subfolder should I be using ?
> >>
> >> You can use “LD_LIBRARY_PATH=. /bin/true” to see which subdirectories
> >> are probed.  I think either neon/vfp or vfp/neon should be among the
> >> subdirectories.  This functionality has since been removed from upstream
> >> glibc because it sometimes results in hundereds of extra openat system
> >> calls.
> >
> > oh, thanks for the trick. Could you confirm my conclusion below
> > (Debian/i386 sid system):
> >
> > % file /usr/lib/i386-linux-gnu/i686/sse2/libx264.so.164
> > /usr/lib/i386-linux-gnu/i686/sse2/libx264.so.164: ELF 32-bit LSB
> > shared object, Intel 80386, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=e66974d10aef77af7ed504266cde974d103484d6, stripped
> >
> > However:
> >
> > % /sbin/ldconfig -v 2>/dev/null | grep -A3 hwcap
> > -> nothing
> >
> > same goes with:
> >
> > % LD_DEBUG=libs LD_LIBRARY_PATH=. /bin/true
> >    3327956:     find library=libc.so.6 [0]; searching
> >    3327956:      search path=.          (LD_LIBRARY_PATH)
> >    3327956:       trying file=./libc.so.6
> >    3327956:      search cache=/etc/ld.so.cache
> >    3327956:       trying file=/lib/i386-linux-gnu/libc.so.6
> >    3327956:
> >    3327956:
> >    3327956:     calling init: /lib/ld-linux.so.2
> >    3327956:
> >    3327956:
> >    3327956:     calling init: /lib/i386-linux-gnu/libc.so.6
> >    3327956:
> >    3327956:
> >    3327956:     initialize program: /bin/true
> >    3327956:
> >    3327956:
> >    3327956:     transferring control: /bin/true
> >    3327956:
> >    3327956:
> >    3327956:     calling fini:  [0]
> >    3327956:
> >    3327956:
> >    3327956:     calling fini: /lib/i386-linux-gnu/libc.so.6 [0]
> >    3327956:
> >    3327956:
> >    3327956:     calling fini: /lib/ld-linux.so.2 [0]
> >    3327956:
> >
> >
> > If I understand correctly the file
> > /usr/lib/i386-linux-gnu/i686/sse2/libx264.so.164 cannot be used on
> > Debian/sid/i386 system since libc used is:
> >
> > % /lib/i386-linux-gnu/libc.so.6
> > GNU C Library (Debian GLIBC 2.37-7) stable release version 2.37.
> > Copyright (C) 2023 Free Software Foundation, Inc.
> > This is free software; see the source for copying conditions.
> > There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
> > PARTICULAR PURPOSE.
> > Compiled by GNU CC version 12.3.0.
> > libc ABIs: UNIQUE IFUNC ABSOLUTE
> > Minimum supported kernel: 3.2.0
> > For bug reporting instructions, please see:
> > <http://www.debian.org/Bugs/>.
>
> From your system configuration, it seems so.  On a ubuntu22 I see:
>
> $ find /usr/ -iname libx264.so*
> /usr/lib/i386-linux-gnu/libx264.so.163
> /usr/lib/i386-linux-gnu/i686/sse2/libx264.so.163
> /usr/lib/x86_64-linux-gnu/libx264.so.163
> $ /sbin/ldconfig -v 2>/dev/null | grep -A3 hwcap
> /lib/i386-linux-gnu/i686: (hwcap: 0x0002000000000000) (from /etc/ld.so.conf.d/i386-linux-gnu.conf:3)
> /lib/i386-linux-gnu/i686/sse2: (hwcap: 0x0002000000000001) (from /etc/ld.so.conf.d/i386-linux-gnu.conf:3)
>         libx264.so.163 -> libx264.so.163
>
> $ cat /etc/ld.so.conf.d/i386-linux-gnu.conf
> # Multiarch support
> /usr/local/lib/i386-linux-gnu
> /lib/i386-linux-gnu
> /usr/lib/i386-linux-gnu
> /usr/local/lib/i686-linux-gnu
> /lib/i686-linux-gnu
> /usr/lib/i686-linux-gnu
>
> $ /lib/ld-linux.so.2 --help
> [...]
> Shared library search path:
>   (libraries located via /etc/ld.so.cache)
>   /lib/i386-linux-gnu (system search path)
>   /usr/lib/i386-linux-gnu (system search path)
>   /lib (system search path)
>   /usr/lib (system search path)
>
> No subdirectories of glibc-hwcaps directories are searched.

^very neat trick. I was not aware of this command.

> Legacy HWCAP subdirectories under library search path directories:
>   i686 (AT_PLATFORM; supported, searched)
>   tls (supported, searched)
>   sse2 (supported, searched)
>
> So if you have '/usr/lib/i386-linux-gnu' add on your ld.so.cache, the
> <path>/i686/see will be search through legacy hwcap support.

This has been confirmed by Florian since: you'd need glibc 2.37 to
reproduce the behavior I am seeing in Debian/sid.

> >
> >>> Conversely how should I read the following:
> >>>
> >>> % sudo ldconfig -p | grep hwcap
> >>>         libfoo.so.160 (libc6,x86-64, hwcap: 0x0004000000000000) =>
> >>> /lib/x86_64-linux-gnu/haswell/libfoo.so.160
> >>>         libfoo.so.160 (libc6,x86-64, hwcap: 0x0000000000000004) =>
> >>> /lib/x86_64-linux-gnu/avx512_1/libfoo.so.160
> >>>
> >>> What does "haswell" / "avx512_1" subfolder implies in terms of gcc
> >>> compile options ?
> >>
> >> It's complicated.  Nowdays, on x86-64, you can use
> >> glibc-hwcaps/x86-64-v3 and glibc-hwcaps/x86-64-v4, which are designed to
> >> correspond to -march=x86-64-v3 and -march=x86-64-v4, and x86-64-v4 is a
> >> superset of x86-64-v3.
> >
> > Ok, thanks for the update.
> >
> > Could you just review what I see on my Debian/sid/armhf system:
> >
> > % cat /proc/cpuinfo | grep Features | uniq
> > Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4
> > idiva idivt lpae evtstrm
> > % LD_DEBUG=libs LD_LIBRARY_PATH=. /bin/true
> >    2964957:     find library=libc.so.6 [0]; searching
> >    2964957:      search path=.          (LD_LIBRARY_PATH)
> >    2964957:       trying file=./libc.so.6
> >    2964957:      search cache=/etc/ld.so.cache
> >    2964957:       trying file=/lib/arm-linux-gnueabihf/libc.so.6
> >    2964957:
> >    2964957:
> >    2964957:     calling init: /lib/ld-linux-armhf.so.3
> >    2964957:
> >    2964957:
> >    2964957:     calling init: /lib/arm-linux-gnueabihf/libc.so.6
> >    2964957:
> >    2964957:
> >    2964957:     initialize program: /bin/true
> >    2964957:
> >    2964957:
> >    2964957:     transferring control: /bin/true
> >    2964957:
> >    2964957:
> >    2964957:     calling fini:  [0]
> >    2964957:
> >    2964957:
> >    2964957:     calling fini: /lib/arm-linux-gnueabihf/libc.so.6 [0]
> >    2964957:
> >    2964957:
> >    2964957:     calling fini: /lib/ld-linux-armhf.so.3 [0]
> >    2964957:
> >
> > Based on the above, it seems glibc is setup to never search hwcaps
> > optimized libraries on this system (even if neon/vfpv4 is supported by
> > system).
>
> This is a ubuntu22 armhf chroot on aarch64:
>
> $ /lib/ld-linux-armhf.so.3 --help
> [...]
> Shared library search path:
>   (libraries located via /etc/ld.so.cache)
>   /lib/arm-linux-gnueabihf (system search path)
>   /usr/lib/arm-linux-gnueabihf (system search path)
>   /lib (system search path)
>   /usr/lib (system search path)
>
> No subdirectories of glibc-hwcaps directories are searched.
>
> Legacy HWCAP subdirectories under library search path directories:
>   v8l (AT_PLATFORM; supported, searched)
>   tls (supported, searched)
>   neon (supported, searched)
>   vfp (supported, searched)
>
> So both neon and vfp; I can not say exactly how debian is configured since both
> ubuntu and debian carry out of tree patches; but I would expect them to be
> similar regarding libc.