glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[Baojun Wang]

Hi libc,

when calling getauxval(AT_HWCAP), it returns 0x2, because of below snippet:

https://code.woboq.org/userspace/glibc/sysdeps/x86/cpu-features.c.html#510

Is there any reason why glibc don't use the values passed from kernel auxv?

Thanks
baojun

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[Carlos O'Donell]

On 10/17/19 11:29 AM, Baojun Wang wrote:
> Hi libc,
> 
> when calling getauxval(AT_HWCAP), it returns 0x2, because of below snippet:
> 
> https://code.woboq.org/userspace/glibc/sysdeps/x86/cpu-features.c.html#510
> 
> Is there any reason why glibc don't use the values passed from kernel auxv?

I don't know of any good reason.

The semantics of getauxval() is to provide the auxiliary vector data.

In this case glibc *does* read AT_HWCAP, but then for x86 it overrides it
with a computed value based on cpuid.

HJ, Is there any reason we do this?

Baojun, Do you see a difference between getauxval() return and real AT_HWCAP?

-- 
Cheers,
Carlos.

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[Baojun Wang]

Yes, the values read from /proc/self/auxv is different (Linux x86_64,
glibc-2.27, tried glibc HEAD without any difference) with
getauxval(AT_HWCAP).

Here is the program I wrote for testing:

https://gist.github.com/wangbj/658da547437fe9b7eff2113cc9fc9741

And here is the output (check AT_HWCAP):

$ ./features

         AT_BASE: getauxval = 0x7f861434a000    , proc_getauxval =
0x7f861434a000

        AT_ENTRY: getauxval = 0x4005c0          , proc_getauxval =
0x4005c0

        AT_HWCAP: getauxval = 0x2               , proc_getauxval =
0xbfebfbff

Best Regards,
baojun

On Thu, Oct 17, 2019 at 4:26 PM Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 10/17/19 11:29 AM, Baojun Wang wrote:
> > Hi libc,
> >
> > when calling getauxval(AT_HWCAP), it returns 0x2, because of below snippet:
> >
> > https://code.woboq.org/userspace/glibc/sysdeps/x86/cpu-features.c.html#510
> >
> > Is there any reason why glibc don't use the values passed from kernel auxv?
>
> I don't know of any good reason.
>
> The semantics of getauxval() is to provide the auxiliary vector data.
>
> In this case glibc *does* read AT_HWCAP, but then for x86 it overrides it
> with a computed value based on cpuid.
>
> HJ, Is there any reason we do this?
>
> Baojun, Do you see a difference between getauxval() return and real AT_HWCAP?
>
> --
> Cheers,
> Carlos.
>

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[H.J. Lu]

On Thu, Oct 17, 2019 at 1:31 PM Baojun Wang <wangbj@gmail.com> wrote:
>
> Yes, the values read from /proc/self/auxv is different (Linux x86_64,
> glibc-2.27, tried glibc HEAD without any difference) with
> getauxval(AT_HWCAP).
>
> Here is the program I wrote for testing:
>
> https://gist.github.com/wangbj/658da547437fe9b7eff2113cc9fc9741
>
> And here is the output (check AT_HWCAP):
>
> $ ./features
>
>          AT_BASE: getauxval = 0x7f861434a000    , proc_getauxval =
> 0x7f861434a000
>
>         AT_ENTRY: getauxval = 0x4005c0          , proc_getauxval =
> 0x4005c0
>
>         AT_HWCAP: getauxval = 0x2               , proc_getauxval =
> 0xbfebfbff
>
> Best Regards,
> baojun
>
> On Thu, Oct 17, 2019 at 4:26 PM Carlos O'Donell <carlos@redhat.com> wrote:
> >
> > On 10/17/19 11:29 AM, Baojun Wang wrote:
> > > Hi libc,
> > >
> > > when calling getauxval(AT_HWCAP), it returns 0x2, because of below snippet:
> > >
> > > https://code.woboq.org/userspace/glibc/sysdeps/x86/cpu-features.c.html#510
> > >
> > > Is there any reason why glibc don't use the values passed from kernel auxv?
> >
> > I don't know of any good reason.
> >
> > The semantics of getauxval() is to provide the auxiliary vector data.
> >
> > In this case glibc *does* read AT_HWCAP, but then for x86 it overrides it
> > with a computed value based on cpuid.
> >
> > HJ, Is there any reason we do this?

dl_hwcap is used to load DSO optimized for a platform.  On i386, we use
AT_HWCAP passed from kernel.  On x86-64, we use a different approach
since AT_HWCAP passed from kernel isn't very useful:

commit 1432d38ea04ab5e96f21a382101856db5b49ad8a
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Wed May 3 13:42:42 2017 -0700

    x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]

    dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
    early during startup.  They are used by dynamic linker to determine
    platform and build an array of hardware capability names, which are
    added to search path when loading shared object.  dl_platform and
    dl_hwcap are unused on x86-64.  On i386, i386, i486, i586 and i686
    platforms were supported and only SSE2 capability was used.

    On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
    processor capabilities is obsolete since all information is available
    in dl_x86_cpu_features.  This patch sets dl_platform and dl_hwcap from
    dl_x86_cpu_features in dynamic linker.  On i386, the available plaforms
    are changed to i586 and i686 since i386 has been deprecated.  On x86-64,
    the available plaforms are haswell, which is for Haswell class processors
    with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which
    is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
    AVX512PF.  A capability, avx512_1, is also added to x86-64 for AVX512
    ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.

and

commit 45ff34638f034877b6a490c217d6a0632ce263f4
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Mon Sep 11 08:18:11 2017 -0700

    x86: Add x86_64 to x86-64 HWCAP [BZ #22093]

    Before glibc 2.26, ld.so set dl_platform to "x86_64" and searched the
    "x86_64" subdirectory when loading a shared library.  ld.so in glibc
    2.26 was changed to set dl_platform to "haswell" or "xeon_phi", based
    on supported ISAs.  This led to shared library loading failure for
    shared libraries placed under the "x86_64" subdirectory.

    This patch adds "x86_64" to x86-64 dl_hwcap so that ld.so will always
    search the "x86_64" subdirectory when loading a shared library.

    NB: We can't set x86-64 dl_platform to "x86-64" since ld.so will skip
    the "haswell" and "xeon_phi" subdirectories on "haswell" and "xeon_phi"
    machines.

> > Baojun, Do you see a difference between getauxval() return and real AT_HWCAP?
> >
> > --
> > Cheers,
> > Carlos.
> >



-- 
H.J.

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[Baojun Wang]

Thanks a lot H.J & Carlos.

On Thu, Oct 17, 2019 at 17:06 H.J. Lu <hjl.tools@gmail.com> wrote:

> On Thu, Oct 17, 2019 at 1:31 PM Baojun Wang <wangbj@gmail.com> wrote:
> >
> > Yes, the values read from /proc/self/auxv is different (Linux x86_64,
> > glibc-2.27, tried glibc HEAD without any difference) with
> > getauxval(AT_HWCAP).
> >
> > Here is the program I wrote for testing:
> >
> > https://gist.github.com/wangbj/658da547437fe9b7eff2113cc9fc9741
> >
> > And here is the output (check AT_HWCAP):
> >
> > $ ./features
> >
> >          AT_BASE: getauxval = 0x7f861434a000    , proc_getauxval =
> > 0x7f861434a000
> >
> >         AT_ENTRY: getauxval = 0x4005c0          , proc_getauxval =
> > 0x4005c0
> >
> >         AT_HWCAP: getauxval = 0x2               , proc_getauxval =
> > 0xbfebfbff
> >
> > Best Regards,
> > baojun
> >
> > On Thu, Oct 17, 2019 at 4:26 PM Carlos O'Donell <carlos@redhat.com>
> wrote:
> > >
> > > On 10/17/19 11:29 AM, Baojun Wang wrote:
> > > > Hi libc,
> > > >
> > > > when calling getauxval(AT_HWCAP), it returns 0x2, because of below
> snippet:
> > > >
> > > >
> https://code.woboq.org/userspace/glibc/sysdeps/x86/cpu-features.c.html#510
> > > >
> > > > Is there any reason why glibc don't use the values passed from
> kernel auxv?
> > >
> > > I don't know of any good reason.
> > >
> > > The semantics of getauxval() is to provide the auxiliary vector data.
> > >
> > > In this case glibc *does* read AT_HWCAP, but then for x86 it overrides
> it
> > > with a computed value based on cpuid.
> > >
> > > HJ, Is there any reason we do this?
>
> dl_hwcap is used to load DSO optimized for a platform.  On i386, we use
> AT_HWCAP passed from kernel.  On x86-64, we use a different approach
> since AT_HWCAP passed from kernel isn't very useful:
>
> commit 1432d38ea04ab5e96f21a382101856db5b49ad8a
> Author: H.J. Lu <hjl.tools@gmail.com>
> Date:   Wed May 3 13:42:42 2017 -0700
>
>     x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]
>
>     dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very
>     early during startup.  They are used by dynamic linker to determine
>     platform and build an array of hardware capability names, which are
>     added to search path when loading shared object.  dl_platform and
>     dl_hwcap are unused on x86-64.  On i386, i386, i486, i586 and i686
>     platforms were supported and only SSE2 capability was used.
>
>     On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and
>     processor capabilities is obsolete since all information is available
>     in dl_x86_cpu_features.  This patch sets dl_platform and dl_hwcap from
>     dl_x86_cpu_features in dynamic linker.  On i386, the available plaforms
>     are changed to i586 and i686 since i386 has been deprecated.  On
> x86-64,
>     the available plaforms are haswell, which is for Haswell class
> processors
>     with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi,
> which
>     is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and
>     AVX512PF.  A capability, avx512_1, is also added to x86-64 for AVX512
>     ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL.
>
> and
>
> commit 45ff34638f034877b6a490c217d6a0632ce263f4
> Author: H.J. Lu <hjl.tools@gmail.com>
> Date:   Mon Sep 11 08:18:11 2017 -0700
>
>     x86: Add x86_64 to x86-64 HWCAP [BZ #22093]
>
>     Before glibc 2.26, ld.so set dl_platform to "x86_64" and searched the
>     "x86_64" subdirectory when loading a shared library.  ld.so in glibc
>     2.26 was changed to set dl_platform to "haswell" or "xeon_phi", based
>     on supported ISAs.  This led to shared library loading failure for
>     shared libraries placed under the "x86_64" subdirectory.
>
>     This patch adds "x86_64" to x86-64 dl_hwcap so that ld.so will always
>     search the "x86_64" subdirectory when loading a shared library.
>
>     NB: We can't set x86-64 dl_platform to "x86-64" since ld.so will skip
>     the "haswell" and "xeon_phi" subdirectories on "haswell" and "xeon_phi"
>     machines.
>
> > > Baojun, Do you see a difference between getauxval() return and real
> AT_HWCAP?
> > >
> > > --
> > > Cheers,
> > > Carlos.
> > >
>
>
>
> --
> H.J.
>

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[Carlos O'Donell]

On 10/17/19 5:06 PM, H.J. Lu wrote:
> dl_hwcap is used to load DSO optimized for a platform.  On i386, we use
> AT_HWCAP passed from kernel.  On x86-64, we use a different approach
> since AT_HWCAP passed from kernel isn't very useful:

The use of dl_hwcap to load DSOs is orthogonal to what getauxval() returns.
They need not be the same.

The question I have is: Why isn't AT_HWCAP returned by getauxval() for x86_64?

We could store AT_HWCAP into dl_hwcap_real, and return that in getauxval()
while still using the *processed* AT_HWCAP in dl_hwcap.

It seems confusing to return an altered AT_AUXVAL from getauxval()?

What do you think?

-- 
Cheers,
Carlos.

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[H.J. Lu]

On Fri, Oct 18, 2019 at 2:24 PM Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 10/17/19 5:06 PM, H.J. Lu wrote:
> > dl_hwcap is used to load DSO optimized for a platform.  On i386, we use
> > AT_HWCAP passed from kernel.  On x86-64, we use a different approach
> > since AT_HWCAP passed from kernel isn't very useful:
>
> The use of dl_hwcap to load DSOs is orthogonal to what getauxval() returns.
> They need not be the same.
>
> The question I have is: Why isn't AT_HWCAP returned by getauxval() for x86_64?
>
> We could store AT_HWCAP into dl_hwcap_real, and return that in getauxval()

This should work.

> while still using the *processed* AT_HWCAP in dl_hwcap.
>
> It seems confusing to return an altered AT_AUXVAL from getauxval()?
>
> What do you think?
>

I have no strong opinion on this.

-- 
H.J.

Re: glibc clears AT_HWCAP and set only HW_CAP_X86_64 (0x2)

[Florian Weimer]

* Carlos O'Donell:

> On 10/17/19 5:06 PM, H.J. Lu wrote:
>> dl_hwcap is used to load DSO optimized for a platform.  On i386, we use
>> AT_HWCAP passed from kernel.  On x86-64, we use a different approach
>> since AT_HWCAP passed from kernel isn't very useful:
>
> The use of dl_hwcap to load DSOs is orthogonal to what getauxval() returns.
> They need not be the same.
>
> The question I have is: Why isn't AT_HWCAP returned by getauxval()
> for x86_64?
>
> We could store AT_HWCAP into dl_hwcap_real, and return that in getauxval()
> while still using the *processed* AT_HWCAP in dl_hwcap.

I think we use GLRO (dl_hwcap) in getauxval so that the return value
of getauxval (AT_HWCAP) is affected by LD_HWCAP_MASK, which is
reasonable as long as GLRO (dl_hwcap) the kernel's AT_HWCAP have the
same meaning.

If we change x86-64, we should add a new symbol version even if the
implementation is the same, so that we can add a compat version if
that proves necessary.  We might need an x86-64-specific replacement
for LD_HWCAP_MASK and another interface to query the flags that
currently land in getauxval (AT_HWCAP), too.