Re: Questions regarding manipulation of IFUNC selection and tunables like glibc.cpu.hwcap_mask

[Stefan Liebler <stli@linux.ibm.com>]

On 25/08/2022 12:47, Szabolcs Nagy wrote:
> The 08/25/2022 11:54, Stefan Liebler via Libc-alpha wrote:
>> on s390x, the IFUNC'ed functions or other hw-dependent code-paths are
>> usually selected by either the HWCAPs or the facility-list retrieved via
>> stfle-instruction.
>>
>> Now we need a possibility to manipulate the IFUNC selection. As the
>> current IFUNC-resolvers always select the functions for the newest
>> features, we only need a possibility to disable features.
>>
>> According to <glibc>/manual/tunables.texi:
>> @deftp Tunable glibc.cpu.hwcap_mask
>> This tunable supersedes the @env{LD_HWCAP_MASK} environment variable and is
>> identical in features.
>>
>> The @code{AT_HWCAP} key in the Auxiliary Vector specifies instruction set
>> extensions available in the processor at runtime for some architectures.
>>  The
>> @code{glibc.cpu.hwcap_mask} tunable allows the user to mask out those
>> capabilities at runtime, thus disabling use of those extensions.
>> @end deftp
> 
> iirc LD_HWCAP_MASK was purely for the old hwcap based
> lib path lookup.
Okay. Thanks for clarification. This also fits to what I've seen in the
sources.
> 
>> But a small testprogram (see attached tst-ifunc-manipulation.c) shows
>> that neither setting the environment variable
>> GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0" nor LD_HWCAP_MASK="0x0
>> influences the HWCAPs.
>>
>> On s390x, the IFUNC-resolvers get the HWCAPs as argument (the most other
>> architectures don't have this argument). Other code-paths can get the
>> HWCAPs via getauxval(AT_HWCAP). In both cases the HWCAPs are loaded from
>> "GLRO(dl_hwcap)". See:
> 
> i think hwcap_mask was not for completely hiding hwcaps
> like that.
Then at least for me, the description of glibc.cpu.hwcap_mask-tunables
is misleading.

> 
> maybe we need a mechanism for hiding too (can be useful
> for testing), however since it is in userspace, the kernel
> and hw will behave as if HWCAP is present which may be
> observable even if the libc tries to hide it.
Yes, you are right, the kernel/hw will not know anything about it.
Somebody could just read the flags from /proc/cpuinfo,
or other details from architecture-specific instructions.
And even glibc would use at least the features which comes indirectly
from the used architecture-level-set even if you mask out older ones.

Thus you think just masking bits in GLRO(dl_hwcap) which would also
influence getauxval(AT_HWCAP) is a bad idea?

> 
> also if this is useful then we need hwcap2 mask too,
Makes sense.

> and allow using hwcap names like "glibc.cpu.hwcaps" does
> on x86.
For my purpose of manipulating the s390-glibc-internal
IFUNCs/code-paths, I could implement a s390 version of
"glibc.cpu.hwcaps" which allows to influence the selectors based on
AT_HWCAP and stfle-bits in one tunable. As soon as AT_HWCAP2 is used on
s390x, this tunables can also influence those features.
For s390 it would also make sense to be able to set a baseline
architecture-level and then enable further features like:
GLIBC_TUNABLES="glibc.cpu.hwcaps=z14,vxe2"

Of course the s390 version of "glibc.cpu.hwcaps" would not influence
other libs/applications which are using getauxval(AT_HWCAP).

What do you think about adding such a s390 version of "glibc.cpu.hwcaps"
and reworking the s390-specific-selectors to use a special
features-struct instead of GLRO(dl_hwcap)?
> 
> 
> 
>> -
>> https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/s390/dl-irel.h;h=20e4887467d80a1b3f95da00bb98386e3eadfe47;hb=HEAD#l33
>> -
>> https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/getauxval.c;h=714ce5bd62ec33c38356b187e6ec067b72b77afb;hb=HEAD#l32
>>
>>
>> Is it a bug or the intention that the HWCAP values are not influenced by
>> glibc.cpu.hwcap_mask tunable?
>>
>> If it is a bug, would it be possible to apply the mask after
>> __tunables_init() like this:
>> GLRO(dl_hwcap) &= GET_HWCAP_MASK();
>> This would affect the IFUNC-resolver, getauxval(AT_HWCAP) and more (e.g.
>> if lock-elision is available or not) on all architectures. This would
>> also change the behavior of programs/libraries using getauxval(AT_HWCAP).
>>
>> As alternative, we could also introduce a new s390-specific tunable
>> like: glibc.cpu.s390.hwcap_mask which influences only the s390-IFUNCS /
>> s390-code-pathes within glibc. The behaviour of programs/libraries using
>> getauxval(AT_HWCAP) is not changed.
>>
>> Independent of the HWCAPs, we need to introduce a new s390-specific
>> tunable like glibc.cpu.s390.stfle_mask in order to influence the
>> s390-IFUNCs within glibc which are dependent on the facility-list.
>>
>> Are there other hints?
>>
>> Thanks in advance,
>> Stefan
> 
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <sys/auxv.h>
>>
>> /* Run with/without environment variables does not influence HWCAPs:
>>
>>    GLIBC_TUNABLES=""
>>    LD_HWCAP_MASK=""
>>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
>>
>>    GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0"
>>    LD_HWCAP_MASK=""
>>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
>>
>>    GLIBC_TUNABLES=""
>>    LD_HWCAP_MASK="0x0"
>>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
>> */
>>
>> static unsigned long global_hwcaps = 0;
>>
>> static int impl(void)
>> {
>>   printf ("HWCAPs passed as argument to ifunc-resolver: 0x%lx\n",
>> 	  global_hwcaps);
>>   return 42;
>> }
>> static void *resolver(unsigned long hwcap)
>> {
>>   global_hwcaps = hwcap;
>>   return impl;
>> }
>> int ifunc(void) __attribute__((ifunc("resolver")));
>>
>> int
>> main (void)
>> {
>>   printf ("GLIBC_TUNABLES=\"%s\"\n", getenv ("GLIBC_TUNABLES") ? : "");
>>   printf ("LD_HWCAP_MASK=\"%s\"\n", getenv ("LD_HWCAP_MASK") ? : "");
>>
>>   ifunc ();
>>   printf ("HWCAPs from getauxval (AT_HWCAP): 0x%lx\n", getauxval (AT_HWCAP));
>>
>>   return EXIT_SUCCESS;
>> }
>