Questions regarding manipulation of IFUNC selection and tunables like glibc.cpu.hwcap_mask

Questions regarding manipulation of IFUNC selection and tunables like glibc.cpu.hwcap_mask

[Stefan Liebler]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

Hi,

on s390x, the IFUNC'ed functions or other hw-dependent code-paths are
usually selected by either the HWCAPs or the facility-list retrieved via
stfle-instruction.

Now we need a possibility to manipulate the IFUNC selection. As the
current IFUNC-resolvers always select the functions for the newest
features, we only need a possibility to disable features.

According to <glibc>/manual/tunables.texi:
@deftp Tunable glibc.cpu.hwcap_mask
This tunable supersedes the @env{LD_HWCAP_MASK} environment variable and is
identical in features.

The @code{AT_HWCAP} key in the Auxiliary Vector specifies instruction set
extensions available in the processor at runtime for some architectures.
 The
@code{glibc.cpu.hwcap_mask} tunable allows the user to mask out those
capabilities at runtime, thus disabling use of those extensions.
@end deftp

But a small testprogram (see attached tst-ifunc-manipulation.c) shows
that neither setting the environment variable
GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0" nor LD_HWCAP_MASK="0x0
influences the HWCAPs.

On s390x, the IFUNC-resolvers get the HWCAPs as argument (the most other
architectures don't have this argument). Other code-paths can get the
HWCAPs via getauxval(AT_HWCAP). In both cases the HWCAPs are loaded from
"GLRO(dl_hwcap)". See:
-
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/s390/dl-irel.h;h=20e4887467d80a1b3f95da00bb98386e3eadfe47;hb=HEAD#l33
-
https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/getauxval.c;h=714ce5bd62ec33c38356b187e6ec067b72b77afb;hb=HEAD#l32


Is it a bug or the intention that the HWCAP values are not influenced by
glibc.cpu.hwcap_mask tunable?

If it is a bug, would it be possible to apply the mask after
__tunables_init() like this:
GLRO(dl_hwcap) &= GET_HWCAP_MASK();
This would affect the IFUNC-resolver, getauxval(AT_HWCAP) and more (e.g.
if lock-elision is available or not) on all architectures. This would
also change the behavior of programs/libraries using getauxval(AT_HWCAP).

As alternative, we could also introduce a new s390-specific tunable
like: glibc.cpu.s390.hwcap_mask which influences only the s390-IFUNCS /
s390-code-pathes within glibc. The behaviour of programs/libraries using
getauxval(AT_HWCAP) is not changed.

Independent of the HWCAPs, we need to introduce a new s390-specific
tunable like glibc.cpu.s390.stfle_mask in order to influence the
s390-IFUNCs within glibc which are dependent on the facility-list.

Are there other hints?

Thanks in advance,
Stefan

[-- Attachment #2: tst-ifunc-manipulation.c --]
[-- Type: text/x-csrc, Size: 1177 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <sys/auxv.h>

/* Run with/without environment variables does not influence HWCAPs:

   GLIBC_TUNABLES=""
   LD_HWCAP_MASK=""
   HWCAPs passed as argument to ifunc-resolver: 0x67ffff
   HWCAPs from getauxval (AT_HWCAP): 0x67ffff

   GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0"
   LD_HWCAP_MASK=""
   HWCAPs passed as argument to ifunc-resolver: 0x67ffff
   HWCAPs from getauxval (AT_HWCAP): 0x67ffff

   GLIBC_TUNABLES=""
   LD_HWCAP_MASK="0x0"
   HWCAPs passed as argument to ifunc-resolver: 0x67ffff
   HWCAPs from getauxval (AT_HWCAP): 0x67ffff
*/

static unsigned long global_hwcaps = 0;

static int impl(void)
{
  printf ("HWCAPs passed as argument to ifunc-resolver: 0x%lx\n",
	  global_hwcaps);
  return 42;
}
static void *resolver(unsigned long hwcap)
{
  global_hwcaps = hwcap;
  return impl;
}
int ifunc(void) __attribute__((ifunc("resolver")));

int
main (void)
{
  printf ("GLIBC_TUNABLES=\"%s\"\n", getenv ("GLIBC_TUNABLES") ? : "");
  printf ("LD_HWCAP_MASK=\"%s\"\n", getenv ("LD_HWCAP_MASK") ? : "");

  ifunc ();
  printf ("HWCAPs from getauxval (AT_HWCAP): 0x%lx\n", getauxval (AT_HWCAP));

  return EXIT_SUCCESS;
}

Re: Questions regarding manipulation of IFUNC selection and tunables like glibc.cpu.hwcap_mask

[Szabolcs Nagy]

The 08/25/2022 11:54, Stefan Liebler via Libc-alpha wrote:
> on s390x, the IFUNC'ed functions or other hw-dependent code-paths are
> usually selected by either the HWCAPs or the facility-list retrieved via
> stfle-instruction.
> 
> Now we need a possibility to manipulate the IFUNC selection. As the
> current IFUNC-resolvers always select the functions for the newest
> features, we only need a possibility to disable features.
> 
> According to <glibc>/manual/tunables.texi:
> @deftp Tunable glibc.cpu.hwcap_mask
> This tunable supersedes the @env{LD_HWCAP_MASK} environment variable and is
> identical in features.
> 
> The @code{AT_HWCAP} key in the Auxiliary Vector specifies instruction set
> extensions available in the processor at runtime for some architectures.
>  The
> @code{glibc.cpu.hwcap_mask} tunable allows the user to mask out those
> capabilities at runtime, thus disabling use of those extensions.
> @end deftp

iirc LD_HWCAP_MASK was purely for the old hwcap based
lib path lookup.

> But a small testprogram (see attached tst-ifunc-manipulation.c) shows
> that neither setting the environment variable
> GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0" nor LD_HWCAP_MASK="0x0
> influences the HWCAPs.
> 
> On s390x, the IFUNC-resolvers get the HWCAPs as argument (the most other
> architectures don't have this argument). Other code-paths can get the
> HWCAPs via getauxval(AT_HWCAP). In both cases the HWCAPs are loaded from
> "GLRO(dl_hwcap)". See:

i think hwcap_mask was not for completely hiding hwcaps
like that.

maybe we need a mechanism for hiding too (can be useful
for testing), however since it is in userspace, the kernel
and hw will behave as if HWCAP is present which may be
observable even if the libc tries to hide it.

also if this is useful then we need hwcap2 mask too,
and allow using hwcap names like "glibc.cpu.hwcaps" does
on x86.



> -
> https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/s390/dl-irel.h;h=20e4887467d80a1b3f95da00bb98386e3eadfe47;hb=HEAD#l33
> -
> https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/getauxval.c;h=714ce5bd62ec33c38356b187e6ec067b72b77afb;hb=HEAD#l32
> 
> 
> Is it a bug or the intention that the HWCAP values are not influenced by
> glibc.cpu.hwcap_mask tunable?
> 
> If it is a bug, would it be possible to apply the mask after
> __tunables_init() like this:
> GLRO(dl_hwcap) &= GET_HWCAP_MASK();
> This would affect the IFUNC-resolver, getauxval(AT_HWCAP) and more (e.g.
> if lock-elision is available or not) on all architectures. This would
> also change the behavior of programs/libraries using getauxval(AT_HWCAP).
> 
> As alternative, we could also introduce a new s390-specific tunable
> like: glibc.cpu.s390.hwcap_mask which influences only the s390-IFUNCS /
> s390-code-pathes within glibc. The behaviour of programs/libraries using
> getauxval(AT_HWCAP) is not changed.
> 
> Independent of the HWCAPs, we need to introduce a new s390-specific
> tunable like glibc.cpu.s390.stfle_mask in order to influence the
> s390-IFUNCs within glibc which are dependent on the facility-list.
> 
> Are there other hints?
> 
> Thanks in advance,
> Stefan

> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/auxv.h>
> 
> /* Run with/without environment variables does not influence HWCAPs:
> 
>    GLIBC_TUNABLES=""
>    LD_HWCAP_MASK=""
>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
> 
>    GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0"
>    LD_HWCAP_MASK=""
>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
> 
>    GLIBC_TUNABLES=""
>    LD_HWCAP_MASK="0x0"
>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
> */
> 
> static unsigned long global_hwcaps = 0;
> 
> static int impl(void)
> {
>   printf ("HWCAPs passed as argument to ifunc-resolver: 0x%lx\n",
> 	  global_hwcaps);
>   return 42;
> }
> static void *resolver(unsigned long hwcap)
> {
>   global_hwcaps = hwcap;
>   return impl;
> }
> int ifunc(void) __attribute__((ifunc("resolver")));
> 
> int
> main (void)
> {
>   printf ("GLIBC_TUNABLES=\"%s\"\n", getenv ("GLIBC_TUNABLES") ? : "");
>   printf ("LD_HWCAP_MASK=\"%s\"\n", getenv ("LD_HWCAP_MASK") ? : "");
> 
>   ifunc ();
>   printf ("HWCAPs from getauxval (AT_HWCAP): 0x%lx\n", getauxval (AT_HWCAP));
> 
>   return EXIT_SUCCESS;
> }

Re: Questions regarding manipulation of IFUNC selection and tunables like glibc.cpu.hwcap_mask

[Stefan Liebler]

On 25/08/2022 12:47, Szabolcs Nagy wrote:
> The 08/25/2022 11:54, Stefan Liebler via Libc-alpha wrote:
>> on s390x, the IFUNC'ed functions or other hw-dependent code-paths are
>> usually selected by either the HWCAPs or the facility-list retrieved via
>> stfle-instruction.
>>
>> Now we need a possibility to manipulate the IFUNC selection. As the
>> current IFUNC-resolvers always select the functions for the newest
>> features, we only need a possibility to disable features.
>>
>> According to <glibc>/manual/tunables.texi:
>> @deftp Tunable glibc.cpu.hwcap_mask
>> This tunable supersedes the @env{LD_HWCAP_MASK} environment variable and is
>> identical in features.
>>
>> The @code{AT_HWCAP} key in the Auxiliary Vector specifies instruction set
>> extensions available in the processor at runtime for some architectures.
>>  The
>> @code{glibc.cpu.hwcap_mask} tunable allows the user to mask out those
>> capabilities at runtime, thus disabling use of those extensions.
>> @end deftp
> 
> iirc LD_HWCAP_MASK was purely for the old hwcap based
> lib path lookup.
Okay. Thanks for clarification. This also fits to what I've seen in the
sources.
> 
>> But a small testprogram (see attached tst-ifunc-manipulation.c) shows
>> that neither setting the environment variable
>> GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0" nor LD_HWCAP_MASK="0x0
>> influences the HWCAPs.
>>
>> On s390x, the IFUNC-resolvers get the HWCAPs as argument (the most other
>> architectures don't have this argument). Other code-paths can get the
>> HWCAPs via getauxval(AT_HWCAP). In both cases the HWCAPs are loaded from
>> "GLRO(dl_hwcap)". See:
> 
> i think hwcap_mask was not for completely hiding hwcaps
> like that.
Then at least for me, the description of glibc.cpu.hwcap_mask-tunables
is misleading.

> 
> maybe we need a mechanism for hiding too (can be useful
> for testing), however since it is in userspace, the kernel
> and hw will behave as if HWCAP is present which may be
> observable even if the libc tries to hide it.
Yes, you are right, the kernel/hw will not know anything about it.
Somebody could just read the flags from /proc/cpuinfo,
or other details from architecture-specific instructions.
And even glibc would use at least the features which comes indirectly
from the used architecture-level-set even if you mask out older ones.

Thus you think just masking bits in GLRO(dl_hwcap) which would also
influence getauxval(AT_HWCAP) is a bad idea?

> 
> also if this is useful then we need hwcap2 mask too,
Makes sense.

> and allow using hwcap names like "glibc.cpu.hwcaps" does
> on x86.
For my purpose of manipulating the s390-glibc-internal
IFUNCs/code-paths, I could implement a s390 version of
"glibc.cpu.hwcaps" which allows to influence the selectors based on
AT_HWCAP and stfle-bits in one tunable. As soon as AT_HWCAP2 is used on
s390x, this tunables can also influence those features.
For s390 it would also make sense to be able to set a baseline
architecture-level and then enable further features like:
GLIBC_TUNABLES="glibc.cpu.hwcaps=z14,vxe2"

Of course the s390 version of "glibc.cpu.hwcaps" would not influence
other libs/applications which are using getauxval(AT_HWCAP).

What do you think about adding such a s390 version of "glibc.cpu.hwcaps"
and reworking the s390-specific-selectors to use a special
features-struct instead of GLRO(dl_hwcap)?
> 
> 
> 
>> -
>> https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/s390/dl-irel.h;h=20e4887467d80a1b3f95da00bb98386e3eadfe47;hb=HEAD#l33
>> -
>> https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/getauxval.c;h=714ce5bd62ec33c38356b187e6ec067b72b77afb;hb=HEAD#l32
>>
>>
>> Is it a bug or the intention that the HWCAP values are not influenced by
>> glibc.cpu.hwcap_mask tunable?
>>
>> If it is a bug, would it be possible to apply the mask after
>> __tunables_init() like this:
>> GLRO(dl_hwcap) &= GET_HWCAP_MASK();
>> This would affect the IFUNC-resolver, getauxval(AT_HWCAP) and more (e.g.
>> if lock-elision is available or not) on all architectures. This would
>> also change the behavior of programs/libraries using getauxval(AT_HWCAP).
>>
>> As alternative, we could also introduce a new s390-specific tunable
>> like: glibc.cpu.s390.hwcap_mask which influences only the s390-IFUNCS /
>> s390-code-pathes within glibc. The behaviour of programs/libraries using
>> getauxval(AT_HWCAP) is not changed.
>>
>> Independent of the HWCAPs, we need to introduce a new s390-specific
>> tunable like glibc.cpu.s390.stfle_mask in order to influence the
>> s390-IFUNCs within glibc which are dependent on the facility-list.
>>
>> Are there other hints?
>>
>> Thanks in advance,
>> Stefan
> 
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <sys/auxv.h>
>>
>> /* Run with/without environment variables does not influence HWCAPs:
>>
>>    GLIBC_TUNABLES=""
>>    LD_HWCAP_MASK=""
>>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
>>
>>    GLIBC_TUNABLES="glibc.cpu.hwcap_mask=0"
>>    LD_HWCAP_MASK=""
>>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
>>
>>    GLIBC_TUNABLES=""
>>    LD_HWCAP_MASK="0x0"
>>    HWCAPs passed as argument to ifunc-resolver: 0x67ffff
>>    HWCAPs from getauxval (AT_HWCAP): 0x67ffff
>> */
>>
>> static unsigned long global_hwcaps = 0;
>>
>> static int impl(void)
>> {
>>   printf ("HWCAPs passed as argument to ifunc-resolver: 0x%lx\n",
>> 	  global_hwcaps);
>>   return 42;
>> }
>> static void *resolver(unsigned long hwcap)
>> {
>>   global_hwcaps = hwcap;
>>   return impl;
>> }
>> int ifunc(void) __attribute__((ifunc("resolver")));
>>
>> int
>> main (void)
>> {
>>   printf ("GLIBC_TUNABLES=\"%s\"\n", getenv ("GLIBC_TUNABLES") ? : "");
>>   printf ("LD_HWCAP_MASK=\"%s\"\n", getenv ("LD_HWCAP_MASK") ? : "");
>>
>>   ifunc ();
>>   printf ("HWCAPs from getauxval (AT_HWCAP): 0x%lx\n", getauxval (AT_HWCAP));
>>
>>   return EXIT_SUCCESS;
>> }
>