Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Dr. Todor Dimitrov]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

Hello,

we’ve recently updated an `aarch64` system to Glibc 2.31 and we’ve started seeing a lot of failing 403 and 407 syscalls. The kernel version is 4.1.52 and the userland is 32-bit. Looking at the implementation of `__clock_gettime64`, it seems that we are hitting the following problem:

  /* Old 32-bit ABI with possible 64-bit time_t support.  */
# ifdef __NR_clock_gettime64
  /* Avoid issue a __NR_clock_gettime64 syscall on kernels that do not
     support 64-bit time_t.  */
  static int time64_support = 1;
  if (atomic_load_relaxed (&time64_support) != 0)
    {
#  ifdef HAVE_CLOCK_GETTIME64_VSYSCALL
      r = INLINE_VSYSCALL (clock_gettime64, 2, clock_id, tp);
#  else
      r = INLINE_SYSCALL_CALL (clock_gettime64, clock_id, tp);
#  endif
      if (r == 0 || errno != ENOSYS)
	return r;

      atomic_store_relaxed (&time64_support, 0);
    }
# endif

`__NR_clock_gettime64` is defined for the 32-bit `arm` architecture but not for `aarch64`. Is this a known issue? What would be the best way to overcome it?

Thanks in advance,
Todor


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3895 bytes --]

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Florian Weimer]

* Todor Dimitrov:

> we’ve recently updated an `aarch64` system to Glibc 2.31 and we’ve
> started seeing a lot of failing 403 and 407 syscalls. The kernel
> version is 4.1.52 and the userland is 32-bit. Looking at the
> implementation of `__clock_gettime64`, it seems that we are hitting
> the following problem:

Are you sure this is an aarch64 system?

> `__NR_clock_gettime64` is defined for the 32-bit `arm` architecture
> but not for `aarch64`. Is this a known issue? What would be the best
> way to overcome it?

Upgrade your kernel to something that has 64-bit time_t support.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Dr. Todor Dimitrov]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

Yes:

uname -a
Linux  4.1.52 #1 SMP PREEMPT Mon Jan 4 13:54:08 CST 2021 aarch64 GNU/Linux

Unfortunately updating the kernel is not an option as it would require a new (not existing) BSP package. The only other option would be downgrading Glibc?

Thanks, Todor

> On 18. Jan 2021, at 18:51, Florian Weimer <fweimer@redhat.com> wrote:
> 
> * Todor Dimitrov:
> 
>> we’ve recently updated an `aarch64` system to Glibc 2.31 and we’ve
>> started seeing a lot of failing 403 and 407 syscalls. The kernel
>> version is 4.1.52 and the userland is 32-bit. Looking at the
>> implementation of `__clock_gettime64`, it seems that we are hitting
>> the following problem:
> 
> Are you sure this is an aarch64 system?
> 
>> `__NR_clock_gettime64` is defined for the 32-bit `arm` architecture
>> but not for `aarch64`. Is this a known issue? What would be the best
>> way to overcome it?
> 
> Upgrade your kernel to something that has 64-bit time_t support.
> 
> Thanks,
> Florian
> -- 
> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
> 


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3895 bytes --]

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Adhemerval Zanella]

On 18/01/2021 14:43, Dr. Todor Dimitrov wrote:
> Hello,
> 
> we’ve recently updated an `aarch64` system to Glibc 2.31 and we’ve started seeing a lot of failing 403 and 407 syscalls. The kernel version is 4.1.52 and the userland is 32-bit. Looking at the implementation of `__clock_gettime64`, it seems that we are hitting the following problem:
> 
>   /* Old 32-bit ABI with possible 64-bit time_t support.  */
> # ifdef __NR_clock_gettime64
>   /* Avoid issue a __NR_clock_gettime64 syscall on kernels that do not
>      support 64-bit time_t.  */
>   static int time64_support = 1;
>   if (atomic_load_relaxed (&time64_support) != 0)
>     {
> #  ifdef HAVE_CLOCK_GETTIME64_VSYSCALL
>       r = INLINE_VSYSCALL (clock_gettime64, 2, clock_id, tp);
> #  else
>       r = INLINE_SYSCALL_CALL (clock_gettime64, clock_id, tp);
> #  endif
>       if (r == 0 || errno != ENOSYS)
> 	return r;
> 
>       atomic_store_relaxed (&time64_support, 0);
>     }
> # endif
> 
> `__NR_clock_gettime64` is defined for the 32-bit `arm` architecture but not for `aarch64`. Is this a known issue? What would be the best way to overcome it?

This is the expected behavior, running this small example:

--
#include <time.h>

int main (int argc, char *argv[])
{
  clock_gettime (CLOCK_REALTIME, &((struct timespec) {}));
  clock_gettime (CLOCK_BOOTTIME, &((struct timespec) {}));
  return 0;
}
--

On a aarch64 kernel without 64-bit compat time_t support (4.12.13),
this is the generated syscalls:

--
clock_gettime64(CLOCK_REALTIME, 0xfffebb58) = -1 ENOSYS (Function not implemented)
clock_gettime(CLOCK_REALTIME, {tv_sec=1610997887, tv_nsec=516302599}) = 0
clock_gettime(CLOCK_BOOTTIME, {tv_sec=2750059, tv_nsec=271773400}) = 0
--

The catch here the 64-bit kernel when running 32-bit userland will
use compat entrypoints to provide the expected kernel ABI. For kernel 
older than v5.1, it means that even if kernel is 64-bit and support 
64-bit time_t, the 32-bit syscall entrypoints won't have 64-bit time_t
support.

The glibc Linux clock_gettime thus probes by first issuing the new
syscall with provides 64-bit time_t support and then falling back
to older in case kernel does not support it.

For clock_gettime and a some other handfull implementations, it would
incur in only one ENOSYS.  However there are a bunch of implementations
that do not have this optimization.

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Dr. Todor Dimitrov]

[-- Attachment #1: Type: text/plain, Size: 2655 bytes --]

OK, thank you for the detailed explanation.

> On 18. Jan 2021, at 20:31, Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote:
> 
> 
> 
> On 18/01/2021 14:43, Dr. Todor Dimitrov wrote:
>> Hello,
>> 
>> we’ve recently updated an `aarch64` system to Glibc 2.31 and we’ve started seeing a lot of failing 403 and 407 syscalls. The kernel version is 4.1.52 and the userland is 32-bit. Looking at the implementation of `__clock_gettime64`, it seems that we are hitting the following problem:
>> 
>>  /* Old 32-bit ABI with possible 64-bit time_t support.  */
>> # ifdef __NR_clock_gettime64
>>  /* Avoid issue a __NR_clock_gettime64 syscall on kernels that do not
>>     support 64-bit time_t.  */
>>  static int time64_support = 1;
>>  if (atomic_load_relaxed (&time64_support) != 0)
>>    {
>> #  ifdef HAVE_CLOCK_GETTIME64_VSYSCALL
>>      r = INLINE_VSYSCALL (clock_gettime64, 2, clock_id, tp);
>> #  else
>>      r = INLINE_SYSCALL_CALL (clock_gettime64, clock_id, tp);
>> #  endif
>>      if (r == 0 || errno != ENOSYS)
>> 	return r;
>> 
>>      atomic_store_relaxed (&time64_support, 0);
>>    }
>> # endif
>> 
>> `__NR_clock_gettime64` is defined for the 32-bit `arm` architecture but not for `aarch64`. Is this a known issue? What would be the best way to overcome it?
> 
> This is the expected behavior, running this small example:
> 
> --
> #include <time.h>
> 
> int main (int argc, char *argv[])
> {
>  clock_gettime (CLOCK_REALTIME, &((struct timespec) {}));
>  clock_gettime (CLOCK_BOOTTIME, &((struct timespec) {}));
>  return 0;
> }
> --
> 
> On a aarch64 kernel without 64-bit compat time_t support (4.12.13),
> this is the generated syscalls:
> 
> --
> clock_gettime64(CLOCK_REALTIME, 0xfffebb58) = -1 ENOSYS (Function not implemented)
> clock_gettime(CLOCK_REALTIME, {tv_sec=1610997887, tv_nsec=516302599}) = 0
> clock_gettime(CLOCK_BOOTTIME, {tv_sec=2750059, tv_nsec=271773400}) = 0
> --
> 
> The catch here the 64-bit kernel when running 32-bit userland will
> use compat entrypoints to provide the expected kernel ABI. For kernel 
> older than v5.1, it means that even if kernel is 64-bit and support 
> 64-bit time_t, the 32-bit syscall entrypoints won't have 64-bit time_t
> support.
> 
> The glibc Linux clock_gettime thus probes by first issuing the new
> syscall with provides 64-bit time_t support and then falling back
> to older in case kernel does not support it.
> 
> For clock_gettime and a some other handfull implementations, it would
> incur in only one ENOSYS.  However there are a bunch of implementations
> that do not have this optimization.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3895 bytes --]

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Florian Weimer]

* Adhemerval Zanella via Libc-help:

> This is the expected behavior, running this small example:
>
> --
> #include <time.h>
>
> int main (int argc, char *argv[])
> {
>   clock_gettime (CLOCK_REALTIME, &((struct timespec) {}));
>   clock_gettime (CLOCK_BOOTTIME, &((struct timespec) {}));
>   return 0;
> }
> --
>
> On a aarch64 kernel without 64-bit compat time_t support (4.12.13),
> this is the generated syscalls:
>
> --
> clock_gettime64(CLOCK_REALTIME, 0xfffebb58) = -1 ENOSYS (Function not implemented)
> clock_gettime(CLOCK_REALTIME, {tv_sec=1610997887, tv_nsec=516302599}) = 0
> clock_gettime(CLOCK_BOOTTIME, {tv_sec=2750059, tv_nsec=271773400}) = 0
> --
>
> The catch here the 64-bit kernel when running 32-bit userland will
> use compat entrypoints to provide the expected kernel ABI. For kernel 
> older than v5.1, it means that even if kernel is 64-bit and support 
> 64-bit time_t, the 32-bit syscall entrypoints won't have 64-bit time_t
> support.

Ugh.  Is this for ILP32 support?

Should we just undefined __NR_clock_gettime64 for 64-bit AArch64?

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Adhemerval Zanella]

On 18/01/2021 17:14, Florian Weimer wrote:
> * Adhemerval Zanella via Libc-help:
> 
>> This is the expected behavior, running this small example:
>>
>> --
>> #include <time.h>
>>
>> int main (int argc, char *argv[])
>> {
>>   clock_gettime (CLOCK_REALTIME, &((struct timespec) {}));
>>   clock_gettime (CLOCK_BOOTTIME, &((struct timespec) {}));
>>   return 0;
>> }
>> --
>>
>> On a aarch64 kernel without 64-bit compat time_t support (4.12.13),
>> this is the generated syscalls:
>>
>> --
>> clock_gettime64(CLOCK_REALTIME, 0xfffebb58) = -1 ENOSYS (Function not implemented)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1610997887, tv_nsec=516302599}) = 0
>> clock_gettime(CLOCK_BOOTTIME, {tv_sec=2750059, tv_nsec=271773400}) = 0
>> --
>>
>> The catch here the 64-bit kernel when running 32-bit userland will
>> use compat entrypoints to provide the expected kernel ABI. For kernel 
>> older than v5.1, it means that even if kernel is 64-bit and support 
>> 64-bit time_t, the 32-bit syscall entrypoints won't have 64-bit time_t
>> support.
> 
> Ugh.  Is this for ILP32 support?
> 
> Should we just undefined __NR_clock_gettime64 for 64-bit AArch64?

No, this is armhf default binaries.  This situation is similar to
i686 running on x86_64 kernels.

AArch64 (and any other architecture with default 64-bit time_t)
will build Linux clock_gettime.c as:

int
__clock_gettime64 (clockid_t clock_id, struct __timespec64 *tp)
{
  int r;
# define __NR_clock_gettime64 __NR_clock_gettime
  if (true)
    {
      r = INLINE_VSYSCALL (clock_gettime64, 2, clock_id, tp);
      if (r == 0 || errno != ENOSYS)
        return r;
    }
  return r;
}

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Florian Weimer]

* Adhemerval Zanella:

>> Should we just undefined __NR_clock_gettime64 for 64-bit AArch64?
>
> No, this is armhf default binaries.  This situation is similar to
> i686 running on x86_64 kernels.

It sure looks like it.  I wasn't aware that it's possible to run armhfp
userspace on an AArch64 kernel.

What kind of hardware is needed for that?

(For some reason, Fedora builds its armhfp distribution in a virtualized
environment.)

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Adhemerval Zanella]

On 18/01/2021 17:35, Florian Weimer wrote:
> * Adhemerval Zanella:
> 
>>> Should we just undefined __NR_clock_gettime64 for 64-bit AArch64?
>>
>> No, this is armhf default binaries.  This situation is similar to
>> i686 running on x86_64 kernels.
> 
> It sure looks like it.  I wasn't aware that it's possible to run armhfp
> userspace on an AArch64 kernel.
> 
> What kind of hardware is needed for that?
> 
> (For some reason, Fedora builds its armhfp distribution in a virtualized
> environment.)

You need a hardware that supports ARMv8 on EL0, was described by 
Armv8-A architecture profile (page D13-3255):

EL3, bits [15:12]
          EL3 Exception level handling. Defined values are:

          0b0000      EL3 is not implemented.
          0b0001      EL3 can be executed in AArch64 state only.
          0b0010      EL3 can be executed in either AArch64 or AArch32 state.

          All other values are reserved.

EL2, bits [11:8]
          EL2 Exception level handling. Defined values are:

          0b0000      EL2 is not implemented.
          0b0001      EL2 can be executed in AArch64 state only.
          0b0010      EL2 can be executed in either AArch64 or AArch32 state.

          All other values are reserved.

EL1, bits [7:4]
          EL1 Exception level handling. Defined values are:

          0b0001      EL1 can be executed in AArch64 state only.
          0b0010      EL1 can be executed in either AArch64 or AArch32 state.

          All other values are reserved.

EL0, bits [3:0]
          EL0 Exception level handling. Defined values are:

          0b0001      EL0 can be executed in AArch64 state only.
          0b0010      EL0 can be executed in either AArch64 or AArch32 state.

          All other values are reserved.

And unfortunately there is no way to access this special register without
kernel support [2] even with 'cpuid' support (HWCAP_CPUID from hwcap) [3]
this information is masked away due security issues.

I don't know a easy way to find out without actually issuing a arm32 binary 
to find out if the chip does support A32 and T32 state.

The kernel also limits the compat syscall mechanism (A32 state running on
arm64 chip):

arch/arm64/Kconfig

1213 menuconfig COMPAT
1214         bool "Kernel support for 32-bit EL0"
1215         depends on ARM64_4K_PAGES || EXPERT
1216         select COMPAT_BINFMT_ELF if BINFMT_ELF
1217         select HAVE_UID16
1218         select OLD_SIGSUSPEND3
1219         select COMPAT_OLD_SIGACTION
1220         help
1221           This option enables support for a 32-bit EL0 running under a 64-bit
1222           kernel at EL1. AArch32-specific components such as system calls,
1223           the user helper functions, VFP support and the ptrace interface are
1224           handled appropriately by the kernel.
1225 
1226           If you use a page size other than 4KB (i.e, 16KB or 64KB), please be aware
1227           that you will only be able to execute AArch32 binaries that were compiled
1228           with page size aligned segments.

[1] https://developer.arm.com/documentation/ddi0487/latest/ 
[2] https://stackoverflow.com/questions/60071590/reading-armv8-a-registers-with-devmem-from-gnu-linux-shell
[3] https://www.kernel.org/doc/html/latest/arm64/cpu-feature-registers.html

Re: Glibc 2.31 - time64 with 64-bit kernel and 32-bit userland

[Adhemerval Zanella]

On 18/01/2021 18:22, Adhemerval Zanella wrote:
> 
> 
> On 18/01/2021 17:35, Florian Weimer wrote:
>> * Adhemerval Zanella:
>>
>>>> Should we just undefined __NR_clock_gettime64 for 64-bit AArch64?
>>>
>>> No, this is armhf default binaries.  This situation is similar to
>>> i686 running on x86_64 kernels.
>>
>> It sure looks like it.  I wasn't aware that it's possible to run armhfp
>> userspace on an AArch64 kernel.
>>
>> What kind of hardware is needed for that?
>>
>> (For some reason, Fedora builds its armhfp distribution in a virtualized
>> environment.)
> 
> You need a hardware that supports ARMv8 on EL0, was described by 
> Armv8-A architecture profile (page D13-3255):
> 
> EL3, bits [15:12]
>           EL3 Exception level handling. Defined values are:
> 
>           0b0000      EL3 is not implemented.
>           0b0001      EL3 can be executed in AArch64 state only.
>           0b0010      EL3 can be executed in either AArch64 or AArch32 state.
> 
>           All other values are reserved.
> 
> EL2, bits [11:8]
>           EL2 Exception level handling. Defined values are:
> 
>           0b0000      EL2 is not implemented.
>           0b0001      EL2 can be executed in AArch64 state only.
>           0b0010      EL2 can be executed in either AArch64 or AArch32 state.
> 
>           All other values are reserved.
> 
> EL1, bits [7:4]
>           EL1 Exception level handling. Defined values are:
> 
>           0b0001      EL1 can be executed in AArch64 state only.
>           0b0010      EL1 can be executed in either AArch64 or AArch32 state.
> 
>           All other values are reserved.
> 
> EL0, bits [3:0]
>           EL0 Exception level handling. Defined values are:
> 
>           0b0001      EL0 can be executed in AArch64 state only.
>           0b0010      EL0 can be executed in either AArch64 or AArch32 state.
> 
>           All other values are reserved.
> 
> And unfortunately there is no way to access this special register without
> kernel support [2] even with 'cpuid' support (HWCAP_CPUID from hwcap) [3]
> this information is masked away due security issues.
> 
> I don't know a easy way to find out without actually issuing a arm32 binary 
> to find out if the chip does support A32 and T32 state.
> 
> The kernel also limits the compat syscall mechanism (A32 state running on
> arm64 chip):
> 
> arch/arm64/Kconfig
> 
> 1213 menuconfig COMPAT
> 1214         bool "Kernel support for 32-bit EL0"
> 1215         depends on ARM64_4K_PAGES || EXPERT
> 1216         select COMPAT_BINFMT_ELF if BINFMT_ELF
> 1217         select HAVE_UID16
> 1218         select OLD_SIGSUSPEND3
> 1219         select COMPAT_OLD_SIGACTION
> 1220         help
> 1221           This option enables support for a 32-bit EL0 running under a 64-bit
> 1222           kernel at EL1. AArch32-specific components such as system calls,
> 1223           the user helper functions, VFP support and the ptrace interface are
> 1224           handled appropriately by the kernel.
> 1225 
> 1226           If you use a page size other than 4KB (i.e, 16KB or 64KB), please be aware
> 1227           that you will only be able to execute AArch32 binaries that were compiled
> 1228           with page size aligned segments.
> 
> [1] https://developer.arm.com/documentation/ddi0487/latest/ 
> [2] https://stackoverflow.com/questions/60071590/reading-armv8-a-registers-with-devmem-from-gnu-linux-shell
> [3] https://www.kernel.org/doc/html/latest/arm64/cpu-feature-registers.html
> 

In fact checking with ARM guys there is a way to find it out on Linux:

 * if the personality(PER_LINUX32) syscall succeeds from an AArch64
   process, then 32-bit process support is present

From the commandline:
 * run 'lscpu': if its output includes the line
      CPU op-mode(s): 32-bit, 64-bit
   then both modes are supported; if it says
      CPU op-mode(s): 64-bit
   then only 64-bit is supported.
   If it says neither then the 'lscpu' binary is too old 
   (lscpu is running "personality(PER_LINUX32)" to find this info.)