aarch64 lacks HP_TIMING support in glibc

aarch64 lacks HP_TIMING support in glibc

[Rafael Eduardo Ruviaro Christ]

Hello,

I noticed that aarch64 lacks HP_TIMING support in glibc. Although this architecture has generic timers, glibc is still using the generic hp-timing.h for aarch64.
I see HP_TIMING support for other architectures. I'd like to know why there is no HP_TIMING support for aarch64?
Also, if there's planning to support it for aarch64?


Thank you,
Kind regards,
Rafael
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.

Re: aarch64 lacks HP_TIMING support in glibc

[Andrew Pinski]

On Tue, Jun 22, 2021 at 11:30 AM Rafael Eduardo Ruviaro Christ via
Libc-alpha <libc-alpha@sourceware.org> wrote:
>
> Hello,
>
> I noticed that aarch64 lacks HP_TIMING support in glibc. Although this architecture has generic timers, glibc is still using the generic hp-timing.h for aarch64.
> I see HP_TIMING support for other architectures. I'd like to know why there is no HP_TIMING support for aarch64?
> Also, if there's planning to support it for aarch64?

See https://sourceware.org/legacy-ml/libc-alpha/2015-06/msg00736.html
and the previous emails in the thread.

Thanks,
Andrew Pinski

>
>
> Thank you,
> Kind regards,
> Rafael
> -- The information ....


This disclaimer is useless for public lists, remove it.

Re: aarch64 lacks HP_TIMING support in glibc

[Adhemerval Zanella]

On 22/06/2021 15:38, Andrew Pinski via Libc-alpha wrote:
> On Tue, Jun 22, 2021 at 11:30 AM Rafael Eduardo Ruviaro Christ via
> Libc-alpha <libc-alpha@sourceware.org> wrote:
>>
>> Hello,
>>
>> I noticed that aarch64 lacks HP_TIMING support in glibc. Although this architecture has generic timers, glibc is still using the generic hp-timing.h for aarch64.
>> I see HP_TIMING support for other architectures. I'd like to know why there is no HP_TIMING support for aarch64?
>> Also, if there's planning to support it for aarch64?
> 
> See https://sourceware.org/legacy-ml/libc-alpha/2015-06/msg00736.html
> and the previous emails in the thread.
> 


Do you have an outcome from kernel regarding it? It seems that the vDSO
seems to use cntvct_el0 regardless, even if it traps for emulation (not
sure if kernel rewrites the vDSO depending of underlying kernel support).

It should be possible support hp-timing.h on aarch64 to use cntvct_el0
if the minimum supported kernel also supports it (3.2).  Afaik the
hp-timing will be used only for LD_DEBUG=statistics, so default usage
would not be penalized. 

Re: aarch64 lacks HP_TIMING support in glibc

[Andreas Schwab]

On Jun 22 2021, Adhemerval Zanella via Libc-alpha wrote:

> It should be possible support hp-timing.h on aarch64 to use cntvct_el0
> if the minimum supported kernel also supports it (3.2).  Afaik the
> hp-timing will be used only for LD_DEBUG=statistics, so default usage
> would not be penalized. 

But then, why is it such a big deal?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: aarch64 lacks HP_TIMING support in glibc

[Adhemerval Zanella]

On 22/06/2021 16:27, Andreas Schwab wrote:
> On Jun 22 2021, Adhemerval Zanella via Libc-alpha wrote:
> 
>> It should be possible support hp-timing.h on aarch64 to use cntvct_el0
>> if the minimum supported kernel also supports it (3.2).  Afaik the
>> hp-timing will be used only for LD_DEBUG=statistics, so default usage
>> would not be penalized. 
> 
> But then, why is it such a big deal?

Do you mean support hp-timing on aarch64 or eventually trap on kernel
on such usage? 

Re: aarch64 lacks HP_TIMING support in glibc

[Andreas Schwab]

On Jun 22 2021, Adhemerval Zanella wrote:

> On 22/06/2021 16:27, Andreas Schwab wrote:
>> On Jun 22 2021, Adhemerval Zanella via Libc-alpha wrote:
>> 
>>> It should be possible support hp-timing.h on aarch64 to use cntvct_el0
>>> if the minimum supported kernel also supports it (3.2).  Afaik the
>>> hp-timing will be used only for LD_DEBUG=statistics, so default usage
>>> would not be penalized. 
>> 
>> But then, why is it such a big deal?
>
> Do you mean support hp-timing on aarch64 or eventually trap on kernel
> on such usage? 

hp-timing in general.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: aarch64 lacks HP_TIMING support in glibc

[Adhemerval Zanella]

On 22/06/2021 17:18, Andreas Schwab wrote:
> On Jun 22 2021, Adhemerval Zanella wrote:
> 
>> On 22/06/2021 16:27, Andreas Schwab wrote:
>>> On Jun 22 2021, Adhemerval Zanella via Libc-alpha wrote:
>>>
>>>> It should be possible support hp-timing.h on aarch64 to use cntvct_el0
>>>> if the minimum supported kernel also supports it (3.2).  Afaik the
>>>> hp-timing will be used only for LD_DEBUG=statistics, so default usage
>>>> would not be penalized. 
>>>
>>> But then, why is it such a big deal?
>>
>> Do you mean support hp-timing on aarch64 or eventually trap on kernel
>> on such usage? 
> 
> hp-timing in general.

It helps on loader statistics, but I guess clock_gettime might be good
enough.  Or maybe even debug statistics are not that useful.

aarch64 lacks HP_TIMING support in glibc

[Wilco Dijkstra]

Hi Rafael,

> I noticed that aarch64 lacks HP_TIMING support in glibc. Although this architecture has generic timers, glibc is still using the generic hp-timing.h for aarch64.
> I see HP_TIMING support for other architectures. I'd like to know why there is no HP_TIMING support for aarch64?
> Also, if there's planning to support it for aarch64?

I added HP_TIMING support for all targets a while back - this uses fast timers
via the vDSO mechanism. Some targets also have a header with an inline counter.

However it's not an external header and it is hardly used inside GLIBC, so it's
not clear whether it's worth the effort to ensure that cntvct_el0 is never trapped
on AArch64 or just declare the systems that do trap it obsolete.

Cheers,
Wilco

Re: aarch64 lacks HP_TIMING support in glibc

[Adhemerval Zanella]

On 23/06/2021 08:43, Wilco Dijkstra via Libc-alpha wrote:
> Hi Rafael,
> 
>> I noticed that aarch64 lacks HP_TIMING support in glibc. Although this architecture has generic timers, glibc is still using the generic hp-timing.h for aarch64.
>> I see HP_TIMING support for other architectures. I'd like to know why there is no HP_TIMING support for aarch64?
>> Also, if there's planning to support it for aarch64?
> 
> I added HP_TIMING support for all targets a while back - this uses fast timers
> via the vDSO mechanism. Some targets also have a header with an inline counter.
> 
> However it's not an external header and it is hardly used inside GLIBC, so it's
> not clear whether it's worth the effort to ensure that cntvct_el0 is never trapped
> on AArch64 or just declare the systems that do trap it obsolete.
> 
> Cheers,
> Wilco
> 

Is the trap really an issue here? Assuming that we can make it only issued for
LD_DEBUG=statistics (I need to check if they are used outside it), it will be
basically the cost of calling clock_gettime on the architecture without the
support.

Re: aarch64 lacks HP_TIMING support in glibc

[Wilco Dijkstra]

Hi Adhemerval,

> Is the trap really an issue here? Assuming that we can make it only issued for
> LD_DEBUG=statistics (I need to check if they are used outside it), it will be
> basically the cost of calling clock_gettime on the architecture without the
> support.

I checked and found an old server that traps. The emulation is surprisingly fast -
only a factor of 10 overhead. On several cores using cntvct_el0 is twice as fast
as clock_gettime. This remains true even if trapped (ie. vDSO is disabled).

That means there is no issue with using cntvct_el0 in hp_timing.h since it doesn't
crash and is never slower. So we could just add the header without needing
kernel fixes.

Cheers,
Wilco

RE: aarch64 lacks HP_TIMING support in glibc

[Rafael Eduardo Ruviaro Christ]

Dear Wilco,

Thank you for your message. Currently, we're using glibc 2.28. As soon as I am able to update to the latest glibc I will repeat the tests in our platform to evaluate the clock_getres() behavior.

Kind regards,
Rafael

-----Original Message-----
From: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Sent: Wednesday, June 23, 2021 1:43 PM
To: 'GNU C Library' <libc-alpha@sourceware.org>; Rafael Eduardo Ruviaro Christ <rafaeleduardoruviaro.christ@asml.com>
Subject: aarch64 lacks HP_TIMING support in glibc

CAUTION: This message is from an external sender

Hi Rafael,

> I noticed that aarch64 lacks HP_TIMING support in glibc. Although this architecture has generic timers, glibc is still using the generic hp-timing.h for aarch64.
> I see HP_TIMING support for other architectures. I'd like to know why there is no HP_TIMING support for aarch64?
> Also, if there's planning to support it for aarch64?

I added HP_TIMING support for all targets a while back - this uses fast timers via the vDSO mechanism. Some targets also have a header with an inline counter.

However it's not an external header and it is hardly used inside GLIBC, so it's not clear whether it's worth the effort to ensure that cntvct_el0 is never trapped on AArch64 or just declare the systems that do trap it obsolete.

Cheers,
Wilco
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.

Re: aarch64 lacks HP_TIMING support in glibc

[Wilco Dijkstra]

Hi Rafael,

> Thank you for your message. Currently, we're using glibc 2.28. As soon as I am able to
> update to the latest glibc I will repeat the tests in our platform to evaluate the
> clock_getres() behavior.

I don't believe there has been any change in behaviour since 2.28 - clock_getres does not
use hp-timing in any way (and never did on AArch64). So is there something specific that
you have an issue with?

Cheers,
Wilco

RE: aarch64 lacks HP_TIMING support in glibc

[Rafael Eduardo Ruviaro Christ]

Hi Wilco,

The function clock_getres returns hardcoded 1ns if platform has high-resolution timers, instead of the actual resolution based on clock frequency. According to linux/include/linux/hrtimer.h:
/*
* The resolution of the clocks. The resolution value is returned in
* the clock_getres() system call to give application programmers an
* idea of the (in)accuracy of timers. Timer values are rounded up to
* this resolution values.
*/
# define HIGH_RES_NSEC          1
We consider this a bad design, but the Linux kernel community has decided that it meets their needs and have therefore not pursued any new implementation to return the real resolution.  For instance, if system clock is 25MHz, correct resolution should be 40ns.

I cannot patch the kernel because it is considered internally to risky. I was hoping to fix this in user space.
For AArch64, it is possible to retrieve the system clock frequency:
asm volatile("mrs %0, cntfrq_el0" : "=r" (freq));
The ideal would be to implement the solution in glibc.
What do you think Wilco?

Kind regards,
Rafael



From: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Sent: Monday, June 28, 2021 9:25 PM
To: Rafael Eduardo Ruviaro Christ <rafaeleduardoruviaro.christ@asml.com>; 'GNU C Library' <libc-alpha@sourceware.org>
Subject: Re: aarch64 lacks HP_TIMING support in glibc

CAUTION: This message is from an external sender
Hi Rafael,

> Thank you for your message. Currently, we're using glibc 2.28. As soon as I am able to
> update to the latest glibc I will repeat the tests in our platform to evaluate the
> clock_getres() behavior.

I don't believe there has been any change in behaviour since 2.28 - clock_getres does not
use hp-timing in any way (and never did on AArch64). So is there something specific that
you have an issue with?

Cheers,
Wilco
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.

Re: aarch64 lacks HP_TIMING support in glibc

[Wilco Dijkstra]

Hi Rafael,

> The function clock_getres returns hardcoded 1ns if platform has high-resolution
> timers, instead of the actual resolution based on clock frequency. 

Yes that is a long standing issue in Linux...

> I cannot patch the kernel because it is considered internally to risky.
> I was hoping to fix this in user space.

Well there were some hacks in older GLIBCs for a few targets but they have been
removed since GLIBC 2.30. Any solution would have to be generic and work on
all targets. For example one could compute the resolution from multiple calls to
a timer.

> For AArch64, it is possible to retrieve the system clock frequency:
> asm volatile("mrs %0, cntfrq_el0" : "=r" (freq));
> The ideal would be to implement the solution in glibc.

This does not give the resolution, just the count frequency (some machines
increment by more than 1). Either way, the clock frequency this returns applies
to cntvct_el0, so you can use these in your own code, but you can't assume 
clock_gettime has the same resolution (it will be true in most cases, however
Linux might switch to a different timer).

You could propose a RFC to improve the clock_getres implementation and see
what people say. It might help if you have use cases where knowing the exact
resolution is essential.

Cheers,
Wilco

Re: aarch64 lacks HP_TIMING support in glibc

[Adhemerval Zanella]

On 02/07/2021 10:13, Wilco Dijkstra via Libc-alpha wrote:
> You could propose a RFC to improve the clock_getres implementation and see
> what people say. It might help if you have use cases where knowing the exact
> resolution is essential.
> 

I really prefer if we avoid trying to hack this on userspace and let the
kernel handle it instead, specially if requires a arch-specific implementation.
I think it is really unlikely that we would get a better resolution in 
userspace that kernel can't provide more efficiently.