Improving math function wrappers

Improving math function wrappers

[Wilco Dijkstra]

Hi,

When looking at profiles of various benchmarks that use math functions, I noticed
that the wrappers that set errno have a significant overhead. This is particularly
noticeable when the wrapper tests the return value of the math function - this requires
the input operand(s) to be saved, creating a stack frame and several callee-saves.

While the wrappers may be bypassed when building with -Ofast, this uses C header
magic (ie. it doesn't work in languages other than C and C++). Given other languages
don't even have the concept of errno, C99 doesn't require it, and no application ever
reads errno, it is inefficient to force the use of the wrappers in almost all cases.

One way of improving this would be to reverse the current logic and default to the
unwrapped IEEE functions. The errno wrapper would then only be selected when the 
standard requires it (C89), on old (obsolete?) UNIX variants that assume errno is
always set, or if the user explicitly requests it.

Another possibility would be to merge the wrappers into their respective math functions.
This is feasible since the IEEE versions must check for the same exceptional cases
anyway. While this would increase complexity of math functions, it shouldn't affect the
critical path for the common cases. However it would mean errno is sometimes set
when it otherwise wouldn't be (with Ofast or when an IEEE math function is called
internally). Also on some odd systems it might even result in extra messages being
printed by __kernel_standard (can we obsolete that???).

What do you think?

Wilco    

Re: Improving math function wrappers

[Joseph Myers]

On Thu, 16 Mar 2017, Wilco Dijkstra wrote:

> Given other languages don't even have the concept of errno, C99 doesn't 
> require it, and no application ever reads errno, it is inefficient to 
> force the use of the wrappers in almost all cases.

C99 not requiring it was an incompatible quiet change from C90.

> Another possibility would be to merge the wrappers into their respective 
> math functions. This is feasible since the IEEE versions must check for 
> the same exceptional cases anyway. While this would increase complexity 
> of math functions, it shouldn't affect the critical path for the common 
> cases. However it would mean errno is sometimes set when it otherwise 
> wouldn't be (with Ofast or when an IEEE math function is called 
> internally). Also on some odd systems it might even result in extra 
> messages being printed by __kernel_standard (can we obsolete that???).

My view is that merging errno-setting into the main implementations of the 
various functions is reasonable - but it also involves a significant 
amount of architecture-specific work where architecture-specific 
implementations of those functions are involved, and it's quite likely the 
code to set the TLS errno might vary depending on PIC versus non-PIC, with 
the non-PIC case then not being well-tested.  The case for it is a lot 
more straightforward where there aren't any assembly implementations for 
particular architectures.  Adding *_noerrno exports for selected 
functions, based on performance evidence, would also be reasonable (GCC 5 
and above define __NO_MATH_ERRNO__ for -fno-math-errno, which can be used 
to select such variants; of course for non-C-family languages the compiler 
would need to know about such function variants).

(There may be cases where errno setting in the main implementations will 
require adding checks there not currently present, if e.g. the function 
does "return x - x;" in the non-finite argument case to return NaN and 
also raise "invalid" for Inf arguments - but such cases should not 
generally be significant for performance; the performance relevant cases 
are those involving finite arguments and results, not error cases or NaNs 
as arguments.)

As for __kernel_standard obsoletion, there are several useful steps that 
can be done towards that:

* We now have the new wrappers that avoid checking _LIB_VERSION or calling 
__kernel_standard, used for float128 to avoid matherr / _LIB_VERSION 
support ever being part of the ABI for new types.

* The next step would be to make _LIB_VERSION, matherr and the ia64 
variants matherrf, matherrl into compat symbols, so that new programs 
cannot use that functionality.  Documentation for matherr error handling 
would be removed.  libieee.a would no longer be installed.  Something 
would need to be done about the test for matherr support.  Things would be 
arranged so that new architectures no longer get exported symbols (even 
compat symbols) for _LIB_VERSION or matherr (although some related code 
might still be present internally).

* The next step after that would be to ensure that the new wrappers are 
used in static libm, and in shared libm when the minimum symbol version 
postdates the making of _LIB_VERSION into a compat symbol.  New 
architectures, and static libm would no longer have any __kernel_standard 
code or code related to _LIB_VERSION / matherr.

* An optional step would be to use the new wrappers also on existing 
architectures as new symbol versions for existing symbols that have 
_LIB_VERSION / matherr support at their old symbol versions.

* If a function is changed to set errno directly in all its 
implementations, so it no longer needs a wrapper except for legacy 
_LIB_VERSION / matherr support, of course in the above for "new wrappers" 
you can read "direct export of the main implementation without a wrapper" 
(and the compat symbols for _LIB_VERSION / matherr support would just end 
up setting errno redundantly in both the main implementation and the 
wrapper).

* Given the different structure of ia64 libm, it seems plausible that 
while the making into compat symbols of _LIB_VERSION etc. should be done 
globally (to keep the API exported the same everywhere), some of the 
subsequent steps might only be applied to non-ia64 architectures.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Improving math function wrappers

[Joseph Myers]

On Thu, 16 Mar 2017, Joseph Myers wrote:

> * The next step after that would be to ensure that the new wrappers are 
> used in static libm, and in shared libm when the minimum symbol version 
> postdates the making of _LIB_VERSION into a compat symbol.  New 
> architectures, and static libm would no longer have any __kernel_standard 
> code or code related to _LIB_VERSION / matherr.

Also note care would be needed about testing at this point, as you'd then 
have significantly different code used in static libm (not well tested) 
from shared libm.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Improving math function wrappers

[Szabolcs Nagy]

On 16/03/17 14:29, Joseph Myers wrote:
> On Thu, 16 Mar 2017, Wilco Dijkstra wrote:
> 
>> Given other languages don't even have the concept of errno, C99 doesn't 
>> require it, and no application ever reads errno, it is inefficient to 
>> force the use of the wrappers in almost all cases.
> 
> C99 not requiring it was an incompatible quiet change from C90.
> 

i think glibc only has to set errno when user code is
compiled with c89 compatibility and even then only for
math functions that were defined in c89 and then only
guarantee correct errno setting in default rounding mode
(c89 code cannot access the fenv).

aiming for more math errno support has little use since
it is a non-portable and thus obsolete feature.

i think gcc should do this too: sqrt(x) should be
inlined as a single instruction by default, instead of
calling libc just in case it may want to set errno
(i.e. -std=c99 and -std=c11 can and thus should imply
-fno-math-errno). i saw various targets wanting this,
but it would be better to do this in general.
https://gcc.gnu.org/ml/gcc/2011-02/msg00095.html

>> Another possibility would be to merge the wrappers into their respective 
>> math functions. This is feasible since the IEEE versions must check for 
>> the same exceptional cases anyway. While this would increase complexity 
>> of math functions, it shouldn't affect the critical path for the common 
>> cases. However it would mean errno is sometimes set when it otherwise 
>> wouldn't be (with Ofast or when an IEEE math function is called 
>> internally). Also on some odd systems it might even result in extra 
>> messages being printed by __kernel_standard (can we obsolete that???).
> 
> My view is that merging errno-setting into the main implementations of the 
> various functions is reasonable - but it also involves a significant 
> amount of architecture-specific work where architecture-specific 
> implementations of those functions are involved, and it's quite likely the 
> code to set the TLS errno might vary depending on PIC versus non-PIC, with 
> the non-PIC case then not being well-tested.  The case for it is a lot 
> more straightforward where there aren't any assembly implementations for 
> particular architectures.  Adding *_noerrno exports for selected 
> functions, based on performance evidence, would also be reasonable (GCC 5 
> and above define __NO_MATH_ERRNO__ for -fno-math-errno, which can be used 
> to select such variants; of course for non-C-family languages the compiler 
> would need to know about such function variants).
> 
> (There may be cases where errno setting in the main implementations will 
> require adding checks there not currently present, if e.g. the function 
> does "return x - x;" in the non-finite argument case to return NaN and 
> also raise "invalid" for Inf arguments - but such cases should not 
> generally be significant for performance; the performance relevant cases 
> are those involving finite arguments and results, not error cases or NaNs 
> as arguments.)
> 

i don't think setting errno in asm would be necessary,
just turn "return x-x;" into "return specialcase1(x);"
where specialcase1 is generic c code shared by targets,
these are only called in cold code paths (if the errno
setting is in a hot path then wrapper vs no wrapper
does not make much difference and a wrapper can always
be in c)

one issue is that underflow/overflow thresholds can depend
on the rounding mode if errno is set based on inputs (instead
of based on the result in a wrapper), but i think errno
can be set spuriously in general so it should be possible
to choose one threshold covering all rounding modes or
alternatively only supporting errno in default rounding mode.

i think we need to decide to either (1) make it easy (default)
to call math functions without errno wrapper or (2) always
set errno, but optimize it so it does not affect performance
for common inputs (by merging the wrapper into the math code).

> As for __kernel_standard obsoletion, there are several useful steps that 
> can be done towards that:
> 
> * We now have the new wrappers that avoid checking _LIB_VERSION or calling 
> __kernel_standard, used for float128 to avoid matherr / _LIB_VERSION 
> support ever being part of the ABI for new types.
> 
> * The next step would be to make _LIB_VERSION, matherr and the ia64 
> variants matherrf, matherrl into compat symbols, so that new programs 
> cannot use that functionality.  Documentation for matherr error handling 
> would be removed.  libieee.a would no longer be installed.  Something 
> would need to be done about the test for matherr support.  Things would be 
> arranged so that new architectures no longer get exported symbols (even 
> compat symbols) for _LIB_VERSION or matherr (although some related code 
> might still be present internally).
> 
> * The next step after that would be to ensure that the new wrappers are 
> used in static libm, and in shared libm when the minimum symbol version 
> postdates the making of _LIB_VERSION into a compat symbol.  New 
> architectures, and static libm would no longer have any __kernel_standard 
> code or code related to _LIB_VERSION / matherr.
> 
> * An optional step would be to use the new wrappers also on existing 
> architectures as new symbol versions for existing symbols that have 
> _LIB_VERSION / matherr support at their old symbol versions.
> 
> * If a function is changed to set errno directly in all its 
> implementations, so it no longer needs a wrapper except for legacy 
> _LIB_VERSION / matherr support, of course in the above for "new wrappers" 
> you can read "direct export of the main implementation without a wrapper" 
> (and the compat symbols for _LIB_VERSION / matherr support would just end 
> up setting errno redundantly in both the main implementation and the 
> wrapper).
> 
> * Given the different structure of ia64 libm, it seems plausible that 
> while the making into compat symbols of _LIB_VERSION etc. should be done 
> globally (to keep the API exported the same everywhere), some of the 
> subsequent steps might only be applied to non-ia64 architectures.
> 

Re: Improving math function wrappers

[Joseph Myers]

On Thu, 16 Mar 2017, Szabolcs Nagy wrote:

> On 16/03/17 14:29, Joseph Myers wrote:
> > On Thu, 16 Mar 2017, Wilco Dijkstra wrote:
> > 
> >> Given other languages don't even have the concept of errno, C99 doesn't 
> >> require it, and no application ever reads errno, it is inefficient to 
> >> force the use of the wrappers in almost all cases.
> > 
> > C99 not requiring it was an incompatible quiet change from C90.
> 
> i think glibc only has to set errno when user code is
> compiled with c89 compatibility and even then only for
> math functions that were defined in c89 and then only
> guarantee correct errno setting in default rounding mode
> (c89 code cannot access the fenv).

Incompatible quiet change means a good idea to implement the compatible 
mode ((math_errhandling & MATH_ERRNO) != 0) as far as possible.  Just like 
it's a good idea to avoid size_t wider than unsigned long.

Compilers for non-C languages without the concept of these functions 
setting errno can of course default to -fno-math-errno by default (and 
built-in no-errno functions that can be used with -fmath-errno is 
otherwise enabled would also be desirable for fma, sqrt, etc.).

> > (There may be cases where errno setting in the main implementations will 
> > require adding checks there not currently present, if e.g. the function 
> > does "return x - x;" in the non-finite argument case to return NaN and 
> > also raise "invalid" for Inf arguments - but such cases should not 
> > generally be significant for performance; the performance relevant cases 
> > are those involving finite arguments and results, not error cases or NaNs 
> > as arguments.)
> 
> i don't think setting errno in asm would be necessary,
> just turn "return x-x;" into "return specialcase1(x);"
> where specialcase1 is generic c code shared by targets,
> these are only called in cold code paths (if the errno
> setting is in a hot path then wrapper vs no wrapper
> does not make much difference and a wrapper can always
> be in c)

Whether the asm code calls a special-case function or __errno_location or 
sets TLS errno directly doesn't make much difference, you still need lots 
of target-specific checks for NaNs / overflow etc.  (Some cases may 
already be inside cold paths, some, e.g. overflow / underflow where the 
exception is implicit from the calculation but errno needs an extra check, 
may not, although the value to check should at least already be in a 
register.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Improving math function wrappers

[Szabolcs Nagy]

On 16/03/17 18:24, Joseph Myers wrote:
> On Thu, 16 Mar 2017, Szabolcs Nagy wrote:
>> On 16/03/17 14:29, Joseph Myers wrote:
>>> On Thu, 16 Mar 2017, Wilco Dijkstra wrote:
>>>
>>>> Given other languages don't even have the concept of errno, C99 doesn't 
>>>> require it, and no application ever reads errno, it is inefficient to 
>>>> force the use of the wrappers in almost all cases.
>>>
>>> C99 not requiring it was an incompatible quiet change from C90.
>>
>> i think glibc only has to set errno when user code is
>> compiled with c89 compatibility and even then only for
>> math functions that were defined in c89 and then only
>> guarantee correct errno setting in default rounding mode
>> (c89 code cannot access the fenv).
> 
> Incompatible quiet change means a good idea to implement the compatible 
> mode ((math_errhandling & MATH_ERRNO) != 0) as far as possible.  Just like 
> it's a good idea to avoid size_t wider than unsigned long.

i think this would be a valid argument if

 (math_errhandling & MATH_ERRNO) != 0

was the only way to get backward compatibility,
but that's not the case: errno can be clobbered
any time and thus c89 backward compatibility can
be maintained with

 (math_errhandling & MATH_ERRNO) == 0

in which case there is no requirement for non-c89
math functions to set errno (or getting errno
right in non-default fenv) and thus the wrapper
can be removed from all single and long double
precision functions (and kept for double prec
c89 functions if glibc cares about compatibility).

> Compilers for non-C languages without the concept of these functions 
> setting errno can of course default to -fno-math-errno by default (and 
> built-in no-errno functions that can be used with -fmath-errno is 
> otherwise enabled would also be desirable for fma, sqrt, etc.).

then fortran can only use no-errno functions
when those can be inlined, since extern calls
use the standard libc symbols which set errno
in glibc.

Re: Improving math function wrappers

[Joseph Myers]

On Tue, 4 Apr 2017, Szabolcs Nagy wrote:

> > Compilers for non-C languages without the concept of these functions 
> > setting errno can of course default to -fno-math-errno by default (and 
> > built-in no-errno functions that can be used with -fmath-errno is 
> > otherwise enabled would also be desirable for fma, sqrt, etc.).
> 
> then fortran can only use no-errno functions
> when those can be inlined, since extern calls
> use the standard libc symbols which set errno
> in glibc.

No, you should think of the implementation as a whole.  The Fortran 
compiler can know about what functions outside the standard are available 
in the C libm, and so generate calls to such nonstandard functions where 
appropriate.  That can either be hardcoded information about functions in 
particular glibc versions, or a system (e.g. file in the sysroot read by 
the compiler) for glibc to communicate such information to the Fortran 
compiler.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Improving math function wrappers

[Szabolcs Nagy]

On 04/04/17 18:25, Szabolcs Nagy wrote:
> On 16/03/17 18:24, Joseph Myers wrote:
>> On Thu, 16 Mar 2017, Szabolcs Nagy wrote:
>>> On 16/03/17 14:29, Joseph Myers wrote:
>>>> On Thu, 16 Mar 2017, Wilco Dijkstra wrote:
>>>>
>>>>> Given other languages don't even have the concept of errno, C99 doesn't 
>>>>> require it, and no application ever reads errno, it is inefficient to 
>>>>> force the use of the wrappers in almost all cases.
>>>>
>>>> C99 not requiring it was an incompatible quiet change from C90.
>>>
>>> i think glibc only has to set errno when user code is
>>> compiled with c89 compatibility and even then only for
>>> math functions that were defined in c89 and then only
>>> guarantee correct errno setting in default rounding mode
>>> (c89 code cannot access the fenv).
>>
>> Incompatible quiet change means a good idea to implement the compatible 
>> mode ((math_errhandling & MATH_ERRNO) != 0) as far as possible.  Just like 
>> it's a good idea to avoid size_t wider than unsigned long.
> 
> i think this would be a valid argument if
> 
>  (math_errhandling & MATH_ERRNO) != 0
> 
> was the only way to get backward compatibility,
> but that's not the case: errno can be clobbered
> any time and thus c89 backward compatibility can
> be maintained with
> 
>  (math_errhandling & MATH_ERRNO) == 0
> 
> in which case there is no requirement for non-c89
> math functions to set errno (or getting errno
> right in non-default fenv) and thus the wrapper
> can be removed from all single and long double
> precision functions (and kept for double prec
> c89 functions if glibc cares about compatibility).
> 

hm this was not correct: errno cannot be arbitrarily
clobbered, it either has to be set correctly or left
unchanged, but this does not change my argument:
backward compatibility can be maintained more easily
with (math_errhandling & MATH_ERRNO) == 0.

http://port70.net/~nsz/c/c11/n1570.html#7.12.1p7

>> Compilers for non-C languages without the concept of these functions 
>> setting errno can of course default to -fno-math-errno by default (and 
>> built-in no-errno functions that can be used with -fmath-errno is 
>> otherwise enabled would also be desirable for fma, sqrt, etc.).
> 
> then fortran can only use no-errno functions
> when those can be inlined, since extern calls
> use the standard libc symbols which set errno
> in glibc.
> 

Re: Improving math function wrappers

[Zack Weinberg]

On Thu, Mar 16, 2017 at 2:24 PM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Thu, 16 Mar 2017, Szabolcs Nagy wrote:
>> On 16/03/17 14:29, Joseph Myers wrote:
>> > On Thu, 16 Mar 2017, Wilco Dijkstra wrote:
>> >
>> >> Given other languages don't even have the concept of errno, C99 doesn't
>> >> require it, and no application ever reads errno, it is inefficient to
>> >> force the use of the wrappers in almost all cases.
>> >
>> > C99 not requiring it was an incompatible quiet change from C90.
>>
>> i think glibc only has to set errno when user code is
>> compiled with c89 compatibility and even then only for
>> math functions that were defined in c89 and then only
>> guarantee correct errno setting in default rounding mode
>> (c89 code cannot access the fenv).
>
> Incompatible quiet change means a good idea to implement the compatible
> mode ((math_errhandling & MATH_ERRNO) != 0) as far as possible.  Just like
> it's a good idea to avoid size_t wider than unsigned long.

In this case, I seriously question whether anyone _wants_ math
functions to set errno, ever, even in C89-era code.

Can we try to find out?  It ought to be possible to trawl
numerical-computation codebases for references to errno... (I don't
have anything like the time to do it, alas)

Re: Improving math function wrappers

[Florian Weimer]

On 03/16/2017 02:52 PM, Wilco Dijkstra wrote:
> While the wrappers may be bypassed when building with -Ofast, this uses C header
> magic (ie. it doesn't work in languages other than C and C++). Given other languages
> don't even have the concept of errno, C99 doesn't require it, and no application ever
> reads errno, it is inefficient to force the use of the wrappers in almost all cases.

What's the expensive part?  Computing what errno value to set, or 
obtaining the location of errno?

We have recently removed the PID cache from the TCB.  We could store the 
errno value there, instead of going through __errno_location.  (Code 
outside glibc would still use __errno_location, of course.)

Florian

Re: Improving math function wrappers

[Joseph Myers]

On Thu, 16 Mar 2017, Florian Weimer wrote:

> We have recently removed the PID cache from the TCB.  We could store the errno
> value there, instead of going through __errno_location.  (Code outside glibc
> would still use __errno_location, of course.)

Code in glibc's libraries should already be setting the TLS initial-exec 
errno variable directly (that's how __set_errno is defined in 
include/errno.h).  But any case that sets errno should not be 
performance-critical; the issue is the effects on performance of non-error 
cases with finite arguments (which should just be checking if the result 
is non-finite or zero, depending on the function - but the issue raised 
here was consequent need to save the arguments for subsequent checks in 
the unlikely case to see whether e.g. a NaN result was an error or from a 
NaN argument, and it's possible the code in the error case might have 
other prologue/epilogue effects that don't get shrink-wrapped away).

Most of the time the checks either duplicate ones already present in the 
main function implementation, or could be localized to an unlikely case 
within that implementation, hence the notion that setting errno in the 
main function implementation makes sense (along with avoiding extra stack 
frame setup etc.).

If we set errno in some main function implementations I think we should be 
clear that adding a wrapper is still a fine way to fix one of the 
remaining bugs about missing errno setting in cases where 
architecture-specific implementations are involved, with the option of 
optimizing later by moving away from that wrapper.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Improving math function wrappers

[Wilco Dijkstra]

Joseph Myers <joseph@codesourcery.com> wrote:

> Code in glibc's libraries should already be setting the TLS initial-exec 
> errno variable directly (that's how __set_errno is defined in 
> include/errno.h).  But any case that sets errno should not be 
> performance-critical; the issue is the effects on performance of non-error 
> cases with finite arguments (which should just be checking if the result 
> is non-finite or zero, depending on the function - but the issue raised 
> here was consequent need to save the arguments for subsequent checks in 
> the unlikely case to see whether e.g. a NaN result was an error or from a 
> NaN argument, and it's possible the code in the error case might have 
> other prologue/epilogue effects that don't get shrink-wrapped away).

Setting errno is not in any way performance critical indeed - even a function
call would be fast enough. It is cases like this that show up as having a high
overhead:

double
__pow (double x, double y)
{
  double z = __ieee754_pow (x, y);
  if (__glibc_unlikely (!isfinite (z)))
     ...
  else if (__builtin_expect (z == 0.0, 0)
	   && isfinite (x) && x != 0 && isfinite (y)
	   && _LIB_VERSION != _IEEE_)
     ...

This means saving and restoring x and y across the call (ie. setting up a stack
frame with 3-4 callee-saves) and 2 if statements on the critical path - all
completely unnecessary.

Inlining __ieee754_pow would help but still keep the extra checks - so moving
them into existing checks for special cases is best if we can't remove errno support.

> If we set errno in some main function implementations I think we should be 
> clear that adding a wrapper is still a fine way to fix one of the 
> remaining bugs about missing errno setting in cases where 
> architecture-specific implementations are involved, with the option of 
> optimizing later by moving away from that wrapper.

Yes, we'd still have to keep the wrapper if there are any targets that require it.
It could be removed once such code has been updated or if the generic
implementation ends up faster (quite likely given this happened for several string
functions)...

Wilco