[PATCH] Optimized generic expf and exp2f

[PATCH] Optimized generic expf and exp2f

[Szabolcs Nagy]

[-- Attachment #1: Type: text/plain, Size: 2908 bytes --]

Based on new expf and exp2f code from
https://github.com/ARM-software/optimized-routines/

with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
expf reciprocal-throughput: 3.3x faster
expf latency: 1.7x faster
libm.so size: -8byte data, -2140byte text
expf/exp2f worst case nearest rounding ulp error: 0.502

Error checks are inline and actual error handling is
in separate functions that are tail called and expected
to be reusable when other math functions are implemented
without wrappers. (expf, __expf, __ieee754_expf symbols
are aliases, _LIB_VERSION is not checked, errno is set
unconditionally according to POSIX rules.)

Double precision arithmetics is used which is expected
to be faster on most targets (including soft-float) than
using single precision and it is easier to get good
precision result with it.

Const data is kept in a separate translation unit which
complicates maintenance a bit, but is expected to give
good code for literal loads on most targets and allows
sharing data across expf, exp2f and powf.

Some configuration is in a new math_config.h which is
currently kept similar to the one in optimized-routines
repo which had to be self-contained, I expect that this
will need more adjustments for glibc.

Some details may need target specific tweaks:
- best convert and round to int operation in the arg
reduction may be different across targets.
- code was optimized on fma target, optimal polynomial
eval may be different without fma.
- gcc does not always generate good code for fp bit
representation access via unions or it may be inherently
slow on some target.

One issue with the argument reduction is that it only
works right with nearest rounded rint, otherwise the
interval is [0,2c] or [-2c,0] instead of [-c,c]. The
polynomial is optimized for [-c,c] but it has sufficent
extra precision that it gives acceptable results on
[-2c,2c] too assuming users are less interested in
non-nearest rounded precision, however this means some
glibc ulp error limits will need adjustment.

Another issue is with the polynomial evaluation which
causes 1 ulp errors for tiny inputs in some non-nearest
rounding, but i think it is worth the trade off for
better pipelined polynomial.


2017-09-05  Szabolcs Nagy  <szabolcs.nagy@arm.com>

	* math/Makefile (type-float-routines): Add math_errf and e_exp2f_data.
	* sysdeps/ieee754/flt-32/w_expf_compat.c: Move to...
	* math/w_expf_compat.c: ... here.
	* sysdeps/aarch64/fpu/math_private.h (TOINT_INTRINSICS): Define.
	(roundtoint, converttoint): Likewise.
	* sysdeps/ieee754/flt-32/e_expf.c: New implementation.
	* sysdeps/ieee754/flt-32/e_exp2f.c: New implementation.
	* sysdeps/ieee754/flt-32/e_exp2f_data.c: New file.
	* sysdeps/ieee754/flt-32/math_config.h: New file.
	* sysdeps/ieee754/flt-32/math_errf.c: New file.
	* sysdeps/ieee754/flt-32/w_exp2f_compat.c: New file.
	* sysdeps/x86_64/fpu/w_expf_compat.c: New file.

[-- Attachment #2: expf.diff --]
[-- Type: text/x-patch, Size: 24259 bytes --]

diff --git a/math/Makefile b/math/Makefile
index 0601f3ac43..04586156f8 100644
--- a/math/Makefile
+++ b/math/Makefile
@@ -115,7 +115,7 @@ type-double-routines := branred doasin dosincos halfulp mpa mpatan2	\
 
 # float support
 type-float-suffix := f
-type-float-routines := k_rem_pio2f
+type-float-routines := k_rem_pio2f math_errf e_exp2f_data
 
 # _Float128 support
 type-float128-suffix := f128
diff --git a/math/w_expf_compat.c b/math/w_expf_compat.c
new file mode 100644
index 0000000000..8a1fa51e46
--- /dev/null
+++ b/math/w_expf_compat.c
@@ -0,0 +1,35 @@
+/* Copyright (C) 2011-2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+   Contributed by Ulrich Drepper <drepper@gmail.com>, 2011.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include <math.h>
+#include <math_private.h>
+#include <math-svid-compat.h>
+
+/* wrapper expf */
+float
+__expf (float x)
+{
+  float z = __ieee754_expf (x);
+  if (__builtin_expect (!isfinite (z) || z == 0, 0)
+      && isfinite (x) && _LIB_VERSION != _IEEE_)
+    return __kernel_standard_f (x, x, 106 + !!signbit (x));
+
+  return z;
+}
+hidden_def (__expf)
+weak_alias (__expf, expf)
diff --git a/sysdeps/aarch64/fpu/math_private.h b/sysdeps/aarch64/fpu/math_private.h
index 807111ea5a..775237ffb2 100644
--- a/sysdeps/aarch64/fpu/math_private.h
+++ b/sysdeps/aarch64/fpu/math_private.h
@@ -319,6 +319,26 @@ libc_feresetround_noex_aarch64_ctx (struct rm_ctx *ctx)
 #define libc_feresetround_noexf_ctx	libc_feresetround_noex_aarch64_ctx
 #define libc_feresetround_noexl_ctx	libc_feresetround_noex_aarch64_ctx
 
+/* Hack: only include the large arm_neon.h when needed.  */
+#ifdef _MATH_CONFIG_H
+#include <arm_neon.h>
+
+/* ACLE intrinsics for frintn and fcvtns instructions.  */
+#define TOINT_INTRINSICS 1
+
+static inline double_t
+roundtoint (double_t x)
+{
+  return vget_lane_f64 (vrndn_f64 (vld1_f64 (&x)), 0);
+}
+
+static inline uint64_t
+converttoint (double_t x)
+{
+  return vcvtnd_s64_f64 (x);
+}
+#endif
+
 #include_next <math_private.h>
 
 #endif
diff --git a/sysdeps/ieee754/flt-32/e_exp2f.c b/sysdeps/ieee754/flt-32/e_exp2f.c
index 567d3ff6d0..dfadd85f93 100644
--- a/sysdeps/ieee754/flt-32/e_exp2f.c
+++ b/sysdeps/ieee754/flt-32/e_exp2f.c
@@ -1,7 +1,6 @@
-/* Single-precision floating point 2^x.
-   Copyright (C) 1997-2017 Free Software Foundation, Inc.
+/* Single-precision 2^x function.
+   Copyright (C) 2017 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
-   Contributed by Geoffrey Keating <geoffk@ozemail.com.au>
 
    The GNU C Library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
@@ -17,116 +16,75 @@
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */
 
-/* The basic design here is from
-   Shmuel Gal and Boris Bachelis, "An Accurate Elementary Mathematical
-   Library for the IEEE Floating Point Standard", ACM Trans. Math. Soft.,
-   17 (1), March 1991, pp. 26-45.
-   It has been slightly modified to compute 2^x instead of e^x, and for
-   single-precision.
-   */
-#ifndef _GNU_SOURCE
-# define _GNU_SOURCE
-#endif
-#include <stdlib.h>
-#include <float.h>
-#include <ieee754.h>
 #include <math.h>
-#include <fenv.h>
-#include <inttypes.h>
-#include <math_private.h>
-
-#include "t_exp2f.h"
-
-static const float TWOM100 = 7.88860905e-31;
-static const float TWO127 = 1.7014118346e+38;
+#include <stdint.h>
+#include "math_config.h"
+
+/*
+EXP2F_TABLE_BITS = 5
+EXP2F_POLY_ORDER = 3
+
+ULP error: 0.502 (nearest rounding.)
+Relative error: 1.69 * 2^-34 in [-1/64, 1/64] (before rounding.)
+Wrong count: 168353 (all nearest rounding wrong results with fma.)
+Non-nearest ULP error: 1 (rounded ULP error)
+*/
+
+#define N (1 << EXP2F_TABLE_BITS)
+#define T __exp2f_data.tab
+#define C __exp2f_data.poly
+#define SHIFT __exp2f_data.shift_scaled
+
+static inline uint32_t
+top12 (float x)
+{
+  return asuint (x) >> 20;
+}
 
 float
-__ieee754_exp2f (float x)
+__exp2f (float x)
 {
-  static const float himark = (float) FLT_MAX_EXP;
-  static const float lomark = (float) (FLT_MIN_EXP - FLT_MANT_DIG - 1);
-
-  /* Check for usual case.  */
-  if (isless (x, himark) && isgreaterequal (x, lomark))
+  uint32_t abstop;
+  uint64_t ki, t;
+  /* double_t for better performance on targets with FLT_EVAL_METHOD==2.  */
+  double_t kd, xd, z, r, r2, y, s;
+
+  xd = (double_t) x;
+  abstop = top12 (x) & 0x7ff;
+  if (__builtin_expect (abstop >= top12 (128.0f), 0))
     {
-      static const float THREEp14 = 49152.0;
-      int tval, unsafe;
-      float rx, x22, result;
-      union ieee754_float ex2_u, scale_u;
-
-      if (fabsf (x) < FLT_EPSILON / 4.0f)
-	return 1.0f + x;
-
-      {
-	SET_RESTORE_ROUND_NOEXF (FE_TONEAREST);
-
-	/* 1. Argument reduction.
-	   Choose integers ex, -128 <= t < 128, and some real
-	   -1/512 <= x1 <= 1/512 so that
-	   x = ex + t/512 + x1.
-
-	   First, calculate rx = ex + t/256.  */
-	rx = x + THREEp14;
-	rx -= THREEp14;
-	x -= rx;  /* Compute x=x1. */
-	/* Compute tval = (ex*256 + t)+128.
-	   Now, t = (tval mod 256)-128 and ex=tval/256  [that's mod, NOT %;
-	   and /-round-to-nearest not the usual c integer /].  */
-	tval = (int) (rx * 256.0f + 128.0f);
-
-	/* 2. Adjust for accurate table entry.
-	   Find e so that
-	   x = ex + t/256 + e + x2
-	   where -7e-4 < e < 7e-4, and
-	   (float)(2^(t/256+e))
-	   is accurate to one part in 2^-64.  */
-
-	/* 'tval & 255' is the same as 'tval%256' except that it's always
-	   positive.
-	   Compute x = x2.  */
-	x -= __exp2f_deltatable[tval & 255];
-
-	/* 3. Compute ex2 = 2^(t/255+e+ex).  */
-	ex2_u.f = __exp2f_atable[tval & 255];
-	tval >>= 8;
-	/* x2 is an integer multiple of 2^-30; avoid intermediate
-	   underflow from the calculation of x22 * x.  */
-	unsafe = abs(tval) >= -FLT_MIN_EXP - 32;
-	ex2_u.ieee.exponent += tval >> unsafe;
-	scale_u.f = 1.0;
-	scale_u.ieee.exponent += tval - (tval >> unsafe);
-
-	/* 4. Approximate 2^x2 - 1, using a second-degree polynomial,
-	   with maximum error in [-2^-9 - 2^-14, 2^-9 + 2^-14]
-	   less than 1.3e-10.  */
-
-	x22 = (.24022656679f * x + .69314736128f) * ex2_u.f;
-      }
-
-      /* 5. Return (2^x2-1) * 2^(t/512+e+ex) + 2^(t/512+e+ex).  */
-      result = x22 * x + ex2_u.f;
-
-      if (!unsafe)
-	return result;
-      else
-	{
-	  result *= scale_u.f;
-	  math_check_force_underflow_nonneg (result);
-	  return result;
-	}
-    }
-  /* Exceptional cases:  */
-  else if (isless (x, himark))
-    {
-      if (isinf (x))
-	/* e^-inf == 0, with no error.  */
-	return 0;
-      else
-	/* Underflow */
-	return TWOM100 * TWOM100;
+      /* |x| >= 128 or x is nan.  */
+      if (asuint (x) == asuint (-INFINITY))
+	return 0.0f;
+      if (abstop >= top12 (INFINITY))
+	return x + x;
+      if (x > 0.0f)
+	return __math_oflowf (0);
+      if (x <= -150.0f)
+	return __math_uflowf (0);
+#if WANT_ERRNO_UFLOW
+      if (x < -149.0f)
+	return __math_may_uflowf (0);
+#endif
     }
-  else
-    /* Return x, if x is a NaN or Inf; or overflow, otherwise.  */
-    return TWO127*x;
+
+  /* x = k/N + r with r in [-1/(2N), 1/(2N)] and int k.  */
+  kd = (double) (xd + SHIFT); /* Rounding to double precision is required.  */
+  ki = asuint64 (kd);
+  kd -= SHIFT; /* k/N for int k.  */
+  r = xd - kd;
+
+  /* exp2(x) = 2^(k/N) * 2^r ~= s * (C0*r^3 + C1*r^2 + C2*r + 1) */
+  t = T[ki % N];
+  t += ki << (52 - EXP2F_TABLE_BITS);
+  s = asdouble (t);
+  z = C[0] * r + C[1];
+  r2 = r * r;
+  y = C[2] * r + 1;
+  y = z * r2 + y;
+  y = y * s;
+  return (float) y;
 }
-strong_alias (__ieee754_exp2f, __exp2f_finite)
+weak_alias (__exp2f, exp2f)
+strong_alias (__exp2f, __ieee754_exp2f)
+strong_alias (__exp2f, __exp2f_finite)
diff --git a/sysdeps/ieee754/flt-32/e_exp2f_data.c b/sysdeps/ieee754/flt-32/e_exp2f_data.c
new file mode 100644
index 0000000000..390dcae333
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/e_exp2f_data.c
@@ -0,0 +1,44 @@
+/* Shared data between expf, exp2f and powf.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include "math_config.h"
+
+#define N (1 << EXP2F_TABLE_BITS)
+
+const struct exp2f_data __exp2f_data = {
+  /* tab[i] = uint(2^(i/N)) - (i << 52-BITS)
+     used for computing 2^(k/N) for an int |k| < 150 N as
+     double(tab[k%N] + (k << 52-BITS)) */
+  .tab = {
+0x3ff0000000000000, 0x3fefd9b0d3158574, 0x3fefb5586cf9890f, 0x3fef9301d0125b51,
+0x3fef72b83c7d517b, 0x3fef54873168b9aa, 0x3fef387a6e756238, 0x3fef1e9df51fdee1,
+0x3fef06fe0a31b715, 0x3feef1a7373aa9cb, 0x3feedea64c123422, 0x3feece086061892d,
+0x3feebfdad5362a27, 0x3feeb42b569d4f82, 0x3feeab07dd485429, 0x3feea47eb03a5585,
+0x3feea09e667f3bcd, 0x3fee9f75e8ec5f74, 0x3feea11473eb0187, 0x3feea589994cce13,
+0x3feeace5422aa0db, 0x3feeb737b0cdc5e5, 0x3feec49182a3f090, 0x3feed503b23e255d,
+0x3feee89f995ad3ad, 0x3feeff76f2fb5e47, 0x3fef199bdd85529c, 0x3fef3720dcef9069,
+0x3fef5818dcfba487, 0x3fef7c97337b9b5f, 0x3fefa4afa2a490da, 0x3fefd0765b6e4540,
+  },
+  .shift_scaled = 0x1.8p+52 / N,
+  .poly = { 0x1.c6af84b912394p-5, 0x1.ebfce50fac4f3p-3, 0x1.62e42ff0c52d6p-1 },
+  .shift = 0x1.8p+52,
+  .invln2_scaled = 0x1.71547652b82fep+0 * N,
+  .poly_scaled = {
+0x1.c6af84b912394p-5/N/N/N, 0x1.ebfce50fac4f3p-3/N/N, 0x1.62e42ff0c52d6p-1/N
+  },
+};
diff --git a/sysdeps/ieee754/flt-32/e_expf.c b/sysdeps/ieee754/flt-32/e_expf.c
index 782072f213..d1eabdae08 100644
--- a/sysdeps/ieee754/flt-32/e_expf.c
+++ b/sysdeps/ieee754/flt-32/e_expf.c
@@ -1,7 +1,6 @@
-/* Single-precision floating point e^x.
-   Copyright (C) 1997-2017 Free Software Foundation, Inc.
+/* Single-precision e^x function.
+   Copyright (C) 2017 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
-   Contributed by Geoffrey Keating <geoffk@ozemail.com.au>
 
    The GNU C Library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
@@ -17,117 +16,90 @@
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */
 
-/* How this works:
-
-   The input value, x, is written as
-
-   x = n * ln(2) + t/512 + delta[t] + x;
-
-   where:
-   - n is an integer, 127 >= n >= -150;
-   - t is an integer, 177 >= t >= -177
-   - delta is based on a table entry, delta[t] < 2^-28
-   - x is whatever is left, |x| < 2^-10
-
-   Then e^x is approximated as
-
-   e^x = 2^n ( e^(t/512 + delta[t])
-	       + ( e^(t/512 + delta[t])
-		   * ( p(x + delta[t] + n * ln(2)) - delta ) ) )
-
-   where
-   - p(x) is a polynomial approximating e(x)-1;
-   - e^(t/512 + delta[t]) is obtained from a table.
-
-   The table used is the same one as for the double precision version;
-   since we have the table, we might as well use it.
-
-   It turns out to be faster to do calculations in double precision than
-   to perform an 'accurate table method' expf, because of the range reduction
-   overhead (compare exp2f).
-   */
-#include <float.h>
-#include <ieee754.h>
 #include <math.h>
-#include <fenv.h>
-#include <inttypes.h>
-#include <math_private.h>
-
-extern const float __exp_deltatable[178];
-extern const double __exp_atable[355] /* __attribute__((mode(DF))) */;
-
-static const float TWOM100 = 7.88860905e-31;
-static const float TWO127 = 1.7014118346e+38;
+#include <stdint.h>
+#include "math_config.h"
+
+/*
+EXP2F_TABLE_BITS = 5
+EXP2F_POLY_ORDER = 3
+
+ULP error: 0.502 (nearest rounding.)
+Relative error: 1.69 * 2^-34 in [-ln2/64, ln2/64] (before rounding.)
+Wrong count: 170635 (all nearest rounding wrong results with fma.)
+Non-nearest ULP error: 1 (rounded ULP error)
+*/
+
+#define N (1 << EXP2F_TABLE_BITS)
+#define InvLn2N __exp2f_data.invln2_scaled
+#define T __exp2f_data.tab
+#define C __exp2f_data.poly_scaled
+
+static inline uint32_t
+top12 (float x)
+{
+  return asuint (x) >> 20;
+}
 
 float
-__ieee754_expf (float x)
+__expf (float x)
 {
-  static const float himark = 88.72283935546875;
-  static const float lomark = -103.972084045410;
-  /* Check for usual case.  */
-  if (isless (x, himark) && isgreater (x, lomark))
+  uint32_t abstop;
+  uint64_t ki, t;
+  /* double_t for better performance on targets with FLT_EVAL_METHOD==2.  */
+  double_t kd, xd, z, r, r2, y, s;
+
+  xd = (double_t) x;
+  abstop = top12 (x) & 0x7ff;
+  if (__builtin_expect (abstop >= top12 (88.0f), 0))
     {
-      static const float THREEp42 = 13194139533312.0;
-      static const float THREEp22 = 12582912.0;
-      /* 1/ln(2).  */
-#undef M_1_LN2
-      static const float M_1_LN2 = 1.44269502163f;
-      /* ln(2) */
-#undef M_LN2
-      static const double M_LN2 = .6931471805599452862;
-
-      int tval;
-      double x22, t, result, dx;
-      float n, delta;
-      union ieee754_double ex2_u;
-
-      {
-	SET_RESTORE_ROUND_NOEXF (FE_TONEAREST);
-
-	/* Calculate n.  */
-	n = x * M_1_LN2 + THREEp22;
-	n -= THREEp22;
-	dx = x - n*M_LN2;
-
-	/* Calculate t/512.  */
-	t = dx + THREEp42;
-	t -= THREEp42;
-	dx -= t;
-
-	/* Compute tval = t.  */
-	tval = (int) (t * 512.0);
-
-	if (t >= 0)
-	  delta = - __exp_deltatable[tval];
-	else
-	  delta = __exp_deltatable[-tval];
-
-	/* Compute ex2 = 2^n e^(t/512+delta[t]).  */
-	ex2_u.d = __exp_atable[tval+177];
-	ex2_u.ieee.exponent += (int) n;
-
-	/* Approximate e^(dx+delta) - 1, using a second-degree polynomial,
-	   with maximum error in [-2^-10-2^-28,2^-10+2^-28]
-	   less than 5e-11.  */
-	x22 = (0.5000000496709180453 * dx + 1.0000001192102037084) * dx + delta;
-      }
-
-      /* Return result.  */
-      result = x22 * ex2_u.d + ex2_u.d;
-      return (float) result;
+      /* |x| >= 88 or x is nan.  */
+      if (asuint (x) == asuint (-INFINITY))
+	return 0.0f;
+      if (abstop >= top12 (INFINITY))
+	return x + x;
+      if (x > 0x1.62e42ep6f) /* x > log(0x1p128) ~= 88.72 */
+	return __math_oflowf (0);
+      if (x < -0x1.9fe368p6f) /* x < log(0x1p-150) ~= -103.97 */
+	return __math_uflowf (0);
+#if WANT_ERRNO_UFLOW
+      if (x < -0x1.9d1d9ep6f) /* x < log(0x1p-149) ~= -103.28 */
+	return __math_may_uflowf (0);
+#endif
     }
-  /* Exceptional cases:  */
-  else if (isless (x, himark))
-    {
-      if (isinf (x))
-	/* e^-inf == 0, with no error.  */
-	return 0;
-      else
-	/* Underflow */
-	return TWOM100 * TWOM100;
-    }
-  else
-    /* Return x, if x is a NaN or Inf; or overflow, otherwise.  */
-    return TWO127*x;
+
+  /* x*N/Ln2 = k + r with r in [-1/2, 1/2] and int k.  */
+  z = InvLn2N * xd;
+
+  /* Round and convert z to int, the result is in [-150*N, 128*N] and
+     ideally ties-to-even rule is used, otherwise the magnitude of r
+     can be bigger which gives larger approximation error.  */
+#if TOINT_INTRINSICS
+  kd = roundtoint (z);
+  ki = converttoint (z);
+#elif TOINT_RINT
+  kd = rint (z);
+  ki = (long) kd;
+#elif TOINT_SHIFT
+# define SHIFT __exp2f_data.shift
+  kd = (double) (z + SHIFT); /* Rounding to double precision is required.  */
+  ki = asuint64 (kd);
+  kd -= SHIFT;
+#endif
+  r = z - kd;
+
+  /* exp(x) = 2^(k/N) * 2^(r/N) ~= s * (C0*r^3 + C1*r^2 + C2*r + 1) */
+  t = T[ki % N];
+  t += ki << (52 - EXP2F_TABLE_BITS);
+  s = asdouble (t);
+  z = C[0] * r + C[1];
+  r2 = r * r;
+  y = C[2] * r + 1;
+  y = z * r2 + y;
+  y = y * s;
+  return (float) y;
 }
-strong_alias (__ieee754_expf, __expf_finite)
+hidden_def (__expf)
+weak_alias (__expf, expf)
+strong_alias (__expf, __ieee754_expf)
+strong_alias (__expf, __expf_finite)
diff --git a/sysdeps/ieee754/flt-32/math_config.h b/sysdeps/ieee754/flt-32/math_config.h
new file mode 100644
index 0000000000..aaaa90067d
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/math_config.h
@@ -0,0 +1,128 @@
+/* Configuration for math routines.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#ifndef _MATH_CONFIG_H
+#define _MATH_CONFIG_H
+
+#include <math.h>
+#include <math_private.h>
+#include <stdint.h>
+
+#ifndef WANT_ROUNDING
+/* Correct special case results in non-nearest rounding modes.  */
+#define WANT_ROUNDING 1
+#endif
+#ifndef WANT_ERRNO
+/* Set errno according to ISO C with (math_errhandling & MATH_ERRNO) != 0.  */
+#define WANT_ERRNO 1
+#endif
+#ifndef WANT_ERRNO_UFLOW
+/* Set errno to ERANGE if result underflows to 0 (in all rounding modes).  */
+#define WANT_ERRNO_UFLOW (WANT_ROUNDING && WANT_ERRNO)
+#endif
+
+#ifndef TOINT_INTRINSICS
+#define TOINT_INTRINSICS 0
+#endif
+#ifndef TOINT_RINT
+#define TOINT_RINT 0
+#endif
+#ifndef TOINT_SHIFT
+#define TOINT_SHIFT 1
+#endif
+
+static inline uint32_t
+asuint (float f)
+{
+  union
+  {
+    float f;
+    uint32_t i;
+  } u = {f};
+  return u.i;
+}
+
+static inline float
+asfloat (uint32_t i)
+{
+  union
+  {
+    uint32_t i;
+    float f;
+  } u = {i};
+  return u.f;
+}
+
+static inline uint64_t
+asuint64 (double f)
+{
+  union
+  {
+    double f;
+    uint64_t i;
+  } u = {f};
+  return u.i;
+}
+
+static inline double
+asdouble (uint64_t i)
+{
+  union
+  {
+    uint64_t i;
+    double f;
+  } u = {i};
+  return u.f;
+}
+
+static inline int
+ieee_2008_issignaling (float x)
+{
+  uint32_t ix = asuint (x);
+  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
+  return 2 * ix > 2u * 0x7fc00000;
+}
+
+#ifdef __GNUC__
+#define HIDDEN __attribute__ ((__visibility__ ("hidden")))
+#define NOINLINE __attribute__ ((noinline))
+#else
+#define HIDDEN
+#define NOINLINE
+#endif
+
+HIDDEN float __math_oflowf (unsigned long);
+HIDDEN float __math_uflowf (unsigned long);
+HIDDEN float __math_may_uflowf (unsigned long);
+HIDDEN float __math_divzerof (unsigned long);
+HIDDEN float __math_invalidf (float);
+
+/* Shared between expf, exp2f and powf.  */
+#define EXP2F_TABLE_BITS 5
+#define EXP2F_POLY_ORDER 3
+extern const struct exp2f_data
+{
+  uint64_t tab[1 << EXP2F_TABLE_BITS];
+  double shift_scaled;
+  double poly[EXP2F_POLY_ORDER];
+  double shift;
+  double invln2_scaled;
+  double poly_scaled[EXP2F_POLY_ORDER];
+} __exp2f_data;
+
+#endif
diff --git a/sysdeps/ieee754/flt-32/math_errf.c b/sysdeps/ieee754/flt-32/math_errf.c
new file mode 100644
index 0000000000..11fbdc0b92
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/math_errf.c
@@ -0,0 +1,77 @@
+/* Single-precision math error handling.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include "math_config.h"
+
+#if WANT_ERRNO
+#include <errno.h>
+/* NOINLINE reduces code size and avoids making math functions non-leaf
+   when the error handling is inlined.  */
+NOINLINE static float
+with_errnof (float y, int e)
+{
+  errno = e;
+  return y;
+}
+#else
+#define with_errnof(x, e) (x)
+#endif
+
+/* NOINLINE prevents fenv semantics breaking optimizations.  */
+NOINLINE static float
+xflowf (unsigned long sign, float y)
+{
+  y = (sign ? -y : y) * y;
+  return with_errnof (y, ERANGE);
+}
+
+HIDDEN float
+__math_uflowf (unsigned long sign)
+{
+  return xflowf (sign, 0x1p-95f);
+}
+
+#if WANT_ERRNO_UFLOW
+/* Underflows to zero in some non-nearest rounding mode, setting errno
+   is valid even if the result is non-zero, but in the subnormal range.  */
+HIDDEN float
+__math_may_uflowf (unsigned long sign)
+{
+  return xflowf (sign, 0x1.4p-75f);
+}
+#endif
+
+HIDDEN float
+__math_oflowf (unsigned long sign)
+{
+  return xflowf (sign, 0x1p97f);
+}
+
+HIDDEN float
+__math_divzerof (unsigned long sign)
+{
+  float y = 0;
+  return with_errnof ((sign ? -1 : 1) / y, ERANGE);
+}
+
+HIDDEN float
+__math_invalidf (float x)
+{
+  float y = (x - x) / (x - x);
+  return isnan (x) ? y : with_errnof (y, EDOM);
+}
diff --git a/sysdeps/ieee754/flt-32/w_exp2f_compat.c b/sysdeps/ieee754/flt-32/w_exp2f_compat.c
new file mode 100644
index 0000000000..c37de4df43
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/w_exp2f_compat.c
@@ -0,0 +1 @@
+/* Empty file to disable the exp2f wrapper.  */
diff --git a/sysdeps/ieee754/flt-32/w_expf_compat.c b/sysdeps/ieee754/flt-32/w_expf_compat.c
index 8a1fa51e46..578cf4d4a7 100644
--- a/sysdeps/ieee754/flt-32/w_expf_compat.c
+++ b/sysdeps/ieee754/flt-32/w_expf_compat.c
@@ -1,35 +1 @@
-/* Copyright (C) 2011-2017 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Contributed by Ulrich Drepper <drepper@gmail.com>, 2011.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Lesser General Public
-   License as published by the Free Software Foundation; either
-   version 2.1 of the License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Lesser General Public License for more details.
-
-   You should have received a copy of the GNU Lesser General Public
-   License along with the GNU C Library; if not, see
-   <http://www.gnu.org/licenses/>.  */
-
-#include <math.h>
-#include <math_private.h>
-#include <math-svid-compat.h>
-
-/* wrapper expf */
-float
-__expf (float x)
-{
-  float z = __ieee754_expf (x);
-  if (__builtin_expect (!isfinite (z) || z == 0, 0)
-      && isfinite (x) && _LIB_VERSION != _IEEE_)
-    return __kernel_standard_f (x, x, 106 + !!signbit (x));
-
-  return z;
-}
-hidden_def (__expf)
-weak_alias (__expf, expf)
+/* Empty file to disable the expf wrapper.  */
diff --git a/sysdeps/x86_64/fpu/w_expf_compat.c b/sysdeps/x86_64/fpu/w_expf_compat.c
new file mode 100644
index 0000000000..dc454638c3
--- /dev/null
+++ b/sysdeps/x86_64/fpu/w_expf_compat.c
@@ -0,0 +1 @@
+#include <math/w_expf_compat.c>

Re: [PATCH] Optimized generic expf and exp2f

[Szabolcs Nagy]

On 05/09/17 17:54, Szabolcs Nagy wrote:
> Based on new expf and exp2f code from
> https://github.com/ARM-software/optimized-routines/
> 
> with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
> expf reciprocal-throughput: 3.3x faster
> expf latency: 1.7x faster

forgot to mention that this is on an aarch64 cpu
on an x86_64 cpu i measure

expf reciprocal-throughput: 1.7x faster than current asm
expf latency: 1.5x faster than current asm

Re: [PATCH] Optimized generic expf and exp2f

[Arjan van de Ven]

On 9/5/2017 9:54 AM, Szabolcs Nagy wrote:
> Based on new expf and exp2f code from
> https://github.com/ARM-software/optimized-routines/
> 
> with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
> expf reciprocal-throughput: 3.3x faster
> expf latency: 1.7x faster
> libm.so size: -8byte data, -2140byte text
> expf/exp2f worst case nearest rounding ulp error: 0.502
> 


you mentioned x86 data.. is that based on current git after
the recent optimizations (on a cpu with fma)?

Re: [PATCH] Optimized generic expf and exp2f

[Joseph Myers]

On Tue, 5 Sep 2017, Szabolcs Nagy wrote:

> Error checks are inline and actual error handling is
> in separate functions that are tail called and expected
> to be reusable when other math functions are implemented
> without wrappers. (expf, __expf, __ieee754_expf symbols
> are aliases, _LIB_VERSION is not checked, errno is set
> unconditionally according to POSIX rules.)

The _LIB_VERSION and matherr handling is part of the ABI for the existing 
symbol versions of expf and exp2f.  Thus, if you wish to use integrated 
error handling that does not handle _LIB_VERSION and matherr the same way 
as the existing code, you need a new symbol version (with the old version 
then probably keeping the existing wrapper but having it wrap the new 
implementation - duplicate errno setting for existing binaries should be 
fine).

> One issue with the argument reduction is that it only
> works right with nearest rounded rint, otherwise the
> interval is [0,2c] or [-2c,0] instead of [-c,c]. The
> polynomial is optimized for [-c,c] but it has sufficent
> extra precision that it gives acceptable results on
> [-2c,2c] too assuming users are less interested in
> non-nearest rounded precision, however this means some
> glibc ulp error limits will need adjustment.

By ulp error limits do you mean the entries in libm-test-ulps, or the 
global max_valid_error limit in libm-test-support.c which no 
libm-test-ulps entry is allowed to exceed?

> 	* sysdeps/ieee754/flt-32/w_expf_compat.c: Move to...
> 	* math/w_expf_compat.c: ... here.

I'd expect all the w_exp{,f,l}_compat.c files to move at the same time 
(modulo probably needing to add an ldbl-opt version to deal with the long 
double versioning in the ldbl-64-128 and ldbl-128ibm versions).  I think 
they are all in fact generic and are only in sysdeps directories because 
older versions of them used to hardcode bounds on arguments to the exp 
functions that gave finite nonzero results for a particular format.

> 	* sysdeps/x86_64/fpu/w_expf_compat.c: New file.

Is this something to do with x86_64 having its own expf implementations?

It seems i386, ia64, m68k, powerpc64 and x86_64 all have their own 
implementations of expf, exp2f or both (sometimes multiarch, sometimes the 
multiarch variants may have a fallback to using the generic C version, or 
using it built with particular options).  I'd expect an expf replacement 
patch like this one to explain explicitly how it affects those 
architectures.  If on any architecture the answer is that the new C 
versions are not used at all, then I'd expect that architecture to get its 
own dummy versions of math_errf.c and e_exp2f_data.c to avoid building 
unused code into libm.  Of course, if an architecture uses either the 
generic expf or the generic exp2f under any circumstances, it needs the 
relevant support code built into libm when there is code referencing it in 
libm.

Now, I do not make assertions about the performance merits of any of those 
architecture-specific variants; it's entirely possible that some of them 
should be removed if their performance is inferior to the performance of 
this C version (once the C version has been appropriately configured for 
that architecture) (or replaced by building it with specific options to 
generate multiarch variants, if that is beneficial on a particular 
architecture).

> +  if (__builtin_expect (abstop >= top12 (128.0f), 0))

We use __glibc_unlikely in glibc.

> +  if (__builtin_expect (abstop >= top12 (88.0f), 0))

Likewise.

> +#ifndef WANT_ROUNDING
> +/* Correct special case results in non-nearest rounding modes.  */
> +#define WANT_ROUNDING 1
> +#endif

glibc style uses "# define" inside #if, etc.

> +static inline int
> +ieee_2008_issignaling (float x)
> +{
> +  uint32_t ix = asuint (x);
> +  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
> +  return 2 * ix > 2u * 0x7fc00000;
> +}

This doesn't seem to be used, but if you need issignaling tests in future 
functions (powf?), you need to respect HIGH_ORDER_BIT_IS_SET_FOR_SNAN from 
nan-high-order-bit.h.

> +#ifdef __GNUC__
> +#define HIDDEN __attribute__ ((__visibility__ ("hidden")))
> +#define NOINLINE __attribute__ ((noinline))

We don't generally want __GNUC__ conditionals in glibc code, unless it's 
actually shared verbatim with an external repository such as gnulib, or 
it's an installed header.  So attribute_hidden can be used 
unconditionally.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Optimized generic expf and exp2f

[Joseph Myers]

On Tue, 5 Sep 2017, Arjan van de Ven wrote:

> you mentioned x86 data.. is that based on current git after
> the recent optimizations (on a cpu with fma)?

Really you need to compare with both the fma and non-fma versions (and 
compare the C version built both with and without fma, since one 
possibility would be that the C version can replace the x86_64 ones but 
should be built twice, with and without fma, to achieve that replacement).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Optimized generic expf and exp2f

[Szabolcs Nagy]

On 05/09/17 21:58, Joseph Myers wrote:
> On Tue, 5 Sep 2017, Arjan van de Ven wrote:
> 
>> you mentioned x86 data.. is that based on current git after
>> the recent optimizations (on a cpu with fma)?
> 
> Really you need to compare with both the fma and non-fma versions (and 
> compare the C version built both with and without fma, since one 
> possibility would be that the C version can replace the x86_64 ones but 
> should be built twice, with and without fma, to achieve that replacement).
> 

i don't have a machine with fma, i tested it on
an older cpu (but i did test against git tip,
using gcc-7).

Re: [PATCH] Optimized generic expf and exp2f

[Gabriel F. T. Gomes]

On Tue, 5 Sep 2017 20:45:44 +0000
Joseph Myers <joseph@codesourcery.com> wrote:

>On Tue, 5 Sep 2017, Szabolcs Nagy wrote:
>
>> +static inline int
>> +ieee_2008_issignaling (float x)
>> +{
>> +  uint32_t ix = asuint (x);
>> +  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
>> +  return 2 * ix > 2u * 0x7fc00000;
>> +}  
>
>This doesn't seem to be used, but if you need issignaling tests in future 
>functions (powf?), you need to respect HIGH_ORDER_BIT_IS_SET_FOR_SNAN from 
>nan-high-order-bit.h.

Is that also valid for _Float128 (meaning that test-math-issignaling.cc
needs to check for that, as well)?  It was my understanding that it isn't.

Re: [PATCH] Optimized generic expf and exp2f

[Joseph Myers]

On Thu, 7 Sep 2017, Gabriel F. T. Gomes wrote:

> On Tue, 5 Sep 2017 20:45:44 +0000
> Joseph Myers <joseph@codesourcery.com> wrote:
> 
> >On Tue, 5 Sep 2017, Szabolcs Nagy wrote:
> >
> >> +static inline int
> >> +ieee_2008_issignaling (float x)
> >> +{
> >> +  uint32_t ix = asuint (x);
> >> +  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
> >> +  return 2 * ix > 2u * 0x7fc00000;
> >> +}  
> >
> >This doesn't seem to be used, but if you need issignaling tests in future 
> >functions (powf?), you need to respect HIGH_ORDER_BIT_IS_SET_FOR_SNAN from 
> >nan-high-order-bit.h.
> 
> Is that also valid for _Float128 (meaning that test-math-issignaling.cc
> needs to check for that, as well)?  It was my understanding that it isn't.

This applies just as much to _Float128.  test-math-issignaling.cc only 
does _Float128 tests in the __HAVE_DISTINCT_FLOAT128 case, which does not 
apply to any architectures using the other NaN convention, so there is no 
problem with that test, but once we support _FloatN/_FloatNx types that 
are ABI-aliases of other types, MIPS64 _Float128 / _Float64x will follow 
the same NaN convention as long double that those types alias (with which 
convention that is depending on the compiler configuration).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Optimized generic expf and exp2f

[Szabolcs Nagy]

On 05/09/17 21:45, Joseph Myers wrote:
> On Tue, 5 Sep 2017, Szabolcs Nagy wrote:
> 
>> Error checks are inline and actual error handling is
>> in separate functions that are tail called and expected
>> to be reusable when other math functions are implemented
>> without wrappers. (expf, __expf, __ieee754_expf symbols
>> are aliases, _LIB_VERSION is not checked, errno is set
>> unconditionally according to POSIX rules.)
> 
> The _LIB_VERSION and matherr handling is part of the ABI for the existing 
> symbol versions of expf and exp2f.  Thus, if you wish to use integrated 
> error handling that does not handle _LIB_VERSION and matherr the same way 
> as the existing code, you need a new symbol version (with the old version 
> then probably keeping the existing wrapper but having it wrap the new 
> implementation - duplicate errno setting for existing binaries should be 
> fine).
> 

i tried to do this but it became complicated dealing with
targets that have their own implementation.

so my plan now is to first post the new code with the
wrapper around it (so there is redundant errno setting),
then in a separate patch try to bump the symbol version
and move to errno-only-wrapper (on all targets for expf
and exp2f) and then finally remove the wrapper for the
new generic code.

it is not yet clear to me how to do the errno-only-wrapper,
since the existing wrapper-template machinery does not
work on a per function basis, but that code would be nice
to reuse, some guidance on that would be helpful.

>> One issue with the argument reduction is that it only
>> works right with nearest rounded rint, otherwise the
>> interval is [0,2c] or [-2c,0] instead of [-c,c]. The
>> polynomial is optimized for [-c,c] but it has sufficent
>> extra precision that it gives acceptable results on
>> [-2c,2c] too assuming users are less interested in
>> non-nearest rounded precision, however this means some
>> glibc ulp error limits will need adjustment.
> 
> By ulp error limits do you mean the entries in libm-test-ulps, or the 
> global max_valid_error limit in libm-test-support.c which no 
> libm-test-ulps entry is allowed to exceed?
> 

limb-test-ulps.

expf failures on x86_64 (no-fma, lrint using +-shift):

testing float (without inline functions)
Failure: Test: exp_downward (-0x1p-20)
Result:
 is:          9.99998986e-01   0x1.ffffdep-1
 should be:   9.99999046e-01   0x1.ffffe0p-1
 difference:  5.96046447e-08   0x1.000000p-24
 ulp       :  1.0000
 max.ulp   :  0.0000
Failure: Test: exp_downward (0x5.8b90b8p+4)
Result:
 is:          3.40279831e+38   0x1.ffff06p+127
 should be:   3.40279851e+38   0x1.ffff08p+127
 difference:  2.02824096e+31   0x1.000000p+104
 ulp       :  1.0000
 max.ulp   :  0.0000
Maximal error of `exp_downward'
 is      : 1 ulp
 accepted: 0 ulp
Failure: Test: exp_towardzero (-0x1p-20)
Result:
 is:          9.99998986e-01   0x1.ffffdep-1
 should be:   9.99999046e-01   0x1.ffffe0p-1
 difference:  5.96046447e-08   0x1.000000p-24
 ulp       :  1.0000
 max.ulp   :  0.0000
Failure: Test: exp_towardzero (0x5.8b90b8p+4)
Result:
 is:          3.40279831e+38   0x1.ffff06p+127
 should be:   3.40279851e+38   0x1.ffff08p+127
 difference:  2.02824096e+31   0x1.000000p+104
 ulp       :  1.0000
 max.ulp   :  0.0000
Maximal error of `exp_towardzero'
 is      : 1 ulp
 accepted: 0 ulp

Re: [PATCH] Optimized generic expf and exp2f

[Joseph Myers]

On Mon, 11 Sep 2017, Szabolcs Nagy wrote:

> it is not yet clear to me how to do the errno-only-wrapper,
> since the existing wrapper-template machinery does not
> work on a per function basis, but that code would be nice
> to reuse, some guidance on that would be helpful.

I'd expect something like

#include <math-type-macros-float.h>
#undef __USE_WRAPPER_TEMPLATE
#define __USE_WRAPPER_TEMPLATE 1
#include <w_exp_template.c>

should work.  (Modulo possibly needing to do something extra to give expf 
the right symbol version.)  A sysdeps w_expf.c should always override the 
default automatic generation based on the template.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Optimized generic expf and exp2f

[Arjan van de Ven]

On 9/6/2017 8:04 AM, Wilco Dijkstra wrote:


btw just to avoid confusion; I like the performance I'm seeing and
the math differences (against the x86-64 version) are minor,
so consider this a "yes please".
Once its in I'll poke at it more and pretty likely we'll end
up deleting the x86-64 asm version
(and add whatever glue we need to make this work for both FMA and non-FMA)

Re: [PATCH] Optimized generic expf and exp2f

[Joseph Myers]

On Wed, 6 Sep 2017, Arjan van de Ven wrote:

> interesting; it takes 2 independent FP adds and a compare (in C) to 
> detect nearest rounding being in effect (which in time can overlap with 
> the float->double conversion) so if there's an option to reduce the 
> algorithm by more than that for a fast path...

My understanding from the patch submission was that round-to-nearest was 
only relevant for the conversion to nearest integer.  (The rest of the 
calculation is done in double, so intermediate rounding errors are of no 
significance there.)

The TOINT_RINT and TOINT_SHIFT cases depend on round-to-nearest to get an 
actual nearest integer result (otherwise we get the larger errors 
discussed from applying a polynomial to a larger range than it was 
optimized for).  TOINT_INTRINSICS does whatever those intrinsics do which 
might round to nearest rather than the current rounding mode.  Use of 
round or roundeven instead of rint would also be possible.  (GCC doesn't 
have built-in roundeven at present, but it would make sense to add.  
SSE4.1 supports encoding all four binary IEEE rounding modes in the 
instruction; glibc only has floor/ceil IFUNCs using that facility, not 
trunc (bug 20142) or roundeven, and likewise for math_private.h __floor 
etc. inlines.)  Adding a constant so that the value is always positive and 
then casting to int (defined to truncate towards 0) is also a possibility.

> (also, some CPUs (like newer Intel) support an instruction prefix 
> encoding to force rounding modes on a FP instruction independent of the 
> global rounding mode, which at some point maybe should be a gcc pragma 
> or attribute or something, and then used in such C code)

That's #pragma STDC FENV_ROUND <direction> from TS 18661-1.  That makes 
sense to implement (it has both compiler and library aspects), but 
probably can't work reliably without first sorting out (conditional on 
appropriate compiler options) the general issues with optimizations not 
respecting exceptions / rounding modes.  And using it would be slow on 
architectures without the hardware support for constant rounding modes in 
instructions (at least in the case where changing rounding modes is slow), 
as it needs to insert dynamic rounding mode saves and restores in that 
case.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Optimized generic expf and exp2f

[Wilco Dijkstra]

Arjan van de Ven wrote:
    
>On 9/6/2017 7:14 AM, Szabolcs Nagy wrote:
>>> interesting; it takes 2 independent FP adds and a compare (in C) to detect nearest rounding
>>> being in effect (which in time can overlap with the float->double conversion)
>>> so if there's an option to reduce the algorithm by more than that for a fast
>>> path...
>>>
>>> (also, some CPUs (like newer Intel) support an instruction prefix encoding to force
>>> rounding modes on a FP instruction independent of the global rounding mode,
>>> which at some point maybe should be a gcc pragma or attribute or something,
>>> and then used in such C code)
>>
> 
>> i don't think reducing the polynomial (from order 3 to order 2)
>> is possible without bigger lookup table, if less accuracy is
>> enough then reducing the table size is possible though:
>> 
>> poly order / table len / ulp error / non-nearest ulp error (rounded)
>> 2          / 64        / 0.61      /
>> 2          / 128       / 0.51      /
>> 2          / 256       / 0.502     /
>> 3          / 8         / 0.91      / > 10
>> 3          / 16        / 0.526     / 2
>> 3          / 32        / 0.502     / 1
>> 3          / 64        / 0.5001    / 1
>> 4          / 8         / 0.54      /
>> 4          / 16        / 0.501     /
>> 4          / 32        / 0.50004   /
>> 4          / 64        / 0.5       /
>> 
>> the c code uses order=3/table=32, the x86_64 asm uses order=4/table=64
>> 
>
> yeah I don't think it'll work out in terms of saving cycles; on Intel at least
> FMA is 4 cycles, but an ADD is 4 cycles as well, so there's no net savings
> by doing the 2xADD+compare to save an FMA.
> (since the ADDs execute in parallel it's also not likely to be more expensive)

Most of a rounding mode test is already there given expf does range reduction.
So you just need to test whether the remainder is outside the [-C,C] interval and
then adjust as necessary.

Note adding a compare does not increase latency as it is all off the critical path.
So I believe further latency reduction is feasible while keeping throughput similar.

It all depends on how much people care about getting near perfect results for
non-nearest rounding modes...

Wilco

Re: [PATCH] Optimized generic expf and exp2f

[Arjan van de Ven]

On 9/6/2017 7:14 AM, Szabolcs Nagy wrote:
>> interesting; it takes 2 independent FP adds and a compare (in C) to detect nearest rounding
>> being in effect (which in time can overlap with the float->double conversion)
>> so if there's an option to reduce the algorithm by more than that for a fast
>> path...
>>
>> (also, some CPUs (like newer Intel) support an instruction prefix encoding to force
>> rounding modes on a FP instruction independent of the global rounding mode,
>> which at some point maybe should be a gcc pragma or attribute or something,
>> and then used in such C code)
>>
> 
> i don't think reducing the polynomial (from order 3 to order 2)
> is possible without bigger lookup table, if less accuracy is
> enough then reducing the table size is possible though:
> 
> poly order / table len / ulp error / non-nearest ulp error (rounded)
> 2          / 64        / 0.61      /
> 2          / 128       / 0.51      /
> 2          / 256       / 0.502     /
> 3          / 8         / 0.91      / > 10
> 3          / 16        / 0.526     / 2
> 3          / 32        / 0.502     / 1
> 3          / 64        / 0.5001    / 1
> 4          / 8         / 0.54      /
> 4          / 16        / 0.501     /
> 4          / 32        / 0.50004   /
> 4          / 64        / 0.5       /
> 
> the c code uses order=3/table=32, the x86_64 asm uses order=4/table=64
> 

yeah I don't think it'll work out in terms of saving cycles; on Intel at least
FMA is 4 cycles, but an ADD is 4 cycles as well, so there's no net savings
by doing the 2xADD+compare to save an FMA.
(since the ADDs execute in parallel it's also not likely to be more expensive)

being able to force rounding might still be interesting  since it avoids the whole
right column of your table

Re: [PATCH] Optimized generic expf and exp2f

[Szabolcs Nagy]

On 06/09/17 14:41, Arjan van de Ven wrote:
> On 9/6/2017 6:16 AM, Wilco Dijkstra wrote:
>> Arjan van de Ven wrote:
>>>
>>> I'm seeing a 16% throughput increase (not 1.5x) but still impressive.
>>
>> Was that using the expf trace input or something else? And with wrapper?
>>
>>> I do see different numerical answers between the two (I had to disable
>>> the code in my bench that detects differences) and sampling a few
>>> it seems that the C code is a little bit less accurate in places,
>>> likely a simpler polynomal.
>>> (for example for  20.636783599853515625    as input)
>>
>> It's still way more accurate than necessary. The only reason is to
>> minimize ULP error for non-nearest rounding modes. If you don't
>> care about worst-case ULP for non-standard rounding modes, the
>> polynomial can be further simplified within 1ULP max error in round
>> to nearest.
> 
> interesting; it takes 2 independent FP adds and a compare (in C) to detect nearest rounding
> being in effect (which in time can overlap with the float->double conversion)
> so if there's an option to reduce the algorithm by more than that for a fast
> path...
> 
> (also, some CPUs (like newer Intel) support an instruction prefix encoding to force
> rounding modes on a FP instruction independent of the global rounding mode,
> which at some point maybe should be a gcc pragma or attribute or something,
> and then used in such C code)
> 

i don't think reducing the polynomial (from order 3 to order 2)
is possible without bigger lookup table, if less accuracy is
enough then reducing the table size is possible though:

poly order / table len / ulp error / non-nearest ulp error (rounded)
2          / 64        / 0.61      /
2          / 128       / 0.51      /
2          / 256       / 0.502     /
3          / 8         / 0.91      / > 10
3          / 16        / 0.526     / 2
3          / 32        / 0.502     / 1
3          / 64        / 0.5001    / 1
4          / 8         / 0.54      /
4          / 16        / 0.501     /
4          / 32        / 0.50004   /
4          / 64        / 0.5       /

the c code uses order=3/table=32, the x86_64 asm uses order=4/table=64

Re: [PATCH] Optimized generic expf and exp2f

[Arjan van de Ven]

On 9/6/2017 6:16 AM, Wilco Dijkstra wrote:
> Arjan van de Ven wrote:
>>
>> I'm seeing a 16% throughput increase (not 1.5x) but still impressive.
> 
> Was that using the expf trace input or something else? And with wrapper?
> 
>> I do see different numerical answers between the two (I had to disable
>> the code in my bench that detects differences) and sampling a few
>> it seems that the C code is a little bit less accurate in places,
>> likely a simpler polynomal.
>> (for example for  20.636783599853515625    as input)
> 
> It's still way more accurate than necessary. The only reason is to
> minimize ULP error for non-nearest rounding modes. If you don't
> care about worst-case ULP for non-standard rounding modes, the
> polynomial can be further simplified within 1ULP max error in round
> to nearest.

interesting; it takes 2 independent FP adds and a compare (in C) to detect nearest rounding
being in effect (which in time can overlap with the float->double conversion)
so if there's an option to reduce the algorithm by more than that for a fast
path...

(also, some CPUs (like newer Intel) support an instruction prefix encoding to force
rounding modes on a FP instruction independent of the global rounding mode,
which at some point maybe should be a gcc pragma or attribute or something,
and then used in such C code)

Re: [PATCH] Optimized generic expf and exp2f

[Wilco Dijkstra]

Arjan van de Ven wrote:
>
> I'm seeing a 16% throughput increase (not 1.5x) but still impressive.

Was that using the expf trace input or something else? And with wrapper?

> I do see different numerical answers between the two (I had to disable
> the code in my bench that detects differences) and sampling a few
> it seems that the C code is a little bit less accurate in places,
> likely a simpler polynomal.
> (for example for  20.636783599853515625    as input)

It's still way more accurate than necessary. The only reason is to
minimize ULP error for non-nearest rounding modes. If you don't
care about worst-case ULP for non-standard rounding modes, the
polynomial can be further simplified within 1ULP max error in round
to nearest.

Wilco
    

Re: [PATCH] Optimized generic expf and exp2f

[Arjan van de Ven]

On 9/6/2017 5:55 AM, Wilco Dijkstra wrote:
> Arjan van de Ven wrote:
>>>
>>> expf reciprocal-throughput: 1.5x faster
>>> expf latency: 1.4x faster
>>
>> interesting; the fma existing expf for x86_64 is 10 to 11 cycles reciprocal-throughput ;-)
>> time to see how you got that 1.5x faster....
> 
> Just using a good algorithm :-)  It's not the first time generic C
> code beats "optimized" assembler implementations in GLIBC,
> and it won't be the last... This shows that optimizing generic
> code is a much better strategy overall.

I'm seeing a 16% throughput increase (not 1.5x) but still impressive.
I do see different numerical answers between the two (I had to disable
the code in my bench that detects differences) and sampling a few
it seems that the C code is a little bit less accurate in places,
likely a simpler polynomal.
(for example for  20.636783599853515625    as input)

I'm all in favor of C code, it's hard to beat compilers, unless
you have a magic trick for which the compiler doesn't know how
to generate code.

Re: [PATCH] Optimized generic expf and exp2f

[Wilco Dijkstra]

Arjan van de Ven wrote:
>> 
>> expf reciprocal-throughput: 1.5x faster
>> expf latency: 1.4x faster
>
> interesting; the fma existing expf for x86_64 is 10 to 11 cycles reciprocal-throughput ;-)
> time to see how you got that 1.5x faster....

Just using a good algorithm :-)  It's not the first time generic C
code beats "optimized" assembler implementations in GLIBC,
and it won't be the last... This shows that optimizing generic
code is a much better strategy overall.

Wilco
    

Re: [PATCH] Optimized generic expf and exp2f

[Szabolcs Nagy]

On 06/09/17 13:36, Wilco Dijkstra wrote:
> Szabolcs Nagy wrote:
>> On 05/09/17 21:58, Joseph Myers wrote:
>>> On Tue, 5 Sep 2017, Arjan van de Ven wrote:
>>
>>>> you mentioned x86 data.. is that based on current git after
>>>> the recent optimizations (on a cpu with fma)?
>>
>>> Really you need to compare with both the fma and non-fma versions (and 
>>> compare the C version built both with and without fma, since one 
>>> possibility would be that the C version can replace the x86_64 ones but 
>>> should be built twice, with and without fma, to achieve that replacement).
> 
> My machine has AVX2 and FMA, and when building the new generic expf
> with -mavx2 -mfma I get:
> 
> expf reciprocal-throughput: 1.5x faster
> expf latency: 1.4x faster
> 
> I verified in both cases FMA was used.

note that the used algorithm is similar but
the c code uses smaller polynomial and another
fma latency is removed from the arg reduction
by using 2^(x/N) polynomial instead of e^x.

..and the c code has no wrapper and it has
less branches to handle special cases.

so the c code does less computation than the
asm, in exchange it does not pass the current
math tests for non-nearest rounding.

Re: [PATCH] Optimized generic expf and exp2f

[Arjan van de Ven]

> 
> expf reciprocal-throughput: 1.5x faster
> expf latency: 1.4x faster

interesting; the fma existing expf for x86_64 is 10 to 11 cycles reciprocal-throughput ;-)
time to see how you got that 1.5x faster....

Re: [PATCH] Optimized generic expf and exp2f

[Wilco Dijkstra]

Szabolcs Nagy wrote:
> On 05/09/17 21:58, Joseph Myers wrote:
> > On Tue, 5 Sep 2017, Arjan van de Ven wrote:
> 
>>> you mentioned x86 data.. is that based on current git after
>>> the recent optimizations (on a cpu with fma)?
> 
>> Really you need to compare with both the fma and non-fma versions (and 
>> compare the C version built both with and without fma, since one 
>> possibility would be that the C version can replace the x86_64 ones but 
>> should be built twice, with and without fma, to achieve that replacement).

My machine has AVX2 and FMA, and when building the new generic expf
with -mavx2 -mfma I get:

expf reciprocal-throughput: 1.5x faster
expf latency: 1.4x faster

I verified in both cases FMA was used.

Wilco

Re: [PATCH] Optimized generic expf and exp2f

[Wilco Dijkstra]

On 05/09/17 17:54, Szabolcs Nagy wrote:
> Based on new expf and exp2f code from
> https://github.com/ARM-software/optimized-routines/
> 
> with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
> expf reciprocal-throughput: 3.3x faster
> expf latency: 1.7x faster
>
> forgot to mention that this is on an aarch64 cpu
> on an x86_64 cpu i measure
>
> expf reciprocal-throughput: 1.7x faster than current asm
> expf latency: 1.5x faster than current asm

Also compared to https://sourceware.org/ml/libc-alpha/2017-09/msg00116.html
when running on AArch64:

expf reciprocal-throughput: 1.8x faster
expf latency: 1.1x faster

Wilco