Re: Ideas for rewrite of libm-test

Re: Ideas for rewrite of libm-test

[Andreas Jaeger]

On Tue, 21 Sep 1999, Zack Weinberg wrote:

> Philip Blundell wrote:
> > What happened to Zack's dejagnu patches?
> 
> It doesn't use dejagnu.
> 
> I haven't had time to think about integrating it or even keeping it up
> to date since, um, mid-April.  AJ expressed interest in getting it
> merged back in July? but I haven't heard back from him since. The last

Just three weeks ago;-).  Sorry, I didn't answer at that time.  I haven't
found time so far to look further into it.

> snapshot is still at
> rabi.phys.columbia.edu:/pub/zack/testsuite-0.9.tar.gz.

I'm just downloading it and should have time next week to look further
into it.  Is it worth spending time on it?  Uli, will you incorporate the
testsuite?

Andreas
-- 
 Andreas Jaeger   aj@suse.de    aj@arthur.rhein-neckar.de
  for pgp-key finger ajaeger@aixd1.rhrk.uni-kl.de

Re: Ideas for rewrite of libm-test

[Ulrich Drepper]

Andreas Jaeger <aj@suse.de> writes:

> I'm just downloading it and should have time next week to look further
> into it.  Is it worth spending time on it?  Uli, will you incorporate the
> testsuite?

Bad timing.  I have absolutely no time the next one and a half week.
I can comment afterwards.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Ideas for rewrite of libm-test

[Geoff Keating]

> From: Andreas Jaeger <aj@suse.de>
> Date: 30 Sep 1999 10:53:43 +0200

> I've looked in more details in the rewrite of libm-test and have one
> more question:
> 
>      How should we define the error tolerances?
> 
> Currently libm-test uses absolute errors (the difference between a
> precalculated value and the calculated value).
> 
> But if we want to describe the overall error of a function, we should
> use relative errors like 2^-63.  
> 
> Should we also specify intervals with their errors?

I think what you probably want is to specify errors in units in the
last binary place.  For f(x), a ulp is `nextafter(f(x), 2*f(x)) -
f(x)', assuming f(x) is some reasonable number.  Perfect rounding is a
maximum difference of 0.5 ulp from the true value, many routines
specify 1 ulp as the maximum tolerance.

The advantage of this is that tolerances become nice integers or
fractions, like 1 or 2 or 129/256, and it gives you some idea of how
bad the routine is---if it has low numbers it's good, numbers like 10
or 32 or 1024 indicate some problem.

-- 
Geoffrey Keating <geoffk@cygnus.com>

Re: Ideas for rewrite of libm-test

[Ulrich Drepper]

Andreas Jaeger <aj@suse.de> writes:

> But if we want to describe the overall error of a function, we should
> use relative errors like 2^-63.  

Actually, the unit should be ULP (units of least precision).  Take a
look at the libm test web page I've written
( http://www.cygnus.com/~drepper/libm/ ) to see how to compute it.

> Btw. I'd like to read a little bit more about this topic.  I've read
> already a number of articles from the different ACM journals but if
> anybody has more pointers to relevant literature, please tell me.

The most important article is "What Every Computer Scientist Should
Know About FLoating-Point Arithmetic" by David Goldberg in ACM
Computing Surveys, March 1991.  For testing "The UNIX System Math
Library, a Status Report" by Silverstein, Sommars, and Tao in Usenix
Winter '90 is interesting.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Ideas for rewrite of libm-test

[Andreas Jaeger]

I've looked in more details in the rewrite of libm-test and have one
more question:

     How should we define the error tolerances?

Currently libm-test uses absolute errors (the difference between a
precalculated value and the calculated value).

But if we want to describe the overall error of a function, we should
use relative errors like 2^-63.  

Should we also specify intervals with their errors?

Btw. I'd like to read a little bit more about this topic.  I've read
already a number of articles from the different ACM journals but if
anybody has more pointers to relevant literature, please tell me.

Andreas
-- 
 Andreas Jaeger   aj@suse.de	aj@arthur.rhein-neckar.de
  for pgp-key finger ajaeger@aixd1.rhrk.uni-kl.de

Re: Ideas for rewrite of libm-test

[Richard Henderson]

On Wed, Sep 22, 1999 at 07:58:59AM +1000, Geoff Keating wrote:
> I think sparc and ARM are similar, they're both normal
> IEEE machines.  For Alpha, it should also be similar enough that the
> calculated margins will do---certainly, you want to know if not.

Supposing that you've compiled with -mieee, and sysdeps/alpha/Makefile
turns that on, Alpha should be IEEE compliant modulo Inexact exceptions.
(You can use -mieee-inexact for that, but it's soooo slow...)

Discrepencies may be the result of kernel bugs.


r~

Re: Ideas for rewrite of libm-test

[Andreas Jaeger]

On Wed, 22 Sep 1999, Geoff Keating wrote:

> > Date: Tue, 21 Sep 1999 22:41:46 +0100
> > From: Philip Blundell <Philip.Blundell@pobox.com>
> > 
> > >It just seems that basing the test cases on what the math functions
> > >produce (as is done now in practise, and will be done formally with
> > >your scheme) is doing it the wrong way around.
> > 
> > You can make arguments both ways.  If the required epsilons are greater than 
> > the calculated error margin then clearly something is wrong (in either libc, 
> > the compiler or the floatng point system). But it seems that compiler and 
> > FP bugs are quite common, and we don't really want "make check" to fail on 
> > platforms where the floating point support is substandard.
> 
> I do want 'make check' to fail, at least on ppc.  ppc should always
> do at least as well as the calculated error margin, or something is
> very wrong.  I think sparc and ARM are similar, they're both normal
> IEEE machines.  For Alpha, it should also be similar enough that the
> calculated margins will do---certainly, you want to know if not.

That's the reason why I suggested platform specific error margins that
will be merged with a general file.  The general file would contain the
theoretical limits - and the platform specific files would contain
additional, platform specific, bounds if neccessary.  If we agree that the
theoretical bounds are all that's needed for e.g. PPC, then there will be
no platform specific file.

> 
> Of course, sometimes the platform is sufficiently different to IEEE
> arithmetic that the maximum errors should be recalculated, in which
> case you can allow some flexibility---I'm thinking of x86 here.
> 
> > Perhaps it would be worth including two sets; the theoretical values
> > and those that are actually achievable in practice on each platform.
> > Or alternatively with a more sophisticated test suite that allowed
> > for expected failures I suppose this would come out in the wash.
> > What happened to Zack's dejagnu patches?
> 
> Hmmm.  This makes sense, but not quite the way you phrased it.  I'd
> like to have two _test cases_, one the theoretical values, and one the
> (better) values that were achieved, either of which could fail.  This
> way I'd know about regressions, too; and new porters would know when
> their port's FP support wasn't working.

This should be possible (but I did not consider it before).  In the
framework I suggested we could generate a file with only the theoretical
limits and use it.

Andreas
-- 
 Andreas Jaeger   aj@suse.de    aj@arthur.rhein-neckar.de
  for pgp-key finger ajaeger@aixd1.rhrk.uni-kl.de

Re: Ideas for rewrite of libm-test

[Geoff Keating]

> cc: aj@suse.de, libc-hacker@sourceware.cygnus.com
> Date: Tue, 21 Sep 1999 22:41:46 +0100
> From: Philip Blundell <Philip.Blundell@pobox.com>
> 
> >It just seems that basing the test cases on what the math functions
> >produce (as is done now in practise, and will be done formally with
> >your scheme) is doing it the wrong way around.
> 
> You can make arguments both ways.  If the required epsilons are greater than 
> the calculated error margin then clearly something is wrong (in either libc, 
> the compiler or the floatng point system). But it seems that compiler and 
> FP bugs are quite common, and we don't really want "make check" to fail on 
> platforms where the floating point support is substandard.

I do want 'make check' to fail, at least on ppc.  ppc should always
do at least as well as the calculated error margin, or something is
very wrong.  I think sparc and ARM are similar, they're both normal
IEEE machines.  For Alpha, it should also be similar enough that the
calculated margins will do---certainly, you want to know if not.

Of course, sometimes the platform is sufficiently different to IEEE
arithmetic that the maximum errors should be recalculated, in which
case you can allow some flexibility---I'm thinking of x86 here.

> Perhaps it would be worth including two sets; the theoretical values
> and those that are actually achievable in practice on each platform.
> Or alternatively with a more sophisticated test suite that allowed
> for expected failures I suppose this would come out in the wash.
> What happened to Zack's dejagnu patches?

Hmmm.  This makes sense, but not quite the way you phrased it.  I'd
like to have two _test cases_, one the theoretical values, and one the
(better) values that were achieved, either of which could fail.  This
way I'd know about regressions, too; and new porters would know when
their port's FP support wasn't working.

-- 
Geoffrey Keating <geoffk@cygnus.com>

Re: Ideas for rewrite of libm-test

[Ulrich Drepper]

Geoff Keating <geoffk@ozemail.com.au> writes:

> Wouldn't it be better to decide what error tolerances we want to allow
> for the math functions _first_, then see what the math functions
> actually produce?

Well, this is easy to say: we want to allow zero errors.  The question
is that we cannot do this right now and therefore have to live with
greater errors.

> It just seems that basing the test cases on what the math functions
> produce (as is done now in practise, and will be done formally with
> your scheme) is doing it the wrong way around.

I agree.  But this is only to describe the status quo, catch build
problems, and (most importantly for me) to be able to document the
errors.  This is one of the requirement of the a standrd compliant
system.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: Ideas for rewrite of libm-test

[Philip Blundell]

>It just seems that basing the test cases on what the math functions
>produce (as is done now in practise, and will be done formally with
>your scheme) is doing it the wrong way around.

You can make arguments both ways.  If the required epsilons are greater than 
the calculated error margin then clearly something is wrong (in either libc, 
the compiler or the floatng point system). But it seems that compiler and 
FP bugs are quite common, and we don't really want "make check" to fail on 
platforms where the floating point support is substandard.

Perhaps it would be worth including two sets; the theoretical values and those 
that are actually achievable in practice on each platform.  Or alternatively 
with a more sophisticated test suite that allowed for expected failures I 
suppose this would come out in the wash.  What happened to Zack's dejagnu 
patches?

p.

Re: Ideas for rewrite of libm-test

[Geoff Keating]

> From: Andreas Jaeger <aj@suse.de>
> Date: 21 Sep 1999 17:28:37 +0200
> 
> 
> Uli suggested to me to rewrite libm-test so that it contains platform
> specific epsilons and told me some ideas.  I've discussed this further 
> with Andreas Schwab and like to tell you the current state of our
> discussion.  I haven't implemented anything yet - I first like to
> discuss the ideas before I start.
...
> I'd also like to enhance libm-test so that it generates automatically
> the "Epsilons" file - this allows easier switching to the new format
> and helps with porting to new targets.
> 
> What do you think?

Wouldn't it be better to decide what error tolerances we want to allow
for the math functions _first_, then see what the math functions
actually produce?

It just seems that basing the test cases on what the math functions
produce (as is done now in practise, and will be done formally with
your scheme) is doing it the wrong way around.

A lot of the libm functions have the maximum error carefully
calculated and visible in their headers.  If those were exceeded, that
would be a bad thing (it probably means a compiler bug).

We could even document the error bounds, so that people using the libc
would know what to expect...

-- 
Geoffrey Keating <geoffk@cygnus.com>

Ideas for rewrite of libm-test

[Andreas Jaeger]

Uli suggested to me to rewrite libm-test so that it contains platform
specific epsilons and told me some ideas.  I've discussed this further 
with Andreas Schwab and like to tell you the current state of our
discussion.  I haven't implemented anything yet - I first like to
discuss the ideas before I start.


libm-test.c will be slightly changed to give us the chance for
automatic building of platform specific files.  The platform specific
files will generated by a perl script.  Since we don't expect that
perl is installed on user system, all generated files are added to
glibc.

I think it's best to start with an example explaining the ideas we
had:

These lines from libm-test.c:
  result = FUNC(modf) (89.6, &intpart);
  check_eps ("modf (89.6, &x) returns 0.6", result, 0.6,
	     CHOOSE (6e-15L, 6e-15, 2e-6));
  check ("modf (89.6, &x) sets x to 89", intpart, 89);

will be changed for libm-test.template to:
  result = FUNC(modf) (89.6, &intpart);
  check ("modf (89.6, &x) returns 0.6", result, 0.6, REPLACE);
  check ("modf (89.6, &x) sets x to 89", intpart, 89, REPLACE);


Platform specific epsilons will be generated (please note that the
epsilons are just examples!):

Epsilons (for alpha, lives in sysdeps/alpha/Epsilons):
Test "modf (89.6, &x) returns 0.6":
float: 1e-7
Test "modf (89.6, &x) sets x to 89":
float: fail

Epsilons (for i386, lives in sysdeps/i386/Epsilons):
Test "modf (89.6, &x) returns 0.6":
ldouble: 1e-16

Epsilons (for i686, lives in sysdeps/i386/i686/Epsilons):
Test "modf (89.6, &x) returns 0.6":
double: 1e-14

Epsilons (generic, might live in sysdeps/ieee754/Epsilons or
sysdeps/generic/Epsilons):
Test "modf (89.6, &x) sets x to 89":
double: 1e15

Some notes:
- The tag for the tests is the string that's used in the call to check.
- The special word "fail" allows us to mark known failures, e.g. complex
  float support on alpha.
- We could easily extend this to hardware platform and operating
  system files if this becomes neccessary.

A perl script will read libm-test.template and all Epsilons that are
relevant for a platform (like the reading of Versions) and generate a
number of files:
- libm-test.c and libm-test.h
- platform specific files epsilon.h 
libm-test.c includes the platform specific epsilon.h and then libm-test.h

The example fragment of libm-test.c might be transformed to:
  result = FUNC(modf) (89.6, &intpart);
  check ("modf (89.6, &x) returns 0.6", result, 0.6, DELTA47, FAIL47);
  check ("modf (89.6, &x) sets x to 89", intpart, 89, DELTA48, FAIL48);

DELTAxx and FAILxx are numbered automatically.

The epsilon.h fragments are:

epsilon.h (for sysdeps/alpha):
#define DELTA47 CHOOSE(0,1e15,1e-7)
#define FAIL48 CHOOSE(0,0,1)

epsilon.h (for sysdeps/i386/i686):
#define DELTA47 CHOOSE(0,1e-14,0)

epsilon.h (for sysdeps/i386):
#define DELTA47 CHOOSE(1e-16,1e15,0)

libm-test.h contains a fall back for all macros since the epsilon.h
files only contain the needed macros:
#ifndef DELTA47
# define DELTA47 0
#endif
#ifndef DELTA48
# define DELTA48 0
#endif
#ifndef FAIL47
# define FAIL47 0
#endif
#ifndef FAIL48
# define FAIL48 0
#endif

I'd also like to enhance libm-test so that it generates automatically
the "Epsilons" file - this allows easier switching to the new format
and helps with porting to new targets.

What do you think?

Andreas
-- 
 Andreas Jaeger   aj@suse.de    aj@arthur.rhein-neckar.de
  for pgp-key finger ajaeger@aixd1.rhrk.uni-kl.de