Re: GCC and Floating-Point

Re: GCC and Floating-Point

[Uros Bizjak]

Hello!

> I'm writing an extensive article about floating-point programming on
> Linux, including a focus on GCC's compilers. This is an outgrowth of
> many debates about topics like -ffast-math and -mfpmath=sse|387, and I
> hope it will prove enlightening for myself and others.

  I would like to point out that for applications that crunch data from real
world (no infinites or nans, where precision is not critical) such as various
simulations, -ffast-math is something that can speed up application a lot.

Regarding i387, -ffast-math does following:

- because NaNs and infinets are not present, fp compares are simplified, as
there is no need for bypases and secondary compares (due to C0 C2 and C3 bits of
FP status word) and cmove instruction can be used in all cases.

- simple builtin functions (sqrt, sin and cos, etc) can be used with hardware
implemented fsqrt, fsin and fcos insn instead of calling library functions.
These can handle arguments up to (something)*2^63, so it is quite enough for
normal apps. This way, a call overhead (saving all FP registers, overhead of
call insn itself) is eliminated, and as an added bonus, sin and cos of the same
argument are combined as fsincos insn.

- not-so-simple (I won't use the word complex here :)) builtin functions (exp,
asin, etc) are expanded on RTL level and CSE is used to eliminate duplicate
calculations.

- floor and ceil functions are implemented as builtin functions and further 
simplified to direct conversion, for example:
(int)floor(double) -> __builtin_lfloor(double).
__builtin_lfloor (and similar builtins) can be implemented directly in i387
using fist(p) insn with appropriate rounding control bits set in control word.

- in addition to this target specific effects, various (otherwise unsafe)
transformations are enabled on middle-level when -ffast-math is used.

  Due to outdated i386 ABI, where all FP parameters are passed on stack, SSE
code does not show all its power when used. When math library function is
called, SSE regs are pushed on stack and called math library function (that is
currently implemented again with i387 insns) pulls these values from stack to
x87 registers. In contrast, x86_64 ABI specifies that FP values are passed in
SSE registers, so they avoid costly SSE reg->stack moves. Until i386 ABI
(together with supporting math functions) is changed to something similar to
x86_64, use of -mfpmath=sse won't show all its power. Another fact is, that x87
intrinsics are currently disabled for -mfpmath=sse, because it was shown that
SSE math libraries (with SSE ABI) are faster for x86_64. Somehow annoying fact
is, that intrinsics are disabled also for i386, where we are still waiting for
ABI to change ;) [Please note that use of SSE intrinsic functions does not rely
on -mfpmath=... settings].

  So, for real-world applications, using i387 with -ffast-math could be
substantially faster than using SSE code. However, the problem lies in math
library headers. These define a lot of inlined asm functions in mathinline.h
header (included when math.h is used). These functions interfear with gcc's
builtins, so -D__NO_MATH_INLINES is needed to fix this problem. The situation is
even worse when SSE code is used. Asm inlines from mathinline.h are implemented
using i387 instructions, so these instructions force parameters to move from SSE
registers to x87 regs (via stack) and the result to move back to SSE reg the
same way. This can be seen when sin(a + 1.0) is compiled with math.h header
included. With -mfpmath=sse, SSE->mem->FP reg moves are needed to satisfy
constraints of inlined sin() code.

  Because SSE->x87 moves are costly, -mfpmath=sse,387 produces unoptimal code.
This option in fact confuses register allocator, and wrong register set is
choosen sometimes. As there is no separate resources for SSE and x87
instructions, the use of -mfpmath=sse,387 is a bit questionable. However, with
-mfpmath=sse, x87 registers could be used for temporary storage in MEM->MEM
moves, they can even do conversions from (double)<->(float) on the fly...

  As an example for this writing, try to benchmark povray with the combination
of following parameters:

-ffast-math
-mpfmath=sse, -mfpmath=387 (and perhaps -mfpmath=387,sse)
-D__NO_MATH_INLINES (this depends on the version of your libc)

Uros.

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-24 09:04:11 +0200, Uros Bizjak wrote:
>   I would like to point out that for applications that crunch data
> from real world (no infinites or nans, where precision is not
> critical) such as various simulations, -ffast-math is something that
> can speed up application a lot.

But note that even when precision is not critical, you may get
consistency problems or annoying unintuitive side effects. For
instance, see

  http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15134

where

float x = 30.0;
int main()
{
  if ( 90.0/x != 3.0)
    abort();
  return 0;
}

fails with -ffast-math (on x86). I would not recommend it, unless
the user knows all the consequences.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point

[chris jefferson]

Vincent Lefevre wrote:

>On 2005-05-24 09:04:11 +0200, Uros Bizjak wrote:
>  
>
>>  I would like to point out that for applications that crunch data
>>from real world (no infinites or nans, where precision is not
>>critical) such as various simulations, -ffast-math is something that
>>can speed up application a lot.
>>    
>>
>
>But note that even when precision is not critical, you may get
>consistency problems or annoying unintuitive side effects. For
>instance, see
>
>  http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15134
>
>where
>
>float x = 30.0;
>int main()
>{
>  if ( 90.0/x != 3.0)
>    abort();
>  return 0;
>}
>
>fails with -ffast-math (on x86). I would not recommend it, unless
>the user knows all the consequences.
>
>  
>
On the other hand, in general using != and == on floating point numbers 
is always dangerous if you do not know all the consequences. For 
example, on your above program if I use 30.1 and 90.3, the program fails 
without -ffast-math.

Chris

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-25 15:22:31 +0100, chris jefferson wrote:
> On the other hand, in general using != and == on floating point
> numbers is always dangerous if you do not know all the consequences.
> For example, on your above program if I use 30.1 and 90.3, the
> program fails without -ffast-math.

Yes, but != and == are not the only problems with floating point.
Any programmer who uses floating-point should have read Goldberg's
article. However, -ffast-math is particularly evil since it turns
a correct program (when assuming some basic properties of the
floating-point system[*]) into an incorrect one.

[*] that have not always been true, but just to give some sense to
what one has written.

BTW, other interesting recent articles:

  http://csdl.computer.org/dl/mags/co/2005/05/r5091.pdf
  "An Open Question to Developers of Numerical Software", by
  W. Kahan and D. Zuras

  http://www.cs.berkeley.edu/~wkahan/Mindless.pdf
  "How Futile are Mindless Assessments of Roundoff in Floating-Point
  Computation?", by W. Kahan

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

RE: GCC and Floating-Point

[Dave Korn]

----Original Message----
>From: chris jefferson
>Sent: 25 May 2005 15:23

> Vincent Lefevre wrote:
 

>> 
>> float x = 30.0;
>> int main()
>> {
>>  if ( 90.0/x != 3.0)
>>    abort();
>>  return 0;
>> }
>> 
>> fails with -ffast-math (on x86). I would not recommend it, unless
>> the user knows all the consequences.
>> 
>> 
>> 
> On the other hand, in general using != and == on floating point numbers
> is always dangerous if you do not know all the consequences. For
> example, on your above program if I use 30.1 and 90.3, the program fails
> without -ffast-math.

  I second that.  It's simply not valid to use != or == to compare floating
point numbers; the standard idiom is and always has been to subtract them
and then see if the delta is less than or greater than some suitably-chosen
epsilon.  You have to regard floating point numbers as if they were
more-or-less random in the lowest few bits after any amount of computation
on them; the chance of getting an exact match is small.

  So never mind -ffast-math; I wouldn't recommend using floating point *at
all* unless the user is well aware of the consequences!


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-25 15:59:28 +0100, Dave Korn wrote:
> ----Original Message----
> >From: chris jefferson
> >Sent: 25 May 2005 15:23
> > On the other hand, in general using != and == on floating point numbers
> > is always dangerous if you do not know all the consequences. For
> > example, on your above program if I use 30.1 and 90.3, the program fails
> > without -ffast-math.
> 
>   I second that. It's simply not valid to use != or == to compare
> floating point numbers;

No, it is valid. Whether you should do that or use another method
depends on the context. In particular, if you can make sure that your
computations are exact, there's no problem in using != or ==. You
can also choose to control the rounding (when possible).

> the standard idiom is and always has been to subtract them and then
> see if the delta is less than or greater than some suitably-chosen
> epsilon. You have to regard floating point numbers as if they were
> more-or-less random in the lowest few bits after any amount of
> computation on them; the chance of getting an exact match is small.

If you do computational geometry, such kinds of things would not be
sufficient.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point

[Allan Sandfeld Jensen]

On Wednesday 25 May 2005 16:22, chris jefferson wrote:
>
> On the other hand, in general using != and == on floating point numbers
> is always dangerous if you do not know all the consequences. For
> example, on your above program if I use 30.1 and 90.3, the program fails
> without -ffast-math.
>
Yes. I still don't understand why gcc doesn't do -ffast-math by default like 
all other compilers. The people who needs perfect standard behavior are a lot 
fewer than all the packagers who doesn't understand which optimization flags 
gcc should _always_ be called with.

`Allan

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-25 19:27:21 +0200, Allan Sandfeld Jensen wrote:
> Yes. I still don't understand why gcc doesn't do -ffast-math by
> default like all other compilers.

No! And I really don't think that other compilers do that.
It would be very bad, would not conform to the C standard[*]
and would make lots of codes fail.

[*] See for instance:

       5.1.2.3  Program execution
[...]
       [#14] EXAMPLE 5 Rearrangement for floating-point expressions
       is  often  restricted because of limitations in precision as
       well as range.  The implementation  cannot  generally  apply
       the   mathematical   associative   rules   for  addition  or
       multiplication,  nor  the  distributive  rule,  because   of
       roundoff   error,  even  in  the  absence  of  overflow  and
       underflow.   Likewise,  implementations   cannot   generally
       replace decimal constants in order to rearrange expressions.
       In  the  following  fragment,  rearrangements  suggested  by
       mathematical rules for real numbers are often not valid (see
       F.8).

               double x, y, z;
               /* ... */
               x = (x * y) * z;  // not equivalent to x *= y * z;
               z = (x - y) + y ; // not equivalent to z = x;
               z = x + x * y;    // not equivalent to z = x * (1.0 + y);
               y = x / 5.0;      // not equivalent to y = x * 0.2;

> The people who needs perfect standard behavior are a lot fewer than
> all the packagers who doesn't understand which optimization flags
> gcc should _always_ be called with.

Standard should be the default.

(Is this a troll or what?)

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point (A proposal)

[Scott Robert Ladd]

Allan Sandfeld Jensen wrote:
>>Yes. I still don't understand why gcc doesn't do -ffast-math by
>>default like all other compilers.

Vincent Lefevre wrote:
> No! And I really don't think that other compilers do that.
> It would be very bad, would not conform to the C standard[*]
> and would make lots of codes fail.

Perhaps what needs to be changed is the definition of -ffast-math
itself. Some people (myself included) view it from the standpoint of
using the full capabilities of our processors' hardware intrinsics;
however, -ffast-math *also* implies the rearrangement of code that
violates Standard behavior. Thus it does two things that perhaps should
not be combined.

To be more pointed, it is -funsafe-math-optimizations (implied by
-ffast-math) that is in need of adjustment.

May I be so bold as to suggest that -funsafe-math-optimizations be
reduced in scope to perform exactly what it's name implies:
transformations that may slightly alter the meanding of code. Then move
the use of hardware intrinsics to a new -fhardware-math switch.

Does anyone object if I experiment a bit with this modification? Or am I
completely wrong in my understanding?

..Scott

Re: GCC and Floating-Point (A proposal)

[Richard Guenther]

On 5/26/05, Scott Robert Ladd <scott.ladd@coyotegulch.com> wrote:
> Allan Sandfeld Jensen wrote:
> >>Yes. I still don't understand why gcc doesn't do -ffast-math by
> >>default like all other compilers.
> 
> Vincent Lefevre wrote:
> > No! And I really don't think that other compilers do that.
> > It would be very bad, would not conform to the C standard[*]
> > and would make lots of codes fail.
> 
> Perhaps what needs to be changed is the definition of -ffast-math
> itself. Some people (myself included) view it from the standpoint of
> using the full capabilities of our processors' hardware intrinsics;
> however, -ffast-math *also* implies the rearrangement of code that
> violates Standard behavior. Thus it does two things that perhaps should
> not be combined.
> 
> To be more pointed, it is -funsafe-math-optimizations (implied by
> -ffast-math) that is in need of adjustment.
> 
> May I be so bold as to suggest that -funsafe-math-optimizations be
> reduced in scope to perform exactly what it's name implies:
> transformations that may slightly alter the meanding of code. Then move
> the use of hardware intrinsics to a new -fhardware-math switch.

I think the other options implied by -ffast-math apart from
-funsafe-math-optimizations should (and do?) enable the use of
hardware intrinsics already.  It's only that some of the optimzations
guarded by -funsafe-math-optimizations could be applied in general.
A good start may be to enumerate the transformations done on a
Wiki page and list the flags it is guarded with.

Richard.

Re: GCC and Floating-Point

[Daniel Berlin]

On Thu, 26 May 2005, Vincent Lefevre wrote:

> On 2005-05-25 19:27:21 +0200, Allan Sandfeld Jensen wrote:
>> Yes. I still don't understand why gcc doesn't do -ffast-math by
>> default like all other compilers.
>
> No! And I really don't think that other compilers do that.

Have you looked, or are you just guessing?

I know for a fact that XLC does it at -O3+, and unless i'm 
misremembering, icc does it at -O2+.

Both require flags to turn the behavior *off* at those opt levels.

XLC will give you a warning when it sees itself making an optimization 
that may affect precision, saying that if you don't want this to happen, 
to use a flag.

Re: GCC and Floating-Point (A proposal)

[Scott Robert Ladd]

Scott Robert Ladd <scott.ladd@coyotegulch.com> wrote:
>>May I be so bold as to suggest that -funsafe-math-optimizations be
>>reduced in scope to perform exactly what it's name implies:
>>transformations that may slightly alter the meanding of code. Then move
>>the use of hardware intrinsics to a new -fhardware-math switch.

Richard Guenther wrote:
> I think the other options implied by -ffast-math apart from
> -funsafe-math-optimizations should (and do?) enable the use of
> hardware intrinsics already.  It's only that some of the optimzations
> guarded by -funsafe-math-optimizations could be applied in general.
> A good start may be to enumerate the transformations done on a
> Wiki page and list the flags it is guarded with.

Unless I've missed something obvious, -funsafe-math-optimizations alone
enables most hardware floating-point intrinsics -- on x86_64 and x86, at
least --. For example, consider a simple line of code that takes the
sine of a constant:

	x = sin(1.0);

On the Pentium 4, with GCC 4.0, various command lines produced the
following code:

    gcc -S -O3 -march=pentium4

	movl	$1072693248, 4(%esp)
	call	sin
	fstpl	4(%esp)

    gcc -S -O3 -march=pentium4 -D__NO_MATH_INLINES

	movl	$1072693248, 4(%esp)
	call	sin
	fstpl	4(%esp)

    gcc -S -O3 -march=pentium4 -funsafe-math-optimizations

	fld1
	fsin
	fstpl	4(%esp)

    gcc -S -O3 -march=pentium4 -funsafe-math-optimizations \
                                  -D__NO_MATH_INLINES

	fld1
	fsin
	fstpl	4(%esp)

As you can see, it is -funsafe-math-optimizations alone that determines
the use of hardware intrinsics, on the P4 at least.

As a side note, GCC 4.0 on the Opteron produces the same result with all
four command-line variations:

    gcc -S -O3 -march=k8
	movlpd	.LC2(%rip), %xmm0
	call	sin

    gcc -S -O3 -march=k8 -D__NO_MATH_INLINES
	movlpd	.LC2(%rip), %xmm0
	call	sin

    gcc -S -O3 -march=k8 -funsafe-math-optimizations
	movlpd	.LC2(%rip), %xmm0
	call	sin

    gcc -S -O3 -march=k8 -funsafe-math-optimizations -D__NO_MATH_INLINES
	movlpd	.LC2(%rip), %xmm0
	call	sin


..Scott

Re: GCC and Floating-Point (A proposal)

[Paul Brook]

On Thursday 26 May 2005 14:25, Scott Robert Ladd wrote:
> Scott Robert Ladd <scott.ladd@coyotegulch.com> wrote:
> >>May I be so bold as to suggest that -funsafe-math-optimizations be
> >>reduced in scope to perform exactly what it's name implies:
> >>transformations that may slightly alter the meanding of code. Then move
> >>the use of hardware intrinsics to a new -fhardware-math switch.
>
> Richard Guenther wrote:
> > I think the other options implied by -ffast-math apart from
> > -funsafe-math-optimizations should (and do?) enable the use of
> > hardware intrinsics already.  It's only that some of the optimzations
> > guarded by -funsafe-math-optimizations could be applied in general.
> > A good start may be to enumerate the transformations done on a
> > Wiki page and list the flags it is guarded with.
>
> Unless I've missed something obvious, -funsafe-math-optimizations alone
> enables most hardware floating-point intrinsics -- on x86_64 and x86, at
> least --. For example, consider a simple line of code that takes the
> sine of a constant:

I thought the x86 sin/cos intrinsics were unsafe. ie. they don't gave accurate 
results in all cases.

Paul

Re: GCC and Floating-Point (A proposal)

[Gabriel Dos Reis]

Paul Brook <paul@codesourcery.com> writes:

| On Thursday 26 May 2005 14:25, Scott Robert Ladd wrote:
| > Scott Robert Ladd <scott.ladd@coyotegulch.com> wrote:
| > >>May I be so bold as to suggest that -funsafe-math-optimizations be
| > >>reduced in scope to perform exactly what it's name implies:
| > >>transformations that may slightly alter the meanding of code. Then move
| > >>the use of hardware intrinsics to a new -fhardware-math switch.
| >
| > Richard Guenther wrote:
| > > I think the other options implied by -ffast-math apart from
| > > -funsafe-math-optimizations should (and do?) enable the use of
| > > hardware intrinsics already.  It's only that some of the optimzations
| > > guarded by -funsafe-math-optimizations could be applied in general.
| > > A good start may be to enumerate the transformations done on a
| > > Wiki page and list the flags it is guarded with.
| >
| > Unless I've missed something obvious, -funsafe-math-optimizations alone
| > enables most hardware floating-point intrinsics -- on x86_64 and x86, at
| > least --. For example, consider a simple line of code that takes the
| > sine of a constant:
| 
| I thought the x86 sin/cos intrinsics were unsafe. ie. they don't
| gave accurate results in all cases.

Indeed.

-- Gaby

Re: GCC and Floating-Point (A proposal)

[Scott Robert Ladd]

Paul Brook wrote:
> I thought the x86 sin/cos intrinsics were unsafe. ie. they don't gave accurate 
> results in all cases.

If memory serves, Intel's fsin (for example) has an error > 1 ulp for
value flose to multiples of pi (2pi, for example).

Now, I'm not certain this is true for the K8 and later Pentiums. Looks
like I need to run another round of tests. ;)

..Scott

Re: GCC and Floating-Point

[Allan Sandfeld Jensen]

On Thursday 26 May 2005 10:15, Vincent Lefevre wrote:
> On 2005-05-25 19:27:21 +0200, Allan Sandfeld Jensen wrote:
> > Yes. I still don't understand why gcc doesn't do -ffast-math by
> > default like all other compilers.
>
> No! And I really don't think that other compilers do that.

I can't speak of all compilers, only the ones I've tried. ICC enables it 
always, Sun CC, Dec CXX, and HP CC at certain levels of optimizations 
(equivalent to -O2). 

Basically any compiler that cares about benchmarks have it enabled by default.

Many of them however have multiple levels of relaxed floating point. The 
lowest levels will try to be as accurate as possible, while the higher will 
loosen the accuracy and just try to be as fast as possible.

>
> > The people who needs perfect standard behavior are a lot fewer than
> > all the packagers who doesn't understand which optimization flags
> > gcc should _always_ be called with.
>
> Standard should be the default.
>
> (Is this a troll or what?)

So why isn't -ansi or -pendantic default?



`Allan

Re: GCC and Floating-Point

[Scott Robert Ladd]

Allan Sandfeld Jensen wrote:
> Basically any compiler that cares about benchmarks have it enabled by default.
> 
> Many of them however have multiple levels of relaxed floating point. The 
> lowest levels will try to be as accurate as possible, while the higher will 
> loosen the accuracy and just try to be as fast as possible.

Perhaps we need something along these lines:

When -ansi or -pedantic is used, the compiler should disallow anything
"unsafe" that may break compliance, warning if someone uses a paradox
like "-ansi -funsafe-math-optimizations".

As has been pointed out elsewhere in this thread,
-funsafe-math-optimizations implies too many different things, and is
vaguely documented. I'd like to see varying levels of floating-point
optimization, including an option that uses an internal library
optimized for both speed and correctness, which are not mutually exclusive.

..Scott

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-26 09:14:40 -0400, Daniel Berlin wrote:
> On Thu, 26 May 2005, Vincent Lefevre wrote:
> >On 2005-05-25 19:27:21 +0200, Allan Sandfeld Jensen wrote:
> >>Yes. I still don't understand why gcc doesn't do -ffast-math by
> >>default like all other compilers.
> >
> >No! And I really don't think that other compilers do that.
> 
> Have you looked, or are you just guessing?

Just guessing (I said "I don't think").

> I know for a fact that XLC does it at -O3+, and unless i'm 
> misremembering, icc does it at -O2+.

But do they use -O3 and -O2 by default?

Even if they don't, maths optimization breaking IEEE-754 (if they
claim to support it) is bad, IMHO, but well...

I don't have these compilers to test them. If you have simple C code
showing how they "break" it, I'd be interesting...

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point

[Daniel Berlin]

On Fri, 2005-05-27 at 12:47 +0200, Vincent Lefevre wrote:
> On 2005-05-26 09:14:40 -0400, Daniel Berlin wrote:
> > On Thu, 26 May 2005, Vincent Lefevre wrote:
> > >On 2005-05-25 19:27:21 +0200, Allan Sandfeld Jensen wrote:
> > >>Yes. I still don't understand why gcc doesn't do -ffast-math by
> > >>default like all other compilers.
> > >
> > >No! And I really don't think that other compilers do that.
> > 
> > Have you looked, or are you just guessing?
> 
> Just guessing (I said "I don't think").
> 
> > I know for a fact that XLC does it at -O3+, and unless i'm 
> > misremembering, icc does it at -O2+.
> 
> But do they use -O3 and -O2 by default?

ICC does, yes.
I don't remember the XLC default opt level.
As someone else pointed out, DEC, HP,etc, do this stuff by default too.

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-26 20:49:13 +0200, Allan Sandfeld Jensen wrote:
> I can't speak of all compilers, only the ones I've tried. ICC
> enables it always, Sun CC, Dec CXX, and HP CC at certain levels of
> optimizations (equivalent to -O2).

I've tried with "cc: Sun WorkShop 6 2000/04/07 C 5.1", and there
doesn't seem to be unsafe optimizations. Here's an example:

#include <stdio.h>

int main (void)
{
  int i;

  for (i = 2; i >= 0; i--)
    {
      double x = (double) i;
      x = -x;
      printf ("%2g %2g\n", x, x + 0.0);
    }

  for (i = 0; i < 100; i += 30)
    {
      long double x = (long double) i;
      printf ("%.20Lg\n", x / 30.0);
    }

  return 0;
}

craffe:~/src/fp> cc -xO5 -o tst tst.c -lsunmath
craffe:~/src/fp> ./tst
-2 -2
-1 -1
-0  0
0
1
2
3

But with gcc, using -ffast-math:

craffe:~/src/fp> gcc -O -ffast-math -o tst tst.c
craffe:~/src/fp> ./tst
-2 -2
-1 -1
-0 -0
0
1
2
3

See the third line: -0 -0.

And indeed, icc does some unsafe optimizations by default, when
assuming IEEE-754, but nothing buggy...

latour:~/src/fp> /localdisk/icc/opt/intel_cc_80/bin/icc -o tst tst.c
latour:~/src/fp> ./tst
-2 -2
-1 -1
-0 -0
0
1
2
3

contrary to gcc:

latour:~/src/fp> gcc -O -ffast-math -o tst tst.c
latour:~/src/fp> ./tst
-2 -2
-1 -1
-0 -0
0
1
2
3.0000000000000000002

So, yes, -ffast-math by default would really be a bad idea and would
make gcc much worse than other compilers.

> > Standard should be the default.
> >
> > (Is this a troll or what?)
> 
> So why isn't -ansi or -pendantic default?

-ansi is broken.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point

[Scott Robert Ladd]

Vincent Lefevre wrote:
> So, yes, -ffast-math by default would really be a bad idea and would
> make gcc much worse than other compilers.

Who has advocated making -ffast-math the default? I certainly haven't.

..Scott

Re: GCC and Floating-Point (A proposal)

[Vincent Lefevre]

On 2005-05-26 14:40:33 +0100, Paul Brook wrote:
> I thought the x86 sin/cos intrinsics were unsafe. ie. they don't
> gave accurate results in all cases.

Yes, and here, this is a bug. See:

http://web.archive.org/web/20040409144725/http://www.naturalbridge.com/floatingpoint/intelfp.html

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-27 07:58:28 -0400, Scott Robert Ladd wrote:
> Vincent Lefevre wrote:
> > So, yes, -ffast-math by default would really be a bad idea and would
> > make gcc much worse than other compilers.
> 
> Who has advocated making -ffast-math the default? I certainly haven't.

Allan Sandfeld Jensen has:

From: Allan Sandfeld Jensen <linux@carewolf.com>
Message-Id: <200505251927.22272.linux@carewolf.com>
| Yes. I still don't understand why gcc doesn't do -ffast-math by default like
| all other compilers. The people who needs perfect standard behavior are a lot
| fewer than all the packagers who doesn't understand which optimization flags
| gcc should _always_ be called with.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: GCC and Floating-Point

[Allan Sandfeld Jensen]

On Friday 27 May 2005 13:51, Vincent Lefevre wrote:
> So, yes, -ffast-math by default would really be a bad idea and would
> make gcc much worse than other compilers.
>
Thanks for the tests. I had no idea GCCs fast-math was that different from  
other compilers.

 Maybe the real goal like other have mentioned should be to divide the 
-ffast-math into multiple switches.

`Allan

Re: GCC and Floating-Point

[Paul Koning]

>>>>> "Allan" == Allan Sandfeld Jensen <linux@carewolf.com> writes:

 Allan> Maybe the real goal like other have mentioned should be to
 Allan> divide the -ffast-math into multiple switches.

Yes, and document them so both users and implementers can tell what
they mean, which is not currently the case.

     paul

Re: GCC and Floating-Point

[Scott Robert Ladd]

Paul Koning wrote:
>  Allan> Maybe the real goal like other have mentioned should be to
>  Allan> divide the -ffast-math into multiple switches.
> 
> Yes, and document them so both users and implementers can tell what
> they mean, which is not currently the case.

I'm more than willing to do this, and relatively quickly, but as I said
in a prior message, I need some sort of assurance that I wouldn't be
wasting my time.

I can post a suggested set of switches and their meanings? We could
actually document what the switches do *before* implementing them.

..Scott

Re: GCC and Floating-Point

[Richard Guenther]

On 5/27/05, Scott Robert Ladd <scott.ladd@coyotegulch.com> wrote:
> Paul Koning wrote:
> >  Allan> Maybe the real goal like other have mentioned should be to
> >  Allan> divide the -ffast-math into multiple switches.
> >
> > Yes, and document them so both users and implementers can tell what
> > they mean, which is not currently the case.
> 
> I'm more than willing to do this, and relatively quickly, but as I said
> in a prior message, I need some sort of assurance that I wouldn't be
> wasting my time.
>
> I can post a suggested set of switches and their meanings? We could
> actually document what the switches do *before* implementing them.

Well, as I said before, actually enumerating the transformations we
currently do would be a start.  And it would aid at finding useful
subclasses of transformations.  As well as then having documentation
by enumerating of (current) transformations together with the switches
that affect them.

Richard.

Re: GCC and Floating-Point

[Scott Robert Ladd]

Richard Guenther wrote:
> On 5/27/05, Scott Robert Ladd <scott.ladd@coyotegulch.com> wrote:
>>I can post a suggested set of switches and their meanings? We could
>>actually document what the switches do *before* implementing them.
> 
> Well, as I said before, actually enumerating the transformations we
> currently do would be a start.  And it would aid at finding useful
> subclasses of transformations.  As well as then having documentation
> by enumerating of (current) transformations together with the switches
> that affect them.

This is where I've already started, just trying to get an idea of what
is going on inside the compiler. Is there any documentation of this sort
already? I think I know the answer to that question, but figure it
doesn't hurt to ask... ;)

I'm likely to pull my old UltraSPARC out of retirement (it hasn't been
turne don in months), just so I have something other than x86_64 and x86
to look at.

..Scott

Re: GCC and Floating-Point

[Paul Koning]

>>>>> "Scott" == Scott Robert Ladd <scott.ladd@coyotegulch.com> writes:

 Scott> Paul Koning wrote:
 Allan> Maybe the real goal like other have mentioned should be to
 Allan> divide the -ffast-math into multiple switches.
 >> Yes, and document them so both users and implementers can tell
 >> what they mean, which is not currently the case.

 Scott> I'm more than willing to do this, and relatively quickly, but
 Scott> as I said in a prior message, I need some sort of assurance
 Scott> that I wouldn't be wasting my time.

 Scott> I can post a suggested set of switches and their meanings? We
 Scott> could actually document what the switches do *before*
 Scott> implementing them.

I would support that.  Given some proposal with discernable semantics,
it's possible to have a discussion on whether those semantics are the
right ones.

      paul

Re: GCC and Floating-Point

[Toon Moene]

Allan Sandfeld Jensen wrote:

> On Friday 27 May 2005 13:51, Vincent Lefevre wrote:
> 
>>So, yes, -ffast-math by default would really be a bad idea and would
>>make gcc much worse than other compilers.

> Thanks for the tests. I had no idea GCCs fast-math was that different from  
> other compilers.
> 
>  Maybe the real goal like other have mentioned should be to divide the 
> -ffast-math into multiple switches.

Good Luck :-)

	http://gcc.gnu.org/ml/gcc/2001-08/msg00368.html

(it's only four years ago - I can also show you my contributions in this 
thread in 1999, i.e., the previous millennium).

-- 
Toon Moene - e-mail: toon@moene.indiv.nluug.nl - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
A maintainer of GNU Fortran 95: http://gcc.gnu.org/fortran/
News on GNU Fortran 95: http://gfortran.org/

Re: GCC and Floating-Point

[Scott Robert Ladd]

Toon Moene wrote:
> Good Luck :-)
> 
>     http://gcc.gnu.org/ml/gcc/2001-08/msg00368.html
> 
> (it's only four years ago - I can also show you my contributions in this
> thread in 1999, i.e., the previous millennium).

Just out of curiosity, why did previous efforts fail in this regard? Was
it simply too much effort to identify all the transformations? Did the
GCC community fail to come to a consensus? Or was it simply -- as I'm
coming to suspect -- that the work involved is not justified by the result?

I do know this: Many, many scientific and mathematical programmers find
GCC frustrating and annoying, and most of those folk know far more about
numbers than I do. I wish more of these people would feel comfortable
posting to the GCC list, rather than sending private e-mails to my
inbox. However, the atmosphere of GCC development is... well, let's just
say that my investment in asbestos underware has not been wasted. ;)

..Scott

Re: GCC and Floating-Point

[Joseph S. Myers]

On Sat, 28 May 2005, Scott Robert Ladd wrote:

> Toon Moene wrote:
> > Good Luck :-)
> > 
> >     http://gcc.gnu.org/ml/gcc/2001-08/msg00368.html
> > 
> > (it's only four years ago - I can also show you my contributions in this
> > thread in 1999, i.e., the previous millennium).
> 
> Just out of curiosity, why did previous efforts fail in this regard? Was
> it simply too much effort to identify all the transformations? Did the
> GCC community fail to come to a consensus? Or was it simply -- as I'm
> coming to suspect -- that the work involved is not justified by the result?

They didn't fail altogether; -ffast-math was split into multiple options 
(-funsafe-math-optimizations -fno-trapping-math etc.) in March 2001, 
following an analysis of everything that checked flag_fast_math 
<http://gcc.gnu.org/ml/gcc-patches/2000-12/msg00690.html>.

Unfortunately this never got followed up properly by defining separate 
preprocessor macros for each option, so if you use the individual options 
then glibc headers checking __FAST_MATH__ may not enable optimizations 
which would in fact be safe with the subset of options specified.  And as 
the current discussion illustrates, -funsafe-math-optimizations itself has 
become a catch-all for many different optimizations, such as -ffast-math 
was.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: GCC and Floating-Point

[Scott Robert Ladd]

Joseph S. Myers wrote:
> They didn't fail altogether; -ffast-math was split into multiple 
> options (-funsafe-math-optimizations -fno-trapping-math etc.) in 
> March 2001, following an analysis of everything that checked 
> flag_fast_math 
> <http://gcc.gnu.org/ml/gcc-patches/2000-12/msg00690.html>.
> 
> Unfortunately this never got followed up properly by defining 
> separate preprocessor macros for each option, so if you use the 
> individual options then glibc headers checking __FAST_MATH__ may not 
> enable optimizations which would in fact be safe with the subset of 
> options specified.

Which implies, of course, that breaking -funsafe-math-optimizations into
components involves modifying glibc for corresponding __???_MATH__
definitions.

The separation of glibc and GCC leads to many problems not faced by
commercial compilers; not only can a commercial compiler focus on a
single platform, it also ships a C library as an integral part of the
compiler package. Thus a commercial compiler, with a narrow focus, can
ensure that the C library tracks changes in the compiler, while such is
not the case with GCC.

GCC ships with an integral Standard C++ library; it seems to me that it
could also provide a Standard C90/99 library as well. But I suspect that
is a kettle of fish that should stay closed. ;)

> And as the current discussion illustrates, 
> -funsafe-math-optimizations itself has become a catch-all for many 
> different optimizations, such as -ffast-math was.

Breaking up -funsafe-math-optimizations would also require a policy for
how future optimizations should be organized, lest we end up right where
we started with one option that implies many things.

..Scott

Re: GCC and Floating-Point

[Joseph S. Myers]

On Sat, 28 May 2005, Scott Robert Ladd wrote:

> The separation of glibc and GCC leads to many problems not faced by
> commercial compilers; not only can a commercial compiler focus on a
> single platform, it also ships a C library as an integral part of the
> compiler package. Thus a commercial compiler, with a narrow focus, can
> ensure that the C library tracks changes in the compiler, while such is
> not the case with GCC.

People who care about glibc in this regard make the necessary changes.  
People not concerned with it do not need to do so; they just change the 
compiler and leave the glibc changes for others.  In practice glibc gets 
updated fine for changes in the compiler, e.g.:

2004-05-27  Jakub Jelinek  <jakub@redhat.com>

        * sysdeps/i386/fpu/bits/mathinline.h (__expm1_code): Define using
        __builtin_expm1l for GCC 3.5+.
        (__expl): Define using __builtin_expl for GCC 3.4+.
        (exp, expf, expl): Don't define for GCC 3.4+.
        (tan, tanf, tanl): Don't define for GCC 3.5+.
        (__atan2l): Define using __builtin_atan2l for GCC 3.4+.
        (atan2, atan2f, atan2l): Don't define for GCC 3.4+ or !__FAST_MATH__.
        (fmod, fmodf, fmodl): Don't define for GCC 3.5+ or !__FAST_MATH__.
        (fabsf, fabsl): Only provide if __USE_MISC or __USE_ISOC99.
        (sin, sinf, sinl, cos, cosf, cosl, log, logf, logl): Don't define
        for GCC 3.4+.
        (log10, log10f, log10l, asin, asinf, asinl, acos, acosf, acosl):
        Don't define for GCC 3.5+.
        (atan, atanf, atanl): Don't define for GCC 3.4+ or !__FAST_MATH__.
        (log1p, log1pf, log1pl, logb, logbf, logbl, log2, log2f, log2l): Don't
        define for GCC 3.5+.
        (drem, dremf, dreml): Don't define for GCC 3.5+ or !__FAST_MATH__.
        * sysdeps/sparc/fpu/bits/mathinline.h (sqrt, sqrtf, sqrtl): Don't
        define for GCC 3.2+.

It is not hard to get this sort of clean patch accepted for glibc.  If 
more precise macros were added to GCC to define what optimizations are OK 
it should be easy to get a patch to use them into glibc, probably 
including the 2.3 stable branch.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: GCC and Floating-Point

[Marc Espie]

In article <20050525145835.GA5967@ay.vinc17.org> you write:
>  http://csdl.computer.org/dl/mags/co/2005/05/r5091.pdf
>  "An Open Question to Developers of Numerical Software", by
>  W. Kahan and D. Zuras

Doesn't look publically accessible from my machine...

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-29 19:22:57 +0200, Marc Espie wrote:
> In article <20050525145835.GA5967@ay.vinc17.org> you write:
> >  http://csdl.computer.org/dl/mags/co/2005/05/r5091.pdf
> >  "An Open Question to Developers of Numerical Software", by
> >  W. Kahan and D. Zuras
> 
> Doesn't look publically accessible from my machine...

Hmm... yes, I think you need to be in a lab that has subscribed
to the publications by the IEEE Computer Society.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

RE: GCC and Floating-Point

[Menezes, Evandro]

Uros, 

> > Actually, in many cases, SSE did help x86 performance as 
> well.  That 
> > happens in FP-intensive applications which spend a lot of time in 
> > loops when the XMM register set can be used more 
> efficiently than the x87 stack.
> 
>   This code could be a perfect example how XMM register file 
> beats x87 reg stack.
> However, contrary to all expectations, x87 code is 20% 
> faster(!!) /on p4, but it would be interesting to see this 
> comparison on x86_64, or perhaps on 32bit AMD/.
> The code structure, produced with -mfpmath=sse, is the same 
> as the code structure produced with -mfpmath=x87, so IMO 
> there is no register allocator effects in play.

I'll look into it and share what I see.
 
>   I was trying to look into this problem, but on first sight, 
> code seems optimal to me...

FWIW, here's some old data I got almost 2 years ago (run-times and geometric means of the ratios using SPEC's bases):

CPU2000	A	B
164.gzip	205s	203s
175.vpr	185s	188s
176.gcc	117s	116s
181.mcf	313s	314s
186.crafty	112s	112s
197.parser	268s	268s
252.eon	147s	167s
253.perlbmk	175s	180s
254.gap	148s	148s
255.vortex	178s	178s
256.bzip2	211s	202s
300.twolf	313s	328s
Int Geomean	812	801
177.mesa	173s	187s
179.art	346s	690s
183.equake	163s	162s
188.ammp	325s	336s
FP Geomean	757	620

Using GCC 3.3.3 from 3_3-hammer branch with the options for runs in column B were "-m32 -O3 -march=k8 -ffast-math -fomit-frame-pointer -malign-double +FDO", for column A, the same ones plus "-mfpmath=sse".  The system was a 1.4GHz Athlon 64 with PC2100 RAM.

Because things were so much better with SSE, I haven't run with x87 lately...

-- 
Evandro

Re: GCC and Floating-Point

[Steven G. Johnson]

Vincent Lefevre wrote:
> You may be interested in my web page "Linux and the Extended Precision
> on x86 Processors":
> 
>   http://www.vinc17.org/research/extended.en.html
> 
> Also, comments are welcome.

William Kahan seems to take the opposite viewpoint.  See e.g.:

	http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf

RE: GCC and Floating-Point

[Uros Bizjak]

Hello Evandro!

>> x87 registers. In contrast, x86_64 ABI specifies that FP 
>> values are passed in SSE registers, so they avoid costly SSE 
>> reg->stack moves. Until i386 ABI (together with supporting 
>> math functions) is changed to something similar to x86_64, 
>> use of -mfpmath=sse won't show all its power.

> Actually, in many cases, SSE did help x86 performance as well.  That
> happens in FP-intensive applications which spend a lot of time in loops
> when the XMM register set can be used more efficiently than the x87 stack.

  There is an annoying piece of code attached to PR19780
(http://gcc.gnu.org/bugzilla/show_bug.cgi?id=19780), a loop that shuffles
registers around a lot:

  int i;

  real v1x, v1y, v1z;
  real v2x, v2y, v2z;
  real v3x, v3y, v3z;

  for (i = 0; i < 100000000; i++)
    {
      v3x = v1y * v2z - v1z * v2y;
      v3y = v1z * v2x - v1x * v2z;
      v3z = v1x * v2y - v1y * v2x;

      v1x = v2x;
      v1y = v2y;
      v1z = v2z;

      v2x = v3x;
      v2y = v3y;
      v2z = v3z;
    }

  This code could be a perfect example how XMM register file beats x87 reg stack.
However, contrary to all expectations, x87 code is 20% faster(!!) /on p4, but it
would be interesting to see this comparison on x86_64, or perhaps on 32bit AMD/.
The code structure, produced with -mfpmath=sse, is the same as the code structure produced
with -mfpmath=x87, so IMO there is no register allocator effects in play.

  I was trying to look into this problem, but on first sight, code seems optimal to me...

Uros.

RE: GCC and Floating-Point

[Menezes, Evandro]

Hi, Uros.

> Due to outdated i386 ABI, where all FP parameters are 
> passed on stack, SSE code does not show all its power when 
> used. When math library function is called, SSE regs are 
> pushed on stack and called math library function (that is 
> currently implemented again with i387 insns) pulls these 
> values from stack to
> x87 registers. In contrast, x86_64 ABI specifies that FP 
> values are passed in SSE registers, so they avoid costly SSE 
> reg->stack moves. Until i386 ABI (together with supporting 
> math functions) is changed to something similar to x86_64, 
> use of -mfpmath=sse won't show all its power.

Actually, in many cases, SSE did help x86 performance as well.  That happens in FP-intensive applications which spend a lot of time in loops when the XMM register set can be used more efficiently than the x87 stack.

> Another fact 
> is, that x87 intrinsics are currently disabled for 
> -mfpmath=sse, because it was shown that SSE math libraries 
> (with SSE ABI) are faster for x86_64.

That's because the x87 microcode operates in 80-bit precision, whereas the SSE routines in just 32 or 64-bit precision.  Yet, their precision is better over their domains.

> These functions 
> interfear with gcc's builtins, so -D__NO_MATH_INLINES is 
> needed to fix this problem. The situation is even worse when 
> SSE code is used. Asm inlines from mathinline.h are 
> implemented using i387 instructions, so these instructions 
> force parameters to move from SSE registers to x87 regs (via 
> stack) and the result to move back to SSE reg the same way. 
> This can be seen when sin(a + 1.0) is compiled with math.h 
> header included. With -mfpmath=sse, SSE->mem->FP reg moves 
> are needed to satisfy constraints of inlined sin() code.

Yep, that's a killer especially in x86_64, which defaults to -mfpmath=sse...

Evandro

Re: GCC and Floating-Point

[Vincent Lefevre]

On 2005-05-23 11:14:52 -0400, Scott Robert Ladd wrote:
[...]
> Naturally, due to hardware availability, I'll be focusing on x86 and
> x86_64 systems.
> 
> I'd appreciate comments, ideas, and suggestions from the folk on this
> mailing list.

You may be interested in my web page "Linux and the Extended Precision
on x86 Processors":

  http://www.vinc17.org/research/extended.en.html

Also, comments are welcome.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

GCC and Floating-Point

[Scott Robert Ladd]

Hello.

I'm writing an extensive article about floating-point programming on
Linux, including a focus on GCC's compilers. This is an outgrowth of
many debates about topics like -ffast-math and -mfpmath=sse|387, and I
hope it will prove enlightening for myself and others.

Among the subjects covered: accuracy, precision, data organization, code
speed, threading issues, and a bit about interval arithmetic and
significant digits.

Naturally, due to hardware availability, I'll be focusing on x86 and
x86_64 systems.

I'd appreciate comments, ideas, and suggestions from the folk on this
mailing list.

Thank you.

..Scott