* Re: What is acceptable for -ffast-math? (Was: associative law incombine)

* Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> Why not make this easier for the user:
> - default to non-IEEE, but "reasonably safe".

The default GCC behavior is not to allow any optimization
that could change the value of an expression.  That rule is easy
for users to understand and surely it is a good rule to keep.

Floating point arithmetic, IEEE or otherwise, does not obey the
associative or distributive laws.  To pretend otherwise, without
paying close attention, quickly leads to unexpected numerical
failures.  Currently the default is to prohibit associative law
"optimizations" of floating-point expressions, and it ought to stay
that way.


> - make "-ffast-math" mean exactly that: fast.

Historically fast-math has allowed some transformations that could
lose a bit of precision but not, for example, transformations that
could destroy all precision through cancellation error.  The addition
of one or more switch levels beyond fast-math retains a
"reasonably-safe speed" level and introduces a level of "break-neck
speed" that may be hazardous to your health.


> - "-mieee":

Associative law optimizations must ordinarily be disallowed even if
not IEEE.  The IEEE compliance switch implies further special features
like exception flags and numbers that are Not A Number.

Re: * Re: What is acceptable for -ffast-math? (Was: associative lawin combine)

[Linus Torvalds]

On Sun, 29 Jul 2001, Stephen L Moshier wrote:
>
> > Why not make this easier for the user:
> > - default to non-IEEE, but "reasonably safe".
>
> The default GCC behavior is not to allow any optimization
> that could change the value of an expression.  That rule is easy
> for users to understand and surely it is a good rule to keep.
>
> Floating point arithmetic, IEEE or otherwise, does not obey the
> associative or distributive laws.

This is YOUR opinion.

But most people who write floating point code do not think that way. They
consider FP to be "mathematically exact" (which _does_ obey associative
and distributive laws), and they write their programs that way.

And that is exactly my point: to the people who do not care about the
exact details of what it means from a mathematical standpoint, using
associative and distributive rules is _fine_. Their algorithms do not
care, simply because the person writing the code doesn't even _know_ about
the issues of limited precision and range of FP code.

See?

What I'm saying is that the compiler cannot fix "thinking errors" of the
programmer. Trying to fix them is self-defeating and ultimately stupid.
I'm claiming that 99%+ of all programmers do not know or care about the
fact that

	(a+b)*c

is not necessarily the same as

	a*c+b*c

when we're talking about floating point.

My _second_ point is that the ones that DO KNOW and DO CARE are also aware
enough that _they_ can be counted on to add a "-mieee" or other flags.

They are also knowledgeable enough that they know what "associative law"
means, which is already a step up from most programmers, I suspect. So
they can understand why the compiler has done an optimization, and can (if
it is documented) be instructed to use the appropriate switch for not
doing that optimization.

So: put the onus of understanding different FP optimizations on the people
who CAN understand them. Don't make the average person who wants to do his
own small project have to worry about it. By the time the person
understands about the range of FP arithmetic enough to care, he will also
have the capability to do something about it.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> > Floating point arithmetic, IEEE or otherwise, does not obey the
> > associative or distributive laws.
> 
> This is YOUR opinion.

That is not just an opinion, it is simple arithmetic.

Further, GCC is supposed to be standards compliant by default.  You
seem to be advocating that GCC should, without warning, generate
programs that violate the standard and are mathematically wrong.
The C99 standard says the following, among other things, about floating
point transformations:


       [#14] EXAMPLE 5 Rearrangement for floating-point expressions
       is  often  restricted because of limitations in precision as
       well as range.  The implementation  cannot  generally  apply
       the   mathematical   associative   rules   for  addition  or
       multiplication,  nor  the  distributive  rule,  because   of
       roundoff   error,  even  in  the  absence  of  overflow  and
       underflow.   Likewise,  implementations   cannot   generally
       replace decimal constants in order to rearrange expressions.
       In  the  following  fragment,  rearrangements  suggested  by
       mathematical rules for real numbers are often not valid (see
       F.8).

               double x, y, z;
               /* ... */
               x = (x * y) * z;  // not equivalent to x *= y * z;
               z = (x - y) + y ; // not equivalent to z = x;
               z = x + x * y;    // not equivalent to z = x * (1.0 +
               y);
               y = x / 5.0;      // not equivalent to y = x * 0.2;

Re: * Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| On Sun, 29 Jul 2001, Stephen L Moshier wrote:
| >
| > > Why not make this easier for the user:
| > > - default to non-IEEE, but "reasonably safe".
| >
| > The default GCC behavior is not to allow any optimization
| > that could change the value of an expression.  That rule is easy
| > for users to understand and surely it is a good rule to keep.
| >
| > Floating point arithmetic, IEEE or otherwise, does not obey the
| > associative or distributive laws.
| 
| This is YOUR opinion.
| 
| But most people who write floating point code do not think that way. They
| consider FP to be "mathematically exact" (which _does_ obey associative
| and distributive laws), and they write their programs that way.

Wrong.  

Floating point arithemic is neither associative nor distributive.
And people who actually write numerical programs for living do know
that. 

If correctness is not a criterium then, it is faster not to compte at
all. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Sun, 29 Jul 2001, Stephen L Moshier wrote:
>
> > > Floating point arithmetic, IEEE or otherwise, does not obey the
> > > associative or distributive laws.
> >
> > This is YOUR opinion.
>
> That is not just an opinion, it is simple arithmetic.

No, it's actually a very complex issue of representing classical
arithmetic with finite values. It's not "simple arithmetic" at all. In the
simple arithmetic, operations _are_ associative and distriubutive.

And what I'm really claiming is that until gcc is a full Artifical
Intelligence system, it cannot add information to a program.

In 99% of all cases, the information of the subtleties of floating point
arithmetic simply was never there in the program gcc tries to compile.
Why? Becuase the programmer didn't even _intend_ any of the subtleties, or
even knew about them.

Gcc trying to enforce them doesn't add any value.

> Further, GCC is supposed to be standards compliant by default.

That is a separate argument, and I don't think you'r enecessarily wrong on
that front. On the other hand, people are already used to having command
line switches for strict standards adherence, so I don't think that this
argument is necessarily a very strong one, more a matter of opinion.

Especially true since the standard you quote is a fairly recent one (has
it been finalized now?)

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Fergus Henderson]

On 29-Jul-2001, Stephen L Moshier <moshier@mediaone.net> wrote:
> 
> Further, GCC is supposed to be standards compliant by default.

?

I think it would be nice if GCC was standards compliant by default,
but that is certainly not the status quo.

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: < http://www.cs.mu.oz.au/~fjh >  |     -- the last words of T. S. Garp.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Fergus Henderson <fjh@cs.mu.oz.au> writes:

| On 29-Jul-2001, Stephen L Moshier <moshier@mediaone.net> wrote:
| > 
| > Further, GCC is supposed to be standards compliant by default.
| 
| ?
| 
| I think it would be nice if GCC was standards compliant by default,
| but that is certainly not the status quo.

Agreed.  But that should be the ideal we should strive for.
And also, I don't think that is an argument for not computing
correctly, even if we're asked to compute fast. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:

> Fergus Henderson <fjh@cs.mu.oz.au> writes:
> |
> | I think it would be nice if GCC was standards compliant by default,
> | but that is certainly not the status quo.
>
> Agreed.  But that should be the ideal we should strive for.
> And also, I don't think that is an argument for not computing
> correctly, even if we're asked to compute fast.

Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
expression.

So it's certainly very very debatable whether you can say "not computing
correctly" for it - the user has not actually shown a strong preference
for the double division, and it is entirely possible that the double
division has come about as a result of macro expansion and/or inline
function expansion.

Who are you to say that the user wouldn't like the code to be run faster?
I don't think this was well-defined in C before C99, for example.

Note that with the current -ffast-math on x86, for example, gcc will
already inline things like "sin()" etc using the x87 instruction directly.
Despite the fact that that instruction is well-defined only for the range
-2**63 to +2**63.

And now there are people that say that something simple as "a/b/c" should
not be optimized to "a/(b*c)" even with -ffast-math.

What planet are you people from?

We are already in the situation that -ffast-math gives a value for "sin()"
that is not even _close_ to the range of [-1,1] under various
circumstances. And you're complaining about the fact that b*c can
overflow? We've already effectively limited our argument range to +-2**63,
and there's no way a*b can overflow, guys.

So even if you don't want to make it an optimization by default with the
C99 argument (hey, I don't really disagree), there is absolutely _no_
valid argument for not doing it with -ffast-math.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| On 30 Jul 2001, Gabriel Dos Reis wrote:
| 
| > Fergus Henderson <fjh@cs.mu.oz.au> writes:
| > |
| > | I think it would be nice if GCC was standards compliant by default,
| > | but that is certainly not the status quo.
| >
| > Agreed.  But that should be the ideal we should strive for.
| > And also, I don't think that is an argument for not computing
| > correctly, even if we're asked to compute fast.
| 
| Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
| expression.

Abstract Algebra is *different* from Computing.  That is lesson 0
from real-world numerical computations. 

| Who are you to say that the user wouldn't like the code to be run faster?

Who are you to say that it is more important to compute wrong and fast
than computing correctly?  

If correctness is not an issue then the fastest way is to avoid the
computation entirely or return 0.0 unconditionally.

| Note that with the current -ffast-math on x86, for example, gcc will
| already inline things like "sin()" etc using the x87 instruction directly.

I don't buy the argument 
 
  That is broken, then we should break this as well

If GCC behaves incorrectly in some cases, the appropriate action to
take is to make it behave correctly, -not- to break other parts.

| What planet are you people from?

A planet where we do numerical computation for living.

As von Neumann said once

   Numerical computation is too serious to be left to the computer
   scientist. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Olivier Galibert]

On Mon, Jul 30, 2001 at 08:53:37PM +0200, Gabriel Dos Reis wrote:
> A planet where we do numerical computation for living.

Then you're supposed to know better than to use -ffast-math.

  OG.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Olivier Galibert <galibert@pobox.com> writes:

| On Mon, Jul 30, 2001 at 08:53:37PM +0200, Gabriel Dos Reis wrote:
| > A planet where we do numerical computation for living.
| 
| Then you're supposed to know better than to use -ffast-math.

Certainly.  

Is that supposed to be  an argument for making GCC using ill-formed
computation rules?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:
>
> | Who are you to say that the user wouldn't like the code to be run faster?
>
> Who are you to say that it is more important to compute wrong and fast
> than computing correctly?

Why do you think it's necessarily wrong?

The _only_ person who can judge whether it's wrong to do so it the person
who wrote the program. He may be happy with the faster code. He's almost
certainly happy if he used "-ffast-math".

> If correctness is not an issue then the fastest way is to avoid the
> computation entirely or return 0.0 unconditionally.

Blather blather. According to your notion of correctness.

> | Note that with the current -ffast-math on x86, for example, gcc will
> | already inline things like "sin()" etc using the x87 instruction directly.
>
> I don't buy the argument
>
>   That is broken, then we should break this as well

It's not broken, you silly boy.

THAT IS THE WHOLE REASON FOR THE "-ffast-math" option.

Go back and look at why it was added. Right - exactly so that people could
get _fast_ math, when they do not care about the inexactness that isn't
even an issue for most people.

> As von Neumann said once
>
>    Numerical computation is too serious to be left to the computer
>    scientist.

Yeah. And as Linus once said: most numerical problems today in pure CPU
cycles are actually 3D games.

Your arguments about "numerical computation" are just silly, as you don't
seem to realize that there are tons of problems where your theoretical
issues are nothing more than noise.

It's not "incorrect" to say that you want the result faster, even if that
result doesn't match your theoretical models.

And remember: in the end, the goodness of _any_ program (including a
compiler) is how well it solves real-world problems. Whether you like the
transformation or not as anumerical scientist has absolutely nothing to do
with that.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| > As von Neumann said once
| >
| >    Numerical computation is too serious to be left to the computer
| >    scientist.
| 
| Yeah. And as Linus once said: most numerical problems today in pure CPU
| cycles are actually 3D games.

Then Linus need to be educated about numerical problems.

| Your arguments about "numerical computation" are just silly, as you don't
| seem to realize that there are tons of problems where your theoretical
| issues are nothing more than noise.

Not just because you claim they are silly makes them so.
More to the point, you don't seem to realize that there are tons of
problems where your theoretical issues of speed are nothing more than
noise. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Olivier Galibert]

On Tue, Jul 31, 2001 at 11:04:14AM +0200, Gabriel Dos Reis wrote:
> Not just because you claim they are silly makes them so.
> More to the point, you don't seem to realize that there are tons of
> problems where your theoretical issues of speed are nothing more than
> noise. 

But for these problems, you're supposed to use -mieee, not
-ffast-math.  Or nothing at all.  But we're talking about -ffast-math
here, which is already too incorrect for a simulations physicist, but
perfect for somebody doing 3D rendering, speech analysis, picture
analysis, etc...

In particular, if a/b/c won't overflow but a/(b*c) may you're already
into so tightly controlled conditions that using -ffast-math would be
somewhere between insanity and incompetence.

We're having 3 levels of "correctness" here:
-ffast-math: loose, works perfectly if you stay within reasonable
             boundaries and don't try to get the last bit of precision

nothing:     stricter, does what you write and don't try to be clever

-mieee:      military-level following of the rules, including anything
             and everything that happens for rouding, overflowing, etc...

There is a whole world of users there outside of numerical
computations as the physicist understand them.  And for them speed is
important, and they know how to keep their values within the bounds
that the compiler and the processor handle efficiently.  Don't go and
cripple _them_.

  OG.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Olivier Galibert <galibert@pobox.com> writes:

| There is a whole world of users there outside of numerical
| computations as the physicist understand them.  And for them speed is
| important, and they know how to keep their values within the bounds
| that the compiler and the processor handle efficiently.  Don't go and
| cripple _them_.

I don't want to "cripple" them.  And, I don't think "cripple" is a fair
characterization. 

You seem to imply that for someone who does Image Analysis,
correctness is less important than speed.  I disagree, if that were
indeed your intent.  If you expect the compiler to evaluate a/b/c as
a/(b*c) then you would have written a/(b*c).  Wouldn't you?

And speed does matter for numerical programmers also.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

On 31 Jul 2001, Gabriel Dos Reis wrote:
> |
> | Yeah. And as Linus once said: most numerical problems today in pure CPU
> | cycles are actually 3D games.
>
> Then Linus need to be educated about numerical problems.

You want to bet?

Sure, go ahead and count up your supercomputers, and I'll try to estimate
how many PS2's etc there are in the world.

That's by CPU cycles - I think you'll lose. Big.

Do the same by economic factors, and I suspect you're on slippery ground
too.

Do the same by number of users, and I _know_ you're done for.

I know, it's nice and comfy up in your ivory tower. Go on, ignore the rest
of the world..

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 1 Aug 2001, Gabriel Dos Reis wrote:
> |
> | That's not a definition.
>
> Do you have one?
>
> Changing 0.125 to 0.0 is a dramatic change.

Not actually necessarily.

Remember: in order to get that difference you had to be operating with
values in the 2**1024 range.

Compared to those values, the difference between 0.0 and 0.125 can often
be considered truly insignificant.

What "often" means depends on what you're doing with the values, of
course.

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| and probably a whole lot of other things that are equally "dubious".
| 
| 
| > But then, if they would have written that form in the first place.
| 
| But, as I said, there might be cases where you can't write it in an
| optimized way in the first place, such as in
|   inline double f() { return a/b; }
|   inline double g() { return c;   }
|   x = f()/g();
| plus inlining. 

But then is 0.0 is acceptable for 0.125 for speed?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| On Wed, 1 Aug 2001 dewar@gnat.com wrote:
| >
| > <<The above rules, btw, only apply for exact IEEE math, and on a number of
| > machines you _will_ see zero for the first one.
| > >>
| >
| > Hello! it was you who restricted the discussion to the x86. If we are allowed
| > to go to other machines, we have lots more examples up our sleeves :-)
| 
| Fair enough. But note how I didn't claim that you cannot get a difference
| in the ULP.

0.125 vs. 0.0 isn't a difference in the ULP.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
| >
| > Do you think changing 0.125 to 0.0 is not drastic?
| 
| Ehh. It's 0.125 to 0.125.
| 
| Try it yourself.

I ran an actual program on sparc-v8/solaris-2.7.  That machine is
known to operate with a 64-bits for double -- I specically mentionned
that.  Just try it yourself. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001 dewar@gnat.com wrote:
>
> <<The above rules, btw, only apply for exact IEEE math, and on a number of
> machines you _will_ see zero for the first one.
> >>
>
> Hello! it was you who restricted the discussion to the x86. If we are allowed
> to go to other machines, we have lots more examples up our sleeves :-)

Fair enough. But note how I didn't claim that you cannot get a difference
in the ULP. I _do_ claim that it is not "wrong" unless you specifically
ask for IEEE math - by virtue of pointing to lots of other machines that
do the same (including, apparently, the most modern architecture on the
planet. I don't have one, I can't test).

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
| >
| > | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
| >
| > On a 64-bit double processor just take
| 
| Oooh.. Quoting me out-of-context.

You requested an example. I gave one.

Then for your the purpose of your rhetorics, you said 

   Let's confine this to x86 - it is the most common case by far, and hey,
   especially when it comes to games etc I doubt it matters what HP-PA does,
   for example. We can easily make the optimizations be enabled on a
   per-architecture basis.

I gave a pretty much fair example to illustrate the point that the
transformation does affect the result on 64-bit double processors
example -- they are quite common.

Be honest and admit that you're getting to the point of requesting a
feature for a very specific target where you know the transformation
is safe.  Which is pretty much different from the previous "it won't
matter" stance.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
>
> Do you think changing 0.125 to 0.0 is not drastic?

Ehh. It's 0.125 to 0.125.

Try it yourself.

That's not a "drastic difference", not if you actually do what I
explained. If you don't have access to a x86 machine, maybe you can ask
somebody else to test the following appended for you.

Surprise, surprise, they come out exactly the same. Even down to the bit
representation (in case you care).

Not that I'm arguing that they will _always_ be the same down to the ULP,
but I _am_ arguing that at least on x86 it's actually a safe optimization.
You should disable it for -mieee.

		Linus

----
#include <math.h>

void test(double, double, double);

main()
{
        test(0.5*DBL_MAX, 2*sqrt(DBL_MAX), 2*sqrt(DBL_MAX));
}

void test(double a, double b, double c)
{
        printf("%f\n%f\n", a/b/c, a/(b*c));
}

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
>
> | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
>
> On a 64-bit double processor just take

Oooh.. Quoting me out-of-context.

Please bother to read the next line in my email and not snip it off.

Your arguments are getting weaker and weaker, and dishonest to boot.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001 dewar@gnat.com wrote:
>
> a/2.0/2.0 will never yield zero if a is non-zero
> a/4.0 may indeed yield zero if a is the smallest number

Didn't you already agree that FTZ was acceptable even for a default mode,
much less -ffast-math?

The above rules, btw, only apply for exact IEEE math, and on a number of
machines you _will_ see zero for the first one.

So I don't think your example is a valid argument for not using the
optimization - even without -ffast-math.

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?

On a 64-bit double processor just take 

	b = c = 2 * sqrt(DBL_MAX);
	a = 0.5 * DBL_MAX

a/(b*c) is 0.0 whereas a/b/c yields 0.125.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Wolfgang Bangerth]

> The transformations in question aren't documented.  So, those having
> "valid reasons to request them" don't even know what there are
> requesting with -fdubious-math.

Well, but then we're talking about a/b/c -> a/(b*c)  "and the like", as
  a*c+b*c -> (a+b)*c
  a*x/a   -> x
and probably a whole lot of other things that are equally "dubious".


> But then, if they would have written that form in the first place.

But, as I said, there might be cases where you can't write it in an
optimized way in the first place, such as in
  inline double f() { return a/b; }
  inline double g() { return c;   }
  x = f()/g();
plus inlining. You can't just rewrite this into a form that does the
optimized computation without the help of the compiler. However, I'd bet
that for the lower parts of the compiler the resulting stream of commands
look just the same as if it were  x=a/b/c  right away.

Regards
  Wolfgang

-------------------------------------------------------------------------
Wolfgang Bangerth          email: wolfgang.bangerth@iwr.uni-heidelberg.de
                             www: http://gaia.iwr.uni-heidelberg.de/~wolf

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

The transformations in question aren't documented.  So, those having
"valid reasons to request them" don't even know what there are
requesting with -fdubious-math.

Right now, we're discussing the ability of the compiler to transform
something like a/b/c into a/(b*c).  Some argue on the ground of speed
considerations, that the programmer would expect the compiler to do
that. But then, if they would have written that form in the first 
place.

The issue isn't about "freedom vs. no-freedom".

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Tim Hollebeek wrote:
>
> Rewriting the programmers program in such a way that he gets
> significantly less accuracy than he could reasonably expect is just
> silly.

Ehh.. "significantly less accuracy"?

Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
Let's confine this to x86 - it is the most common case by far, and hey,
especially when it comes to games etc I doubt it matters what HP-PA does,
for example. We can easily make the optimizations be enabled on a
per-architecture basis.

The one obvious case is (b*c) overflows and turns into Inf, at which point
the end result will be +-0 with the optimization. Now, in order for the
original to _not_ have been close to zero, we're talking about 'a' being
at the very limits of the number space. And remember that on x86 register
arithmetic - regardless of precision setting - the exponent is 15 bits. So
in order to get the overflow, AND to get a original value that wasn't zero
(ie to get an error), 'a' would have had to be on the order of

	2**(16383-64)

which won't even _fit_ in a double anyway (the -64 is to take even
denormals into account, but others have said that flush-to-zero would have
been acceptable anyway by default). Did I mis-do my calculations.

(Repeat argument for underflow).

So the overflow case actually doesn't exist on x86 if we keep the values
in registers or flush to stack in extended real format (and if we do NOT
flush to stack in extended real format, x86 has other problems like the
fact that the results depend on register flushes etc).

Do we have other problems with the conversion? Sure, it might be 1ULP off
even in the non-overflow case, but as we've seen others do that
optimization by default, never mind even with -ffast-math.

So tell me, what makes the optimization so dangerous? It doesn't seem to
do anything that we don't already do.

Did I miss some noteworthy case?

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Wolfgang Bangerth]

Tim Hollebeek wrote:
> Rewriting the programmers program in such a way that he gets
> significantly less accuracy than he could reasonably expect is just
> silly.

Of course it is, but that is not the point that is being discussed: nobody
proposed doing these transformations at -O2. If you don't want these
transformations, then don't switch on -fsomething. But if, for some
certainly valid reason you don't want to use these transformations, why
shouldn't other people have valid reasons to request them?

Regards
  Wolfgang


-------------------------------------------------------------------------
Wolfgang Bangerth          email: wolfgang.bangerth@iwr.uni-heidelberg.de
                             www: http://gaia.iwr.uni-heidelberg.de/~wolf

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Tue, 31 Jul 2001 dewar@gnat.com wrote:
>
> I still see *no* quantitative data showing that the transformation most
> discussed here (associative redistribution) is effective in improving
> performance.

Actually, the transformation that was discussed here was really

	a/b/c -> a/(b*c)

and I don't know where the associative redistribution thing really came
from. Probably just an example of another thing like it..

> Linus seems to be getting a little emotional in this discussion but swearing
> does not replace data.

Hey, I called people silly, not <censored>. You must have a very low
tolerance ;)

And the replacement-with-reciprocal kind of thing can certainly make a
difference. I'm too lazy to dig out my old quake3 source CD to try to dig
up what I ended up doing there, though.

> This seems such a reasonable position that I really have a heck of a time
> understanding why Linus is so incensed by it.

I'm incensed by people who think that the current -ffast-math already does
something that is positively EVIL - even though the current setup with
inline replacements of "fsin" etc has been there for years.

> Let's take an example, which is multiplication by a reciprocal. This of
> course does give "wrong" results. But division is often quite slow. On the
> other hand, I can't imagine floating-point programmers, including those
> doing game stuff, not knowing this, and thus writing the multiplication
> in the first place.

Actually, in the case of quake3, I distinctly remember how the x86 version
actually used division on purpose, because the x86 code was hand-tuned
assembler and they had counted cycles (for a Pentium machine) where the
division overlapped the other arithmetic.

The x86 hand-tuned stuff also took care to fit inside the FP stack etc -
they had started off with a C version and compiled that, and then once
they got as far as they could they just extracted the asm and did the rest
by hand.

It was also the much more obvious algorithm to do a division.

On alpha (and hey, this was before the 21264), the same did not hold true,
and it was faster to do a reciprocal. But it's been too long.

> <<Oh, round-to-zero is definitely acceptable in the world of "who cares
> about IEEE, we want fast math, and we'll use fixed arithmetic if the FP
> code is too slow".
> >>
>
> That's probably a bad idea, on most modern machines, full IEEE floating-point
> is faster than integer arithmetic (particularly in the case of multiplication)

Ehh.. You seem to think that "hardware == fast".

Denormal handling on x86 is done in microcode as a internal trap in the
pipeline. Round-to-zero is noticeably faster - even when the _code_ is the
same. Which is why Intel has special mode flags - where it internally
rounds to zero instead of making a denormal.

I'm not kidding. Look at the "Denormals are zero flag", bit 6 in the
MXCSR.

And I quote (all typos are probably mine):

 "The denormals-are-zero mode is not compatible with IEEE Standard 754.
  The denormals-are-zeros mode is provided to improve processor
  performance for applications such as streaming media processing, where
  rounding a denormal to zero does not appreciably affect the quality of
  the procesed data"

What I'm saying is that a buge company like Intel chose to do this
optimization IN HARDWARE because they thought that (a) it was meaningful
and (b) they thought the problem warranted it.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Tue, 31 Jul 2001, Joern Rennecke wrote:
>
> > That said, I don't mind about -fdubious-math as far as it is not the
> > default, nor turned on at any level of optimization.
>
> For a user, it's hard to guess or remember user that -fdubious-math
> and -ffast-math are related.  How about -ffast-and-loose-math ?

What are you guys smoking?

The _definition_ of "-ffast-math" is "fast and loose". It says so. Try
this:

	gcc -v --help 2>&1 | grep fast-math

Don't introduce new flags and names.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 31 Jul 2001, Gabriel Dos Reis wrote:
>
> What I intended to mean is that the compiler has, first, to prove that
> such transformmation yields the same result.

Ehh..

You're _wrong_.

A lot of compiler optimizations yield different results. The question is
whether it is acceptable or not.

It _clearly_ is acceptable to do when the user has explicitly asked for
"-ffast-math". The whole meaning of the flag is to say "I want it fast,
even if you might lose some range or precision".

It's only a matter of defining the semantics.

There are other optimizations like that. Signed integer overflow is
undefined, so you can optimize integer operations in ways that do not give
the same results for all integers.

Alias analysis was added several years ago, even though it was _clearly_ a
"illegal optimization" from a traditional C standpoint. That one broke
tons of real programs. Again, it was a question of re-defining the
semantics, and allowing people to take advantage of them.

Already, the _default_ behaviour for gcc at least on x86 with floating
point has very little to do with IEEE. It's been discussed several times
how to fix it - and very little real work on it has been done for the last
several years. Why? Because it really doesn't matter for most people, and
the current work-arounds are too much of a performance hit.

So gcc _already_ has "better fast than correct to the last ULP in IEEE".
Live with it. Even by default. Your argument that it cannot do the same
even when the user asks for it is completely ludicrous.

Your continual insistence that it is "wrong" only shows that you have
built a cage around your brain.

Have you even _looked_ at what the gcc definition for "fast-math" are?
Look again:

  {"fast-math", &flag_fast_math, 1,
   "Improve FP speed by violating ANSI & IEEE rules" },

and realize that even something simple as "double" doesn't historcially
have an exact meaning in C (ie you can _not_ assume specific ranges or
specific IEEE rounding in traditional C - it was left up to the
implementor to do as best he could with what hardware was available).

That means that traditionally, you don't even know what overflows or not,
or what precision you get. It's historically "undefined". Which means that
the semantics are not clear.

Now, this is obviously why a lot of computational people disliked,
distrusted and didn't use C historically, preferring to use Fortran which
had more defined rules. And which is, of course, why C99 integrated the
math extensions that some people have been working on for a loong time.

But you have to realize that this is (a) fairly recent and (b) not
everybody needs or wants it.

And you should also realize that YOU are not the grand poobah that gets to
tell people what semantics they want.

[ Neither am I, of course, but I'm arguing for choice and freedom, while
  you're arguing for small-mindedness everywhere. ]

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Tue, 31 Jul 2001 dewar@gnat.com wrote:
>
> Well it sure would be nice to here from some of these mythical numerical
> programmers (I don't care if they are writing games or nuclear reactor codes)
> who would be happier, so far we haven't heard this! And in my experience,
> even quite inexperienced floating-point numerical programmers are very
> disturbed when optimization changes the results of their programs.

I used -ffast-math myself, when I worked on the quake3 port to Linux (it's
been five years, how time flies).

It didn't make much difference at that point, because the x86 had
hand-written assembly, and gcc for the alpha didn't do much (anything?)
with -ffast-math.

But I tried _everything_. The main FP work I did on that thing on the
alpha improved the framerate by about 50% on alpha - FP was _that_
critical for it.  Most of it was by looking at gcc output and trying to
re-organize the C code to make it be better (because gcc didn't do much on
its own).

And yes, it was exactly things like multiplying by reciprocals.

> > Your arguments about "numerical computation" are just silly, as you don't
> > seem to realize that there are tons of problems where your theoretical
> > issues are nothing more than noise.
>
> If you think the arguments are silly, then I really fear you lack the full
> context for this discussion, a discussion that has, as you should know raged
> for well over thirty years.

Most of the kind of "numercial" work that you seem to be talking about has
probably rather little to do with FP performance. Most of the traditional
heavy FP code tends to be _much_ more about cache layout and good memory
access patterns.

I'm personally aquainted with tryign to make a game engine go fast, where
the memory effects are fewer, and the FP itself is the bottleneck.

> Sure -ffast-math is precisely intended to allow transformations that would
> not otherwise be allowed (let's not call them optimizations, that's just
> too controversial a word in the context of this argument).

Why not call them optimizations? They are. The only thing we change is the
boundary of valid ranges.

> The question is what is the boundary of allowable transformations. No one
> agrees that there should be no boundaries (even you don't like the change
> results to zero, though note that abandoning denormals has exactly this
> effect, and might be considered acceptable).

Oh, round-to-zero is definitely acceptable in the world of "who cares
about IEEE, we want fast math, and we'll use fixed arithmetic if the FP
code is too slow".

In fact, it is _so_ acceptable that CPU designers design for it. Look at
MMX2, and wonder why they have a RTZ mode? Because it makes the _hardware_
go faster.

That should tell you something. Big companies that have billion-dollar
fabs spend time optimizing their chips that take several years to design
for _games_. Not for IEEE traditional Fortran-kind math.

But apparently some gcc developers don't think that is even a worthy
market, because you just want to do fluid dynamics.

> So, what is the boundary, can one for instance forget about denormals and
> flush to zero to save a bit of time, can one truncate instead of round,
> can one ignore negative zeroes, or infinity semantics, can one ignore
> intermediate overflow (note: nearly all the discussed transformations are
> implicitly saying yes to this last question).

Do them all by default with -ffast-math.

Then, you can have specific flags for people who want just _one_
optimization. I doubt you'll find many users who do that, but maybe I'm
wrong. Giving people the choice is always a good idea.

> I have not seen anyone writing from the point of view of serious numerical
> coding saying [ .. ]

There you go again. What the hell do you call "serious numerical coding"?

Take a look at the computer game market today. It's a lot more serious
than most matematicians puttering around in their labs, let me tell you.
That's a BIG industry.

Also note that _nobody_ in your kind of "serious numerical coding"
community would ever worry about "-ffast-math" in the first place. Why the
hell would they, when 99% of the time it doesn't make any difference at
all. The people you apparently consider serious are a lot more interested
in fast communication (so that they can solve the thing in parallell) and
incredible memory bandwidth.

I doubt you'll find many of your "serious numerical coding" people who
would even _notice_ the raw FP throughput. Look at SpecFP - CPU's are fast
enough, it spends most of its time waiting on memory.

> Should -ffast-math allow full precision operation? I would think so,
> since it definitely improves performance, and reduces surprises.

Ehh.. gcc right now allows full precision operation BY DEFAULT. No
-ffast-math required.

Same goes for negative zero as far as I remember - on some HP-PA stuff at
least. Simply because it was too painful to get "right" on their earlier
hardware.

In short, what you seem to argue that -ffast-math should means are all
things gcc _already_ does, with no

> By the way, I said I would be shocked to find a Fortran compiler that did
> associative redistribution in the absence of parens. I am somewhat surprised
> that no one stepped forward with a counter-example, but I suspect in fact that
> there may not be any shocking Fortran implementations around.

I would suspect that very few people have Fortran compilers around and
bother to check it.

> It is an old argument, the one that says that fpt is approximate, so why bother
> to be persnickety about it. Seymour Cray always tool this viewpoint, and it
> did not bother him that 81.0/3.0 did not give exactly 27.0 on the CDC 6000
> class machines.

..and he was universally respected for making the fastest machines around.

What you forget to mention is that these days it's so _cheap_ to get
IEEE, that from a hardware standpoint pretty much everybody includes it
anyway.

But they then often throw it away because it ends up having expensive
run-time issues (alpha with exception handling and proper denormals, Intel
with special RTZ modes etc).

Why? Because in many areas Seymour Cray is _still_ right. The thing that
killed off non-IEEE was not that he was wrong, but the fact that _some_
people do need IEEE "exact FP". Not everybody. Not even the majority. But
because some people do need it, you need to support it. Which is why
everybody does, today.

But do you see the difference between

  "We have to support it because a portion of the user base has to have
   it, and if we don't have it we won't be able to sell to any of that
   user base"

and

  "Everybody must use it, because anything else is wrong"

Eh?

Do you see that Seymours approach didn't fail because he was always wrong?
It failed because he was _sometimes_ wrong.

And you know what? He was right enough of the time to have built up an
empire for a while. That's something not everybody can say about
themselves. And that is something that you should respect.

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Mon, 30 Jul 2001 dewar@gnat.com wrote:

> >>Well, I won't be so rash as to say that it should do that *by default*,
>
> But that of course is the issue under discussoin :-)

No it is not.

See the subject line.

Doing it by default was just one possible suggestion.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Mark Mitchell]

>> OTOH, you could re-use (abuse) statement expressions for this purpose:
>
>> ({a+b;})*c

Goodness.  I have been away from email for a few days, and now I'm back,
but I'm considering going away from email again and not coming back. :-)
:-)

If the worst problem in GCC is that you cannot specify optimization
barriers in the midst of expressions then we are in very good shape
indeed!

Let's not go down this path.  In my opinion, we really do not want
to introduce this kind of thing.  There is much more win simply in
fixing bugs and limitations in the current optimizers.

-- 
Mark Mitchell                mark@codesourcery.com
CodeSourcery, LLC            http://www.codesourcery.com

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Joseph S. Myers]

On 30 Jul 2001, Alexandre Oliva wrote:

> Would you prefer
> 
> __extension__ __attribute__ ((optimize=0)) (a*b) +
> __extension__ __attribute__ ((optimize=0)) (a*c)
> 
> ? :-)
> 
> I'd much rather write ((a*b)) + ((a*c)), which will even compile with
> other compilers, and might be picked up in a future revision of the C
> and C++ Standards.

I'd rather you use #pragma STDC FP_CONTRACT OFF - while this seems to be
intended to disable optimisations that give more accurate results, it
should probably disable optimisations here that may give less accurate
results as well.

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> > Further, GCC is supposed to be standards compliant by default.
> 
> Really?

Well, then I'd like to understand the actual policy better.  You have
GNU extensions but otherwise seem to include what's in the language
definition.  The apparent conflicts arise when you say the
GNU extensions have to be turned off in order for the compiler
to be strictly compliant with the standard.

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Mon, 30 Jul 2001 dewar@gnat.com wrote:
>
> But if you look at real Fortran codes, they are full of parentheses to stop
> the compiler from messing around, and in fact most Fortran compilers do only
> very limited mucking even when no parens are present. I would be shocked
> to find a FOrtraqn compiler that changed
>
> a*c + b*c
>
> into
>
> (a+b)*c
>
> a totally diffrerent expression!

You'd be shocked?

What if I told you that it's actually legal? Go check it out.

Note that the mark of a language that generates fast code is that it both
(a) allows the compiler the maximum possible flexibility to do the best
job it can, and (b) allows the programmer to specify what the limits are.

Fortran, while being a pretty horrible language for anything else, does
exactly this for FP math. It doesn't do it for much anything else, of
course.

Now, the problem here is that what is permissible in Fortran is not
permissible in C, but I think we should _allow_ it to happen.

Note that I'm not really advocating breaking standards, like so many
people seem to think. If you actually go back and read my emails instead
of just reacting to them by gut feel, you'll notice that what I really
suggested was to make it _easy_ for the non-math programmer to tell the
compiler "do the best you can with this".

I argue _against_ a lot of magic compiler flags that are version-
dependent and hard to use, and I argue _for_ "-ffast-math" being very
strong.

I also argue that since we already have more standards than we can shake a
stick at (traditional, c89, gnu89, c99, etc), and C traditionally has not
been very strict about what optimizations are permissible, the argument of
"conforms to C99 by default" is not a very strong one. But I also noted
that it _is_ a very valid argument, although not as absolute as some
people seem to think it is (exactly because we do have multiple standards,
and gcc already defaults to a standard of its own rather than anything
else).

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Sun, 29 Jul 2001 dewar@gnat.com wrote:
>
> <<Further, GCC is supposed to be standards compliant by default.  You
> seem to be advocating that GCC should, without warning, generate
> programs that violate the standard and are mathematically wrong.
> >>
>
> and that violate the expectations of programmers who *do* know what they
> are doing!

Note that most of the people I know who _do_ know what they are doing also
want to have the option o ftelling the compiler when they don't care, and
want to allow the compiler to do "illegal" things if that allows the
compiler to generate better code.

It's sad that C99 doesn't seem to allow the Fortran kind of "you go and
make the best you can of this expression unless I tell you otherwise".
Which means that we'd _have_ to be non-standard to allow the optimization
at all. Which obviously means a flag (one way or the other).

Rather than have a good way to do it on a case-by-case basis, which is
probably what the people who _really_ care would prefer.

In C, you can't reasonably use the Fortran "parenthesis are significant"
approach - the way the C pre-processor works you often have lots of extra
parenthesis for that reason. However, C does have the well-established
sequence point rule, so it would make sense to have that kind of
extension.

And hey, if you want to make it be an extra flag, just make it a _simple_
one. Like "-ffast-math". Don't make people have to know 100 esoteric flags
that are compiler-release-specific. That's completely useless.

		Linus