* Re: What is acceptable for -ffast-math? (Was: associative law incombine)

* Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> Why not make this easier for the user:
> - default to non-IEEE, but "reasonably safe".

The default GCC behavior is not to allow any optimization
that could change the value of an expression.  That rule is easy
for users to understand and surely it is a good rule to keep.

Floating point arithmetic, IEEE or otherwise, does not obey the
associative or distributive laws.  To pretend otherwise, without
paying close attention, quickly leads to unexpected numerical
failures.  Currently the default is to prohibit associative law
"optimizations" of floating-point expressions, and it ought to stay
that way.


> - make "-ffast-math" mean exactly that: fast.

Historically fast-math has allowed some transformations that could
lose a bit of precision but not, for example, transformations that
could destroy all precision through cancellation error.  The addition
of one or more switch levels beyond fast-math retains a
"reasonably-safe speed" level and introduces a level of "break-neck
speed" that may be hazardous to your health.


> - "-mieee":

Associative law optimizations must ordinarily be disallowed even if
not IEEE.  The IEEE compliance switch implies further special features
like exception flags and numbers that are Not A Number.

Re: * Re: What is acceptable for -ffast-math? (Was: associative lawin combine)

[Linus Torvalds]

On Sun, 29 Jul 2001, Stephen L Moshier wrote:
>
> > Why not make this easier for the user:
> > - default to non-IEEE, but "reasonably safe".
>
> The default GCC behavior is not to allow any optimization
> that could change the value of an expression.  That rule is easy
> for users to understand and surely it is a good rule to keep.
>
> Floating point arithmetic, IEEE or otherwise, does not obey the
> associative or distributive laws.

This is YOUR opinion.

But most people who write floating point code do not think that way. They
consider FP to be "mathematically exact" (which _does_ obey associative
and distributive laws), and they write their programs that way.

And that is exactly my point: to the people who do not care about the
exact details of what it means from a mathematical standpoint, using
associative and distributive rules is _fine_. Their algorithms do not
care, simply because the person writing the code doesn't even _know_ about
the issues of limited precision and range of FP code.

See?

What I'm saying is that the compiler cannot fix "thinking errors" of the
programmer. Trying to fix them is self-defeating and ultimately stupid.
I'm claiming that 99%+ of all programmers do not know or care about the
fact that

	(a+b)*c

is not necessarily the same as

	a*c+b*c

when we're talking about floating point.

My _second_ point is that the ones that DO KNOW and DO CARE are also aware
enough that _they_ can be counted on to add a "-mieee" or other flags.

They are also knowledgeable enough that they know what "associative law"
means, which is already a step up from most programmers, I suspect. So
they can understand why the compiler has done an optimization, and can (if
it is documented) be instructed to use the appropriate switch for not
doing that optimization.

So: put the onus of understanding different FP optimizations on the people
who CAN understand them. Don't make the average person who wants to do his
own small project have to worry about it. By the time the person
understands about the range of FP arithmetic enough to care, he will also
have the capability to do something about it.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> > Floating point arithmetic, IEEE or otherwise, does not obey the
> > associative or distributive laws.
> 
> This is YOUR opinion.

That is not just an opinion, it is simple arithmetic.

Further, GCC is supposed to be standards compliant by default.  You
seem to be advocating that GCC should, without warning, generate
programs that violate the standard and are mathematically wrong.
The C99 standard says the following, among other things, about floating
point transformations:


       [#14] EXAMPLE 5 Rearrangement for floating-point expressions
       is  often  restricted because of limitations in precision as
       well as range.  The implementation  cannot  generally  apply
       the   mathematical   associative   rules   for  addition  or
       multiplication,  nor  the  distributive  rule,  because   of
       roundoff   error,  even  in  the  absence  of  overflow  and
       underflow.   Likewise,  implementations   cannot   generally
       replace decimal constants in order to rearrange expressions.
       In  the  following  fragment,  rearrangements  suggested  by
       mathematical rules for real numbers are often not valid (see
       F.8).

               double x, y, z;
               /* ... */
               x = (x * y) * z;  // not equivalent to x *= y * z;
               z = (x - y) + y ; // not equivalent to z = x;
               z = x + x * y;    // not equivalent to z = x * (1.0 +
               y);
               y = x / 5.0;      // not equivalent to y = x * 0.2;

Re: * Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| On Sun, 29 Jul 2001, Stephen L Moshier wrote:
| >
| > > Why not make this easier for the user:
| > > - default to non-IEEE, but "reasonably safe".
| >
| > The default GCC behavior is not to allow any optimization
| > that could change the value of an expression.  That rule is easy
| > for users to understand and surely it is a good rule to keep.
| >
| > Floating point arithmetic, IEEE or otherwise, does not obey the
| > associative or distributive laws.
| 
| This is YOUR opinion.
| 
| But most people who write floating point code do not think that way. They
| consider FP to be "mathematically exact" (which _does_ obey associative
| and distributive laws), and they write their programs that way.

Wrong.  

Floating point arithemic is neither associative nor distributive.
And people who actually write numerical programs for living do know
that. 

If correctness is not a criterium then, it is faster not to compte at
all. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Sun, 29 Jul 2001, Stephen L Moshier wrote:
>
> > > Floating point arithmetic, IEEE or otherwise, does not obey the
> > > associative or distributive laws.
> >
> > This is YOUR opinion.
>
> That is not just an opinion, it is simple arithmetic.

No, it's actually a very complex issue of representing classical
arithmetic with finite values. It's not "simple arithmetic" at all. In the
simple arithmetic, operations _are_ associative and distriubutive.

And what I'm really claiming is that until gcc is a full Artifical
Intelligence system, it cannot add information to a program.

In 99% of all cases, the information of the subtleties of floating point
arithmetic simply was never there in the program gcc tries to compile.
Why? Becuase the programmer didn't even _intend_ any of the subtleties, or
even knew about them.

Gcc trying to enforce them doesn't add any value.

> Further, GCC is supposed to be standards compliant by default.

That is a separate argument, and I don't think you'r enecessarily wrong on
that front. On the other hand, people are already used to having command
line switches for strict standards adherence, so I don't think that this
argument is necessarily a very strong one, more a matter of opinion.

Especially true since the standard you quote is a fairly recent one (has
it been finalized now?)

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Fergus Henderson]

On 29-Jul-2001, Stephen L Moshier <moshier@mediaone.net> wrote:
> 
> Further, GCC is supposed to be standards compliant by default.

?

I think it would be nice if GCC was standards compliant by default,
but that is certainly not the status quo.

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: < http://www.cs.mu.oz.au/~fjh >  |     -- the last words of T. S. Garp.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Fergus Henderson <fjh@cs.mu.oz.au> writes:

| On 29-Jul-2001, Stephen L Moshier <moshier@mediaone.net> wrote:
| > 
| > Further, GCC is supposed to be standards compliant by default.
| 
| ?
| 
| I think it would be nice if GCC was standards compliant by default,
| but that is certainly not the status quo.

Agreed.  But that should be the ideal we should strive for.
And also, I don't think that is an argument for not computing
correctly, even if we're asked to compute fast. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:

> Fergus Henderson <fjh@cs.mu.oz.au> writes:
> |
> | I think it would be nice if GCC was standards compliant by default,
> | but that is certainly not the status quo.
>
> Agreed.  But that should be the ideal we should strive for.
> And also, I don't think that is an argument for not computing
> correctly, even if we're asked to compute fast.

Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
expression.

So it's certainly very very debatable whether you can say "not computing
correctly" for it - the user has not actually shown a strong preference
for the double division, and it is entirely possible that the double
division has come about as a result of macro expansion and/or inline
function expansion.

Who are you to say that the user wouldn't like the code to be run faster?
I don't think this was well-defined in C before C99, for example.

Note that with the current -ffast-math on x86, for example, gcc will
already inline things like "sin()" etc using the x87 instruction directly.
Despite the fact that that instruction is well-defined only for the range
-2**63 to +2**63.

And now there are people that say that something simple as "a/b/c" should
not be optimized to "a/(b*c)" even with -ffast-math.

What planet are you people from?

We are already in the situation that -ffast-math gives a value for "sin()"
that is not even _close_ to the range of [-1,1] under various
circumstances. And you're complaining about the fact that b*c can
overflow? We've already effectively limited our argument range to +-2**63,
and there's no way a*b can overflow, guys.

So even if you don't want to make it an optimization by default with the
C99 argument (hey, I don't really disagree), there is absolutely _no_
valid argument for not doing it with -ffast-math.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| On 30 Jul 2001, Gabriel Dos Reis wrote:
| 
| > Fergus Henderson <fjh@cs.mu.oz.au> writes:
| > |
| > | I think it would be nice if GCC was standards compliant by default,
| > | but that is certainly not the status quo.
| >
| > Agreed.  But that should be the ideal we should strive for.
| > And also, I don't think that is an argument for not computing
| > correctly, even if we're asked to compute fast.
| 
| Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
| expression.

Abstract Algebra is *different* from Computing.  That is lesson 0
from real-world numerical computations. 

| Who are you to say that the user wouldn't like the code to be run faster?

Who are you to say that it is more important to compute wrong and fast
than computing correctly?  

If correctness is not an issue then the fastest way is to avoid the
computation entirely or return 0.0 unconditionally.

| Note that with the current -ffast-math on x86, for example, gcc will
| already inline things like "sin()" etc using the x87 instruction directly.

I don't buy the argument 
 
  That is broken, then we should break this as well

If GCC behaves incorrectly in some cases, the appropriate action to
take is to make it behave correctly, -not- to break other parts.

| What planet are you people from?

A planet where we do numerical computation for living.

As von Neumann said once

   Numerical computation is too serious to be left to the computer
   scientist. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Olivier Galibert]

On Mon, Jul 30, 2001 at 08:53:37PM +0200, Gabriel Dos Reis wrote:
> A planet where we do numerical computation for living.

Then you're supposed to know better than to use -ffast-math.

  OG.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Olivier Galibert <galibert@pobox.com> writes:

| On Mon, Jul 30, 2001 at 08:53:37PM +0200, Gabriel Dos Reis wrote:
| > A planet where we do numerical computation for living.
| 
| Then you're supposed to know better than to use -ffast-math.

Certainly.  

Is that supposed to be  an argument for making GCC using ill-formed
computation rules?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:
>
> | Who are you to say that the user wouldn't like the code to be run faster?
>
> Who are you to say that it is more important to compute wrong and fast
> than computing correctly?

Why do you think it's necessarily wrong?

The _only_ person who can judge whether it's wrong to do so it the person
who wrote the program. He may be happy with the faster code. He's almost
certainly happy if he used "-ffast-math".

> If correctness is not an issue then the fastest way is to avoid the
> computation entirely or return 0.0 unconditionally.

Blather blather. According to your notion of correctness.

> | Note that with the current -ffast-math on x86, for example, gcc will
> | already inline things like "sin()" etc using the x87 instruction directly.
>
> I don't buy the argument
>
>   That is broken, then we should break this as well

It's not broken, you silly boy.

THAT IS THE WHOLE REASON FOR THE "-ffast-math" option.

Go back and look at why it was added. Right - exactly so that people could
get _fast_ math, when they do not care about the inexactness that isn't
even an issue for most people.

> As von Neumann said once
>
>    Numerical computation is too serious to be left to the computer
>    scientist.

Yeah. And as Linus once said: most numerical problems today in pure CPU
cycles are actually 3D games.

Your arguments about "numerical computation" are just silly, as you don't
seem to realize that there are tons of problems where your theoretical
issues are nothing more than noise.

It's not "incorrect" to say that you want the result faster, even if that
result doesn't match your theoretical models.

And remember: in the end, the goodness of _any_ program (including a
compiler) is how well it solves real-world problems. Whether you like the
transformation or not as anumerical scientist has absolutely nothing to do
with that.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| > As von Neumann said once
| >
| >    Numerical computation is too serious to be left to the computer
| >    scientist.
| 
| Yeah. And as Linus once said: most numerical problems today in pure CPU
| cycles are actually 3D games.

Then Linus need to be educated about numerical problems.

| Your arguments about "numerical computation" are just silly, as you don't
| seem to realize that there are tons of problems where your theoretical
| issues are nothing more than noise.

Not just because you claim they are silly makes them so.
More to the point, you don't seem to realize that there are tons of
problems where your theoretical issues of speed are nothing more than
noise. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Olivier Galibert]

On Tue, Jul 31, 2001 at 11:04:14AM +0200, Gabriel Dos Reis wrote:
> Not just because you claim they are silly makes them so.
> More to the point, you don't seem to realize that there are tons of
> problems where your theoretical issues of speed are nothing more than
> noise. 

But for these problems, you're supposed to use -mieee, not
-ffast-math.  Or nothing at all.  But we're talking about -ffast-math
here, which is already too incorrect for a simulations physicist, but
perfect for somebody doing 3D rendering, speech analysis, picture
analysis, etc...

In particular, if a/b/c won't overflow but a/(b*c) may you're already
into so tightly controlled conditions that using -ffast-math would be
somewhere between insanity and incompetence.

We're having 3 levels of "correctness" here:
-ffast-math: loose, works perfectly if you stay within reasonable
             boundaries and don't try to get the last bit of precision

nothing:     stricter, does what you write and don't try to be clever

-mieee:      military-level following of the rules, including anything
             and everything that happens for rouding, overflowing, etc...

There is a whole world of users there outside of numerical
computations as the physicist understand them.  And for them speed is
important, and they know how to keep their values within the bounds
that the compiler and the processor handle efficiently.  Don't go and
cripple _them_.

  OG.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Olivier Galibert <galibert@pobox.com> writes:

| There is a whole world of users there outside of numerical
| computations as the physicist understand them.  And for them speed is
| important, and they know how to keep their values within the bounds
| that the compiler and the processor handle efficiently.  Don't go and
| cripple _them_.

I don't want to "cripple" them.  And, I don't think "cripple" is a fair
characterization. 

You seem to imply that for someone who does Image Analysis,
correctness is less important than speed.  I disagree, if that were
indeed your intent.  If you expect the compiler to evaluate a/b/c as
a/(b*c) then you would have written a/(b*c).  Wouldn't you?

And speed does matter for numerical programmers also.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

On 31 Jul 2001, Gabriel Dos Reis wrote:
> |
> | Yeah. And as Linus once said: most numerical problems today in pure CPU
> | cycles are actually 3D games.
>
> Then Linus need to be educated about numerical problems.

You want to bet?

Sure, go ahead and count up your supercomputers, and I'll try to estimate
how many PS2's etc there are in the world.

That's by CPU cycles - I think you'll lose. Big.

Do the same by economic factors, and I suspect you're on slippery ground
too.

Do the same by number of users, and I _know_ you're done for.

I know, it's nice and comfy up in your ivory tower. Go on, ignore the rest
of the world..

		Linus