Re: What is acceptable for -ffast-math? (Was: associative law in combine)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<What I intended to mean is that the compiler has, first, to prove that
such transformmation yields the same result.  The only argument I've
seen so far is that, GCC should make the program run fast...
>>

Not quite, "the same result" is too strong. Instead, the rule is that
the transformation must yield a result that is still conforming to the
requirements of the standard. Since the standard is (deliberately) not
100% precise in saying what this result should be, there is some legitimate
wiggle room.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

dewar@gnat.com writes:

| <<What I intended to mean is that the compiler has, first, to prove that
| such transformmation yields the same result.  The only argument I've
| seen so far is that, GCC should make the program run fast...
| >>
| 
| Not quite, "the same result" is too strong. Instead, the rule is that
| the transformation must yield a result that is still conforming to the
| requirements of the standard. Since the standard is (deliberately) not
| 100% precise in saying what this result should be, there is some legitimate
| wiggle room.

Transforming a/b/c into a/(b*c) where (b*c) overflows whereas (a/b)/c
does not isn't something acceptable (speaking here as a user or a
numerical programmer).  Even if C++98 is as vague as C90 was about
floating point arithmetic, the move is that the next revision of the
Standard will provide a sound framework and support for numerical
computations; I don't think speed is enough to justify incorrectness.
I don't think that the impreciseness in C++98 makes it OK to apply
dubious rules.

That doesn't mean I don't want optimization.  I do love optimization.
By optimization, I really mean optimization.

That said, I don't mind about -fdubious-math as far as it is not the
default, nor turned on at any level of optimization.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Joern Rennecke]

> That said, I don't mind about -fdubious-math as far as it is not the
> default, nor turned on at any level of optimization.

For a user, it's hard to guess or remember user that -fdubious-math
and -ffast-math are related.  How about -ffast-and-loose-math ?

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Joern Rennecke <amylaar@redhat.com> writes:

| > That said, I don't mind about -fdubious-math as far as it is not the
| > default, nor turned on at any level of optimization.
| 
| For a user, it's hard to guess or remember user that -fdubious-math
| and -ffast-math are related.  How about -ffast-and-loose-math ?

I was actually proposing to rename -ffast-math to -fdubious-math :-)
but your proposal has also my support :-))

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Tue, 31 Jul 2001, Joern Rennecke wrote:
>
> > That said, I don't mind about -fdubious-math as far as it is not the
> > default, nor turned on at any level of optimization.
>
> For a user, it's hard to guess or remember user that -fdubious-math
> and -ffast-math are related.  How about -ffast-and-loose-math ?

What are you guys smoking?

The _definition_ of "-ffast-math" is "fast and loose". It says so. Try
this:

	gcc -v --help 2>&1 | grep fast-math

Don't introduce new flags and names.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Theodore Papadopoulo]

While I agree that by default the compiler must follow "good rules", I 
think that always disallowing optimisations which may lose some bits of 
accuracy (when specified with a option switch of course) just because 
it may hurt someone, somewhere, sometime is a little too strong.
Full safety is IMHO impossible with floating point... Only the 
programmer knows the accuracy of its input data (Linus called that 
noise) and the accuracy he expects on its result.

Since the example has been given, let us speak about image processing.
The data (when using real images) are often 'char' based so you have 
basically 8 bits or less of information per channel.  Then converting
these images to float and working with them, there is usually little risk of 
introducing some error due to some bad rounding. float is enough for 
most image analysis and computation, and lots of people are even 
using double (which sometimes are also faster). In such cases, I 
would argue that absolute floating point accuracy is not worth. To be 
meaningful the results have to be robust to errors (in the input and 
thus in the subsequent calculations that are many orders of magnitude 
higher than the machine accuracy). If they are not, then the approach 
is certainly not valid.

gdr@codesourcery.com said:
> That doesn't mean I don't want optimization.  I do love optimization.
> By optimization, I really mean optimization. 

No by optimisation, you mean mathematically provable safe optimisation and 
unfortunately, the provability of such things by mathematics is usually
very difficult. I agree that sometimes you have to rely only on such 
optimisations, so that, by default, the compiler should only allow them.
But, you have to allow for people:

- for whom provably has an experimental meaning. There are a lot of 
fields where computationnal difficulty is avoided by addind some 
random noise over the existing values just because doing so usually
removes some degeneracies (with 99.999999% of the cases) in the computation
that are just too difficult to deal with and that can be considered
as artifacts of a configuration that will not impact the result (a 
choice between equally good solutions is just made).

- who know that the allowed input data (for their particular program) is well
within the range of the floating point range accuracy they have selected to work with.
Image analysis, games, and lots of real world problem when the data 
are corrupted with noise often enters in this category (measurement 
apparatus seldom give 24 or 53 bits of accuracy).

Proving anything with floating point computation is very difficult in 
some cases even for knowledgeable people. Often, people who really want
to deal with accuracy often go beyond simple floating point computation by
using multiple arithmetics with increasing accuracy or even better, range arithmetic.

> Moreover, if the drawbacks of some optimizations are too dangerous
> (very reduced domain, etc), a solution could be (yes, I realize there
> are too many already) to allow the user to specify how hard one wants
> the compiler to try and speed up the code using -ffast-math-<N>, as in
> -O<N>, or adding --fmath-allow-domain-shrinking, or -fmath-ignore-nans
> or whatever could be used by the programmer to tell the compiler what
> is important to him/her and what can be sacrificed for sped. 

I like the -ffast-math-<N> idea.
By default -ffast-math (equivalent to eg -ffast-math-0) would be very conservative.
Then with N increasing you would allow optimisations that lose more and 
more bits of accuracy.

While I agree that it would only be useful to add things that indeed 
increase the speed by more than a few percents, this is something 
very difficult to state with modern machines, all the more that things 
may add up in a complex fashion, if multiple small optimisations are 
added.

wolfgang.bangerth@iwr.uni-heidelberg.de said:
> - denormals: if you run into the risk of getting NaNs one way but not
>   the other, you most certainly already have gone beyond the limits of
>   your program, i.e. your linear solver diverged, you have chosen an
>   inherently unstable formulation for your problem, etc. In practice,

I totally agree.
All the more that denormals is certainly something that can cost a lot in term of 
computation time since often this is not really implemented in the 
hardware (trap to microcode or to library code). I remember having 
seen decreases in speed by a factor of at least 10, for things that 
would have been equally well be treated as zero.

Often, when using floating point in some tricky situations, I wonder 
whether it would not be better to have very fragile floating point 
arithmetic by default, so that each potential problematic spot is 
easily located as one generating a NaN or an obviously bad result.
I certainly go too far here, but I'm not sure it would not be good in 
the long run...

--------------------------------------------------------------------
Theodore Papadopoulo
Email: Theodore.Papadopoulo@sophia.inria.fr Tel: (33) 04 92 38 76 01
 --------------------------------------------------------------------


-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Exmh version 2.2 06/23/2000

iD8DBQE7aCZYIzTj8qrxOU4RAtXKAJ4zEqZKrUYxF+8rsk4+KVJA1hvAjgCg4KZk
0uhEmXIpS200Azv/xR+bQxM=
=TwfB
-----END PGP SIGNATURE-----

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Theodore Papadopoulo <Theodore.Papadopoulo@sophia.inria.fr> writes:

| gdr@codesourcery.com said:
| > That doesn't mean I don't want optimization.  I do love optimization.
| > By optimization, I really mean optimization. 
| 
| No by optimisation, you mean mathematically provable safe optimisation and 

No, I do mean a transformation which does not dramatically transmute 
the computations.  

Where did you get the idea that I meant "mathematically provable safe
optimisation"?

We're talking here of transformations which we do know able to
drastically change the results.
Going into abstract considerations like NP-completness is beside the
point. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Hollebeek]

> While I agree that by default the compiler must follow "good rules", I 
> think that always disallowing optimisations which may lose some bits of 
> accuracy (when specified with a option switch of course) just because 
> it may hurt someone, somewhere, sometime is a little too strong.
> Full safety is IMHO impossible with floating point... Only the 
> programmer knows the accuracy of its input data (Linus called that 
> noise) and the accuracy he expects on its result.

Exactly ... and it isn't at all rare that the programmer knows exactly
how much noise is in the input data (little or none) and the how much
accuracy he expects in the result (as much as possible!).

Rewriting the programmers program in such a way that he gets
significantly less accuracy than he could reasonably expect is just
silly.

Noise is a red herring.  There is no reason why computation should
introduce more innaccuracy than they need to, especially in a ad hoc
way that is beyond the programmer's control.

-Tim

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Tim Hollebeek wrote:
>
> Rewriting the programmers program in such a way that he gets
> significantly less accuracy than he could reasonably expect is just
> silly.

Ehh.. "significantly less accuracy"?

Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
Let's confine this to x86 - it is the most common case by far, and hey,
especially when it comes to games etc I doubt it matters what HP-PA does,
for example. We can easily make the optimizations be enabled on a
per-architecture basis.

The one obvious case is (b*c) overflows and turns into Inf, at which point
the end result will be +-0 with the optimization. Now, in order for the
original to _not_ have been close to zero, we're talking about 'a' being
at the very limits of the number space. And remember that on x86 register
arithmetic - regardless of precision setting - the exponent is 15 bits. So
in order to get the overflow, AND to get a original value that wasn't zero
(ie to get an error), 'a' would have had to be on the order of

	2**(16383-64)

which won't even _fit_ in a double anyway (the -64 is to take even
denormals into account, but others have said that flush-to-zero would have
been acceptable anyway by default). Did I mis-do my calculations.

(Repeat argument for underflow).

So the overflow case actually doesn't exist on x86 if we keep the values
in registers or flush to stack in extended real format (and if we do NOT
flush to stack in extended real format, x86 has other problems like the
fact that the results depend on register flushes etc).

Do we have other problems with the conversion? Sure, it might be 1ULP off
even in the non-overflow case, but as we've seen others do that
optimization by default, never mind even with -ffast-math.

So tell me, what makes the optimization so dangerous? It doesn't seem to
do anything that we don't already do.

Did I miss some noteworthy case?

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?

On a 64-bit double processor just take 

	b = c = 2 * sqrt(DBL_MAX);
	a = 0.5 * DBL_MAX

a/(b*c) is 0.0 whereas a/b/c yields 0.125.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
>
> | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
>
> On a 64-bit double processor just take

Oooh.. Quoting me out-of-context.

Please bother to read the next line in my email and not snip it off.

Your arguments are getting weaker and weaker, and dishonest to boot.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
| >
| > | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
| >
| > On a 64-bit double processor just take
| 
| Oooh.. Quoting me out-of-context.

You requested an example. I gave one.

Then for your the purpose of your rhetorics, you said 

   Let's confine this to x86 - it is the most common case by far, and hey,
   especially when it comes to games etc I doubt it matters what HP-PA does,
   for example. We can easily make the optimizations be enabled on a
   per-architecture basis.

I gave a pretty much fair example to illustrate the point that the
transformation does affect the result on 64-bit double processors
example -- they are quite common.

Be honest and admit that you're getting to the point of requesting a
feature for a very specific target where you know the transformation
is safe.  Which is pretty much different from the previous "it won't
matter" stance.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Aug  1, 2001, Gabriel Dos_Reis <gdosreis@sophia.inria.fr> wrote:

> | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?

> On a 64-bit double processor just take 

> 	b = c = 2 * sqrt(DBL_MAX);
> 	a = 0.5 * DBL_MAX

> a/(b*c) is 0.0 whereas a/b/c yields 0.125.

I'm no expert in FP arithmetic, but it appears to me that this chunk
of code is a materialization of what Linus wrote:

> The one obvious case is (b*c) overflows and turns into Inf, at which
> point the end result will be +-0 with the optimization.

If there is a difference, will you care to enlighten me?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Theodore Papadopoulo]

gdr@codesourcery.com said:
> No, I do mean a transformation which does not dramatically transmute
> the computations.  

That's not a definition. Dramatically, can mean very different things 
depending on the people and on the algorithms. That's basically the 
point that I'm trying to make. In many applications and algorithm
the optimization a/b/c to a/(b*c) is meaningful and relevant. Not 
everyone will have floats that are 2*sqrt(DOUBLE_MAX) (or any other 
combination that might overflow). Actually, a lot of people use a 
representation to make computation that is much wider than the 
accuracy of the input and the accuracy you can get on the output.

Let's face it, there are two ways of using floats.

1) you take enough margin to not care about the accuracy problems (ie 
you take several bits more than ``needed''), and then want to write 
your program as naturally as possible and expect the compiler to do 
what is possible for the computation to be as fast as possible.

2) You are constrained either by the accuracy you would like to get 
or by the size of your data. Then you tend to write without these 
extra bits and pay much more attention to the rounding effects.

NONE OF THE MODELS IS BETTER THAN THE OTHER. There should be 
support in gcc for both communities. You can argue that wanting to 
speed up a computation made with doubles that could have been with 
simple floats is stupid, but:

a) double are often as fast or even faster than floats. Until fairly 
recently, a lot of processor were doing float computation by truncating
the results of a double computation. Recent Sparc processor indeed 
have hardware floats, that go faster than doubles but not 2 times 
IIRC.

b) Writing code that extracts every single possible bit of accuracy 
tends to be very difficult to understand and maintain (see eg DNRM2 
or FNRM2 in blas that just computes the Euclidean norm of a vector 
and you will understand what I mean. By the way, I'm sure that 
nowadays, implementing FNMR2 with double would be faster at the same 
accuracy and much simpler to maintain. For DNMR2 things are more 
compilcated as we would have to switch to long double with which I 
have little experience).


The fact that "95% of the folks out there are completely clueless about
floating point. (J.G. 28 Feb. 1998)" have to be taken into account.
For at least quite many of those 95%, floating point works because 
they are not experiencing the "bounds" of floting point and merely 
stay in the safe area provided by the extra bits of accuracy.

By the way the previous sentence is  cited in a report of Kahan and Darcy
about "Java floating point", where they also state the rule (that can be broken 
eventually):

"Evaluate arithmetic expressions and, except possibly for gargantuan 
arrays, declare temporary (local) variable ALL with the widest finite 
precision that is not too slow nor too narrow."

> Where did you get the idea that I meant "mathematically provable safe
> optimisation"? 

First to be sure we understand each other, by mathematically here, I 
reasonning in a mathematical fashion using the rules governing 
floating point computation and trying to assert that two different 
expressions will give the same result.

That is exactly your point, no. You do not want to allow 
optimisations that would not lead to the same results in boundary 
situations (and again, I agree that this is perfectly fine for some 
community of users, my point is that there is another communitie 
which does not care).

> Going into abstract considerations like NP-completness is beside the
> point.  

Agreed, I was not going into such considerations.

--------------------------------------------------------------------
Theodore Papadopoulo
Email: Theodore.Papadopoulo@sophia.inria.fr Tel: (33) 04 92 38 76 01
 --------------------------------------------------------------------


-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Exmh version 2.2 06/23/2000

iD8DBQE7aEiPIzTj8qrxOU4RAg7OAJ9PnH74nWu7XiH23MteF3JrVfdPmQCfQw60
fY1qP8z2WT8l1kpBcABPL9I=
=YCbP
-----END PGP SIGNATURE-----

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Alexandre Oliva <aoliva@redhat.com> writes:

| On Aug  1, 2001, Gabriel Dos_Reis <gdosreis@sophia.inria.fr> wrote:
| 
| > | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
| 
| > On a 64-bit double processor just take 
| 
| > 	b = c = 2 * sqrt(DBL_MAX);
| > 	a = 0.5 * DBL_MAX
| 
| > a/(b*c) is 0.0 whereas a/b/c yields 0.125.
| 
| I'm no expert in FP arithmetic, but it appears to me that this chunk
| of code is a materialization of what Linus wrote:
| 
| > The one obvious case is (b*c) overflows and turns into Inf, at which
| > point the end result will be +-0 with the optimization.
| 
| If there is a difference, will you care to enlighten me?

You're quoting just a tiny part of his rhetorics and that can cause
confusion. 

He used the argument above to derive that for a/b/c not to yeild 0.0,
a should have been very very big, which can't happen in practice.

My example is there to show that is untrue.  A 64-bit double processor
isn't something uncommon and a, b, and c are both representable.

Now, if that transformation is considered only on a x86 target then
that is en entirely different debate -- but then we the discussion
started, that wasn't the assumption.  That restrcition was introduced
at the very end.

This is just an example of where quick extrapolation doesn't work.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Theodore Papadopoulo <Theodore.Papadopoulo@sophia.inria.fr> writes:

| [1  <text/plain; us-ascii (7bit)>]
| 
| 
| gdr@codesourcery.com said:
| > No, I do mean a transformation which does not dramatically transmute
| > the computations.  
| 
| That's not a definition.

Do you have one?

Changing 0.125 to 0.0 is a dramatic change.

| Dramatically, can mean very different things 
| depending on the people and on the algorithms. That's basically the 
| point that I'm trying to make. In many applications and algorithm
| the optimization a/b/c to a/(b*c) is meaningful and relevant.

In the case I showed, I was tring to compute a unit-length vector by
first scaling in order to avoid overflow (the /c part).

I'm not dismissing the fact that there are algorithms which would
benefit from optimizations.  As I said earlier, I'm a fervent
proponent of optimizations.  Not transformations which change the
computation in a very unexpected ways. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 1 Aug 2001, Gabriel Dos Reis wrote:
> |
> | That's not a definition.
>
> Do you have one?
>
> Changing 0.125 to 0.0 is a dramatic change.

Not actually necessarily.

Remember: in order to get that difference you had to be operating with
values in the 2**1024 range.

Compared to those values, the difference between 0.0 and 0.125 can often
be considered truly insignificant.

What "often" means depends on what you're doing with the values, of
course.

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Aug  1, 2001, Gabriel Dos Reis <gdr@codesourcery.com> wrote:

> Alexandre Oliva <aoliva@redhat.com> writes:
> | On Aug  1, 2001, Gabriel Dos_Reis <gdosreis@sophia.inria.fr> wrote:
> | 
> | > | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
> | 
> | > On a 64-bit double processor just take 
> | 
> | > 	b = c = 2 * sqrt(DBL_MAX);
> | > 	a = 0.5 * DBL_MAX
> | 
> | > a/(b*c) is 0.0 whereas a/b/c yields 0.125.
> | 
> | I'm no expert in FP arithmetic, but it appears to me that this chunk
> | of code is a materialization of what Linus wrote:
> | 
> | > The one obvious case is (b*c) overflows and turns into Inf, at which
> | > point the end result will be +-0 with the optimization.
> | 
> | If there is a difference, will you care to enlighten me?

> You're quoting just a tiny part of his rhetorics

Indeed, I failed to quote the other relevant part.  Linus wrote:

> Now, in order for the original to _not_ have been close to zero,
> we're talking about 'a' being at the very limits of the number
> space.

It appears to me that 0.5 * DBL_MAX is indeed at the very limits of
the number space.  So, I still fail to see the difference.

> He used the argument above to derive that for a/b/c not to yeild 0.0,
> a should have been very very big, which can't happen in practice.

But he was explicitly talking about x86's extended precision when he
made this argument.

> My example is there to show that is untrue.  A 64-bit double processor
> isn't something uncommon and a, b, and c are both representable.

I see.  Anyway, it's still true that 0.5 * DBL_MAX is ``at the very
limits of the number space'', so all bets about numbers being ``far
from overflowing'' are off, and optimizations that rely on this
assumption are indeed likely to break programs.

As we have seen, not all FP programs work in ranges close to overflow
limits, so these could benefit from such optimizations M-backspace
transformations.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos_Reis]

| > | > The one obvious case is (b*c) overflows and turns into Inf, at which
| > | > point the end result will be +-0 with the optimization.
| > | 
| > | If there is a difference, will you care to enlighten me?
| 
| > You're quoting just a tiny part of his rhetorics
| 
| Indeed, I failed to quote the other relevant part.  Linus wrote:

More accurately he wrote:

  The one obvious case is (b*c) overflows and turns into Inf, at which point
  the end result will be +-0 with the optimization. Now, in order for the
  original to _not_ have been close to zero, we're talking about 'a' being
  at the very limits of the number space. And remember that on x86 register
  arithmetic - regardless of precision setting - the exponent is 15 bits. So
  in order to get the overflow, AND to get a original value that wasn't zero
  (ie to get an error), 'a' would have had to be on the order of

	  2**(16383-64)

  which won't even _fit_ in a double anyway (the -64 is to take even
  denormals into account, but others have said that flush-to-zero would have
  been acceptable anyway by default). Did I mis-do my calculations.

Emphasis is not mine.  Please note, what he showed: That the
situation (i.e. a/b/c) couldn't have been well-formed in the first
place.  For x86, that may be true.  But that is a specific case. 

| > Now, in order for the original to _not_ have been close to zero,
| > we're talking about 'a' being at the very limits of the number
| > space.
| 
| It appears to me that 0.5 * DBL_MAX is indeed at the very limits of
| the number space.  So, I still fail to see the difference.

I don't consider 0.5 * DBL_MAX to be at the "very limits of the number
space".

| > He used the argument above to derive that for a/b/c not to yeild 0.0,
| > a should have been very very big, which can't happen in practice.
| 
| But he was explicitly talking about x86's extended precision when he
| made this argument.

Yes, he was talking about x86.  But again, when we started discussing
the transformation, there were no limitation to targets.  Again, I
exhibit the example to show that there are out there bunch of
processors where the stance "it won't matter " isn't true.  Now
if the transformation were to be restricted to x86, then that is a
different issue.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Toon Moene]

Gabriel Dos_Reis wrote:

> I don't consider 0.5 * DBL_MAX to be at the "very limits of the number
> space".

Gaby, I'll try a different approach - I'm not going to say you are an
<elided> or your analysis is wrong, flawed, or biased.

I'm just going to point you at the topic of this thread: "What is
acceptable for -ffast-math ?".

Now, try for a moment to think with me.  During the hoolabaloo around
the Pentium FDIV error, I actually went into our weather forecasting
code to count the floating point divisions per grid cell per time step
(70).

The grand total for the whole forecast period was 2.2x10^9.

That was end-of-1994.  Nowadays, we have about 15 times more grid cells
and a 33 % longer forecast period.  Because I don't want to do this
counting exercise again, I'll just assume the same number of divisions
per grid cell, per time step, i.e. 70.

That makes the total 20 x 2.2 x 10^9 ~ 0.5 x 10^11 divisions per
forecast.

If we further assume that:

1) All of these divisions are of the form that allow the transformation.

2) Some forecast will hit your example.

3) The probability distribution for this hit is uniform (this assumption
   is doubtful, because of the value involved).

this means that one in 2^53 / (0.5x10^11) ~ 10^5 forecasts will hit the
error.

I'm sorry, but I do not have the time to wait for the off-chance that
this error will occur, so I'll happily apply this transformation.

Now, even given this, I would certainly heed Roberts advice that the
transformation should actually be a perceptable "optimisation".

In our case, I doubt this (but then, I - personally - am not interested
in anything less than a 10 % improvement).

Cheers,

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
Join GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Nick Ing-Simmons]

Gabriel Dos Reis <gdr@codesourcery.com> writes:
>Joern Rennecke <amylaar@redhat.com> writes:
>
>| > That said, I don't mind about -fdubious-math as far as it is not the
>| > default, nor turned on at any level of optimization.
>| 
>| For a user, it's hard to guess or remember user that -fdubious-math
>| and -ffast-math are related.  How about -ffast-and-loose-math ?
>
>I was actually proposing to rename -ffast-math to -fdubious-math :-)
>but your proposal has also my support :-))

quick-and-dirty-math ?
 
>
>-- Gaby
-- 
Nick Ing-Simmons
http://www.ni-s.u-net.com/

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| and probably a whole lot of other things that are equally "dubious".
| 
| 
| > But then, if they would have written that form in the first place.
| 
| But, as I said, there might be cases where you can't write it in an
| optimized way in the first place, such as in
|   inline double f() { return a/b; }
|   inline double g() { return c;   }
|   x = f()/g();
| plus inlining. 

But then is 0.0 is acceptable for 0.125 for speed?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| On Wed, 1 Aug 2001 dewar@gnat.com wrote:
| >
| > <<The above rules, btw, only apply for exact IEEE math, and on a number of
| > machines you _will_ see zero for the first one.
| > >>
| >
| > Hello! it was you who restricted the discussion to the x86. If we are allowed
| > to go to other machines, we have lots more examples up our sleeves :-)
| 
| Fair enough. But note how I didn't claim that you cannot get a difference
| in the ULP.

0.125 vs. 0.0 isn't a difference in the ULP.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
| >
| > Do you think changing 0.125 to 0.0 is not drastic?
| 
| Ehh. It's 0.125 to 0.125.
| 
| Try it yourself.

I ran an actual program on sparc-v8/solaris-2.7.  That machine is
known to operate with a 64-bits for double -- I specically mentionned
that.  Just try it yourself. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001 dewar@gnat.com wrote:
>
> <<The above rules, btw, only apply for exact IEEE math, and on a number of
> machines you _will_ see zero for the first one.
> >>
>
> Hello! it was you who restricted the discussion to the x86. If we are allowed
> to go to other machines, we have lots more examples up our sleeves :-)

Fair enough. But note how I didn't claim that you cannot get a difference
in the ULP. I _do_ claim that it is not "wrong" unless you specifically
ask for IEEE math - by virtue of pointing to lots of other machines that
do the same (including, apparently, the most modern architecture on the
planet. I don't have one, I can't test).

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Gabriel Dos_Reis wrote:
>
> Do you think changing 0.125 to 0.0 is not drastic?

Ehh. It's 0.125 to 0.125.

Try it yourself.

That's not a "drastic difference", not if you actually do what I
explained. If you don't have access to a x86 machine, maybe you can ask
somebody else to test the following appended for you.

Surprise, surprise, they come out exactly the same. Even down to the bit
representation (in case you care).

Not that I'm arguing that they will _always_ be the same down to the ULP,
but I _am_ arguing that at least on x86 it's actually a safe optimization.
You should disable it for -mieee.

		Linus

----
#include <math.h>

void test(double, double, double);

main()
{
        test(0.5*DBL_MAX, 2*sqrt(DBL_MAX), 2*sqrt(DBL_MAX));
}

void test(double a, double b, double c)
{
        printf("%f\n%f\n", a/b/c, a/(b*c));
}

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001 dewar@gnat.com wrote:
>
> a/2.0/2.0 will never yield zero if a is non-zero
> a/4.0 may indeed yield zero if a is the smallest number

Didn't you already agree that FTZ was acceptable even for a default mode,
much less -ffast-math?

The above rules, btw, only apply for exact IEEE math, and on a number of
machines you _will_ see zero for the first one.

So I don't think your example is a valid argument for not using the
optimization - even without -ffast-math.

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Wolfgang Bangerth]

> The transformations in question aren't documented.  So, those having
> "valid reasons to request them" don't even know what there are
> requesting with -fdubious-math.

Well, but then we're talking about a/b/c -> a/(b*c)  "and the like", as
  a*c+b*c -> (a+b)*c
  a*x/a   -> x
and probably a whole lot of other things that are equally "dubious".


> But then, if they would have written that form in the first place.

But, as I said, there might be cases where you can't write it in an
optimized way in the first place, such as in
  inline double f() { return a/b; }
  inline double g() { return c;   }
  x = f()/g();
plus inlining. You can't just rewrite this into a form that does the
optimized computation without the help of the compiler. However, I'd bet
that for the lower parts of the compiler the resulting stream of commands
look just the same as if it were  x=a/b/c  right away.

Regards
  Wolfgang

-------------------------------------------------------------------------
Wolfgang Bangerth          email: wolfgang.bangerth@iwr.uni-heidelberg.de
                             www: http://gaia.iwr.uni-heidelberg.de/~wolf

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

The transformations in question aren't documented.  So, those having
"valid reasons to request them" don't even know what there are
requesting with -fdubious-math.

Right now, we're discussing the ability of the compiler to transform
something like a/b/c into a/(b*c).  Some argue on the ground of speed
considerations, that the programmer would expect the compiler to do
that. But then, if they would have written that form in the first 
place.

The issue isn't about "freedom vs. no-freedom".

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Wolfgang Bangerth]

Tim Hollebeek wrote:
> Rewriting the programmers program in such a way that he gets
> significantly less accuracy than he could reasonably expect is just
> silly.

Of course it is, but that is not the point that is being discussed: nobody
proposed doing these transformations at -O2. If you don't want these
transformations, then don't switch on -fsomething. But if, for some
certainly valid reason you don't want to use these transformations, why
shouldn't other people have valid reasons to request them?

Regards
  Wolfgang


-------------------------------------------------------------------------
Wolfgang Bangerth          email: wolfgang.bangerth@iwr.uni-heidelberg.de
                             www: http://gaia.iwr.uni-heidelberg.de/~wolf

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Tue, 31 Jul 2001 dewar@gnat.com wrote:
>
> I still see *no* quantitative data showing that the transformation most
> discussed here (associative redistribution) is effective in improving
> performance.

Actually, the transformation that was discussed here was really

	a/b/c -> a/(b*c)

and I don't know where the associative redistribution thing really came
from. Probably just an example of another thing like it..

> Linus seems to be getting a little emotional in this discussion but swearing
> does not replace data.

Hey, I called people silly, not <censored>. You must have a very low
tolerance ;)

And the replacement-with-reciprocal kind of thing can certainly make a
difference. I'm too lazy to dig out my old quake3 source CD to try to dig
up what I ended up doing there, though.

> This seems such a reasonable position that I really have a heck of a time
> understanding why Linus is so incensed by it.

I'm incensed by people who think that the current -ffast-math already does
something that is positively EVIL - even though the current setup with
inline replacements of "fsin" etc has been there for years.

> Let's take an example, which is multiplication by a reciprocal. This of
> course does give "wrong" results. But division is often quite slow. On the
> other hand, I can't imagine floating-point programmers, including those
> doing game stuff, not knowing this, and thus writing the multiplication
> in the first place.

Actually, in the case of quake3, I distinctly remember how the x86 version
actually used division on purpose, because the x86 code was hand-tuned
assembler and they had counted cycles (for a Pentium machine) where the
division overlapped the other arithmetic.

The x86 hand-tuned stuff also took care to fit inside the FP stack etc -
they had started off with a C version and compiled that, and then once
they got as far as they could they just extracted the asm and did the rest
by hand.

It was also the much more obvious algorithm to do a division.

On alpha (and hey, this was before the 21264), the same did not hold true,
and it was faster to do a reciprocal. But it's been too long.

> <<Oh, round-to-zero is definitely acceptable in the world of "who cares
> about IEEE, we want fast math, and we'll use fixed arithmetic if the FP
> code is too slow".
> >>
>
> That's probably a bad idea, on most modern machines, full IEEE floating-point
> is faster than integer arithmetic (particularly in the case of multiplication)

Ehh.. You seem to think that "hardware == fast".

Denormal handling on x86 is done in microcode as a internal trap in the
pipeline. Round-to-zero is noticeably faster - even when the _code_ is the
same. Which is why Intel has special mode flags - where it internally
rounds to zero instead of making a denormal.

I'm not kidding. Look at the "Denormals are zero flag", bit 6 in the
MXCSR.

And I quote (all typos are probably mine):

 "The denormals-are-zero mode is not compatible with IEEE Standard 754.
  The denormals-are-zeros mode is provided to improve processor
  performance for applications such as streaming media processing, where
  rounding a denormal to zero does not appreciably affect the quality of
  the procesed data"

What I'm saying is that a buge company like Intel chose to do this
optimization IN HARDWARE because they thought that (a) it was meaningful
and (b) they thought the problem warranted it.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 31 Jul 2001, Gabriel Dos Reis wrote:
>
> What I intended to mean is that the compiler has, first, to prove that
> such transformmation yields the same result.

Ehh..

You're _wrong_.

A lot of compiler optimizations yield different results. The question is
whether it is acceptable or not.

It _clearly_ is acceptable to do when the user has explicitly asked for
"-ffast-math". The whole meaning of the flag is to say "I want it fast,
even if you might lose some range or precision".

It's only a matter of defining the semantics.

There are other optimizations like that. Signed integer overflow is
undefined, so you can optimize integer operations in ways that do not give
the same results for all integers.

Alias analysis was added several years ago, even though it was _clearly_ a
"illegal optimization" from a traditional C standpoint. That one broke
tons of real programs. Again, it was a question of re-defining the
semantics, and allowing people to take advantage of them.

Already, the _default_ behaviour for gcc at least on x86 with floating
point has very little to do with IEEE. It's been discussed several times
how to fix it - and very little real work on it has been done for the last
several years. Why? Because it really doesn't matter for most people, and
the current work-arounds are too much of a performance hit.

So gcc _already_ has "better fast than correct to the last ULP in IEEE".
Live with it. Even by default. Your argument that it cannot do the same
even when the user asks for it is completely ludicrous.

Your continual insistence that it is "wrong" only shows that you have
built a cage around your brain.

Have you even _looked_ at what the gcc definition for "fast-math" are?
Look again:

  {"fast-math", &flag_fast_math, 1,
   "Improve FP speed by violating ANSI & IEEE rules" },

and realize that even something simple as "double" doesn't historcially
have an exact meaning in C (ie you can _not_ assume specific ranges or
specific IEEE rounding in traditional C - it was left up to the
implementor to do as best he could with what hardware was available).

That means that traditionally, you don't even know what overflows or not,
or what precision you get. It's historically "undefined". Which means that
the semantics are not clear.

Now, this is obviously why a lot of computational people disliked,
distrusted and didn't use C historically, preferring to use Fortran which
had more defined rules. And which is, of course, why C99 integrated the
math extensions that some people have been working on for a loong time.

But you have to realize that this is (a) fairly recent and (b) not
everybody needs or wants it.

And you should also realize that YOU are not the grand poobah that gets to
tell people what semantics they want.

[ Neither am I, of course, but I'm arguing for choice and freedom, while
  you're arguing for small-mindedness everywhere. ]

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Tue, 31 Jul 2001 dewar@gnat.com wrote:
>
> Well it sure would be nice to here from some of these mythical numerical
> programmers (I don't care if they are writing games or nuclear reactor codes)
> who would be happier, so far we haven't heard this! And in my experience,
> even quite inexperienced floating-point numerical programmers are very
> disturbed when optimization changes the results of their programs.

I used -ffast-math myself, when I worked on the quake3 port to Linux (it's
been five years, how time flies).

It didn't make much difference at that point, because the x86 had
hand-written assembly, and gcc for the alpha didn't do much (anything?)
with -ffast-math.

But I tried _everything_. The main FP work I did on that thing on the
alpha improved the framerate by about 50% on alpha - FP was _that_
critical for it.  Most of it was by looking at gcc output and trying to
re-organize the C code to make it be better (because gcc didn't do much on
its own).

And yes, it was exactly things like multiplying by reciprocals.

> > Your arguments about "numerical computation" are just silly, as you don't
> > seem to realize that there are tons of problems where your theoretical
> > issues are nothing more than noise.
>
> If you think the arguments are silly, then I really fear you lack the full
> context for this discussion, a discussion that has, as you should know raged
> for well over thirty years.

Most of the kind of "numercial" work that you seem to be talking about has
probably rather little to do with FP performance. Most of the traditional
heavy FP code tends to be _much_ more about cache layout and good memory
access patterns.

I'm personally aquainted with tryign to make a game engine go fast, where
the memory effects are fewer, and the FP itself is the bottleneck.

> Sure -ffast-math is precisely intended to allow transformations that would
> not otherwise be allowed (let's not call them optimizations, that's just
> too controversial a word in the context of this argument).

Why not call them optimizations? They are. The only thing we change is the
boundary of valid ranges.

> The question is what is the boundary of allowable transformations. No one
> agrees that there should be no boundaries (even you don't like the change
> results to zero, though note that abandoning denormals has exactly this
> effect, and might be considered acceptable).

Oh, round-to-zero is definitely acceptable in the world of "who cares
about IEEE, we want fast math, and we'll use fixed arithmetic if the FP
code is too slow".

In fact, it is _so_ acceptable that CPU designers design for it. Look at
MMX2, and wonder why they have a RTZ mode? Because it makes the _hardware_
go faster.

That should tell you something. Big companies that have billion-dollar
fabs spend time optimizing their chips that take several years to design
for _games_. Not for IEEE traditional Fortran-kind math.

But apparently some gcc developers don't think that is even a worthy
market, because you just want to do fluid dynamics.

> So, what is the boundary, can one for instance forget about denormals and
> flush to zero to save a bit of time, can one truncate instead of round,
> can one ignore negative zeroes, or infinity semantics, can one ignore
> intermediate overflow (note: nearly all the discussed transformations are
> implicitly saying yes to this last question).

Do them all by default with -ffast-math.

Then, you can have specific flags for people who want just _one_
optimization. I doubt you'll find many users who do that, but maybe I'm
wrong. Giving people the choice is always a good idea.

> I have not seen anyone writing from the point of view of serious numerical
> coding saying [ .. ]

There you go again. What the hell do you call "serious numerical coding"?

Take a look at the computer game market today. It's a lot more serious
than most matematicians puttering around in their labs, let me tell you.
That's a BIG industry.

Also note that _nobody_ in your kind of "serious numerical coding"
community would ever worry about "-ffast-math" in the first place. Why the
hell would they, when 99% of the time it doesn't make any difference at
all. The people you apparently consider serious are a lot more interested
in fast communication (so that they can solve the thing in parallell) and
incredible memory bandwidth.

I doubt you'll find many of your "serious numerical coding" people who
would even _notice_ the raw FP throughput. Look at SpecFP - CPU's are fast
enough, it spends most of its time waiting on memory.

> Should -ffast-math allow full precision operation? I would think so,
> since it definitely improves performance, and reduces surprises.

Ehh.. gcc right now allows full precision operation BY DEFAULT. No
-ffast-math required.

Same goes for negative zero as far as I remember - on some HP-PA stuff at
least. Simply because it was too painful to get "right" on their earlier
hardware.

In short, what you seem to argue that -ffast-math should means are all
things gcc _already_ does, with no

> By the way, I said I would be shocked to find a Fortran compiler that did
> associative redistribution in the absence of parens. I am somewhat surprised
> that no one stepped forward with a counter-example, but I suspect in fact that
> there may not be any shocking Fortran implementations around.

I would suspect that very few people have Fortran compilers around and
bother to check it.

> It is an old argument, the one that says that fpt is approximate, so why bother
> to be persnickety about it. Seymour Cray always tool this viewpoint, and it
> did not bother him that 81.0/3.0 did not give exactly 27.0 on the CDC 6000
> class machines.

..and he was universally respected for making the fastest machines around.

What you forget to mention is that these days it's so _cheap_ to get
IEEE, that from a hardware standpoint pretty much everybody includes it
anyway.

But they then often throw it away because it ends up having expensive
run-time issues (alpha with exception handling and proper denormals, Intel
with special RTZ modes etc).

Why? Because in many areas Seymour Cray is _still_ right. The thing that
killed off non-IEEE was not that he was wrong, but the fact that _some_
people do need IEEE "exact FP". Not everybody. Not even the majority. But
because some people do need it, you need to support it. Which is why
everybody does, today.

But do you see the difference between

  "We have to support it because a portion of the user base has to have
   it, and if we don't have it we won't be able to sell to any of that
   user base"

and

  "Everybody must use it, because anything else is wrong"

Eh?

Do you see that Seymours approach didn't fail because he was always wrong?
It failed because he was _sometimes_ wrong.

And you know what? He was right enough of the time to have built up an
empire for a while. That's something not everybody can say about
themselves. And that is something that you should respect.

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Mon, 30 Jul 2001 dewar@gnat.com wrote:

> >>Well, I won't be so rash as to say that it should do that *by default*,
>
> But that of course is the issue under discussoin :-)

No it is not.

See the subject line.

Doing it by default was just one possible suggestion.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Mark Mitchell]

>> OTOH, you could re-use (abuse) statement expressions for this purpose:
>
>> ({a+b;})*c

Goodness.  I have been away from email for a few days, and now I'm back,
but I'm considering going away from email again and not coming back. :-)
:-)

If the worst problem in GCC is that you cannot specify optimization
barriers in the midst of expressions then we are in very good shape
indeed!

Let's not go down this path.  In my opinion, we really do not want
to introduce this kind of thing.  There is much more win simply in
fixing bugs and limitations in the current optimizers.

-- 
Mark Mitchell                mark@codesourcery.com
CodeSourcery, LLC            http://www.codesourcery.com

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Joseph S. Myers]

On 30 Jul 2001, Alexandre Oliva wrote:

> Would you prefer
> 
> __extension__ __attribute__ ((optimize=0)) (a*b) +
> __extension__ __attribute__ ((optimize=0)) (a*c)
> 
> ? :-)
> 
> I'd much rather write ((a*b)) + ((a*c)), which will even compile with
> other compilers, and might be picked up in a future revision of the C
> and C++ Standards.

I'd rather you use #pragma STDC FP_CONTRACT OFF - while this seems to be
intended to disable optimisations that give more accurate results, it
should probably disable optimisations here that may give less accurate
results as well.

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> > Further, GCC is supposed to be standards compliant by default.
> 
> Really?

Well, then I'd like to understand the actual policy better.  You have
GNU extensions but otherwise seem to include what's in the language
definition.  The apparent conflicts arise when you say the
GNU extensions have to be turned off in order for the compiler
to be strictly compliant with the standard.

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:

> Fergus Henderson <fjh@cs.mu.oz.au> writes:
> |
> | I think it would be nice if GCC was standards compliant by default,
> | but that is certainly not the status quo.
>
> Agreed.  But that should be the ideal we should strive for.
> And also, I don't think that is an argument for not computing
> correctly, even if we're asked to compute fast.

Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
expression.

So it's certainly very very debatable whether you can say "not computing
correctly" for it - the user has not actually shown a strong preference
for the double division, and it is entirely possible that the double
division has come about as a result of macro expansion and/or inline
function expansion.

Who are you to say that the user wouldn't like the code to be run faster?
I don't think this was well-defined in C before C99, for example.

Note that with the current -ffast-math on x86, for example, gcc will
already inline things like "sin()" etc using the x87 instruction directly.
Despite the fact that that instruction is well-defined only for the range
-2**63 to +2**63.

And now there are people that say that something simple as "a/b/c" should
not be optimized to "a/(b*c)" even with -ffast-math.

What planet are you people from?

We are already in the situation that -ffast-math gives a value for "sin()"
that is not even _close_ to the range of [-1,1] under various
circumstances. And you're complaining about the fact that b*c can
overflow? We've already effectively limited our argument range to +-2**63,
and there's no way a*b can overflow, guys.

So even if you don't want to make it an optimization by default with the
C99 argument (hey, I don't really disagree), there is absolutely _no_
valid argument for not doing it with -ffast-math.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Mon, 30 Jul 2001 dewar@gnat.com wrote:
>
> But if you look at real Fortran codes, they are full of parentheses to stop
> the compiler from messing around, and in fact most Fortran compilers do only
> very limited mucking even when no parens are present. I would be shocked
> to find a FOrtraqn compiler that changed
>
> a*c + b*c
>
> into
>
> (a+b)*c
>
> a totally diffrerent expression!

You'd be shocked?

What if I told you that it's actually legal? Go check it out.

Note that the mark of a language that generates fast code is that it both
(a) allows the compiler the maximum possible flexibility to do the best
job it can, and (b) allows the programmer to specify what the limits are.

Fortran, while being a pretty horrible language for anything else, does
exactly this for FP math. It doesn't do it for much anything else, of
course.

Now, the problem here is that what is permissible in Fortran is not
permissible in C, but I think we should _allow_ it to happen.

Note that I'm not really advocating breaking standards, like so many
people seem to think. If you actually go back and read my emails instead
of just reacting to them by gut feel, you'll notice that what I really
suggested was to make it _easy_ for the non-math programmer to tell the
compiler "do the best you can with this".

I argue _against_ a lot of magic compiler flags that are version-
dependent and hard to use, and I argue _for_ "-ffast-math" being very
strong.

I also argue that since we already have more standards than we can shake a
stick at (traditional, c89, gnu89, c99, etc), and C traditionally has not
been very strict about what optimizations are permissible, the argument of
"conforms to C99 by default" is not a very strong one. But I also noted
that it _is_ a very valid argument, although not as absolute as some
people seem to think it is (exactly because we do have multiple standards,
and gcc already defaults to a standard of its own rather than anything
else).

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Sun, 29 Jul 2001 dewar@gnat.com wrote:
>
> <<Further, GCC is supposed to be standards compliant by default.  You
> seem to be advocating that GCC should, without warning, generate
> programs that violate the standard and are mathematically wrong.
> >>
>
> and that violate the expectations of programmers who *do* know what they
> are doing!

Note that most of the people I know who _do_ know what they are doing also
want to have the option o ftelling the compiler when they don't care, and
want to allow the compiler to do "illegal" things if that allows the
compiler to generate better code.

It's sad that C99 doesn't seem to allow the Fortran kind of "you go and
make the best you can of this expression unless I tell you otherwise".
Which means that we'd _have_ to be non-standard to allow the optimization
at all. Which obviously means a flag (one way or the other).

Rather than have a good way to do it on a case-by-case basis, which is
probably what the people who _really_ care would prefer.

In C, you can't reasonably use the Fortran "parenthesis are significant"
approach - the way the C pre-processor works you often have lots of extra
parenthesis for that reason. However, C does have the well-established
sequence point rule, so it would make sense to have that kind of
extension.

And hey, if you want to make it be an extra flag, just make it a _simple_
one. Like "-ffast-math". Don't make people have to know 100 esoteric flags
that are compiler-release-specific. That's completely useless.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Sun, 29 Jul 2001, Stephen L Moshier wrote:
>
> > > Floating point arithmetic, IEEE or otherwise, does not obey the
> > > associative or distributive laws.
> >
> > This is YOUR opinion.
>
> That is not just an opinion, it is simple arithmetic.

No, it's actually a very complex issue of representing classical
arithmetic with finite values. It's not "simple arithmetic" at all. In the
simple arithmetic, operations _are_ associative and distriubutive.

And what I'm really claiming is that until gcc is a full Artifical
Intelligence system, it cannot add information to a program.

In 99% of all cases, the information of the subtleties of floating point
arithmetic simply was never there in the program gcc tries to compile.
Why? Becuase the programmer didn't even _intend_ any of the subtleties, or
even knew about them.

Gcc trying to enforce them doesn't add any value.

> Further, GCC is supposed to be standards compliant by default.

That is a separate argument, and I don't think you'r enecessarily wrong on
that front. On the other hand, people are already used to having command
line switches for strict standards adherence, so I don't think that this
argument is necessarily a very strong one, more a matter of opinion.

Especially true since the standard you quote is a fairly recent one (has
it been finalized now?)

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> > Floating point arithmetic, IEEE or otherwise, does not obey the
> > associative or distributive laws.
> 
> This is YOUR opinion.

That is not just an opinion, it is simple arithmetic.

Further, GCC is supposed to be standards compliant by default.  You
seem to be advocating that GCC should, without warning, generate
programs that violate the standard and are mathematically wrong.
The C99 standard says the following, among other things, about floating
point transformations:


       [#14] EXAMPLE 5 Rearrangement for floating-point expressions
       is  often  restricted because of limitations in precision as
       well as range.  The implementation  cannot  generally  apply
       the   mathematical   associative   rules   for  addition  or
       multiplication,  nor  the  distributive  rule,  because   of
       roundoff   error,  even  in  the  absence  of  overflow  and
       underflow.   Likewise,  implementations   cannot   generally
       replace decimal constants in order to rearrange expressions.
       In  the  following  fragment,  rearrangements  suggested  by
       mathematical rules for real numbers are often not valid (see
       F.8).

               double x, y, z;
               /* ... */
               x = (x * y) * z;  // not equivalent to x *= y * z;
               z = (x - y) + y ; // not equivalent to z = x;
               z = x + x * y;    // not equivalent to z = x * (1.0 +
               y);
               y = x / 5.0;      // not equivalent to y = x * 0.2;