Re: associative law in combine

Re: associative law in combine

[Jan Hubicka]

> Jan Hubicka wrote:
> 
> > > Sure, but not in fold-CONST.c :-)
> > OK, what is the apropriate place for that then? We simply need it to be done
> > at tree level.  It is FOLD-const after all :)
> 
> Unfortunately, I don't know.
> 
> Perhaps just asking it on the list is best.
OK, so to speak loud - where is the proper place to convert
a/b/c to a/(b*c) at tree level. fold-const or some other?

Honza

Re: associative law in combine

[Dima Volodin]

On Tue, 17 Jul 2001 21:21:25 +0200, you wrote:

>> Jan Hubicka wrote:
>> 
>> > > Sure, but not in fold-CONST.c :-)
>> > OK, what is the apropriate place for that then? We simply need it to be done
>> > at tree level.  It is FOLD-const after all :)
>> 
>> Unfortunately, I don't know.
>> 
>> Perhaps just asking it on the list is best.
>OK, so to speak loud - where is the proper place to convert
>a/b/c to a/(b*c) at tree level. fold-const or some other?

Are you sure you want to do that at all? These two expressions have
different overflow properties and might produce different results.

>Honza

Dima

Re: associative law in combine

[Joern Rennecke]

> 
> > Jan Hubicka wrote:
> > 
> > > > Sure, but not in fold-CONST.c :-)
> > > OK, what is the apropriate place for that then? We simply need it to be done
> > > at tree level.  It is FOLD-const after all :)
> > 
> > Unfortunately, I don't know.
> > 
> > Perhaps just asking it on the list is best.
> OK, so to speak loud - where is the proper place to convert
> a/b/c to a/(b*c) at tree level. fold-const or some other?

Only if b and c are constants, the operations are floating point, and
b can be multiplied with c without loss of precision or overflow.
Or if b and/or c is a power of two, and b can be multiplied with c without
overflow.

Re: associative law in combine

[Toon Moene]

Joern Rennecke wrote:

> > > Jan Hubicka wrote:

> > OK, so to speak loud - where is the proper place to convert
> > a/b/c to a/(b*c) at tree level. fold-const or some other?
> 
> Only if b and c are constants, the operations are floating point, and
> b can be multiplied with c without loss of precision or overflow.
> Or if b and/or c is a power of two, and b can be multiplied with c without
> overflow.

Joern is right, Jan.  One can argue about the loss of precision (under
unsafe math optimisations), but not the overflow.  I overlooked that
issue in my reply to you.  Because overflow can only be determined at
compile time with constants, this conversion cannot be right for
variables.

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
Join GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

What is acceptable for -ffast-math? (Was: associative law in combine)

[Jan Hubicka]

> Joern Rennecke wrote:
> 
> > > > Jan Hubicka wrote:
> 
> > > OK, so to speak loud - where is the proper place to convert
> > > a/b/c to a/(b*c) at tree level. fold-const or some other?
> > 
> > Only if b and c are constants, the operations are floating point, and
> > b can be multiplied with c without loss of precision or overflow.
> > Or if b and/or c is a power of two, and b can be multiplied with c without
> > overflow.
> 
> Joern is right, Jan.  One can argue about the loss of precision (under
> unsafe math optimisations), but not the overflow.  I overlooked that
> issue in my reply to you.  Because overflow can only be determined at
> compile time with constants, this conversion cannot be right for
> variables.
OK, I believe, that our concept of unsafe_math_optimizations allows such
transformation, but I see, that the line between acceptable
unsafe_math_optimization and unacceptable one is pretty fragine.

In case -ffast-math is not enought to ask for such transformation,
we probably should invent switch for that.  This change (plus the
other changes to avoid divisions) seems to play important role
on optimizing some FP software (due to extreme cost of fp division).

Honza

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Prince]

----- Original Message -----
From: "Jan Hubicka" <jh@suse.cz>
To: "Toon Moene" <toon@moene.indiv.nluug.nl>
Cc: "Joern Rennecke" <amylaar@redhat.com>; "Jan Hubicka" <jh@suse.cz>;
<gcc@gcc.gnu.org>
Sent: Wednesday, July 18, 2001 1:44 AM
Subject: What is acceptable for -ffast-math? (Was: associative law in
combine)


> > Joern Rennecke wrote:
> >
> > > > > Jan Hubicka wrote:
> >
> > > > OK, so to speak loud - where is the proper place to convert
> > > > a/b/c to a/(b*c) at tree level. fold-const or some other?
> > >
> > > Only if b and c are constants, the operations are floating point,
and
> > > b can be multiplied with c without loss of precision or overflow.
> > > Or if b and/or c is a power of two, and b can be multiplied with c
without
> > > overflow.
> >
> > Joern is right, Jan.  One can argue about the loss of precision
(under
> > unsafe math optimisations), but not the overflow.  I overlooked that
> > issue in my reply to you.  Because overflow can only be determined
at
> > compile time with constants, this conversion cannot be right for
> > variables.
> OK, I believe, that our concept of unsafe_math_optimizations allows
such
> transformation, but I see, that the line between acceptable
> unsafe_math_optimization and unacceptable one is pretty fragine.
>
> In case -ffast-math is not enought to ask for such transformation,
> we probably should invent switch for that.  This change (plus the
> other changes to avoid divisions) seems to play important role
> on optimizing some FP software (due to extreme cost of fp division).
>
> Honza

This optimization is specifically permitted for appropriate data types
by the Fortran standard, with the reservation that parentheses employed
to prevent re-association must be observed.  As gcc is unable to
distinguish between the expressions (a/b)/c and a/b/c, this optimization
would violate the standard in the former case, so is undesirable for
enabling under -ffast-math.

When Honza and I first discussed this, and also the transformation

for(i=0;i<n;++i)
    a[i]=b[i]/c;

to

for(i=0;i<n;++i)
    a[i]=b[i]*(1/c);

I mentioned that these would put too many risks in the -ffast-math
category.  I would like to see -ffast-math permit only those
optimizations outside of IEEE compliance which are generally expected
(e.g. according to Fortran standard).

For gcc-3.1, I would like to see an additional switch to permit
optimizations such as the ones on floating point division mentioned
above.  I don't care about the name, but it should be more mnemonic than
the ones used by commercial compilers; something like -ffast-div.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Jan Hubicka]

> 
> This optimization is specifically permitted for appropriate data types
> by the Fortran standard, with the reservation that parentheses employed
> to prevent re-association must be observed.  As gcc is unable to
> distinguish between the expressions (a/b)/c and a/b/c, this optimization
> would violate the standard in the former case, so is undesirable for
> enabling under -ffast-math.
Our documentation of -ffast-math seems to be wondefully incomplette:

This option causes the preprocessor macro @code{__FAST_MATH__} to be defined.

This option should never be turned on by any @option{-O} option since
it can result in incorrect output for programs which depend on
an exact implementation of IEEE or ISO rules/specifications for
math functions.

But my understanding is, that -ffast-math allow compiler to do transformations
not strictly correct, but working in was majority of cases.  The
transofrmations mentioned above looks like good example of these.

I would like to keep them under -ffast-math for simple purpose, that users
tends to know about -ffast-math and use it, but don't know about possibly new
functions.  I believe that the typical usage of Joe user will be to try
"-ffast-math" and if don't work and he is really interested in the
perofrmance, he will try the suboptions.

GLIBC for instance does inlining of some math functions on -ffast-math that
gives inexact results.

I agree with idea of adding new options to control this feature, but as
options of -ffast-math, as the -funsafe-math-optimizations, -ftrapping-math
and -ferrno-math is currently implemented.
> 
> When Honza and I first discussed this, and also the transformation
> 
> for(i=0;i<n;++i)
>     a[i]=b[i]/c;
> 
> to
> 
> for(i=0;i<n;++i)
>     a[i]=b[i]*(1/c);
I've already implemented this transformation.
> 
> I mentioned that these would put too many risks in the -ffast-math
> category.  I would like to see -ffast-math permit only those
> optimizations outside of IEEE compliance which are generally expected
> (e.g. according to Fortran standard).
You do have experience with this options.  Do you know examples of programs
where it fails?

For instance we already do reassociation with -ffast-math
> 
> For gcc-3.1, I would like to see an additional switch to permit
> optimizations such as the ones on floating point division mentioned
> above.  I don't care about the name, but it should be more mnemonic than
> the ones used by commercial compilers; something like -ffast-div.
This sounds resonable name.  If no one complains, I will add it in the
sense mentioned above.

Honza

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Prince]

----- Original Message -----
From: "Jan Hubicka" <jh@suse.cz>
To: "Tim Prince" <tprince@computer.org>
Cc: "Jan Hubicka" <jh@suse.cz>; "Toon Moene"
<toon@moene.indiv.nluug.nl>; "Joern Rennecke" <amylaar@redhat.com>;
<gcc@gcc.gnu.org>
Sent: Sunday, July 29, 2001 6:31 AM
Subject: Re: What is acceptable for -ffast-math? (Was: associative law
in combine)


> >
> > This optimization is specifically permitted for appropriate data
types
> > by the Fortran standard, with the reservation that parentheses
employed
> > to prevent re-association must be observed.  As gcc is unable to
> > distinguish between the expressions (a/b)/c and a/b/c, this
optimization
> > would violate the standard in the former case, so is undesirable for
> > enabling under -ffast-math.
> Our documentation of -ffast-math seems to be wondefully incomplette:
>
> This option causes the preprocessor macro @code{__FAST_MATH__} to be
defined.
>
> This option should never be turned on by any @option{-O} option since
> it can result in incorrect output for programs which depend on
> an exact implementation of IEEE or ISO rules/specifications for
> math functions.
>
> But my understanding is, that -ffast-math allow compiler to do
transformations
> not strictly correct, but working in was majority of cases.  The
> transofrmations mentioned above looks like good example of these.
>
> I would like to keep them under -ffast-math for simple purpose, that
users
> tends to know about -ffast-math and use it, but don't know about
possibly new
> functions.  I believe that the typical usage of Joe user will be to
try
> "-ffast-math" and if don't work and he is really interested in the
> perofrmance, he will try the suboptions.
>
> GLIBC for instance does inlining of some math functions on -ffast-math
that
> gives inexact results.
Thanks for bringing this up.  This is an example of inflicting too many
dangerous shortcuts
under one option; worse, an option which kicks in by default already
at -O, in violation
of the rule you quoted above.  I myself use a saner version of
<mathinline.h>.  It allows
gcc-3.1 to out-perform g77, without all those crashes on exceptional
arguments.  (OT)
I don't see the decisions about options made by gcc having much
influence on glibc, nor
should the decisions made for glibc influence gcc when they don't make
sense.  I see a
need now for some changes in math library support for P4, but the level
of interest may
be insufficient.
>
> I agree with idea of adding new options to control this feature, but
as
> options of -ffast-math, as
the -funsafe-math-optimizations, -ftrapping-math
> and -ferrno-math is currently implemented.
This seems OK if we have a documented way to turn off individually the
sub-options
of -ffast-math.  I have been experimenting this morning with -ffast-math
and trying to
over-ride the individual options.  I think it works,
but -march=pentiumpro -ffast-math
has lost its good effect for P-III in g77-3.1 now that short IF blocks
are producing
jumps out of line.  gcc has not picked up this behavior, but I see now
that -ffast-math
is no longer required to obtain efficient code for floating point
compare and branch.
> >
> > When Honza and I first discussed this, and also the transformation
> >
> > for(i=0;i<n;++i)
> >     a[i]=b[i]/c;
> >
> > to
> >
> > for(i=0;i<n;++i)
> >     a[i]=b[i]*(1/c);
> I've already implemented this transformation.
> >
> > I mentioned that these would put too many risks in the -ffast-math
> > category.  I would like to see -ffast-math permit only those
> > optimizations outside of IEEE compliance which are generally
expected
> > (e.g. according to Fortran standard).
> You do have experience with this options.  Do you know examples of
programs
> where it fails?
The pre-inversion may make noticeable differences in accuracy, and those
may well
be of concern.  It shouldn't fail on an IEEE-compliant architecture, in
the normal
course of events.  As others pointed out, the re-association of division
makes
opportunities for changes in over- and under-flow behavior, from which
IA32
architectures will no longer be protected, with the increasing use of
SSE
instructions.

In a large application, if something breaks when turning on a family of
unsafe
optimizations, when there is no documented way of turning them on
individually,
the only practical solution may be to do without the optimizations.  I
have spent
weeks on a binary search to find out which methods in a C++ application
were
breaking with optimization, on a platform for which g++ is not
available.
>
> For instance we already do reassociation with -ffast-math
> >
> > For gcc-3.1, I would like to see an additional switch to permit
> > optimizations such as the ones on floating point division mentioned
> > above.  I don't care about the name, but it should be more mnemonic
than
> > the ones used by commercial compilers; something like -ffast-div.
> This sounds resonable name.  If no one complains, I will add it in the
> sense mentioned above.
>
> Honza

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Jan Hubicka]

> Thanks for bringing this up.  This is an example of inflicting too many
> dangerous shortcuts
> under one option; worse, an option which kicks in by default already
> at -O, in violation
> of the rule you quoted above.  I myself use a saner version of
> <mathinline.h>.  It allows

The mathinline is mostly mess.  I was thinking about rewriting it myself.
It seems to make sense to make target specific builtins for the i387
features the header needs and use them, instead of using mostly unusable
i387 inline assembly.

If you have sanified version and you can get the copyring issues,
please try to contribute the changed to glibc.

> I don't see the decisions about options made by gcc having much
> influence on glibc, nor
> should the decisions made for glibc influence gcc when they don't make
> sense.  I see a
> need now for some changes in math library support for P4, but the level
> of interest may
> be insufficient.

The description quoted above states, that FLAG_FAST_MATH is defined and
so glibc is correct IMO to change the behaviour.

> >
> > I agree with idea of adding new options to control this feature, but
> as
> > options of -ffast-math, as
> the -funsafe-math-optimizations, -ftrapping-math
> > and -ferrno-math is currently implemented.
> This seems OK if we have a documented way to turn off individually the
> sub-options
> of -ffast-math.  I have been experimenting this morning with -ffast-math
> and trying to
> over-ride the individual options.  I think it works,
> but -march=pentiumpro -ffast-math
> has lost its good effect for P-III in g77-3.1 now that short IF blocks
> are producing
> jumps out of line.  gcc has not picked up this behavior, but I see now
> that -ffast-math
> is no longer required to obtain efficient code for floating point
> compare and branch.

OK, I will document the new options in -ffast-math, as we document these
on -Ox and send a patch shortly.
> The pre-inversion may make noticeable differences in accuracy, and those
> may well
> be of concern.  It shouldn't fail on an IEEE-compliant architecture, in
> the normal
> course of events.  As others pointed out, the re-association of division
> makes
> opportunities for changes in over- and under-flow behavior, from which
> IA32
> architectures will no longer be protected, with the increasing use of
> SSE
> instructions.

We may jump into something like -fapproximate-fp-math allowing transformation
with loss of effciency if that makes sense.
> 
> In a large application, if something breaks when turning on a family of
> unsafe
> optimizations, when there is no documented way of turning them on
> individually,

You can turn them individually and if they are not documented, I will do
so shortly.

Honza

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Prince]

----- Original Message -----
From: "Jan Hubicka" <jh@suse.cz>
To: "Tim Prince" <tprince@computer.org>
Cc: "Jan Hubicka" <jh@suse.cz>; "Toon Moene"
<toon@moene.indiv.nluug.nl>; "Joern Rennecke" <amylaar@redhat.com>;
<gcc@gcc.gnu.org>
Sent: Sunday, July 29, 2001 10:24 AM
Subject: Re: What is acceptable for -ffast-math? (Was: associative law
in combine)


> The mathinline is mostly mess.  I was thinking about rewriting it
myself.
> It seems to make sense to make target specific builtins for the i387
> features the header needs and use them, instead of using mostly
unusable
> i387 inline assembly.
glibc as a whole is so dependent on i386 assembly that it would be a
major project to change.  It might be interesting to include direct
support for more math intrinsics in gcc, but that begins to encroach on
the traditional separation between gcc and support libraries.  I suppose
this would involve built-ins for frexp(), ldexp() et al. in each
supported precision.  I fear that a noticeable amount of efficiency
could be lost, and we would remain with the question of full accuracy
and safety vs speed in certain cases.  A beginning would be to organize
mathinline.h so as to be based on such a group of builtins in order to
test them in the current framework.  This project may not offer
sufficient return without assurance of glibc taking advantage of it and
becoming more portable.
>
> If you have sanified version and you can get the copyring issues,
> please try to contribute the changed to glibc.
I took this far enough with the glibc maintainers to see that their
point of view does not allow introduction of any "safety features."  Now
I will be even further away, wanting to improve support for P4.
> >
> > In a large application, if something breaks when turning on a family
of
> > unsafe
> > optimizations, when there is no documented way of turning them on
> > individually,
>
> You can turn them individually and if they are not documented, I will
do
> so shortly.

Thanks
>
> Honza

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Jan Hubicka]

> glibc as a whole is so dependent on i386 assembly that it would be a
> major project to change.  It might be interesting to include direct
Gcc currently have an method to add machine dependent builtins used
by the SSE code.  I think in same sense we can add machine dependent
builtins for the few remaining i387 instructions we don't generate
directly and then use mathinline.h to do right job converting them
to 'C' way.

I believe this is clean solution.
> support for more math intrinsics in gcc, but that begins to encroach on
> the traditional separation between gcc and support libraries.  I suppose
> this would involve built-ins for frexp(), ldexp() et al. in each
> supported precision.  I fear that a noticeable amount of efficiency
I must say I don't follow the above...
> could be lost, and we would remain with the question of full accuracy
> and safety vs speed in certain cases.  A beginning would be to organize
> mathinline.h so as to be based on such a group of builtins in order to
> test them in the current framework.  This project may not offer
This sounds exactly what I say :) So we seems to be in tune.

Honza
> sufficient return without assurance of glibc taking advantage of it and
> becoming more portable.
> >
> > If you have sanified version and you can get the copyring issues,
> > please try to contribute the changed to glibc.
> I took this far enough with the glibc maintainers to see that their
> point of view does not allow introduction of any "safety features."  Now
> I will be even further away, wanting to improve support for P4.
I

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

In article < 004501c117f4$84a48c80$9865fea9@timayum4srqln4 > you write:
>
>This optimization is specifically permitted for appropriate data types
>by the Fortran standard, with the reservation that parentheses employed
>to prevent re-association must be observed.  As gcc is unable to
>distinguish between the expressions (a/b)/c and a/b/c, this optimization
>would violate the standard in the former case, so is undesirable for
>enabling under -ffast-math.

Note that with FP math, especially with C, there are mainly two kinds of
people:

 - those who know and care about underflows, overflows and precision.
   They _sometimes_ also want exact IEEE semantics, but not necessarily.
 - those who don't, and don't want to, and just want fast code.

Why not make this easier for the user:

 - default to non-IEEE, but "reasonably safe". Allow optimizations that
   underflow or aren't precise to ULP - or optimizations that extend the
   range of calculations (ie keep things in "double" or "long double"
   longer than strictly allowed), but don't allow optimizations that can
   make the range of operations smaller (ie extra traps are bad).

 - make "-ffast-math" mean exactly that: fast.  Even at the expense of
   being even more incorrect, ie allowing optimizations that might
   overflow. Let's face it - they are quite rare, especially in code
   that doesn't care. Make "-ffast-math" be the generic "if you don't
   know, and if you don't care, use this option" for people who just
   want ultimate speed for their quake3 thing.

 - make the people who _do_ know what they are doing and _do_ know what
   the options mean have extra options on just how careful they can be.
   It's silly and counter-productive to assume that most people would
   know what "-ffast-div" is.

So for real math, you'd have

 - "-mieee": This turns us into anal animals, and often implies using
   different math libraries etc too. No "reasonable safe" optimizations
   except the ones explicitly allowed by standards.

 - "-ffloat-precision": allow optimizations that might lose precision.

 - "-ffloat-div-reciprocal": allow optimizations with turning divides
   into reciprocals etc.

  ...

But don't make "-ffast-math" imply "it's still safe". Please. 

So do allow Honza's optimization under fast-math. Don't allow it by
default, and NEVER allow it with "-mieee" unless we start doign the
Fortran thing.

That way, we can also have a simple benchmarking rule: use "-O2
-ffast-math" for benchmarks. No need to tell people about the latest
"-ffloat-random-optimization" of the day, which changes with compiler
releases anyway.

		Linus

Re: associative law in combine

[Joern Rennecke]

> issue in my reply to you.  Because overflow can only be determined at
> compile time with constants, this conversion cannot be right for
> variables.

Actually, as with most rules, there is an exception here: if b and c
are widened, or are constants that fit into a type with less precision,
we can widen before the multiply, or, even better, we can use a wideining
multiply (if available).

e.g.:

 long long a;
 int b, c;

or:

 double a;
 float b, c;

or:

 double a;
 float b;

 a = a / b / 42

However, that might be to rare to be worth the effort.

Re: * Re: What is acceptable for -ffast-math? (Was: associative lawin combine)

[Linus Torvalds]

On Sun, 29 Jul 2001, Stephen L Moshier wrote:
>
> > Why not make this easier for the user:
> > - default to non-IEEE, but "reasonably safe".
>
> The default GCC behavior is not to allow any optimization
> that could change the value of an expression.  That rule is easy
> for users to understand and surely it is a good rule to keep.
>
> Floating point arithmetic, IEEE or otherwise, does not obey the
> associative or distributive laws.

This is YOUR opinion.

But most people who write floating point code do not think that way. They
consider FP to be "mathematically exact" (which _does_ obey associative
and distributive laws), and they write their programs that way.

And that is exactly my point: to the people who do not care about the
exact details of what it means from a mathematical standpoint, using
associative and distributive rules is _fine_. Their algorithms do not
care, simply because the person writing the code doesn't even _know_ about
the issues of limited precision and range of FP code.

See?

What I'm saying is that the compiler cannot fix "thinking errors" of the
programmer. Trying to fix them is self-defeating and ultimately stupid.
I'm claiming that 99%+ of all programmers do not know or care about the
fact that

	(a+b)*c

is not necessarily the same as

	a*c+b*c

when we're talking about floating point.

My _second_ point is that the ones that DO KNOW and DO CARE are also aware
enough that _they_ can be counted on to add a "-mieee" or other flags.

They are also knowledgeable enough that they know what "associative law"
means, which is already a step up from most programmers, I suspect. So
they can understand why the compiler has done an optimization, and can (if
it is documented) be instructed to use the appropriate switch for not
doing that optimization.

So: put the onus of understanding different FP optimizations on the people
who CAN understand them. Don't make the average person who wants to do his
own small project have to worry about it. By the time the person
understands about the range of FP arithmetic enough to care, he will also
have the capability to do something about it.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Stephen L Moshier]

> > Floating point arithmetic, IEEE or otherwise, does not obey the
> > associative or distributive laws.
> 
> This is YOUR opinion.

That is not just an opinion, it is simple arithmetic.

Further, GCC is supposed to be standards compliant by default.  You
seem to be advocating that GCC should, without warning, generate
programs that violate the standard and are mathematically wrong.
The C99 standard says the following, among other things, about floating
point transformations:


       [#14] EXAMPLE 5 Rearrangement for floating-point expressions
       is  often  restricted because of limitations in precision as
       well as range.  The implementation  cannot  generally  apply
       the   mathematical   associative   rules   for  addition  or
       multiplication,  nor  the  distributive  rule,  because   of
       roundoff   error,  even  in  the  absence  of  overflow  and
       underflow.   Likewise,  implementations   cannot   generally
       replace decimal constants in order to rearrange expressions.
       In  the  following  fragment,  rearrangements  suggested  by
       mathematical rules for real numbers are often not valid (see
       F.8).

               double x, y, z;
               /* ... */
               x = (x * y) * z;  // not equivalent to x *= y * z;
               z = (x - y) + y ; // not equivalent to z = x;
               z = x + x * y;    // not equivalent to z = x * (1.0 +
               y);
               y = x / 5.0;      // not equivalent to y = x * 0.2;

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Fergus Henderson]

On 29-Jul-2001, Stephen L Moshier <moshier@mediaone.net> wrote:
> 
> Further, GCC is supposed to be standards compliant by default.

?

I think it would be nice if GCC was standards compliant by default,
but that is certainly not the status quo.

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: < http://www.cs.mu.oz.au/~fjh >  |     -- the last words of T. S. Garp.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Fergus Henderson <fjh@cs.mu.oz.au> writes:

| On 29-Jul-2001, Stephen L Moshier <moshier@mediaone.net> wrote:
| > 
| > Further, GCC is supposed to be standards compliant by default.
| 
| ?
| 
| I think it would be nice if GCC was standards compliant by default,
| but that is certainly not the status quo.

Agreed.  But that should be the ideal we should strive for.
And also, I don't think that is an argument for not computing
correctly, even if we're asked to compute fast. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<Further, GCC is supposed to be standards compliant by default.  You
seem to be advocating that GCC should, without warning, generate
programs that violate the standard and are mathematically wrong.
>>

and that violate the expectations of programmers who *do* know what they
are doing!

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Sun, 29 Jul 2001 dewar@gnat.com wrote:
>
> <<Further, GCC is supposed to be standards compliant by default.  You
> seem to be advocating that GCC should, without warning, generate
> programs that violate the standard and are mathematically wrong.
> >>
>
> and that violate the expectations of programmers who *do* know what they
> are doing!

Note that most of the people I know who _do_ know what they are doing also
want to have the option o ftelling the compiler when they don't care, and
want to allow the compiler to do "illegal" things if that allows the
compiler to generate better code.

It's sad that C99 doesn't seem to allow the Fortran kind of "you go and
make the best you can of this expression unless I tell you otherwise".
Which means that we'd _have_ to be non-standard to allow the optimization
at all. Which obviously means a flag (one way or the other).

Rather than have a good way to do it on a case-by-case basis, which is
probably what the people who _really_ care would prefer.

In C, you can't reasonably use the Fortran "parenthesis are significant"
approach - the way the C pre-processor works you often have lots of extra
parenthesis for that reason. However, C does have the well-established
sequence point rule, so it would make sense to have that kind of
extension.

And hey, if you want to make it be an extra flag, just make it a _simple_
one. Like "-ffast-math". Don't make people have to know 100 esoteric flags
that are compiler-release-specific. That's completely useless.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 30, 2001, Linus Torvalds <torvalds@transmeta.com> wrote:

> In C, you can't reasonably use the Fortran "parenthesis are significant"
> approach - the way the C pre-processor works you often have lots of extra
> parenthesis for that reason.

But we could use new tokens, say (( )) and ((( ))) to mark expressions
the preprocessor and the compiler shouldn't muck up with.  It would
probably be tricky to get this right in a grammar, but the integrated
preprocessor probably makes it easier.  The separate preprocessor
would have to insert blanks between consecutive ( tokens generated by
preprocessing to ensure they're not handled as a single token (or at
least marked as optimization barriers, since it'll probably be more
reasonable to keep them as separate tokens.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Neil Booth]

Alexandre Oliva wrote:-

> But we could use new tokens, say (( )) and ((( ))) to mark expressions
> the preprocessor and the compiler shouldn't muck up with.

You can't be serious?  Any non-trivial CPP expansion tends to have
multiple redundant parentheses all over the place.

Neil.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 30, 2001, Neil Booth <neil@daikokuya.demon.co.uk> wrote:

> Alexandre Oliva wrote:-
>> But we could use new tokens, say (( )) and ((( ))) to mark expressions
>> the preprocessor and the compiler shouldn't muck up with.

> You can't be serious?  Any non-trivial CPP expansion tends to have
> multiple redundant parentheses all over the place.

I am serious.  The preprocessor should protect individual parentheses
from becoming consecutive by introducing whitespace.  We should tell:

((foo))

from

#define a (foo)
#define b (a)
b

so that the latter expands to say:

( (foo) )

This is something the preprocessor already does in certain cases, to
prevent certain sequences of characters from becoming different
tokens.  I'm just suggesting that we do this for parentheses too, so
that we can recognize sequences of parentheses explicitly introduced
by the user, as optimization barriers.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Neil Booth]

Alexandre Oliva wrote:-

> I am serious.  The preprocessor should protect individual parentheses
> from becoming consecutive by introducing whitespace.

And what about existing user code?

Neil.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 30, 2001, Neil Booth <neil@daikokuya.demon.co.uk> wrote:

> Alexandre Oliva wrote:-
>> I am serious.  The preprocessor should protect individual parentheses
>> from becoming consecutive by introducing whitespace.

> And what about existing user code?

Well, depending on whether we enable this feature by default or not,
we might get different code generated or not.  If we don't apply
unsafe transformations by default, existing user code certainly won't
be affected.  If we do, we may lose optimization opportunities that
were not intended to be avoided by users unaware of this feature.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Joern Rennecke]

> But we could use new tokens, say (( )) and ((( ))) to mark expressions
> the preprocessor and the compiler shouldn't muck up with.  It would

It's hard to distinguish an opening (( from two opening (s that
are closes separately, e.g.:

a*((b+c)*d)

OTOH, you could re-use (abuse) statement expressions for this purpose:

({a+b;})*c

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 30, 2001, Joern Rennecke <amylaar@redhat.com> wrote:

>> But we could use new tokens, say (( )) and ((( ))) to mark expressions
>> the preprocessor and the compiler shouldn't muck up with.  It would

> It's hard to distinguish an opening (( from two opening (s that
> are closes separately, e.g.:

> a*((b+c)*d)

Indeed.  That's why I suggested to only mark such duplicate
parentheses as optimization barriers, at the end of my message.  We'd
still see two different tokens, but if the opening and closing
parentheses of an expression of the form '(' expression ')' are both
marked as optimization barriers, we generate the expression tree
inside an optimization barrier.  The lexer/preprocessor would have to
mark all '('s but the first in a sequence, and all ')'s but the last
in a sequence, as candidate markers of optimization barriers.

> OTOH, you could re-use (abuse) statement expressions for this purpose:

> ({a+b;})*c

Yeah, but this won't compile with compilers other than GCC.  OTOH, it
might already do what we want in older releases of GCC.  Does it?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Alexandre Oliva <aoliva@redhat.com> writes:

| On Jul 30, 2001, Joern Rennecke <amylaar@redhat.com> wrote:
| 
| >> But we could use new tokens, say (( )) and ((( ))) to mark expressions
| >> the preprocessor and the compiler shouldn't muck up with.  It would
| 
| > It's hard to distinguish an opening (( from two opening (s that
| > are closes separately, e.g.:
| 
| > a*((b+c)*d)
| 
| Indeed.  That's why I suggested to only mark such duplicate
| parentheses as optimization barriers, at the end of my message.

I'm puzzled to see "optimization" used to designate a transformation
applying dubious rules to transmute the meaning of a computation :-(

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 30, 2001, Gabriel Dos Reis <gdr@codesourcery.com> wrote:

> I'm puzzled to see "optimization" used to designate a transformation
> applying dubious rules to transmute the meaning of a computation :-(

In what sense is this different from Fortran's rules on parentheses?
I'm just trying to bring in the same rule to C, given the additional
complexities brought in by the preprocessor mentioned by Linus.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Alexandre Oliva <aoliva@redhat.com> writes:

| On Jul 30, 2001, Gabriel Dos Reis <gdr@codesourcery.com> wrote:
| 
| > I'm puzzled to see "optimization" used to designate a transformation
| > applying dubious rules to transmute the meaning of a computation :-(
| 
| In what sense is this different from Fortran's rules on parentheses?

The C/C++ Standard, for example, say that a/b/c is evaluated as
(a/b)/c. Someone proposed that it should be evaluated as a/(b*c)
-- which of course is incorrect -- when optimizing.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Joern Rennecke <amylaar@redhat.com> writes:

| OTOH, you could re-use (abuse) statement expressions for this purpose:
| 
| ({a+b;})*c

Please, let's get serious :-)

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<In 99% of all cases, the information of the subtleties of floating point
arithmetic simply was never there in the program gcc tries to compile.
Why? Becuase the programmer didn't even _intend_ any of the subtleties, or
even knew about them.

Gcc trying to enforce them doesn't add any value.
>>

The 99% is a highly excessive estimate here. Many many people understand
and expect that the compiler will NOT mess around with their floating-point
expressions. It has always been the case in Fortran that you can depend
on parentheses being respected, and programmers do indeed count on this.

Especially on an IEEE machine, fpt is not just some vague approximation
of real numbers, it is a well defined arithmetic system with well defined
rules, and in the world of numerical programming, people are not as
incompetent as you claim. Sure, a casual C programmer using an occasional
bit of fpt arithmetic may well not know what they are doing, but a compiler
should aim its implementation of fpt at those writing numeric codes. Giving
wrong answers fast is definitely not useful.

Part of the trouble here is that many compiler writers and systems people
simply don't know floating-point semantics well enough, and have often made
improper assumptions about optimizations etc, which have caused a lot of trouble
in the past. 

I cannot believe that any expert in floating-point would agree that it is just
fine to do the kind of optimizations you suggest by default, and indeed I have
trouble seeing that this kind of "optimization" is ever anything but a bug!

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<It's sad that C99 doesn't seem to allow the Fortran kind of "you go and
make the best you can of this expression unless I tell you otherwise".
>>

But if you look at real Fortran codes, they are full of parentheses to stop
the compiler from messing around, and in fact most Fortran compilers do only
very limited mucking even when no parens are present. I would be shocked
to find a FOrtraqn compiler that changed

a*c + b*c

into

(a+b)*c

a totally diffrerent expression!

I would have written the second expression if I had wanted that effect.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Toon Moene]

dewar@gnat.com wrote:

> <<It's sad that C99 doesn't seem to allow the Fortran kind of "you go and
> make the best you can of this expression unless I tell you otherwise".
> >>

> But if you look at real Fortran codes, they are full of parentheses to stop
> the compiler from messing around, and in fact most Fortran compilers do only
> very limited mucking even when no parens are present. I would be shocked
> to find a FOrtraqn compiler that changed
> 
> a*c + b*c
> 
> into
> 
> (a+b)*c
> 
> a totally diffrerent expression!
> 
> I would have written the second expression if I had wanted that effect.

No, you should have written (a*c) + (b*c) if you wanted the first
expression *instead* of the second one - heck, you say so yourself, in
the above (... full of parentheses ...)

[ Note 7.24 in the Fortran 95 Standard ].

Of course this only works one way - the compiler is not free to change
(a+b)*c into (a*c)+(b*c), because of the parentheses in the first
expression.

[ And in another mail: ]

> It certainly would be interesting to hear from someone with extensive
> experience in writing numerical codes supporting Linus' position (I have
> that extensive experience, but for sure I do not support the position 
> that a compiler should by default do completely different things from 
> what it is told.

Well, I won't be so rash as to say that it should do that *by default*,
but there is a lot of fp code out there that's robust against small
differences in evaluation of floating point expressions.  The reason for
this is very simple: The majority of physics processes that are modelled
on a computer can only be computed approximately *even if you used the
real "reals" for it*.  Only very few fluid mechanics problems are
usefully "idealized" enough to allow analytic solutions - all others
have to be discretized one way or another (finite differences, finite
elements, spectral decomposition, vortex decomposition, etc.).

Once you allow that approximation, the fact that fp computations are not
"exact" is not very interesting (you still have to watch out for fatal
cancellation, but that's why you should use parentheses where
necessary).

That said, I do agree that I rather use something like -ffast-math to
indicate that I know what I'm doing, and I would be rather disappointed
if -ffast-math wouldn't honor parentheses [in the sense that I wouldn't
use that option anymore] :-)

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
Join GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

It certainly would be interesting to hear from someone with extensive
experience in writing numerical codes supporting Linus' position (I have
that extensive experience, but for sure I do not support the position that
a compiler should by default do completely different things from what it
is told.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Hollebeek]

On Mon, Jul 30, 2001 at 09:01:17AM -0400, dewar@gnat.com wrote:
> It certainly would be interesting to hear from someone with extensive
> experience in writing numerical codes supporting Linus' position (I have
> that extensive experience, but for sure I do not support the position that
> a compiler should by default do completely different things from what it
> is told.

His position is even worse: the compiler should abandon well specified
semantics in favor of less predictable ones (where what result you get
depends on which optimizations were applied).

When people suggest similar things regarding inlining and assembly
fragments, which certainly are understood by far less than 1% of the
users, Linus flames the list for breaking his code which depends on
very subtle aspects of the behavior of those features.

So hopefully he'll reverse his position, and take a position
consistent with his previous positions: optimization should not
transform working code into code that produces different results,
no matter how fast it does so.

-Tim

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

Another issue here is that the optimizations in question are unlikely to
be that significant in practice. In real programs where performance is an
issue, there will be relatively few opportunities for this kind of
optimization, and assuming that the program was reasonably competently
written, those few cases are likely to be just those that should not
be molested.

Consdier again the assoicative case. The expression

a*b + a*c

Now there are three cases

1. The really competent programmer, who knows that the associat9ive 
transformation will blow up the error analysis, and wrote it that way
for a good reason.

2. The somewhat competent programmer, who knows nothing about fpt details,
but who wants their program to run fast. Such a programmer is likely to
rewrite this manually as

  a +(b*c)

anyway, because that's obviously "faster".

3. The incompetent programmer who knows nothing about fpt details, wants
their program to run fast, would be perfectly happy to have that
transformation done, but does not do it themselves.

These optimizations are aimed only at class 3 programmers. I think that

a) there are fewer such programmers than people think, certainly nothing
like Linus' 99%

b) even for them, it will be rare for these kind of improper optimizations
to make a difference (that's why Fortran compilers don't really mess around
that much, even though the standard allows them to).

Floating-point is always tricky. If you see

a*a*a*a

in a program, it may seem very tempting to "optimize" this
as 

T := a*a;
T*T

but even though the latter has only two operations instead of three, it
has significantly worse error bounds.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Kevin Handy]

dewar@gnat.com wrote:
> 
> Another issue here is that the optimizations in question are unlikely to
> be that significant in practice. In real programs where performance is an
> issue, there will be relatively few opportunities for this kind of
> optimization, and assuming that the program was reasonably competently
> written, those few cases are likely to be just those that should not
> be molested.
> 
> Consdier again the assoicative case. The expression
> 
> a*b + a*c
> 

...

> 2. The somewhat competent programmer, who knows nothing about fpt details,
> but who wants their program to run fast. Such a programmer is likely to
> rewrite this manually as
> 
>   a +(b*c)

I would hope that a competent programmer would not transform the
expression into this. I would hope he would transform it into

    a*(b+c)

instead, unless my algebra has become hopelessly munged since
high school.

> anyway, because that's obviously "faster".

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[mike stump]

> Date: Sun, 29 Jul 2001 17:03:16 -0400 (EDT)
> From: Stephen L Moshier <moshier@mediaone.net>

> Further, GCC is supposed to be standards compliant by default.

Really?

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:

> Fergus Henderson <fjh@cs.mu.oz.au> writes:
> |
> | I think it would be nice if GCC was standards compliant by default,
> | but that is certainly not the status quo.
>
> Agreed.  But that should be the ideal we should strive for.
> And also, I don't think that is an argument for not computing
> correctly, even if we're asked to compute fast.

Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
expression.

So it's certainly very very debatable whether you can say "not computing
correctly" for it - the user has not actually shown a strong preference
for the double division, and it is entirely possible that the double
division has come about as a result of macro expansion and/or inline
function expansion.

Who are you to say that the user wouldn't like the code to be run faster?
I don't think this was well-defined in C before C99, for example.

Note that with the current -ffast-math on x86, for example, gcc will
already inline things like "sin()" etc using the x87 instruction directly.
Despite the fact that that instruction is well-defined only for the range
-2**63 to +2**63.

And now there are people that say that something simple as "a/b/c" should
not be optimized to "a/(b*c)" even with -ffast-math.

What planet are you people from?

We are already in the situation that -ffast-math gives a value for "sin()"
that is not even _close_ to the range of [-1,1] under various
circumstances. And you're complaining about the fact that b*c can
overflow? We've already effectively limited our argument range to +-2**63,
and there's no way a*b can overflow, guys.

So even if you don't want to make it an optimization by default with the
C99 argument (hey, I don't really disagree), there is absolutely _no_
valid argument for not doing it with -ffast-math.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| On 30 Jul 2001, Gabriel Dos Reis wrote:
| 
| > Fergus Henderson <fjh@cs.mu.oz.au> writes:
| > |
| > | I think it would be nice if GCC was standards compliant by default,
| > | but that is certainly not the status quo.
| >
| > Agreed.  But that should be the ideal we should strive for.
| > And also, I don't think that is an argument for not computing
| > correctly, even if we're asked to compute fast.
| 
| Note that "a/b/c" _is_ the same as "a/(b*c)" when seen as a mathematical
| expression.

Abstract Algebra is *different* from Computing.  That is lesson 0
from real-world numerical computations. 

| Who are you to say that the user wouldn't like the code to be run faster?

Who are you to say that it is more important to compute wrong and fast
than computing correctly?  

If correctness is not an issue then the fastest way is to avoid the
computation entirely or return 0.0 unconditionally.

| Note that with the current -ffast-math on x86, for example, gcc will
| already inline things like "sin()" etc using the x87 instruction directly.

I don't buy the argument 
 
  That is broken, then we should break this as well

If GCC behaves incorrectly in some cases, the appropriate action to
take is to make it behave correctly, -not- to break other parts.

| What planet are you people from?

A planet where we do numerical computation for living.

As von Neumann said once

   Numerical computation is too serious to be left to the computer
   scientist. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Olivier Galibert]

On Mon, Jul 30, 2001 at 08:53:37PM +0200, Gabriel Dos Reis wrote:
> A planet where we do numerical computation for living.

Then you're supposed to know better than to use -ffast-math.

  OG.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Olivier Galibert <galibert@pobox.com> writes:

| On Mon, Jul 30, 2001 at 08:53:37PM +0200, Gabriel Dos Reis wrote:
| > A planet where we do numerical computation for living.
| 
| Then you're supposed to know better than to use -ffast-math.

Certainly.  

Is that supposed to be  an argument for making GCC using ill-formed
computation rules?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

On 30 Jul 2001, Gabriel Dos Reis wrote:
>
> | Who are you to say that the user wouldn't like the code to be run faster?
>
> Who are you to say that it is more important to compute wrong and fast
> than computing correctly?

Why do you think it's necessarily wrong?

The _only_ person who can judge whether it's wrong to do so it the person
who wrote the program. He may be happy with the faster code. He's almost
certainly happy if he used "-ffast-math".

> If correctness is not an issue then the fastest way is to avoid the
> computation entirely or return 0.0 unconditionally.

Blather blather. According to your notion of correctness.

> | Note that with the current -ffast-math on x86, for example, gcc will
> | already inline things like "sin()" etc using the x87 instruction directly.
>
> I don't buy the argument
>
>   That is broken, then we should break this as well

It's not broken, you silly boy.

THAT IS THE WHOLE REASON FOR THE "-ffast-math" option.

Go back and look at why it was added. Right - exactly so that people could
get _fast_ math, when they do not care about the inexactness that isn't
even an issue for most people.

> As von Neumann said once
>
>    Numerical computation is too serious to be left to the computer
>    scientist.

Yeah. And as Linus once said: most numerical problems today in pure CPU
cycles are actually 3D games.

Your arguments about "numerical computation" are just silly, as you don't
seem to realize that there are tons of problems where your theoretical
issues are nothing more than noise.

It's not "incorrect" to say that you want the result faster, even if that
result doesn't match your theoretical models.

And remember: in the end, the goodness of _any_ program (including a
compiler) is how well it solves real-world problems. Whether you like the
transformation or not as anumerical scientist has absolutely nothing to do
with that.

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| > As von Neumann said once
| >
| >    Numerical computation is too serious to be left to the computer
| >    scientist.
| 
| Yeah. And as Linus once said: most numerical problems today in pure CPU
| cycles are actually 3D games.

Then Linus need to be educated about numerical problems.

| Your arguments about "numerical computation" are just silly, as you don't
| seem to realize that there are tons of problems where your theoretical
| issues are nothing more than noise.

Not just because you claim they are silly makes them so.
More to the point, you don't seem to realize that there are tons of
problems where your theoretical issues of speed are nothing more than
noise. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Olivier Galibert]

On Tue, Jul 31, 2001 at 11:04:14AM +0200, Gabriel Dos Reis wrote:
> Not just because you claim they are silly makes them so.
> More to the point, you don't seem to realize that there are tons of
> problems where your theoretical issues of speed are nothing more than
> noise. 

But for these problems, you're supposed to use -mieee, not
-ffast-math.  Or nothing at all.  But we're talking about -ffast-math
here, which is already too incorrect for a simulations physicist, but
perfect for somebody doing 3D rendering, speech analysis, picture
analysis, etc...

In particular, if a/b/c won't overflow but a/(b*c) may you're already
into so tightly controlled conditions that using -ffast-math would be
somewhere between insanity and incompetence.

We're having 3 levels of "correctness" here:
-ffast-math: loose, works perfectly if you stay within reasonable
             boundaries and don't try to get the last bit of precision

nothing:     stricter, does what you write and don't try to be clever

-mieee:      military-level following of the rules, including anything
             and everything that happens for rouding, overflowing, etc...

There is a whole world of users there outside of numerical
computations as the physicist understand them.  And for them speed is
important, and they know how to keep their values within the bounds
that the compiler and the processor handle efficiently.  Don't go and
cripple _them_.

  OG.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Olivier Galibert <galibert@pobox.com> writes:

| There is a whole world of users there outside of numerical
| computations as the physicist understand them.  And for them speed is
| important, and they know how to keep their values within the bounds
| that the compiler and the processor handle efficiently.  Don't go and
| cripple _them_.

I don't want to "cripple" them.  And, I don't think "cripple" is a fair
characterization. 

You seem to imply that for someone who does Image Analysis,
correctness is less important than speed.  I disagree, if that were
indeed your intent.  If you expect the compiler to evaluate a/b/c as
a/(b*c) then you would have written a/(b*c).  Wouldn't you?

And speed does matter for numerical programmers also.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Linus Torvalds]

On 31 Jul 2001, Gabriel Dos Reis wrote:
> |
> | Yeah. And as Linus once said: most numerical problems today in pure CPU
> | cycles are actually 3D games.
>
> Then Linus need to be educated about numerical problems.

You want to bet?

Sure, go ahead and count up your supercomputers, and I'll try to estimate
how many PS2's etc there are in the world.

That's by CPU cycles - I think you'll lose. Big.

Do the same by economic factors, and I suspect you're on slippery ground
too.

Do the same by number of users, and I _know_ you're done for.

I know, it's nice and comfy up in your ivory tower. Go on, ignore the rest
of the world..

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<What if I told you that it's actually legal? Go check it out.
>>

Of course it's legal - I *do* know the Fortran standard :-)

Indeed, read what I said more carefully, what I was saying was that *even
though* the Fortran standard allows this transformation, I would be shocked
at a compiler that actually did this particular one, since it is seriously
misguided to do so. Permission in the standard to do unwise things does
not mean that you have to do them.

<<Fortran, while being a pretty horrible language for anything else, does
exactly this for FP math. It doesn't do it for much anything else, of
course.
>>

Fortran is by no means such a horrible language for numerical calculations
(I say this having written many hundreds of thousands of lines of Fortran
numerical code). Yes, I know perfectly well that the standard allows this,
and that is why competent Fortran programmers use parentheses extensively,
but at least in my experience, Fortran compilers are fairly judicious about
taking advantage of the permissions in the absence of parens. Most certainly
one would expect that a Fortran compiler would NOT bother to preserve the
"proper" negative zero semantics in the case where parens are not used,
and so would for example feel free to transform 

a-b

into

-(b-a)

which is of course not a valid transformation from a strict IEEE point of view.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<I would hope that a competent programmer would not transform the
expression into this. I would hope he would transform it into

    a*(b+c)

instead, unless my algebra has become hopelessly munged since
high school.
>>

:-)

oops, yes indeed

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

>>Well, I won't be so rash as to say that it should do that *by default*,

But that of course is the issue under discussoin :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<No, you should have written (a*c) + (b*c) if you wanted the first
expression *instead* of the second one - heck, you say so yourself, in
the above (... full of parentheses ...)
>>

I really don't understand the "No" here. Please reread my message. Yes
of course, to fully follow the standard you put in the parens, as I noted
earlier, Fortran code tends to be full of parens for this reason.

But ... my point was that *EVEN IF* no parens were present, I would be shocked
to find the Fortran compiler using the associative law to rewrite my 
expression.

After all, if you really believe that a Fortran compiler is not only allowed
(true) but reasonably may (false) perform any kind of such "optimization",
there are some pretty weird results that can occur in the extreme.

Actually most Fortran programmers, even those not doing careful error
analysis, tend to be quite disturbed if optimization changes results,
even if this is allowed by the standard.

If you write

  A = B * C

  IF (A.EQ.B*C) ...

it is a surprise for the IF to yield False, even though the standard permits
it. Kahane in fact declares this to be a bug, and cannot be mollified by
being told that the standard permits this "bug".

I know that IBM in their Fortran compiler for the Power architecture (and
remember IBM is targetting knowledgable numeric programmers there) decided
that in optimization mode they should still round to single precision when
calculations are done in registers, and avoid the phenomenon of getting
unexpected extra accuracy for register calculations [the reason incidentally
behind the misbehavior of the IF above].

The x86 of course is quite a nightmare from this point of view (at least
the traditional 80-bit stack based arithmetic is a nightmare, the newer
fpt on the later Pentiums and Athlons is better).

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Toon Moene]

dewar@gnat.com wrote:

> If you write
> 
>   A = B * C
> 
>   IF (A.EQ.B*C) ...
> 
> it is a surprise for the IF to yield False, even though the standard permits
> it. Kahane in fact declares this to be a bug, and cannot be mollified by
> being told that the standard permits this "bug".

Yes, I know this - it was discussed thoroughly in 1999.

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
Join GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<I am serious.  The preprocessor should protect individual parentheses
from becoming consecutive by introducing whitespace.  We should tell:

>>

Sometimes I must say that people seem to work hard to dream up the most
infeasible and ugly syntactical ideas that one could possibly imagine :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 30, 2001, dewar@gnat.com wrote:

> <<I am serious.  The preprocessor should protect individual parentheses
> from becoming consecutive by introducing whitespace.  We should tell:
>>> 

> Sometimes I must say that people seem to work hard to dream up the most
> infeasible and ugly syntactical ideas that one could possibly imagine :-)

:-)

Would you prefer

__extension__ __attribute__ ((optimize=0)) (a*b) +
__extension__ __attribute__ ((optimize=0)) (a*c)

? :-)

I'd much rather write ((a*b)) + ((a*c)), which will even compile with
other compilers, and might be picked up in a future revision of the C
and C++ Standards.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Kevin Handy]

Alexandre Oliva wrote:
> 
> On Jul 30, 2001, dewar@gnat.com wrote:
> 
> > <<I am serious.  The preprocessor should protect individual parentheses
> > from becoming consecutive by introducing whitespace.  We should tell:
> >>>
> 
> > Sometimes I must say that people seem to work hard to dream up the most
> > infeasible and ugly syntactical ideas that one could possibly imagine :-)
> 
> :-)
> 
> Would you prefer
> 
> __extension__ __attribute__ ((optimize=0)) (a*b) +
> __extension__ __attribute__ ((optimize=0)) (a*c)
> 
> ? :-)
> 
> I'd much rather write ((a*b)) + ((a*c)), which will even compile with
> other compilers, and might be picked up in a future revision of the C
> and C++ Standards.

I'd think that trying to write a parser for this would be much
harder than necessary. You'd have to be able to handle

   ((a*b) + (a*c))

as well as

   ((a*b) + c)

and what about three parentheses

   (((a * b) +  c) + (a*d))

and you'd have to keep track of which parentheses had another
parentheses attached to them (both before and after), or track 
white space as part of the parsing.

If you want to use something like

   __extension__ __attribute__ ((optimize=0)) (a*b) + (a*c)

you could always wrap it in a macro

   #define noopt(x) __extension__ __attribute__ ((optimize=0)) (x)

   noopt((a*b) + (a*c))

which would give the benefit to non-gcc users that you could document
what it means, and if other compilers had other ways of specifying
the same thing, you could handle it in macros.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<Well, then I'd like to understand the actual policy better.  You have
GNU extensions but otherwise seem to include what's in the language
definition.  The apparent conflicts arise when you say the
GNU extensions have to be turned off in order for the compiler
to be strictly compliant with the standard.
>>

Does the standard forbid upwards compatible extensions?

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Joe Buck]

>>...  The apparent conflicts arise when you say the
>> GNU extensions have to be turned off in order for the compiler
>> to be strictly compliant with the standard.

> Does the standard forbid upwards compatible extensions?

The standard specifies that some names, including names beginning with
two underscores, are reserved to the implementation.  Extensions that
use reserved names don't conflict with the standard.  But extensions that
introduce new keywords, where the keywords aren't in the implementation
namespace, do conflict with the standard because a programmer can legally
use that word as an identifier.

Example: "typeof".

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<I'd much rather write ((a*b)) + ((a*c)), which will even compile with
other compilers, and might be picked up in a future revision of the C
and C++ Standards.
>>

I very much doubt that the standards committee would even consider, let
alone adopt such a lexically dubious extension :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<I'd rather you use #pragma STDC FP_CONTRACT OFF - while this seems to be
intended to disable optimisations that give more accurate results, it
should probably disable optimisations here that may give less accurate
results as well.
>>

Well to be fair, Alexandre is trying to provide a mechanism for very
specifically controlling on an operation by operation basis whether
"optimizations" of the type we are discussing are permitted.

My own view is that, if you absolutely must persue this, then just use
parentheses as in Fortran. So what if some PP output is unnecessarily
signalled as not being optimizable, this is a second order effect (my
own guess is that even the first order effect of such optimizations is
small in any case).

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<The standard specifies that some names, including names beginning with
two underscores, are reserved to the implementation.  Extensions that
use reserved names don't conflict with the standard.  But extensions that
introduce new keywords, where the keywords aren't in the implementation
namespace, do conflict with the standard because a programmer can legally
use that word as an identifier.
>>

OK, yes of course new reserved words are not allowed (COBOL is the only
standard language in which new keywords are allowed, well at least this
was in 74 COBOL, not sure if this was retained in later standards).

But it is interesting that many of the GNU extensions do not conflict with
the standard (e.g. nested procedures, dynamic arrays), so these are not
examples of GNU-C not meeting the criterion of being standard by default.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

Note incidentally that by contrast, no extensions of any kind are permitted
in Ada, and the Ada test suite contains a number of tests that try to guess
at possible extensions and make sure they are not accepted :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

> ({a+b;})*c

OK, I admit, the (( .. )) suggestion was not the most awful one after all :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

dewar@gnat.com writes:

| > ({a+b;})*c
| 
| OK, I admit, the (( .. )) suggestion was not the most awful one after all :-)

:-)

The idea that GCC should require users to jump through hoops, using
non-standard constructs in order to have their programs interpreted
according to standard rules is just frightening.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Jul 31, 2001, Gabriel Dos Reis <gdr@codesourcery.com> wrote:

> dewar@gnat.com writes:
> | > ({a+b;})*c
> | 
> | OK, I admit, the (( .. )) suggestion was not the most awful one after all :-)

> :-)

> The idea that GCC should require users to jump through hoops, using
> non-standard constructs in order to have their programs interpreted
> according to standard rules is just frightening.

Aren't we talking about -ffast-math?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Alexandre Oliva <aoliva@redhat.com> writes:

| On Jul 31, 2001, Gabriel Dos Reis <gdr@codesourcery.com> wrote:
| 
| > dewar@gnat.com writes:
| > | > ({a+b;})*c
| > | 
| > | OK, I admit, the (( .. )) suggestion was not the most awful one after all :-)
| 
| > :-)
| 
| > The idea that GCC should require users to jump through hoops, using
| > non-standard constructs in order to have their programs interpreted
| > according to standard rules is just frightening.
| 
| Aren't we talking about -ffast-math?

Certainly.  The point is that an optimization should transmute the
meaning of program.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<The C/C++ Standard, for example, say that a/b/c is evaluated as
(a/b)/c. Someone proposed that it should be evaluated as a/(b*c)
-- which of course is incorrect -- when optimizing.
>>

Well now, let's be a little careful here ....

When the C standard says a/b/c is evaluated as (a/b)/c, it means that the
result is *as if* it were computed that way, and a compiler is of course
free to compute a/(b*c) if the result is still correct.

Now for the integer case we know exactly what "still correct" means, it
means that if we can prove that a/(b*c) does not overflow, and always
gives the same result as (a/b)/c, then of course the compiler can compute
it the first way.

What we are discussing is what "still correct" means for floating-point.
Most certainly the C and C++ standards do not give an exact prescription
for floating-point values. You actually have to say quite a bit to even
get near to the point where you can really answer the question accurately.

Let's look at the exact rules in Ada for example. Of course Ada requires
that a/b/c be computed as if it were (a/b)/c. But Ada also defines a
concept of model numbers and model intervals, and any evaluation scheme
that provides a guarantee that the result is in the appropriate model
interval is permissible. Let's give an example.

If you see  a / 2.0 / 2.0

it is absolutely definition OK, to compute this particular expression
as a / 4.0 in any language no matter WHAT the standard says, since the
result is exactly the same. But for other divisors this is not necessarily
true.

So the "which of course is incorrect" is a bit too glib here.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

dewar@gnat.com writes:

| <<The C/C++ Standard, for example, say that a/b/c is evaluated as
| (a/b)/c. Someone proposed that it should be evaluated as a/(b*c)
| -- which of course is incorrect -- when optimizing.
| >>
| 
| Well now, let's be a little careful here ....
| 
| When the C standard says a/b/c is evaluated as (a/b)/c, it means that the
| result is *as if* it were computed that way, and a compiler is of course
| free to compute a/(b*c) if the result is still correct.

Completely agreed.

[...]

| So the "which of course is incorrect" is a bit too glib here.

What I intended to mean is that the compiler has, first, to prove that
such transformmation yields the same result.  The only argument I've
seen so far is that, GCC should make the program run fast...

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Joern Rennecke]

> If you see  a / 2.0 / 2.0
> 
> it is absolutely definition OK, to compute this particular expression
> as a / 4.0 in any language no matter WHAT the standard says, since the
> result is exactly the same. But for other divisors this is not necessarily

No, if you have IEEE denormals, double rounding can give a different result.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<What I intended to mean is that the compiler has, first, to prove that
such transformmation yields the same result.  The only argument I've
seen so far is that, GCC should make the program run fast...
>>

Not quite, "the same result" is too strong. Instead, the rule is that
the transformation must yield a result that is still conforming to the
requirements of the standard. Since the standard is (deliberately) not
100% precise in saying what this result should be, there is some legitimate
wiggle room.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

dewar@gnat.com writes:

| <<What I intended to mean is that the compiler has, first, to prove that
| such transformmation yields the same result.  The only argument I've
| seen so far is that, GCC should make the program run fast...
| >>
| 
| Not quite, "the same result" is too strong. Instead, the rule is that
| the transformation must yield a result that is still conforming to the
| requirements of the standard. Since the standard is (deliberately) not
| 100% precise in saying what this result should be, there is some legitimate
| wiggle room.

Transforming a/b/c into a/(b*c) where (b*c) overflows whereas (a/b)/c
does not isn't something acceptable (speaking here as a user or a
numerical programmer).  Even if C++98 is as vague as C90 was about
floating point arithmetic, the move is that the next revision of the
Standard will provide a sound framework and support for numerical
computations; I don't think speed is enough to justify incorrectness.
I don't think that the impreciseness in C++98 makes it OK to apply
dubious rules.

That doesn't mean I don't want optimization.  I do love optimization.
By optimization, I really mean optimization.

That said, I don't mind about -fdubious-math as far as it is not the
default, nor turned on at any level of optimization.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Joern Rennecke]

> That said, I don't mind about -fdubious-math as far as it is not the
> default, nor turned on at any level of optimization.

For a user, it's hard to guess or remember user that -fdubious-math
and -ffast-math are related.  How about -ffast-and-loose-math ?

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Joern Rennecke <amylaar@redhat.com> writes:

| > That said, I don't mind about -fdubious-math as far as it is not the
| > default, nor turned on at any level of optimization.
| 
| For a user, it's hard to guess or remember user that -fdubious-math
| and -ffast-math are related.  How about -ffast-and-loose-math ?

I was actually proposing to rename -ffast-math to -fdubious-math :-)
but your proposal has also my support :-))

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Nick Ing-Simmons]

Gabriel Dos Reis <gdr@codesourcery.com> writes:
>Joern Rennecke <amylaar@redhat.com> writes:
>
>| > That said, I don't mind about -fdubious-math as far as it is not the
>| > default, nor turned on at any level of optimization.
>| 
>| For a user, it's hard to guess or remember user that -fdubious-math
>| and -ffast-math are related.  How about -ffast-and-loose-math ?
>
>I was actually proposing to rename -ffast-math to -fdubious-math :-)
>but your proposal has also my support :-))

quick-and-dirty-math ?
 
>
>-- Gaby
-- 
Nick Ing-Simmons
http://www.ni-s.u-net.com/

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Theodore Papadopoulo]

While I agree that by default the compiler must follow "good rules", I 
think that always disallowing optimisations which may lose some bits of 
accuracy (when specified with a option switch of course) just because 
it may hurt someone, somewhere, sometime is a little too strong.
Full safety is IMHO impossible with floating point... Only the 
programmer knows the accuracy of its input data (Linus called that 
noise) and the accuracy he expects on its result.

Since the example has been given, let us speak about image processing.
The data (when using real images) are often 'char' based so you have 
basically 8 bits or less of information per channel.  Then converting
these images to float and working with them, there is usually little risk of 
introducing some error due to some bad rounding. float is enough for 
most image analysis and computation, and lots of people are even 
using double (which sometimes are also faster). In such cases, I 
would argue that absolute floating point accuracy is not worth. To be 
meaningful the results have to be robust to errors (in the input and 
thus in the subsequent calculations that are many orders of magnitude 
higher than the machine accuracy). If they are not, then the approach 
is certainly not valid.

gdr@codesourcery.com said:
> That doesn't mean I don't want optimization.  I do love optimization.
> By optimization, I really mean optimization. 

No by optimisation, you mean mathematically provable safe optimisation and 
unfortunately, the provability of such things by mathematics is usually
very difficult. I agree that sometimes you have to rely only on such 
optimisations, so that, by default, the compiler should only allow them.
But, you have to allow for people:

- for whom provably has an experimental meaning. There are a lot of 
fields where computationnal difficulty is avoided by addind some 
random noise over the existing values just because doing so usually
removes some degeneracies (with 99.999999% of the cases) in the computation
that are just too difficult to deal with and that can be considered
as artifacts of a configuration that will not impact the result (a 
choice between equally good solutions is just made).

- who know that the allowed input data (for their particular program) is well
within the range of the floating point range accuracy they have selected to work with.
Image analysis, games, and lots of real world problem when the data 
are corrupted with noise often enters in this category (measurement 
apparatus seldom give 24 or 53 bits of accuracy).

Proving anything with floating point computation is very difficult in 
some cases even for knowledgeable people. Often, people who really want
to deal with accuracy often go beyond simple floating point computation by
using multiple arithmetics with increasing accuracy or even better, range arithmetic.

> Moreover, if the drawbacks of some optimizations are too dangerous
> (very reduced domain, etc), a solution could be (yes, I realize there
> are too many already) to allow the user to specify how hard one wants
> the compiler to try and speed up the code using -ffast-math-<N>, as in
> -O<N>, or adding --fmath-allow-domain-shrinking, or -fmath-ignore-nans
> or whatever could be used by the programmer to tell the compiler what
> is important to him/her and what can be sacrificed for sped. 

I like the -ffast-math-<N> idea.
By default -ffast-math (equivalent to eg -ffast-math-0) would be very conservative.
Then with N increasing you would allow optimisations that lose more and 
more bits of accuracy.

While I agree that it would only be useful to add things that indeed 
increase the speed by more than a few percents, this is something 
very difficult to state with modern machines, all the more that things 
may add up in a complex fashion, if multiple small optimisations are 
added.

wolfgang.bangerth@iwr.uni-heidelberg.de said:
> - denormals: if you run into the risk of getting NaNs one way but not
>   the other, you most certainly already have gone beyond the limits of
>   your program, i.e. your linear solver diverged, you have chosen an
>   inherently unstable formulation for your problem, etc. In practice,

I totally agree.
All the more that denormals is certainly something that can cost a lot in term of 
computation time since often this is not really implemented in the 
hardware (trap to microcode or to library code). I remember having 
seen decreases in speed by a factor of at least 10, for things that 
would have been equally well be treated as zero.

Often, when using floating point in some tricky situations, I wonder 
whether it would not be better to have very fragile floating point 
arithmetic by default, so that each potential problematic spot is 
easily located as one generating a NaN or an obviously bad result.
I certainly go too far here, but I'm not sure it would not be good in 
the long run...

--------------------------------------------------------------------
Theodore Papadopoulo
Email: Theodore.Papadopoulo@sophia.inria.fr Tel: (33) 04 92 38 76 01
 --------------------------------------------------------------------


-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Exmh version 2.2 06/23/2000

iD8DBQE7aCZYIzTj8qrxOU4RAtXKAJ4zEqZKrUYxF+8rsk4+KVJA1hvAjgCg4KZk
0uhEmXIpS200Azv/xR+bQxM=
=TwfB
-----END PGP SIGNATURE-----

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Theodore Papadopoulo <Theodore.Papadopoulo@sophia.inria.fr> writes:

| gdr@codesourcery.com said:
| > That doesn't mean I don't want optimization.  I do love optimization.
| > By optimization, I really mean optimization. 
| 
| No by optimisation, you mean mathematically provable safe optimisation and 

No, I do mean a transformation which does not dramatically transmute 
the computations.  

Where did you get the idea that I meant "mathematically provable safe
optimisation"?

We're talking here of transformations which we do know able to
drastically change the results.
Going into abstract considerations like NP-completness is beside the
point. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Theodore Papadopoulo]

gdr@codesourcery.com said:
> No, I do mean a transformation which does not dramatically transmute
> the computations.  

That's not a definition. Dramatically, can mean very different things 
depending on the people and on the algorithms. That's basically the 
point that I'm trying to make. In many applications and algorithm
the optimization a/b/c to a/(b*c) is meaningful and relevant. Not 
everyone will have floats that are 2*sqrt(DOUBLE_MAX) (or any other 
combination that might overflow). Actually, a lot of people use a 
representation to make computation that is much wider than the 
accuracy of the input and the accuracy you can get on the output.

Let's face it, there are two ways of using floats.

1) you take enough margin to not care about the accuracy problems (ie 
you take several bits more than ``needed''), and then want to write 
your program as naturally as possible and expect the compiler to do 
what is possible for the computation to be as fast as possible.

2) You are constrained either by the accuracy you would like to get 
or by the size of your data. Then you tend to write without these 
extra bits and pay much more attention to the rounding effects.

NONE OF THE MODELS IS BETTER THAN THE OTHER. There should be 
support in gcc for both communities. You can argue that wanting to 
speed up a computation made with doubles that could have been with 
simple floats is stupid, but:

a) double are often as fast or even faster than floats. Until fairly 
recently, a lot of processor were doing float computation by truncating
the results of a double computation. Recent Sparc processor indeed 
have hardware floats, that go faster than doubles but not 2 times 
IIRC.

b) Writing code that extracts every single possible bit of accuracy 
tends to be very difficult to understand and maintain (see eg DNRM2 
or FNRM2 in blas that just computes the Euclidean norm of a vector 
and you will understand what I mean. By the way, I'm sure that 
nowadays, implementing FNMR2 with double would be faster at the same 
accuracy and much simpler to maintain. For DNMR2 things are more 
compilcated as we would have to switch to long double with which I 
have little experience).


The fact that "95% of the folks out there are completely clueless about
floating point. (J.G. 28 Feb. 1998)" have to be taken into account.
For at least quite many of those 95%, floating point works because 
they are not experiencing the "bounds" of floting point and merely 
stay in the safe area provided by the extra bits of accuracy.

By the way the previous sentence is  cited in a report of Kahan and Darcy
about "Java floating point", where they also state the rule (that can be broken 
eventually):

"Evaluate arithmetic expressions and, except possibly for gargantuan 
arrays, declare temporary (local) variable ALL with the widest finite 
precision that is not too slow nor too narrow."

> Where did you get the idea that I meant "mathematically provable safe
> optimisation"? 

First to be sure we understand each other, by mathematically here, I 
reasonning in a mathematical fashion using the rules governing 
floating point computation and trying to assert that two different 
expressions will give the same result.

That is exactly your point, no. You do not want to allow 
optimisations that would not lead to the same results in boundary 
situations (and again, I agree that this is perfectly fine for some 
community of users, my point is that there is another communitie 
which does not care).

> Going into abstract considerations like NP-completness is beside the
> point.  

Agreed, I was not going into such considerations.

--------------------------------------------------------------------
Theodore Papadopoulo
Email: Theodore.Papadopoulo@sophia.inria.fr Tel: (33) 04 92 38 76 01
 --------------------------------------------------------------------


-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Exmh version 2.2 06/23/2000

iD8DBQE7aEiPIzTj8qrxOU4RAg7OAJ9PnH74nWu7XiH23MteF3JrVfdPmQCfQw60
fY1qP8z2WT8l1kpBcABPL9I=
=YCbP
-----END PGP SIGNATURE-----

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Theodore Papadopoulo <Theodore.Papadopoulo@sophia.inria.fr> writes:

| [1  <text/plain; us-ascii (7bit)>]
| 
| 
| gdr@codesourcery.com said:
| > No, I do mean a transformation which does not dramatically transmute
| > the computations.  
| 
| That's not a definition.

Do you have one?

Changing 0.125 to 0.0 is a dramatic change.

| Dramatically, can mean very different things 
| depending on the people and on the algorithms. That's basically the 
| point that I'm trying to make. In many applications and algorithm
| the optimization a/b/c to a/(b*c) is meaningful and relevant.

In the case I showed, I was tring to compute a unit-length vector by
first scaling in order to avoid overflow (the /c part).

I'm not dismissing the fact that there are algorithms which would
benefit from optimizations.  As I said earlier, I'm a fervent
proponent of optimizations.  Not transformations which change the
computation in a very unexpected ways. 

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Hollebeek]

> While I agree that by default the compiler must follow "good rules", I 
> think that always disallowing optimisations which may lose some bits of 
> accuracy (when specified with a option switch of course) just because 
> it may hurt someone, somewhere, sometime is a little too strong.
> Full safety is IMHO impossible with floating point... Only the 
> programmer knows the accuracy of its input data (Linus called that 
> noise) and the accuracy he expects on its result.

Exactly ... and it isn't at all rare that the programmer knows exactly
how much noise is in the input data (little or none) and the how much
accuracy he expects in the result (as much as possible!).

Rewriting the programmers program in such a way that he gets
significantly less accuracy than he could reasonably expect is just
silly.

Noise is a red herring.  There is no reason why computation should
introduce more innaccuracy than they need to, especially in a ad hoc
way that is beyond the programmer's control.

-Tim

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On Wed, 1 Aug 2001, Tim Hollebeek wrote:
>
> Rewriting the programmers program in such a way that he gets
> significantly less accuracy than he could reasonably expect is just
> silly.

Ehh.. "significantly less accuracy"?

Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
Let's confine this to x86 - it is the most common case by far, and hey,
especially when it comes to games etc I doubt it matters what HP-PA does,
for example. We can easily make the optimizations be enabled on a
per-architecture basis.

The one obvious case is (b*c) overflows and turns into Inf, at which point
the end result will be +-0 with the optimization. Now, in order for the
original to _not_ have been close to zero, we're talking about 'a' being
at the very limits of the number space. And remember that on x86 register
arithmetic - regardless of precision setting - the exponent is 15 bits. So
in order to get the overflow, AND to get a original value that wasn't zero
(ie to get an error), 'a' would have had to be on the order of

	2**(16383-64)

which won't even _fit_ in a double anyway (the -64 is to take even
denormals into account, but others have said that flush-to-zero would have
been acceptable anyway by default). Did I mis-do my calculations.

(Repeat argument for underflow).

So the overflow case actually doesn't exist on x86 if we keep the values
in registers or flush to stack in extended real format (and if we do NOT
flush to stack in extended real format, x86 has other problems like the
fact that the results depend on register flushes etc).

Do we have other problems with the conversion? Sure, it might be 1ULP off
even in the non-overflow case, but as we've seen others do that
optimization by default, never mind even with -ffast-math.

So tell me, what makes the optimization so dangerous? It doesn't seem to
do anything that we don't already do.

Did I miss some noteworthy case?

		Linus

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?

On a 64-bit double processor just take 

	b = c = 2 * sqrt(DBL_MAX);
	a = 0.5 * DBL_MAX

a/(b*c) is 0.0 whereas a/b/c yields 0.125.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Aug  1, 2001, Gabriel Dos_Reis <gdosreis@sophia.inria.fr> wrote:

> | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?

> On a 64-bit double processor just take 

> 	b = c = 2 * sqrt(DBL_MAX);
> 	a = 0.5 * DBL_MAX

> a/(b*c) is 0.0 whereas a/b/c yields 0.125.

I'm no expert in FP arithmetic, but it appears to me that this chunk
of code is a materialization of what Linus wrote:

> The one obvious case is (b*c) overflows and turns into Inf, at which
> point the end result will be +-0 with the optimization.

If there is a difference, will you care to enlighten me?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Alexandre Oliva <aoliva@redhat.com> writes:

| On Aug  1, 2001, Gabriel Dos_Reis <gdosreis@sophia.inria.fr> wrote:
| 
| > | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
| 
| > On a 64-bit double processor just take 
| 
| > 	b = c = 2 * sqrt(DBL_MAX);
| > 	a = 0.5 * DBL_MAX
| 
| > a/(b*c) is 0.0 whereas a/b/c yields 0.125.
| 
| I'm no expert in FP arithmetic, but it appears to me that this chunk
| of code is a materialization of what Linus wrote:
| 
| > The one obvious case is (b*c) overflows and turns into Inf, at which
| > point the end result will be +-0 with the optimization.
| 
| If there is a difference, will you care to enlighten me?

You're quoting just a tiny part of his rhetorics and that can cause
confusion. 

He used the argument above to derive that for a/b/c not to yeild 0.0,
a should have been very very big, which can't happen in practice.

My example is there to show that is untrue.  A 64-bit double processor
isn't something uncommon and a, b, and c are both representable.

Now, if that transformation is considered only on a x86 target then
that is en entirely different debate -- but then we the discussion
started, that wasn't the assumption.  That restrcition was introduced
at the very end.

This is just an example of where quick extrapolation doesn't work.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Alexandre Oliva]

On Aug  1, 2001, Gabriel Dos Reis <gdr@codesourcery.com> wrote:

> Alexandre Oliva <aoliva@redhat.com> writes:
> | On Aug  1, 2001, Gabriel Dos_Reis <gdosreis@sophia.inria.fr> wrote:
> | 
> | > | Now, show me a case where a/b/c -> a/(b*c) is significantly less accurate?
> | 
> | > On a 64-bit double processor just take 
> | 
> | > 	b = c = 2 * sqrt(DBL_MAX);
> | > 	a = 0.5 * DBL_MAX
> | 
> | > a/(b*c) is 0.0 whereas a/b/c yields 0.125.
> | 
> | I'm no expert in FP arithmetic, but it appears to me that this chunk
> | of code is a materialization of what Linus wrote:
> | 
> | > The one obvious case is (b*c) overflows and turns into Inf, at which
> | > point the end result will be +-0 with the optimization.
> | 
> | If there is a difference, will you care to enlighten me?

> You're quoting just a tiny part of his rhetorics

Indeed, I failed to quote the other relevant part.  Linus wrote:

> Now, in order for the original to _not_ have been close to zero,
> we're talking about 'a' being at the very limits of the number
> space.

It appears to me that 0.5 * DBL_MAX is indeed at the very limits of
the number space.  So, I still fail to see the difference.

> He used the argument above to derive that for a/b/c not to yeild 0.0,
> a should have been very very big, which can't happen in practice.

But he was explicitly talking about x86's extended precision when he
made this argument.

> My example is there to show that is untrue.  A 64-bit double processor
> isn't something uncommon and a, b, and c are both representable.

I see.  Anyway, it's still true that 0.5 * DBL_MAX is ``at the very
limits of the number space'', so all bets about numbers being ``far
from overflowing'' are off, and optimizations that rely on this
assumption are indeed likely to break programs.

As we have seen, not all FP programs work in ranges close to overflow
limits, so these could benefit from such optimizations M-backspace
transformations.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist    *Please* write to mailing lists, not to me

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos_Reis]

| > | > The one obvious case is (b*c) overflows and turns into Inf, at which
| > | > point the end result will be +-0 with the optimization.
| > | 
| > | If there is a difference, will you care to enlighten me?
| 
| > You're quoting just a tiny part of his rhetorics
| 
| Indeed, I failed to quote the other relevant part.  Linus wrote:

More accurately he wrote:

  The one obvious case is (b*c) overflows and turns into Inf, at which point
  the end result will be +-0 with the optimization. Now, in order for the
  original to _not_ have been close to zero, we're talking about 'a' being
  at the very limits of the number space. And remember that on x86 register
  arithmetic - regardless of precision setting - the exponent is 15 bits. So
  in order to get the overflow, AND to get a original value that wasn't zero
  (ie to get an error), 'a' would have had to be on the order of

	  2**(16383-64)

  which won't even _fit_ in a double anyway (the -64 is to take even
  denormals into account, but others have said that flush-to-zero would have
  been acceptable anyway by default). Did I mis-do my calculations.

Emphasis is not mine.  Please note, what he showed: That the
situation (i.e. a/b/c) couldn't have been well-formed in the first
place.  For x86, that may be true.  But that is a specific case. 

| > Now, in order for the original to _not_ have been close to zero,
| > we're talking about 'a' being at the very limits of the number
| > space.
| 
| It appears to me that 0.5 * DBL_MAX is indeed at the very limits of
| the number space.  So, I still fail to see the difference.

I don't consider 0.5 * DBL_MAX to be at the "very limits of the number
space".

| > He used the argument above to derive that for a/b/c not to yeild 0.0,
| > a should have been very very big, which can't happen in practice.
| 
| But he was explicitly talking about x86's extended precision when he
| made this argument.

Yes, he was talking about x86.  But again, when we started discussing
the transformation, there were no limitation to targets.  Again, I
exhibit the example to show that there are out there bunch of
processors where the stance "it won't matter " isn't true.  Now
if the transformation were to be restricted to x86, then that is a
different issue.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Toon Moene]

Gabriel Dos_Reis wrote:

> I don't consider 0.5 * DBL_MAX to be at the "very limits of the number
> space".

Gaby, I'll try a different approach - I'm not going to say you are an
<elided> or your analysis is wrong, flawed, or biased.

I'm just going to point you at the topic of this thread: "What is
acceptable for -ffast-math ?".

Now, try for a moment to think with me.  During the hoolabaloo around
the Pentium FDIV error, I actually went into our weather forecasting
code to count the floating point divisions per grid cell per time step
(70).

The grand total for the whole forecast period was 2.2x10^9.

That was end-of-1994.  Nowadays, we have about 15 times more grid cells
and a 33 % longer forecast period.  Because I don't want to do this
counting exercise again, I'll just assume the same number of divisions
per grid cell, per time step, i.e. 70.

That makes the total 20 x 2.2 x 10^9 ~ 0.5 x 10^11 divisions per
forecast.

If we further assume that:

1) All of these divisions are of the form that allow the transformation.

2) Some forecast will hit your example.

3) The probability distribution for this hit is uniform (this assumption
   is doubtful, because of the value involved).

this means that one in 2^53 / (0.5x10^11) ~ 10^5 forecasts will hit the
error.

I'm sorry, but I do not have the time to wait for the off-chance that
this error will occur, so I'll happily apply this transformation.

Now, even given this, I would certainly heed Roberts advice that the
transformation should actually be a perceptable "optimisation".

In our case, I doubt this (but then, I - personally - am not interested
in anything less than a 10 % improvement).

Cheers,

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
Join GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

> The _only_ person who can judge whether it's wrong to do so it the person
> who wrote the program. He may be happy with the faster code. He's almost
> certainly happy if he used "-ffast-math".

Well it sure would be nice to here from some of these mythical numerical
programmers (I don't care if they are writing games or nuclear reactor codes)
who would be happier, so far we haven't heard this! And in my experience,
even quite inexperienced floating-point numerical programmers are very
disturbed when optimization changes the results of their programs.

> Your arguments about "numerical computation" are just silly, as you don't
> seem to realize that there are tons of problems where your theoretical
> issues are nothing more than noise.

If you think the arguments are silly, then I really fear you lack the full
context for this discussion, a discussion that has, as you should know raged
for well over thirty years.

> Go back and look at why it was added. Right - exactly so that people could
> get _fast_ math, when they do not care about the inexactness that isn't
> even an issue for most people.

Sure -ffast-math is precisely intended to allow transformations that would
not otherwise be allowed (let's not call them optimizations, that's just
too controversial a word in the context of this argument).

The question is what is the boundary of allowable transformations. No one
agrees that there should be no boundaries (even you don't like the change
results to zero, though note that abandoning denormals has exactly this
effect, and might be considered acceptable).

So, what is the boundary, can one for instance forget about denormals and
flush to zero to save a bit of time, can one truncate instead of round,
can one ignore negative zeroes, or infinity semantics, can one ignore
intermediate overflow (note: nearly all the discussed transformations are
implicitly saying yes to this last question).

I have not seen anyone writing from the point of view of serious numerical
coding saying "sure, go ahead and transform a*b + a*c, I write that all
the time and would be happy if the compiler would speed it up in the
obvious manner". So there is real doubt as to whether this particular
transformation (which can introduce overflow where none existed before)
is one that we can agree to include.

It seems to me that the criteria for including a transformation are

  a) it is well understood and clear
  b) it really makes a difference in performance
  c) it does not introduce surprises

Unfortunately, these are often in conflict. For instance, in C, we usually
degrade the precision of the x86 floating-point unit to minimize surprises,
even though operating in full precision would be more efficient in some
cases. Should -ffast-math allow full precision operation? I would think so,
since it definitely improves performance, and reduces surprises.

Similarly, I would go for ignoring accuracy in negative zero handling, since
that so clearly meets criteria a) and c). 

But really it would be very valuable to have quantitative data that show that
a particular optimization is worth while. I will say again that the particular
case that has been discussed, which is associative law redistribution, has not
carried that burden for me AT ALL. It is a dubious transformation, violating
condition c) above, and you can see that it is dubious, given the reaction of
many, not just me, on this list. Furthermore, promoting something like this
gives an impression to serious numerical analysts that "those GCC guys" simply
don't understand the requirements of floating-point arithmetic, and that's a
cost one does not want to freely pay unless you are sure you are getting 
something in return.

We can't look to the standard here, because the question is one of taste. 
Even in Ada, in relaxed mode (the equivalent of -ffast-math, but acknowledged
and defined somewhat in the stadard), the standard leaves these kind of
decisions to the implementor. However, I cannot imagine any Ada implementor
being as cavalier as you would suggest.

By the way, I said I would be shocked to find a Fortran compiler that did
associative redistribution in the absence of parens. I am somewhat surprised
that no one stepped forward with a counter-example, but I suspect in fact that
there may not be any shocking Fortran implementations around.

It is an old argument, the one that says that fpt is approximate, so why bother
to be persnickety about it. Seymour Cray always tool this viewpoint, and it
did not bother him that 81.0/3.0 did not give exactly 27.0 on the CDC 6000
class machines.

But the advent, and essentially universal adoption, of IEEE arithmetic, really
signifies that Seymour's viewpoint (and yours in this discussion) have lost the
argument and that indeed we want floating-point to be well behaved even though
we know many programmers won't know what this means. Our experience is that
even these inexperienced programmers gain from a better floating-point model,
and that, surprise! the cost in hardware to give such a model turns out to be
minimal.

Now compiler writers always get carried away with the supposed importance
of miscellaneous optimizations, and every compiler writer has the experience
of working hard on some optimization, only to find that it has little or
no effect.

If indeed we could show that associative redistribution had a large effect
on important programs (be they games, SPEC tests, or nuclear reactor codes),
then we would have a different situation from the hardware folks. But I suspect
that careful measurement would show that we are in the same boat, and that the
cost of providing decent floating-point semantics is too small for it to be
worth worrying about very much.


Robert

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Toon Moene]

dewar@gnat.com wrote:

> By the way, I said I would be shocked to find a Fortran compiler that did
> associative redistribution in the absence of parens. I am somewhat surprised
> that no one stepped forward with a counter-example, but I suspect in fact that
> there may not be any shocking Fortran implementations around.

Well, that might be because we're on a public list and I, certainly, am
not going to name names of competitors.  Granted, I do not have an
actual proof of a Fortran compiler that substituted (a+b)*c for a*b+a*c,
but I have seen those that:

1. Included an option to ignore parentheses in the source program.

2. Included an option to use floating point induction variables, i.e.,
   change:

      DO I = 1, N
         A(I) = I * 0.1
      ENDDO

   into:

      T = 0.0
      DO I = 1, N
         T = T + 0.1
         A(I) = T
      ENDDO

> It is an old argument, the one that says that fpt is approximate, so why bother
> to be persnickety about it. Seymour Cray always tool this viewpoint, and it
> did not bother him that 81.0/3.0 did not give exactly 27.0 on the CDC 6000
> class machines.

In one of the papers I read during the 1999 discussion, I recall seeing
a quote by Kahan along the lines (paraphrased, because I can't find it
anymore): Yes, there are floating point codes that are robust against
Cray's floating point implementation ...

The discussion actually boils down to: "What is the answer we want to
obtain using these calculations ?".

In physics, the computational model in itself is often already a
compromise - if I just confine myself to weather forecasting, which I'm
involved with now for about a decade:

1. Spherical Earth, rotating with constant angular velocity,
2. Ignoring compressibility of the air, assuming hydrostatic equilibrium
   at all time.
3. Ignoring the vertical component of the Coriolis force,
4. Constant acceleration of gravity, independent of height.

Now the compromises to make this actually *computable*:

1. Finite difference discretisation of the continuous equations,
2. Filters to damp numerical noise (both in space and time dimensions),
3. Approximations to compute the compound effect of sub-grid scale
   physics (radiation, phase changes of water, etc.).

Now let's look at the question above, again: 

"it did not bother [Cray] that 81.0/3.0 did not give exactly 27.0 on 
 the CDC 6000 class machines."

In the light of the above, does this observation of "Cray floating
point" matter ?  It depends, of course:

1. Help, on the CDC's 81.0 / 3.0 is zero !  Yes, *that* would be a 
   problem.
2. Hmmm, CDC 6000's seem to get as much as 5 ULP's wrong on division
....
   Well, I wouldn't care.

So, I wouldn't mind a "-fcray-fp" option in the latter case.  Sure, the
answers from our weather forecast model would be different under that
floating point model than under the IEEE-754 one.  However, I'm not much
interested in the *computational* answer of the model, I'm interested in
the weather forecast I can derive from it.  So if one answer is 10.2
degrees Celsius tomorrow morning at 6 a.m. and the other is 10.1 degrees
Celsius, which is the "correct" one ?  The actual temperature will
probably be somewhere between 11 and 12 ... not bad.

"Predicting is hard, especially as far as the future is concerned"

Again, the issue is: In my use of these computational cycles to solve a
physics problem, what constitutes an answer ?  *Is* there only one,
correct, answer ?  Is an answer in the vicinity of the computationally
correct answer really "incorrect" ?  If no, what is "vicinity" in this
case ?  Do I have a method to determine "vicinity" based on the
computational compromises I made ?  On the physical compromises ?

Fortunately, due to the physical nature of the problem I'm trying to
solve, I can actually use a measurement to determine the "success" of my
endeavor.  So what's the accuracy of the measurement device ?  How does
the fact that the computational model deals with masses of air 20km x
20km x 100 meter instead of the few cubic meters of air the instrument
deals with disturb my "verification" ?  *Is* the difference I measure
due to this effect, or due to the poor physics approximation, or due to
the computational approximation, ... or due to the 5 ULP's of the Crayzy
division ?

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
Join GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Prince]

----- Original Message -----
From: "Toon Moene" <toon@moene.indiv.nluug.nl>
To: <dewar@gnat.com>
Cc: <gdr@codesourcery.com>; <torvalds@transmeta.com>; <fjh@cs.mu.oz.au>;
<gcc@gcc.gnu.org>; <moshier@moshier.ne.mediaone.net>;
<tprince@computer.org>
Sent: Tuesday, July 31, 2001 3:57 PM
Subject: Re: What is acceptable for -ffast-math? (Was: associative law
in combine)


> dewar@gnat.com wrote:
>
> > By the way, I said I would be shocked to find a Fortran compiler
that did
> > associative redistribution in the absence of parens. I am somewhat
surprised
> > that no one stepped forward with a counter-example, but I suspect in
fact that
> > there may not be any shocking Fortran implementations around.
>
> Well, that might be because we're on a public list and I, certainly,
am
> not going to name names of competitors.  Granted, I do not have an
> actual proof of a Fortran compiler that substituted (a+b)*c for
a*b+a*c,
> but I have seen those that:
>
> 1. Included an option to ignore parentheses in the source program.
>
> 2. Included an option to use floating point induction variables, i.e.,
>    change:
>
>       DO I = 1, N
>          A(I) = I * 0.1
>       ENDDO
>
>    into:
>
>       T = 0.0
>       DO I = 1, N
>          T = T + 0.1
>          A(I) = T
>       ENDDO
>
The usual meaning of an option to permit floating point induction
variables would be to permit the usage which was covered by the f77
standard and later ruled out, not to introduce them where the source
code is written with integer induction variables.  The Kuck KAP
pre-processors worked the other way, converting the second version into
the first, with documented options provided to prevent it.  That
transformation was often desired to promote parallelizaton.
> In physics, the computational model in itself is often already a
> compromise - if I just confine myself to weather forecasting, which
I'm
> involved with now for about a decade:
>
> 1. Spherical Earth, rotating with constant angular velocity,
> 2. Ignoring compressibility of the air, assuming hydrostatic
equilibrium
>    at all time.
> 3. Ignoring the vertical component of the Coriolis force,
> 4. Constant acceleration of gravity, independent of height.
>
> Now the compromises to make this actually *computable*:
>
> 1. Finite difference discretisation of the continuous equations,
> 2. Filters to damp numerical noise (both in space and time
dimensions),
> 3. Approximations to compute the compound effect of sub-grid scale
>    physics (radiation, phase changes of water, etc.).
>
> Now let's look at the question above, again:
>
> "it did not bother [Cray] that 81.0/3.0 did not give exactly 27.0 on
>  the CDC 6000 class machines."
>
Typical compilers for IA-64 are off frequently by 1 ULP on simple
division calculations at -O3, and on sqrt() at -O2.  This is a normal
consequence of substituting a series of iterations without extra
precision for division.  Some may consider this an excessive reversion
toward the Cray tradition.
>

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Per Abrahamsen]

dewar@gnat.com writes:

> Well it sure would be nice to here from some of these mythical numerical
> programmers (I don't care if they are writing games or nuclear reactor codes)
> who would be happier, so far we haven't heard this! 

I'm writing agricultural simulations[1], and I'd be happy to get
transformations enabled with -ffast-math that would be safe with real
numbers, but not with floating point numbers.

And I'm one of those incompetant programmers that are forced to write
a * b + a * c rather than a * (b + c), when the former is easier to
understand, i.e. closer to the original problem.  Otherwise my small
brain can simply not grasp what the code was meant to signify, when I
reread it later.

So we aren't quite mythical.  Maybe we really ought to be shoot and
replaced by Real Programmers(TM), but there seem to be a shortage of
those, so that is not likely to happen soon.

Footnotes: 
[1]  See <URL: http://www.dina.kvl.dk/~daisy/ >.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[mike stump]

> Date: Mon, 30 Jul 2001 18:45:49 -0400 (EDT)
> From: Stephen L Moshier <moshier@mediaone.net>
> To: mike stump <mrs@windriver.com>
> cc: gcc@gcc.gnu.org

> > > Further, GCC is supposed to be standards compliant by default.
> > 
> > Really?

> Well, then I'd like to understand the actual policy better.  You
> have GNU extensions but otherwise seem to include what's in the
> language definition.  The apparent conflicts arise when you say the
> GNU extensions have to be turned off in order for the compiler to be
> strictly compliant with the standard.

Sounds like you have a pretty good idea already.  Mostly compliant
isn't the same as compliant.  If you want to understand the
differences better, take a look at all the code that -mieee, -ansi,
-std... and -pedantic-errors changes.  I think I got all the right
flags, but I may have missed one, also, I'm not up on the ieee stuff,
so the first one could be wrong.

It seems that the extensions fall on enough peoples radars for this to
be an issue, which rises this to a level higher than just an
insignificant nit; I tend to think of it as a minor nit.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[mike stump]

> From: dewar@gnat.com
> To: moshier@moshier.ne.mediaone.net, mrs@windriver.com
> Date: Mon, 30 Jul 2001 21:00:23 -0400 (EDT)

> Does the standard forbid upwards compatible extensions?

Roughly speaking, when something is undefined, we are allowed to
define it.  When behavior is defined, we are not allow to behave as if
we differed from it.  When something is unspecified, we are allowed to
specify it.

For example:

#include <alloca.h>

isn't defined, therefore, we can define it to do anything.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

>isn't defined, therefore, we can define it to do anything.

Right, that's what I mean by an upwards compatible extension, i.e. one that
cannot affect any legal defined program. So the answer is indeed for C and
C++ that extensions are allowed freely if they are upwards compatible (this
is not the case in COBOL and Ada), where the standard requires a mode in 
which no extensions are made.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

surely -mieee is not required to be standards compliant?

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[mike stump]

> From: dewar@gnat.com
> To: dewar@gnat.com, jbuck@synopsys.COM
> Cc: gcc@gcc.gnu.org, moshier@moshier.ne.mediaone.net, mrs@windriver.com
> Date: Mon, 30 Jul 2001 21:38:24 -0400 (EDT)

> But it is interesting that many of the GNU extensions do not
> conflict with the standard (e.g. nested procedures, dynamic arrays),
> so these are not examples of GNU-C not meeting the criterion of
> being standard by default.

?

When there is wording that states that the value inside the [] shall
be a constant greater than 0 and non-negative, there is absolutely no
room for gcc to extend the sematics to include dynamicly sized arrays,
sorry.

  If the constant-expression (5.19) is present, it shall be an
  integral constant expression and its value shall be greater than
  zero.

From:

1 In a declaration  T  D where  D has the form

          D1 [constant-expressionopt]

  and the type of the identifier in the declaration  T  D1 is   derived-
  declarator-type-list    T, then the type of the identifier of  D is an
  array type.   T is called the array  element  type;  this  type  shall
  not  be  a  reference  type,  the  type    void, a function type or an
  abstract class type.  If the  constant-expression (5.19)  is  present,
  it  shall  be  an  integral constant expression and its value shall be
  greater than zero.  The constant expression specifies the    bound  of
  (number  of  elements  in)  the  array.   If the value of the constant
  expression is  N, the array has  N elements numbered  0 to   N-1,  and
  the type of the identifier of  D is derived-declarator-type-list array
  of  N  T.  An object of array type contains a  contiguously  allocated


  8.3.4  Arrays           DRAFT: 2 December 1996       Declarators  8-11

  non-empty  set  of    N  sub-objects  of  type    T.   If the constant
  expression is omitted, the type of the identifier of   D  is  derived-
  declarator-type-list  array  of  unknown  bound  of   T, an incomplete
  object type.  The type derived-declarator-type-list array of  N  T  is
  a  different  type from the type derived-declarator-type-list array of
  unknown bound of  T, see 3.9.  Any type of the  form  cv-qualifier-seq
  array  of    N   T is adjusted to array of  N cv-qualifier-seq  T, and
  similarly for array of unknown bound of  T.  [Example:

          typedef int A[5], AA[2][3];
          typedef const A CA;     // type is ``array of 5 const int''
          typedef const AA CAA;   // type is ``array of 2 array of 3 const int'\
'

   -end example] [Note: an array of   N  cv-qualifier-seq    T  has  cv-
  qualified  type;  such an array has internal linkage unless explicitly
  declared  extern (7.1.5.1) and must be  initialized  as  specified  in
  8.5.  ]

I don't have any clue how you were so misled on this.  Similar I
suspect for nested procedures.  When the standard says that violation
of a mandate of the standard must result in a diagnostic:

2 Every conforming C++ implementation shall, within its resource limits,
  accept and correctly execute well-formed C++ programs, and shall issue
  at least one diagnostic message when  presented  with  any  ill-formed
  program  that contains a violation of any diagnosable semantic rule or
  of any syntax rule.

this _does_ result in failure to comply with the standard and gcc does
fail to meet a criterion of the standard, as you put it.  These are
reasonably safe extensions, as no conforming program can use the
standard to complain about the semantics then, but that is all.

#include <alloca.h> is an example of an allowed extension.  It isn't
defined in the standard, and the standard doesn't mandate a
diagnostic.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<When there is wording that states that the value inside the [] shall
be a constant greater than 0 and non-negative, there is absolutely no
room for gcc to extend the sematics to include dynamicly sized arrays,
sorry.
>>

I disagree, this is giving the rules for legal, well-define programs, and
the standard has nothing to say about the behavior of a program that does
not follow the above rules.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[mike stump]

> From: dewar@gnat.com
> To: dewar@gnat.com, moshier@moshier.ne.mediaone.net, mrs@windriver.com
> Cc: gcc@gcc.gnu.org
> Date: Tue, 31 Jul 2001 11:35:04 -0400 (EDT)

> >isn't defined, therefore, we can define it to do anything.

> Right, that's what I mean by an upwards compatible extension, i.e. one that
> cannot affect any legal defined program. So the answer is indeed for C and
> C++ that extensions are allowed freely if they are upwards compatible (this
> is not the case in COBOL and Ada), where the standard requires a mode in 
> which no extensions are made.

What did you mean by `are allowed freely'?  Did you mean are permitted
by the standard?  If so, then you are wrong.  They are not allowed,
there is a requirement upon all processors of the language that they
issue a diagnostic.  This requirement of a diagnostic is not what I
would call `are allowed freely'.  If you meant anything else, you can
ignore this message.

> From: dewar@gnat.com
> To: dewar@gnat.com, jbuck@synopsys.COM, mrs@windriver.com
> Cc: gcc@gcc.gnu.org, moshier@moshier.ne.mediaone.net
> Date: Tue, 31 Jul 2001 11:37:02 -0400 (EDT)

> <<When there is wording that states that the value inside the [] shall
> be a constant greater than 0 and non-negative, there is absolutely no
> room for gcc to extend the sematics to include dynamicly sized arrays,
> sorry.
> >>

> I disagree, this is giving the rules for legal, well-define
> programs, and the standard has nothing to say about the behavior of
> a program that does not follow the above rules.

In case it wasn't perfectly clean in the above, I meant, there is no
room within the confines of the standard.  If you don't disagree with
that, you can stop reading now; however, if you do, read on...

The standard says nothing about the behavior of any program, you
misunderstand the standard.  The standard only give requirements on
processors of the language.  Any violation of any requirement of the
standard upon a processor of the lanuage, is a violation of the
standard.  There is no difference in the standard wether the
requirement is for a diagnostic, or for some behavior of running code.
Any differentiation made is extra-standard.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<Well, that might be because we're on a public list and I, certainly, am
not going to name names of competitors.  Granted, I do not have an
actual proof of a Fortran compiler that substituted (a+b)*c for a*b+a*c,
but I have seen those that:
>>

Well you should certainly not hesitate to name compilers here, this is hardly
a sensitive issue, and indeed there are those on this list who think that
associative rearrangement *is* an appopriate "optimization" even if it
introduces overflow where none existed for the canonical evaluation.

But in fact, as you make clear, you don't in fact know of any Fortran compiler
that does associative rearrangement, and I repeat my point ... I would be
surprised if there is any such in existence.

<<2. Included an option to use floating point induction variables, i.e.,
   change:

      DO I = 1, N
         A(I) = I * 0.1
      ENDDO

   into:

      T = 0.0
      DO I = 1, N
         T = T + 0.1
         A(I) = T
      ENDDO
>>

Must be an old compiler. I can't think of any modern machine where such a
change would make a difference. Floating-point IEEE addition is typically
at least as difficult, sometimes one pipeline stage harder, than floating-point
multiplication, so that's a pretty pointless transformation on modern machines
as well as being a real horror (this not a case where the error is limited
to a few ULP!)

And that's really the point. For modern machines, are we really talking
about transformations that make any difference to the performance? Not
once in this long discussoin has anyone provided even the tiniest smidgeon
of quantitative support for the position that you do see a real performance
difference in real programs. Linus may declare that he knows that this is
a significant issue, and call people silly if they disagree, but that hardly
counts as quantitative data :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

I still see *no* quantitative data showing that the transformation most
discussed here (associative redistribution) is effective in improving
performance.

Linus seems to be getting a little emotional in this discussion but swearing
does not replace data.

My viewpoint, which I will repeat briefly is that we have to look at each
proposed transformation individually, and determine, by quantitative
measurement (many of these measurements are in the literature) how great
the gain is, and then the gain has to be traded off against the effect of
the transformation.

This seems such a reasonable position that I really have a heck of a time
understanding why Linus is so incensed by it.

Note that the requirement that there be a significant gain is a gate. If a
given transformation cannot be shown to be an effective contributor to
performance, then we don't need to discuss the effect of the transformation.
I suspect that in practice, many of the proposed transformations simply won't
get past this gate, and that thus a lot of hot air is expended uselessly 
arguing about the second stage.

In the case that a given transformation *is* effective, then we can discuss.

Let's take an example, which is multiplication by a reciprocal. This of
course does give "wrong" results. But division is often quite slow. On the
other hand, I can't imagine floating-point programmers, including those
doing game stuff, not knowing this, and thus writing the multiplication
in the first place. 

<<Oh, round-to-zero is definitely acceptable in the world of "who cares
about IEEE, we want fast math, and we'll use fixed arithmetic if the FP
code is too slow".
>>

That's probably a bad idea, on most modern machines, full IEEE floating-point
is faster than integer arithmetic (particularly in the case of multiplication)

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Linus Torvalds]

On 31 Jul 2001, Gabriel Dos Reis wrote:
>
> What I intended to mean is that the compiler has, first, to prove that
> such transformmation yields the same result.

Ehh..

You're _wrong_.

A lot of compiler optimizations yield different results. The question is
whether it is acceptable or not.

It _clearly_ is acceptable to do when the user has explicitly asked for
"-ffast-math". The whole meaning of the flag is to say "I want it fast,
even if you might lose some range or precision".

It's only a matter of defining the semantics.

There are other optimizations like that. Signed integer overflow is
undefined, so you can optimize integer operations in ways that do not give
the same results for all integers.

Alias analysis was added several years ago, even though it was _clearly_ a
"illegal optimization" from a traditional C standpoint. That one broke
tons of real programs. Again, it was a question of re-defining the
semantics, and allowing people to take advantage of them.

Already, the _default_ behaviour for gcc at least on x86 with floating
point has very little to do with IEEE. It's been discussed several times
how to fix it - and very little real work on it has been done for the last
several years. Why? Because it really doesn't matter for most people, and
the current work-arounds are too much of a performance hit.

So gcc _already_ has "better fast than correct to the last ULP in IEEE".
Live with it. Even by default. Your argument that it cannot do the same
even when the user asks for it is completely ludicrous.

Your continual insistence that it is "wrong" only shows that you have
built a cage around your brain.

Have you even _looked_ at what the gcc definition for "fast-math" are?
Look again:

  {"fast-math", &flag_fast_math, 1,
   "Improve FP speed by violating ANSI & IEEE rules" },

and realize that even something simple as "double" doesn't historcially
have an exact meaning in C (ie you can _not_ assume specific ranges or
specific IEEE rounding in traditional C - it was left up to the
implementor to do as best he could with what hardware was available).

That means that traditionally, you don't even know what overflows or not,
or what precision you get. It's historically "undefined". Which means that
the semantics are not clear.

Now, this is obviously why a lot of computational people disliked,
distrusted and didn't use C historically, preferring to use Fortran which
had more defined rules. And which is, of course, why C99 integrated the
math extensions that some people have been working on for a loong time.

But you have to realize that this is (a) fairly recent and (b) not
everybody needs or wants it.

And you should also realize that YOU are not the grand poobah that gets to
tell people what semantics they want.

[ Neither am I, of course, but I'm arguing for choice and freedom, while
  you're arguing for small-mindedness everywhere. ]

			Linus

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos Reis]

Linus Torvalds <torvalds@transmeta.com> writes:

| Look again:
| 
|   {"fast-math", &flag_fast_math, 1,
|    "Improve FP speed by violating ANSI & IEEE rules" },
| 
| and realize that even something simple as "double" doesn't historcially
| have an exact meaning in C (ie you can _not_ assume specific ranges or
| specific IEEE rounding in traditional C - it was left up to the
| implementor to do as best he could with what hardware was available).
| 
| That means that traditionally, you don't even know what overflows or not,

You don't need to have an IEEE implementation before speaking of
overflow.  That is the point you seem to miss.

|   you're arguing for small-mindedness everywhere. ]

like yours?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

[ Neither am I, of course, but I'm arguing for choice and freedom, while
  you're arguing for small-mindedness everywhere. ]

Well that's not really the case. Because if your argument leads to
-ffast-math going overboard and going too far in giving what most people
regard as wrong results, then you HAVE taken away choice and freedom,
because -ffast-math is no longer usable for these people. That's why it
is perfectly reasonable to have a technical argument on a case by case
basis.

Trying to argue from a general ideological point of view (I favor freedom,
you do not, therfore you are wrong) is not particularly helpful in this
discussion.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<The _definition_ of "-ffast-math" is "fast and loose". It says so. Try
this:
>>

I hardly thing "fast and loose" is a definition. We know perfectly well 
that there are transformations that Linus would find unacceptable, so no
one is in favor of unrestrained looseness. We are simply (trying to) have
a technical discussion about how loose is loose.

By the way, this entire discussion has been played out in the Ada context,
since in the Ada standard, the two modes are recognized at the standard
level, but the standard does not attempt to define fast and loose.

Note that in classical C, there was no statement about floating-point
accuracy at all. So from a definitional point of view, the compiler could
do anything, but people had some general agreement over what was or was
not appropriate. We are simply trying to reach that general agreement in
this -ffast-math context.

By the way, going back to an earlier post, Linus claims that GCC by default
allows extra precision. That's interesting, since it would mean that on
Power, optimization changes the results. I mentioned earlier that IBM decided
after careful study that this was not acceptable to their numeric users, so
it is interesting for gcc to take an opposite point of view (I suspect that
this was done without careful study :-)

Actually what I had in mind was 80-bit on x86, and I don't think gcc operates
that way by default?

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Joern Rennecke]

> Actually what I had in mind was 80-bit on x86, and I don't think gcc operates
> that way by default?

It just leaves the rounding mode alone, but some library code
(glibc? newlib?) generally assumes it to be for 64 bit mantissa, i.e. 80 bit
XFmode.

OTOH, if a pseudo gets spilled, double gets a 64 bit stack slot, and float
32 bit.  So we get double rounding effects and inconsistent precision.
The -ffloat-store option goes some way to mitigate these problems, at a
significant cost in performance.  There was some discussion to set the
rounding mode to 64 bit, but that would break library code that assumes
XFmode values, and would still get excess precision in borderline cases:
values that are IEEE denormals in 64 bit format are not denormal in the
register format, because the exponent part of the fp registers remains
16 bit no matter what the roundig mode is set to.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Tim Prince]

----- Original Message -----
From: "Joern Rennecke" <amylaar@redhat.com>
To: <dewar@gnat.com>
Cc: <amylaar@redhat.com>; <torvalds@transmeta.com>; <aoliva@redhat.com>;
<gcc@gcc.gnu.org>; <gdr@codesourcery.com>;
<moshier@moshier.ne.mediaone.net>; <tprince@computer.org>
Sent: Tuesday, July 31, 2001 6:50 PM
Subject: Re: What is acceptable for -ffast-math? (Was: associative law
in combine)


> > Actually what I had in mind was 80-bit on x86, and I don't think gcc
operates
> > that way by default?
>
> It just leaves the rounding mode alone, but some library code
> (glibc? newlib?) generally assumes it to be for 64 bit mantissa, i.e.
80 bit
> XFmode.
>
> OTOH, if a pseudo gets spilled, double gets a 64 bit stack slot, and
float
> 32 bit.  So we get double rounding effects and inconsistent precision.
> The -ffloat-store option goes some way to mitigate these problems, at
a
> significant cost in performance.  There was some discussion to set the
> rounding mode to 64 bit, but that would break library code that
assumes
> XFmode values, and would still get excess precision in borderline
cases:
> values that are IEEE denormals in 64 bit format are not denormal in
the
> register format, because the exponent part of the fp registers remains
> 16 bit no matter what the roundig mode is set to.
Spilling float calculations in double may  produce double rounding only
when there are at least 2 operations between spills.  If the rounding
precision is set to 53-bit double, it will make no difference whether
there is a spill.  That may be an argument for this practice; I don't
that it is a primary one.
 As newlib is intended to be independent of the presence of 80-bit
registers or not, and contains a great deal of code originally written
for SPARC, I don't see where it will be depending on XFmode.  I believe
even glibc has been employed with fair success in 53-bit precision mode.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<You want to bet?


I would take that bet ... think of all the spread sheet stuff in the world,
and all the image processing independent of games. The percentage of PC's
used to play 3D games is smaller than you think. Even most home computers
are used for email etc more than games.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

by the way, I find that flush to zero is a perfectly reasonable semantics
for a C compiler, especially on machines that do this naturally. For example,
on an Alpha, I would do this by default, and only do real denormals in -mieee
mode, because otherwise the cost is indeed high, same think for the R10K.
There is certainly nothing in the C or C++ standard requiring denormals.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Vincent Penquerc'h]

Hi,

I'd like to point out that it seems right to me to allow
non-perfect results when using -ffast-math, for the simple
reason that if we get two methods for computing FP operations
and one is faster, there is no point using the slowest (unless
you happen to use -Os and this one is smaller, etc, but this
is special case). Therefore, the very existence of such a
-ffast-math option acknowledges that its results are less
perfect than the default behavior, be it on average error,
domain, etc. This has been mentionned already, but I wanted
to state it clearly.
As long as the differences and gotchas between these two
modes is well documented, I see little reason to dismiss
-ffast-math on the grounds that it is less than perfect
compared to the default behavior.
Moreover, if the drawbacks of some optimizations are too
dangerous (very reduced domain, etc), a solution could be
(yes, I realize there are too many already) to allow the
user to specify how hard one wants the compiler to try and
speed up the code using -ffast-math-<N>, as in -O<N>, or
adding --fmath-allow-domain-shrinking, or -fmath-ignore-nans
or whatever could be used by the programmer to tell the
compiler what is important to him/her and what can be
sacrificed for sped.

-- 
Lyrian 

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<As long as the differences and gotchas between these two
modes is well documented, I see little reason to dismiss
-ffast-math on the grounds that it is less than perfect
compared to the default behavior.
>>

No one is dismissing -ffast-math on any grounds, we are just discussing what
it should allow. Clearly everyone has a limit on what is reasonable, we need
to discuss to get a common agreement on reasonable limits. What we are talking
about here is just how tolerant we want to be. For example, can one tolerate
computing ** using log and exp? The result can be hundreds of ULP wrong, but
perhaps this is still tolerable IF it really saves time.

The point I keep plugging is that there is no point in providing less
accurate results unless you get some significant gain from the transformation.
For example, providing true -mieee results is really expensive on the Alpha,
so it is well worth compromising. Similarly, providing exact Java semantics
on the x86 is very expensive so it is worth compromising. 

Whether the specific transformation

 a/b/c => a/(b*c)

is worthwhile is unproven. In practice divisions are far less frequent than
multiplications and additions in any case, and the number of cases where
an expression with two divisions is written in the form a/b/c is small.
Furthermore, one would expect that since programmers obviously know that
division is in general slow, they would tend to rewrite as a/(b*c) anyway,
so perhaps it is in fact quite likely that many of the cases in which
a/b/c is written are deliberate.

Yes, Linus points to a case in which due to pipeline overlap the division
is faster, but compiler writers and systems programmers are always good
at finding theoretical cases. For me, the only justification for the
transformation above is if tests show that real programs noticably benefit.
This is a fairly stiff burden.

When I was actively working on SPITBOL, I solicited optimization suggestions
for SNOBOL-4 for the SPITBOL newsletter. I got many submissions. Then I put
into place a rule that said that you must be able to show one program that
speeds up by at least 5% (people were allowed to construct a program for the
purposes). The number of suggestions dropped to close to zero :-)

RE: What is acceptable for -ffast-math? (Was: associative law in combine)

[Vincent Penquerc'h]

> <<As long as the differences and gotchas between these two
> modes is well documented, I see little reason to dismiss
> -ffast-math on the grounds that it is less than perfect
> compared to the default behavior.
> >>
>
> No one is dismissing -ffast-math on any grounds, we are just
> discussing what
> it should allow. Clearly everyone has a limit on what is
> reasonable, we need

I was mostly referring to the following statement, which,
while true, is referring to the current -ffast-math behavior
as "broken", whereas it can be seen as "not perfect but still
acceptable" when optimizing:

> | Note that with the current -ffast-math on x86, for example, gcc will
> | already inline things like "sin()" etc using the x87 instruction
directly.
>
> I don't buy the argument
>
>   That is broken, then we should break this as well
>
> If GCC behaves incorrectly in some cases, the appropriate action to
> take is to make it behave correctly, -not- to break other parts.

Hence the second part of my post about different flags to tell
the compiler what is acceptable and what is not.

> The point I keep plugging is that there is no point in providing less
> accurate results unless you get some significant gain from the
> transformation.
> For example, providing true -mieee results is really expensive on
> the Alpha,
> so it is well worth compromising. Similarly, providing exact Java
> semantics
> on the x86 is very expensive so it is worth compromising.

Such flags would be well suited to cover different optimizations
with different architectures (as the -O flags already trigger
different optimizations depending on the architecture).

--
Lyrian

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Eric W. Biederman]

dewar@gnat.com writes:

> <<As long as the differences and gotchas between these two
> modes is well documented, I see little reason to dismiss
> -ffast-math on the grounds that it is less than perfect
> compared to the default behavior.
> >>
> 
> No one is dismissing -ffast-math on any grounds, we are just discussing what
> it should allow. Clearly everyone has a limit on what is reasonable, we need
> to discuss to get a common agreement on reasonable limits. What we are talking
> about here is just how tolerant we want to be. For example, can one tolerate
> computing ** using log and exp? The result can be hundreds of ULP wrong, but
> perhaps this is still tolerable IF it really saves time.

For reasonable limits I suggest that we look at the accuracy lost with
the traditional tools.  And say if an optimization or set of
optimizations result into too much lost accuracy the are unacceptable.

In particular there is a large body of literature that delves into
calculating exactly how much error any given floating point
operation introduces into a calculation.  That is why -mieee is useful
for some poeople.  

So since we can calculate the additional error term introduced by a
given optimization or set of optimizations we should use that when
deciding if that code change is allowable.  Then we only have to 
see if it the code change is actually an  optimization.

Eric

RE: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<Such flags would be well suited to cover different optimizations
with different architectures (as the -O flags already trigger
different optimizations depending on the architecture).
>>

The trouble with proliferation of flags is that it gets complicated for
people to even learn about them, let alone figure out what they do or
do not want.

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

The transformations in question aren't documented.  So, those having
"valid reasons to request them" don't even know what there are
requesting with -fdubious-math.

Right now, we're discussing the ability of the compiler to transform
something like a/b/c into a/(b*c).  Some argue on the ground of speed
considerations, that the programmer would expect the compiler to do
that. But then, if they would have written that form in the first 
place.

The issue isn't about "freedom vs. no-freedom".

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Wolfgang Bangerth]

> The transformations in question aren't documented.  So, those having
> "valid reasons to request them" don't even know what there are
> requesting with -fdubious-math.

Well, but then we're talking about a/b/c -> a/(b*c)  "and the like", as
  a*c+b*c -> (a+b)*c
  a*x/a   -> x
and probably a whole lot of other things that are equally "dubious".


> But then, if they would have written that form in the first place.

But, as I said, there might be cases where you can't write it in an
optimized way in the first place, such as in
  inline double f() { return a/b; }
  inline double g() { return c;   }
  x = f()/g();
plus inlining. You can't just rewrite this into a form that does the
optimized computation without the help of the compiler. However, I'd bet
that for the lower parts of the compiler the resulting stream of commands
look just the same as if it were  x=a/b/c  right away.

Regards
  Wolfgang

-------------------------------------------------------------------------
Wolfgang Bangerth          email: wolfgang.bangerth@iwr.uni-heidelberg.de
                             www: http://gaia.iwr.uni-heidelberg.de/~wolf

Re: What is acceptable for -ffast-math? (Was: associative law incombine)

[Gabriel Dos_Reis]

| and probably a whole lot of other things that are equally "dubious".
| 
| 
| > But then, if they would have written that form in the first place.
| 
| But, as I said, there might be cases where you can't write it in an
| optimized way in the first place, such as in
|   inline double f() { return a/b; }
|   inline double g() { return c;   }
|   x = f()/g();
| plus inlining. 

But then is 0.0 is acceptable for 0.125 for speed?

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Theodore Papadopoulo]

Gabriel.Dos_Reis@sophia.inria.fr said:
> | and probably a whole lot of other things that are equally "dubious".
> |  |  | > But then, if they would have written that form in the first
> place. |  | But, as I said, there might be cases where you can't write
> it in an | optimized way in the first place, such as in |   inline
> double f() { return a/b; } |   inline double g() { return c;   } |   x
> = f()/g(); | plus inlining. 

> But then is 0.0 is acceptable for 0.125 for speed? 

Yes, if you know by advance that this cannot happen because of your 
input data accuracy. Then you will get the speed advantage without 
the loss of accuracy since it cannot happen.


--------------------------------------------------------------------
Theodore Papadopoulo
Email: Theodore.Papadopoulo@sophia.inria.fr Tel: (33) 04 92 38 76 01
 --------------------------------------------------------------------


-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Exmh version 2.2 06/23/2000

iD8DBQE7aEn/IzTj8qrxOU4RAnxLAJ9fzLZKK+35iIN5MAtX6sb+OAs55QCfTbiJ
jUnXpCa+JAuS9mmYBTBb+/g=
=lUNo
-----END PGP SIGNATURE-----

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos_Reis]

| Gabriel.Dos_Reis@sophia.inria.fr said:
| > | and probably a whole lot of other things that are equally "dubious".
| > |  |  | > But then, if they would have written that form in the first
| > place. |  | But, as I said, there might be cases where you can't write
| > it in an | optimized way in the first place, such as in |   inline
| > double f() { return a/b; } |   inline double g() { return c;   } |   x
| > = f()/g(); | plus inlining. 
| 
| > But then is 0.0 is acceptable for 0.125 for speed? 
| 
| Yes, if you know by advance that this cannot happen because of your 
| input data accuracy. Then you will get the speed advantage without 
| the loss of accuracy since it cannot happen.

Data for the supposed speed in such case are still missing.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Nick Ing-Simmons]

Gabriel Dos_reis <gdosreis@sophia.inria.fr> writes:
>| Gabriel.Dos_Reis@sophia.inria.fr said:
>| > | and probably a whole lot of other things that are equally "dubious".
>| > |  |  | > But then, if they would have written that form in the first
>| > place. |  | But, as I said, there might be cases where you can't write
>| > it in an | optimized way in the first place, such as in |   inline
>| > double f() { return a/b; } |   inline double g() { return c;   } |   x
>| > = f()/g(); | plus inlining. 
>| 
>| > But then is 0.0 is acceptable for 0.125 for speed? 
>| 
>| Yes, if you know by advance that this cannot happen because of your 
>| input data accuracy. Then you will get the speed advantage without 
>| the loss of accuracy since it cannot happen.
>
>Data for the supposed speed in such case are still missing.

   Divide is very slow compared to multiply in modern hardware.
   r = a/(b*c) has one divide r = a/b/c has two.
   Divide also seldom pipelines as well.

In real-time image processing (say) failing to compute the frame in 20mS
is catastrophic, but having a pixel black (0.0) rather 
than 0.125 white (Dark gray) is of minimal impact.


>
>-- Gaby
-- 
Nick Ing-Simmons
http://www.ni-s.u-net.com/

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Per Abrahamsen]

Gabriel Dos_Reis <gdosreis@sophia.inria.fr> writes:

> But then is 0.0 is acceptable for 0.125 for speed?

Sure, whether the NO3 leaching is 0.0 g/ha/y or 0.125 g/ha/y doesn't
matter, both numbers are far below what is measurable.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<Full safety is IMHO impossible with floating point... Only the
programmer knows the accuracy of its input data (Linus called that
noise) and the accuracy he expects on its result.
>>

Now that statement really cannot be let go by.

Of course it is possible to write safe and reliable floating-point code, but
in general this is only feasible if you have a sensible floating-point model
(e.g. IEEE), and you have a well understood mapping between the high level
language and the floating-point model.

THe viewpoint that floating-point is inherently sloppy is an unhelpful
and archaic one. It is the sort of thing that gives rise to idiotic advice
like "never use absolute equality on floating-point values".

Now of course a lot of code is written in a very much less safe and reliable
manner, and indeed, one can considerably relax the model much of the time.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<We're talking here of transformations which we do know able to
drastically change the results.
>>

Words like drastically are not particularly helpful in this discussion,
plain "change" would be just fine.

THe issue is whether the "change" is worth the speed up.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Gabriel Dos_Reis]

| <<We're talking here of transformations which we do know able to
| drastically change the results.
| >>
| 
| Words like drastically are not particularly helpful in this discussion,
| plain "change" would be just fine.

Do you think changing 0.125 to 0.0 is not drastic?  I used
"drastically" to stress on the magnitude or error -- no other
conotation was implied. 

| THe issue is whether the "change" is worth the speed up.

I do fully understand that.

-- Gaby

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<Exactly ... and it isn't at all rare that the programmer knows exactly
how much noise is in the input data (little or none) and the how much
accuracy he expects in the result (as much as possible!).
>>

No, no, "as much as possible" is often quite wrong. There are stable algorithms
that become unstable with unexpected extra precision.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

a/2.0/2.0 will never yield zero if a is non-zero
a/4.0 may indeed yield zero if a is the smallest number

Indeed if one saw a/2.0/2.0 that's pretty peculiar, and it would not be
unreasonable to guess that the properly of not yielding zero was being
exploited :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

<<The above rules, btw, only apply for exact IEEE math, and on a number of
machines you _will_ see zero for the first one.
>>

Hello! it was you who restricted the discussion to the x86. If we are allowed
to go to other machines, we have lots more examples up our sleeves :-)

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

>>Your arguments are getting weaker and weaker, and dishonest to boot.


I really see no reason for ad hominem remarks, I am now killing the thread,
when things deteriorate that far, no further useful exchange is likely.

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Phil Edwards]

  To contribute, or not to contribute, that is the question
  Whether it is better in the long run
  To continue to post messages to this thread
  Which goeth round in circles, further and further off-topic.
  Or to stop posting to it
  And allow silly arguments to go unchallenged
  Or on the other hand to stop reading it altogether
  And by ignoring it, end it.

(apologies to Andrew Rilstone)

I'm just sort of wondering if there's going to be a patch posted here or on
gcc-patches, something in the way of a concrete suggestion.  Thus far there's
been some name calling, lots of heat, very little light.  Maybe someone
would care to summarize the topic (e.g., are we restricting the discussion
to a particular platform, what exactly is the source of disagreement, etc)?

If I recall my flamewar protocol correctly, the next step is ad hominem
attacks.  Those are always entertaining.


Phil
...I wrote "exactly" in a FP thread... that was a mistake...

-- 
Would I had phrases that are not known, utterances that are strange, in
new language that has not been used, free from repetition, not an utterance
which has grown stale, which men of old have spoken.
                                     - anonymous Egyptian scribe, c.1700 BC

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Carlo Wood]

When I see a thread as long as this one, I always think:
here is some inefficient "communication" going on :p.
Allow me to give my 0.02.

> > Changing 0.125 to 0.0 is a dramatic change.

Some may think that dramatic changes are acceptable.

When an optimization results in

99%   of practical cases it is faster,
0.99% of practical cases it is slower and
0.01% of the cases it results in a dramatic change;

then that might be acceptable because the number
of cases were the difference is not dramatic and also
not close to the normal FP accuracy is 0%.

A dramatic change will be noted.  Developers have to watch
for overflow and underflow already anyway.

My definition of what is unacceptable is when an optimization
causes possibly errors for example in the 32th bit, while the
expected accuracy is 64 bit...

Imho, this difference (dramatic versus subtile) is would you
tried to discuss, but failed to make clear.

-- 
Carlo Wood <carlo@alinoe.com>

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[Vincent Penquerc'h]

May I suggest breaking this into two separate topics, in an
attempt to avoid attribution of some arguments to the wrong
category.
I think everyone is in agreement that -ffast-math should
not be enabled by default by -O. So I propose:

1: Should -ffast-math include transformations that can
   under certain circumstances yield substantially different
   results than the original code ?

2: What transformations should be allowed when -ffast-math
   is *not* specified (eg the a/b/c example).

-- 
Lyrian 

Re: What is acceptable for -ffast-math? (Was: associative law in combine)

[dewar]

Stephen says

> I'm writing agricultural simulations[1], and I'd be happy to get
> transformations enabled with -ffast-math that would be safe with real
> numbers, but not with floating point numbers.

Well that's at least a coherent position, but I am afraid that no one would
really accept this criterion. for example, this would allow replacing:

  (a*b) / (c*d)

by

  (a * b * 10.0**1000) / (c * d * 10.0**10000)

and introduce overflow where none was expected, and if you think such
a transformation is just obviously unreasonable consider:

   x := b * 10.0**1000;
   y := d * 10.0**1000;

   now compute (a*b) / (c*d)

and to improve efficiency transform this to (a*x)/(c*y), since x and y
happen to be available in registers

Now you might be inclined to say, OK, "transformations that would be safe
with real numbers providing they do not introduce overflow".

But you don't mean that, because many of the transformations we have
discussed introduce overflow in "marginal" cases, and now we are back in
undefined territory.