Re: GCC options and floating-point correctness (benchmarks)

Re: GCC options and floating-point correctness (benchmarks)

[Joost VandeVondele]

Using the fortran version of the paranoia test :
http://www.netlib.org/paranoia/

xlf -qfloat=nomaf:rndsngl -qstrict -O3 dpara.f
 No failures, defects nor flaws have been discovered.
 Rounding appears to conform to the proposed IEEE standard  P754
 The arithmetic diagnosed appears to be Excellent!
 End of Test.

f95 -ieee=full dpara.f
 The number of  FLAWs  discovered =              1
 The arithmetic diagnosed seems Satisfactory though flawed.
 End of Test.
(flaw -0.0**0 -> NaN)

Joost

Re: GCC options and floating-point correctness (benchmarks)

[Richard Kenner]

    I would bet that Intel is *very* familiar with Paranoia. I would have
    assumed that gcc developers would also be very familiar  with this
    test, but surprisingly, this seems incorrect!

GCC developers certainly *used* to be familiar with Paranoia ...

GCC options and floating-point correctness (benchmarks)

[Scott Robert Ladd]

Hello,

One of the most prominent numerical programmers, Dr. William Kahan, 
developed the PARANOIA benchmark. While it doesn't cover every aspect of 
numerical "correctness", it does provide an interesting benchmakr for 
comparing the effects of different compiler options.

Among numerical programmers, Mr. Kahan holds a reputation similar to 
that of Donald Knuth.

I produced the following results for a C version of PARANOIA on a 2.8GHz 
Pentium 4 (Northwood core), HT enabled, using a very recent build of the 
tree-ssa branch. I'll be glad to send copies of the benchmark source to 
anyone who requests it privately.

Conclusions and observations:

The best results (and the only potentially acceptable one in Mr. Kahan's 
opinion) required the use of the -march=pentium4 and -mfpmath=sse options.

VERY IMPORTANT: The -ffast-math switch *DOES NOT* introduce failures and 
defects *UNLESS COMBINED* with one of -O1, -O2, or -O3. And yes, 
-ffast-math does alter code generation even if no optimization is specified.

In fact, the use of any optimizations (-O1, -O2, or -O3) introduces 
numerous failures and defects (with and without -ffast-math), *unless* 
those switches are combined with -march=pentium4 and -mfpmath=sse. It is 
the -O options that should be our concern, *NOT* -ffast-math.

Adding -ffast-math to -O3 (without any other code generation options) 
actually *reduces* the number of failures over a simple -O3.

gcc -o paranoiag -lm paranoia.c

(no failures!)
The number of  SERIOUS DEFECTs  discovered = 1.
The number of  DEFECTs  discovered =         1.
The number of  FLAWs  discovered =           1.

gcc -o paranoiag -lm -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -O1 -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -O2 -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -O3 -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -ffast-math -march=pentium4 -mfpmath=sse paranoia.c

(no failures)
(no serious defects)
The number of  DEFECTs  discovered =         1.
(no flaws)

gcc -o paranoiag -lm -O1 paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         5.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O2 paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         5.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 paranoia.c

The number of  FAILUREs  encountered =       4.
The number of  SERIOUS DEFECTs  discovered = 4.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 -ffast-math paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 4.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 -ffast-math -march=pentium4 paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 -ffast-math -march=pentium4 -mfpmath=sse paranoia.c

The number of  SERIOUS DEFECTs  discovered = 1.
The number of  DEFECTs  discovered =         1.
The number of  FLAWs  discovered =           2.

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: GCC options and floating-point correctness (benchmarks)

[Joseph S. Myers]

On Thu, 25 Mar 2004, Scott Robert Ladd wrote:

> The best results (and the only potentially acceptable one in Mr. Kahan's 
> opinion) required the use of the -march=pentium4 and -mfpmath=sse options.

What if -ffloat-store is used in alternative to those?  It doesn't
actually work as far as causing conformance (e.g., casts to same type
still get discarded rather than reducing precision), but it is supposed to
help.

-- 
Joseph S. Myers
jsm@polyomino.org.uk

Re: GCC options and floating-point correctness (benchmarks)

[Scott Robert Ladd]

Joseph S. Myers wrote:
> What if -ffloat-store is used in alternative to those?  It doesn't
> actually work as far as causing conformance (e.g., casts to same type
> still get discarded rather than reducing precision), but it is supposed to
> help.

-ffloat-store and -mieee-fp don't change the results.

I've delayed the new release of Acovea, my evolutionary compiler option 
analysis tool, (http://www.coyotegulch.com/acovea) until early next 
week, so that I can add a variant of Kahan's benchmark and test how 
options affect the number of failures, defects, and flaws.

The new Acovea uses XML confirugation files for compiler definitions, as 
opposed to the previous versions that used hard-coded C++ classes for 
that purpose.

..Scott

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: GCC options and floating-point correctness (benchmarks)

[Dan Nicolaescu]

Scott Robert Ladd <coyote@coyotegulch.com> writes:

  > Joseph S. Myers wrote:
  >> What if -ffloat-store is used in alternative to those?  It doesn't
  >> actually work as far as causing conformance (e.g., casts to same type
  >> still get discarded rather than reducing precision), but it is supposed to
  >> help.
  >
  > -ffloat-store and -mieee-fp don't change the results.

This is contradicted by my results. 

From /proc/cpuinfo
model name      : Intel(R) XEON(TM) CPU 2.00GHz

gcc is gcc-3.2

wget http://www.netlib.org/paranoia/paranoia.c
gcc -O2 -ffloat-store paranoia.c -lm 
./a.out

The number of  DEFECTs  discovered =         1.
The number of  FLAWs  discovered =           1.

Without -ffloat-store: 

gcc -O2 paranoia.c -lm 

The number of  FAILUREs  encountered =       2.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         4.
The number of  FLAWs  discovered =           2.

With -mieee-fp:

gcc -O2 -mieee-fp paranoia.c -lm 

The number of  FAILUREs  encountered =       2.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         4.
The number of  FLAWs  discovered =           2.


Now, how about ICC-8.0? 

icc -O2 paranoia.c

The number of  FAILUREs  encountered =       2.
The number of  SERIOUS DEFECTs  discovered = 2.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           1.

icc -march=pentium4 paranoia.c
paranoia.c(416) : (col. 2) remark: BLOCK WAS VECTORIZED.
[Note that vectorization was performed even though no optimization
flag was passed]

The number of  FLAWs  discovered =           1.
[I haven't checked if this result was due to some code being
optimized away at compile time. If anyone wants to check I can
provide the assembly.]


icc -march=pentiumiii paranoia.c
paranoia.c(416) : (col. 2) remark: BLOCK WAS VECTORIZED.

The number of  FAILUREs  encountered =       2.
The number of  SERIOUS DEFECTs  discovered = 4.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           1.

icc -march=pentiumiii -O2 -mp1 paranoia.c
paranoia.c(416) : (col. 2) remark: BLOCK WAS VECTORIZED.

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           1.

The results for the Portland Group compiler version 5.0-2

pgcc -O2 paranoia.c 

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 2.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

pgcc paranoia.c 
... the test hangs ... 


On sparc-sun-solaris2.7 all the tests pass with both gcc and cc.


Draw your own conclusions from this data. 

Please don't CC me on replies for this thread, I'll read them from
the list.

Re: GCC options and floating-point correctness (benchmarks)

[Scott Robert Ladd]

Dan Nicolaescu wrote:
> Scott Robert Ladd <coyote@coyotegulch.com> writes:
> 
>   > Joseph S. Myers wrote:
>   >> What if -ffloat-store is used in alternative to those?  It doesn't
>   >> actually work as far as causing conformance (e.g., casts to same type
>   >> still get discarded rather than reducing precision), but it is supposed to
>   >> help.
>   >
>   > -ffloat-store and -mieee-fp don't change the results.
> 
> This is contradicted by my results. 

Okey-dokey, as we say here in the wild, wild west.

 From my proc/cpuingo, I get:

vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Pentium 4 (Northwood)
stepping        : 7
cpu MHz         : 2785.075
cache size      : 512 KB

The following two command lines produce the exact same result from 
paranoia (1 defect):

   gcc -o paranoiag -ffloat-store -march=pentium4 -ffast-math \
               -mfpmath=sse -lm paranoia.c

   gcc -o paranoiag -march=pentium4 -ffast-math -mfpmath=sse \
		-lm paranoia.c

In other words, -ffloat-store made no difference -- and using the either 
command above produces a better (fewer problems) result than does your:

   gcc -O2 -ffloat-store paranoia.c -lm

Note that I am running tree-ssa as my compiler; this is very different 
from gcc 3.2.

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: GCC options and floating-point correctness (benchmarks)

[Theodore Papadopoulo]

coyote@coyotegulch.com said:
> I produced the following results for a C version of PARANOIA on a
> 2.8GHz  Pentium 4 (Northwood core), HT enabled, using a very recent
> build of the  tree-ssa branch. I'll be glad to send copies of the
> benchmark source to  anyone who requests it privately. 

Do you have those for intel's compiler ??

--------------------------------------------------------------------
Theodore Papadopoulo
Email: Theodore.Papadopoulo@sophia.inria.fr Tel: (33) 04 92 38 76 01
 --------------------------------------------------------------------

Re: GCC options and floating-point correctness (benchmarks)

[Scott Robert Ladd]

Theodore Papadopoulo wrote:
> coyote@coyotegulch.com said:
> 
>>I produced the following results for a C version of PARANOIA on a
>>2.8GHz  Pentium 4 (Northwood core), HT enabled, using a very recent
>>build of the  tree-ssa branch. I'll be glad to send copies of the
>>benchmark source to  anyone who requests it privately. 
> 
> 
> Do you have those for intel's compiler ??

icc -o paranoiai -xN -tpp7 paranoia.c
icc -o paranoiai -xN -tpp7 -mp1 paranoia.c
icc -o paranoiai -xN -tpp7 -mp paranoia.c
icc -o paranoiai -xN -tpp7 -ipo paranoia.c
icc -o paranoiai -xN -tpp7 -mp1 paranoia.c
icc -o paranoiai -xN -tpp7 -mp paranoia.c
icc -o paranoiai -xN -tpp7 -O1 paranoia.c
icc -o paranoiai -xN -tpp7 -O2 paranoia.c
icc -o paranoiai -xN -tpp7 -O3 paranoia.c
icc -o paranoiai -xN -tpp7 -O1 -ipo paranoia.c
icc -o paranoiai -xN -tpp7 -O2 -ipo paranoia.c
icc -o paranoiai -xN -tpp7 -O3 -ipo paranoia.c
icc -o paranoiai -xN -tpp7 -O1 -ipo -mp1 paranoia.c
icc -o paranoiai -xN -tpp7 -O2 -ipo -mp1 paranoia.c
icc -o paranoiai -xN -tpp7 -O3 -ipo -mp1 paranoia.c
icc -o paranoiai -xN -tpp7 -O1 -ipo -mp paranoia.c
icc -o paranoiai -xN -tpp7 -O2 -ipo -mp paranoia.c
icc -o paranoiai -xN -tpp7 -O3 -ipo -mp paranoia.c

(no failures)
(no serious defects)
(no defects)
The number of  FLAWs  discovered =         1.
  - - - - - - - - - - - - - - -

Yes, I checked to see if the code changed with each combination of 
options; it did. In fact, the "flaw" is different with different options.

With -mp, the single test that failed was:

  - - - - - - - - - - - - - - -
Sticky bit used incorrectly or not at all.
FLAW:  lack(s) of guard digits or failure(s) to correctly round or chop
  - - - - - - - - - - - - - - -

Without -mp, the single flaw was:

  - - - - - - - - - - - - - - -
FLAW:  X = 3.05947655544740190e-308
         is not equal to Z = 2.22507385850720138e-308 .
yet X - Z yields 0.00000000000000000e+00 .
  - - - - - - - - - - - - - - -

Given that GCC exhibits FAILURES and DEFECTS, even under the best of 
circumstances, suggests that ICC may have a leg up when it comes to this 
benchmark.


-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: GCC options and floating-point correctness (benchmarks)

[Joe Buck]

On Thu, Mar 25, 2004 at 01:36:14PM -0500, Scott Robert Ladd wrote:
> [ paranoia results ]

> Given that GCC exhibits FAILURES and DEFECTS, even under the best of 
> circumstances, suggests that ICC may have a leg up when it comes to this 
> benchmark.

But there's a problem.  People have posted trivial programs where icc
majorly violates IEEE rules.  Could Intel be tuning specifically to work
well on paranoia, but take impermissible shortcuts elsewhere?

Re: GCC options and floating-point correctness (benchmarks)

[Scott Robert Ladd]

Joe Buck wrote:
> But there's a problem.  People have posted trivial programs where icc
> majorly violates IEEE rules.  Could Intel be tuning specifically to work
> well on paranoia, but take impermissible shortcuts elsewhere?

That's certainly possible. It's not like we haven't seen such creativity 
in other situations. However, I'd want to find some proof before making 
any accusations in this case.

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: GCC options and floating-point correctness (benchmarks)

[Joe Buck]

On Thu, Mar 25, 2004 at 02:32:58PM -0500, Scott Robert Ladd wrote:
> Joe Buck wrote:
> > But there's a problem.  People have posted trivial programs where icc
> > majorly violates IEEE rules.  Could Intel be tuning specifically to work
> > well on paranoia, but take impermissible shortcuts elsewhere?
> 
> That's certainly possible. It's not like we haven't seen such creativity 
> in other situations. However, I'd want to find some proof before making 
> any accusations in this case.

Agreed; we should be cautious before deciding that there is a problem.
Are we certain, for example, that icc's optimization of

	(a+b)+c == a+(b+c)

for double a, b, and c, is impermissibly aggressive?
 

RE: GCC options and floating-point correctness (benchmarks)

[Dave Korn]

> -----Original Message-----
> From: gcc-owner On Behalf Of Joe Buck
> Sent: 25 March 2004 19:44

> On Thu, Mar 25, 2004 at 02:32:58PM -0500, Scott Robert Ladd wrote:
> > Joe Buck wrote:
> > > But there's a problem.  People have posted trivial programs where 
> > > icc majorly violates IEEE rules.  Could Intel be tuning 
> specifically 
> > > to work well on paranoia, but take impermissible 
> shortcuts elsewhere?
> > 
> > That's certainly possible. It's not like we haven't seen such 
> > creativity in other situations. However, I'd want to find 
> some proof 
> > before making any accusations in this case.
> 
> Agreed; we should be cautious before deciding that there is a problem.
> Are we certain, for example, that icc's optimization of
> 
> 	(a+b)+c == a+(b+c)
> 
> for double a, b, and c, is impermissibly aggressive?
>  


<innocent> Why doesn't someone who has a licensed copy of ICC raise a bug
report with Intel? </innocent> 



    cheers, 
      DaveK
-- 
Can't think of a witty .sigline today....

Re: GCC options and floating-point correctness (benchmarks)

[Robert Dewar]

Joe Buck wrote:

> Agreed; we should be cautious before deciding that there is a problem.
> Are we certain, for example, that icc's optimization of
> 
> 	(a+b)+c == a+(b+c)
> 
> for double a, b, and c, is impermissibly aggressive?

In my view, yes.
>  

Re: GCC options and floating-point correctness (benchmarks)

[Joe Buck]

On Fri, Mar 26, 2004 at 07:55:56PM -0500, Robert Dewar wrote:
> Joe Buck wrote:
> 
> > Agreed; we should be cautious before deciding that there is a problem.
> > Are we certain, for example, that icc's optimization of
> > 
> > 	(a+b)+c == a+(b+c)
> > 
> > for double a, b, and c, is impermissibly aggressive?
> 
> In my view, yes.

I have not seen the assembly code that they are producing.  Evidently
they do not simply reduce it to a constant, though they take shortcuts.
So let's be sure before we accuse them.

Re: GCC options and floating-point correctness (benchmarks)

[Roger Sayle]

On Fri, 26 Mar 2004, Joe Buck wrote:
> I have not seen the assembly code that they are producing.  Evidently
> they do not simply reduce it to a constant, though they take shortcuts.
> So let's be sure before we accuse them.

int foo(double a, double b, double c)
{
  return (a+b)+c == a+(b+c);
}


; -- Machine type IA32
; mark_description "Intel(R) C++ Compiler for 32-bit applications, Version 7.1   Build 20030307Z";
; mark_description "-Qvc6 -Qlocation,link,C:\\Program Files\\Microsoft Visual Studio\\VC98\\Bin -nologo -FA -O2 -Ob2";
;ident "Intel(R) C++ Compiler for 32-bit applications, Version 7.1   Build 20030307Z"

_foo	PROC NEAR
.B1.1:                          ; Preds .B1.0
        push      ebx                                           ;2.1
        mov       ebx, esp                                      ;2.1
        and       esp, -8                                       ;2.1
        fld       QWORD PTR [ebx+8]                             ;1.5
        fadd      QWORD PTR [ebx+16]                            ;3.13
        fadd      QWORD PTR [ebx+24]                            ;3.26
        mov       edx, 1                                        ;3.26
        fcomp     st(0)                                         ;3.26
        fnstsw    ax                                            ;3.26
        xor       ecx, ecx                                      ;3.26
        sahf                                                    ;3.26
        mov       eax, edx                                      ;3.26
        je        .L1           ; Prob 50%                      ;3.26
        mov       eax, ecx                                      ;3.26
.L1:                                                            ;
        mov       esp, ebx                                      ;3.26
        pop       ebx                                           ;3.26
        ret                                                     ;3.26


Roger
--

Re: GCC options and floating-point correctness (benchmarks)

[Robert Dewar]

Joe Buck wrote:

> But there's a problem.  People have posted trivial programs where icc
> majorly violates IEEE rules.  Could Intel be tuning specifically to work
> well on paranoia

I would expect so. I would expect ANY compiler to be tested against
Paranoia.

but take impermissible shortcuts elsewhere?

We already have some examples, so this is definitely the case.

Re: GCC options and floating-point correctness (benchmarks)

[Robert Dewar]

> Given that GCC exhibits FAILURES and DEFECTS, even under the best of 
> circumstances, suggests that ICC may have a leg up when it comes to this 
> benchmark.

I would bet that Intel is *very* familiar with Paranoia. I would have
assumed that gcc developers would also be very familiar  with this
test, but surprisingly, this seems incorrect!