povray: a benchmark with mainline, tree-ssa, and icc

povray: a benchmark with mainline, tree-ssa, and icc

[Scott Robert Ladd]

Hello.

Given some questions posed in the "Contributing tree-ssa to mainline" 
thread, I decided to run a little benchmark.

The target of the benchmark is povray, a well-known ray-tracing program 
that is also getting some heavy use in my household at the moment. I 
pulled fresh source code for version 3.50c from their web site at 
http://www.povray.org. I used the official povray "benchmark.pov" with 
the settings in "benchmark.ini".

I ran these tests on a 2.8GHz Pentium 4 (Northwood) with 512MB RAM, 
under Linux 2.6.0, using mainline and tree-ssa code pulled form CVS mere 
hours ago. I also included results for Intel C++ 8.0, as a matter of 
comparison.

All times are in minutes:seconds, as reported by povray's internal 
timer. Switches used were:

gcc: -O3 -march=pentium4 --finline-functions
      -ffast-math -fomit-frame-pointer -funroll-loops
      -fexpensive-optimizations -malign-double
      -foptimize-sibling-calls -minline-all-stringops
      -Wno-multichar

icc: -O3 -xN -tpp7 -ipo

The GCC switches are the default defined by povray, with the exception 
of my changing the -march switch to specify the Pentium 4. I chose the 
ICC switches myself.

A drum-roll, please.

                compile  benchmark
                 time      time
               --------  ---------
gcc mainline    1:43      7:59
gcc tree-ssa    2:40      7:24
icc 8.0         1:53      5:50


I don't find these numbers comforting.

Tree-ssa is substantially slower at compiling than mainline, although it 
does produce slightly faster code.

The ICC numbers speak for themselves. Yes, I verified the output as correct.

CAVEAT: This is only *one* platform, albeit a popular one, and only one 
benchmark. I'm going to run a full set of tests in the next few days, 
all on real world code.

Is povray a good benchmark? I think so, at least in terms of the kind of 
work I do, which is mostly number crunching.

Some anecdotal observations: On a DSP application for one of my clients, 
ICC is consistently 20% faster than GCC; tree-ssa offers a 5% 
improvement over mainline. Alas, the application is under NDA, so I 
can't provide more details. I do know that, with a very few exceptions, 
the povray numbers above reflect the relative performance of these 
compilers on my projects.

I realize some people do not see ICC as a valid competitor... however, I 
don't see why GCC can't at least approach ICC's level of performance.

At this point, tree-ssa is a marginal improvement over mainline in code 
quality, and a significant regression in terms of compile speed. Those 
who know tree-ssa may have some insight into whether or not more 
optimization can be squeezed out while also improving compile time.

..Scott

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Andrew Pinski]

On Jan 17, 2004, at 12:01, Scott Robert Ladd wrote:
> A drum-roll, please.
>
>                compile  benchmark
>                 time      time
>               --------  ---------
> gcc mainline    1:43      7:59
> gcc tree-ssa    2:40      7:24
> icc 8.0         1:53      5:50
>
>
> I don't find these numbers comforting.
>
> Tree-ssa is substantially slower at compiling than mainline, although 
> it does produce slightly faster code.


Well you did not do a fare comparison for compile time at all unless 
you configured without checking turned on.  The tree-ssa does more 
checking than what the mainline or ICC does.


Thanks,
Andrew Pinski

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Daniel Berlin]

On Jan 17, 2004, at 3:01 PM, Scott Robert Ladd wrote:

> Hello.
>
> Given some questions posed in the "Contributing tree-ssa to mainline" 
> thread, I decided to run a little benchmark.
>
> The target of the benchmark is povray, a well-known ray-tracing 
> program that is also getting some heavy use in my household at the 
> moment. I pulled fresh source code for version 3.50c from their web 
> site at http://www.povray.org. I used the official povray 
> "benchmark.pov" with the settings in "benchmark.ini".
>
> I ran these tests on a 2.8GHz Pentium 4 (Northwood) with 512MB RAM, 
> under Linux 2.6.0, using mainline and tree-ssa code pulled form CVS 
> mere hours ago. I also included results for Intel C++ 8.0, as a matter 
> of comparison.

>
> All times are in minutes:seconds, as reported by povray's internal 
> timer. Switches used were:
>
> gcc: -O3 -march=pentium4 --finline-functions
>      -ffast-math -fomit-frame-pointer -funroll-loops
>      -fexpensive-optimizations -malign-double
>      -foptimize-sibling-calls -minline-all-stringops
>      -Wno-multichar
>
> icc: -O3 -xN -tpp7 -ipo
>
> The GCC switches are the default defined by povray, with the exception 
> of my changing the -march switch to specify the Pentium 4. I chose the 
> ICC switches myself.
>
> A drum-roll, please.
>
>                compile  benchmark
>                 time      time
>               --------  ---------
> gcc mainline    1:43      7:59
> gcc tree-ssa    2:40      7:24

This is a pretty surprising number.
Are you sure you disabled checking?
tree-ssa does a lot more checking than the mainline does, so with 
checking on, it will be significantly slower in some cases.
>

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Daniel Berlin]

>>                compile  benchmark
>>                 time      time
>>               --------  ---------
>> gcc mainline    1:43      7:59
>> gcc tree-ssa    2:40      7:24
>
> This is a pretty surprising number.
> Are you sure you disabled checking?
> tree-ssa does a lot more checking than the mainline does, so with 
> checking on, it will be significantly slower in some cases.

I should also note that the benchmark time is ~6% *faster*.
6% is not insignificant.

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Scott Robert Ladd]

Andrew Pinski wrote:
> Well you did not do a fare comparison for compile time at all unless you 
> configured without checking turned on.  The tree-ssa does more checking 
> than what the mainline or ICC does.

Then please illuminate me as to the proper option to turn off this extra 
checking.


-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Zack Winkles]

Scott Robert Ladd <coyote@coyotegulch.com> said:
> Andrew Pinski wrote:
> > Well you did not do a fare comparison for compile time at all unless
> > you configured without checking turned on.  The tree-ssa does more
> > checking than what the mainline or ICC does.
> 
> Then please illuminate me as to the proper option to turn off this
> extra checking.

--disable-checking

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Jan Hubicka]

> Andrew Pinski wrote:
> >Well you did not do a fare comparison for compile time at all unless you 
> >configured without checking turned on.  The tree-ssa does more checking 
> >than what the mainline or ICC does.
> 
> Then please illuminate me as to the proper option to turn off this extra 
> checking.

You need to recompile compiler with --disable-checking passed to
configure script.

Honza
> 
> 
> -- 
> Scott Robert Ladd
> Coyote Gulch Productions (http://www.coyotegulch.com)
> Software Invention for High-Performance Computing
> 

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Scott Robert Ladd]

Daniel Berlin wrote:
SRL>                compile  benchmark
SRL>                 time      time
SRL>               --------  ---------
SRL> gcc mainline    1:43      7:59
SRL> gcc tree-ssa    2:40      7:24

> I should also note that the benchmark time is ~6% *faster*.
> 6% is not insignificant.

I pointed this out in my original post: In general, tree-ssa produces 
code that is 5% faster than does mainline. And I certainly consider that 
to be a valid benefit of tree-ssa.


-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing

Re: povray: a benchmark with mainline, tree-ssa, and icc

[Diego Novillo]

On Sat, 2004-01-17 at 15:17, Scott Robert Ladd wrote:
> Andrew Pinski wrote:
> > Well you did not do a fare comparison for compile time at all unless you 
> > configured without checking turned on.  The tree-ssa does more checking 
> > than what the mainline or ICC does.
> 
> Then please illuminate me as to the proper option to turn off this extra 
> checking.
> 
Scott, starting tomorrow the yum repository with x86 tree-ssa binaries
will be built with checking disabled (see the project's page at
http://gcc.gnu.org/projects/tree-ssa/).

To build your own, use --disable-checking when configuring the compiler.


Diego.