Re: [GCC 3.x] Performance testing for QA

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<Now of course doing things this blatantly would be considered cheating, but
in practice you can do things almost this outrageous in a much more
reserved and non-obvious way.
>>

By the way, some company made a video board for the PC that had a special
bit map built into the ROM that happened to be a string something like
"PC Video Font Benchmark Test" in some particular font, predecoded. 
PC Magazine was not amused when they discovered this :-)

Re: [GCC 3.x] Performance testing for QA

[Jan Hubicka]

> On Tue, Sep 03, 2002 at 04:25:13PM -0400, Robert Dewar wrote:
> > <<Since the benchmark gcc is never fed any erroneous source while running
> > the benchmark, none of the error-handling stuff is ever executed.  With
> > profile-based optimization it's possible to move a lot of that stuff so
> > it never gets paged in; that's a significant win.  The same conditions
> > don't apply when running gcc normally, so this is fairly artificial.
> > >>
> > 
> > Why would it be paged in if it is not executed?
> 
> If it's on the same page as something which is executed.  This is one
> of the major benefits of profile-driven optimization.
Right :)
BTW I was experimenting with patching makefiles to use GCC profile
feedback when building GCC.  The resulting compiler is about 7% faster
to build binutils then.

Honza
> 
> -- 
> Daniel Jacobowitz
> MontaVista Software                         Debian GNU/Linux Developer

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<Than you need to use a wide variety of encoders to avoid this problem.
Also since the source code for gcc is freely available such a thing will
not go unnoticed for long. I seams that you are trying to say that making
tight loops run fast is a pointless exercise by the compiler.  If a large
number of programs tend to use a particular type of loop than optimizing
for those loops would be a huge win in terms of performance.
>>

There are plenty of benchmarks in SPEC and in other standard suites that
have tight loops. So you are misinterpreting my remarks entirely.

The point is that benchmarks are designed to be good as benchmarks, the issue
of whether they perform a useful calculation that lots of people run a lot of
the time is quite besides the point. I would guess that you have not even
studied the SPEC suite closely before you made remarks about it :-)

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<and unroll it the right number of times (in some cases, by emitting
canned
assembly as you suggest).  The benchmark was removed in later SPEC's...
>>

And of course the breaking of Matrix300 is a legend in benchmarking history :-)

Re: [GCC 3.x] Performance testing for QA

[Dale Johannesen]

On Tuesday, September 3, 2002, at 01:55 PM, Robert Dewar wrote:

> <<The type of operations that encoder decoders use (lots of arithmetic 
> and
> data movement) should somehow be incorporate as a benchmark test gcc
> should optimize for.  The ideal goal would be to make the code fast 
> enough
> that hand written assembly would not be necessary.  Thus using an 
> encoder as
> a benchmark would be beneficial.  Even if it concentrates on tight 
> loops.
>>>
>
> You need to remember that it is trivial to make a compiler make any ONE
> simple program as fast as optimal assembly language.
>
> How? Simply by recognizing the program, and saying "ah ha, this is 
> program
> number 234c, so replace it by the super efficient asm that Joe wrote".
>
> Now of course doing things this blatantly would be considered 
> cheating, but
> in practice you can do things almost this outrageous in a much more
> reserved and non-obvious way.

One of the SPEC92 benchmarks, eqntott, was dominated by a single 
function
that looped through a linked data structure.  Unrolling the loop the 
right
number of times was critical, but you couldn't figure this out 
legitimately
because the right number of times depended on the data in the structure,
and that was read in from a file.  By the end-of-life of Spec92 every
commercial compiler I know of had special code to recognize this 
function
and unroll it the right number of times (in some cases, by emitting 
canned
assembly as you suggest).  The benchmark was removed in later SPEC's...

Re: [GCC 3.x] Performance testing for QA

[Kevin Atkinson]

On Tue, 3 Sep 2002, Robert Dewar wrote:

> <<The type of operations that encoder decoders use (lots of arithmetic and
> data movement) should somehow be incorporate as a benchmark test gcc
> should optimize for.  The ideal goal would be to make the code fast enough
> that hand written assembly would not be necessary.  Thus using an encoder as
> a benchmark would be beneficial.  Even if it concentrates on tight loops.
> >>
> 
> You need to remember that it is trivial to make a compiler make any ONE
> simple program as fast as optimal assembly language.

Than you need to use a wide variety of encoders to avoid this problem.  
Also since the source code for gcc is freely available such a thing will
not go unnoticed for long. I seams that you are trying to say that making
tight loops run fast is a pointless exercise by the compiler.  If a large
number of programs tend to use a particular type of loop than optimizing
for those loops would be a huge win in terms of performance.

--- 
http://kevin.atkinson.dhs.org

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<The type of operations that encoder decoders use (lots of arithmetic and
data movement) should somehow be incorporate as a benchmark test gcc
should optimize for.  The ideal goal would be to make the code fast enough
that hand written assembly would not be necessary.  Thus using an encoder as
a benchmark would be beneficial.  Even if it concentrates on tight loops.
>>

You need to remember that it is trivial to make a compiler make any ONE
simple program as fast as optimal assembly language.

How? Simply by recognizing the program, and saying "ah ha, this is program
number 234c, so replace it by the super efficient asm that Joe wrote".

Now of course doing things this blatantly would be considered cheating, but
in practice you can do things almost this outrageous in a much more 
reserved and non-obvious way.

So a benchmark that has a single inner loop that takes most of the time
is almost always a bad idea.

Re: [GCC 3.x] Performance testing for QA

[Kevin Atkinson]

On Tue, 3 Sep 2002, Robert Dewar wrote:

> <<However, encoding/decoding (especially encoding) is one area were speed
> really matters.  Encoding in real time (with out sacrificing quality) is
> still not passable with many codecs except on the newest machines.
> Unfortunately, because video work is so computationally extensive significant
> portions of the code are often rewritten is assembly to take advantage of
> MMX etc instructions that many compilers do not produce normally.  This
> means that that pure C versions may not exist or if they do they are not
> coded optimally.
> >>
> 
> This is all true.
> 
> But what does it have to do with being a good benchmark?

The type of operations that encoder decoders use (lots of arithmetic and 
data movement) should somehow be incorporate as a benchmark test gcc 
should optimize for.  The ideal goal would be to make the code fast enough 
that hand written assembly would not be necessary.  Thus using an encoder as 
a benchmark would be beneficial.  Even if it concentrates on tight loops.

--- 
http://kevin.atkinson.dhs.org

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<However, encoding/decoding (especially encoding) is one area were speed
really matters.  Encoding in real time (with out sacrificing quality) is
still not passable with many codecs except on the newest machines.
Unfortunately, because video work is so computationally extensive significant
portions of the code are often rewritten is assembly to take advantage of
MMX etc instructions that many compilers do not produce normally.  This
means that that pure C versions may not exist or if they do they are not
coded optimally.
>>

This is all true.

But what does it have to do with being a good benchmark?

Nothing as far as I can see.

Re: [GCC 3.x] Performance testing for QA

[Daniel Jacobowitz]

On Tue, Sep 03, 2002 at 04:25:13PM -0400, Robert Dewar wrote:
> <<Since the benchmark gcc is never fed any erroneous source while running
> the benchmark, none of the error-handling stuff is ever executed.  With
> profile-based optimization it's possible to move a lot of that stuff so
> it never gets paged in; that's a significant win.  The same conditions
> don't apply when running gcc normally, so this is fairly artificial.
> >>
> 
> Why would it be paged in if it is not executed?

If it's on the same page as something which is executed.  This is one
of the major benefits of profile-driven optimization.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<Since the benchmark gcc is never fed any erroneous source while running
the benchmark, none of the error-handling stuff is ever executed.  With
profile-based optimization it's possible to move a lot of that stuff so
it never gets paged in; that's a significant win.  The same conditions
don't apply when running gcc normally, so this is fairly artificial.
>>

Why would it be paged in if it is not executed?

Re: [GCC 3.x] Performance testing for QA

[Kevin Atkinson]

On Mon, 2 Sep 2002, Robert Dewar wrote:

> I disagree, gcc is a very interesting benchmark because of its locality
> patterns. It precisely is NOT a collection of tight loops, and unlike
> some of the other tests in SPEC, it is very hard to tune a compiler 
> to do artifically well on the gcc benchmark.
> 
> Of course you want a big collection of benchmarks, and actually we advise
> users of GNAT to always test on their own code rather than benchmarks,
> but GCC is probably one of the better benchmarks around. 
> The trouble with the encoder/decoder examples is that they tend
> to concentrate on specific inner loops. That means that they are
> much more open to subversion (although interestingly, GCC is probably
> the one compiler where there is NOT a lot of effort spent specifically
> on getting benchmarks to work better).

However, encoding/decoding (especially encoding) is one area were speed 
really matters.  Encoding in real time (with out sacrificing quality) is 
still not passable with many codecs except on the newest machines.  
Unfortunately, because video work is so computationally extensive significant
portions of the code are often rewritten is assembly to take advantage of 
MMX etc instructions that many compilers do not produce normally.  This 
means that that pure C versions may not exist or if they do they are not 
coded optimally.

For example my PIII 500 is fast enough for me for most of my work but when 
I need to do video work I use my family P4.

--- 
http://kevin.atkinson.dhs.org

Re: [GCC 3.x] Performance testing for QA

[Dale Johannesen]

On Monday, September 2, 2002, at 03:49 PM, Robert Dewar wrote:

> <<   I only reason I think its nice to have the compiler bootstrap as 
> part
> of the benchmark is to make sure that the compiler is fast.  Instead, I
> think it would be more interesting to see how fast the compiler could
> build all the other benchmarks -- not how fast it could build itself :)
>>>
>
> I disagree, gcc is a very interesting benchmark because of its locality
> patterns. It precisely is NOT a collection of tight loops, and unlike
> some of the other tests in SPEC, it is very hard to tune a compiler
> to do artifically well on the gcc benchmark.

Since the benchmark gcc is never fed any erroneous source while running
the benchmark, none of the error-handling stuff is ever executed.  With
profile-based optimization it's possible to move a lot of that stuff so
it never gets paged in; that's a significant win.  The same conditions
don't apply when running gcc normally, so this is fairly artificial.

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<   (It's also great to have a bunch of scientific code as benchmarks,
but code that Joe Bob might use should make an appearance too).
>>

There is of course the Livermoor Loops benchmark in this area ...

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

<<   I only reason I think its nice to have the compiler bootstrap as part
of the benchmark is to make sure that the compiler is fast.  Instead, I
think it would be more interesting to see how fast the compiler could
build all the other benchmarks -- not how fast it could build itself :)
>>

I disagree, gcc is a very interesting benchmark because of its locality
patterns. It precisely is NOT a collection of tight loops, and unlike
some of the other tests in SPEC, it is very hard to tune a compiler 
to do artifically well on the gcc benchmark.

Of course you want a big collection of benchmarks, and actually we advise
users of GNAT to always test on their own code rather than benchmarks,
but GCC is probably one of the better benchmarks around. 
The trouble with the encoder/decoder examples is that they tend
to concentrate on specific inner loops. That means that they are
much more open to subversion (although interestingly, GCC is probably
the one compiler where there is NOT a lot of effort spent specifically
on getting benchmarks to work better).

Re: [GCC 3.x] Performance testing for QA

[Timothy J. Wood]

On Monday, September 2, 2002, at 02:47  PM, Robert Dewar wrote:

>> Dear GCC people,
>>
>> Allow me the humble question to ask you if it would possible to
>> employ a simple yet efficient testing method from real life as
>> opposed to the SPEC* (artificial) test?
>
>
> Interesting that someone should think that the gcc compiler is an
> artificial test (it is one of the SPEC tests :-)

   This doesn't seem odd at all.  A vanishingly small number of people 
actually build the compiler itself.  It makes sense to use code that 
normal people actually use every day as benchmarks, so things like 
MP3/Ogg encoders/decoders, DivX encoders/decoders, and possibly 
rendering glyphs with FreeType could provide another set of useful 
benchmarks.  Clearly a bunch of work would need to be done to isolate 
the effects of the compiler from the effects of improvements in the 
algorithms and underlying OS, but more people actually use these 
applications than the compiler itself.

   (It's also great to have a bunch of scientific code as benchmarks, 
but code that Joe Bob might use should make an appearance too).

   I only reason I think its nice to have the compiler bootstrap as part 
of the benchmark is to make sure that the compiler is fast.  Instead, I 
think it would be more interesting to see how fast the compiler could 
build all the other benchmarks -- not how fast it could build itself :)

-tim

Re: [GCC 3.x] Performance testing for QA

[Robert Dewar]

> Dear GCC people,
>
> Allow me the humble question to ask you if it would possible to
> employ a simple yet efficient testing method from real life as
> opposed to the SPEC* (artificial) test?


Interesting that someone should think that the gcc compiler is an
artificial test (it is one of the SPEC tests :-)

Re: [GCC 3.x] Performance testing for QA

[Andreas Jaeger]

I've trimmed the CC list, no need to cross-post between GCC and
mplayer lists.

Sasi Péter <Peter.Sasi@t-systems.co.hu> writes:

> Dear GCC people,
>
> Allow me the humble question to ask you if it would possible to
> employ a simple yet efficient testing method from real life as
> opposed to the SPEC* (artificial) test?

SPEC* are not really artificial benchmarks, those are real life
programs that have been changed into tests.

> Specifically I was thinking of running a few encoding/decoding/manipulation runs with mplayer on sample multimedia streams.
>
> Main reasons I could think of are:
> - Easy to test: all you need (sources, sample files) is available free at www.mplayerhq.hu (for Linux(intel, alpha), Solaris, Irix, etc.)
> - Important for masses: it is one of the most widely used free software in the area gaining probably the most momentum nowadays (amongst performace hungry software)
> - There is a lot of space for improvements: even the 2.95 series perform better than the ongoing 3.x series, but if going for the long term Intel's compiler currently wins hands down
>
> I know it is not a very scientific measurement, still it does apply to a lot of people, andd is a reason why a lot of people still stick to the older versions.
> New features are really an important thing to have, still they should not make it instead of the good performance, but exploiting it.
>
> Summarizing the above:
> - Do you think such measurements are feasible/worth and can supply usable input for the development?

They can.

> - Do you lack the interest, the resources, or what else?

Resources and volunteers looking at the tests.

> - Ifyou are not interested, is there a reasoning behind that? (point me to RTFM if applicable)
> - In case of positive answers do you think profiling, code analysis or what else could be the way to improve the current situation? (get 3.x to the perf. level of 2.95.x)

It might be.  But it's not something you can just "give" us.  If you
set up some automatic tests - like I do with my SPEC tests -, you
should make the data available to developers and work with them on
performance issues.  Bug reports for specific problems or just: Your
patch causes a regression are helpful.

Note that GCC is a volunteer effort.  If you're not doing this, I fear
that nobody does it.

>
> Please CC my personal address in your reply, since I am not subscribed to any of your mailing lists.
>
> Thank you for your precious time.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

[GCC 3.x] Performance testing for QA

[Sasi Péter]

Dear GCC people,

Allow me the humble question to ask you if it would possible to employ a simple yet efficient testing method from real life as opposed to the SPEC* (artificial) test?
Specifically I was thinking of running a few encoding/decoding/manipulation runs with mplayer on sample multimedia streams.

Main reasons I could think of are:
- Easy to test: all you need (sources, sample files) is available free at www.mplayerhq.hu (for Linux(intel, alpha), Solaris, Irix, etc.)
- Important for masses: it is one of the most widely used free software in the area gaining probably the most momentum nowadays (amongst performace hungry software)
- There is a lot of space for improvements: even the 2.95 series perform better than the ongoing 3.x series, but if going for the long term Intel's compiler currently wins hands down

I know it is not a very scientific measurement, still it does apply to a lot of people, andd is a reason why a lot of people still stick to the older versions.
New features are really an important thing to have, still they should not make it instead of the good performance, but exploiting it.

Summarizing the above:
- Do you think such measurements are feasible/worth and can supply usable input for the development?
- Do you lack the interest, the resources, or what else?
- Ifyou are not interested, is there a reasoning behind that? (point me to RTFM if applicable)
- In case of positive answers do you think profiling, code analysis or what else could be the way to improve the current situation? (get 3.x to the perf. level of 2.95.x)

Please CC my personal address in your reply, since I am not subscribed to any of your mailing lists.

Thank you for your precious time.

Best regards,

Peter
-- Sasi Péter                   E-mail:  peter.sasi@t-systems.co.hu 
   Sr. Systems Engineer       Internet: <http://www.t-systems.co.hu/> 
   T-Systems Unisoftware Ltd.              Phone:  +36 (1)  456 54 13 
   Hungary                                 Fax:    +36 (1)  456 54 99 
Postal address: H-1453 Budapest 92, Pf.28. Mobile: +36 (30) 280 34 31 
Address: H-1097 Budapest, Könyves Kálmán krt. 12-14. 3.em., Lurdy-Ház