Benchmark suite...

[Jan Hubicka <hubicka@atrey.karlin.mff.cuni.cz>]

Hi
After a week spent by tunning my patch I decided that I really need some testsuite.
It is quite time consuming to test changes at real programs, since lots of thinks
needs to be compiled. I would like to have some testsuite to verify my changes.
So I've put together few programs and made a script to compile/test them.

Because others seemsed to like benchmark suite idea too, I've decided to make something
more definitive, wich should possibly become part of egcs, if it turns out to be usefull.
I've made benchmarking library that uses interupt to stop test, so you might change time
of test as you wish and benchmarking is quite accurate since speed of system calls is not
significant. Also I've made script able to automatically run tests at various data types
(long long, long, short and char), display results and compare them immediately with
saved files. So you might see the changes before test is finished. It is also able to
run specified tests, so when you see some problem in one test, you don't need to run whole
suite etc...

My goal is not to make comparsion benchmark for various compilers, so I am not doing any overall
results. It is targeted only for developpers who want to see behaviour of their changes.
Thats why I want to make it consisting mainly of simple tests, where you may browse the assembly
result and see whats is going wrong and so on.

So far I've implemented just very few tests mostly grabbed from my programs - Mset calcualtion
loop, quicksort and recursive hanoi. Jeffrey pointed out that complex tests are also neccesary
so I've added also mathing algorithm from XaoS and palette approximation from Allegro.
I would love to have more tests. Both simple loops wich needs some tricks to optimize
and complex code (like complression algorithm or whatever else)

Current version is avalable at my ftp site ftp://ftp.ta.jcu.cz/pub/linux/hubicka/experimental/egcs-bench.tgz

I have only gcc2.7.2.3 and egcc here, but results seems to be very interesting. I will
send comparsions of current snapshot tommorow.

So here is place for yours ideas :) I am open to all suggestions....


PS: here is comparsion of gcc2.7.2 and egcs1.0 at Pentium/120. For every test
the number of loops per socond is printed and comparsion to gcc (100%=no change,
50% = twice as slow, 200% = twice as fast).
First line is signed types, second unsigned. The types are sorted by size, so
for integer test it is:
long long, long, short, char
and for fp it is long double,double,char.

Benchmarks Tue Oct 20 14:36:33 MET DST 1998
egcc -O2 -pipe

Integer tests
Hanoi (tests/hanoi.c)
      836.6337 101%     1270.2970  96%      854.0000 107%     1036.0000 106%
      827.0000 100%     1265.0000  96%      905.0000 110%     1204.9020 116%
Mandelbrot set calculation loop (tests/mset.c)
   764000.0000  92%  2093137.2549  90%  1726732.6733 111%  1666000.0000  99%
   767647.0588  92%  2100990.0990  91%  1826732.6733 116%  1902970.2970 113%
Unrolled mandelbrot set calculation loop (tests/umset.c)
   864000.0000  95%  2406930.6931  91%  2190099.0099 116%  2080198.0198 102%
   856435.6436  96%  2428431.3725  91%  2244554.4554 119%  2310891.0891 113%
Quicksort (tests/qsort.c)
       37.8641  84%       96.0396 128%       67.3267  89%      109.9010 102%
       38.2353  84%       95.0495 125%       67.3267  89%      110.8911  99%
Bressemham line drawing algorithm (tests/bressemham.c)
  6554613.8614 100%  8417720.0000 103%  7112316.8317 101%  8328500.0000  99%
  6642680.0000 102%  8444198.0198 103%  7191702.9703 103%  8421333.3333 100%
Palette approximation (tests/pal.c)
  -------.---- ---%        7.2072 112%  -------.---- ---%  -------.---- ---%
  -------.---- ---%  -------.---- ---%  -------.---- ---%  -------.---- ---%
XaoS internal loop (tests/xaos.c)
  -------.---- ---%      449.0196  91%  -------.---- ---%  -------.---- ---%
  -------.---- ---%  -------.---- ---%  -------.---- ---%  -------.---- ---%

Floating point tests
Mandelbrot set calculation loop (tests/mset.c)
  2853465.3465 107%  2888118.8119 109%  2903960.3960 109%
Unrolled mandelbrot set calculation loop (tests/umset.c)
  6158415.8416 136%  6052475.2475 134%  5890196.0784 131%
Quicksort (tests/qsort.c)
       38.6139  77%       32.6733  66%       28.1553  78%

Complex type tests
Mandelbrot set calculation loop using complex numbers (tests/cmset.c)
  2947000.0000  88%  3051485.1485  92%  2975247.5248  89%
Unrolled mandelbrot set calculation loop using complex numbers (tests/cumset.c)
  5826732.6733  97%  5884158.4158  99%  5399009.9010  91%


Honza