Re: Benchmark recommendations needed

[andras@tantosonline.com]

On 2022-02-22 16:59, Patrick McGehearty wrote:
> I studied Dhrystone about 30 years ago and found it
> had a number of flaws back then. For example, most of
> the loops in the code are only executed 1-3 times, which
> minimizes the value of hoisting values out of inner loops.
> Read the Dhrystone wikipedia article for more information.
> 
> Going back to what benchmarks might be useful...
> 
> you might consider the Livermore Loops
> http://www.netlib.org/benchmark/livermorec
> 
> These are 24 kernels (tight loops) originally in Fortran but
> ported to C 30 years ago. They are reasonably representative
> of floating point computational kernels. They are available
> without a fee.
> 
> Even if you have no interest in floating point computation for
> your target architecture, examining the assembly output of these
> kernels will be helpful in finding where your port of gcc
> is doing well and where the machine architecture input to the
> various optimizer phases need some tuning.
> 
> You also might review that code and write some modest
> test loops of your own for integer code patterns.
> 
> Developing good benchmarks is a skill which requires the
> developer to know the intended purpose of the benchmark.
> I.e. is our goal to compare optimizer implementations?
> or different architectures (i.e. arm vs x86)?
> or different implementations of an architecture
>    (i.e. intel vs amd or early x86 vs current x86)
> or ...
> well, you get the idea.
> 
> Good luck,
> 
> - Patrick McGehearty
> 
> 
> 
> On 2/22/2022 3:49 PM, Paul Koning via Gcc wrote:
>> 
>>> On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> 
>>> wrote:
>>> 
>>> Andras,
>>> 
>>> The whole point of benchmarks is to judge a processor's performance.
>>> That being said, just crippling GCC is not reasonable because
>>> processors must be judged in the appropriate context and that
>>> includes the current state of the art compiler technology. If you 
>>> have
>>> a new processor I'd benchmark it using the applications you built it
>>> for.
>> Exactly.  Part of what you want to see is that GCC optimizes well for 
>> the new machine, i.e., that there aren't artifacts of the machine 
>> description that get in the way of optimization.
>> 
>> So you'd want to use applications that are good exercises not just of 
>> the code generator but also the optimizer.  Dhrystone isn't really 
>> that, because it has evolved into mostly an optimizer test, not a 
>> machine or code generator test.
>> 
>> 	paul
>> 

Thank you for all the feedback. When I said 'crippling' GCC, I didn't 
mean to make it generally worse, just to make sure I don't trigger 
dhrystone-specific optimizations, if there are any.

Your point about comparing processors is a fair one and that's where I'm 
heading, but I need to start small. Even just comparing object file 
(.text section) size between architectures have been hugely 
enlightening.

There is another point though: my port of GCC is certainly not state of 
the art (well, it technically is, because that's the only port in 
existence, but you get what I mean). Part of the process is to hone in 
the port in terms of tuning the instruction selection, peepholes, etc. 
In some sense this work benchmarks compilers not processors. At any 
rate, I do get that dhrystone is a poor substitute for a real benchmark 
and I'll try to move away from it.

I wasn't aware of Livermore loops being ported to C (and I don't yet 
have a Fortran port), so that's a great suggestion. I'll check it out.

Thanks again for all the help,
Andras