Benchmark recommendations needed

Benchmark recommendations needed

[Andras Tantos]

Hello all!

I'm working on porting GCC to a new processor architecture. I think
I've finally got to a fairly stable stage, so the next logical step
would be to test and optimize. For that, I would need some benchmarks,
and this is where I'm seeking your help.

This being a hobby project, I can't shell out $1000+ for the spec
suite. On top of that, I only have newlib ported as the runtime, which
means very limited support for OS facilities.

I already have dhrystone, but what else would you recommend using?

I'm looking for 'general purpose' payloads, things, where I can judge
object code size, instruction set utilization, look for tuning
opportunities, etc.

I would like to also be able to compare the results with other
architectures (FPGA cores, such as nios2 as well as some low-end cores,
such as 32-bit arm/thumb and riscv-RV32IMFC).

So, can you suggest some benchmarks or applications to be used as ones?

Thanks a bunch!
Andras

Re: Benchmark recommendations needed

[Richard Biener]

On Tue, Feb 15, 2022 at 3:25 PM Andras Tantos <andras@tantosonline.com> wrote:
>
> Hello all!
>
> I'm working on porting GCC to a new processor architecture. I think
> I've finally got to a fairly stable stage, so the next logical step
> would be to test and optimize. For that, I would need some benchmarks,
> and this is where I'm seeking your help.
>
> This being a hobby project, I can't shell out $1000+ for the spec
> suite. On top of that, I only have newlib ported as the runtime, which
> means very limited support for OS facilities.
>
> I already have dhrystone, but what else would you recommend using?
>
> I'm looking for 'general purpose' payloads, things, where I can judge
> object code size, instruction set utilization, look for tuning
> opportunities, etc.
>
> I would like to also be able to compare the results with other
> architectures (FPGA cores, such as nios2 as well as some low-end cores,
> such as 32-bit arm/thumb and riscv-RV32IMFC).
>
> So, can you suggest some benchmarks or applications to be used as ones?

There's scimark and nbench and for scientific there's fortran polyhedron.
The botan crypto library also has some extensive benchmark setup
for crypto kernels.

Richard.

> Thanks a bunch!
> Andras
>
>

Re: Benchmark recommendations needed

[Hans-Peter Nilsson]

On Mon, 14 Feb 2022, Andras Tantos wrote:
> Hello all!
>
> I'm working on porting GCC to a new processor architecture. I think
> I've finally got to a fairly stable stage, so the next logical step
> would be to test and optimize. For that, I would need some benchmarks,
> and this is where I'm seeking your help.
>
> This being a hobby project, I can't shell out $1000+ for the spec
> suite. On top of that, I only have newlib ported as the runtime, which
> means very limited support for OS facilities.
>
> I already have dhrystone, but what else would you recommend using?
>
> I'm looking for 'general purpose' payloads, things, where I can judge
> object code size, instruction set utilization, look for tuning
> opportunities, etc.
>
> I would like to also be able to compare the results with other
> architectures (FPGA cores, such as nios2 as well as some low-end cores,
> such as 32-bit arm/thumb and riscv-RV32IMFC).
>
> So, can you suggest some benchmarks or applications to be used as ones?

I don't see Coremark being mentioned yet.  It's at
https://github.com/eembc/coremark.git and certainly
general-purpose; just have follow README.md.  I suggest the 10
iterations mentioned in README.md and outputting the number of
cycles from your simulator at exit (through a hook or by a
simulator option).

brgds, H-P

Re: Benchmark recommendations needed

[andras]

Thanks H-P! I'll certainly check it out.

Andras

February 17, 2022 9:15 AM, "Hans-Peter Nilsson" <hp@bitrange.com> wrote:

> On Mon, 14 Feb 2022, Andras Tantos wrote:
> 
>> Hello all!
>> 
>> I'm working on porting GCC to a new processor architecture. I think
>> I've finally got to a fairly stable stage, so the next logical step
>> would be to test and optimize. For that, I would need some benchmarks,
>> and this is where I'm seeking your help.
>> 
>> This being a hobby project, I can't shell out $1000+ for the spec
>> suite. On top of that, I only have newlib ported as the runtime, which
>> means very limited support for OS facilities.
>> 
>> I already have dhrystone, but what else would you recommend using?
>> 
>> I'm looking for 'general purpose' payloads, things, where I can judge
>> object code size, instruction set utilization, look for tuning
>> opportunities, etc.
>> 
>> I would like to also be able to compare the results with other
>> architectures (FPGA cores, such as nios2 as well as some low-end cores,
>> such as 32-bit arm/thumb and riscv-RV32IMFC).
>> 
>> So, can you suggest some benchmarks or applications to be used as ones?
> 
> I don't see Coremark being mentioned yet. It's at
> https://github.com/eembc/coremark.git and certainly
> general-purpose; just have follow README.md. I suggest the 10
> iterations mentioned in README.md and outputting the number of
> cycles from your simulator at exit (through a hook or by a
> simulator option).
> 
> brgds, H-P

Re: Benchmark recommendations needed

[Gary Oblock]

Trying to use the dhrystone isn't going to be very useful. It has many downsides not the least is that gcc's optimizer can run rings about it.

Gary

________________________________
From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org>
Sent: Tuesday, February 15, 2022 6:25 AM
To: gcc@gcc.gnu.org <gcc@gcc.gnu.org>
Subject: Re:

[EXTERNAL EMAIL NOTICE: This email originated from an external sender. Please be mindful of safe email handling and proprietary information protection practices.]


Send Gcc mailing list submissions to
        gcc@gcc.gnu.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://gcc.gnu.org/mailman/listinfo/gcc
or, via email, send a message with subject or body 'help' to
        gcc-request@gcc.gnu.org

You can reach the person managing the list at
        gcc-owner@gcc.gnu.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gcc digest..."

Re: Benchmark recommendations needed

[Andras Tantos]

That's true, I did notice GCC being rather ... peculiar about
drhystone. Is there a way to make it less clever about the benchmark?

Or is there some alteration to the benchmark I can make to not trigger
the special behavior in GCC?

Andras

On Mon, 2022-02-21 at 03:19 +0000, Gary Oblock via Gcc wrote:
> Trying to use the dhrystone isn't going to be very useful. It has
> many downsides not the least is that gcc's optimizer can run rings
> about it.
> 
> Gary
> 
> ________________________________
> From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on
> behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org>
> Sent: Tuesday, February 15, 2022 6:25 AM
> To: gcc@gcc.gnu.org <gcc@gcc.gnu.org>
> Subject: Re:
> 
> [EXTERNAL EMAIL NOTICE: This email originated from an external
> sender. Please be mindful of safe email handling and proprietary
> information protection practices.]
> 
> 
> Send Gcc mailing list submissions to
>         gcc@gcc.gnu.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://gcc.gnu.org/mailman/listinfo/gcc
> or, via email, send a message with subject or body 'help' to
>         gcc-request@gcc.gnu.org
> 
> You can reach the person managing the list at
>         gcc-owner@gcc.gnu.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Gcc digest..."

Re: Benchmark recommendations needed

[Richard Earnshaw]

Dhrystone is (and probably always was) a bogus benchmark.  It's a 
well-known truism that MIPS stands for Meaningless Indication of 
Processor Speed, and dhrystone scores are equally meaningless. 
Dhrystone fell out of common usage over 20 years ago.

It's not GCC that is being peculiar, it's just Dhrystone is pointless.

R.

On 22/02/2022 05:22, Andras Tantos wrote:
> That's true, I did notice GCC being rather ... peculiar about
> drhystone. Is there a way to make it less clever about the benchmark?
> 
> Or is there some alteration to the benchmark I can make to not trigger
> the special behavior in GCC?
> 
> Andras
> 
> On Mon, 2022-02-21 at 03:19 +0000, Gary Oblock via Gcc wrote:
>> Trying to use the dhrystone isn't going to be very useful. It has
>> many downsides not the least is that gcc's optimizer can run rings
>> about it.
>>
>> Gary
>>
>> ________________________________
>> From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on
>> behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org>
>> Sent: Tuesday, February 15, 2022 6:25 AM
>> To: gcc@gcc.gnu.org <gcc@gcc.gnu.org>
>> Subject: Re:
>>
>> [EXTERNAL EMAIL NOTICE: This email originated from an external
>> sender. Please be mindful of safe email handling and proprietary
>> information protection practices.]
>>
>>
>> Send Gcc mailing list submissions to
>>          gcc@gcc.gnu.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>          https://gcc.gnu.org/mailman/listinfo/gcc
>> or, via email, send a message with subject or body 'help' to
>>          gcc-request@gcc.gnu.org
>>
>> You can reach the person managing the list at
>>          gcc-owner@gcc.gnu.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Gcc digest..."
> 

Re: Benchmark recommendations needed

[Gary Oblock]

Andras,

The whole point of benchmarks is to judge a processor's performance.
That being said, just crippling GCC is not reasonable because
processors must be judged in the appropriate context and that
includes the current state of the art compiler technology. If you have
a new processor I'd benchmark it using the applications you built it
for.

Gary

________________________________
From: Andras Tantos <andras@tantosonline.com>
Sent: Monday, February 21, 2022 9:22 PM
To: Gary Oblock <gary@amperecomputing.com>; gcc@gcc.gnu.org <gcc@gcc.gnu.org>
Subject: Re: Benchmark recommendations needed

[EXTERNAL EMAIL NOTICE: This email originated from an external sender. Please be mindful of safe email handling and proprietary information protection practices.]


That's true, I did notice GCC being rather ... peculiar about
drhystone. Is there a way to make it less clever about the benchmark?

Or is there some alteration to the benchmark I can make to not trigger
the special behavior in GCC?

Andras

On Mon, 2022-02-21 at 03:19 +0000, Gary Oblock via Gcc wrote:
> Trying to use the dhrystone isn't going to be very useful. It has
> many downsides not the least is that gcc's optimizer can run rings
> about it.
>
> Gary
>
> ________________________________
> From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on
> behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org>
> Sent: Tuesday, February 15, 2022 6:25 AM
> To: gcc@gcc.gnu.org <gcc@gcc.gnu.org>
> Subject: Re:
>
> [EXTERNAL EMAIL NOTICE: This email originated from an external
> sender. Please be mindful of safe email handling and proprietary
> information protection practices.]
>
>
> Send Gcc mailing list submissions to
>         gcc@gcc.gnu.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://gcc.gnu.org/mailman/listinfo/gcc
> or, via email, send a message with subject or body 'help' to
>         gcc-request@gcc.gnu.org
>
> You can reach the person managing the list at
>         gcc-owner@gcc.gnu.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Gcc digest..."

Re: Benchmark recommendations needed

[Paul Koning]

> On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> wrote:
> 
> Andras,
> 
> The whole point of benchmarks is to judge a processor's performance.
> That being said, just crippling GCC is not reasonable because
> processors must be judged in the appropriate context and that
> includes the current state of the art compiler technology. If you have
> a new processor I'd benchmark it using the applications you built it
> for.

Exactly.  Part of what you want to see is that GCC optimizes well for the new machine, i.e., that there aren't artifacts of the machine description that get in the way of optimization.

So you'd want to use applications that are good exercises not just of the code generator but also the optimizer.  Dhrystone isn't really that, because it has evolved into mostly an optimizer test, not a machine or code generator test.

	paul

Re: Benchmark recommendations needed

[Patrick McGehearty]

I studied Dhrystone about 30 years ago and found it
had a number of flaws back then. For example, most of
the loops in the code are only executed 1-3 times, which
minimizes the value of hoisting values out of inner loops.
Read the Dhrystone wikipedia article for more information.

Going back to what benchmarks might be useful...

you might consider the Livermore Loops
http://www.netlib.org/benchmark/livermorec

These are 24 kernels (tight loops) originally in Fortran but
ported to C 30 years ago. They are reasonably representative
of floating point computational kernels. They are available
without a fee.

Even if you have no interest in floating point computation for
your target architecture, examining the assembly output of these
kernels will be helpful in finding where your port of gcc
is doing well and where the machine architecture input to the
various optimizer phases need some tuning.

You also might review that code and write some modest
test loops of your own for integer code patterns.

Developing good benchmarks is a skill which requires the
developer to know the intended purpose of the benchmark.
I.e. is our goal to compare optimizer implementations?
or different architectures (i.e. arm vs x86)?
or different implementations of an architecture
    (i.e. intel vs amd or early x86 vs current x86)
or ...
well, you get the idea.

Good luck,

- Patrick McGehearty



On 2/22/2022 3:49 PM, Paul Koning via Gcc wrote:
>
>> On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> wrote:
>>
>> Andras,
>>
>> The whole point of benchmarks is to judge a processor's performance.
>> That being said, just crippling GCC is not reasonable because
>> processors must be judged in the appropriate context and that
>> includes the current state of the art compiler technology. If you have
>> a new processor I'd benchmark it using the applications you built it
>> for.
> Exactly.  Part of what you want to see is that GCC optimizes well for the new machine, i.e., that there aren't artifacts of the machine description that get in the way of optimization.
>
> So you'd want to use applications that are good exercises not just of the code generator but also the optimizer.  Dhrystone isn't really that, because it has evolved into mostly an optimizer test, not a machine or code generator test.
>
> 	paul
>

Re: Benchmark recommendations needed

[andras]

On 2022-02-22 16:59, Patrick McGehearty wrote:
> I studied Dhrystone about 30 years ago and found it
> had a number of flaws back then. For example, most of
> the loops in the code are only executed 1-3 times, which
> minimizes the value of hoisting values out of inner loops.
> Read the Dhrystone wikipedia article for more information.
> 
> Going back to what benchmarks might be useful...
> 
> you might consider the Livermore Loops
> http://www.netlib.org/benchmark/livermorec
> 
> These are 24 kernels (tight loops) originally in Fortran but
> ported to C 30 years ago. They are reasonably representative
> of floating point computational kernels. They are available
> without a fee.
> 
> Even if you have no interest in floating point computation for
> your target architecture, examining the assembly output of these
> kernels will be helpful in finding where your port of gcc
> is doing well and where the machine architecture input to the
> various optimizer phases need some tuning.
> 
> You also might review that code and write some modest
> test loops of your own for integer code patterns.
> 
> Developing good benchmarks is a skill which requires the
> developer to know the intended purpose of the benchmark.
> I.e. is our goal to compare optimizer implementations?
> or different architectures (i.e. arm vs x86)?
> or different implementations of an architecture
>    (i.e. intel vs amd or early x86 vs current x86)
> or ...
> well, you get the idea.
> 
> Good luck,
> 
> - Patrick McGehearty
> 
> 
> 
> On 2/22/2022 3:49 PM, Paul Koning via Gcc wrote:
>> 
>>> On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> 
>>> wrote:
>>> 
>>> Andras,
>>> 
>>> The whole point of benchmarks is to judge a processor's performance.
>>> That being said, just crippling GCC is not reasonable because
>>> processors must be judged in the appropriate context and that
>>> includes the current state of the art compiler technology. If you 
>>> have
>>> a new processor I'd benchmark it using the applications you built it
>>> for.
>> Exactly.  Part of what you want to see is that GCC optimizes well for 
>> the new machine, i.e., that there aren't artifacts of the machine 
>> description that get in the way of optimization.
>> 
>> So you'd want to use applications that are good exercises not just of 
>> the code generator but also the optimizer.  Dhrystone isn't really 
>> that, because it has evolved into mostly an optimizer test, not a 
>> machine or code generator test.
>> 
>> 	paul
>> 

Thank you for all the feedback. When I said 'crippling' GCC, I didn't 
mean to make it generally worse, just to make sure I don't trigger 
dhrystone-specific optimizations, if there are any.

Your point about comparing processors is a fair one and that's where I'm 
heading, but I need to start small. Even just comparing object file 
(.text section) size between architectures have been hugely 
enlightening.

There is another point though: my port of GCC is certainly not state of 
the art (well, it technically is, because that's the only port in 
existence, but you get what I mean). Part of the process is to hone in 
the port in terms of tuning the instruction selection, peepholes, etc. 
In some sense this work benchmarks compilers not processors. At any 
rate, I do get that dhrystone is a poor substitute for a real benchmark 
and I'll try to move away from it.

I wasn't aware of Livermore loops being ported to C (and I don't yet 
have a Fortran port), so that's a great suggestion. I'll check it out.

Thanks again for all the help,
Andras