* Re: Benchmark recommendations needed [not found] <mailman.7085.1644935119.2100866.gcc@gcc.gnu.org> @ 2022-02-21 3:19 ` Gary Oblock 2022-02-22 5:22 ` Andras Tantos 0 siblings, 1 reply; 11+ messages in thread From: Gary Oblock @ 2022-02-21 3:19 UTC (permalink / raw) To: gcc Trying to use the dhrystone isn't going to be very useful. It has many downsides not the least is that gcc's optimizer can run rings about it. Gary ________________________________ From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org> Sent: Tuesday, February 15, 2022 6:25 AM To: gcc@gcc.gnu.org <gcc@gcc.gnu.org> Subject: Re: [EXTERNAL EMAIL NOTICE: This email originated from an external sender. Please be mindful of safe email handling and proprietary information protection practices.] Send Gcc mailing list submissions to gcc@gcc.gnu.org To subscribe or unsubscribe via the World Wide Web, visit https://gcc.gnu.org/mailman/listinfo/gcc or, via email, send a message with subject or body 'help' to gcc-request@gcc.gnu.org You can reach the person managing the list at gcc-owner@gcc.gnu.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Gcc digest..." ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-21 3:19 ` Benchmark recommendations needed Gary Oblock @ 2022-02-22 5:22 ` Andras Tantos 2022-02-22 11:42 ` Richard Earnshaw 2022-02-22 21:26 ` Gary Oblock 0 siblings, 2 replies; 11+ messages in thread From: Andras Tantos @ 2022-02-22 5:22 UTC (permalink / raw) To: Gary Oblock, gcc That's true, I did notice GCC being rather ... peculiar about drhystone. Is there a way to make it less clever about the benchmark? Or is there some alteration to the benchmark I can make to not trigger the special behavior in GCC? Andras On Mon, 2022-02-21 at 03:19 +0000, Gary Oblock via Gcc wrote: > Trying to use the dhrystone isn't going to be very useful. It has > many downsides not the least is that gcc's optimizer can run rings > about it. > > Gary > > ________________________________ > From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on > behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org> > Sent: Tuesday, February 15, 2022 6:25 AM > To: gcc@gcc.gnu.org <gcc@gcc.gnu.org> > Subject: Re: > > [EXTERNAL EMAIL NOTICE: This email originated from an external > sender. Please be mindful of safe email handling and proprietary > information protection practices.] > > > Send Gcc mailing list submissions to > gcc@gcc.gnu.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://gcc.gnu.org/mailman/listinfo/gcc > or, via email, send a message with subject or body 'help' to > gcc-request@gcc.gnu.org > > You can reach the person managing the list at > gcc-owner@gcc.gnu.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Gcc digest..." ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-22 5:22 ` Andras Tantos @ 2022-02-22 11:42 ` Richard Earnshaw 2022-02-22 21:26 ` Gary Oblock 1 sibling, 0 replies; 11+ messages in thread From: Richard Earnshaw @ 2022-02-22 11:42 UTC (permalink / raw) To: Andras Tantos, Gary Oblock, gcc Dhrystone is (and probably always was) a bogus benchmark. It's a well-known truism that MIPS stands for Meaningless Indication of Processor Speed, and dhrystone scores are equally meaningless. Dhrystone fell out of common usage over 20 years ago. It's not GCC that is being peculiar, it's just Dhrystone is pointless. R. On 22/02/2022 05:22, Andras Tantos wrote: > That's true, I did notice GCC being rather ... peculiar about > drhystone. Is there a way to make it less clever about the benchmark? > > Or is there some alteration to the benchmark I can make to not trigger > the special behavior in GCC? > > Andras > > On Mon, 2022-02-21 at 03:19 +0000, Gary Oblock via Gcc wrote: >> Trying to use the dhrystone isn't going to be very useful. It has >> many downsides not the least is that gcc's optimizer can run rings >> about it. >> >> Gary >> >> ________________________________ >> From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on >> behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org> >> Sent: Tuesday, February 15, 2022 6:25 AM >> To: gcc@gcc.gnu.org <gcc@gcc.gnu.org> >> Subject: Re: >> >> [EXTERNAL EMAIL NOTICE: This email originated from an external >> sender. Please be mindful of safe email handling and proprietary >> information protection practices.] >> >> >> Send Gcc mailing list submissions to >> gcc@gcc.gnu.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://gcc.gnu.org/mailman/listinfo/gcc >> or, via email, send a message with subject or body 'help' to >> gcc-request@gcc.gnu.org >> >> You can reach the person managing the list at >> gcc-owner@gcc.gnu.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Gcc digest..." > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-22 5:22 ` Andras Tantos 2022-02-22 11:42 ` Richard Earnshaw @ 2022-02-22 21:26 ` Gary Oblock 2022-02-22 21:49 ` Paul Koning 1 sibling, 1 reply; 11+ messages in thread From: Gary Oblock @ 2022-02-22 21:26 UTC (permalink / raw) To: Andras Tantos, gcc Andras, The whole point of benchmarks is to judge a processor's performance. That being said, just crippling GCC is not reasonable because processors must be judged in the appropriate context and that includes the current state of the art compiler technology. If you have a new processor I'd benchmark it using the applications you built it for. Gary ________________________________ From: Andras Tantos <andras@tantosonline.com> Sent: Monday, February 21, 2022 9:22 PM To: Gary Oblock <gary@amperecomputing.com>; gcc@gcc.gnu.org <gcc@gcc.gnu.org> Subject: Re: Benchmark recommendations needed [EXTERNAL EMAIL NOTICE: This email originated from an external sender. Please be mindful of safe email handling and proprietary information protection practices.] That's true, I did notice GCC being rather ... peculiar about drhystone. Is there a way to make it less clever about the benchmark? Or is there some alteration to the benchmark I can make to not trigger the special behavior in GCC? Andras On Mon, 2022-02-21 at 03:19 +0000, Gary Oblock via Gcc wrote: > Trying to use the dhrystone isn't going to be very useful. It has > many downsides not the least is that gcc's optimizer can run rings > about it. > > Gary > > ________________________________ > From: Gcc <gcc-bounces+gary=amperecomputing.com@gcc.gnu.org> on > behalf of gcc-request@gcc.gnu.org <gcc-request@gcc.gnu.org> > Sent: Tuesday, February 15, 2022 6:25 AM > To: gcc@gcc.gnu.org <gcc@gcc.gnu.org> > Subject: Re: > > [EXTERNAL EMAIL NOTICE: This email originated from an external > sender. Please be mindful of safe email handling and proprietary > information protection practices.] > > > Send Gcc mailing list submissions to > gcc@gcc.gnu.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://gcc.gnu.org/mailman/listinfo/gcc > or, via email, send a message with subject or body 'help' to > gcc-request@gcc.gnu.org > > You can reach the person managing the list at > gcc-owner@gcc.gnu.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Gcc digest..." ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-22 21:26 ` Gary Oblock @ 2022-02-22 21:49 ` Paul Koning 2022-02-23 0:59 ` Patrick McGehearty 0 siblings, 1 reply; 11+ messages in thread From: Paul Koning @ 2022-02-22 21:49 UTC (permalink / raw) To: Gary Oblock; +Cc: Andras Tantos, GCC Development > On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> wrote: > > Andras, > > The whole point of benchmarks is to judge a processor's performance. > That being said, just crippling GCC is not reasonable because > processors must be judged in the appropriate context and that > includes the current state of the art compiler technology. If you have > a new processor I'd benchmark it using the applications you built it > for. Exactly. Part of what you want to see is that GCC optimizes well for the new machine, i.e., that there aren't artifacts of the machine description that get in the way of optimization. So you'd want to use applications that are good exercises not just of the code generator but also the optimizer. Dhrystone isn't really that, because it has evolved into mostly an optimizer test, not a machine or code generator test. paul ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-22 21:49 ` Paul Koning @ 2022-02-23 0:59 ` Patrick McGehearty 2022-02-23 17:01 ` andras 0 siblings, 1 reply; 11+ messages in thread From: Patrick McGehearty @ 2022-02-23 0:59 UTC (permalink / raw) To: gcc I studied Dhrystone about 30 years ago and found it had a number of flaws back then. For example, most of the loops in the code are only executed 1-3 times, which minimizes the value of hoisting values out of inner loops. Read the Dhrystone wikipedia article for more information. Going back to what benchmarks might be useful... you might consider the Livermore Loops http://www.netlib.org/benchmark/livermorec These are 24 kernels (tight loops) originally in Fortran but ported to C 30 years ago. They are reasonably representative of floating point computational kernels. They are available without a fee. Even if you have no interest in floating point computation for your target architecture, examining the assembly output of these kernels will be helpful in finding where your port of gcc is doing well and where the machine architecture input to the various optimizer phases need some tuning. You also might review that code and write some modest test loops of your own for integer code patterns. Developing good benchmarks is a skill which requires the developer to know the intended purpose of the benchmark. I.e. is our goal to compare optimizer implementations? or different architectures (i.e. arm vs x86)? or different implementations of an architecture (i.e. intel vs amd or early x86 vs current x86) or ... well, you get the idea. Good luck, - Patrick McGehearty On 2/22/2022 3:49 PM, Paul Koning via Gcc wrote: > >> On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> wrote: >> >> Andras, >> >> The whole point of benchmarks is to judge a processor's performance. >> That being said, just crippling GCC is not reasonable because >> processors must be judged in the appropriate context and that >> includes the current state of the art compiler technology. If you have >> a new processor I'd benchmark it using the applications you built it >> for. > Exactly. Part of what you want to see is that GCC optimizes well for the new machine, i.e., that there aren't artifacts of the machine description that get in the way of optimization. > > So you'd want to use applications that are good exercises not just of the code generator but also the optimizer. Dhrystone isn't really that, because it has evolved into mostly an optimizer test, not a machine or code generator test. > > paul > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-23 0:59 ` Patrick McGehearty @ 2022-02-23 17:01 ` andras 0 siblings, 0 replies; 11+ messages in thread From: andras @ 2022-02-23 17:01 UTC (permalink / raw) To: Patrick McGehearty; +Cc: gcc On 2022-02-22 16:59, Patrick McGehearty wrote: > I studied Dhrystone about 30 years ago and found it > had a number of flaws back then. For example, most of > the loops in the code are only executed 1-3 times, which > minimizes the value of hoisting values out of inner loops. > Read the Dhrystone wikipedia article for more information. > > Going back to what benchmarks might be useful... > > you might consider the Livermore Loops > http://www.netlib.org/benchmark/livermorec > > These are 24 kernels (tight loops) originally in Fortran but > ported to C 30 years ago. They are reasonably representative > of floating point computational kernels. They are available > without a fee. > > Even if you have no interest in floating point computation for > your target architecture, examining the assembly output of these > kernels will be helpful in finding where your port of gcc > is doing well and where the machine architecture input to the > various optimizer phases need some tuning. > > You also might review that code and write some modest > test loops of your own for integer code patterns. > > Developing good benchmarks is a skill which requires the > developer to know the intended purpose of the benchmark. > I.e. is our goal to compare optimizer implementations? > or different architectures (i.e. arm vs x86)? > or different implementations of an architecture > (i.e. intel vs amd or early x86 vs current x86) > or ... > well, you get the idea. > > Good luck, > > - Patrick McGehearty > > > > On 2/22/2022 3:49 PM, Paul Koning via Gcc wrote: >> >>> On Feb 22, 2022, at 4:26 PM, Gary Oblock via Gcc <gcc@gcc.gnu.org> >>> wrote: >>> >>> Andras, >>> >>> The whole point of benchmarks is to judge a processor's performance. >>> That being said, just crippling GCC is not reasonable because >>> processors must be judged in the appropriate context and that >>> includes the current state of the art compiler technology. If you >>> have >>> a new processor I'd benchmark it using the applications you built it >>> for. >> Exactly. Part of what you want to see is that GCC optimizes well for >> the new machine, i.e., that there aren't artifacts of the machine >> description that get in the way of optimization. >> >> So you'd want to use applications that are good exercises not just of >> the code generator but also the optimizer. Dhrystone isn't really >> that, because it has evolved into mostly an optimizer test, not a >> machine or code generator test. >> >> paul >> Thank you for all the feedback. When I said 'crippling' GCC, I didn't mean to make it generally worse, just to make sure I don't trigger dhrystone-specific optimizations, if there are any. Your point about comparing processors is a fair one and that's where I'm heading, but I need to start small. Even just comparing object file (.text section) size between architectures have been hugely enlightening. There is another point though: my port of GCC is certainly not state of the art (well, it technically is, because that's the only port in existence, but you get what I mean). Part of the process is to hone in the port in terms of tuning the instruction selection, peepholes, etc. In some sense this work benchmarks compilers not processors. At any rate, I do get that dhrystone is a poor substitute for a real benchmark and I'll try to move away from it. I wasn't aware of Livermore loops being ported to C (and I don't yet have a Fortran port), so that's a great suggestion. I'll check it out. Thanks again for all the help, Andras ^ permalink raw reply [flat|nested] 11+ messages in thread
* Benchmark recommendations needed @ 2022-02-14 22:23 Andras Tantos 2022-02-15 14:31 ` Richard Biener ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Andras Tantos @ 2022-02-14 22:23 UTC (permalink / raw) To: gcc Hello all! I'm working on porting GCC to a new processor architecture. I think I've finally got to a fairly stable stage, so the next logical step would be to test and optimize. For that, I would need some benchmarks, and this is where I'm seeking your help. This being a hobby project, I can't shell out $1000+ for the spec suite. On top of that, I only have newlib ported as the runtime, which means very limited support for OS facilities. I already have dhrystone, but what else would you recommend using? I'm looking for 'general purpose' payloads, things, where I can judge object code size, instruction set utilization, look for tuning opportunities, etc. I would like to also be able to compare the results with other architectures (FPGA cores, such as nios2 as well as some low-end cores, such as 32-bit arm/thumb and riscv-RV32IMFC). So, can you suggest some benchmarks or applications to be used as ones? Thanks a bunch! Andras ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-14 22:23 Andras Tantos @ 2022-02-15 14:31 ` Richard Biener 2022-02-17 17:15 ` Hans-Peter Nilsson 2022-02-17 18:27 ` andras 2 siblings, 0 replies; 11+ messages in thread From: Richard Biener @ 2022-02-15 14:31 UTC (permalink / raw) To: Andras Tantos; +Cc: GCC Development On Tue, Feb 15, 2022 at 3:25 PM Andras Tantos <andras@tantosonline.com> wrote: > > Hello all! > > I'm working on porting GCC to a new processor architecture. I think > I've finally got to a fairly stable stage, so the next logical step > would be to test and optimize. For that, I would need some benchmarks, > and this is where I'm seeking your help. > > This being a hobby project, I can't shell out $1000+ for the spec > suite. On top of that, I only have newlib ported as the runtime, which > means very limited support for OS facilities. > > I already have dhrystone, but what else would you recommend using? > > I'm looking for 'general purpose' payloads, things, where I can judge > object code size, instruction set utilization, look for tuning > opportunities, etc. > > I would like to also be able to compare the results with other > architectures (FPGA cores, such as nios2 as well as some low-end cores, > such as 32-bit arm/thumb and riscv-RV32IMFC). > > So, can you suggest some benchmarks or applications to be used as ones? There's scimark and nbench and for scientific there's fortran polyhedron. The botan crypto library also has some extensive benchmark setup for crypto kernels. Richard. > Thanks a bunch! > Andras > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-14 22:23 Andras Tantos 2022-02-15 14:31 ` Richard Biener @ 2022-02-17 17:15 ` Hans-Peter Nilsson 2022-02-17 18:27 ` andras 2 siblings, 0 replies; 11+ messages in thread From: Hans-Peter Nilsson @ 2022-02-17 17:15 UTC (permalink / raw) To: Andras Tantos; +Cc: gcc On Mon, 14 Feb 2022, Andras Tantos wrote: > Hello all! > > I'm working on porting GCC to a new processor architecture. I think > I've finally got to a fairly stable stage, so the next logical step > would be to test and optimize. For that, I would need some benchmarks, > and this is where I'm seeking your help. > > This being a hobby project, I can't shell out $1000+ for the spec > suite. On top of that, I only have newlib ported as the runtime, which > means very limited support for OS facilities. > > I already have dhrystone, but what else would you recommend using? > > I'm looking for 'general purpose' payloads, things, where I can judge > object code size, instruction set utilization, look for tuning > opportunities, etc. > > I would like to also be able to compare the results with other > architectures (FPGA cores, such as nios2 as well as some low-end cores, > such as 32-bit arm/thumb and riscv-RV32IMFC). > > So, can you suggest some benchmarks or applications to be used as ones? I don't see Coremark being mentioned yet. It's at https://github.com/eembc/coremark.git and certainly general-purpose; just have follow README.md. I suggest the 10 iterations mentioned in README.md and outputting the number of cycles from your simulator at exit (through a hook or by a simulator option). brgds, H-P ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Benchmark recommendations needed 2022-02-14 22:23 Andras Tantos 2022-02-15 14:31 ` Richard Biener 2022-02-17 17:15 ` Hans-Peter Nilsson @ 2022-02-17 18:27 ` andras 2 siblings, 0 replies; 11+ messages in thread From: andras @ 2022-02-17 18:27 UTC (permalink / raw) To: Hans-Peter Nilsson; +Cc: gcc Thanks H-P! I'll certainly check it out. Andras February 17, 2022 9:15 AM, "Hans-Peter Nilsson" <hp@bitrange.com> wrote: > On Mon, 14 Feb 2022, Andras Tantos wrote: > >> Hello all! >> >> I'm working on porting GCC to a new processor architecture. I think >> I've finally got to a fairly stable stage, so the next logical step >> would be to test and optimize. For that, I would need some benchmarks, >> and this is where I'm seeking your help. >> >> This being a hobby project, I can't shell out $1000+ for the spec >> suite. On top of that, I only have newlib ported as the runtime, which >> means very limited support for OS facilities. >> >> I already have dhrystone, but what else would you recommend using? >> >> I'm looking for 'general purpose' payloads, things, where I can judge >> object code size, instruction set utilization, look for tuning >> opportunities, etc. >> >> I would like to also be able to compare the results with other >> architectures (FPGA cores, such as nios2 as well as some low-end cores, >> such as 32-bit arm/thumb and riscv-RV32IMFC). >> >> So, can you suggest some benchmarks or applications to be used as ones? > > I don't see Coremark being mentioned yet. It's at > https://github.com/eembc/coremark.git and certainly > general-purpose; just have follow README.md. I suggest the 10 > iterations mentioned in README.md and outputting the number of > cycles from your simulator at exit (through a hook or by a > simulator option). > > brgds, H-P ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-02-23 17:01 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <mailman.7085.1644935119.2100866.gcc@gcc.gnu.org> 2022-02-21 3:19 ` Benchmark recommendations needed Gary Oblock 2022-02-22 5:22 ` Andras Tantos 2022-02-22 11:42 ` Richard Earnshaw 2022-02-22 21:26 ` Gary Oblock 2022-02-22 21:49 ` Paul Koning 2022-02-23 0:59 ` Patrick McGehearty 2022-02-23 17:01 ` andras 2022-02-14 22:23 Andras Tantos 2022-02-15 14:31 ` Richard Biener 2022-02-17 17:15 ` Hans-Peter Nilsson 2022-02-17 18:27 ` andras
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).