* Resuming SPEC performance tracking at RedHat
@ 2007-06-15 13:20 Vladimir N. Makarov
2007-06-15 14:05 ` Richard Guenther
2007-06-15 14:32 ` Diego Novillo
0 siblings, 2 replies; 5+ messages in thread
From: Vladimir N. Makarov @ 2007-06-15 13:20 UTC (permalink / raw)
To: gcc mailing list
People from gcc community found that GCC performance tracking at
RedHat stopped after Diego left RedHat. As I understand this was
helpful for some of them. Therefore we decided to resume GCC
Performance Tracking on GCC. This work is based on Diego Novillo's
scripts (Diego, thanks for the good scripts) with the following
changes:
o Only Intel Core2 machine is used now for this. I don't plan to do
performance tracking for other machines (like PPC, AMD, and Itanium)
because they are already tested by SUSE and I don't want a
duplication and to use rare not mainstream machines which could be
used for development purposes in RedHat.
o SPEC2000 will be tested for 64-bit and 32-bit mode (-m32).
o I am not going to use SPEC95 for performance tracking. It is too
small benchmarks which does not reflect modern size programs.
o I've added SPEC2006 performance tracking. It will be done once a
week because SPEC2006 takes > 10 times more
time for execution. The SPEC2006 will be run only for -O2
-mtune=generic and only for 64-bit mode .
o Now SPEC2000 and SPEC2006 will run 3 times (as it is used for
reportable results). It permits to generate graphs with the error
marks (minimal and maximum values will be presented on the graph. For
geom. mean scores, the geom. mean of minimal scores and maximum scores
will be present). I think it will be useful even if it may look
ugly.
o Sometimes people saw unexpected drop or increase in scores.
Usually it was result of one or more SPEC program failure. Now
scores with failed programs will be marked on the graphs.
o I removed elapsed compilation times which was measured with one
second accuracy. I found them not useful
especially for individual programs (e.g. compilation time of gzip
was 1, 2 or 3 seconds depended from how lucky we were).
o I've added CPU (user time) compilation time for all SPEC only. In
general case it has no sense to measure them for individual
benchmarks because of runspec start time (measuring them without
runspec start time would require SPEC testsuite modification).
The compiler is built with release checking to see the final
result to see how the release would look like.
o I am going to do release performance tracking too. Now there is
comparison of 3.2 to 4.2 releases.
The result for SPEC2000 will be ready at 6am ET for 64-bit mode at
12pm for 32-bit mode from Monday to Saturday. The result for SPEC2006
will be ready at 2pm on Sunday.
There is a possibility that the results will be not posted on next week
because I'll be on vacation and because the used site is in beta state
but they will be posted later and not lost. I hope Intel server
survive such rigorous testing too.
The performance tracking results is on
https://vmakarov.108.redhat.com/nonav/spec/index.html
Vlad
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Resuming SPEC performance tracking at RedHat
2007-06-15 13:20 Resuming SPEC performance tracking at RedHat Vladimir N. Makarov
@ 2007-06-15 14:05 ` Richard Guenther
2007-06-15 14:32 ` Vladimir N. Makarov
2007-06-18 10:11 ` Michael Matz
2007-06-15 14:32 ` Diego Novillo
1 sibling, 2 replies; 5+ messages in thread
From: Richard Guenther @ 2007-06-15 14:05 UTC (permalink / raw)
To: Vladimir N. Makarov; +Cc: gcc mailing list, David Edelsohn
On 6/15/07, Vladimir N. Makarov <vmakarov@redhat.com> wrote:
> People from gcc community found that GCC performance tracking at
> RedHat stopped after Diego left RedHat. As I understand this was
> helpful for some of them. Therefore we decided to resume GCC
> Performance Tracking on GCC. This work is based on Diego Novillo's
> scripts (Diego, thanks for the good scripts) with the following
> changes:
First off, thanks!
> o Only Intel Core2 machine is used now for this. I don't plan to do
> performance tracking for other machines (like PPC, AMD, and Itanium)
> because they are already tested by SUSE and I don't want a
> duplication and to use rare not mainstream machines which could be
> used for development purposes in RedHat.
Just to summarize what we test at SUSE currently:
- SPEC2000 is tested in various variants on AMD x86_64, including
32bit results and FDO runs.
- SPEC2000 is tested on ia64
- SPEC2006 is tested on AMD x86_64, but results are currently not
accessible to the outside (we need to go over the SPEC reporting
rules)
so, no PPC testing from us (the old testing machine died and we don't
have a replacement for it). So, if you happen to have a machine for
PPC testing then this would be nice. Maybe IBM can set up such as well.
> The performance tracking results is on
>
> https://vmakarov.108.redhat.com/nonav/spec/index.html
To beat Gerald here - can you update gcc.gnu.org/benchmarks.html?
Thanks,
Richard.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Resuming SPEC performance tracking at RedHat
2007-06-15 14:05 ` Richard Guenther
@ 2007-06-15 14:32 ` Vladimir N. Makarov
2007-06-18 10:11 ` Michael Matz
1 sibling, 0 replies; 5+ messages in thread
From: Vladimir N. Makarov @ 2007-06-15 14:32 UTC (permalink / raw)
To: Richard Guenther; +Cc: gcc mailing list, David Edelsohn
Richard Guenther wrote:
> On 6/15/07, Vladimir N. Makarov <vmakarov@redhat.com> wrote:
>
> Just to summarize what we test at SUSE currently:
>
Richard, thanks for the information.
> - SPEC2000 is tested in various variants on AMD x86_64, including
> 32bit results and FDO runs.
>
> - SPEC2000 is tested on ia64
>
I have only one available ia64 machine (Montecito) but it is used for
development.
> - SPEC2006 is tested on AMD x86_64, but results are currently not
> accessible to the outside (we need to go over the SPEC reporting
> rules)
>
> so, no PPC testing from us (the old testing machine died and we don't
> have a replacement for it). So, if you happen to have a machine for
> PPC testing then this would be nice. Maybe IBM can set up such as well.
>
I have a ppc machine which IBM kindly sent me. But now it is actively
used for my register allocation project. May be in a few months, I
could start to use it.
Diego run all tests only once therefore it was natural to use available
machines over nights. Some machines were preproduction ones. I don't
think it has a sence to run the tests once (because the result is
inaccurate in this case) and to use the preproduction machines (I think
we should be oriented to new processors). So it is hard to resume
testing all machines which Diego used.
>> The performance tracking results is on
>>
>> https://vmakarov.108.redhat.com/nonav/spec/index.html
>
>
> To beat Gerald here - can you update gcc.gnu.org/benchmarks.html?
>
Gerald is pretty good at updating html pages. But ok, I'll do it.
Vlad
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Resuming SPEC performance tracking at RedHat
2007-06-15 13:20 Resuming SPEC performance tracking at RedHat Vladimir N. Makarov
2007-06-15 14:05 ` Richard Guenther
@ 2007-06-15 14:32 ` Diego Novillo
1 sibling, 0 replies; 5+ messages in thread
From: Diego Novillo @ 2007-06-15 14:32 UTC (permalink / raw)
To: Vladimir N. Makarov; +Cc: gcc mailing list
On 6/15/07 9:21 AM, Vladimir N. Makarov wrote:
> https://vmakarov.108.redhat.com/nonav/spec/index.html
Awesome! Could you send me the scripts? You had mentioned a few
modifications that sounded very useful.
Thanks.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Resuming SPEC performance tracking at RedHat
2007-06-15 14:05 ` Richard Guenther
2007-06-15 14:32 ` Vladimir N. Makarov
@ 2007-06-18 10:11 ` Michael Matz
1 sibling, 0 replies; 5+ messages in thread
From: Michael Matz @ 2007-06-18 10:11 UTC (permalink / raw)
To: Richard Guenther; +Cc: Vladimir N. Makarov, gcc mailing list, David Edelsohn
Hi,
On Fri, 15 Jun 2007, Richard Guenther wrote:
> so, no PPC testing from us (the old testing machine died and we don't
> have a replacement for it).
Actually it's back, and just needs to be partitioned and set up.
Ciao,
Michael.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2007-06-18 9:06 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-06-15 13:20 Resuming SPEC performance tracking at RedHat Vladimir N. Makarov
2007-06-15 14:05 ` Richard Guenther
2007-06-15 14:32 ` Vladimir N. Makarov
2007-06-18 10:11 ` Michael Matz
2007-06-15 14:32 ` Diego Novillo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).