re: Request for Steering Committee: official release criteria for 4.x

re: Request for Steering Committee: official release criteria for 4.x

[Dan Kegel]

"Giovanni Bajo" <giovannibajo@libero.it> wrote:
> - Do we want a small set of files/applications to closely monitor code
> generation quality and compile time behaviour? 

How 'bout the set specified by SPECcpu2000?  One could do worse.

(In case anyone doesn't have the URL handy, there's an up-to-date
graph of runtime performance on each of those apps vs. day vs. branch at
http://www.suse.de/~aj/SPEC/amd64/
Dig down several pages, it's worth it.
My only gripe with these pages is that they require better
color vision than I possess to distinguish the traces on the graphs.)
- Dan


-- 
Trying to get a job as a c++ developer?  See http://kegel.com/academy/getting-hired.html

Re: Request for Steering Committee: official release criteria for 4.x

[Aaron W. LaFramboise]

Dan Kegel wrote:

> "Giovanni Bajo" <giovannibajo@libero.it> wrote:
> 
>>- Do we want a small set of files/applications to closely monitor code
>>generation quality and compile time behaviour? 
> 
> How 'bout the set specified by SPECcpu2000?  One could do worse.

Is there any way for Joe GCC User to get a copy of SPEC?  Or is SPEC
proprietary or secret or otherwise non-free?

Aaron W. LaFramboise

Re: Request for Steering Committee: official release criteria for 4.x

[Steven Bosscher]

On Sunday 31 October 2004 06:34, Aaron W. LaFramboise wrote:
> Is there any way for Joe GCC User to get a copy of SPEC?  Or is SPEC
> proprietary or secret or otherwise non-free?

SPEC is not free.

Gr.
Steven

Re: Request for Steering Committee: official release criteria for 4.x

[Dan Kegel]

Steven Bosscher wrote:
> On Sunday 31 October 2004 06:34, Aaron W. LaFramboise wrote:
> 
>>Is there any way for Joe GCC User to get a copy of SPEC?  Or is SPEC
>>proprietary or secret or otherwise non-free?
> 
> 
> SPEC is not free.

However, it is made up of a bunch of little programs,
some of which are free.  We can probably scrape together
a free subset of something very much like SPECcpu, using
SPECcpu as a guide.
- Dan

-- 
Trying to get a job as a c++ developer?  See http://kegel.com/academy/getting-hired.html

Re: Request for Steering Committee: official release criteria for4.x

[Giovanni Bajo]

Dan Kegel wrote:

>>> Is there any way for Joe GCC User to get a copy of SPEC?  Or is SPEC
>>> proprietary or secret or otherwise non-free?
>>
>> SPEC is not free.
>
> However, it is made up of a bunch of little programs,
> some of which are free.  We can probably scrape together
> a free subset of something very much like SPECcpu, using
> SPECcpu as a guide.

That was my plan for make check-performance: having something like SPEC within
GCC's own testsuite which could be used by all the developers to check the
performance. It does not need to be close to SPECcpu itself though.

Giovanni Bajo

Re: Request for Steering Committee: official release criteria for 4.x

[Jeffrey A Law]

On Sun, 2004-10-31 at 06:23 -0800, Dan Kegel wrote:
> Steven Bosscher wrote:
> > On Sunday 31 October 2004 06:34, Aaron W. LaFramboise wrote:
> > 
> >>Is there any way for Joe GCC User to get a copy of SPEC?  Or is SPEC
> >>proprietary or secret or otherwise non-free?
> > 
> > 
> > SPEC is not free.
> 
> However, it is made up of a bunch of little programs,
> some of which are free.  We can probably scrape together
> a free subset of something very much like SPECcpu, using
> SPECcpu as a guide.
Very true.  But one of the things that is particularly important
is the input vectors for the benchmarks.  

However, I don't think we can go wrong building a benchmark suite
based (when possible) on the same code as SPEC with our own
input vectors.

jeff

Re: Request for Steering Committee: official release criteria for 4.x

[Vladimir Makarov]

Jeffrey A Law wrote:

>On Sun, 2004-10-31 at 06:23 -0800, Dan Kegel wrote:
>  
>
>>Steven Bosscher wrote:
>>    
>>
>>>On Sunday 31 October 2004 06:34, Aaron W. LaFramboise wrote:
>>>
>>>      
>>>
>>>>Is there any way for Joe GCC User to get a copy of SPEC?  Or is SPEC
>>>>proprietary or secret or otherwise non-free?
>>>>        
>>>>
>>>SPEC is not free.
>>>      
>>>
>>However, it is made up of a bunch of little programs,
>>some of which are free.  We can probably scrape together
>>a free subset of something very much like SPECcpu, using
>>SPECcpu as a guide.
>>    
>>
I am not sure all programs are free.  Authors of the benchmarks give 
permission to use the program only as a part of Spec bechmark.  That is 
what SPEC requires in order to include a program in SPEC testsuite.  But 
we could ask the authors to give us permission to use the programs as a 
part of gcc testsuite.

I think that the bigger work is to write framework analogous (or better) 
to SPEC.

>Very true.  But one of the things that is particularly important
>is the input vectors for the benchmarks.  
>
>However, I don't think we can go wrong building a benchmark suite
>based (when possible) on the same code as SPEC with our own
>input vectors.
>
>  
>
Inputs and correct outputs are provided by the authors.  So we could 
also ask the authors to use the data as a part of gcc testsuite too.

Vlad

Request for Steering Committee: official release criteria for 4.x

[Giovanni Bajo]

Hello,

this is a request for the Steering Committee.

I think we need to revive a page holding a list of the official release
criteria for 4.x releases. Right now, it is all pretty arbitrary and heuristic,
with Bugmasters knowing probably more rules than many mainteners (having
deduced those rules, eg., carefully watching the Bugzilla work being done by
RMs, which is something not all maintainers are supposed to do). I would like
such a page to answer [at least] these questions:

- Which are the so-called "primary platforms" for GCC 4.x? [what is,
technically, a primary platform? I would say that a primary platform is
platform whose specific regressions might stop a release. Is this true?]

- Which are the so-called "secondary platforms" for GCC 4.x? [what is,
technically, a secondary platform? I would say that it is a platform for which
broken bootstrap might be a show-stopper for a release. If a bug happens on a
platform which is neither primary nor secondary, should it be targeted at the
next major release automatically or not?]

- Which are the real-world applications that should absolutey be tested as
working? [Kernel, Glibc, POOMA, Boost...]

- Which is the oldest release against which regressions are checked? [2.95
probably]

- Which is the compile-time threshold at -O0 for a bug to be considered a
regression? [used to be 25%]

- Which is the compile-time threshold (if any!) at -O1/-O2/-O3/-Os for a bug to
be considered a regression? [used to be 25%]

- Which is the memory occupation threshold at -O0/-O1/-O2/-O3/-Os for a bug to
be considered a regression?

- Which is the binary size threshold at -Os for a bug to be considered a
regression?

- Which is the binary size threshold (if any!) at -Os for a bug to be
considered a regression?

- Do we want a small set of files/applications to closely monitor code
generation quality and compile time behaviour? The old release criteria
suggested some (LAPACK, Stepanov abstraction test, gzip). We could isolate a
small number of preprocessed (compilable and runnable, where possible) sources
against which run a set of script-based performance tests. For these files,
probably the thresholds for regressions could be smaller (5% or so). I believe
this could be automated in "make check-performance", using another specified
compiler as the baseline (for instance, it could be the bootstrapping compiler
by default).


If these questions are officially answered and/or other criteria officially
specified, I volunteer to do the html legwork to revive the criteria page on
the site.

Thanks
Giovanni Bajo