Revised release criteria for GCC 4.0

Revised release criteria for GCC 4.0

[Mark Mitchell]

Working with the SC, I have prepared revised release criteria for GCC 
4.0, which are available here:

   http://gcc.gnu.org/gcc-4.0/criteria.html

These revisions include changes to the set of primary and secondary 
platforms to more accurately reflect the platforms currently thought to 
be important, and also include more realistic goals for validation.

Comments are welcome, and we might make changes if there is sufficient 
momentum in a particular direction.  However, I would suggest that you 
not spend too much energy picking nits; our release criteria are 
guidelines, not absolutes.

Yours,

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[Richard Earnshaw]

On Mon, 2004-12-13 at 02:09, Mark Mitchell wrote:
> Working with the SC, I have prepared revised release criteria for GCC 
> 4.0, which are available here:
> 
>    http://gcc.gnu.org/gcc-4.0/criteria.html
> 
> These revisions include changes to the set of primary and secondary 
> platforms to more accurately reflect the platforms currently thought to 
> be important, and also include more realistic goals for validation.
> 
> Comments are welcome, and we might make changes if there is sufficient 
> momentum in a particular direction.  However, I would suggest that you 
> not spend too much energy picking nits; our release criteria are 
> guidelines, not absolutes.

How come mips-elf is a primary platform, but arm-elf is only secondary?

R.

Re: Revised release criteria for GCC 4.0

[Mark Mitchell]

Richard Earnshaw wrote:
> On Mon, 2004-12-13 at 02:09, Mark Mitchell wrote:
> 
>>Working with the SC, I have prepared revised release criteria for GCC 
>>4.0, which are available here:
>>
>>   http://gcc.gnu.org/gcc-4.0/criteria.html
>>
>>These revisions include changes to the set of primary and secondary 
>>platforms to more accurately reflect the platforms currently thought to 
>>be important, and also include more realistic goals for validation.
>>
>>Comments are welcome, and we might make changes if there is sufficient 
>>momentum in a particular direction.  However, I would suggest that you 
>>not spend too much energy picking nits; our release criteria are 
>>guidelines, not absolutes.
> 
> 
> How come mips-elf is a primary platform, but arm-elf is only secondary?

Historical accident, as much as anything.

There was a MIPS platform before (IRIX).  The SC seemed to agree that 
IRIX was no longer a major focus, but wanted to keep a MIPS platform.

I certainly have no objection to adding arm-none-elf to the primary 
platform lists; it seems like a good idea to me.  Would you like to ask 
the SC directly, or would you prefer that I do it?

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[Richard Earnshaw]

On Mon, 2004-12-13 at 15:23, Mark Mitchell wrote:
> I certainly have no objection to adding arm-none-elf to the primary 
> platform lists; it seems like a good idea to me.  Would you like to ask 
> the SC directly, or would you prefer that I do it?

I'd be happy to let you handle it.

R.

Re: Revised release criteria for GCC 4.0

[Scott Robert Ladd]

Mark, 

Am I correct in assuming that the state of gfortran is not a
requirement for the 4.0 release?

Also, do the current commitment restrictions prohibit enhancements for
gfortran? Since it's a new front end, it doesn't have regressions, per
se, unless you consider it in comparison to g77 (i.e., gfortran
failures to compile old g77-compatible code are regressions).

Thanks.

..Scott

Re: Revised release criteria for GCC 4.0

[Giovanni Bajo]

Scott Robert Ladd <scott.ladd@gmail.com> wrote:

> Also, do the current commitment restrictions prohibit enhancements for
> gfortran?

I would not agree with that.

> Since it's a new front end, it doesn't have regressions, per
> se, unless you consider it in comparison to g77 (i.e., gfortran
> failures to compile old g77-compatible code are regressions).

When we speak of regression, we do not strictly speak of a techinical
regression in the compiler code, rather a regression in the user experience
of the product. If somebody was using GCC 3.4 for his Fortran code, and GCC
4.0 is unable to compile it because of a bug[*], then it is a regression,
and will probably prevent the user from upgrading to the newer version. The
fact that the frontend is brand new is a non-point.

-- 
Giovanni Bajo

Re: Revised release criteria for GCC 4.0

[Mark Mitchell]

Giovanni Bajo wrote:

>>Since it's a new front end, it doesn't have regressions, per
>>se, unless you consider it in comparison to g77 (i.e., gfortran
>>failures to compile old g77-compatible code are regressions).
> 
> 
> When we speak of regression, we do not strictly speak of a techinical
> regression in the compiler code, rather a regression in the user experience
> of the product. If somebody was using GCC 3.4 for his Fortran code, and GCC
> 4.0 is unable to compile it because of a bug[*], then it is a regression,
> and will probably prevent the user from upgrading to the newer version. The
> fact that the frontend is brand new is a non-point.

I agree -- but, realistically, with a complete rewrite there are going 
to be regressions for years to come.  It's good to fix them, but when we 
made the choice to write a new front end, we also made the choice to 
accept bugs in return for features.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[Björn Haase]

 Hi,

 I have just seen some discussion concerning running the test suite with the
 avr simulator.

 Using simulavr, I have so far succeeded the vast majority of the execution
 tests. Among the whole test suite there are only about 3 tests that cannot be
 executed since they need too much memory. (If interested: look at the 
corresponding  threads at the avr-libc-dev mailing list for a how-to). I have 
just posted the first test result at gcc-testresult@gcc.gnu.org.

 Having in mind that the avr community is fairly active, it is, in my opionion
 not justified to consider this platform to be not even secondary target.
 In my opinion it could be a big benefit also for the bigger platforms 
development:
 Those like us, who are working with tiny systems actually still look at their
 assembler code. We will recognize immediately if a change in the compiler
 mid-end, e.g. the register allocator results in strange or not optimal
 code.

 Yours,

 Björn

Re: Revised release criteria for GCC 4.0

[Bernardo Innocenti]

Hans-Peter Nilsson wrote:

> The presence of a turn-key simulator as per simtest-howto.html
> makes it *immensely* more simple for others than hardware
> holders to regression-test their GCC changes, verify bugs and
> fix them.  So all the shouting from AVR enthusiasts is suggested
> be better spent getting an AVR simulator and a newlib port in
> the tree.

I forgot to say that the AVR simulator is a GPL'd project
on Savannah:

  http://savannah.nongnu.org/projects/simulavr

There's also a gdb-remote debugger for JTAG ICEs:

  http://sourceforge.net/projects/avarice/


AFAIK, these projects have never tried merging into
gdb... and perhaps they should consider it.

-- 
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/

Re: Revised release criteria for GCC 4.0

[Hans-Peter Nilsson]

On Wed, 15 Dec 2004, E. Weddington wrote:
> Ok, sure. But realistically, it's going to be the AVR port developers
> who will be running the testsuite, and they're already familiar with
> avr-libc (I'm excluding the RTEMS folks for this argument).

You're still missing the point that the simtest-howto.html stuff
is generally not for port maintainers but for *other* GCC
developers.

> So, from what I gather with that statment, is that using newlib is just
> a convenience; there's nothing inherent in it that is required for
> testing with a simulator?

Correct.  Of course, you need for newlib to generate and for the
simulator to have special calls for e.g. file operations and
memory allocation.

> Hmm, that's what I was hoping to find out, documentation for those types
> of simulators. I've seen SID <http://sources.redhat.com/sid/> and GGEN
> <http://sources.redhat.com/cgen/>, but does anybody know what is
> currently being used to develop those simulators in the sim directory?,
> what should be used?

If there's already an AVR simulator, you shouldn't start from
scratch; just connect it to the sim framework.  See ppc, arm or
sh simulators for instance.  No documentation, but plenty of
source code.  When starting from scratch and for gcc testing,
CGEN or igen would be better than SID, because SID needs more
than is covered by the simtest-howto.html instructions.

brgds, H-P

Re: Revised release criteria for GCC 4.0

[Bernardo Innocenti]

Hans-Peter Nilsson wrote:
> On Mon, 13 Dec 2004, Bernardo Innocenti wrote:
> 
>>We also have a GDB simulator for the AVR, but I'm afraid nobody ever
>>tried running the Dejagnu testsuite with it.
> 
> 
> I don't see an avr simulator in src/sim.  I do see a m68hc11
> simulator.  For this reason *only*, I'd prefer that any
> few-bits-arch attention be covered by the m68hc11 port
> (or hc12 if you prefer).

Agreed.

> The presence of a turn-key simulator as per simtest-howto.html
> makes it *immensely* more simple for others than hardware
> holders to regression-test their GCC changes, verify bugs and
> fix them.  So all the shouting from AVR enthusiasts is suggested
> be better spent getting an AVR simulator and a newlib port in
> the tree.

The avr target uses avr-libc.  newlib has never
been ported to it and libgloss even fails to compile
when I build in the uberbaum tree.

The number of targets who decided to rewrite their
own libc implementations is simply amazing.

One would expect that rewriting yet another C library
from scratch ought to be a huge waste of time and
probably a boring task.

A possible explanation is that existing libraries
such as glibc are huge and hard to understand.
Other libraries are simply not portable enough
(avr-libc) or designed with some OS in mind (uClibc).

When someone ports GCC to a new embedded processor,
it's easier to cherry pick a few useful ANSI functions
until you can get an hello world program to work.

-- 
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/

Re: Revised release criteria for GCC 4.0

[Ian Lance Taylor]

[ CC list trimmed ]

> So, from what I gather with that statment, is that using newlib is
> just a convenience; there's nothing inherent in it that is required
> for testing with a simulator?

Correct.  In order to run the gcc testsuite, you just need to let the
test harness know whether the program called exit() or abort().

> Hmm, that's what I was hoping to find out, documentation for those
> types of simulators. I've seen SID <http://sources.redhat.com/sid/>
> and GGEN <http://sources.redhat.com/cgen/>, but does anybody know what
> is currently being used to develop those simulators in the sim
> directory?, what should be used?

Some use cgen, some use igen (see sim/igen), some are freecoded, they
should all ideally use the support code in sim/common.  You can
probably get better information on the gdb mailing list.

Ian

Re: Revised release criteria for GCC 4.0

[E. Weddington]

Ian Lance Taylor wrote:

>"E. Weddington" <ericw@evcohs.com> writes:
>
>  
>
>>1. Under simtest-howto.html, it mentions that newlib is required for
>>testing, and you mention a newlib port. Why is newlib required? The
>>avr port is a slightly different beast in that, for the vast majority
>>of users (see more below), it does not use newlib as its C library, it
>>uses avr-libc: <http://savannah.nongnu.org/projects/avr-libc/>, and
>>has done so for a few years. The only recent exception to this is the
>>RTEMS folks who are more interested in a newlib port. I don't know
>>offhand how mature the newlib avr port is.
>>    
>>
>
>The goal is to make it easy for gcc developers to run the testsuite.
>Many gcc developers already know how to work with newlib.  In
>particular, it is already in the uberbaum repository on
>sourceware.org, along with gcc and the binutils.  So somebody can
>simply check out the uberbaum repository and build and test
>everything.  Requiring gcc developers to obtain yet another tool in
>order to do testing makes it that much less likely that anybody will
>actually test.
>
>  
>
Ok, sure. But realistically, it's going to be the AVR port developers 
who will be running the testsuite, and they're already familiar with 
avr-libc (I'm excluding the RTEMS folks for this argument). And since 
the AVR hasn't even made the "secondary" platform status :-), it looks 
like it will continue to be this way for a while.

So, from what I gather with that statment, is that using newlib is just 
a convenience; there's nothing inherent in it that is required for 
testing with a simulator?

>
>The best type of simulator lives in the sim directory in the src
>repository on sourceware.org.  This is part of the gdb distribution.
>This type of simulator can be run standalone or as part of gdb.  I
>don't know what documentation there is on these types of simulators.
>
>  
>
Yes, I've seen those too.
Hmm, that's what I was hoping to find out, documentation for those types 
of simulators. I've seen SID <http://sources.redhat.com/sid/> and GGEN 
<http://sources.redhat.com/cgen/>, but does anybody know what is 
currently being used to develop those simulators in the sim directory?, 
what should be used?

Thanks
Eric

Re: Revised release criteria for GCC 4.0

[Ian Lance Taylor]

"E. Weddington" <ericw@evcohs.com> writes:

> 1. Under simtest-howto.html, it mentions that newlib is required for
> testing, and you mention a newlib port. Why is newlib required? The
> avr port is a slightly different beast in that, for the vast majority
> of users (see more below), it does not use newlib as its C library, it
> uses avr-libc: <http://savannah.nongnu.org/projects/avr-libc/>, and
> has done so for a few years. The only recent exception to this is the
> RTEMS folks who are more interested in a newlib port. I don't know
> offhand how mature the newlib avr port is.

The goal is to make it easy for gcc developers to run the testsuite.
Many gcc developers already know how to work with newlib.  In
particular, it is already in the uberbaum repository on
sourceware.org, along with gcc and the binutils.  So somebody can
simply check out the uberbaum repository and build and test
everything.  Requiring gcc developers to obtain yet another tool in
order to do testing makes it that much less likely that anybody will
actually test.

> 2. You speak of an "AVR simulator". I've seen various scattered
> information about generating a simulator (for embedded targets) and
> usage. Is there some doc that explains what the entire process is on
> putting together a simulator? and where to put it? Pointers welcome. I
> understand that DejaGnu is used to perform the actual testing.

The best type of simulator lives in the sim directory in the src
repository on sourceware.org.  This is part of the gdb distribution.
This type of simulator can be run standalone or as part of gdb.  I
don't know what documentation there is on these types of simulators.

Ian

Re: Revised release criteria for GCC 4.0

[E. Weddington]

Hans-Peter Nilsson wrote:

>On Mon, 13 Dec 2004, Bernardo Innocenti wrote:
>  
>
>>We also have a GDB simulator for the AVR, but I'm afraid nobody ever
>>tried running the Dejagnu testsuite with it.
>>    
>>
>
>I don't see an avr simulator in src/sim.  
>
That is correct. The "simulator" that Bernardo was probably referring to 
is a seperate program: SimulAVR
<http://savannah.nongnu.org/projects/simulavr/>

If I have my terminology correct, simulavr can act as a gdbserver, 
providing a backend to GDB to provide simulation for the AVR.

>The presence of a turn-key simulator as per simtest-howto.html
>makes it *immensely* more simple for others than hardware
>holders to regression-test their GCC changes, verify bugs and
>fix them.  So all the shouting from AVR enthusiasts is suggested
>be better spent getting an AVR simulator and a newlib port in
>the tree.
>
>
>  
>

There have been some recent, intense interest in doing something along 
these lines for the AVR. For example, there has been recent work to get 
an AVR testsuite running for binutils. I understand that SimulAVR above 
is very probably inadequate to run the GCC testsuite in it's current form.

Can you answer some basic questions, though?

1. Under simtest-howto.html, it mentions that newlib is required for 
testing, and you mention a newlib port. Why is newlib required? The avr 
port is a slightly different beast in that, for the vast majority of 
users (see more below), it does not use newlib as its C library, it uses 
avr-libc: <http://savannah.nongnu.org/projects/avr-libc/>, and has done 
so for a few years. The only recent exception to this is the RTEMS folks 
who are more interested in a newlib port. I don't know offhand how 
mature the newlib avr port is.

2. You speak of an "AVR simulator". I've seen various scattered 
information about generating a simulator (for embedded targets) and 
usage. Is there some doc that explains what the entire process is on 
putting together a simulator? and where to put it? Pointers welcome. I 
understand that DejaGnu is used to perform the actual testing.

Thanks
Eric

Re: Revised release criteria for GCC 4.0

[Hans-Peter Nilsson]

On Mon, 13 Dec 2004, Bernardo Innocenti wrote:
> We also have a GDB simulator for the AVR, but I'm afraid nobody ever
> tried running the Dejagnu testsuite with it.

I don't see an avr simulator in src/sim.  I do see a m68hc11
simulator.  For this reason *only*, I'd prefer that any
few-bits-arch attention be covered by the m68hc11 port
(or hc12 if you prefer).

The presence of a turn-key simulator as per simtest-howto.html
makes it *immensely* more simple for others than hardware
holders to regression-test their GCC changes, verify bugs and
fix them.  So all the shouting from AVR enthusiasts is suggested
be better spent getting an AVR simulator and a newlib port in
the tree.

brgds, H-P

Re: Revised release criteria for GCC 4.0

[Ed Smith-Rowland]

Mark Mitchell,

I noticed that Objective-C++ is not mentioned in the language list.  
Does this mean that Objective-C++ is set for 4.1 or was that an 
oversight or that it is a 'secondary' language.

Ed Smith-Rowland

Re: Revised release criteria for GCC 4.0

[Steven Bosscher]

On Tuesday 14 December 2004 21:33, Gerald Pfeifer wrote:
> On Tue, 14 Dec 2004, Steven Bosscher wrote:
> > In fact I think that getting release criteria so late in stage 3
> > is just silly and quite pointless
>
> Since the 4.0 release criteria shall serve as a template for the 4.1
> release criteria, I beg to disagree.

My "pointless" remark may be too strong.  When you have to deriving
release criteria based on an existing baseline (like, when you're in
stage 3 already) I guess there is not much more you can do.  Let me
say I'm quite happy that the secondary and primary platforms have been
updated.

Still, for the upcoming releases we should set goals for the releases
early on.  I realize this is hard.  But people rely on the judgment of
the RM and the SC.  If the release criteria for GCC 4.0 are a template
for GCC 4.1, I suppose our users should not expect too much of GCC 4.1.


> > especially now that they are in such a weak form and not very specific.
>
> In that past, we had much stronger criteria but basically ignored them.

I hope that the RM would just say "No!" to patches that don't satisfy
the release requirements for GCC 4.1.  If the release requirements are
set early enough to, say, judge on a branch being merged or not, it is
my opinion that the RM should have the power to say "No" to a branch
merge when mainline after the branch merge does not satisfy the release
criteria.  This was not possible in the past because release criteria
often were not available until very late in the release cycle.  When
the release criteria do not include performance (compile time and code
quality) requirements, the RM is left without power and I think we all
lose in the end.

Gr.
Steven

Re: Revised release criteria for GCC 4.0

[Gerald Pfeifer]

On Tue, 14 Dec 2004, Steven Bosscher wrote:
> In fact I think that getting release criteria so late in stage 3
> is just silly and quite pointless

Since the 4.0 release criteria shall serve as a template for the 4.1
release criteria, I beg to disagree.

> especially now that they are in such a weak form and not very specific.

In that past, we had much stronger criteria but basically ignored them.

The current, weaker set, is something that we hope to be able to really
take seriously.

> For 4.0 there isn't anything that can be done about it; but could
> the SC/RM set release criteria for 4.1 early on in the development
> cycle?  I really liked the projects list that Mark did before we
> went into stage3.

Fully agreed.

Gerald

Re: Revised release criteria for GCC 4.0

[Mark Mitchell]

Paul Schlie wrote:

>  may not well satisfy. Maybe it would be nice to define a few cross-targets
>  distinct from the existing primary and secondary platforms which include
>  a basic 8-bit and 16/32-bit target, with possibly simplified testing?)

I think that might be useful, but not for 4.0.  I think we've got enough 
to worry about; it will simply have to be up to the AVR maintainers to 
do their best to keep the port working.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[E. Weddington]

On 14 Dec 2004 at 6:27, Ralf Corsepius wrote:

> On Mon, 2004-12-13 at 15:50 -0700, E. Weddington wrote:
> 
> > I'd like to see avr added to that list. Note, however, that the C 
> > library wouldn't have to be built since this is just testing the build 
> > of binutils and gcc. The avr target uses its own library: avr-libc.
> Not quite: bare avr targets use avr-libc, other avr-targets might use
> other libc's (e.g. avr-rtems uses newlib).

Absolutely. Thanks for the correction. :-)
Eric

Re: Revised release criteria for GCC 4.0

[Ralf Corsepius]

On Mon, 2004-12-13 at 15:50 -0700, E. Weddington wrote:

> I'd like to see avr added to that list. Note, however, that the C 
> library wouldn't have to be built since this is just testing the build 
> of binutils and gcc. The avr target uses its own library: avr-libc.
Not quite: bare avr targets use avr-libc, other avr-targets might use
other libc's (e.g. avr-rtems uses newlib).

Ralf

Re: Revised release criteria for GCC 4.0

[Mark Mitchell]

Daniel Kegel wrote:
> Peter Barada wrote:
> 
>>> On Mon, Dec 13, 2004 at 07:26:42PM +0100, Bernardo Innocenti wrote:
>>>
>>>> Fine!  So far, I only had an old Amiga (68060 @ 50MHz) to run the
>>>> testsuite on.  It took ages and the CPU would also occasionally
>>>> overheat, failing tests randomly ;-)
>>>>
>>>> Would it be possible to donate one board to GCC or CodeSourcery
>>>> for the automatic regression tester? 
> 
> 
> I can donate an sh-4 (iirc) target board or two.

Speaking for CodeSourcery, we're happy for people to send us free 
hardware, but we're not going to make any promises whatsoever about 
doing anything at all with it.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[Daniel Kegel]

Peter Barada wrote:
>>On Mon, Dec 13, 2004 at 07:26:42PM +0100, Bernardo Innocenti wrote:
>>
>>>Fine!  So far, I only had an old Amiga (68060 @ 50MHz) to run the
>>>testsuite on.  It took ages and the CPU would also occasionally
>>>overheat, failing tests randomly ;-)
>>>
>>>Would it be possible to donate one board to GCC or CodeSourcery
>>>for the automatic regression tester? 

I can donate an sh-4 (iirc) target board or two.
- Dan

Re: Revised release criteria for GCC 4.0

[Peter Barada]

>On Mon, Dec 13, 2004 at 07:26:42PM +0100, Bernardo Innocenti wrote:
>> Fine!  So far, I only had an old Amiga (68060 @ 50MHz) to run the
>> testsuite on.  It took ages and the CPU would also occasionally
>> overheat, failing tests randomly ;-)
>> 
>> Would it be possible to donate one board to GCC or CodeSourcery
>> for the automatic regression tester?  This would ensure in the
>> future nobody will ever break the m68k target by mistake (at least
>> for ColdFire).
>
>I don't think that this is a fair request to make: CodeSourcery people
>are busy, as are the other GCC developers.
>
>If a volunteer wants to take on responsibility for setting up a local
>copy of the automatic regression tester, I'm sure there are plenty of
>people who would help that volunteer get the regression tester set up.
>People who care about the platform (in this case, ColdFire) are those
>who must pay the primary cost.

Ok, I'll bite.

Who's out there that's done this before for a raw target and can help
to do this?  Also who can tell me how to report the results?

-- 
Peter Barada
peter@the-baradas.com

Re: Revised release criteria for GCC 4.0

[Peter Barada]

>Peter Barada wrote:
>>From my past experience with the m68k-elf target, I've learned that many
>> 
>>>tests assume a full-blown libc and an underlying OS.
>> 
>> 
>> Now that the 5475/5485 ColdFire parts/eval boards exist, there's a
>> full Linux that supports it, and can be used to test it :)
>
>Fine!  So far, I only had an old Amiga (68060 @ 50MHz) to run the
>testsuite on.  It took ages and the CPU would also occasionally
>overheat, failing tests randomly ;-)
>
>Would it be possible to donate one board to GCC or CodeSourcery
>for the automatic regression tester?  This would ensure in the
>future nobody will ever break the m68k target by mistake (at least
>for ColdFire).

I think this is something we vhave to either do ourselves, or convince
people that produce boards and distribute GNU cross-toolchains to
support.  Personally, I was in a bread-line begging for toast to get
the 547x board that I have :)

>> In the case of the other less-powerful parts, can uClinux be used to
>> do testing?  If so, then there are *many* more platforms that can be
>> rigorously tested(assuming people volunteer their time to run the
>> tests on the platforms that they have access to).
>
>There were lots of stability issues with uClinux on a 5272C3 board,
>mostly caused by lack of RAM (1MB free after boot), memory
>fragmentation and corruption (some tests *intentionally* trash
>memory and expect it to cause a segfault!).
>
>Besides, uploding the tests via rcp on a JFFS2 filesystem took even
>longer than executing them.
>
>I should have used a trick with NFS (for which there was no ready
>support in Dejagnu) or upload the tests in a tmpfs (which would have
>fragmented memory even more).

Hmm, I haven't thought of all the issues attached to this :)  I didn't
know that the regression test has tests that actually try to
*crash*.  Perhaps it could be culled down to a list that runs without
requiring a full-blown protective model?  As for the mechanics, I
defer to someone who's actually doing remote testing(alas, that list
doens't include me).

>> As the toolchain evolves, its becoming much harder to build a
>> cross-toolchain, and would be next to impossible without the work of
>> Dan Kegel and others to make crosstool capable of building
>> cross-toolchains.
>
>I do agree.  Building a cross-toolchain in uberbaum would be ideal.
>The biggest obstacle is that few people test it and many libc
>variants are not readily pluggable into the src repository, most
>notably glibc and uClibc.

I build uberbaum for m68k-elf as I'm trying to work on issues relating
to ColdFire.  I haven't yet tried to run the regression test.  I've
looked over Dan's great notes but haven't had any spare time to
try out his method.

I'm sure that there are enough of us that regularly use(and abuse)
some of the cross targets that perhaps we should get together to share
information on how to build/test/verify what we work with.

Who's interested(I am!)?  I have a few ColdFire targets I can beat up
given help from someone who's actually gone through this saga before :)

>Testing with newlib is not an option for most targets!

I figured at absolute *minima*, building newlib for the cross-targets
would be a decent exercise of the cross-compiler....

>> I hope that future development does not preclude
>> the creation of working cross-compilers, and I'd like to see the
>> addition to the release criteria that GCC has to configure/build
>> cross-compilers for a set of targets, and perhaps build up a full
>> cross-toolchain as a way to stress it.
>
>I agree here.  In the long term, building cross-toolchains should
>be automated using the top-level machinery.

I agree with you in principal, but unfortunately the full
cross-toolchain includes components outside of GCC, so coordinating
changes between each of the components will be difficult.  I'd still
like to see a release criteria that identified versions of the cross
components(of binutils, newlib and glibc) that would be used to build
cross-targets just to be sure that they still build.

My biggest fear is that as the list of primary  and secondary targets
shrinks and only includes two raw elf targets(mips/arm), that those of
us that create/use cross-compilers orr the other targets may get left
unintentionally in the dark, and have to patch our way to a working
cross-toolchain.

-- 
Peter Barada
peter@the-baradas.com

Re: Revised release criteria for GCC 4.0

[Mark Mitchell]

Steven Bosscher wrote:
> On Tuesday 14 December 2004 01:03, Mark Mitchell wrote:
> (on compile time not in the criteria...)
> 
>>Partly, that's because this isn't something that's easy to fix right
>>before a release.  If you want to fix it, you have to deal with it
>>during the earlier development stages, when there's more flux.
> 
> 
> Then release criteria should be available before stage3, and more
> people would have to care about it.  Probably the latter is a bit
> more important (volunteers, blahblah).
> In fact I think that getting release criteria so late in stage 3
> is just silly and quite pointless, especially now that they are in
> such a weak form and not very specific.

I agree; we should have done this earlier.  Several people still 
requested it, so I think "pointless" is too strong a statement, but I 
agree that it is less pointful than it would have been done had we done 
it earlier. :-)

> For 4.0 there isn't anything that can be done about it; but could
> the SC/RM set release criteria for 4.1 early on in the development
> cycle?  I really liked the projects list that Mark did before we
> went into stage3.

Yes, we'll be doing that in Stage 1 for 4.1.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[Steven Bosscher]

On Tuesday 14 December 2004 01:03, Mark Mitchell wrote:
(on compile time not in the criteria...)
> Partly, that's because this isn't something that's easy to fix right
> before a release.  If you want to fix it, you have to deal with it
> during the earlier development stages, when there's more flux.

Then release criteria should be available before stage3, and more
people would have to care about it.  Probably the latter is a bit
more important (volunteers, blahblah).
In fact I think that getting release criteria so late in stage 3
is just silly and quite pointless, especially now that they are in
such a weak form and not very specific.

For 4.0 there isn't anything that can be done about it; but could
the SC/RM set release criteria for 4.1 early on in the development
cycle?  I really liked the projects list that Mark did before we
went into stage3.

Gr.
Steven

Re: Revised release criteria for GCC 4.0

[Mark Mitchell]

Benjamin Kosnik wrote:
> hi mark. thanks for the update.
> 
> i must say, the clarification is really nice, but puzzling. 

> 1) platform support 
> any chance you could elaborate on the rationale used
> to pick the primary platforms? the decision is cool, but the process
> behind it would be nice to understand. 

It's a combination of factors.  Yes, some aspects are historical.  One 
reason to include some systems is the "canary" aspect of some systems; 
for example, including systems without weak symbols helps to smoke out 
issues with templates.  We'd all love for all systems to become more 
SVR4-ish, but the SC feels that it's important to retain support for a 
relatively wide variety of systems for the forseeable future.  We wanted 
to continue to support the same major processor families as in previous 
releases, with the exception of Alpha, which has been EOL'd.

> 2) complete dropping of code quality, applications
> WTF?? Why drop the glibc and kernel baselines?? I think these have
> helped in the past to keep initial releases from being of the
> brown-paper bag variety.

We're not dropping code quality as a criterion, we're simply treatig 
code quality regressions as a regression like any other.

As for kernel/application baselines, how many releases have I done where 
(a) I had that baseline data to examine, and (b) the results were good? 
  Zero.

Instead, we're taking the point of view that, realistically, we're not 
going to have that data, but that, fortunately, many people use 
prerelease versions of GCC to test their stuff with, and bugs get 
reported, and so we'll get much of the same information in the form of 
bug reports.

> 3) drop compile time performance as a factor. 

Likewise, compile-time performance regressions are regressions, and 
therefore legitimate issues.

But, I'd be unlikely to hold up an otherwise functional release because 
of some compile-time regressions on some inputs.   (I think that's true 
of most software, other than extremely performance-oriented software; 
for example, I doubt Microsoft would hold up a release of Word because 
it was 15% slower in repaginating certain long documents.)

Partly, that's because this isn't something that's easy to fix right 
before a release.  If you want to fix it, you have to deal with it 
during the earlier development stages, when there's more flux.  It would 
be a bad decision to (say) substantially reorganize a tree data 
structure to save space in the week before a release.  However, adding a 
few lines of code to check for error_mark_node, or to deal with an 
obscure argument-passing problem, is quite reasonable.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304

Re: Revised release criteria for GCC 4.0

[Benjamin Kosnik]

hi mark. thanks for the update.

i must say, the clarification is really nice, but puzzling. 

if these two are compared:

http://gcc.gnu.org/gcc-4.0/criteria.html
vs.
http://gcc.gnu.org/gcc-3.4/criteria.html

a couple of differences are obvious. 

1) platform support 
any chance you could elaborate on the rationale used
to pick the primary platforms? the decision is cool, but the process
behind it would be nice to understand. 

it doesn't seem to be popularity, since i'd think the gcc user base is
x86-linux>powerpc-darwin>=x86-cygwin>any proprietary unix>= any
embedded. 

it doesn't seem to be number of maintainers, as powepc-darwin has more
maintainers than either hpux or aix. 

is it people volunteering to test?

perhaps historical reasons?

note, i don't really care what is primary as long as it includes
x86/linux, just curious.

2) complete dropping of code quality, applications
WTF?? Why drop the glibc and kernel baselines?? I think these have
helped in the past to keep initial releases from being of the
brown-paper bag variety.

it is weird to drop the  codegen requirements right before the release
that introduces the new optimization infrastructure. maybe just me??

3) drop compile time performance as a factor. 
granted this has been dropped de facto from every 3.x series release. so
maybe this is just caving in to reality? i understand this, but it is
sad to see, as it seems to be the #1 complaint from users right now.
dropping it makes the gcc project look like it is ignoring feedback.

help!

benjamin

Re: Revised release criteria for GCC 4.0

[E. Weddington]

Daniel Kegel wrote:

> Bernardo Innocenti wrote:
> > Peter Barada wrote:
>
>>>
>>> I hope that future development does not preclude
>>> the creation of working cross-compilers, and I'd like to see the
>>> addition to the release criteria that GCC has to configure/build
>>> cross-compilers for a set of targets, and perhaps build up a full
>>> cross-toolchain as a way to stress it.
>>
>>
>> I agree here.  In the long term, building cross-toolchains should
>> be automated using the top-level machinery.
>>
>> Building them all with a single script would be great
>
>
> I would be happy to provide a single script tailored for this purpose.
> Total runtime for the script would be under a day on a 1GHz machine, I 
> think.
>
> What's the appropriate list of targets?  My first guess is:
>   arm, i686, ia64, m68k, mips, powerpc750, powerpc970, s390, sh4, x86_64
>
I'd like to see avr added to that list. Note, however, that the C 
library wouldn't have to be built since this is just testing the build 
of binutils and gcc. The avr target uses its own library: avr-libc.

Re: Revised release criteria for GCC 4.0

[Daniel Kegel]

Bernardo Innocenti wrote:
 > Peter Barada wrote:
>> As the toolchain evolves, its becoming much harder to build a
>> cross-toolchain, and would be next to impossible without the work of
>> Dan Kegel and others to make crosstool capable of building
>> cross-toolchains. ...
>> I hope that future development does not preclude
>> the creation of working cross-compilers, and I'd like to see the
>> addition to the release criteria that GCC has to configure/build
>> cross-compilers for a set of targets, and perhaps build up a full
>> cross-toolchain as a way to stress it.
> 
> I agree here.  In the long term, building cross-toolchains should
> be automated using the top-level machinery.
> 
> Building them all with a single script would be great

I would be happy to provide a single script tailored for this purpose.
Total runtime for the script would be under a day on a 1GHz machine, I think.

What's the appropriate list of targets?  My first guess is:
   arm, i686, ia64, m68k, mips, powerpc750, powerpc970, s390, sh4, x86_64

What's the desired output format?  Something similar to
http://kegel.com/crosstool/crosstool-0.28-rc37/buildlogs/0.28/
(but for only one version of gcc/glibc/binutils), I imagine.
- Dan

Re: Revised release criteria for GCC 4.0

[Joe Buck]

On Mon, Dec 13, 2004 at 07:26:42PM +0100, Bernardo Innocenti wrote:
> Fine!  So far, I only had an old Amiga (68060 @ 50MHz) to run the
> testsuite on.  It took ages and the CPU would also occasionally
> overheat, failing tests randomly ;-)
> 
> Would it be possible to donate one board to GCC or CodeSourcery
> for the automatic regression tester?  This would ensure in the
> future nobody will ever break the m68k target by mistake (at least
> for ColdFire).

I don't think that this is a fair request to make: CodeSourcery people
are busy, as are the other GCC developers.

If a volunteer wants to take on responsibility for setting up a local
copy of the automatic regression tester, I'm sure there are plenty of
people who would help that volunteer get the regression tester set up.
People who care about the platform (in this case, ColdFire) are those
who must pay the primary cost.

Re: Revised release criteria for GCC 4.0

[Bernardo Innocenti]

Peter Barada wrote:
>From my past experience with the m68k-elf target, I've learned that many
> 
>>tests assume a full-blown libc and an underlying OS.
> 
> 
> Now that the 5475/5485 ColdFire parts/eval boards exist, there's a
> full Linux that supports it, and can be used to test it :)

Fine!  So far, I only had an old Amiga (68060 @ 50MHz) to run the
testsuite on.  It took ages and the CPU would also occasionally
overheat, failing tests randomly ;-)

Would it be possible to donate one board to GCC or CodeSourcery
for the automatic regression tester?  This would ensure in the
future nobody will ever break the m68k target by mistake (at least
for ColdFire).


> In the case of the other less-powerful parts, can uClinux be used to
> do testing?  If so, then there are *many* more platforms that can be
> rigorously tested(assuming people volunteer their time to run the
> tests on the platforms that they have access to).

There were lots of stability issues with uClinux on a 5272C3 board,
mostly caused by lack of RAM (1MB free after boot), memory
fragmentation and corruption (some tests *intentionally* trash
memory and expect it to cause a segfault!).

Besides, uploding the tests via rcp on a JFFS2 filesystem took even
longer than executing them.

I should have used a trick with NFS (for which there was no ready
support in Dejagnu) or upload the tests in a tmpfs (which would have
fragmented memory even more).


> As the toolchain evolves, its becoming much harder to build a
> cross-toolchain, and would be next to impossible without the work of
> Dan Kegel and others to make crosstool capable of building
> cross-toolchains.

I do agree.  Building a cross-toolchain in uberbaum would be ideal.
The biggest obstacle is that few people test it and many libc
variants are not readily pluggable into the src repository, most
notably glibc and uClibc.

Testing with newlib is not an option for most targets!


> I hope that future development does not preclude
> the creation of working cross-compilers, and I'd like to see the
> addition to the release criteria that GCC has to configure/build
> cross-compilers for a set of targets, and perhaps build up a full
> cross-toolchain as a way to stress it.

I agree here.  In the long term, building cross-toolchains should
be automated using the top-level machinery.

Building them all with a single script would be great.

Sometimes we could even run the testsuite with a GDB
simulator.

-- 
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/

Re: Revised release criteria for GCC 4.0

[Paul Schlie]

> From: Peter Barada <peter@the-baradas.com>
>> From my past experience with the m68k-elf target, I've learned that many
>> tests assume a full-blown libc and an underlying OS.
> 
> Now that the 5475/5485 ColdFire parts/eval boards exist, there's a
> full Linux that supports it, and can be used to test it :)
> 
> In the case of the other less-powerful parts, can uClinux be used to
> do testing?  If so, then there are *many* more platforms that can be
> rigorously tested(assuming people volunteer their time to run the
> tests on the platforms that they have access to).
> 
> As the toolchain evolves, its becoming much harder to build a
> cross-toolchain, and would be next to impossible without the work of
> Dan Kegel and others to make crosstool capable of building
> cross-toolchains.  I hope that future development does not preclude
> the creation of working cross-compilers, and I'd like to see the
> addition to the release criteria that GCC has to configure/build
> cross-compilers for a set of targets, and perhaps build up a full
> cross-toolchain as a way to stress it.

Yes true, but unfortunately 68k/ColdFire derivatives won't tend to expose
large target biases that a smaller 8-bit RISC target would:

- one-unit-per-word (8-bit/QI) char
- two-word (16-bit/HI) int/char*
- limited to a four-word (32-bit/SI/SF) long/long-long/float/double
  (i.e can't practically support DI/DF 64-bit+ modes, as di3 built-in's
   tend to imply the requirement for 24 registers just for the operands
   alone; which overall tends to more aggressively help identify large
   target biases, which may otherwise needlessly not be easily identified.)

(but yes, the AVR is not a "platform" per-se and has very limited memory, it
 will only be capable of supporting a limited number of tests, but likely
 an adequate number to verify the a cross build actually builds and can
 correctly generate basic code, which is likely sufficient given the modest
 goal to help keep GCC's code base as target size neutral as reasonably
 possible; and does not preclude testing of platforms capable of hosting
 uClinux such as 68k/ColdFire for other purposes, which existing targets
 may not well satisfy. Maybe it would be nice to define a few cross-targets
 distinct from the existing primary and secondary platforms which include
 a basic 8-bit and 16/32-bit target, with possibly simplified testing?)


 

Re: Revised release criteria for GCC 4.0

[Peter Barada]

From my past experience with the m68k-elf target, I've learned that many
>tests assume a full-blown libc and an underlying OS.

Now that the 5475/5485 ColdFire parts/eval boards exist, there's a
full Linux that supports it, and can be used to test it :)

In the case of the other less-powerful parts, can uClinux be used to
do testing?  If so, then there are *many* more platforms that can be
rigorously tested(assuming people volunteer their time to run the
tests on the platforms that they have access to).

As the toolchain evolves, its becoming much harder to build a
cross-toolchain, and would be next to impossible without the work of
Dan Kegel and others to make crosstool capable of building
cross-toolchains.  I hope that future development does not preclude
the creation of working cross-compilers, and I'd like to see the
addition to the release criteria that GCC has to configure/build
cross-compilers for a set of targets, and perhaps build up a full
cross-toolchain as a way to stress it.

-- 
Peter Barada
peter@the-baradas.com

Re: Revised release criteria for GCC 4.0

[Bernardo Innocenti]

Paul Schlie wrote:
>>Working with the SC, I have prepared revised release criteria for GCC 4.0,
>>which are available here:
>>
>>http://gcc.gnu.org/gcc-4.0/criteria.html
>>
>>These revisions include changes to the set of primary and secondary platforms
>>to more accurately reflect the platforms currently thought to be important,
>>and also include more realistic goals for validation.
>>
>>Comments are welcome, and we might make changes if there is sufficient
>>momentum in a particular direction. However, I would suggest that you not
>>spend too much energy picking nits; our release criteria are guidelines, not
>>absolutes.
> 
> Although not a primary or secondary platform (which are all relatively
> larger 32/64 bit targets), might it make sense to try to at least include
> one small 8-bit secondary target representative of smaller simpler RISC
> machines (such as AVR); with the objective that the target should at least
> build (and ideally generate reasonably correct, albeit possibly not optimal
> code), in an effort to try to maintain a more reasonably target size neutral
> code base, rather than let unintended large target biases unnecessarily
> manifest themselves into GCC?

I second this proposal.  AVR is frequently broken by middle-end
patches, mostly because it's one of the few targets where int
is 16bit and bigger than a CPU register (8bit).

Besides, given the amount of material and development tools you can find
on the Internet for the AVR, this CPU appears to be *very* widely used.

We also have a GDB simulator for the AVR, but I'm afraid nobody ever
tried running the Dejagnu testsuite with it.

From my past experience with the m68k-elf target, I've learned that many
tests assume a full-blown libc and an underlying OS.

-- 
  // Bernardo Innocenti - Develer S.r.l., R&D dept.
\X/  http://www.develer.com/

Re: Revised release criteria for GCC 4.0

[Paul Schlie]

> Working with the SC, I have prepared revised release criteria for GCC 4.0,
> which are available here:
> 
> http://gcc.gnu.org/gcc-4.0/criteria.html
>
> These revisions include changes to the set of primary and secondary platforms
> to more accurately reflect the platforms currently thought to be important,
> and also include more realistic goals for validation.
>
> Comments are welcome, and we might make changes if there is sufficient
> momentum in a particular direction. However, I would suggest that you not
> spend too much energy picking nits; our release criteria are guidelines, not
> absolutes.

Although not a primary or secondary platform (which are all relatively
larger 32/64 bit targets), might it make sense to try to at least include
one small 8-bit secondary target representative of smaller simpler RISC
machines (such as AVR); with the objective that the target should at least
build (and ideally generate reasonably correct, albeit possibly not optimal
code), in an effort to try to maintain a more reasonably target size neutral
code base, rather than let unintended large target biases unnecessarily
manifest themselves into GCC?