new automated testing facility now operational

new automated testing facility now operational

[Geoff Keating]

I am now running a service which automatically and continuously tests
th GCC CVS archive whenever a commit is made, and sends mail to the
people making the commit if a regression is detected.

To allow everyone who contributes to GCC to interpret the results and
fix any problems detected, the test does attempt a bootstrap.
Instead, it builds a cross compiler to the target 'powerpc-eabisim'.
The advantage of this is that no matter what hardware is available to
the contributor, they can always build a simulator and run `make check'
for this target, so there's no excuse for not fixing any problems.

The email sent should be self-explanatory.  I should mention some
things about interpreting it, though:

- The email sent shows the complete list of changes to the ChangeLogs
  since the last successful build.  The service has been running for
  about two weeks now in test mode, and the last successful build was
  on the 22nd, so the email has grown to about 50k.  Sorry.  The easy
  way to fix this is to fix the tree.  (I think the tree was OK for a
  few minutes on the 25th or so, but the test program didn't get a
  chance to run before someone introduced another bug.)

- The rate of change on the GCC tree is so high now that, since a
  build and check takes about two hours, there is no way to test each
  patch individually.  In fact, because of limitations in CVS, it is
  becoming increasingly difficult to work out where each patch starts
  and ends.  So the tester will test all the changes since its last
  run in one batch.  If it detects no changes since its last run, it
  will poll ChangeLog files every 15 minutes and run when it finds a
  change; even then, it's quite likely that there will be more than
  one patch committed in a 15-minute window.  So it may not be your
  patch that causes the problem.

- It's also possible for the checker to check out a half-committed patch.
  This is also a limitation in CVS.  I think this will be very rare.

- It's hosted on sparc-sun-solaris2.6, and is using GCC 2.95.2 to
  build the cross-compiler.

- Yes, it's really a regression.  That part of the tester is quite
  reliable.

The address it sends mail to is the one listed in the ChangeLog.  It
will send mail with a 'From' field like

> From: "GCC regression checker" <geoffk@cygnus.com>

you should sort your mail using the 'GCC regression checker' part of
the string, since eventually I'll get it its own e-mail address.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: new automated testing facility now operational

[Alexandre Oliva]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

On Mar 30, 2000, Geoff Keating <geoffk@cygnus.com> wrote:

> I am now running a service which automatically and continuously tests
> th GCC CVS archive whenever a commit is made, and sends mail to the
> people making the commit if a regression is detected.

Cool!  Could it Cc: some mailing list?  gcc-bugs, or some other
created specifically for this purpose?

> Instead, it builds a cross compiler to the target 'powerpc-eabisim'.

It would be nice to have a couple of different targets, instead of
just powerpc-eabi, and we could hold the state of the last build on
each target.

-- 
Alexandre Oliva    Enjoy Guaraná, see http://www.ic.unicamp.br/~oliva/
Cygnus Solutions, a Red Hat company        aoliva@{redhat, cygnus}.com
Free Software Developer and Evangelist    CS PhD student at IC-Unicamp
oliva@{lsd.ic.unicamp.br, gnu.org}   Write to mailing lists, not to me

Re: new automated testing facility now operational

[Geoff Keating]

> Cc: gcc@gcc.gnu.org
> From: Alexandre Oliva <aoliva@cygnus.com>
> Organization: Cygnus Solutions, a Red Hat Company

> On Mar 30, 2000, Geoff Keating <geoffk@cygnus.com> wrote:
> 
> > I am now running a service which automatically and continuously tests
> > th GCC CVS archive whenever a commit is made, and sends mail to the
> > people making the commit if a regression is detected.
> 
> Cool!  Could it Cc: some mailing list?  gcc-bugs, or some other
> created specifically for this purpose?

It should.  I think it should be a new list since the messages are
automated and relatively frequent.

> > Instead, it builds a cross compiler to the target 'powerpc-eabisim'.
> 
> It would be nice to have a couple of different targets, instead of
> just powerpc-eabi, and we could hold the state of the last build on
> each target.

I chose powerpc-eabi because it was the most stable target which had a
freely available simulator.  The other targets that have simulators
are arm, d10v, d30v, erc32, fr30, h8300, h8500, i960, m32r, mcore,
mips, mn10200, mn10300, sh, v860, w65, and z8k; of those, I think
i960, mips, arm, and sh are also reasonable targets to try running
tests with.

At the moment, though, I have no spare machines and would like to
start small, with just one target.  Maybe in a month or two, it can be
extended.  I'd really like to test on a less-RISC chip, perhaps i960?

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: new automated testing facility now operational

[Alexandre Oliva]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Mar 30, 2000, Geoff Keating <geoffk@cygnus.com> wrote:

> I chose powerpc-eabi because it was the most stable target which had a
> freely available simulator.

OTOH, it is one for which the build takes significantly longer than
for any other target I've ever built, because of the huge number of
multilibs.  Which means it runs tests less often.

-- 
Alexandre Oliva    Enjoy Guaraná, see http://www.ic.unicamp.br/~oliva/
Cygnus Solutions, a Red Hat company        aoliva@{redhat, cygnus}.com
Free Software Developer and Evangelist    CS PhD student at IC-Unicamp
oliva@{lsd.ic.unicamp.br, gnu.org}   Write to mailing lists, not to me

Re: new automated testing facility now operational

[Michael Meissner]

On Thu, Mar 30, 2000 at 04:30:02PM -0800, Geoff Keating wrote:
> > Cc: gcc@gcc.gnu.org
> > From: Alexandre Oliva <aoliva@cygnus.com>
> > Organization: Cygnus Solutions, a Red Hat Company
> 
> > On Mar 30, 2000, Geoff Keating <geoffk@cygnus.com> wrote:
> > 
> > > I am now running a service which automatically and continuously tests
> > > th GCC CVS archive whenever a commit is made, and sends mail to the
> > > people making the commit if a regression is detected.
> > 
> > Cool!  Could it Cc: some mailing list?  gcc-bugs, or some other
> > created specifically for this purpose?
> 
> It should.  I think it should be a new list since the messages are
> automated and relatively frequent.
> 
> > > Instead, it builds a cross compiler to the target 'powerpc-eabisim'.
> > 
> > It would be nice to have a couple of different targets, instead of
> > just powerpc-eabi, and we could hold the state of the last build on
> > each target.
> 
> I chose powerpc-eabi because it was the most stable target which had a
> freely available simulator.  The other targets that have simulators
> are arm, d10v, d30v, erc32, fr30, h8300, h8500, i960, m32r, mcore,
> mips, mn10200, mn10300, sh, v860, w65, and z8k; of those, I think
> i960, mips, arm, and sh are also reasonable targets to try running
> tests with.

Well d10v/d30v aren't yet in the FSF tree, and v860 is really v850.  I suspect
the z8k is rather dead.

> At the moment, though, I have no spare machines and would like to
> start small, with just one target.  Maybe in a month or two, it can be
> extended.  I'd really like to test on a less-RISC chip, perhaps i960?
> 
> -- 
> - Geoffrey Keating <geoffk@cygnus.com>

-- 
Michael Meissner, Cygnus Solutions, a Red Hat company.
PMB 198, 174 Littleton Road #3, Westford, Massachusetts 01886, USA
Work:	  meissner@redhat.com		phone: +1 978-486-9304
Non-work: meissner@spectacle-pond.org	fax:   +1 978-692-4482

Re: new automated testing facility now operational

[Geoff Keating]

> From: Alexandre Oliva <aoliva@cygnus.com>
> Organization: Cygnus Solutions, a Red Hat Company
> Date: 30 Mar 2000 21:58:44 -0300

> On Mar 30, 2000, Geoff Keating <geoffk@cygnus.com> wrote:
> 
> > I chose powerpc-eabi because it was the most stable target which had a
> > freely available simulator.
> 
> OTOH, it is one for which the build takes significantly longer than
> for any other target I've ever built, because of the huge number of
> multilibs.  Which means it runs tests less often.

There are only 10 multilibs, and it spends most of its time actually
running the testsuite.  It would be more scary if I had it running
tests on the multilibs.

I think the other targets I suggested have a similar number of
multilibs, except for i960, and ARM (which have many fewer and many
more, respectively).

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: new automated testing facility now operational

[Jeffrey A Law]

  In message < 200003310129.RAA09770@localhost.cygnus.com >you write:
  > > OTOH, it is one for which the build takes significantly longer than
  > > for any other target I've ever built, because of the huge number of
  > > multilibs.  Which means it runs tests less often.
  > 
  > There are only 10 multilibs, and it spends most of its time actually
  > running the testsuite.  It would be more scary if I had it running
  > tests on the multilibs.
  > 
  > I think the other targets I suggested have a similar number of
  > multilibs, except for i960, and ARM (which have many fewer and many
  > more, respectively).
The mn103 only has 2 multilibs (mn103 & am33).  It's not a particularly
interesting target other than it works, has few multilibs and a simulator.

jeff

Re: new automated testing facility now operational

[Gerald Pfeifer]

On Thu, 30 Mar 2000, Geoff Keating wrote:
> I am now running a service which automatically and continuously tests
> th GCC CVS archive whenever a commit is made, and sends mail to the
> people making the commit if a regression is detected.

This is great!

> - The rate of change on the GCC tree is so high now that, since a
>   build and check takes about two hours, there is no way to test each
>   patch individually.

Is the mail going to the committer having triggered the current test run
or to all addresses in the ChangeLog since the last successful build? (I
guess the later would be preferrable.)

A new mailing list @gcc.gnu.org (with a web archive, of course), that is
Cc:ed on these messages would be nice. Have you already talked to Jeff?

Are you planning to make your scripts (publicly) available, so that we can
set this up for a couple of further targets?


Now, once the tree is stable again, I'll have to commit some patch that
breaks the tree, just to see your nice system in action. :-) (Just
kidding, of course!)

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: new automated testing facility now operational

[Geoff Keating]

> Date: Mon, 3 Apr 2000 00:59:26 +0200 (CEST)
> From: Gerald Pfeifer <pfeifer@dbai.tuwien.ac.at>

> > - The rate of change on the GCC tree is so high now that, since a
> >   build and check takes about two hours, there is no way to test each
> >   patch individually.
> 
> Is the mail going to the committer having triggered the current test run
> or to all addresses in the ChangeLog since the last successful build? (I
> guess the later would be preferrable.)

It goes to all the addresses in the ChangeLog since the last (not
necessarily successful) build.

> A new mailing list @gcc.gnu.org (with a web archive, of course), that is
> Cc:ed on these messages would be nice. Have you already talked to Jeff?

Jeff is working on it.

> Are you planning to make your scripts (publicly) available, so that we can
> set this up for a couple of further targets?

There's not much to them, but yes, I do plan to do that.

However, there's a limit to how many of these things you want running.
They place additional load on the CVS server, and it should be
possible to get reasonable coverage with only two or three targets.

The purpose of this wasn't to act as a substitute for people testing
their patches themselves; in fact, the hope is that people will be
shamed into testing better.

In the long run, I'd like to see a policy that all patches should be
tested on some small group of targets, and that any patch which breaks
one of these targets will be rejected or reverted until it is fixed
and tested properly.

> Now, once the tree is stable again, I'll have to commit some patch that
> breaks the tree, just to see your nice system in action. :-) (Just
> kidding, of course!)

Oh, it's not quite stable yet.

-- 
- Geoffrey Keating <geoffk@cygnus.com>

Re: new automated testing facility now operational

[Mike Stump]

Way cool.

Yeah, I've thought we've needed it for 5 years now.  Next one will be
a, after much testing of nominated changes across lots of targets, the
code gets migrated to the next release branch.  You want your code in
the release, you have to pass the test, then, the _system_ puts it in.
Code that fails testing, is automatically moved to a side branch to
allow other code being put in to pass it up.  The more junk that goes
in, the more that is derailed to a side branch.  If nice code goes in,
the system pats you on the back by arranging for it to go into the
release branch.

Need a well tested snapshot, but don't want to be burned, use the
release branch.  Like helping others, cleap up the stuff on the side
branch.

Now, lots of us folks have spare disk and spare cycles.  Might be nice
if you pass dated cvs command lines to others that you otherwise
believe will work, (since you have nice bandwidth into the server),
and have them bash on them to increase coverage.  I'm thinking about
extending the automated stuff out, not doing any of this by hand.

I know I have a complete target lab with about 56 boards in it (28
totally free at the moment), all with ethernet, APC power control for
dejagnu and so on, across 9 major CPU families.  Certainly I have to
already support these whee beasties.  And it benefits me if problems
are found before I `get' to the software.  Ah, in time.

Anyway, talk and toughts are cheap.  Your actions are awesome and much
harder, thanks.