Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

[Laurent GUERBY]

On Wed, 2004-07-21 at 19:45, Mark Mitchell wrote:
> So, I think testing is good, but I think we should keep an eye on how 
> long it takes.  It might be that someone could run these tests nightly 
> instead of all of us running them all the time for a few weeks and we 
> could see how many errors we catch.  If it's more than zero, that would 
> strenghten the argument for using them all the time.

Would it be ok if someone (or me if no one knows directly OSDL people)
fills http://osdl.org/lab_activities/lab_projects/propose_a_project.html
in the name of the GCC project and ask for access to some machines to
set up some kind of GCC patch tester (bootstrap, testsuite plus may be a
few applications)?

IMHO, trying to keep the running time of a testsuite under tight
limits is not really the right thing to do, first it's hard or
impossible to do, and it wastes precious brain cycles that are very
likely to be useful on other tasks, without taking into account the
increased risk of unnoticed problems.

I prefer a set up where the only use for a local build is debugging
and all build & testing is done in a separate and controlled
environment, shared by everyone in the project and with
adequate ressources to do a much better job than local testing.

It would also be fair to all GCC developpers, even those who
cannot afford more than a 486SX with 2MB of RAM or
who don't work on a Free Software company with appropriate
ressources.

Laurent

Re: Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

[Benjamin Kosnik]

I think the testing you are proposing wouldn't hurt, regardless of the
specific patch in question that started this discussion. Indeed, having
a patch server that can validate proposed patches has been a long
discussed feature.

Besides machine resources, this project also needs an administrator, and
some initial setup. Are you suggesting OSDL help with all of these
needs, or just machine resources?

>It would also be fair to all GCC developpers, even those who
>cannot afford more than a 486SX with 2MB of RAM or
>who don't work on a Free Software company with appropriate
>ressources.

Keeping the development process possible, even for people without
hardware resources, is a worthy goal for the gcc project: I would be
surprised if anybody disagreed with this.

-benjamin

Re: Testing GCC & OSDL

[Laurent GUERBY]

On Thu, 2004-07-22 at 18:25, Benjamin Kosnik wrote:
> Besides machine resources, this project also needs an administrator, and
> some initial setup. Are you suggesting OSDL help with all of these
> needs, or just machine resources?

Machines running some Linux distro with incoming ssh enabled should be
enough from OSDL, root access is not needed (IMHO), but if they
have human ressources to spare that would be welcomed :).
Big cheap non networked non backuped IDE disks would help,
I can buy them for OSDL if they don't have some at hand.

Of course I first need to write some stuff, I'd prefer python over
bourne shell or TCL these days, and my home machines will be enough for
this development.

To run it from day to day, with appropriate documentation anyone
should be able to help.

My available time is about 1 hour per business day, and week-ends
when I'm not doing something else.

Laurent

Re: Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

[Hans-Peter Nilsson]

On Wed, 21 Jul 2004, Laurent GUERBY wrote:
> Would it be ok if someone (or me if no one knows directly OSDL people)
> fills http://osdl.org/lab_activities/lab_projects/propose_a_project.html
> in the name of the GCC project and ask for access to some machines to
> set up some kind of GCC patch tester (bootstrap, testsuite plus may be a
> few applications)?

Seemed like a good idea!  What happened to this?

brgds, H-P

Re: Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

[Randy.Dunlap]

On Thu, 16 Sep 2004 22:42:01 -0400 (EDT) Hans-Peter Nilsson wrote:

| On Wed, 21 Jul 2004, Laurent GUERBY wrote:
| > Would it be ok if someone (or me if no one knows directly OSDL people)
| > fills http://osdl.org/lab_activities/lab_projects/propose_a_project.html
| > in the name of the GCC project and ask for access to some machines to
| > set up some kind of GCC patch tester (bootstrap, testsuite plus may be a
| > few applications)?
| 
| Seemed like a good idea!  What happened to this?
| 
| brgds, H-P

Did anyone propose a project?  If so, I can check on it....

--
~Randy

Re: Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

[Laurent GUERBY]

On Fri, 2004-09-17 at 04:46, Randy.Dunlap wrote:
> On Thu, 16 Sep 2004 22:42:01 -0400 (EDT) Hans-Peter Nilsson wrote:
> 
> | On Wed, 21 Jul 2004, Laurent GUERBY wrote:
> | > Would it be ok if someone (or me if no one knows directly OSDL people)
> | > fills http://osdl.org/lab_activities/lab_projects/propose_a_project.html
> | > in the name of the GCC project and ask for access to some machines to
> | > set up some kind of GCC patch tester (bootstrap, testsuite plus may be a
> | > few applications)?
> | 
> | Seemed like a good idea!  What happened to this?
> | 
> | brgds, H-P
> 
> Did anyone propose a project?  If so, I can check on it....

I did not propose a project to OSDL, as mentionned later in the thread

http://gcc.gnu.org/ml/gcc/2004-07/msg01071.html

my available time has been and is currently far too low to handle
administration and the small developments needed.

My offer to buy disks if needed is still valid though,
it is useful IMHO to keep on line bootstraped and installed compilers to
check quickly later a given piece of code over the time - compile time,
run time or behaviour, disk space is kind of cheap these days.

So feel free to go ahead :).

Laurent

Re: Testing GCC & OSDL (Was: RFC: Generated structure layout tests for gcc.dg/compat/struct-layout-1 (take 2))

[Joseph S. Myers]

On Fri, 17 Sep 2004, Laurent GUERBY wrote:

> My offer to buy disks if needed is still valid though,
> it is useful IMHO to keep on line bootstraped and installed compilers to
> check quickly later a given piece of code over the time - compile time,
> run time or behaviour, disk space is kind of cheap these days.

A regression tester that tests every mainline commit rather than batching 
them would be feasible on a small cluster of machines (about 30 commits a 
day to mainline, a bit more if you want to test release branches as well), 
but keeping all the installed compilers from such a process would take 
about 9GB a day (and if you want them available long-term to identify the 
exact commit at which any newly discovered regression came in, this 
storage does need to be backed up).

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
  http://www.srcf.ucam.org/~jsm28/gcc/#c90status - status of C90 for GCC 4.0
    jsm@polyomino.org.uk (personal mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: Testing GCC & OSDL

[Laurent GUERBY]

On Fri, 2004-09-17 at 10:43, Joseph S. Myers wrote:
> A regression tester that tests every mainline commit rather than batching 
> them would be feasible on a small cluster of machines (about 30 commits a 
> day to mainline, a bit more if you want to test release branches as well), 

BTW, is there a script that assumes a local CVS repository (rsync'ed)
and gives you the list of CVS dates in between commits? (so that
"cvs co -D X" for X in this list gives the interesting list of sources)

> but keeping all the installed compilers from such a process would take 
> about 9GB a day (and if you want them available long-term to identify the 
> exact commit at which any newly discovered regression came in, this 
> storage does need to be backed up).

A few choices can mitigate that since we're just caching sequential
builds:

1. keep only one out of N so you still speed up the binary search but
need to finish by building a few compilers

2. if you have N disks, install build number I on disk I modulo N, this
way when you loose one disk you just degrade a bit the situation

3. keep more of the recent or recently needed installs since they're
more likely to be useful

4. compress and/or use binary deltas (documentation, language, platform
and library patches will affect only one part of the installed files),
but you will have to uncompress before use (not much of a problem given
CPU/disk performance ratio these days).

Laurent

Re: Testing GCC & OSDL

[Joseph S. Myers]

On Fri, 17 Sep 2004, Laurent GUERBY wrote:

> On Fri, 2004-09-17 at 10:43, Joseph S. Myers wrote:
> > A regression tester that tests every mainline commit rather than batching 
> > them would be feasible on a small cluster of machines (about 30 commits a 
> > day to mainline, a bit more if you want to test release branches as well), 
> 
> BTW, is there a script that assumes a local CVS repository (rsync'ed)
> and gives you the list of CVS dates in between commits? (so that
> "cvs co -D X" for X in this list gives the interesting list of sources)

Watching gcc-cvs was the most obvious method that occurred to me for such 
a tester to identify changesets.  Things would be easier with a version 
control system that actually had changesets, but I don't think this is a 
major obstacle if the machines were there to devote to testing every 
commit (covering *every* branch would take about twice as much resources 
as just mainline plus release branches); you can approximate changesets 
well enough from CVS.  For extracting past changesets, scripts to convert 
existing repositories to different version control systems (e.g. the one 
that converts CVS to Subversion) may be of use, though I think too slow to 
use in a realtime tester.

> 1. keep only one out of N so you still speed up the binary search but
> need to finish by building a few compilers

We do indeed effectively have something as good as this - Phil's 
Regression Hunter <http://www.devphil.com/~reghunt/> tracks problems down 
to a daily build and the scripts in contrib/reghunt then serve to do a 
binary search (on timestamps rather than changesets) to locate the 
individual patch that caused a problem, and these methods get used to fill 
out regression bugs with details of what caused the regression.

Keeping builds for every changeset would be a minor refinement to speed up 
the process of locating the cause of a newly found regression that may 
have been there for a while.

Having regression testers test every changeset as it is made and report 
regressions within a few hours would mean that people are told of 
regressions their patches have caused while the patches are still fresh in 
their minds, rather than getting a regression report covering many patches 
at once and suspecting it's probably someone else's patch.  (Checkins 
while there's a build failure cause trouble in this regard, but enough 
regressions appear while the tree stays building throughout that I think 
the principle's still useful.)

> 4. compress and/or use binary deltas (documentation, language, platform
> and library patches will affect only one part of the installed files),
> but you will have to uncompress before use (not much of a problem given
> CPU/disk performance ratio these days).

I think this would be a useful practical approach - e.g. keep one in 32 
builds (so approximately one a day for mainline), keep those half way 
inbetween as binary deltas from those 16 before, etc., so a regression 
test would finish with copying an endpoint tree and applying a delta to 
get a midpoint tree to test (five times).  The 1.5GB disk writing involved 
could be saved (reduced to 0.3GB reading of the startpoint tree) by using 
a big enough filesystem in RAM.

Experiment would be needed to determine how much disk space is needed to 
store trees in deltas in this fashion, and how fast regression testing 
could be with this approach.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
  http://www.srcf.ucam.org/~jsm28/gcc/#c90status - status of C90 for GCC 4.0
    jsm@polyomino.org.uk (personal mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: Testing GCC & OSDL

[Daniel Jacobowitz]

On Fri, Sep 17, 2004 at 11:38:56AM +0000, Joseph S. Myers wrote:
> On Fri, 17 Sep 2004, Laurent GUERBY wrote:
> 
> > On Fri, 2004-09-17 at 10:43, Joseph S. Myers wrote:
> > > A regression tester that tests every mainline commit rather than batching 
> > > them would be feasible on a small cluster of machines (about 30 commits a 
> > > day to mainline, a bit more if you want to test release branches as well), 
> > 
> > BTW, is there a script that assumes a local CVS repository (rsync'ed)
> > and gives you the list of CVS dates in between commits? (so that
> > "cvs co -D X" for X in this list gives the interesting list of sources)
> 
> Watching gcc-cvs was the most obvious method that occurred to me for such 
> a tester to identify changesets.  Things would be easier with a version 
> control system that actually had changesets, but I don't think this is a 
> major obstacle if the machines were there to devote to testing every 
> commit (covering *every* branch would take about twice as much resources 
> as just mainline plus release branches); you can approximate changesets 
> well enough from CVS.  For extracting past changesets, scripts to convert 
> existing repositories to different version control systems (e.g. the one 
> that converts CVS to Subversion) may be of use, though I think too slow to 
> use in a realtime tester.

I maintain a repository of changeset data for gcc, src, and glibc. 
Take a look at it, if you want:
  http://return.false.org/~drow/xcvs/gcc.html

The software is a little quirky, but seems to be holding up OK at the
moment.  HEAD is divided into 47,538 changesets.


-- 
Daniel Jacobowitz

Re: Testing GCC & OSDL

[Mike Stump]

On Friday, September 17, 2004, at 04:38  AM, Joseph S. Myers wrote:
> Watching gcc-cvs was the most obvious method that occurred to me for 
> such
> a tester to identify changesets.

Another radical idea, watch gcc-patches.  :-)  This could be in the 
end, better.