Automated build and test system (overview)

Automated build and test system (overview)

[Kris Van Hees]

As many will know, I have been working on an automated build-and-test
system for Frysk (and generic enough to support other projects at will),
to allow us to do regular, automated build-and-test runs in various
environments.  For our initial setup, we have configured an x86 FC6 box
and an x86_64 FC6 box.  Both boxes are updated with the latest FC6
updates every night.

Right now, each box executes the build_farm scripts once every night
(from cron), and often several times during the day from the command
line.

The main objectives for the build farm system are:

	- Any system that is connected to the Internet, and that has the
	  required software package dependencies installed needed to
	  build frysk should be eligible to become part of the build
	  farm.
	- The build farm should not limit itself to just supporting
	  frysk builds.  Virtually any project can be added to it.
	- Systems in the build farm do not all have to build every
	  single project.  This can be controlled per system.
	- The build farm scripts must support unattended operation as
	  much as possible, including automatic updating of the scripts
	  as new ones are released.

The current implementation consists of a single shell script that is
made available for download from any participating build farm system.
The build_farm script is the main driver for the operation of each build
system, and can be tweaked locally when needed.  It's main purpose is to
retrieve the rest of the build farm scripts (made available for download
as a gzip'd tar), extract them, and source the actual build engine
script along with either a host-specific component or a generic one.

The build engine script defines numerous functions that provide the
actual logic of the build.  It also sources a shell script for each
known project.  These shell scripts define things like location to
retrieve the source code from, location where the source code will be
extracted, location where the build will take place (usually not the
place where the source code was extracted, if at all possible), and
optionally some override functions to handle project-specific ways of
handling steps in the build process (more on that later).

After all files are sourced, control moves to the host-specific (or
generic) script.  This script is always very basic, like:

	<-----snip----->
	global_lock

	build_and_test frysk_fresh
	build_and_test frysk_incr

	global_unlock
	<-----snip----->

The global_lock function creates a lock file to ensure that only a
single instance of the build farm script can run at any given time using
a specific environment (usually, just one per system).  The
build_and_test function takes a project name as argument, and triggers a
build-and-test of that project.  Finally, global_unlock releases the
lock, and the script ends.

The build_and_test function sets up project-specific files (for logging,
checksum tracking, and per-project locking), acquires a lock for the
project, and executed the following steps:

	1. Fetch the source code (using cvs, wget, svn, or rsync)
	2. Compute a checksum of the freshly retrieved tree
	3. If nothing changed in the source code (or the build farm
	   scripts), there is no need to build and test, and the
	   function returns
	4. Configure the build of the project
	5. Build the project
	6. Execute the tests for the project

Each step calls a project-specific pre-action function before doing the
actual action, and a post-action function after doing the action, to
allow for customization.  The actions themselves can also be provided as
a project-specific override.

At the completion of the build_and_test function, the logfile will be
"publish" to a defined location (generic or project-specific).  This can
be uploading it using scp, rsync, or moving it to a specific location on
the local system.

The logfile will be used to report on the success (or failure) of the
build, and is analyzed by scripts (not part of the build farm scripts)
to extract the information needed for the reporting.

Builds have been running for a few days now, and have uncovered a few
issues with frysk along the way (so, early successes already).  The
reporting piece is still being worked on, and I am looking into making
the build farm scripts available to the public, so we can have more
people join the build farm system, if they are interested.  The scripts
are written such that client systems pull and push files from and to the
server(s), so no special permissions need be granted to the server.

Any questions? :)

	Cheers,
	Kris

Re: Automated build and test system (overview)

[Mark Wielaard]

Hi Kris,

On Fri, 2007-03-09 at 09:15 -0800, Kris Van Hees wrote:
> The main objectives for the build farm system are:
> 
> 	- Any system that is connected to the Internet, and that has the
> 	  required software package dependencies installed needed to
> 	  build frysk should be eligible to become part of the build
> 	  farm.
> 	- The build farm should not limit itself to just supporting
> 	  frysk builds.  Virtually any project can be added to it.
> 	- Systems in the build farm do not all have to build every
> 	  single project.  This can be controlled per system.
> 	- The build farm scripts must support unattended operation as
> 	  much as possible, including automatic updating of the scripts
> 	  as new ones are released.

Ambitious goals! :)

> The build_and_test function sets up project-specific files (for logging,
> checksum tracking, and per-project locking), acquires a lock for the
> project, and executed the following steps:
> 
> 	1. Fetch the source code (using cvs, wget, svn, or rsync)
> 	2. Compute a checksum of the freshly retrieved tree
> 	3. If nothing changed in the source code (or the build farm
> 	   scripts), there is no need to build and test, and the
> 	   function returns
> 	4. Configure the build of the project
> 	5. Build the project
> 	6. Execute the tests for the project
> 
> Each step calls a project-specific pre-action function before doing the
> actual action, and a post-action function after doing the action, to
> allow for customization.  The actions themselves can also be provided as
> a project-specific override.
> 
> At the completion of the build_and_test function, the logfile will be
> "publish" to a defined location (generic or project-specific).  This can
> be uploading it using scp, rsync, or moving it to a specific location on
> the local system.
> 
> The logfile will be used to report on the success (or failure) of the
> build, and is analyzed by scripts (not part of the build farm scripts)
> to extract the information needed for the reporting.

Do you have any plans on having more "intelligent" parsing of the result
files? Just having the result files published for various architectures
over time will be of great help. Having something that can pinpoint when
which test result flipped from PASS/FAIL would be great.

Also, have you looked at The Gnome BuildBrigade
http://live.gnome.org/BuildBrigade It would be nice to somehow integrate
with that to make us feel more part of the Gnome world. There goals seem
pretty broad though, you might want to concentrate on Frysk first of
course.

Cheers,

Mark

Re: Automated build and test system (overview)

[Stan Cox]

On Mon, 2007-03-12 at 13:23 +0100, Mark Wielaard wrote:
> Having something that can pinpoint when
> which test result flipped from PASS/FAIL would be great. 

Perhaps something like the gcc regression tester which posts regressions
along with ChangeLog entries since the last run:
 http://gcc.gnu.org/ml/gcc-regression
?

Re: Automated build and test system (overview)

[Kris Van Hees]

On Mon, Mar 12, 2007 at 01:23:28PM +0100, Mark Wielaard wrote:
> Hi Kris,
> 
> On Fri, 2007-03-09 at 09:15 -0800, Kris Van Hees wrote:
> > The main objectives for the build farm system are:
> > 
> > 	- Any system that is connected to the Internet, and that has the
> > 	  required software package dependencies installed needed to
> > 	  build frysk should be eligible to become part of the build
> > 	  farm.
> > 	- The build farm should not limit itself to just supporting
> > 	  frysk builds.  Virtually any project can be added to it.
> > 	- Systems in the build farm do not all have to build every
> > 	  single project.  This can be controlled per system.
> > 	- The build farm scripts must support unattended operation as
> > 	  much as possible, including automatic updating of the scripts
> > 	  as new ones are released.
> 
> Ambitious goals! :)

Fortunately, this has been done (a couple of times over in fact) before,
so in the end it isn't too hard to do.  Note that the objectives do not
mention reporting, which is a whole different thing.  Right now, the
scripts support all goals listed above, although the 'unattended
operation' is a bit of a moving target, especially with kernel bugs
getting triggered during frysk builds :)

The inspiration for the build farm system as I wrote it came from the
Samba build farm (http://build.samba.org/) for anyone interested in the
roots of this design.

<<snip>>
> > The logfile will be used to report on the success (or failure) of the
> > build, and is analyzed by scripts (not part of the build farm scripts)
> > to extract the information needed for the reporting.
> 
> Do you have any plans on having more "intelligent" parsing of the result
> files? Just having the result files published for various architectures
> over time will be of great help. Having something that can pinpoint when
> which test result flipped from PASS/FAIL would be great.

I definitely hope to be able to make the reporting more than just
pass/fail kind of information for the various stages.  Given that CVS
does not provide for a project-wide revision id for commits, and instead
handles it on a per-file basis, it becomes a bit more complicated to
report on all changes that happened between two build runs, but it
certainly can be done.  At a minimum, I am working on storing the
current revision of every file in the source tree along with the logs,
so upon a switch from PASS to FAIL, I should be able to list the files
that got changed from the last build until now, along with their
revision id then and now.  Reporting on changelog entries is possible
also, of course, but since I'd need the revision ids for that anyway, I
figure sticking to the ids being reported for now may be less ambitious.

> Also, have you looked at The Gnome BuildBrigade
> http://live.gnome.org/BuildBrigade It would be nice to somehow integrate
> with that to make us feel more part of the Gnome world. There goals seem
> pretty broad though, you might want to concentrate on Frysk first of
> course.

I'll definitely have a look, but for now I think keeping focus on frysk
is crucial.  The reporting piece is still quite a bit of work.

	Cheers,
	Kris

Re: Automated build and test system (overview)

[Andrew Cagney]

GCC's regression tester is a well cited example of how to not report 
test results ;-)   It e-mails everyone that made a change, no matter how 
irrelevant, having the effect of crying wolf :-(

Testing such as being discussed here gives us access to higher level 
trend data.  Examining that data allows us to identify emerging 
problems, for instance constant breakage with a specific compiler, 
kernel breakages.  When applicable we can then adjust our processes do 
address persistent problems.

Andrew

Stan Cox wrote:
> On Mon, 2007-03-12 at 13:23 +0100, Mark Wielaard wrote:
>   
>> Having something that can pinpoint when
>> which test result flipped from PASS/FAIL would be great. 
>>     
>
> Perhaps something like the gcc regression tester which posts regressions
> along with ChangeLog entries since the last run:
>  http://gcc.gnu.org/ml/gcc-regression
> ?
>
>
>   

Re: Automated build and test system (overview)

[Stan Cox]

On Mon, 2007-03-12 at 14:59 -0400, Andrew Cagney wrote:
> having the effect of crying wolf :-(

Agreed, it certainly is trigger-happy.

Re: Automated build and test system (overview)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

Hi Kris,

On Mon, 2007-03-12 at 10:44 -0700, Kris Van Hees wrote:
> > Do you have any plans on having more "intelligent" parsing of the result
> > files? Just having the result files published for various architectures
> > over time will be of great help. Having something that can pinpoint when
> > which test result flipped from PASS/FAIL would be great.
> 
> I definitely hope to be able to make the reporting more than just
> pass/fail kind of information for the various stages.

If we can add requests, is there a simple way for you to set things up
so that test reports are made/compared between a system with selinux
enabled and disabled? I recently saw an somewhat obscure bug in gdb
being reported that depended on a selinux denial happening/not-happening
(and not properly checking a result code). So I thought it would be good
to see differences between test results for frysk also with it
en/disabled.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Automated build and test system (overview)

[Andrew Cagney]

Hi Kris,

To what extent is this system expected to test the less used build / 
test / release paths?

For instance, a deployed frysk release includes its testsuite (funit, 
fsystest, ftail).  We've found lets us quickly ascertain the state of a 
system and identify existing problems - especially by running fsystest.  
Unfortunately developers rarely run the full "make distcheck" or test 
with no build tree, so that is missing some love.

Similarly, it is possible to run additional stress tests - funit 
--stress - these tests are non-deterministic by design and are expected 
to expose new edge cases.  Any problems they identify are reduced to 
more-deterministic unit tests and added to the default test pool.  
Unfortunately, they are only run occasionally.

There's also the option of running << funit --unresolved >> to confirm 
status of the unresolved bugs; although I suspect more useful is to 
check to see if the bug report is still open :-)

Build.samba.org looks like the old tinerbox system for mozilla; being 
able to drill down to the logs using a web interface, and not be swamped 
by daily e-mail, is certainly appreciated.

Andrew