public inbox for frysk@sourceware.org
 help / color / mirror / Atom feed
* Automated build and test system (overview)
@ 2007-03-09 17:15 Kris Van Hees
  2007-03-12 12:23 ` Mark Wielaard
  2007-03-20 17:19 ` Andrew Cagney
  0 siblings, 2 replies; 8+ messages in thread
From: Kris Van Hees @ 2007-03-09 17:15 UTC (permalink / raw)
  To: frysk

As many will know, I have been working on an automated build-and-test
system for Frysk (and generic enough to support other projects at will),
to allow us to do regular, automated build-and-test runs in various
environments.  For our initial setup, we have configured an x86 FC6 box
and an x86_64 FC6 box.  Both boxes are updated with the latest FC6
updates every night.

Right now, each box executes the build_farm scripts once every night
(from cron), and often several times during the day from the command
line.

The main objectives for the build farm system are:

	- Any system that is connected to the Internet, and that has the
	  required software package dependencies installed needed to
	  build frysk should be eligible to become part of the build
	  farm.
	- The build farm should not limit itself to just supporting
	  frysk builds.  Virtually any project can be added to it.
	- Systems in the build farm do not all have to build every
	  single project.  This can be controlled per system.
	- The build farm scripts must support unattended operation as
	  much as possible, including automatic updating of the scripts
	  as new ones are released.

The current implementation consists of a single shell script that is
made available for download from any participating build farm system.
The build_farm script is the main driver for the operation of each build
system, and can be tweaked locally when needed.  It's main purpose is to
retrieve the rest of the build farm scripts (made available for download
as a gzip'd tar), extract them, and source the actual build engine
script along with either a host-specific component or a generic one.

The build engine script defines numerous functions that provide the
actual logic of the build.  It also sources a shell script for each
known project.  These shell scripts define things like location to
retrieve the source code from, location where the source code will be
extracted, location where the build will take place (usually not the
place where the source code was extracted, if at all possible), and
optionally some override functions to handle project-specific ways of
handling steps in the build process (more on that later).

After all files are sourced, control moves to the host-specific (or
generic) script.  This script is always very basic, like:

	<-----snip----->
	global_lock

	build_and_test frysk_fresh
	build_and_test frysk_incr

	global_unlock
	<-----snip----->

The global_lock function creates a lock file to ensure that only a
single instance of the build farm script can run at any given time using
a specific environment (usually, just one per system).  The
build_and_test function takes a project name as argument, and triggers a
build-and-test of that project.  Finally, global_unlock releases the
lock, and the script ends.

The build_and_test function sets up project-specific files (for logging,
checksum tracking, and per-project locking), acquires a lock for the
project, and executed the following steps:

	1. Fetch the source code (using cvs, wget, svn, or rsync)
	2. Compute a checksum of the freshly retrieved tree
	3. If nothing changed in the source code (or the build farm
	   scripts), there is no need to build and test, and the
	   function returns
	4. Configure the build of the project
	5. Build the project
	6. Execute the tests for the project

Each step calls a project-specific pre-action function before doing the
actual action, and a post-action function after doing the action, to
allow for customization.  The actions themselves can also be provided as
a project-specific override.

At the completion of the build_and_test function, the logfile will be
"publish" to a defined location (generic or project-specific).  This can
be uploading it using scp, rsync, or moving it to a specific location on
the local system.

The logfile will be used to report on the success (or failure) of the
build, and is analyzed by scripts (not part of the build farm scripts)
to extract the information needed for the reporting.

Builds have been running for a few days now, and have uncovered a few
issues with frysk along the way (so, early successes already).  The
reporting piece is still being worked on, and I am looking into making
the build farm scripts available to the public, so we can have more
people join the build farm system, if they are interested.  The scripts
are written such that client systems pull and push files from and to the
server(s), so no special permissions need be granted to the server.

Any questions? :)

	Cheers,
	Kris

^ permalink raw reply	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2007-03-20 17:19 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-03-09 17:15 Automated build and test system (overview) Kris Van Hees
2007-03-12 12:23 ` Mark Wielaard
2007-03-12 15:45   ` Stan Cox
2007-03-12 18:59     ` Andrew Cagney
2007-03-12 19:09       ` Stan Cox
2007-03-12 17:45   ` Kris Van Hees
2007-03-17 19:10     ` Mark Wielaard
2007-03-20 17:19 ` Andrew Cagney

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).