ANNOUNCEMENT: Release of the automated build-and-test system

ANNOUNCEMENT: Release of the automated build-and-test system

[Kris Van Hees]

I am happy to announce that the automated build-and-test system is
available for general participation.  This is a major milestone in the
development of this system, because it allows direct (and safe)
contributions to the overall coverage that automated tests can provide
by anyone interested in donating some cycles on their system(s).

You can find more information (and instructions for download and
installation) at:

	http://build.alchar.org/~aedil/

Please note that this system is currently in *beta* mode, so things are
still changing in various places, and the central reporting is not yet
operational (although upload of logs is).  I look forward to comments.
Some questions have already been answered in the FAQ, and as time
progresses I am sure more topics will be added to that FAQ based on
input from everyone.

Also note that the current location of the system is a temporary
solution.  When the system moves to a more permanent location, a single
manual operation on each node will suffice to make that change take
effect.

	Cheers,
	Kris

Re: ANNOUNCEMENT: Release of the automated build-and-test system

[Mark Wielaard]

Hoi Kris,

On Wed, 2007-03-28 at 22:01 -0700, Kris Van Hees wrote:
> I am happy to announce that the automated build-and-test system is
> available for general participation.  This is a major milestone in the
> development of this system, because it allows direct (and safe)
> contributions to the overall coverage that automated tests can provide
> by anyone interested in donating some cycles on their system(s).

Thanks! Doing a test build now.

> You can find more information (and instructions for download and
> installation) at:
> 
> 	http://build.alchar.org/~aedil/
> 
> Please note that this system is currently in *beta* mode, so things are
> still changing in various places, and the central reporting is not yet
> operational (although upload of logs is).  I look forward to comments.

It is pretty generic, but also very centralistic. Could the build
scripts, or at least the frysk_*.pkg files be stored in frysk CVS?
Likewise for mailing/uploading the results. That way a project could
handle its own tests and result processing.

Asking contributors to request a password is an extra step that I would
not add as long as you can identify different build hosts by email
address or ip-address. Lowering barriers to contribute is good. See for
example how gcc does it, you can do an anonymous checkout and then run
the test scripts that will email the results to a public list so
everybody can easily contribute and get at the raw results:
(See the various build scripts and the test_summary script)
http://gcc.gnu.org/viewcvs/trunk/contrib/
http://gcc.gnu.org/ml/gcc-testresults/current/

You say that you keep version information as a list of CVS revision ids.
Wouldn't a date/time suffice? Or just a revision tag in the tree?

BTW. There are no distribution terms on these sources, could you put
them under the GPL?

Cheers,

Mark

Re: ANNOUNCEMENT: Release of the automated build-and-test system

[Phil Muldoon]

Mark Wielaard wrote:
> Hoi Kris,
>
> On Wed, 2007-03-28 at 22:01 -0700, Kris Van Hees wrote:
>   
>> I am happy to announce that the automated build-and-test system is
>> available for general participation.  This is a major milestone in the
>> development of this system, because it allows direct (and safe)
>> contributions to the overall coverage that automated tests can provide
>> by anyone interested in donating some cycles on their system(s).
>>     
>
> Thanks! Doing a test build now.
>   

I did one this evening, very nice!

In addition to what Mark's points are, my only nit is for the addition 
of a -v flag so I can have an indication of what is doing. Perhaps what 
it module it is building/testing. Whether it is a dist/incr/fresh step?

Regards

Phil

Re: ANNOUNCEMENT: Release of the automated build-and-test system

[Kris Van Hees]

Checking in from vacation (hence the late response)...

On Mon, Apr 02, 2007 at 02:43:39PM +0200, Mark Wielaard wrote:
> Hoi Kris,
> 
> On Wed, 2007-03-28 at 22:01 -0700, Kris Van Hees wrote:
> > I am happy to announce that the automated build-and-test system is
> > available for general participation.  This is a major milestone in the
> > development of this system, because it allows direct (and safe)
> > contributions to the overall coverage that automated tests can provide
> > by anyone interested in donating some cycles on their system(s).
> 
> Thanks! Doing a test build now.

Great!  If you are able to commit a few cycles every night or so, please
let me know (and provide me with the uname -a output for the system you
can run it on) and I will send you a password to enable you to upload
results.

> > You can find more information (and instructions for download and
> > installation) at:
> > 
> > 	http://build.alchar.org/~aedil/
> > 
> > Please note that this system is currently in *beta* mode, so things are
> > still changing in various places, and the central reporting is not yet
> > operational (although upload of logs is).  I look forward to comments.
> 
> It is pretty generic, but also very centralistic. Could the build
> scripts, or at least the frysk_*.pkg files be stored in frysk CVS?
> Likewise for mailing/uploading the results. That way a project could
> handle its own tests and result processing.

The system is not meant to be Frysk specific and truly intended to be
outside of Frysk CVS for all of its functions.  If I were to put the
frysk_*.pkg files in Frysk CVS, then people would not be able to use
those files without having the build_farm scripts, and the other way
around.  It is meant to be an independent unit that pulls the source
tree of Frysk (and any other project we define in it), and performs a
build-and-test run on it.  The project (Frysk in this case) is still
responsible for defining tests in its testsuite.  And (of course) there
is also a project specific piece on the reporting server that parses the
project specific data from the logs.  But I do think that there is more
value in keeping that outside of Frysk rather than putting it in Frysk
CVS.  The reporting piece has more dependencies on the build_farm stuff
than it has on any of the supported projects.

Another reason for being centralistic is that reporting is meant to be
done on a central system, where support for a variety of projects can be
coordinated.  The goal is to have an automated build-and-test system
that is very lightweight, easy to deploy, and with minimal dependencies.

> Asking contributors to request a password is an extra step that I would
> not add as long as you can identify different build hosts by email
> address or ip-address. Lowering barriers to contribute is good. See for
> example how gcc does it, you can do an anonymous checkout and then run
> the test scripts that will email the results to a public list so
> everybody can easily contribute and get at the raw results:
> (See the various build scripts and the test_summary script)
> http://gcc.gnu.org/viewcvs/trunk/contrib/
> http://gcc.gnu.org/ml/gcc-testresults/current/

The main reason for putting up a small barrier is that we obviously are
facing potential malicious uploads.  While a password-protection is only
a limited level of security, it does pose enough of a barrier to avoid
the majority of pests.  Given that the central server will provide access
to the results, both in parsed and raw format, everybody should be able
to get to that data at will.  The advantage of using an upload area and
a central reporting server is that we have a place where (rather large)
logs are stored.  If a bug is found in the parsing scripts, it is easy
enough to re-run reporting on older logs.  Doing this on a central
server also allows us to automatically run comparisons between various
build-and-test runs.  In all, it offers (limited) data mining
possibilities that otherwise would only be available if we were to have
nodes mail the full logs, and that would be rather heavy.

> You say that you keep version information as a list of CVS revision ids.
> Wouldn't a date/time suffice? Or just a revision tag in the tree?

Well, perhaps.  However, note that CVS keeps revisions on a per-file
basis, and that commits are not exactly atomic (when multiple files are
commited at the same time).  Given the granularity of date/time stamps,
there is a potential for differences between the tree used for the build
and one to re-create the build.  What we do is actually a combination.
The revision list contains a date/time stamp as well (in UTC), which is
used as the first indicator of a tree revision.  After doing a checkout
using the date/time stamp found in the rev file, the per-file revision
ids are compared between the checked out tree and the rev file, and any
discrepancies are resolved using the revision id from the rev file.
that guarantees that the tree will be identicial to the one used to
generate the rev file.

Using a revision tag in the tree would work if we were having nodes do
coordinated builds, i.e. builds of specific revision tags, like one per
night.  The trouble is that we'd be cluttering up the CVS space with
tons and tons of tags that have only a limited use in time, and we also
end up having nodes perform build-and-test runs on revisions that may
already be known to be bad.  By having every node set their own schedule
for when they run tests, and have them test against the current CVS
HEAD, we cover a lot more ground.

> BTW. There are no distribution terms on these sources, could you put
> them under the GPL?

Thanks for catching this.  The scripts have now been released under the
GPLv2.

	Cheers,
	Kris

Re: ANNOUNCEMENT: Release of the automated build-and-test system

[Kris Van Hees]

On Tue, Apr 03, 2007 at 10:42:43PM -0500, Phil Muldoon wrote:
> I did one this evening, very nice!

Thanks.  Feel free to join us in doing nightly (or more frequent) runs,
and enable logs to get uploaded.  If you are interested, please let me
know.

> In addition to what Mark's points are, my only nit is for the addition 
> of a -v flag so I can have an indication of what is doing. Perhaps what 
> it module it is building/testing. Whether it is a dist/incr/fresh step?

I choose not to include a verbose flag (or any flag for that matter, at
this point) for build_farm.sh because all information is already being
logged anyway.  The build.log file provides a high level view on what is
going on.  Almost all status lines logged (beginning with --) mention
the project name somewhere in them, albeit not in the same place on the
various lines.  Perhaps printing the project name as part of each line
would help?

When you want to see more details on what the build is doing, you can
look at the build.<pkg>.<host>.log file, found in the build_farm
directory.  Those files (one per project) provide you with the
information that I would envision a verbose flag to provide anyway.
Given that it is already printed to a log file, I'm not sure that
duplicating that output in the build.log file adds much value?

	Cheers,
	Kris