Availability of Build and Test System

Availability of Build and Test System

[Elena Zannoni]

In the hope that the Sytemtap team finds this useful,
Wenji and Kris have added Systemtap to the projects that are tested 
nightly on
some machines we have here at Oracle. The results are published on this 
website:
http://build.alchar.org/~aedil/rep/

As you can see right now we are testing F7 and FC6.
(BTW the builds on FC6 i386 are failing).
We are also collecting gcov data nightly.

Anybody can contribute to the tests by running some simple scripts on 
their machines,
and the results will be uploaded and displayed on the above web page.
You can read instructions on how to do that, and also about the overall 
architecture
of the build system here: http://build.alchar.org/~aedil/

Note that the current website is temporary while we get a more official 
site
from Oracle. The scripts will also be available shortly on an external 
cvs repo.

In addition to Systemtap the system has been testing Frysk as well, and 
for that project
a nightly summary is posted here: 
http://sourceware.org/ml/frysk-testresults/2007-q2/
Similar reports can be generated for Systemtap and mailed to the mailing 
list.

Kris is the author and maintainer of the build/test system and is 
looking for ideas for improvement,
and comments, and of course, most of all, contributions of testruns. He 
will be happy
to discuss more details about the system, and answer questions.

elena

Re: Availability of Build and Test System

[David Smith]

Elena Zannoni wrote:
> 
> In the hope that the Sytemtap team finds this useful,
> Wenji and Kris have added Systemtap to the projects that are tested 
> nightly on
> some machines we have here at Oracle. The results are published on this 
> website:
> http://build.alchar.org/~aedil/rep/

I took a look at the systemtap results from here:

<http://build.alchar.org/~aedil/rep/ca-tools4/systemtap_fresh.20070614-082235.html>

Several notes:

- This is an output of "make check", not "make installcheck".  "make
check" just performs basic sanity checks, and never actually runs any
probes.  For this system to be useful for systemtap, it is going to need
to run "make installcheck".

- When either "make check" or "make installcheck" is run, 2 output files
are produced: systemtap.sum and systemtap.log.  systemtap.sum contains
basic pass/fail information and systemtap.log contains the full output
needed for debugging.  The build log file
(<http://build.alchar.org/~aedil/rep/ca-tools4/dl/systemtap_fresh.20070614-082235.log.bz2>) 

only contains systemtap.sum information.   To debug most failures, we'll
also need the systemtap.log information.

- From looking at the information that is there in the build log, I'd 
guess the test system doesn't have all the needed packages to run 
systemtap.  Be sure that kernel-devel, kernel-debuginfo, and 
kernel-debuginfo-common are installed.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Availability of Build and Test System

[Kris Van Hees]

On Thu, Jun 14, 2007 at 04:13:04PM -0500, David Smith wrote:
> I took a look at the systemtap results from here:
> 
> <http://build.alchar.org/~aedil/rep/ca-tools4/systemtap_fresh.20070614-082235.html>
> 
> Several notes:
> 
> - This is an output of "make check", not "make installcheck".  "make
> check" just performs basic sanity checks, and never actually runs any
> probes.  For this system to be useful for systemtap, it is going to need
> to run "make installcheck".

I most definitely agree.  The choice to run 'make check' for the current
runs is quite deliberate to ensure that the current worker nodes
(executing the builds) are at minimal risk of crashing.  Of course, the
goal is to run the full testsuite.  But that will need a bit of work at
this point, because crashing the worker nodes is a bit of a problem
right now.  It is an important problem to resolve either way, since e.g.
frysk has accomplished a few machine crashes in the past as well.

Part of this will be trying to capture as much data as possible in the
event of a crash.

> - When either "make check" or "make installcheck" is run, 2 output files
> are produced: systemtap.sum and systemtap.log.  systemtap.sum contains
> basic pass/fail information and systemtap.log contains the full output
> needed for debugging.  The build log file
> (<http://build.alchar.org/~aedil/rep/ca-tools4/dl/systemtap_fresh.20070614-082235.log.bz2>) 
> 
> only contains systemtap.sum information.   To debug most failures, we'll
> also need the systemtap.log information.

Adding in the systemtap.log file won't be a problem.  Right now, the
default is to capture the stdout and stderr output in a single stream,
and upload that.  Obviously, for projects that use dejagnu for their
testing framework, the .sum and .log files will need to be included as
well.

> - From looking at the information that is there in the build log, I'd 
> guess the test system doesn't have all the needed packages to run 
> systemtap.  Be sure that kernel-devel, kernel-debuginfo, and 
> kernel-debuginfo-common are installed.

That box was indeed missing kernel-devel at the time that build and test
run was executed.  That has since been corrected.

Thanks for the feedback.  Suggestions from anyone on issues with the
system are more than welcome, and of course, contributing bulds by
running a worker node (very lightweight and easy) is always welcome
also.

	Cheers,
	Kris

Re: Availability of Build and Test System

[David Smith]

Kris Van Hees wrote:
> On Thu, Jun 14, 2007 at 04:13:04PM -0500, David Smith wrote:
>> - This is an output of "make check", not "make installcheck".  "make
>> check" just performs basic sanity checks, and never actually runs any
>> probes.  For this system to be useful for systemtap, it is going to need
>> to run "make installcheck".
> 
> I most definitely agree.  The choice to run 'make check' for the current
> runs is quite deliberate to ensure that the current worker nodes
> (executing the builds) are at minimal risk of crashing.  Of course, the
> goal is to run the full testsuite.  But that will need a bit of work at
> this point, because crashing the worker nodes is a bit of a problem
> right now.  It is an important problem to resolve either way, since e.g.
> frysk has accomplished a few machine crashes in the past as well.
> 
> Part of this will be trying to capture as much data as possible in the
> event of a crash.

BTW, I forgot to mention that for "make installcheck" to work, "make 
install" has to be run first.  I can't remember if you were doing an 
install or not.

Yep, "make installcheck" can crash the system (although hasn't for me 
personally in a while).  You might check with Will Cohen (who currently 
publishes weekly results) to see what he does for system crashes.  I 
don't know if he just restarts by hand or has an automated system.

Another problem (that you are most likely already aware of) is that with 
just "make check" the gcov results aren't going to be very useful.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)