Announcing eCos Test Farm Results

Announcing eCos Test Farm Results

[Alex Schuilenburg]

Hi All

As promised, the eCos test farm results are now being made available to 
all eCos maintainers.  There is still a fair amount of work being done 
and we are not running on as many publically available platforms as we 
would like since resources are being concentrated on testing commercial 
projects, but you can get a flavour of what is to come at:
   http://bugzilla.ecoscentric.com/tfq.cgi

You will need to login (Mark using RH email address, Gary using MLB and 
Andrew using ASCOM) in order to see the results since this is obviously 
one of our commercial add-ons so viewing of results will be limited to 
our customers, staff and the other maintainers.  The system is 
integrated with bugzilla for obvious reasons, and you will also have a 
"Test Results" link in the bottom RHS of your eCos bugzilla pages when 
you are logged in.

Non-eCosCentric maintainers are also restricted to the public results as 
well as eCosCentric customers (who also can acccess their own platform 
results).  That is, you do not have access to the half-dozen or so 
configuration pages from which perms are constructed, board 
configurations are altered, ordering of which perms are executed, when a 
test tarball is to be made, which platforms it is to run on, which tests 
to run/exclude etc.  However, we will not be unreasonable so speak to an 
eCosCentric maintainer if you have specific requests.  I truly hope you 
find this facility useful.

We have only a few public board results (around 23000+) in the farm from 
a two week old tarball created from anoncvs (known as the trunk) from 
vendors who have been good enough to loan or give us boards.  Not all 
boards have been put in (e.g. PID, cerfcube, pc and laki) since they 
were not a high priority.  Updating is also periodic (not automated 
until we create a formal update strategy).

Andrew - the ebsa285 tests were stalled while we tried to get multiple 
boards being used simultaneously and figured out how to test valid reset 
without a serial connection (we were limited by serial ports and our 
terminal server purchased off eBay is not working too well), so they 
should start catching up now.

There obviously is still a bit of work to do:
* Test result summary pages (i.e. totals of passes, failures, etc)
* Some tests are not being driven correctly (bad error reporting)
* XFAIL result
* Triggers/Alerts

As always, comments welcome.  Lurkers are also invited to review the 
results by creating a bugzilla account and emailing me requesting access.

-- Alex

Re: Announcing eCos Test Farm Results

[Alex Schuilenburg]

Hi again folks

Thanks for all your feedback so far and identifying some of the bogus 
failures reported by the farm.  As expected, we have some fine tuning to 
do :-)

The tfq.cgi script has recently been updated, implementing test 
summaries as well as all your suggestions to date.

To help track issues, we have set up a Test Farm product in the same 
database.  If you find a bug or have an enhacement request, please 
report it in bugzilla at http://bugzilla.ecoscentric.com.

Thanks
-- Alex