[ECOS] ecos test infrastructure

[ECOS] ecos test infrastructure

[Klaas Gadeyne]

I am currently looking at the (host side) test infrastructure ecos currently
provides and I have several questions to which I couldn't find an
answer in the docs.

I read in several places, e.g.
<http://ecos.sourceware.org/cgi-bin/cvsweb.cgi/ecos/README.host?rev=1.4&content-type=text/x-cvsweb-markup&cvsroot=ecos>
<http://ecos.sourceware.org/docs-latest/cdl-guide/build.tests.html>
that the test infrastructure will be worked on in future ecos
versions.  The last updates to README.host are dating from feb 2003.
I was wondering if anything is to be expected here in the near future
(say a couple of months)?

Furthermore, Chapter 22 of the user-guide describes how to build/run a test
suite from configtool.   When I try that approach, building goes fine,
but when I try to run a test program, I get the same "failed to get
pty error" as described here
<http://sources.redhat.com/ml/ecos-discuss/2004-11/msg00206.html>
However I have the LEGACY_PTYS option enabled in my kernel.  Any suggestions?

Running a single test program from within gdb works fine.
Would it be possible with the current infrastructure to run a
testsuite on the target system without using the graphical configtool
and report the results to a database instead? [*]

Is there any more documentation about the testing infrastructure
(e.g.~about the use of dejagnu) than what I found in user-, ref-, and
component writers guide and the above described README.host?

thx,

Klaas Gadeyne

[*] I guess the test farm at ecoscentric must do something similar...


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] ecos test infrastructure

[Andrew Lunn]

On Mon, Oct 31, 2005 at 04:54:00PM +0100, Klaas Gadeyne wrote:
> I am currently looking at the (host side) test infrastructure ecos currently
> provides and I have several questions to which I couldn't find an
> answer in the docs.
> 
> I read in several places, e.g.
> <http://ecos.sourceware.org/cgi-bin/cvsweb.cgi/ecos/README.host?rev=1.4&content-type=text/x-cvsweb-markup&cvsroot=ecos>
> <http://ecos.sourceware.org/docs-latest/cdl-guide/build.tests.html>
> that the test infrastructure will be worked on in future ecos
> versions.  The last updates to README.host are dating from feb 2003.
> I was wondering if anything is to be expected here in the near future
> (say a couple of months)?

No, i doubt any work will take place here in the near future. It is
one of the selling points of eCosCentric that they have a testfarm. If
they make it too easy for you to run the tests yourself you would not
pay them money for the use of their test farm. Having said that, if
somebody was to contribute code i would gladly incorpertate it, if it
met the usual quality standards. This is open source after all....

 
> Furthermore, Chapter 22 of the user-guide describes how to build/run a test
> suite from configtool.   When I try that approach, building goes fine,
> but when I try to run a test program, I get the same "failed to get
> pty error" as described here
> <http://sources.redhat.com/ml/ecos-discuss/2004-11/msg00206.html>
> However I have the LEGACY_PTYS option enabled in my kernel.  Any 
> suggestions?

That was just a guess. I had seen a similar problem with another
application with 2.6. You need to debug the problem further. Try
running strace and see what configtool is trying to actually use when
it fails. If you can figure that out you might be able to adjust your
kernel. Alternatively you might be able to fix configtool. There are
instructions on how to compile it at 

http://www.ecoscentric.com/devzone/configtool.shtml

I'll warn you thought, it is not easy.

> Running a single test program from within gdb works fine.
> Would it be possible with the current infrastructure to run a
> testsuite on the target system without using the graphical configtool
> and report the results to a database instead? [*]
>
> Is there any more documentation about the testing infrastructure
> (e.g.~about the use of dejagnu) than what I found in user-, ref-, and
> component writers guide and the above described README.host?

Sure it is possible. You can look at the sources for the configtool. I
can also make an educated guess at how it works. I think it will set
breakpoints in cyg_test_exit() & cyg_assert_fail(). The breakpoints
probably have gdb commands attached which causes gdb to exit. gdb will
be started with a script which loads the image, starts login to a
file, sets the breakpoints and then runs the program. I guess there is
also a timeout machanism to detect stuck testcases.
 
> [*] I guess the test farm at ecoscentric must do something similar...

I've never seen eCosCentric's code, since it is there interlectual
property, but i guess it does.

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] ecos test infrastructure

[Alex Schuilenburg]

Andrew Lunn wrote:
>>Running a single test program from within gdb works fine.
>>Would it be possible with the current infrastructure to run a
>>testsuite on the target system without using the graphical configtool
>>and report the results to a database instead? [*]
>>
>>Is there any more documentation about the testing infrastructure
>>(e.g.~about the use of dejagnu) than what I found in user-, ref-, and
>>component writers guide and the above described README.host?
[...]
>>[*] I guess the test farm at ecoscentric must do something similar...
> 
> 
> I've never seen eCosCentric's code, since it is there interlectual
> property, but i guess it does.

It does :-)  Currently at 5,658,576 tests executed and counting with
results easily searchable and selectable by our engineers.

OOI, while the docs refer to dejagnu, while at Cynus and later planet
Red Hat, it became clear to us that while dejagnu is fine for driving
gcc and gdb tests, unfortunately it does not cut the mustard when it
comes to driving eCos tests. Generally, we have found that the
configtool testing capability fulfils the requirements of most
developers whose use anoncvs so have not taken that further.

As for our test farm, eCosCentric have spent a significant amount of
effort and investment developing a fully-automated eCos testing
infrastructure to help us produce eCosPro. It puts eCos through its
paces in ways that the configtool cannot but requires a similar
investment by anyone wanting to run their own test suite. This generally
puts the usefulness of such a tool beyond the scope of anyone other than
those wanting to make a living supporting eCos, which explains in part
why you have not seen it available in anoncvs :-)

-- Alex


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] ecos test infrastructure

[Klaas Gadeyne]

On Mon, 31 Oct 2005, Andrew Lunn wrote:
>> Furthermore, Chapter 22 of the user-guide describes how to build/run a test
>> suite from configtool.   When I try that approach, building goes fine,
>> but when I try to run a test program, I get the same "failed to get
>> pty error" as described here
>> <http://sources.redhat.com/ml/ecos-discuss/2004-11/msg00206.html>
>> However I have the LEGACY_PTYS option enabled in my kernel.  Any
>> suggestions?
>
> That was just a guess. I had seen a similar problem with another
> application with 2.6. You need to debug the problem further. Try
> running strace and see what configtool is trying to actually use when
> it fails.

It appeared to be a problem with unsufficient rights to access the
/dev/pty* files.

thx for your help!

klaas

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss