Testing frysk.testbed.Funit*

Testing frysk.testbed.Funit*

[Andrew Cagney]

Hi,

I'm looking at ways to more directly test the frysk.testbed.Funit* 
classes (e.g., FunitExec, DetachedAckProcess) that wrap 
PKGLIBDIR/funit-* utilities, but am finding that the most effective 
route is to use frysk.proc's framework - duplicating the existing 
frysk.proc tests exercising frysk.proc functionality that is effectively 
testing that code.

I could duplicate the tests but it seems redundant.  Any thoughts on a 
strategy?

Andrew

Re: Testing frysk.testbed.Funit*

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Hi Andrew,

On Mon, 2007-07-30 at 11:47 -0400, Andrew Cagney wrote:
> I'm looking at ways to more directly test the frysk.testbed.Funit* 
> classes (e.g., FunitExec, DetachedAckProcess) that wrap 
> PKGLIBDIR/funit-* utilities, but am finding that the most effective 
> route is to use frysk.proc's framework - duplicating the existing 
> frysk.proc tests exercising frysk.proc functionality that is effectively 
> testing that code.
> 
> I could duplicate the tests but it seems redundant.  Any thoughts on a 
> strategy?

I might be missing the exact cases you want to test. But can't you just
audit the current frysk.proc tests to see if they cover all relevant
cases already and if not add one or two tests to the existing proc tests
so all cases are covered? That way you will also extend the real proc
tests to handle more cases catching two birds with one stone (if that
isn't a terribly political incorrect saying).

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Testing frysk.testbed.Funit*

[Andrew Cagney]

Mark Wielaard wrote:
> Hi Andrew,
>
> On Mon, 2007-07-30 at 11:47 -0400, Andrew Cagney wrote:
>   
>> I'm looking at ways to more directly test the frysk.testbed.Funit* 
>> classes (e.g., FunitExec, DetachedAckProcess) that wrap 
>> PKGLIBDIR/funit-* utilities, but am finding that the most effective 
>> route is to use frysk.proc's framework - duplicating the existing 
>> frysk.proc tests exercising frysk.proc functionality that is effectively 
>> testing that code.
>>
>> I could duplicate the tests but it seems redundant.  Any thoughts on a 
>> strategy?
>>     
>
>   
Mark,

you describe the current state of play; frysk.proc code is testing both 
itself internally and the funit tools implicitly.  There's nothing 
directly testing units such as FunitExecOffspring; instead it is done 
implicitly via frysk.proc.  That is great when it works, but not so 
great when tests fail as differentiating between an FunitExecOffspring 
or frysk.proc breakage that caused the fail isn't possible.

As a contrasting example.  Say we find the UI is crashing and track it 
down to a dwarf binding bug.  What we do is add a test-case to the dwarf 
bindings testing the problem (which contains the root cause of the 
problem); and then fix it.  Is the core code is now being tested, we're 
confident that our problem won't return.  Is there an effective way to 
do that here with the Funit* bindings?

Andrew

> I might be missing the exact cases you want to test. But can't you just
> audit the current frysk.proc tests to see if they cover all relevant
> cases already and if not add one or two tests to the existing proc tests
> so all cases are covered? That way you will also extend the real proc
> tests to handle more cases catching two birds with one stone (if that
> isn't a terribly political incorrect saying).
>
> Cheers,
>
> Mark
>   

Re: Testing frysk.testbed.Funit*

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2787 bytes --]

Hi Andrew,

On Tue, 2007-07-31 at 11:57 -0400, Andrew Cagney wrote:
> Mark Wielaard wrote:
> > On Mon, 2007-07-30 at 11:47 -0400, Andrew Cagney wrote:
> >   
> >> I'm looking at ways to more directly test the frysk.testbed.Funit* 
> >> classes (e.g., FunitExec, DetachedAckProcess) that wrap 
> >> PKGLIBDIR/funit-* utilities, but am finding that the most effective 
> >> route is to use frysk.proc's framework - duplicating the existing 
> >> frysk.proc tests exercising frysk.proc functionality that is effectively 
> >> testing that code.
> >>
> >> I could duplicate the tests but it seems redundant.  Any thoughts on a 
> >> strategy?
> >
> > I might be missing the exact cases you want to test. But can't you just
> > audit the current frysk.proc tests to see if they cover all relevant
> > cases already and if not add one or two tests to the existing proc tests
> > so all cases are covered? That way you will also extend the real proc
> > tests to handle more cases catching two birds with one stone (if that
> > isn't a terribly political incorrect saying).
> >
> you describe the current state of play; frysk.proc code is testing both 
> itself internally and the funit tools implicitly.  There's nothing 
> directly testing units such as FunitExecOffspring; instead it is done 
> implicitly via frysk.proc.  That is great when it works, but not so 
> great when tests fail as differentiating between an FunitExecOffspring 
> or frysk.proc breakage that caused the fail isn't possible.

Tests that not just cover the bare essentials, but test the code in
actual use scenarios are very important. It makes sure that the code is
tested as it will actually be used. And in this case, if you find
something not covered, the proc code gets also more tests. As you say
that is great if it works. But I get the feeling that is not enough for
your current strategy.

> As a contrasting example.  Say we find the UI is crashing and track it 
> down to a dwarf binding bug.  What we do is add a test-case to the dwarf 
> bindings testing the problem (which contains the root cause of the 
> problem); and then fix it.  Is the core code is now being tested, we're 
> confident that our problem won't return.  Is there an effective way to 
> do that here with the Funit* bindings?

Right. That is what I am actually suggesting. Make sure that there are
enough tests in proc that you feel confident that everything under funit
is covered. Then if some issue is found anyway later on and it is
tracked down to funit then just add an extra test there when you fix the
problem so you can be confident it won't return. Besides that you have
to fall back on your suggestion, slightly redundant, test duplication I
am afraid.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]