test case, exceptions, and tearDown

test case, exceptions, and tearDown

[Andrew Cagney]

Just FYI,

Todays IRC had a bit of discussion regarding how a JUnit test is written 
(as a style thing). I've found the JUnit faq ( 
http://junit.sourceforge.net/doc/faq/faq.htm ), even though it is using 
version 4 code, makes for a good reference for how the authors intended 
the framework to be used.

Two answers I found useful:

> *How do I write a test that fails when an unexpected exception is 
> thrown?*
>
> Declare the exception in the |throws| clause of the test method and 
> don't catch the exception within the test method. Uncaught exceptions 
> will cause the test to fail with an error.
>
> The following is an example test that fails when the 
> |IndexOutOfBoundsException| is raised:
>
> |
>
>     @Test
>     public void testIndexOutOfBoundsExceptionNotRaised() 
>         throws IndexOutOfBoundsException {
>     
>         ArrayList emptyList = new ArrayList();
>         Object o = emptyList.get(0);
>     }
>       |
Notice how the exception doesn't need to be explicitly caught; instead 
the exception being thrown is interpreted as a FAIL.

and:

> *When are tests garbage collected?*
>
> /(Submitted by: Timothy Wall and Kent Beck)/
>
> By design, the tree of Test instances is built in one pass, then the 
> tests are executed in a second pass. The test runner holds strong 
> references to all Test instances for the duration of the test 
> execution. This means that for a very long test run with many Test 
> instances, none of the tests may be garbage collected until the end of 
> the entire test run.
>
> Therefore, if you allocate external or limited resources in a test, 
> you are responsible for freeing those resources. Explicitly setting an 
> object to |null| in the |tearDown()| method, for example, allows it to 
> be garbage collected before the end of the entire test run.
>
Things involving file-descriptors would certainly fall into that 
category.  Something to keep an eye out for :-)


Andrew

Re: test case, exceptions, and tearDown

[Kris Van Hees]

While I agree in principle with the things you quote below, and in
general with most of the junit FAQ (as I have for many years), I do also
believe that we shouldn't necessarily blindly follow whatever is stated
in the FAQ.  I mean that in very generic terms: the purpose of having a
testsuite is to provide us with clear, consistent, and above all useful
results.

One thing I have always had an issue with concerning junit is the
blending of ERROR and FAIL in junit 4.  I believe that the distinction
between a failed test (assertions failed or expected failure mode (e.g.
exception) occcured) and a test execution that failed (unexpected event
caused the test to not complete as expected, i.e. something made it
impossible to make a correct PASS/FAIL determination).  In that sense, I
do not think it to be wise to always take the strict rule of just
letting exceptions bubble up to the framework.  To me, the most
important word in the FAQ about this topic is "unexpected".  If there is
an exception that is known to validly occur in a failure scenario (it
makes the test fail to satisfy it assertions) then that should be
reflected in the result differently from an unexpected exception that
interferes with the test execution.

There are obviously different schools of thought, but given that Frysk
is already using 2 testing frameworks that are set up to return results
that fall within the POSIX categories (as described in the dejagnu
documentation as reference - discussed in past months), it would make
sense that we make use of the full spectrum of result messages in a
meaningful way.

	Cheers,
	Kris
On Tue, Jul 03, 2007 at 05:52:24PM -0400, Andrew Cagney wrote:
> Just FYI,
> 
> Todays IRC had a bit of discussion regarding how a JUnit test is written 
> (as a style thing). I've found the JUnit faq ( 
> http://junit.sourceforge.net/doc/faq/faq.htm ), even though it is using 
> version 4 code, makes for a good reference for how the authors intended 
> the framework to be used.
> 
> Two answers I found useful:
> 
> >*How do I write a test that fails when an unexpected exception is 
> >thrown?*
> >
> >Declare the exception in the |throws| clause of the test method and 
> >don't catch the exception within the test method. Uncaught exceptions 
> >will cause the test to fail with an error.
> >
> >The following is an example test that fails when the 
> >|IndexOutOfBoundsException| is raised:
> >
> >|
> >
> >    @Test
> >    public void testIndexOutOfBoundsExceptionNotRaised() 
> >        throws IndexOutOfBoundsException {
> >    
> >        ArrayList emptyList = new ArrayList();
> >        Object o = emptyList.get(0);
> >    }
> >      |
> Notice how the exception doesn't need to be explicitly caught; instead 
> the exception being thrown is interpreted as a FAIL.
> 
> and:
> 
> >*When are tests garbage collected?*
> >
> >/(Submitted by: Timothy Wall and Kent Beck)/
> >
> >By design, the tree of Test instances is built in one pass, then the 
> >tests are executed in a second pass. The test runner holds strong 
> >references to all Test instances for the duration of the test 
> >execution. This means that for a very long test run with many Test 
> >instances, none of the tests may be garbage collected until the end of 
> >the entire test run.
> >
> >Therefore, if you allocate external or limited resources in a test, 
> >you are responsible for freeing those resources. Explicitly setting an 
> >object to |null| in the |tearDown()| method, for example, allows it to 
> >be garbage collected before the end of the entire test run.
> >
> Things involving file-descriptors would certainly fall into that 
> category.  Something to keep an eye out for :-)
> 
> 
> Andrew
> 

Re: test case, exceptions, and tearDown

[Andrew Cagney]

Kris,

You rase several interesting points; and as you note there are different 
schools of thought.  Can you point me at the JUnit thread where you 
raised this issue?  I'd be interested in reading the discussion.

JUnit gives us a common framework, and a set of conventions, that is 
proving more than sufficient to our needs.  The only real stumbling 
block we've encountered is that JUnit holds to Java's underlying 
assumption that you yoour test can be written-once and run-everywhere.  
Frysk, being system dependent, doesn't have that luxury and so needs 
ways to identify tests that can't work or have problems in specific 
circumstances; and for that we've kludged up a work-around that draws on 
POSIX and its definition of UNSUPPORTED and UNRESOLVED.

Given the success we're seeing with developers adding JUnit tests, I 
consider this more than sufficient.

Andrew


Kris Van Hees wrote:
> While I agree in principle with the things you quote below, and in
> general with most of the junit FAQ (as I have for many years), I do also
> believe that we shouldn't necessarily blindly follow whatever is stated
> in the FAQ.  I mean that in very generic terms: the purpose of having a
> testsuite is to provide us with clear, consistent, and above all useful
> results.
>
> One thing I have always had an issue with concerning junit is the
> blending of ERROR and FAIL in junit 4.  I believe that the distinction
> between a failed test (assertions failed or expected failure mode (e.g.
> exception) occcured) and a test execution that failed (unexpected event
> caused the test to not complete as expected, i.e. something made it
> impossible to make a correct PASS/FAIL determination).  In that sense, I
> do not think it to be wise to always take the strict rule of just
> letting exceptions bubble up to the framework.  To me, the most
> important word in the FAQ about this topic is "unexpected".  If there is
> an exception that is known to validly occur in a failure scenario (it
> makes the test fail to satisfy it assertions) then that should be
> reflected in the result differently from an unexpected exception that
> interferes with the test execution.
>
> There are obviously different schools of thought, but given that Frysk
> is already using 2 testing frameworks that are set up to return results
> that fall within the POSIX categories (as described in the dejagnu
> documentation as reference - discussed in past months), it would make
> sense that we make use of the full spectrum of result messages in a
> meaningful way.
>   

Re: test case, exceptions, and tearDown

[Kris Van Hees]

I have not raised these issues in any JUnit thrad, nor do I believe
there is much point in doing so.  The JUnit developers have stated their
opinion rather strongly on these issues, and I honestly do not believe
they are open to changing their mind on this.  I had numerous
conversations without organizations about this topic, and the feelings
on both sides of the issue seem to be rather strong.

Sorry that there has not been a more public discussion about this, that
I am aware of.

If a lack of distinction between a test failing due to external
influences (or being non-deterministic) and a recognized exception
triggering a failure is acceptable, the suggested use of junit is of
course sufficient.  Given our situation (needing to deal with more
system specific details), and as you say not being able to depend on a
write-one-run-anywhere situation, I don't think we can simply stick to
the overall recommendations by the junit team.  After all, we are *not*
working within their ideal environment.

Anyway, given that there are clearly two positions on this topic,
majority opinion ought to drive the final decision.  If we are to go
with a folding of ERROR and FAIL, I do want to suggest though that
failing tests are given more attention in view of such failure
potentially being a result of a true problem with the test itself rather
than a reflection of the implemented assertions not passing.

	Cheers,
	Kris

On Thu, Jul 05, 2007 at 11:51:49AM -0400, Andrew Cagney wrote:
> Kris,
> 
> You rase several interesting points; and as you note there are different 
> schools of thought.  Can you point me at the JUnit thread where you 
> raised this issue?  I'd be interested in reading the discussion.
> 
> JUnit gives us a common framework, and a set of conventions, that is 
> proving more than sufficient to our needs.  The only real stumbling 
> block we've encountered is that JUnit holds to Java's underlying 
> assumption that you yoour test can be written-once and run-everywhere.  
> Frysk, being system dependent, doesn't have that luxury and so needs 
> ways to identify tests that can't work or have problems in specific 
> circumstances; and for that we've kludged up a work-around that draws on 
> POSIX and its definition of UNSUPPORTED and UNRESOLVED.
> 
> Given the success we're seeing with developers adding JUnit tests, I 
> consider this more than sufficient.
> 
> Andrew
> 
> 
> Kris Van Hees wrote:
> >While I agree in principle with the things you quote below, and in
> >general with most of the junit FAQ (as I have for many years), I do also
> >believe that we shouldn't necessarily blindly follow whatever is stated
> >in the FAQ.  I mean that in very generic terms: the purpose of having a
> >testsuite is to provide us with clear, consistent, and above all useful
> >results.
> >
> >One thing I have always had an issue with concerning junit is the
> >blending of ERROR and FAIL in junit 4.  I believe that the distinction
> >between a failed test (assertions failed or expected failure mode (e.g.
> >exception) occcured) and a test execution that failed (unexpected event
> >caused the test to not complete as expected, i.e. something made it
> >impossible to make a correct PASS/FAIL determination).  In that sense, I
> >do not think it to be wise to always take the strict rule of just
> >letting exceptions bubble up to the framework.  To me, the most
> >important word in the FAQ about this topic is "unexpected".  If there is
> >an exception that is known to validly occur in a failure scenario (it
> >makes the test fail to satisfy it assertions) then that should be
> >reflected in the result differently from an unexpected exception that
> >interferes with the test execution.
> >
> >There are obviously different schools of thought, but given that Frysk
> >is already using 2 testing frameworks that are set up to return results
> >that fall within the POSIX categories (as described in the dejagnu
> >documentation as reference - discussed in past months), it would make
> >sense that we make use of the full spectrum of result messages in a
> >meaningful way.
> >  
> 

Re: test case, exceptions, and tearDown

[Andrew Cagney]

Kris Van Hees wrote:
> Anyway, given that there are clearly two positions on this topic,
> majority opinion ought to drive the final decision.  If we are to go
> with a folding of ERROR and FAIL, I do want to suggest though that
> failing tests are given more attention in view of such failure
> potentially being a result of a true problem with the test itself rather
> than a reflection of the implemented assertions not passing.
>   
To true; for my part I've already closed out two Fedora 7 failures.

Andrew