Improving mauve (Was: Re: Tests, Documentation)

Improving mauve (Was: Re: Tests, Documentation)

[Dalibor Topic]

[-- Attachment #1: Type: text/plain, Size: 4937 bytes --]

Hallo Sascha,

I'm taking the discussion off the Classpath mailing list, and moving it 
to mauve, where it belongs ;)

Sascha Brawer wrote:

>>test-writing has to be made to feel at least as
>>engaging as code-writing, if it's to be successful.
> 
> 
> I imagine that this will be hard: For most people (including myself),
> it's so much more fun to write code that actually does something.

In the long run, test writing needs to become so straightforward that it 
can be largely replaced by automatically generated tests.

Example: you can often find interesting types of bugs in methods by 
throwing somehow preconditioned input patterns at them, and checking if 
the output/object state still adheres to some post-condition.

The hard way to do this is to do it manually, for every bug pattern we 
observe, in each class it could occur. Which is what mauve currently 
makes us do basically, and I don't know if JUnit is much better in that 
respect.

The nice way to do it would be to have helper methods/classes that

a) allow us to generate interesting (i.e. possibly failure inducing) 
input patters.
b) it should be possible to optionally select only the input that 
matches a pre-condition, since some tests for methods may only make 
sense if the input instances (fixtures) are in some specific state [1].
c) have utility methods for running a test case over all fixtures, a 
subset, all pairs, etc.
d) have utility methods for checking post-conditions.

basically, I want to move away from seeing tests like this in every class:

   FieldPosition fp2 = new FieldPosition(21);
   fp2.setBeginIndex(1999);
   fp2.setEndIndex(2001);
   harness.check(fp.equals(fp2) == true, "equals (true)");

to tests like this:

   harness.testAllPairs(getFieldPositionFixtures(), EqualsTestCase);

or in another class, that also implements equals:

   harness.testAllPairs(getSomeOtherFixtures(), EqualsTestCase);

where the post-condition logic of the equals API (or bug pattern, or 
whatever you need to test) is encoded in the test case, and what's being 
tested is if the 'interesting' input triggers a bug (i.e. an unexpected 
deviation) or not. The nice thing is that interesting input can be 
generated quite easily, in general. Even more so, if you can use 
reflection. [2]

I think we need to separate test input generation from test logic to 
make tests easier to write. My typical test for a new class would look 
something like this:

void testClass(TestClass cl) {
   testConstructors(cl);
   testFields(cl);
   testMethods(cl);
}

void testConstructors(TestClass cl) {
   for each constructor
     try creating an instance with a set of parameters
     test if the result meets post-condition of that constructor
       [ like should throw an exception if input is null ]
}

void testMethods(TestClass cl) {
    for each method
      get a set of fixtures that can be used to test the method
      invoke method on fixtures
      test if the result it meets post-condition of that method
}

I hope that the work in writing the test case can move away from 
grinding out similar test cases, to specifying interesting inputs, pre- 
and post-conditions for the methods in something like TestClass.

It's all a work in progress, as I'm trying to figure out nice, simple 
ways to do this. I've converted the FieldPosition tests into a more 
modular form (attached) [3]. I'll use that to try out my ideas in this 
regard. Critique of what I have so far, and the directions I have in 
mind would be very welcome, as most of you guys have been using mauve 
for years without that much of problem ;)

cheers,
dalibor topic

[1] If I want to test for example how BufferedReader.read() behaves on a 
closed stream, I want to test just with the BufferedReader test fixtures 
that are closed. Testing with others wouldn't be that useful. ;) That's 
the simple part.

The read-should-throw-an-exception-if-reader-is-closed is specified by 
the API of Reader.close(). So what I really want is a generic test for 
Reader.close(), that allows me to pass different types of readers that 
overwrite read() so that I can test if they still comply to the API in 
the test case for close() for the classes extending Reader. Otherwise 
I'd still have to copy the test code for each reader I write the test 
code for.

[2] It shouldn't be too hard to write some fixtureGenerator(Constructor 
cons, Object[] types) method for example, that automatically populates a 
fixture set given a constructor, and an array of primitive types the 
constructor receives, by creating all possible 'interesting' instances 
of fixtures.

[3] As FieldPosition has different (accessible) members depending on the 
API version, I separated 1.1 specific fixtures from those that would 
only be un-equal under 1.2. That resulted in a separation of tests into 
a generic part, and an API release specific part that calls the generic 
part with API release specific fixtures.

[-- Attachment #2: fpos.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 4061 bytes --]