Participation in Mauve

Participation in Mauve

[Gerrit Ruelens]

Hi,

Here at ACUNIA we are using the Mauve suite to perform tests on our vm
and use the framework to write new tests.  We would like to contribute
our effort back into Mauve project, but along the way we slightly
altered the TestHarness  and ended up with an altered SimpleTestHarness.

1.due to the system we are testing, we  read files not as an
RandomAccessFile but by using getSystemResourceAsStream.  Since this
change is made in the SimpleTestHarness this will be no cause real
problems.
2.We added an abstract function void setclass(String classname) to the
TestHarness.
3.We changed the way how checks handle a String message.  All those
calls are directed to check(boolean b, String msg).  The checkPoint
method isn't called anymore by those methods ... (the checkPoint stays
the same)
4.We changed the methods which reported the output to the screen in the
SimpleTestHarness

The purpose of points 2 to 4 is to generate a uniform message structure.
It is build with the following rule:

<result> $$<classname>$$<checkPoint>$$<String message>$$ <file name>:
<checkPoint> (number <count>)

where:
 <result> PASS or FAIL (nothing changed here)
 <classname> is the class you want to test (set by setclass)
 <checkPoint> indicates the method that is being tested ( set by
checkPoint)
 <String message> is de string you pass to the check method (or default
"no Message")
  this message could be description of the test.
 <file name> is the name of the java class being used ...
 <count> is still the number of checks after a checkPoint ...

This could lead to the following example:

FAIL $$java.lang.String$$toString()String$$wrong value returned$$
gnu.testlet.java.lang.String: toString()String (number 4)

this structure serves for two purposes:
1.if you run a lot of tests you get a nice overview of what is going
wrong. (class, method, description of test)
2.but more important this string can easily be parsed to generate a nice
report.

It is not that hard for us remove all the setclass function calls in our
code (one line in each test class ), but I feel it might be useful to
have such a method.


 Gerrit Ruelens

Re: Participation in Mauve

[Tom Tromey]

>>>>> "Gerrit" == Gerrit Ruelens <gerrit.ruelens@acunia.com> writes:

Gerrit> We would like to contribute our effort back into Mauve
Gerrit> project, but along the way we slightly altered the TestHarness
Gerrit> and ended up with an altered SimpleTestHarness.

One thing to know is that SimpleTestHarness is just one possible test
harness.  You could always write another one that follows the correct
API.  That's what we do in libgcj.

However, that also isn't a requirement...

Gerrit> 1.due to the system we are testing, we read files not as an
Gerrit> RandomAccessFile but by using getSystemResourceAsStream.
Gerrit> Since this change is made in the SimpleTestHarness this will
Gerrit> be no cause real problems.

It might cause problems for our testing.  I'm not positive.

Maybe we can make the behavior conditional on a switch of some kind,
either to configure or to the test harness itself.

I've been thinking of redoing the build process so that decisions are
made by configure and not via the ugly `KEYS'-at-make-time mechanism
we now use.

Gerrit> 2.We added an abstract function void setclass(String
Gerrit> classname) to the TestHarness.

Do you call this with the name of the class you are testing or the
name of the class you are writing?  If the latter we could just
automate this.

Gerrit> 3.We changed the way how checks handle a String message.  All
Gerrit> those calls are directed to check(boolean b, String msg).  The
Gerrit> checkPoint method isn't called anymore by those methods
Gerrit> ... (the checkPoint stays the same)

Why?

Gerrit> 4.We changed the methods which reported the output to the
Gerrit> screen in the SimpleTestHarness

Gerrit> The purpose of points 2 to 4 is to generate a uniform message
Gerrit> structure.

This sounds good.  But could we make it a switch?  I'd like the
default to be something "human readable" (whatever that means -- I'm
ok with changing the default format if you think it is ugly).

Tom

Re: Participation in Mauve

[Gerrit Ruelens]

Tom Tromey wrote:

> >>>>> "Gerrit" == Gerrit Ruelens <gerrit.ruelens@acunia.com> writes:
>
> Gerrit> 1.due to the system we are testing, we read files not as an
> Gerrit> RandomAccessFile but by using getSystemResourceAsStream.
> Gerrit> Since this change is made in the SimpleTestHarness this will
> Gerrit> be no cause real problems.
>
> It might cause problems for our testing.  I'm not positive.

It is no problem to read files as a RandomAccessFile.


> Gerrit> 2.We added an abstract function void setclass(String
> Gerrit> classname) to the TestHarness.
>
> Do you call this with the name of the class you are testing or the
> name of the class you are writing?  If the latter we could just
> automate this.

the classname is the class you are testing. (the name of the class you
are using to test is known by the SimpleTestHarness )


>
> Gerrit> 3.We changed the way how checks handle a String message.  All
> Gerrit> those calls are directed to check(boolean b, String msg).  The
> Gerrit> checkPoint method isn't called anymore by those methods
> Gerrit> ... (the checkPoint stays the same)
>
> Why?

We started changing the SimpleTestHarness so it would be able to generate
output that could be used
to make list of classes that were tested. So we used the verbose option
(we get all the FAIL and PASS).
this was nice but it meant all the information should be passed in the
checkpoint (the name of the class being tested and the method of
the class that is being tested).  But since the class we test is the
same  all through testclas, we added the method setclass to make our life
a bit easier.
I guess I misinterpreted the meaning of passing a string message in a
check ( I tought it should give extra information instead of setting the
checkPoint ).
So we started to use the checkPoint to pass which method we were testing
(with signature).  Unfortunally it meant changing the implementation of
the TestHarness
Finally we used the string we passed to a check to give a message of what
went wrong ...
With all this information we are able to generate output (with "$$" in
the string) which could be parsed (with the grep, cut and sort) to
generate a
file with all the failures and one of what's being tested.  These files
serve as input for a javadoc style program to generate doc pages were is
mentioned
whether as method is tested or not and if it succeeds al tests (if not it
print a message saying what went wrong).


> Gerrit> 4.We changed the methods which reported the output to the
> Gerrit> screen in the SimpleTestHarnes
>
> Gerrit> The purpose of points 2 to 4 is to generate a uniform message
> Gerrit> structure.
>
> This sounds good.  But could we make it a switch?  I'd like the
> default to be something "human readable" (whatever that means -- I'm
> ok with changing the default format if you think it is ugly).
>
> Tom

It is no problom for us to make our code compliant to the API (basicly
removing one line in each class).
There is no need to change the harness,  it is just a suggestion ...

        Gerrit Ruelens

Re: Participation in Mauve

[Tom Tromey]

>>>>> "Gerrit" == Gerrit Ruelens <gerrit.ruelens@acunia.com> writes:

Gerrit> It is no problom for us to make our code compliant to the API
Gerrit> (basicly removing one line in each class).  There is no need
Gerrit> to change the harness, it is just a suggestion ...

Oh, I don't mind the API change.  It isn't a big deal.

Do you know how to use cvs?  And have cvs and ssh installed on your
machine?

I'd like to give you direct write access to the repository so that you
can check in your changes as well as whatever new tests you write.

I'm happy to give this access to anybody in your group who writes
tests (so they don't all have to go through you).

Please fill out this form to get access:

    http://sources.redhat.com/cgi-bin/pdw/ps_form.cgi

It might take a while (currently the requests are processed by hand),
but I'll try to do them very quickly.


There aren't any real rules for doing Mauve checkins.  Basically,
"play nice" -- don't make huge overhauls without talking about it on
the list first, don't check in tests that aren't free software, etc.

Tom