Mauve Code Coverage

Mauve Code Coverage

[Yan Georget]

Hello,

I've recently discovered the Mauve project.
I was myself thinking about writing tests for Java libraries, but since you 
started before...

Anyway, my email concerns code coverage. Do you compute the code coverage of 
your test suites?

At Koalog, we have developped a powerful code coverage computation tool 
(http://www.koalog.com/php/kover.php). 
You can download an evaluation version. 
If you like it, and since you are an open-source project, we could grant you 
the right to use it with no restrictions.

Let me know what you think.

Best regards,
Yan Georget

Re: Mauve Code Coverage

[Stephen Crawley]

> Anyway, my email concerns code coverage. Do you compute the code coverage of 
> your test suites?

AFAIK, no we don't.

> At Koalog, we have developped a powerful code coverage computation tool 
> (http://www.koalog.com/php/kover.php). 
> You can download an evaluation version. 
> If you like it, and since you are an open-source project, we could grant you 
> the right to use it with no restrictions.
> 
> Let me know what you think.

I think this is a good idea in principle.  But a better idea would be
for you to run the coverage tool on Mauve/Classpath/some VM and post the
results.  If you could set something up to do this automatically, that
would be even better!

Another point is that coverage testing will only really help if people
volunteer to write good test cases to fill in the gaps.

Finally, coverage testing won't tell us about test cases that are incorrect;
e.g. ones that test for implementation specific behavior, or ones that don't
work against various Sun JDK's.  In other words, Mauve's limitations are not
just restricted to coverage.

-- Steve

Re: Mauve Code Coverage

[Yan Georget]

> I think this is a good idea in principle.  But a better idea would be
> for you to run the coverage tool on Mauve/Classpath/some VM and post the
> results.  If you could set something up to do this automatically, that
> would be even better!

Not sure, it is much more rewarding for developpers (here, test writers) to 
run the code coverage themselves than letting a quality engineer (here, 
Koalog) do it for them.

> Another point is that coverage testing will only really help if people
> volunteer to write good test cases to fill in the gaps.

That's the point. See above.

> Finally, coverage testing won't tell us about test cases that are
> incorrect; e.g. ones that test for implementation specific behavior, or
> ones that don't work against various Sun JDK's.  In other words, Mauve's
> limitations are not just restricted to coverage.

In a usual projet, you check that all the test pass under several environments 
(OS and JDK). In your case, you would also check that the coverage results 
are the same under these environments.
Computing the code coverage is indeed a simple but powerful way to test the 
tests, which is exactly what you want to do here.

What do you think?
Yan

Re: Mauve Code Coverage

[Brian Jones]

Yan Georget <yan.georget@koalog.com> writes:

> What do you think?
> Yan
> 

I think it's a cool idea.  Most of our tests are written in a black
box fashion in that we expect them to pass on any correct
implementation of Java.  It could still be enlightening to use the
code coverage tool however.

Brian
-- 
Brian Jones <cbj@gnu.org>

Re: Mauve Code Coverage

[Mark Wielaard]

Hi,

On Wed, 2003-06-25 at 19:49, Yan Georget wrote:
> I've recently discovered the Mauve project.
> I was myself thinking about writing tests for Java libraries, but since you 
> started before...

Please join the fun! As you can see there are lots of standard classes
without mauve tests.

> Anyway, my email concerns code coverage. Do you compute the code coverage of 
> your test suites?

No currently not.

> At Koalog, we have developped a powerful code coverage computation tool 
> (http://www.koalog.com/php/kover.php). 
> You can download an evaluation version. 
> If you like it, and since you are an open-source project, we could grant you 
> the right to use it with no restrictions.

It looks interesting but does "right to use it with no restrictions"
mean that you would distribute a version under a free software license?
Does it run on one of the free VMs?

Thanks,

Mark

Re: Mauve Code Coverage

[Stephen Crawley]

Yan,

I'm being the "Devil's Advocate" here ... bear with me.  Basically,
I think the idea is a good one, if we can pull it off.

> > I think this is a good idea in principle.  But a better idea would be
> > for you to run the coverage tool on Mauve/Classpath/some VM and post the
> > results.  If you could set something up to do this automatically, that
> > would be even better!
> 
> Not sure, it is much more rewarding for developpers (here, test writers) to 
> run the code coverage themselves than letting a quality engineer (here, 
> Koalog) do it for them.

I take your point that it >>could<< be more convenient for testcase
developers to run the coverage tests themselves.  But that assumes that
the "barrier to entry" for using the tool isn't too high.

1)  I assume that Koalog will want to limit the their offer of free licenses
to "testcase developers".  Have you considered how many you will provide,
and your criteria for deciding who to give them to?  Will there be much
paperwork?

2)  How easy will it be to use the coverage tool?  In particular, is it
much work to install the tool?  Is it much work to set up the coverage
tool for a particular project?  [Could the setup files be CVS'd?]

3)  Does the tool cope with calls (and callbacks) from Java and native
code?  A lot of this happens in a typical Classpath-based VM.

> > Another point is that coverage testing will only really help if people
> > volunteer to write good test cases to fill in the gaps.
> 
> That's the point. See above.

I think you missed >>my<< point.  That is that we are desperately short 
of people writing testcases at the moment.  Take a look at the rate of
Mauve checkins as per the Changelog.  A coverage tool won't change this.
(This doesn't mean we shouldn't use a coverage tool ... but lets not
get our expectations too high!)    

> > Finally, coverage testing won't tell us about test cases that are
> > incorrect; e.g. ones that test for implementation specific behavior, or
> > ones that don't work against various Sun JDK's.  In other words, Mauve's
> > limitations are not just restricted to coverage.
> 
> In a usual projet, you check that all the test pass under several environments 
> (OS and JDK). In your case, you would also check that the coverage results 
> are the same under these environments.
> Computing the code coverage is indeed a simple but powerful way to test the 
> tests, which is exactly what you want to do here.

Am I missing something here?  Can a Classpath cleanroom developer really do a
coverage analysis of running Mauve testcases against a Sun JDK?  Doesn't the
developer need to use Sun source code to do this?

-- Steve

Re: Mauve Code Coverage

[Yan Georget]

> Koalog seems to integrate all kinds of free software like Ant, SkinLF,
> JUnit and Log4J so I hoped you would distribute it under a Free Software
> license.

Actually, we just integrate with Log4J.
KCC easily integrates with Ant, but Ant is not redistributed with KCC and you 
can use KCC without Ant.
Same with SkinLF. BTW we gave a KCC license to the developper of SkinLF.
We just use JUnit internally for our tests.

Since we use free software, our policy has always been to give licenses to the 
people of the free software community who were interested in our products.
In the same spirit, we gave several licenses of our constraint solver to 
universities and students.

> I was not refering to the price, but meant Free Software as
> defined by the Free Software Foundation:
> http://www.gnu.org/philosophy/free-sw.html.

Well, we are a small software company ... we can't let people modify and 
distribute our applications. If we become bigger, we could sell support 
only...

> Since I am working on free software like gcj and GNU Classpath to
> provide users a way to migrate away from proprietary software whenever
> possible I am not comfortable with using non-free products if I can
> avoid it.

I fully understand. 

> The GNU Compiler Collection comes with gcj (GNU Compiler for Java) to
> create native binaries from java source code and gij (GNU Interpreter
> for Java) to interpret java byte code from class or jar files.
> <http://gcc.gnu.org/java/>
>
> And there is Kaffe a traditional JVM which includes a Just In Time
> compiler <http://www.kaffe.org/>

Yes, i know these two.  Maybe, i'll try one of them. Which one do you 
recommend?

Yan

Re: Mauve Code Coverage

[Yan Georget]

Stephen,

> I take your point that it >>could<< be more convenient for testcase
> developers to run the coverage tests themselves.  But that assumes that
> the "barrier to entry" for using the tool isn't too high.
>
> 1)  I assume that Koalog will want to limit the their offer of free
> licenses to "testcase developers".  

Yes.

> Have you considered how many you will
> provide, 

As many as developers.

> and your criteria for deciding who to give them to?  

No criteria.

> Will there be
> much paperwork?

No.

> 2)  How easy will it be to use the coverage tool?  In particular, is it
> much work to install the tool?  Is it much work to set up the coverage
> tool for a particular project?  [Could the setup files be CVS'd?]

Install is very simple, it takes a couple of seconds.
It is also very easy to set up the tool for a given project.
For more information about these points, look at the user guide.

>
> 3)  Does the tool cope with calls (and callbacks) from Java and native
> code?  A lot of this happens in a typical Classpath-based VM.

No. To be more precise, KCC will only compute the code coverage for the Java 
byte code.

> Am I missing something here?  Can a Classpath cleanroom developer really do
> a coverage analysis of running Mauve testcases against a Sun JDK?  Doesn't
> the developer need to use Sun source code to do this?

You can use any JDK as long as it is JPDA compliant (Sun and IBM JDKs are).
You don't need any source (since KCC does not instrument the code), you just 
need the jar files.
You will need the source to see the result line per line (in KCC GUI only).

Does it make sense?

Yan
 

Re: Mauve Code Coverage

[Tom Tromey]

>>>>> "Yan" == Yan Georget <yan.georget@koalog.com> writes:

Yan> You can use any JDK as long as it is JPDA compliant (Sun and IBM
Yan> JDKs are).

Is there any free VM that is JPDA compliant?  I don't know of one :-(

If there is one, I'd love to see coverage results for Classpath
libraries when run against Mauve.  That would be really useful.
Assuming anybody would use the results to drive Mauve development, not
necessarily a good assumption.

Tom

Re: Mauve Code Coverage

[Dalibor Topic]

Salut Yan,

--- Yan Georget <yan.georget@koalog.com> wrote:

> > The GNU Compiler Collection comes with gcj (GNU Compiler for Java) to
> > create native binaries from java source code and gij (GNU Interpreter
> > for Java) to interpret java byte code from class or jar files.
> > <http://gcc.gnu.org/java/>
> >
> > And there is Kaffe a traditional JVM which includes a Just In Time
> > compiler <http://www.kaffe.org/>
> 
> Yes, i know these two.  Maybe, i'll try one of them. Which one do you 
> recommend?

Since we don't have a JPDA implementation yet in kaffe, I doubt that the code
coverage tool would run on kaffe. I'm not aware of anyone working on JPDa for
kaffe yet, but there is a debugger for kaffe (bdbj) that maybe could be used as
a starting point for implementing JPDA.

cheers,
dalibor topic

__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com

Re: Mauve Code Coverage

[Stephen Crawley]

Yan> You can use any JDK as long as it is JPDA compliant (Sun and IBM
Yan> JDKs are).

Tom> Is there any free VM that is JPDA compliant?  I don't know of one :-(

For the record, Kissme is not JPDA compliant.