* Mauve Code Coverage
@ 2003-06-25 18:32 Yan Georget
2003-06-26 1:04 ` Stephen Crawley
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Yan Georget @ 2003-06-25 18:32 UTC (permalink / raw)
To: mauve-discuss
Hello,
I've recently discovered the Mauve project.
I was myself thinking about writing tests for Java libraries, but since you
started before...
Anyway, my email concerns code coverage. Do you compute the code coverage of
your test suites?
At Koalog, we have developped a powerful code coverage computation tool
(http://www.koalog.com/php/kover.php).
You can download an evaluation version.
If you like it, and since you are an open-source project, we could grant you
the right to use it with no restrictions.
Let me know what you think.
Best regards,
Yan Georget
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-25 18:32 Mauve Code Coverage Yan Georget
@ 2003-06-26 1:04 ` Stephen Crawley
2003-06-26 8:53 ` Yan Georget
2003-06-26 14:43 ` Mark Wielaard
[not found] ` <200306261718.08841.yan.georget@koalog.com>
2 siblings, 1 reply; 11+ messages in thread
From: Stephen Crawley @ 2003-06-26 1:04 UTC (permalink / raw)
To: yan.georget; +Cc: mauve-discuss
> Anyway, my email concerns code coverage. Do you compute the code coverage of
> your test suites?
AFAIK, no we don't.
> At Koalog, we have developped a powerful code coverage computation tool
> (http://www.koalog.com/php/kover.php).
> You can download an evaluation version.
> If you like it, and since you are an open-source project, we could grant you
> the right to use it with no restrictions.
>
> Let me know what you think.
I think this is a good idea in principle. But a better idea would be
for you to run the coverage tool on Mauve/Classpath/some VM and post the
results. If you could set something up to do this automatically, that
would be even better!
Another point is that coverage testing will only really help if people
volunteer to write good test cases to fill in the gaps.
Finally, coverage testing won't tell us about test cases that are incorrect;
e.g. ones that test for implementation specific behavior, or ones that don't
work against various Sun JDK's. In other words, Mauve's limitations are not
just restricted to coverage.
-- Steve
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-26 1:04 ` Stephen Crawley
@ 2003-06-26 8:53 ` Yan Georget
2003-06-26 11:20 ` Brian Jones
2003-06-26 15:14 ` Stephen Crawley
0 siblings, 2 replies; 11+ messages in thread
From: Yan Georget @ 2003-06-26 8:53 UTC (permalink / raw)
To: Stephen Crawley; +Cc: mauve-discuss
> I think this is a good idea in principle. But a better idea would be
> for you to run the coverage tool on Mauve/Classpath/some VM and post the
> results. If you could set something up to do this automatically, that
> would be even better!
Not sure, it is much more rewarding for developpers (here, test writers) to
run the code coverage themselves than letting a quality engineer (here,
Koalog) do it for them.
> Another point is that coverage testing will only really help if people
> volunteer to write good test cases to fill in the gaps.
That's the point. See above.
> Finally, coverage testing won't tell us about test cases that are
> incorrect; e.g. ones that test for implementation specific behavior, or
> ones that don't work against various Sun JDK's. In other words, Mauve's
> limitations are not just restricted to coverage.
In a usual projet, you check that all the test pass under several environments
(OS and JDK). In your case, you would also check that the coverage results
are the same under these environments.
Computing the code coverage is indeed a simple but powerful way to test the
tests, which is exactly what you want to do here.
What do you think?
Yan
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-26 8:53 ` Yan Georget
@ 2003-06-26 11:20 ` Brian Jones
2003-06-26 15:14 ` Stephen Crawley
1 sibling, 0 replies; 11+ messages in thread
From: Brian Jones @ 2003-06-26 11:20 UTC (permalink / raw)
To: yan.georget; +Cc: Stephen Crawley, mauve-discuss
Yan Georget <yan.georget@koalog.com> writes:
> What do you think?
> Yan
>
I think it's a cool idea. Most of our tests are written in a black
box fashion in that we expect them to pass on any correct
implementation of Java. It could still be enlightening to use the
code coverage tool however.
Brian
--
Brian Jones <cbj@gnu.org>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-26 8:53 ` Yan Georget
2003-06-26 11:20 ` Brian Jones
@ 2003-06-26 15:14 ` Stephen Crawley
2003-06-26 17:36 ` Yan Georget
1 sibling, 1 reply; 11+ messages in thread
From: Stephen Crawley @ 2003-06-26 15:14 UTC (permalink / raw)
To: yan.georget; +Cc: Stephen Crawley, mauve-discuss, crawley
Yan,
I'm being the "Devil's Advocate" here ... bear with me. Basically,
I think the idea is a good one, if we can pull it off.
> > I think this is a good idea in principle. But a better idea would be
> > for you to run the coverage tool on Mauve/Classpath/some VM and post the
> > results. If you could set something up to do this automatically, that
> > would be even better!
>
> Not sure, it is much more rewarding for developpers (here, test writers) to
> run the code coverage themselves than letting a quality engineer (here,
> Koalog) do it for them.
I take your point that it >>could<< be more convenient for testcase
developers to run the coverage tests themselves. But that assumes that
the "barrier to entry" for using the tool isn't too high.
1) I assume that Koalog will want to limit the their offer of free licenses
to "testcase developers". Have you considered how many you will provide,
and your criteria for deciding who to give them to? Will there be much
paperwork?
2) How easy will it be to use the coverage tool? In particular, is it
much work to install the tool? Is it much work to set up the coverage
tool for a particular project? [Could the setup files be CVS'd?]
3) Does the tool cope with calls (and callbacks) from Java and native
code? A lot of this happens in a typical Classpath-based VM.
> > Another point is that coverage testing will only really help if people
> > volunteer to write good test cases to fill in the gaps.
>
> That's the point. See above.
I think you missed >>my<< point. That is that we are desperately short
of people writing testcases at the moment. Take a look at the rate of
Mauve checkins as per the Changelog. A coverage tool won't change this.
(This doesn't mean we shouldn't use a coverage tool ... but lets not
get our expectations too high!)
> > Finally, coverage testing won't tell us about test cases that are
> > incorrect; e.g. ones that test for implementation specific behavior, or
> > ones that don't work against various Sun JDK's. In other words, Mauve's
> > limitations are not just restricted to coverage.
>
> In a usual projet, you check that all the test pass under several environments
> (OS and JDK). In your case, you would also check that the coverage results
> are the same under these environments.
> Computing the code coverage is indeed a simple but powerful way to test the
> tests, which is exactly what you want to do here.
Am I missing something here? Can a Classpath cleanroom developer really do a
coverage analysis of running Mauve testcases against a Sun JDK? Doesn't the
developer need to use Sun source code to do this?
-- Steve
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-26 15:14 ` Stephen Crawley
@ 2003-06-26 17:36 ` Yan Georget
2003-06-26 21:11 ` Tom Tromey
0 siblings, 1 reply; 11+ messages in thread
From: Yan Georget @ 2003-06-26 17:36 UTC (permalink / raw)
To: Stephen Crawley; +Cc: Stephen Crawley, mauve-discuss, crawley
Stephen,
> I take your point that it >>could<< be more convenient for testcase
> developers to run the coverage tests themselves. But that assumes that
> the "barrier to entry" for using the tool isn't too high.
>
> 1) I assume that Koalog will want to limit the their offer of free
> licenses to "testcase developers".
Yes.
> Have you considered how many you will
> provide,
As many as developers.
> and your criteria for deciding who to give them to?
No criteria.
> Will there be
> much paperwork?
No.
> 2) How easy will it be to use the coverage tool? In particular, is it
> much work to install the tool? Is it much work to set up the coverage
> tool for a particular project? [Could the setup files be CVS'd?]
Install is very simple, it takes a couple of seconds.
It is also very easy to set up the tool for a given project.
For more information about these points, look at the user guide.
>
> 3) Does the tool cope with calls (and callbacks) from Java and native
> code? A lot of this happens in a typical Classpath-based VM.
No. To be more precise, KCC will only compute the code coverage for the Java
byte code.
> Am I missing something here? Can a Classpath cleanroom developer really do
> a coverage analysis of running Mauve testcases against a Sun JDK? Doesn't
> the developer need to use Sun source code to do this?
You can use any JDK as long as it is JPDA compliant (Sun and IBM JDKs are).
You don't need any source (since KCC does not instrument the code), you just
need the jar files.
You will need the source to see the result line per line (in KCC GUI only).
Does it make sense?
Yan
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-26 17:36 ` Yan Georget
@ 2003-06-26 21:11 ` Tom Tromey
2003-06-26 23:42 ` Stephen Crawley
0 siblings, 1 reply; 11+ messages in thread
From: Tom Tromey @ 2003-06-26 21:11 UTC (permalink / raw)
To: yan.georget; +Cc: Stephen Crawley, mauve-discuss, crawley
>>>>> "Yan" == Yan Georget <yan.georget@koalog.com> writes:
Yan> You can use any JDK as long as it is JPDA compliant (Sun and IBM
Yan> JDKs are).
Is there any free VM that is JPDA compliant? I don't know of one :-(
If there is one, I'd love to see coverage results for Classpath
libraries when run against Mauve. That would be really useful.
Assuming anybody would use the results to drive Mauve development, not
necessarily a good assumption.
Tom
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Mauve Code Coverage
2003-06-25 18:32 Mauve Code Coverage Yan Georget
2003-06-26 1:04 ` Stephen Crawley
@ 2003-06-26 14:43 ` Mark Wielaard
[not found] ` <200306261718.08841.yan.georget@koalog.com>
2 siblings, 0 replies; 11+ messages in thread
From: Mark Wielaard @ 2003-06-26 14:43 UTC (permalink / raw)
To: yan.georget; +Cc: mauve-discuss
Hi,
On Wed, 2003-06-25 at 19:49, Yan Georget wrote:
> I've recently discovered the Mauve project.
> I was myself thinking about writing tests for Java libraries, but since you
> started before...
Please join the fun! As you can see there are lots of standard classes
without mauve tests.
> Anyway, my email concerns code coverage. Do you compute the code coverage of
> your test suites?
No currently not.
> At Koalog, we have developped a powerful code coverage computation tool
> (http://www.koalog.com/php/kover.php).
> You can download an evaluation version.
> If you like it, and since you are an open-source project, we could grant you
> the right to use it with no restrictions.
It looks interesting but does "right to use it with no restrictions"
mean that you would distribute a version under a free software license?
Does it run on one of the free VMs?
Thanks,
Mark
^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <200306261718.08841.yan.georget@koalog.com>]
end of thread, other threads:[~2003-06-26 23:42 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-06-25 18:32 Mauve Code Coverage Yan Georget
2003-06-26 1:04 ` Stephen Crawley
2003-06-26 8:53 ` Yan Georget
2003-06-26 11:20 ` Brian Jones
2003-06-26 15:14 ` Stephen Crawley
2003-06-26 17:36 ` Yan Georget
2003-06-26 21:11 ` Tom Tromey
2003-06-26 23:42 ` Stephen Crawley
2003-06-26 14:43 ` Mark Wielaard
[not found] ` <200306261718.08841.yan.georget@koalog.com>
[not found] ` <1056642476.3658.109.camel@elsschot>
2003-06-26 16:13 ` Yan Georget
2003-06-26 23:07 ` Dalibor Topic
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).