public inbox for mauve-discuss@sourceware.org
 help / color / mirror / Atom feed
* inconsistent test names
@ 2005-11-18 19:08 Edwin Steiner
  2005-11-24 21:50 ` Mark Wielaard
  2005-11-25 19:28 ` Tom Tromey
  0 siblings, 2 replies; 3+ messages in thread
From: Edwin Steiner @ 2005-11-18 19:08 UTC (permalink / raw)
  To: mauve-discuss

Hello!

I'm one of the developers of the cacao jvm[1]. We use mauve
intensively to test cacao and it is a very good test suite.

Currently I'm developing an automated testing framework for
cacao[2]. You can see preliminary results here (large page!):

http://www.complang.tuwien.ac.at/cacaojvm//tgolem/2005-11-18T03-22/mauve.html

One problem I've run accross is that some mauve tests use
inconsistent names to report PASSes and FAILs. The most troublesome
cases are those, where a *named* check/fail is followed by an
*unnamed* one. For example:

gnu/testlet/java/net/MulticastSocket/MulticastSocketTest.java:207

there is a

    harness.fail("could not create FIRST multicast socket on shared port
    " + sharedMcastPort);

at this line. The next check on line 216 and unnamed. Now if the first check fails,
and the second succeeds, you get a bogus PASS line like:

PASS: gnu.testlet.java.net.MulticastSocket.MulticastSocketTest: could not
create SECOND multicast socket on shared port 1234 (number 2)

from the harness.

If you also think these cases should be fixed (by naming the checks),
I would be happy to provide patches. Makes more sense than teaching
my tgolem to work around these cases.

-Edwin

[1] http://www.cacaojvm.org/
[2] http://www.cacaojvm.org/tgolem
    Most of the tgolem code is not cacao specific and I plan
    to release it as a GPL project within the next months.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: inconsistent test names
  2005-11-18 19:08 inconsistent test names Edwin Steiner
@ 2005-11-24 21:50 ` Mark Wielaard
  2005-11-25 19:28 ` Tom Tromey
  1 sibling, 0 replies; 3+ messages in thread
From: Mark Wielaard @ 2005-11-24 21:50 UTC (permalink / raw)
  To: Edwin Steiner; +Cc: mauve-discuss

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

Hi Edwin,

On Fri, 2005-11-18 at 20:08 +0100, Edwin Steiner wrote:
> Currently I'm developing an automated testing framework for
> cacao[2]. You can see preliminary results here (large page!):
> 
> http://www.complang.tuwien.ac.at/cacaojvm//tgolem/2005-11-18T03-22/mauve.html

Very nice overview!

> One problem I've run accross is that some mauve tests use
> inconsistent names to report PASSes and FAILs. The most troublesome
> cases are those, where a *named* check/fail is followed by an
> *unnamed* one.
> [...]
> If you also think these cases should be fixed (by naming the checks),
> I would be happy to provide patches. Makes more sense than teaching
> my tgolem to work around these cases.

That does make sense to me. Probably the best approach would be to have
an explicit harness.checkPoint("<descriptive string>") after each named
check followed by unnamed ones. But I would be happy with any patch that
solves this issue.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: inconsistent test names
  2005-11-18 19:08 inconsistent test names Edwin Steiner
  2005-11-24 21:50 ` Mark Wielaard
@ 2005-11-25 19:28 ` Tom Tromey
  1 sibling, 0 replies; 3+ messages in thread
From: Tom Tromey @ 2005-11-25 19:28 UTC (permalink / raw)
  To: Edwin Steiner; +Cc: mauve-discuss

>>>>> "Edwin" == Edwin Steiner <edwin.steiner@gmx.net> writes:

Edwin> One problem I've run accross is that some mauve tests use
Edwin> inconsistent names to report PASSes and FAILs. The most troublesome
Edwin> cases are those, where a *named* check/fail is followed by an
Edwin> *unnamed* one. For example:

Yeah, this is a problem.  It is kind of a pain to write this
correctly.

I wonder if we should just give up on the current scheme and do what
was suggested years ago, namely just name the test case after the
.class, and do a single big pass/fail for the entire file as a whole.
Then we would be guaranteed that the test names would be stable (as
stable as the file names).

We could still track down failures by passing a special flag that
would make Mauve dump a stack trace on a failing check().  If compiled
with debugging this would give the line number -- which would be even
more convenient than what we do now.

Thoughts?

Tom

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2005-11-25 19:28 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-11-18 19:08 inconsistent test names Edwin Steiner
2005-11-24 21:50 ` Mark Wielaard
2005-11-25 19:28 ` Tom Tromey

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).