* JUnit integration
@ 2003-11-27 2:51 graydon hoare
2003-11-27 4:34 ` Classpath / Mauve integration Stephen Crawley
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: graydon hoare @ 2003-11-27 2:51 UTC (permalink / raw)
To: mauve-discuss
[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]
hi,
I've recently found myself somewhat unsatisfied with mauve, and have
put together a list of changes I'd like to make. of course it's wildly
optimistic, but, y'know, best to have unrealistic goals than no goals
at all :)
I posted it to classpath list the other day:
http://mail.gnu.org/archive/html/classpath/2003-11/msg00184.html
and have, since there was a generally non-hostile reception there,
gone on to write up a couple bits of useful preparatory work:
- a bridge class which connects classpath to JUnit
( see http://people.redhat.com/graydon/junit-mauve.png )
- a perl script which does what "choose" claims to do for
mauve tests, only with some improvements (much faster,
interprets Uses: lines correctly and recursively, etc)
these are attached. but the more general question -- the next logical
step in my mind -- is whether to start merging mauve into classpath so
it configures and builds "naturally" as part of the day-to-day
development cycle on classpath (using the classpath build dir,
compiler and VM selection, for example, and running as the nominal
"make check" for classpath). I'm wondering how taboo an idea that is,
whether anyone would mind if I had a go at it.
-graydon
[-- Attachment #2: TestletSuite.java --]
[-- Type: application/octet-stream, Size: 3536 bytes --]
// Copyright (c) 2003 Red Hat
// Written by Graydon Hoare <graydon@redhat.com>
// This file is part of Mauve.
// Mauve is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2, or (at your option)
// any later version.
// Mauve is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Mauve; see the file COPYING. If not, write to
// the Free Software Foundation, 59 Temple Place - Suite 330,
// Boston, MA 02111-1307, USA. */
package gnu.testlet;
import java.io.*;
/**
*
* This class adapts older Mauve tests, based on the Testlet / TestHarness
* interfaces, to JUnit's TestCase / TestSuite interfaces. The result is
* that each Testlet runs as a single TestCase, and the complete collection
* of all Testlets-mapped-to-TestCases is run as a TestSuite, itself a type
* of TestCase.
*
* we use explicit package names in the source here because we are bridging
* two packages with very similar vocabulary, and it helps clarify
* intention.
*
*/
public class TestletSuite
extends junit.framework.TestSuite
{
private gnu.testlet.TestHarness harness;
private static class JUnitHarness
extends gnu.testlet.SimpleTestHarness
{
// override parts of gnu.testlet.SimpleTestHarness
public void check (boolean result)
{
junit.framework.Assert.assertTrue (result);
}
public JUnitHarness()
{
super(false,false,false);
}
}
// fill junit.framework.TestSuite with wrapped gnu.testlet.Testlets
private static class TestletCase extends junit.framework.TestCase
{
private gnu.testlet.Testlet testlet;
private gnu.testlet.TestHarness harness;
public TestletCase (gnu.testlet.Testlet t,
gnu.testlet.TestHarness h)
{
super(t.getClass ().getName ());
testlet = t;
harness = h;
}
protected void runTest ()
{
testlet.test (harness);
}
}
public TestletSuite ()
{
harness = new JUnitHarness ();
try
{
BufferedReader r =
new BufferedReader
(new InputStreamReader
(new FileInputStream
(gnu.testlet.config.builddir + File.separator + "classes")));
while (true)
{
String name = null;
name = r.readLine ();
if (name == null)
break;
try
{
gnu.testlet.Testlet tl =
(gnu.testlet.Testlet) Class.forName (name).newInstance ();
this.addTest (new TestletCase (tl, harness));
}
catch (Exception e)
{
System.err.println("Exception building testlet " + name + ": ");
}
}
}
catch (Exception e)
{
System.err.println("Exception building testlet suite: " + e);
}
}
// this is for JUnit's graphical runners to hook in to the class
public static junit.framework.Test suite ()
{
return new TestletSuite ();
}
// this is for the command-line to hook into the class
public static void main (String args[]) {
junit.textui.TestRunner.run (suite ());
}
}
[-- Attachment #3: choose.pl --]
[-- Type: application/octet-stream, Size: 4357 bytes --]
#!/usr/bin/perl -w
# Copyright (c) 2003 Red Hat
# Written by Graydon Hoare <graydon@redhat.com>
# This file is part of Mauve.
# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING. If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA. */
# Choose the tests we want to run.
# Usage: choose.pl output-directory [tag] ...
# Run in the source directory.
# See README for information on tags.
use File::Find;
use Carp;
use strict;
my %tests_to_run = ();
my %tags_to_find = ();
my $basedir = 'gnu/testlet';
my @dirs = qw( java javax BinaryCompatibility );
my @file_patterns = ();
sub scan_deps
{
my $filename = shift(@_);
return if (exists($tests_to_run{$filename}));
$tests_to_run{$filename} = 1;
my @deps = ();
open(FILE, $filename);
while (<FILE>)
{
if (m@^// Uses: (.*)$@o)
{
my $uses = $1;
my @uses = split(/\s+/, $uses);
for my $used (@uses)
{
my $target = $File::Find::dir . '/' . $used . '.java';
-e $target || croak("missing 'Uses:' target $target, in $filename");
push @deps, $target;
}
}
}
close(FILE);
for my $dep (@deps)
{
scan_deps($dep);
}
}
sub finder
{
if ($File::Find::name =~ /.java$/o)
{
my $filename_matches = 0;
my $tag_spec_matches = 0;
if (not @file_patterns)
{
# no file patterns -> everything matches
$filename_matches = 1;
}
else
{
my $simple_name = $File::Find::name;
$simple_name =~ s@^$basedir/@@o;
for my $pat (@file_patterns)
{
if (('!' . $simple_name) =~ /^$pat/)
{
$filename_matches = 0;
}
elsif ($simple_name =~ /^$pat/)
{
$filename_matches = 1;
}
}
}
if ($filename_matches)
{
if (not %tags_to_find)
{
# no tag patterns -> everything matches
$tag_spec_matches = 1;
}
else
{
open(FILE, $File::Find::name);
while (<FILE>)
{
if (m@^// Tags: (.*)$@o)
{
my $tags = $1;
my @tags = split(/\s+/, $tags);
for my $tag (@tags)
{
if ($tag eq 'not-a-test')
{
close(FILE);
return;
}
}
my $included_by_tag = 0;
my $excluded_by_tag = 0;
for my $tag (@tags)
{
if (length($tag) > 2
&& substr($tag,0,1) eq '!'
&& exists($tags_to_find{substr($tag,1)}))
{
$excluded_by_tag = 1;
}
elsif (exists($tags_to_find{$tag}))
{
$included_by_tag = 1;
}
}
if ($included_by_tag && (! $excluded_by_tag))
{
$tag_spec_matches = 1;
}
}
}
close(FILE);
}
}
# scan dependencies of this file
if ($tag_spec_matches)
{
scan_deps($File::Find::name);
}
}
}
# main()
croak "usage: choose <OUTDIR> [KEYS...]" unless @ARGV;
my $outdir = shift(@ARGV);
for my $arg (@ARGV)
{
if ($arg =~ /java/)
{
$arg =~ s@\.@/@;
push @file_patterns, $arg;
}
else
{
$tags_to_find{$arg} = 1;
}
}
# massage arguments
if (not %tags_to_find
|| exists $tags_to_find{'JDK1.1'}
|| exists $tags_to_find{'JDK1.2'})
{
$tags_to_find{'JDK1.1'} = 1;
$tags_to_find{'JDK1.0'} = 1;
}
# search directories
my @qualified_dirs = map { $basedir . '/' . $_; } @dirs;
find({ 'no_chdir' => 1,
'wanted' => \&finder },
@qualified_dirs);
# output results
my $test = '';
open (CLASSES, ">$outdir/classes");
for $test (sort(keys(%tests_to_run)))
{
$test =~ s/.java$//o;
$test =~ s@/@.@go;
print CLASSES $test, "\n";
}
close (CLASSES);
open (CHOICES, ">$outdir/choices");
print CHOICES 'CHOICES=';
for $test (sort(keys(%tests_to_run)))
{
print CHOICES $test, " \\\n";
}
print CHOICES "\n";
print CHOICES 'SimpleTestHarness: ';
for $test (sort(keys(%tests_to_run)))
{
$test =~ s/.java$/.o/o;
print CHOICES $test, " \\\n";
}
print CHOICES "\n";
close (CHOICES);
^ permalink raw reply [flat|nested] 5+ messages in thread
* Classpath / Mauve integration
2003-11-27 2:51 JUnit integration graydon hoare
@ 2003-11-27 4:34 ` Stephen Crawley
2003-11-27 10:29 ` JUnit integration Raif S. Naffah
2003-11-27 19:17 ` Tom Tromey
2 siblings, 0 replies; 5+ messages in thread
From: Stephen Crawley @ 2003-11-27 4:34 UTC (permalink / raw)
To: graydon hoare; +Cc: mauve-discuss
Hi Graydon,
This is a response to two of your recent emails to 'classpath' and
'mauve-discuss' respectively.
in a 'mauve-discuss' email, graydon@redhat.com said:
> the next logical step in my mind -- is whether to start merging mauve
> into classpath so it configures and builds "naturally" as part of the
> day-to-day development cycle on classpath (using the classpath build
> dir, compiler and VM selection, for example, and running as the
> nominal "make check" for classpath). I'm wondering how taboo an idea
> that is, whether anyone would mind if I had a go at it.
I'm not sure I understand how you propose to do this 'merge', but here
are some (IMO) no-nos.
1) Don't merge the Mauve source tree into the Classpath source tree.
[I don't think you intended to but ...]
2) Don't make the Mauve config environment dependent on Classpath.
If the Mauve config environment is 'aware' of Classpath, it should
also work if Classpath isn't there.
3) Don't make the Classpath config/build environment dependent on Mauve.
If the Classpath config/build environment is 'aware' of Classpath,
it should also work if Mauve isn't there.
I also have doubts as to whether adding a 'make check' to Classpath will
work for most VMs that use Classpath. In many cases, Classpath does not
currently 'know' anything about the 'client' VM; e.g. Kissme & SableVM.
In the case of Kissme, Mauve integration works as follows. The Kissme
autoconf setup has some magic to find the Classpath sandbox and
(optionally) the Mauve sandboxes. The locations of these sandboxes are
then expanded into the relevant Makefiles and scripts. Targets are
provided in the Kissme makefile to configure Mauve for Kissme testing,
and to run a regression tests using the Kissme VM. All of this is done
without modifying either the Mauve or Classpath source trees to be aware
of Kissme.
In a 'classpath' email, graydon@redhat.com said:
> my gut feeling (I don't mean to be insulting or dismissive, just
> observing) is that few people -- and progressively fewer as time
> passes -- will have a use for mauve outside testing classpath. if
> they're a proprietary vendor, they'll buy a sun compatibility kit;
> everyone else seems, from my limited perspective, to be slowly coming
> around to the benefits of just throwing their weight behind classpath.
It depends on what you mean by "have a use for mauve outside testing
classpath". I use Mauve to test Kissme, but I do this "outside of
Classpath" in the sense that I run the tests from the Kissme build
environment. [See above.]
More generally, testing against Mauve should be relevant to all Java
implementors, whether or not they use the Sun JCK testsuite. While
Mauve and the JCK tests do largely overlap, there will inevitably be
Mauve testcases that cover aspects that JCK testcases don't cover.
Compatibility of Java implementations in these aspects is of interest to
Java developers, even if vendors don't care. Hence, there is a clear
case for keeping Mauve largely independent of Classpath.
-- Steve
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: JUnit integration
2003-11-27 2:51 JUnit integration graydon hoare
2003-11-27 4:34 ` Classpath / Mauve integration Stephen Crawley
@ 2003-11-27 10:29 ` Raif S. Naffah
2003-11-27 19:17 ` Tom Tromey
2 siblings, 0 replies; 5+ messages in thread
From: Raif S. Naffah @ 2003-11-27 10:29 UTC (permalink / raw)
To: graydon hoare, mauve-discuss
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160
hello all,
On Thu, 27 Nov 2003 01:50 pm, graydon hoare wrote:
> ...
> I've recently found myself somewhat unsatisfied with mauve...
>
> ...the next logical
> step in my mind -- is whether to start merging mauve into classpath
> so it configures and builds "naturally" as part of the day-to-day
> development cycle on classpath (using the classpath build dir,
> compiler and VM selection, for example, and running as the nominal
> "make check" for classpath)...
(this message is in the spirit of sharing experience and not taking
sides.)
when we started GNU Crypto we were using JUnit, then we converted our
tests to Mauve. we did that for few reasons, most importantly:
* Mauve is much lighter weight than JUnit,
* by supplying few classes and scripts we are able to configure and
carry out the testing in the usual GNU 'make check' way. this way the
developer didnt have to download and/or install any additional software
- --except for gcc and the GNU toolchain.
(for more info, see
<http://savannah.gnu.org/cgi-bin/viewcvs/gnu-crypto/gnu-crypto/gcj/source/Makefile.am?rev=1.36&content-type=text/vnd.viewcvs-markup>,
lines 64, 136-139, and 606-612.)
we had to duplicate the classes (in mauve) under gnu.testlet --except
TestSecurityManager-- and use a shell script (choose-classes) written
by Mark Weilaard (see
<http://savannah.gnu.org/cgi-bin/viewcvs/gnu-crypto/gnu-crypto/source/choose-classes?rev=1.1&content-type=text/vnd.viewcvs-markup>)
adapted to recognise the classes hierarchy rooted under gnu.
for Classpath, there will be (a) no need for duplicating any classes,
nor (b) adapting the choose-classes script! all Classpath would have to
provide is a tarball of mauve sources and the incantations (similar to
what we have in GNU Crypto) to configure and invoke mauve.
cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Que du magnifique
iD8DBQE/xdIN+e1AKnsTRiERA9dKAKCw88TqB9kOVneYHWEPX4kGntLieACfVZCu
N53VZAwEQw+tn1IvttxWexU=
=qzLr
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: JUnit integration
2003-11-27 2:51 JUnit integration graydon hoare
2003-11-27 4:34 ` Classpath / Mauve integration Stephen Crawley
2003-11-27 10:29 ` JUnit integration Raif S. Naffah
@ 2003-11-27 19:17 ` Tom Tromey
2003-11-27 20:14 ` graydon hoare
2 siblings, 1 reply; 5+ messages in thread
From: Tom Tromey @ 2003-11-27 19:17 UTC (permalink / raw)
To: graydon hoare; +Cc: mauve-discuss
>>>>> "graydon" == graydon hoare <graydon@redhat.com> writes:
graydon> these are attached. but the more general question -- the next logical
graydon> step in my mind -- is whether to start merging mauve into classpath so
graydon> it configures and builds "naturally" as part of the day-to-day
graydon> development cycle on classpath (using the classpath build dir,
graydon> compiler and VM selection, for example, and running as the nominal
graydon> "make check" for classpath). I'm wondering how taboo an idea that is,
graydon> whether anyone would mind if I had a go at it.
First, different VMs use Classpath differently. So this may matter.
Second, traditionally Mauve hasn't required any copyright assignments
of any sort. Collecting those now would be hard to impossible (e.g.,
we got a bunch of tests from HP years back, no clue who to contact
about those). So it isn't clear Mauve meets GNU requirements for
checkin...
Third, it is useful to be able to run Mauve against non-Classpath
JVMs, in particular the Sun JVM. This lets us do compatibility
testing; in fact an item on my infrastructure to-do list is to run
Mauve against the JDK nightly and compare the results, to see if we're
getting bad tests.
Anyway, those are issues, potential or otherwise. I definitely agree
that making it easier to run Mauve against Classpath changes would be
helpful.
For libgcj we "solved" this problem by having some wrapper code in the
test suite. Many libgcj developers probably don't run this regularly
though :-(.
Tom
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2003-11-27 20:14 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-11-27 2:51 JUnit integration graydon hoare
2003-11-27 4:34 ` Classpath / Mauve integration Stephen Crawley
2003-11-27 10:29 ` JUnit integration Raif S. Naffah
2003-11-27 19:17 ` Tom Tromey
2003-11-27 20:14 ` graydon hoare
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).