JUnit integration

JUnit integration

[graydon hoare]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

hi,

I've recently found myself somewhat unsatisfied with mauve, and have
put together a list of changes I'd like to make. of course it's wildly
optimistic, but, y'know, best to have unrealistic goals than no goals
at all :)

I posted it to classpath list the other day:

http://mail.gnu.org/archive/html/classpath/2003-11/msg00184.html

and have, since there was a generally non-hostile reception there,
gone on to write up a couple bits of useful preparatory work:

  - a bridge class which connects classpath to JUnit
    ( see http://people.redhat.com/graydon/junit-mauve.png )

  - a perl script which does what "choose" claims to do for
    mauve tests, only with some improvements (much faster,
    interprets Uses: lines correctly and recursively, etc)

these are attached. but the more general question -- the next logical
step in my mind -- is whether to start merging mauve into classpath so
it configures and builds "naturally" as part of the day-to-day
development cycle on classpath (using the classpath build dir,
compiler and VM selection, for example, and running as the nominal
"make check" for classpath). I'm wondering how taboo an idea that is,
whether anyone would mind if I had a go at it.

-graydon



[-- Attachment #2: TestletSuite.java --]
[-- Type: application/octet-stream, Size: 3536 bytes --]

// Copyright (c) 2003 Red Hat
// Written by Graydon Hoare <graydon@redhat.com>

// This file is part of Mauve.

// Mauve is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2, or (at your option)
// any later version.

// Mauve is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.

// You should have received a copy of the GNU General Public License
// along with Mauve; see the file COPYING.  If not, write to
// the Free Software Foundation, 59 Temple Place - Suite 330,
// Boston, MA 02111-1307, USA.  */

package gnu.testlet;
import java.io.*;

/**
 *
 * This class adapts older Mauve tests, based on the Testlet / TestHarness
 * interfaces, to JUnit's TestCase / TestSuite interfaces. The result is
 * that each Testlet runs as a single TestCase, and the complete collection
 * of all Testlets-mapped-to-TestCases is run as a TestSuite, itself a type
 * of TestCase.
 *
 * we use explicit package names in the source here because we are bridging
 * two packages with very similar vocabulary, and it helps clarify
 * intention.
 *
 */

public class TestletSuite 
  extends junit.framework.TestSuite
{

  private gnu.testlet.TestHarness harness;
  
  private static class JUnitHarness 
    extends gnu.testlet.SimpleTestHarness            
  {  
    // override parts of gnu.testlet.SimpleTestHarness
    public void check (boolean result)
    {
      junit.framework.Assert.assertTrue (result);
    }
    public JUnitHarness()
    {
      super(false,false,false);
    }
  }

  // fill junit.framework.TestSuite with wrapped gnu.testlet.Testlets

  private static class TestletCase extends junit.framework.TestCase
  {
    private gnu.testlet.Testlet testlet;
    private gnu.testlet.TestHarness harness;

    public TestletCase (gnu.testlet.Testlet t,
                        gnu.testlet.TestHarness h)
    {
      super(t.getClass ().getName ());
      testlet = t;
      harness = h;
    }
   
    protected void runTest ()
    {
      testlet.test (harness);
    }    
  }

  public TestletSuite ()
  {
    harness = new JUnitHarness ();
    try 
      {
        BufferedReader r = 
          new BufferedReader 
           (new InputStreamReader
            (new FileInputStream
             (gnu.testlet.config.builddir + File.separator + "classes")));

        while (true)
          {
            String name = null;
            name = r.readLine ();
            if (name == null)
              break;
            try 
              {
                gnu.testlet.Testlet tl = 
                  (gnu.testlet.Testlet) Class.forName (name).newInstance ();
                this.addTest (new TestletCase (tl, harness));
              }
            catch (Exception e)
              {
                System.err.println("Exception building testlet " + name + ": ");
              }
          }
      }
    catch (Exception e)
      {
        System.err.println("Exception building testlet suite: " + e);
      }
  }

  // this is for JUnit's graphical runners to hook in to the class
  public static junit.framework.Test suite () 
  { 
    return new TestletSuite ();
  }
  
  // this is for the command-line to hook into the class
  public static void main (String args[]) { 
    junit.textui.TestRunner.run (suite ());
  }
}

[-- Attachment #3: choose.pl --]
[-- Type: application/octet-stream, Size: 4357 bytes --]

#!/usr/bin/perl -w

# Copyright (c) 2003 Red Hat
# Written by Graydon Hoare <graydon@redhat.com>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.  */

# Choose the tests we want to run.
# Usage: choose.pl output-directory [tag] ...
# Run in the source directory.
# See README for information on tags.

use File::Find;
use Carp;
use strict;

my %tests_to_run = ();
my %tags_to_find = ();
my $basedir = 'gnu/testlet';
my @dirs = qw( java javax BinaryCompatibility );
my @file_patterns = ();

sub scan_deps
{
    my $filename = shift(@_);
    return if (exists($tests_to_run{$filename}));
    $tests_to_run{$filename} = 1;

    my @deps = ();
    open(FILE, $filename);
    while (<FILE>)
    {
	if (m@^// Uses: (.*)$@o)
	{
	    my $uses = $1;
	    my @uses = split(/\s+/, $uses);
	    for my $used (@uses)
	    {
		my $target = $File::Find::dir . '/' . $used . '.java';
		-e $target || croak("missing 'Uses:' target $target, in $filename");
		push @deps, $target;
	    }
	}
    }
    close(FILE);

    for my $dep (@deps)
    {
	scan_deps($dep);
    }
}

sub finder
{
    if ($File::Find::name =~ /.java$/o)
    {
	my $filename_matches = 0;
	my $tag_spec_matches = 0;

	if (not @file_patterns)
	{
	    # no file patterns -> everything matches
	    $filename_matches = 1;
	}
	else
	{
	    my $simple_name = $File::Find::name;
	    $simple_name =~ s@^$basedir/@@o;
	    for my $pat (@file_patterns)
	    {
		if (('!' . $simple_name) =~ /^$pat/)
		{
		    $filename_matches = 0;
		}
		elsif ($simple_name =~ /^$pat/)
		{
		    $filename_matches = 1;
		}
	    }
	}

	if ($filename_matches)
	{
	    if (not %tags_to_find)
	    {
		# no tag patterns -> everything matches
		$tag_spec_matches = 1;
	    }
	    else
	    {
		open(FILE, $File::Find::name);
		while (<FILE>)
		{
		    if (m@^// Tags: (.*)$@o)
		    {
			my $tags = $1;
			my @tags = split(/\s+/, $tags);
			for my $tag (@tags)
			{
			    if ($tag eq 'not-a-test')
			    {
				close(FILE);
				return;
			    }
			}
			
			my $included_by_tag = 0;
			my $excluded_by_tag = 0;
			
			for my $tag (@tags)
			{
			    if (length($tag) > 2 
				&& substr($tag,0,1) eq '!'
				&& exists($tags_to_find{substr($tag,1)}))
			    {
				$excluded_by_tag = 1;
			    }
			    elsif (exists($tags_to_find{$tag}))
			    {
				$included_by_tag = 1;
			    }
			}
			if ($included_by_tag && (! $excluded_by_tag))
			{
			    $tag_spec_matches = 1;
			}
		    }
		}
		close(FILE);
	    }
	}

	# scan dependencies of this file
	if ($tag_spec_matches)
	{
	    scan_deps($File::Find::name);
	}
    }
}

# main()

croak "usage: choose <OUTDIR> [KEYS...]" unless @ARGV;

my $outdir = shift(@ARGV);
for my $arg (@ARGV)
{
    if ($arg =~ /java/)
    {
	$arg =~ s@\.@/@;
	push @file_patterns, $arg;
    }
    else
    {    
	$tags_to_find{$arg} = 1;
    }
}

# massage arguments
if (not %tags_to_find
    || exists $tags_to_find{'JDK1.1'} 
    || exists $tags_to_find{'JDK1.2'})
{
    $tags_to_find{'JDK1.1'} = 1;
    $tags_to_find{'JDK1.0'} = 1;
}

# search directories
my @qualified_dirs = map { $basedir . '/' . $_; } @dirs;
find({ 'no_chdir' => 1,
       'wanted' => \&finder },
     @qualified_dirs);


# output results 
my $test = '';
open (CLASSES, ">$outdir/classes");
for $test (sort(keys(%tests_to_run)))
{
    $test =~ s/.java$//o;
    $test =~ s@/@.@go;
    print CLASSES $test, "\n";
}
close (CLASSES);


open (CHOICES, ">$outdir/choices");
print CHOICES 'CHOICES=';
for $test (sort(keys(%tests_to_run)))
{
    print CHOICES $test, " \\\n";
}
print CHOICES "\n";

print CHOICES 'SimpleTestHarness: ';
for $test (sort(keys(%tests_to_run)))
{
    $test =~ s/.java$/.o/o;
    print CHOICES $test, " \\\n";
}
print CHOICES "\n";
close (CHOICES);

Classpath / Mauve integration

[Stephen Crawley]

Hi Graydon,

This is a response to two of your recent emails to 'classpath' and
'mauve-discuss' respectively.

in a 'mauve-discuss' email, graydon@redhat.com said:
> the next logical step in my mind -- is whether to start merging mauve
> into classpath so it configures and builds "naturally" as part of the
> day-to-day development cycle on classpath (using the classpath build
> dir, compiler and VM selection, for example, and running as the
> nominal "make check" for classpath). I'm wondering how taboo an idea
> that is, whether anyone would mind if I had a go at it.

I'm not sure I understand how you propose to do this 'merge', but here
are some (IMO) no-nos.

  1)  Don't merge the Mauve source tree into the Classpath source tree.
      [I don't think you intended to but ...]

  2)  Don't make the Mauve config environment dependent on Classpath.
      If the Mauve config environment is 'aware' of Classpath, it should
      also work if Classpath isn't there.
      
  3)  Don't make the Classpath config/build environment dependent on Mauve.
      If the Classpath config/build environment is 'aware' of Classpath, 
      it should also work if Mauve isn't there.
      
I also have doubts as to whether adding a 'make check' to Classpath will
work for most VMs that use Classpath.  In many cases, Classpath does not
currently 'know' anything about the 'client' VM; e.g. Kissme & SableVM.

In the case of Kissme, Mauve integration works as follows.  The Kissme
autoconf setup has some magic to find the Classpath sandbox and
(optionally) the Mauve sandboxes.  The locations of these sandboxes are
then expanded into the relevant Makefiles and scripts.  Targets are
provided in the Kissme makefile to configure Mauve for Kissme testing,
and to run a regression tests using the Kissme VM.  All of this is done
without modifying either the Mauve or Classpath source trees to be aware
of Kissme.

In a 'classpath' email, graydon@redhat.com said:
> my gut feeling (I don't mean to be insulting or dismissive, just
> observing) is that few people -- and progressively fewer as time
> passes -- will have a use for mauve outside testing classpath. if
> they're a proprietary vendor, they'll buy a sun compatibility kit;
> everyone else seems, from my limited perspective, to be slowly coming
> around to the benefits of just throwing their weight behind classpath.

It depends on what you mean by "have a use for mauve outside testing
classpath".  I use Mauve to test Kissme, but I do this "outside of
Classpath" in the sense that I run the tests from the Kissme build
environment.  [See above.]

More generally, testing against Mauve should be relevant to all Java
implementors, whether or not they use the Sun JCK testsuite.  While
Mauve and the JCK tests do largely overlap, there will inevitably be
Mauve testcases that cover aspects that JCK testcases don't cover.
Compatibility of Java implementations in these aspects is of interest to
Java developers, even if vendors don't care.  Hence, there is a clear
case for keeping Mauve largely independent of Classpath.

-- Steve

Re: JUnit integration

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

hello all,

On Thu, 27 Nov 2003 01:50 pm, graydon hoare wrote:
> ...
> I've recently found myself somewhat unsatisfied with mauve...
>
> ...the next logical
> step in my mind -- is whether to start merging mauve into classpath
> so it configures and builds "naturally" as part of the day-to-day
> development cycle on classpath (using the classpath build dir,
> compiler and VM selection, for example, and running as the nominal
> "make check" for classpath)...

(this message is in the spirit of sharing experience and not taking 
sides.)

when we started GNU Crypto we were using JUnit, then we converted our 
tests to Mauve.  we did that for few reasons, most importantly:

* Mauve is much lighter weight than JUnit,
* by supplying few classes and scripts we are able to configure and 
carry out the testing in the usual GNU 'make check' way.  this way the 
developer didnt have to download and/or install any additional software 
- --except for gcc and the GNU toolchain.

(for more info, see 
<http://savannah.gnu.org/cgi-bin/viewcvs/gnu-crypto/gnu-crypto/gcj/source/Makefile.am?rev=1.36&content-type=text/vnd.viewcvs-markup>, 
lines 64, 136-139, and 606-612.)

we had to duplicate the classes (in mauve) under gnu.testlet --except 
TestSecurityManager-- and use a shell script (choose-classes) written 
by Mark Weilaard (see 
<http://savannah.gnu.org/cgi-bin/viewcvs/gnu-crypto/gnu-crypto/source/choose-classes?rev=1.1&content-type=text/vnd.viewcvs-markup>) 
adapted to recognise the classes hierarchy rooted under gnu.

for Classpath, there will be (a) no need for duplicating any classes, 
nor (b) adapting the choose-classes script! all Classpath would have to 
provide is a tarball of mauve sources and the incantations (similar to 
what we have in GNU Crypto) to configure and invoke mauve.


cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE/xdIN+e1AKnsTRiERA9dKAKCw88TqB9kOVneYHWEPX4kGntLieACfVZCu
N53VZAwEQw+tn1IvttxWexU=
=qzLr
-----END PGP SIGNATURE-----

Re: JUnit integration

[Tom Tromey]

>>>>> "graydon" == graydon hoare <graydon@redhat.com> writes:

graydon> these are attached. but the more general question -- the next logical
graydon> step in my mind -- is whether to start merging mauve into classpath so
graydon> it configures and builds "naturally" as part of the day-to-day
graydon> development cycle on classpath (using the classpath build dir,
graydon> compiler and VM selection, for example, and running as the nominal
graydon> "make check" for classpath). I'm wondering how taboo an idea that is,
graydon> whether anyone would mind if I had a go at it.

First, different VMs use Classpath differently.  So this may matter.

Second, traditionally Mauve hasn't required any copyright assignments
of any sort.  Collecting those now would be hard to impossible (e.g.,
we got a bunch of tests from HP years back, no clue who to contact
about those).  So it isn't clear Mauve meets GNU requirements for
checkin...

Third, it is useful to be able to run Mauve against non-Classpath
JVMs, in particular the Sun JVM.  This lets us do compatibility
testing; in fact an item on my infrastructure to-do list is to run
Mauve against the JDK nightly and compare the results, to see if we're
getting bad tests.


Anyway, those are issues, potential or otherwise.  I definitely agree
that making it easier to run Mauve against Classpath changes would be
helpful.

For libgcj we "solved" this problem by having some wrapper code in the
test suite.  Many libgcj developers probably don't run this regularly
though :-(.

Tom

Re: JUnit integration

[graydon hoare]

Tom Tromey <tromey@redhat.com> writes:

> First, different VMs use Classpath differently...
> Second, traditionally Mauve hasn't required any copyright assignments..
> Third, it is useful to be able to run Mauve against non-Classpath..

ah well, it was worth a passing thought :)

-graydon