Some more Mauve observations

Some more Mauve observations

[Mark Wielaard]

Hi,

Here are some things that I would like to improve in Mauve.
It is also the beginning of some documentation. Although I do admit that
it looks more like a collection of random thought and observations :)
Let me know what you think about it.

Mauve doesn't have really simple application/system tests only unit
tests. The unit tests are much more accurate but need some extra
framework. I agree with Andrew that real standalone tests are very nice
to have, especially for exchanging tests with people that don't have
Mauve setup. Dalibor is correct that such application tests are much
more fragile then real unit tests but I think it is more important to
have a really easy way to add more tests. Will try to come up with
something that turns a application test into a Mauve unit test without
to much work.

I looked at the unit test framework of Mauve and JUnit (not how to
actually run the test suite, see below) and I don't think one is really
better then the other. For Mauve it works as follows:

When writing a class you implement gnu.testlet.Testlet which defines
just one method void test (TestHarness harness). The TestHarness gives
you the following methods for which comparable ones can also be found in
JUnit:

checkPoint(String) - Gives (base)name for next check/test.

check(boolean) - Test if argument is true.

fail(String) - Give the test a name and fail.
               Basically, checkPoint(name); check(false);

check(X,Y)
check(X,Y,String) for Objects and all primitive types
- checks if the given arguments are the same.
  The optional third argument can be the name of the test.

It might be nice to have something like the JUnit Assert class in Mauve
if people think that makes people that know about JUnit already help
write Mauve tests:
http://junit.sourceforge.net/javadoc/junit/framework/Assert.html
Except for that class I don't think JUnit is that special.
And some of it relies to much on a good working Java environment and
Mauve is often used in a more challenging situation.

The following features are not present in JUnit:

The following TestHarness methods can be used to give more output:

debug(String) and verbose(String)
debug(String, boolean) for (not) adding newlines.
debug(Trowable) and debug(Object[], String)

The TestHarness also implements the gnu.testlet.config
interface that defines methods that give the Strings representing the
SourceDirectory, TempDirectory, PathSeparator and file Separator. These
values are generated by configure.
(The only test that uses this directly is
gnu/testlet/java/io/File/jdk11.java.)

You can also use the following methods from TestHarness:
getResourceReader() and getResourceStream() which give you a handle to a
file (starting from the top of the source tree). The implementation in
SimpleTestHarness uses the configure values found in config.java. These
methods are certainly handy but I hope that they can be done simpler (so
they don't need the configure output) by for example adding the contents
of those resource to a static byte[] in the class itself.
(All of the resources used are less then 500 bytes except for the
Unicode database which is more then 400K.)

Nice about the Mauve SimpleTestHarness is that it interprets the
"xfails" file (in the current directory) when verbose is true. That
makes it easy to find regressions. (JUnit does not seem to have this
because it is based on the XP methodology which has the philosophy that
it is really, really bad to have failing tests and not fixing them
immediately.)

Setting up and running the Mauve framework is not the easiest thing in
the world. The configure and Makefile are to complex because they have
special rules for using gcj. I would like to push the special support
for gcj out of Mauve and in the libgcj tree so we can use more clear and
simple scripts (and hopefully get rid of the whole configure step).

Alex wrote some simple shell scripts for Kissme to easily run Mauve. The
keys file and the tags should make Mauve very flexible but in practice
it is unclear. Needs more documentation. Each file contains a // Tags:
line (and possible a // Uses: line). Which tags imply which other tags
and the mechanism for generating the classlist from a key file is done
in the choose script. I think this mechanism is fine but it is poorly
documented.

What I am personally really missing is a test batch mode. It would be
really nice to have a mechanism for running the compiler and VM for each
test separately. Currently when the compiler cannot compile a test or if
the VM crashes your complete test run stops. libgcj has something that
can do that using dejagnu but I hope it can be done by a simple shell
script.

I hope to implement some of the above suggestions in the next couple of
days.

Cheers,

Mark

Re: Some more Mauve observations (test batch mode)

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 2072 bytes --]

Hi,

On Fri, 2002-07-19 at 01:36, Mark Wielaard wrote:
> What I am personally really missing is a test batch mode. It would be
> really nice to have a mechanism for running the compiler and VM for each
> test separately. Currently when the compiler cannot compile a test or if
> the VM crashes your complete test run stops. libgcj has something that
> can do that using dejagnu but I hope it can be done by a simple shell
> script.

I made something by adapting/splitting up the 'choose' script and I am
really happy with how it works. I have been testing it with jikes and
gcj -C as compilers and gij or kissme as runtime VMs. It also works for
compiling to native code (completely from source, not yet
source->class->native but that is not so interesting for me at the
moment). If someone has a working kjc and kaffe installation please try
it out. To use it just set a variable in the 'runner' an 'batch_run'
script and run it with ./batch_run > results_file. (Mauve should have
been correctly configured though.)

Attached you find the following files:

choose-classes:
- generates a 'test_classes' file given some keys/tags.

uses-list:
- generates a 'uses_files' file containing all
  the java source files used by a given test.

runner:
- Runs a given test class and generates a FAIL output
  when the test crashes the VM of times out.

batch_run:
- Uses choose-classes to generate a list of all the
  tests in mauve. Tries to compile all the needed files
  (using the uses-list script) and generates a FAIL
  when compilation fails and finally runs the test using
  the runner script.

There is also a small patch for SimpleTestHarness to add a -resultsonly
flag to make it only output the PASS or FAIL results which is used by
the runner script.

These scripts should be usable with the current Mauve installation.
The really tricky part is integrating them with the configure/build
environment. I have not yet really thought about it since the Makefile
looks like black magic to me. Hopefully someone is interested in helping
me integrate them...

Cheers,

Mark

[-- Attachment #2: choose-classes --]
[-- Type: text/x-sh, Size: 6329 bytes --]

#! /bin/sh

# Copyright (c) 1998, 1999 Cygnus Solutions
# Written by Tom Tromey <tromey@cygnus.com>

# Copyright (c) 2002 Free Software Foundation, Inc.
# Adapted by Mark Wielaard <mark@klomp.org>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.  */

# Choose the tests we want to run. Output generated in file 'test_classes'.
# Usage: choose-classes [-verbose] output-directory [tag] ...
# Run in the source directory.
# See README for information on tags.

verbose=
spectags=
tags=
file_specs=

if test "$1" = "-verbose"; then
   verbose=yes
   shift
fi

outdir="$1"
shift
test -z "$outdir" \
    && echo "Usage: choose-classes [-verbose] output-directory [tag] ..." \
    && exit -1

# Some tags imply other tags.  For instance, JDK1.2 implies JDK1.1 and
# JDK1.0.  This makes it easier to keep the tags up to date in the
# source -- we only have to keep track of differences (as opposed to
# editing every source file when a new tag is added).  If we see an
# otherwise-unrecognized tag, and the file `mauve-TAG' exists, then we
# treat the contents of that file as a list of implied tags.
taglist=${1+"$@"}
seen_dashdash=
while test -n "$taglist"; do
   set $taglist
   i="$1"
   shift
   taglist=${1+"$@"}

   case "$i" in
    X--)
       seen_dashdash=yes
       ;;
    !!java.*)
       echo "Invalid tag: $i" 1>&2
       exit 1
       ;;
    !java.* | java.* | !javax.* | javax.*)
       file="`echo $i | tr . /`"
       file_specs="$file_specs $file"
       ;;
    JAVA2)
       tags="$tags JDK1.0 JDK1.1 JDK1.2 $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JDK1.2)
       tags="$tags JDK1.0 JDK1.1 JAVA2 $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JDK1.1)
       tags="$tags JDK1.0 JLS1.1 $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JDK1.0)
       tags="$tags JLS1.0"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JLS1.1)
       tags="$tags JDK1.0 JDK1.1"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    JLS1.0)
       tags="$tags JDK1.0"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
    *)
       file=
       # Always prefer a tag file in the build directory.
       if test -f "$outdir/mauve-$i"; then
	  file="$outdir/mauve-$i"
       else
	  if test -f "mauve-$i"; then
	     file="mauve-$i"
	  fi
       fi
       if test -n "$file"; then
	  # We don't want tags from a file to be treated as specified
	  # on the command line.  So we add a `X--' before the
	  # expansion.  We don't use just `--' as this would confuse
	  # `set'.
	  taglist="$taglist X-- `sed -e 's/#.*$//g' $file`"
       fi
       tags="$tags $i"
       test -z "$seen_dashdash" && spectags="$spectags $i"
       ;;
   esac
done

# By default, use JDK1.1.
test -z "$tags" && tags="JDK1.0 JDK1.1"

test -n "$verbose" && echo "tags = $tags"
test -n "$verbose" && echo "spectags = $spectags"
test -n "$verbose" && echo "file_specs = $file_specs"

classes=/tmp/classes-$$
: > $classes

(cd gnu/testlet; find java javax -name '*.java' -print) |
while read file; do
   exact_match=no
   test -n "$verbose" && echo "Examining $file"
   if test -n "$file_specs"; then
      ok=yes
      for pat in $file_specs; do
	 # PAT can be like `java.lang' or `!java.lang'.
	 # If `!FILE' matches `PAT', then we know the pattern
	 # starts with `!' and is an exclusion pattern.
	 # Otherwise, if `!FILE' matches `!PAT', then we know that
	 # the file should be included.
	 case "!$file" in
	  ${pat}*)
	     ok=no
	     test -n "$verbose" && echo "  ... excluded by $pat"
	     ;;
	  !${pat}.java)
	     # If the tag list includes an exact match, then we
	     # unconditionally accept it.
	     ok=yes
	     exact_match=yes
	     test -n "$verbose" && echo "  ... included by $pat (exact match)"
	     ;;
	  !${pat}*)
	     ok=yes
	     test -n "$verbose" && echo "  ... included by $pat"
	     ;;
	 esac
      done
   else
      ok=yes
   fi

   if test "$ok" = no; then
      continue
   fi

   # Surround value with spaces so that case statements will work
   # correctly.
   taglist=" `grep '^// Tags:' gnu/testlet/$file | sed -e 's,^// Tags:,,'` "
   istest=yes
   case "$taglist" in
    *\ not-a-test\ *)
       istest=no
       ;;
   esac

   # For an exact match, we don't look at the tags.
   if test "$exact_match" = no; then
      ok=no
      # If any tag on the tag list matches, then we are ok.
      for tag in $tags; do
	 case "$taglist" in
	    *" ${tag} "*)
	    ok=yes
	    break
	    ;;
	 esac
      done

      # If any specified tag hits a `not' tag, then we are not ok.
      if test "$ok" = yes; then
	 for tag in $spectags; do
	    case "$taglist" in
	       *" !${tag} "*)
	         ok=no
		 test -n "$verbose" && echo "File $file has tag !$tag"
		 break
		 ;;
	    esac
	 done
      else
	 test -n "$verbose" && echo "  ... excluded because no tag matched"
      fi
   fi

   if test "$ok" = yes; then
      class="gnu.testlet.`echo $file | sed -e 's/\.java$//' | tr / .`"
      if test "$istest" = yes; then
	 echo $class >> $classes
	 test -n "$verbose" && echo "Chose $class"
      else
      	 test -n "$verbose" && echo " ... excluded because not-a-test"
      fi
   fi
done

if test -f $outdir/test_classes \
   && cmp $classes $outdir/test_classes > /dev/null 2>&1; then
   # Files are the same.
   rm $classes
else
   mv $classes $outdir/test_classes
fi

exit 0

[-- Attachment #3: uses-list --]
[-- Type: text/x-sh, Size: 2673 bytes --]

#!/bin/sh

# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>
# Based on the choose script written by Tom Tromey <tromey@cygnus.com>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

# Given a (list of) test classes or java files generates a list of all java
# files used for that test in the file 'uses_files'.

usage="uses-list [-verbose] output-directory classes_or_files"

if test "$1" = "-verbose"; then
    verbose=yes
    shift
else
    verbose=
fi 

outdir="$1"
shift
test -z "$outdir" && echo "$usage" && exit -1

usesfile=/tmp/uses-file-$$
uses=/tmp/uses-$$

class="$1"
shift
test -z "$class" && echo "$usage" && exit -1

while test ! -z "$class"; do
    test -n "$verbose" && echo "Processing $class"

    # Strip trailing .java or .class if given and replace . with /
    file="`echo $class | sed -e 's/\.java$//' | sed -e 's/\.class$//' | tr . /`"
    file="$file.java"
    dir="`dirname $file`"
    test ! -f "$file" && echo "file $file does not exist" && exit -1

    useslist=" `grep '^// Uses:' $file | sed -e 's,^// Uses:,,'` "

    filelist=$file
    echo "$file" >> "$usesfile"
    for used in $useslist; do
	usedfile=$dir/$used.java
	if test ! -f "$usedfile"; then
	    echo "file $used file used by $file does not exist"
	    exit -1
	fi
	echo "$usedfile" >> "$usesfile"
	filelist="$filelist $usedfile"
    done
    test -n "$verbose" && echo " ... uses: $filelist"

    class="$1"
    shift
done

# Now uniquify the list of auxiliary files we used.
# I think neither `uniq' nor `sort -u' is portable, so we do it by hand.
previous=
sort "$usesfile" |
while read next; do
    if test "$previous" != "$next"; then
	echo "$next" >> $uses
	previous=$next
    fi
done

if test -f $outdir/uses_files \
    && cmp $uses $outdir/uses_files > /dev/null 2>&1; then
    # Files are the same.
    rm $uses
else
    mv $uses $outdir/uses_files
fi
rm $usesfile

[-- Attachment #4: runner --]
[-- Type: text/x-sh, Size: 2052 bytes --]

#!/bin/bash

# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

# Tries to run one test case. Produces a FAIL output if the test crashes
# or doesn't finish after some timeout occurs.
# RUNTIME command can be set in the script to select the VM.
# For testing (gcj) native code set RUNTIME to 'native' it will then try to
# execute the program SimpleTestHarness in the current directory.
# Called from batch_run script.
#
# Usage: runner test_class_name
# Run in the source directory.

#RUNTIME="gij"
RUNTIME="Kissme"
#RUNTIME="native"

# Number of seconds to wait for the test to finish
WAIT=60

# Called when timeout occurs
timeout()
{
    kill -9 $PID 2>&1 > /dev/null
}

# Install timeout
trap timeout SIGALRM
sleep $WAIT && kill -s SIGALRM $$ 2>&1 > /dev/null &
TIMEOUT=$!

testclass=$1
# Run process
if test "$RUNTIME" = "native"; then
    echo $testclass | ./SimpleTestHarness -resultsonly &
else
    echo $testclass | $RUNTIME gnu.testlet.SimpleTestHarness -resultsonly &
fi
PID=$!
wait $PID
result=$?

# Terminated unexpectedly?
if test $result != 0 -a $result != 1; then
    echo "FAIL: $testclass abnormal termination $result CRASH or TIMEOUT"
fi

# cleanup timeout
trap '' SIGALRM
kill $TIMEOUT 2>&1 > /dev/null

[-- Attachment #5: batch_run --]
[-- Type: text/x-sh, Size: 2510 bytes --]

#!/bin/bash

# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>

# This file is part of Mauve.

# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING.  If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.

# Tries to compile and run all possible tests.
# Produces a FAIL output if compilation fails.
# COMPILER command can be set in the script
# For testing (gcj) native code compilation set NATIVE to 'true', otherwise
# don't set it.
# Uses runner script to run each individual test.
#
# Usage: run_batch > result_output
# Run in the source directory.

# Include every key/tag we can think of but always exclude the unicode test.
KEYS="JDK1.0 JDK1.1 JDK1.2 JDK1.3 JDK1.4 JLS1.0 JLS1.2 JDBC1.0 JDBC2.0 JAVA2"
KEYS="$KEYS !java.lang.Character.unicode"

#COMPILER="gcj -C"
#COMPILER="gcj"
COMPILER="jikes"

COMPILER_FLAGS="-g"

NATIVE=
#NATIVE="true"

# Cleanup
find gnu/testlet -name "*.class" | xargs rm -f
if test ! -z "$NATIVE"; then
    rm -f SimpleTestHarness
fi

# Create Mauve Framwork
framework_sources="gnu/testlet/SimpleTestHarness.java gnu/testlet/TestHarness.java gnu/testlet/Testlet.java gnu/testlet/ResourceNotFoundException.java gnu/testlet/config.java"

if test -z "$NATIVE"; then
    $COMPILER $COMPILER_FLAGS $framework_sources
fi

# Generate test classes list
./choose-classes . "$KEYS"

# Compile and run all the tests
for testclass in `cat test_classes`; do
    # Get all needed java sources
    ./uses-list . $testclass

    # Try to compile sources
    if test -z "$NATIVE"; then
        $COMPILER $COMPILER_FLAGS @uses_files
    else
        $COMPILER -o SimpleTestHarness --main=gnu.testlet.SimpleTestHarness \
	          $framework_sources `cat uses_files`
    fi
    if test $?  == 0; then
        ./runner $testclass
    else
        echo "FAIL: $testclass COMPILE FAILED"
    fi
done

[-- Attachment #6: SimpleTestHarness.patch --]
[-- Type: text/x-patch, Size: 2786 bytes --]

Index: gnu/testlet/SimpleTestHarness.java
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/SimpleTestHarness.java,v
retrieving revision 1.29
diff -u -r1.29 SimpleTestHarness.java
--- gnu/testlet/SimpleTestHarness.java	18 Feb 2002 01:43:07 -0000	1.29
+++ gnu/testlet/SimpleTestHarness.java	19 Jul 2002 22:57:12 -0000
@@ -39,6 +39,7 @@
   private int total = 0;
   private boolean verbose = false;
   private boolean debug = false;
+  private boolean results_only=false;
   private String description;
   private String last_check;
 
@@ -59,13 +60,13 @@
 	      System.out.println (desc);
 	      ++failures;
 	    }
-	  else if (verbose)
+	  else if (verbose || results_only)
 	    {
 	      System.out.println ("X" + desc);
 	      ++xfailures;
 	    }
 	}
-      else if (verbose)
+      else if (verbose || results_only)
 	{
 	  if (expected_xfails.contains (getDescription ("FAIL")))
 	    {
@@ -235,18 +236,26 @@
 
   protected int done ()
     {
-      System.out.println(failures + " of " + total + " tests failed");
-      if (xpasses > 0)
-        System.out.println(xpasses + " of " + total + " tests unexpectedly passed");
-      if (xfailures > 0)
-        System.out.println(xfailures + " of " + total + " tests expectedly failed");
+      if (!results_only)
+	{
+          System.out.println(failures + " of " + total + " tests failed");
+          if (xpasses > 0)
+            System.out.println(xpasses + " of " + total
+			    + " tests unexpectedly passed");
+          if (xfailures > 0)
+            System.out.println(xfailures + " of " + total
+			    + " tests expectedly failed");
+	}
       return failures > 0 ? 1 : 0;
+
     }
 
-  protected SimpleTestHarness (boolean verbose, boolean debug)
+  protected SimpleTestHarness (boolean verbose, boolean debug,
+		               boolean results_only)
     {
       this.verbose = verbose;
       this.debug = debug;
+      this.results_only= results_only;
 
       try
         {
@@ -269,6 +278,7 @@
     {
       boolean verbose = false;
       boolean debug = false;
+      boolean results_only = false;
       int i;
 
       for (i = 0; i < args.length; i++) 
@@ -277,12 +287,18 @@
 	    verbose = true;
 	  else if (args[i].equals("-debug")) 
 	    debug = true;
+	  else if (args[i].equals("-resultsonly")) 
+	    {
+	      results_only = true;
+	      verbose = false;
+	      debug = false;
+	    }
 	  else
 	    break;
         }
 
       SimpleTestHarness harness
-	= new SimpleTestHarness (verbose, debug);
+	= new SimpleTestHarness (verbose, debug, results_only);
 
       BufferedReader r
 	= new BufferedReader (new InputStreamReader (System.in));