Harness patch to allow code coverage

Harness patch to allow code coverage

[Anthony Balkissoon]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

This patch is mainly to allow easy use of the EMMA tool, but also
contains small bug fixes.  EMMA usage is auto detected so the normal
usage will not be affected.

2006-05-31  Anthony Balkissoon  <abalkiss@redhat.com>

	* Harness.java:
	(getBootclasspath): Accept output to console from the auto detector.
	(finalize): Dump the coverage data.
	(initProces): Confirm that the RunnerProcess started properly.
	(runTest): Use a invalidTest flag to suppress output when -showpasses 
	is used and a file is run that is "not-a-test".  If the RunnerProcess 
	startup confirmation fails, print a message and exit.  Handle the 
	statup confirmation and coverage dump confirmatino from the 
	RunnerProcess.
	* Makefile.in: Regenerated.
	* RunnerProcess.java: 
	(emmaJarLocation): New field.
	(useEMMA): New field.
	(emmaMethod): New field.
	(main): Accept the -emma command line argument to specify the emma.jar 
	location. Setup the EMMA coverage dumping facilities.  Handle the 
	messages from the Harness indicating we should confirm startup or force
	a coverage dump.
	(setupEMMA): New method.
	(dumpCoverageData): New method.
	* configure: Regenerated.
	* configure.in: Added -with-emma method to allow optional setting of 
	emma.jar location.
	* gnu/testlet/TestHarness.java:
	(getEmmaString): New method.
	* gnu/testlet/config.java.in:
	(emmaString): New field.
	(getEmmaString): New method.

--Tony

[-- Attachment #2: HarnessEMMA.diff --]
[-- Type: text/x-patch, Size: 17543 bytes --]

Index: Harness.java
===================================================================
RCS file: /cvs/mauve/mauve/Harness.java,v
retrieving revision 1.7
diff -u -r1.7 Harness.java
--- Harness.java	18 May 2006 20:04:17 -0000	1.7
+++ Harness.java	31 May 2006 17:20:08 -0000
@@ -508,14 +508,13 @@
           if (br.ready())
             {
               bcpOutput = br.readLine();
+              if (bcpOutput == null)
+                // This means the auto-detection failed.
+                return null;
               if (bcpOutput.startsWith("BCP_FINDER:"))
                 return bcpOutput.substring(11);
               else
-                {
-                  // This means the auto-detection failed.
-                  System.out.println(bcpOutput);
-                  return null;
-                }
+                System.out.println(bcpOutput);
             }
         }
     }
@@ -620,8 +619,9 @@
     //Clean up 
     try
       {
+        runTest("_dump_data_");
         runner_in.close();
-        runner_out.close();
+        runner_out.close();        
         runnerProcess.destroy();
       } 
     catch (IOException e) 
@@ -651,8 +651,7 @@
         runner_out = new PrintWriter(runnerProcess.getOutputStream(), true);
         runner_in = 
           new BufferedReader
-          (new InputStreamReader(runnerProcess.getInputStream()));
-        
+          (new InputStreamReader(runnerProcess.getInputStream()));                
         // Create a timer to watch this new process.
         runner_watcher = new TimeoutWatcher(runner_timeout);
       }
@@ -661,6 +660,8 @@
         System.err.println("Problems invoking RunnerProcess: " + e);
         System.exit(1);
       }
+    
+    runTest("_confirm_startup_");
   }
   
   /**
@@ -749,6 +750,7 @@
   {
     String tn = stripPrefix(testName.replace(File.separatorChar, '.'));
     String outputFromTest;
+    boolean invalidTest = false;
     int temp = -1;
 
     // Start the timeout watcher
@@ -768,6 +770,11 @@
         // thread has declared the test hung and if so ends the process.
         if (testIsHung)
           {
+            if (testName.equals("_confirm_startup_"))
+              {
+                System.out.println("ERROR: Cannot create test runner process.  Exit");
+                System.exit(1);
+              }
             synchronized (runner_lock)
             {
               testIsHung = false;
@@ -793,6 +800,7 @@
               outputFromTest = runner_in.readLine();              
               if (outputFromTest.startsWith("RunnerProcess:"))
                 {
+                  invalidTest = false;
                   // This means the test finished properly, now have to see if
                   // it passed or failed.
                   if (outputFromTest.endsWith("pass"))
@@ -804,11 +812,15 @@
                       // Temporarily decrease the total number of tests,
                       // because it will be incremented later even 
                       // though the test was not a real test.
+                      invalidTest = true;
                       total_tests--;
                       temp = 0;
                     }
                   break;
-                }                
+                } 
+              else if (outputFromTest.equals("_startup_okay_") || 
+                  outputFromTest.equals("_data_dump_okay_"))
+                return;
               else
                 // This means it was just output from the test, like a 
                 // System.out.println within the test itself, we should
@@ -839,7 +851,7 @@
     total_tests ++;
     
     // If the test passed and the user wants to know about passes, tell them.
-    if (showPasses && temp == 0 && !verbose)
+    if (showPasses && temp == 0 && !verbose && !invalidTest)
       System.out.println ("PASS: "+tn);
   }  
   
@@ -917,7 +929,7 @@
    * 3) run those tests.
    * @param folderName
    */
-  private static void processFolder(String folderName)
+  private static void processFolder(String folderName)  
   {
     File dir = new File(folderName);
     String dirPath = dir.getPath();    
Index: RunnerProcess.java
===================================================================
RCS file: /cvs/mauve/mauve/RunnerProcess.java,v
retrieving revision 1.3
diff -u -r1.3 RunnerProcess.java
--- RunnerProcess.java	25 Apr 2006 16:22:36 -0000	1.3
+++ RunnerProcess.java	31 May 2006 17:20:08 -0000
@@ -29,6 +29,7 @@
 import gnu.testlet.TestResult;
 import gnu.testlet.TestSecurityManager;
 import gnu.testlet.Testlet;
+import gnu.testlet.config;
 
 import java.io.BufferedReader;
 import java.io.File;
@@ -39,6 +40,10 @@
 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.io.Reader;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.URL;
+import java.net.URLClassLoader;
 import java.util.Vector;
 
 public class RunnerProcess
@@ -49,6 +54,12 @@
   
   // Total number of harness.check calls since the last checkpoint
   private int count = 0;
+  
+  // The location of the emma.jar file
+  private static String emmaJarLocation = null;
+  
+  // Whether or not to use EMMA
+  private static boolean useEMMA = true;
 
   // Total number of harness.check fails plus harness.fail calls
   private int failures = 0;
@@ -66,8 +77,7 @@
   private int total = 0;
 
   // True if we should run in verbose (noisy) mode
-  private static boolean verbose = false;
-  
+  private static boolean verbose = false;  
 
   // True if failing calls to harness.check(Object, Object) should print the
   // toString methods of each Object
@@ -91,6 +101,8 @@
   // The result of the current test
   private TestResult currentResult = null;
   
+  // The EMMA forced data dump method
+  private static Method emmaMethod = null;
   
   protected RunnerProcess()
   {    
@@ -113,7 +125,7 @@
       }
   }
 
-  public static void main(String[] args) throws Exception
+  public static void main(String[] args)
   {    
     // The test that Harness wants us to run.
     String testname = null;
@@ -141,13 +153,39 @@
               throw new RuntimeException("No file path after '-xmlout'.");
             xmlfile = args[i];
           }
+        else if (args[i].equalsIgnoreCase("-emma"))
+          {
+            // User is specifying the location of the eclipse-ecj.jar file
+            // to use for compilation.
+            if (++i >= args.length)
+              throw new RuntimeException("No file path " +
+                    "after '-emma'.  Exit");
+            emmaJarLocation = args[i];
+          }
       }
     // If the user wants an xml report, create a new TestReport.
     if (xmlfile != null)
       {
         report = new TestReport(System.getProperties());
       }
-
+    
+    // Setup the data coverage dumping mechanism.  The default configuration
+    // is to auto-detect EMMA, meaning if the emma classes are found on the 
+    // classpath then we should force a dump of coverage data.  Also, the user
+    // can configure with -with-emma=JARLOCATION or can specify -emma 
+    // JARLOCATION on the command line to explicitly specify an emma.jar to use
+    // to dump coverage data.
+    if (emmaJarLocation == null)
+      emmaJarLocation = config.emmaString;
+    try
+    {
+      setupEMMA(!emmaJarLocation.equals("_auto_detect_emma_"));
+    }
+    catch (Exception emmaException)
+    {
+      useEMMA = false;
+    }
+    
     while (true)
       {
         // Ask Harness for a test to run, run it, report back to Harness, and
@@ -155,8 +193,20 @@
         try
         {
           testname = in.readLine();
-          RunnerProcess harness = new RunnerProcess();
-          runAndReport(harness, testname);
+          if (testname.equals("_dump_data_"))
+            {
+              if (useEMMA)
+                dumpCoverageData();
+              else
+                System.out.println("_data_dump_okay_");
+            }
+          else if (testname.equals("_confirm_startup_"))
+            System.out.println("_startup_okay_");
+          else
+            {
+              RunnerProcess harness = new RunnerProcess();
+              runAndReport(harness, testname);
+            }
         }
         catch (IOException ioe)
         {          
@@ -733,4 +783,74 @@
                          + " tests expectedly failed");
     return failures > 0 ? 1 : 0;
   }
+  
+  /**
+   * Sets up the compiler by reflection, sets up the compiler options,
+   * and the PrintWriters to get error messages from the compiler.
+   * 
+   * @throws Exception if the emma jar can't be found and the sources
+   * aren't in the proper place.
+   */
+  private static void setupEMMA(boolean useJar) throws Exception
+  {
+    ClassNotFoundException cnfe = null;
+    Class klass = null;
+    String classname = "com.vladium.emma.rt.RT";
+    if (!useJar)
+      {                
+        try
+        {
+          klass = Class.forName(classname);
+        }
+        catch (ClassNotFoundException e)
+        {
+          cnfe = e;
+          useJar = true;
+        }
+      }
+    
+    if (useJar)
+      {
+        File jar = new File(emmaJarLocation);
+        if (! jar.exists() || ! jar.canRead())
+          throw cnfe;
+        
+        ClassLoader loader = new URLClassLoader(new URL[] { jar.toURL() });
+        try
+        {
+          klass = loader.loadClass(classname);
+        }
+        catch (ClassNotFoundException f)
+        {
+          throw cnfe;
+        }
+      }
+    
+    emmaMethod = 
+      klass.getMethod
+      ("dumpCoverageData", new Class[] 
+          { File.class, boolean.class, boolean.class });
+  }  
+  
+  /**
+   * This method forces EMMA to dump its coverage data.  We do this
+   * when all tests have been completed and only if the user either
+   * configured with --with-emma-jar or specified -emma-jar on the
+   * command line.
+   */
+  private static void dumpCoverageData()
+  {
+    try
+    {
+      emmaMethod.invoke(null, new Object[] {
+                                            new File("coverage.ec"),
+                                            Boolean.TRUE,
+                                            Boolean.TRUE });
+    }
+    catch (Exception e)
+    {
+      // This shouldn't happen.
+    }
+    System.out.println("_data_dump_okay_");
+  }
 }
Index: configure
===================================================================
RCS file: /cvs/mauve/mauve/configure,v
retrieving revision 1.29
diff -u -r1.29 configure
--- configure	26 Apr 2006 18:40:37 -0000	1.29
+++ configure	31 May 2006 17:20:08 -0000
@@ -272,7 +272,7 @@
 PACKAGE_BUGREPORT=
 
 ac_unique_file="gnu/testlet/Testlet.java"
-ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS INSTALL_PROGRAM INSTALL_SCRIPT INSTALL_DATA CYGPATH_W PACKAGE VERSION ACLOCAL AUTOCONF AUTOMAKE AUTOHEADER MAKEINFO install_sh STRIP ac_ct_STRIP INSTALL_STRIP_PROGRAM mkdir_p AWK SET_MAKE am__leading_dot AMTAR am__tar am__untar build build_cpu build_vendor build_os host host_cpu host_vendor host_os CHECK_PATH_SEPARATOR CHECK_FILE_SEPARATOR CC CFLAGS LDFLAGS CPPFLAGS ac_ct_CC EXEEXT OBJEXT DEPDIR am__include am__quote AMDEP_TRUE AMDEP_FALSE AMDEPBACKSLASH CCDEPMODE am__fastdepCC_TRUE am__fastdepCC_FALSE TEST_JAVA USE_GCJ_TRUE USE_GCJ_FALSE CP_INSTALL_DIR ECJ_JAR AUTO_COMPILE CLASS_FILES_TRUE CLASS_FILES_FALSE JAVA JAVAC GCJ GCJFLAGS GCJDEPMODE am__fastdepGCJ_TRUE am__fastdepGCJ_FALSE SRCDIR TMPDIR MAIL_HOST LIBOBJS LTLIBOBJS'
+ac_subst_vars='SHELL PATH_SEPARATOR PACKAGE_NAME PACKAGE_TARNAME PACKAGE_VERSION PACKAGE_STRING PACKAGE_BUGREPORT exec_prefix prefix program_transform_name bindir sbindir libexecdir datadir sysconfdir sharedstatedir localstatedir libdir includedir oldincludedir infodir mandir build_alias host_alias target_alias DEFS ECHO_C ECHO_N ECHO_T LIBS INSTALL_PROGRAM INSTALL_SCRIPT INSTALL_DATA CYGPATH_W PACKAGE VERSION ACLOCAL AUTOCONF AUTOMAKE AUTOHEADER MAKEINFO install_sh STRIP ac_ct_STRIP INSTALL_STRIP_PROGRAM mkdir_p AWK SET_MAKE am__leading_dot AMTAR am__tar am__untar build build_cpu build_vendor build_os host host_cpu host_vendor host_os CHECK_PATH_SEPARATOR CHECK_FILE_SEPARATOR CC CFLAGS LDFLAGS CPPFLAGS ac_ct_CC EXEEXT OBJEXT DEPDIR am__include am__quote AMDEP_TRUE AMDEP_FALSE AMDEPBACKSLASH CCDEPMODE am__fastdepCC_TRUE am__fastdepCC_FALSE TEST_JAVA USE_GCJ_TRUE USE_GCJ_FALSE CP_INSTALL_DIR EMMA ECJ_JAR AUTO_COMPILE CLASS_FILES_TRUE CLASS_FILES_FALSE JAVA JAVAC GCJ GCJFLAGS GCJDEPMODE am__fastdepGCJ_TRUE am__fastdepGCJ_FALSE SRCDIR TMPDIR MAIL_HOST LIBOBJS LTLIBOBJS'
 ac_subst_files=''
 
 # Initialize some variables set by options.
@@ -820,7 +820,10 @@
   --with-vm=TESTJVM  				Run the tests with TESTJVM
   --with-gcj                       Use gcj to compile .class files
   --with-bootclasspath=BOOTCP   Specify the bootclasspath for the compiler
-  --with-ecj-jar=JARLOCATION       Use the ecj jar found at JARLOCATION for auto-compilation
+  --with-emma(=JARLOCATION)       Use emma, either unpacked in classpath
+folder or at the specified JARLOCATION
+  --with-ecj-jar=JARLOCATION       Use the ecj jar found at JARLOCATION for
+auto-compilation
 
   --with-tmpdir=DIR                Put temporary files in DIR /tmp
 
@@ -2940,6 +2943,19 @@
 
 
 
+# Check whether --with-emma or --without-emma was given.
+if test "${with_emma+set}" = set; then
+  withval="$with_emma"
+  EMMA="$with_emma"
+else
+  EMMA="yes"
+fi;
+if test "$EMMA" = "yes"
+then
+EMMA="_auto_detect_emma_"
+fi
+
+
 
 # Check whether --with-ecj-jar or --without-ecj-jar was given.
 if test "${with_ecj_jar+set}" = set; then
@@ -4014,6 +4030,7 @@
 s,@USE_GCJ_TRUE@,$USE_GCJ_TRUE,;t t
 s,@USE_GCJ_FALSE@,$USE_GCJ_FALSE,;t t
 s,@CP_INSTALL_DIR@,$CP_INSTALL_DIR,;t t
+s,@EMMA@,$EMMA,;t t
 s,@ECJ_JAR@,$ECJ_JAR,;t t
 s,@AUTO_COMPILE@,$AUTO_COMPILE,;t t
 s,@CLASS_FILES_TRUE@,$CLASS_FILES_TRUE,;t t
Index: configure.in
===================================================================
RCS file: /cvs/mauve/mauve/configure.in,v
retrieving revision 1.17
diff -u -r1.17 configure.in
--- configure.in	26 Apr 2006 18:40:37 -0000	1.17
+++ configure.in	31 May 2006 17:20:08 -0000
@@ -26,9 +26,19 @@
 CP_INSTALL_DIR="$with_bootclasspath")
 AC_SUBST(CP_INSTALL_DIR)
 
+AC_ARG_WITH(emma,
+[  --with-emma(=JARLOCATION)       Use emma, either unpacked in classpath 
+folder or at the specified JARLOCATION],
+EMMA="$with_emma", EMMA="yes")
+if test "$EMMA" = "yes"
+then
+EMMA="_auto_detect_emma_"
+fi
+AC_SUBST(EMMA)
 
 AC_ARG_WITH(ecj-jar,
-[  --with-ecj-jar=JARLOCATION       Use the ecj jar found at JARLOCATION for auto-compilation],
+[  --with-ecj-jar=JARLOCATION       Use the ecj jar found at JARLOCATION for 
+auto-compilation],
 ECJ_JAR="$with_ecj_jar", ECJ_JAR=yes)
 if test "$ECJ_JAR" = "yes"
 then
Index: Makefile.in
===================================================================
RCS file: /cvs/mauve/mauve/Makefile.in,v
retrieving revision 1.36
diff -u -r1.36 Makefile.in
--- Makefile.in	18 May 2006 20:10:50 -0000	1.36
+++ Makefile.in	31 May 2006 17:20:08 -0000
@@ -120,6 +120,7 @@
 ECHO_N = @ECHO_N@
 ECHO_T = @ECHO_T@
 ECJ_JAR = @ECJ_JAR@
+EMMA = @EMMA@
 EXEEXT = @EXEEXT@
 GCJ = @GCJ@
 GCJDEPMODE = @GCJDEPMODE@
Index: gnu/testlet/TestHarness.java
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/TestHarness.java,v
retrieving revision 1.25
diff -u -r1.25 TestHarness.java
--- gnu/testlet/TestHarness.java	24 Apr 2006 19:42:02 -0000	1.25
+++ gnu/testlet/TestHarness.java	31 May 2006 17:20:08 -0000
@@ -295,6 +295,11 @@
   {
     return ecjJar;
   }
+  
+  public String getEmmaString()
+  {
+    return emmaString;
+  }
 
   public String getTestJava()
   {
Index: gnu/testlet/config.java.in
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/config.java.in,v
retrieving revision 1.5
diff -u -r1.5 config.java.in
--- gnu/testlet/config.java.in	24 Apr 2006 19:42:02 -0000	1.5
+++ gnu/testlet/config.java.in	31 May 2006 17:20:08 -0000
@@ -26,6 +26,7 @@
   public static final String autoCompile = "@AUTO_COMPILE@";
   public static final String testJava = "@TEST_JAVA@";
   public static final String ecjJar = "@ECJ_JAR@";
+  public static final String emmaString = "@EMMA@";
   public static final String srcdir = "@SRCDIR@";
   public static final String tmpdir = "@TMPDIR@";
   public static final String pathSeparator = "@CHECK_PATH_SEPARATOR@";
@@ -36,6 +37,7 @@
   public abstract String getAutoCompile ();
   public abstract String getTestJava ();
   public abstract String getEcjJar ();
+  public abstract String getEmmaString ();
   public abstract String getSourceDirectory ();
   public abstract String getTempDirectory ();
   public abstract String getPathSeparator ();

Re: Harness patch to allow code coverage

[David Gilbert]

Anthony Balkissoon wrote:

>This patch is mainly to allow easy use of the EMMA tool, but also
>contains small bug fixes.  EMMA usage is auto detected so the normal
>usage will not be affected.
>  
>
Great!  Do you have this working?  Is there a report anywhere to see?

Regards,

Dave

Re: Harness patch to allow code coverage

[Anthony Balkissoon]

On Wed, 2006-05-31 at 18:31 +0100, David Gilbert wrote:
> Anthony Balkissoon wrote:
> 
> >This patch is mainly to allow easy use of the EMMA tool, but also
> >contains small bug fixes.  EMMA usage is auto detected so the normal
> >usage will not be affected.
> >  
> >
> Great!  Do you have this working?  Is there a report anywhere to see?
> 
> Regards,
> 
> Dave

Hi David,

yes, it's working now, I have a report viewable internally (ie - Red
Hat).  I have no public space so I couldn't put them up anywhere else.
They will be up on builder.classpath.org soon, Tom Fitzsimmons is now
working on that. He also wants to have a combined Japi/code coverage
report so you can link to the source code right from the Japi scores and
see which lines are being tested.  This is all coming soon.

--Tony

Re: Harness patch to allow code coverage

[David Gilbert]

Anthony Balkissoon wrote:

>
>Hi David,
>
>yes, it's working now, I have a report viewable internally (ie - Red
>Hat).  I have no public space so I couldn't put them up anywhere else.
>They will be up on builder.classpath.org soon, Tom Fitzsimmons is now
>working on that. He also wants to have a combined Japi/code coverage
>report so you can link to the source code right from the Japi scores and
>see which lines are being tested.  This is all coming soon.
>
>--Tony
>
>  
>
Great, I'm looking forward to that.  Another thing that would be great 
is just a set of results for Mauve generated nightly.  From time to 
time, I run this one (that's linked from the GNU Classpath home page):

http://www.object-refinery.com/classpath/mauve/report/

...but the code that generates this is quite flaky...a replacement would 
be very welcome!

Regards,

Dave

Re: Harness patch to allow code coverage

[Thomas Fitzsimmons]

David Gilbert wrote:
> Anthony Balkissoon wrote:
> 
>>
>> Hi David,
>>
>> yes, it's working now, I have a report viewable internally (ie - Red
>> Hat).  I have no public space so I couldn't put them up anywhere else.
>> They will be up on builder.classpath.org soon, Tom Fitzsimmons is now
>> working on that. He also wants to have a combined Japi/code coverage
>> report so you can link to the source code right from the Japi scores and
>> see which lines are being tested.  This is all coming soon.
>>
>> --Tony
>>
>>  
>>
> Great, I'm looking forward to that.  Another thing that would be great 
> is just a set of results for Mauve generated nightly.  From time to 
> time, I run this one (that's linked from the GNU Classpath home page):
> 
> http://www.object-refinery.com/classpath/mauve/report/
> 
> ...but the code that generates this is quite flaky...a replacement would 
> be very welcome!

Yes, we need a nightly script to generate Mauve results.  But the report page 
should be more detailed, laid out like this:

Package:	Total:	 	Passed: 	Failed:
java.awt 	1284		1270		14

And the Failed number should be a link to a list of the specific failing tests, 
with the failure messages produced by the new harness:

Package: java.awt

FAIL: gnu.testlet.java.awt.Frame.size1
   line 54: got 29 but expected 53
   line 45: boolean passed to check was false
   line 45: boolean passed to check was false
   line 45: boolean passed to check was false

That would tell people which tests to work on.

We also need a percentage bar beside each package in the JAPI output, indicating 
the number passing and failing tests.

Tom