Re: Project Mauve: A Free Java Regression Test and Compatibility Package

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Aaron M. Renn]

>I was wondering - is there an org.gnu class hierarchy
>already? I guess that a lot of this stuff (make, depend,
>config maybe - how'd you do this in Java, getopt and
>regex, bash....) might crop up over time, especially
>when people want to deploy on Win32. It might be worth
>considering dropping this into a good place and maintain
>them in a unified way?

The GNU project is using simply "gnu." not "org.gnu" for its pacakges.
Trying to unify some of the various GNU Java programs and library packages
availabe is one of the items on my infinite TODO list, but it probably won't
happen soon.  At least not if I'm the one that does it.  I want to
concentrate on Classpath first.

--
Aaron M. Renn (arenn@urbanophile.com) http://www.urbanophile.com/arenn/

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Brian Jones]

Stuart Ballard <sballard@netreach.net> writes:

> How about the gjt ( http://www.gjt.org/ )?
> 
> Sounds like a great place to put gnu.* packages...

I was thinking more along the lines of a place to 

a) register a package (with name, location, etc)
b) maintain viewable tree with explanations of what is where
c) impose some order, at least on the first identifier after gnu.

The current list on http://www.gnu.org/software/java/ shows that the
current packages are all over the namespace.

Brian
-- 
|-------------------------------|Software Engineer
|Brian Jones			|cbj@nortel.net
|cbj@gnu.org			| http://www.nortel.net
| http://www.classpath.org/      |------------------------------

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Stuart Ballard]

Brian Jones wrote:
> 
> Bernd Kreimeier <bk@gamers.org> writes:
> 
> > I was wondering - is there an org.gnu class hierarchy
> > already? I guess that a lot of this stuff (make, depend,
> > config maybe - how'd you do this in Java, getopt and
> > regex, bash....) might crop up over time, especially
> > when people want to deploy on Win32. It might be worth
> > considering dropping this into a good place and maintain
> > them in a unified way?
> 
> I was talking to Paul (rao@gnu.org) about this the other day.  There
> really has to be a place to maintain some sort of order in the gnu.*
> or org.gnu.* hierarchy of java stuff.  There are already regex and
> getopt implementations btw.  I'm not sure who fills this role or what
> keeps track of it.  But, this is probably of interest to more groups
> than are receiving this email.

How about the gjt ( http://www.gjt.org/ )?

Sounds like a great place to put gnu.* packages...

Stuart.

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Brian Jones]

Bernd Kreimeier <bk@gamers.org> writes:

> Tom Tromey writes:
>  > I believe we currently require GNU make, sh, a Java runtime, and a
>  > Java compiler.  It would probably be possible to get rid of the GNU
>  > make dependency.  I'm not particularly motivated to do it.
> 
> There is a JavaMake ( and a JavaDepend
> http://www.cs.mcgill.ca/~stever/software/JavaDeps/
> http://www.gnu.org/software/java/java.html
> 
> separate projects, both listed at gnu.org. JavaDepend
> creates GNU Make dependecies, but it is easy to tweak
> it into satisfying the different syntax JavaMake uses.
> The one thing I am missing from JavaMake is a replacement
> for the UNIX find (I am using a simplistic Makefile that
> traverses the *.java source tree from the root), but for
> your purposes the existing operators might be sufficient.
> 
> I was wondering - is there an org.gnu class hierarchy
> already? I guess that a lot of this stuff (make, depend,
> config maybe - how'd you do this in Java, getopt and 
> regex, bash....) might crop up over time, especially 
> when people want to deploy on Win32. It might be worth 
> considering dropping this into a good place and maintain
> them in a unified way?

I was talking to Paul (rao@gnu.org) about this the other day.  There
really has to be a place to maintain some sort of order in the gnu.*
or org.gnu.* hierarchy of java stuff.  There are already regex and
getopt implementations btw.  I'm not sure who fills this role or what
keeps track of it.  But, this is probably of interest to more groups
than are receiving this email.

Brian
-- 
|-------------------------------|Software Engineer
|Brian Jones			|cbj@nortel.net
|cbj@gnu.org			| http://www.nortel.net
| http://www.classpath.org/      |------------------------------

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Bernd Kreimeier]

Tom Tromey writes:
 > I believe we currently require GNU make, sh, a Java runtime, and a
 > Java compiler.  It would probably be possible to get rid of the GNU
 > make dependency.  I'm not particularly motivated to do it.
 
There is a JavaMake ( and a JavaDepend
http://www.cs.mcgill.ca/~stever/software/JavaDeps/
http://www.gnu.org/software/java/java.html

separate projects, both listed at gnu.org. JavaDepend
creates GNU Make dependecies, but it is easy to tweak
it into satisfying the different syntax JavaMake uses.
The one thing I am missing from JavaMake is a replacement
for the UNIX find (I am using a simplistic Makefile that
traverses the *.java source tree from the root), but for
your purposes the existing operators might be sufficient.

I was wondering - is there an org.gnu class hierarchy
already? I guess that a lot of this stuff (make, depend,
config maybe - how'd you do this in Java, getopt and 
regex, bash....) might crop up over time, especially 
when people want to deploy on Win32. It might be worth 
considering dropping this into a good place and maintain
them in a unified way?
 
                                         b.

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Tom Tromey]

>>>>> "Godmar" == Godmar Back <gback@cs.utah.edu> writes:

Godmar> Here's an example:  can your testing framework handle this test?
Godmar> [ ... ]
Godmar> For instance, Sun 1.1.3 would deadlock on this one, 1.1.6
Godmar> would produce correct output.

No, you're right, we can't do that one.

Tom

RE: Project Mauve: A Free Java Regression Test and Compatibility Package

[John Keiser]

> From: Godmar Back [ mailto:gback@cs.utah.edu ]
> Subject: Re: Project Mauve: A Free Java Regression Test and
> Compatibility Package
>
>
> >
> > I'm afraid I don't understand either of these objections.
> >
> > What is it about this approach that makes it impossible to test JNI
> > compatibility?
> >
>
>  I meant to test C programs that make use of the particular engine's JNI
> interfaces.
>

I expect (but am not sure) that there will be provisions to build test
libraries that you can interface to as native methods.  This will give you
90% of the JNI functionality.

However, the invocation interface is one thing that needs a C test with
main().  Having a section for generic tests would be good.

> <snip>
>
> >
> > I guess that's too bad.  I personally don't like it, but there it is
> > anyway.  Some of the untestable things really approach being tests of
> > the runtime and not the library per se.  I don't know if these really
> > fall within mauve's remit.
>
> That's what my concern was:  the testing requirements for the runtime
> and for the library are different, at least potentially.
> What I'd like is something that could do both.
>
> <snip>

I agree entirely.  In some areas, the runtime and library blend anyway.  It
would be nice to have the free software equivalent of the JCK here.  Test of
JNI, invocation interface, JVMDI, runtime (casting, main(), etc.), and
libraries.  Nothing that tests *internal* pieces of the runtime.  Just the
interface.

--John Keiser

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Godmar Back]

> 
> I'm afraid I don't understand either of these objections.
> 
> What is it about this approach that makes it impossible to test JNI
> compatibility?
> 

 I meant to test C programs that make use of the particular engine's JNI
interfaces.

> For the second point: are you concerned about testing for exceptions
> thrown in `main' in particular?  Or do you want to be able to test
> constructors that cause ExceptionInInitializerError to be thrown?  You
> can certainly test the latter; you just need two classes.

Here's an example:  can your testing framework handle this test?

/*
 * Try a join before main().
 */
public class JoinBeforeMain implements Runnable {
    public void run() {
        System.out.println("in run");
    }

    static {
        System.out.println("static init");
        JoinBeforeMain jbm = new JoinBeforeMain();

        Thread t = new Thread(jbm);
        t.start();
        try {
            t.join();
        } catch (Exception _) { System.out.println("caught " + _); }
    }

    public static void main(String av[]) {
    }
}

For instance, Sun 1.1.3 would deadlock on this one, 1.1.6 would
produce correct output.

> 
> I guess that's too bad.  I personally don't like it, but there it is
> anyway.  Some of the untestable things really approach being tests of
> the runtime and not the library per se.  I don't know if these really
> fall within mauve's remit.

That's what my concern was:  the testing requirements for the runtime
and for the library are different, at least potentially.
What I'd like is something that could do both.

However, something that only tests the libraries would be useful too,
of course.

> Godmar> It would also be nice if it didn't require anything but
> Godmar> /bin/sh to run.
> 
> I believe we currently require GNU make, sh, a Java runtime, and a
> Java compiler.  It would probably be possible to get rid of the GNU
> make dependency.  I'm not particularly motivated to do it.

GNU make, I think, is not ideal, but tolerable.  
I was just concerned developers would have to have guile or a particular
version of perl or something like that.

	- Godmar

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Tom Tromey]

>>>>> "Godmar" == Godmar Back <gback@cs.utah.edu> writes:

README> You'll notice that each test has a public constructor.

Godmar> That wouldn't allow me to test JNI compatibility, plus it
Godmar> would allow me to run any of the tests that deal the
Godmar> complexities of the threading and initialization parts.  (For
Godmar> instance: what happens if you get an
Godmar> ExceptionInInitializerError in the class whose main() method
Godmar> is invoked, etc.)

I'm afraid I don't understand either of these objections.

What is it about this approach that makes it impossible to test JNI
compatibility?

For the second point: are you concerned about testing for exceptions
thrown in `main' in particular?  Or do you want to be able to test
constructors that cause ExceptionInInitializerError to be thrown?  You
can certainly test the latter; you just need two classes.

Can you give me an actual example of a test you can't write in this
framework?  I mean, based on one of the above things?  I know there
are tests you can't write -- for instance, you can't test that a
running daemon thread won't prevent the test program from exiting.
Also, you can't test finalization on exit.

I guess that's too bad.  I personally don't like it, but there it is
anyway.  Some of the untestable things really approach being tests of
the runtime and not the library per se.  I don't know if these really
fall within mauve's remit.

Or maybe you're asking about what happens if there is a bug in the
test case's constructor.  All the existing constructors are empty.
That seems like a wise course to take in general.  So I'd say this
isn't an important consideration.

Godmar> I would prefer something that would simply diff the expected
Godmar> against the actual output, and voila.

I'm with you there, but most everybody else was against this.

Anyway, the plan is to add some code to the harness to let you do this
in a certain way -- you'll be able to embed the expected output in the
test case, and the harness will do the comparison.  This isn't written
yet.

Godmar> It would also be nice if it didn't require anything but
Godmar> /bin/sh to run.

I believe we currently require GNU make, sh, a Java runtime, and a
Java compiler.  It would probably be possible to get rid of the GNU
make dependency.  I'm not particularly motivated to do it.

Godmar> On the other hand, I would like it if I could use it to run a
Godmar> group of tests or single tests quickly.

You can do this.  Feed the class list you care about into the test
harness' stdin.

Note also that SimpleTestHarness is just a simple, generic test
harness.  You can write a more complicated one if you have different
needs.  We're going to write another harness in-house so we can do
DejaGNU-based embedded system testing (though we might use the simple
harness for ordinary testing)

Godmar> That is not to say that test using Class.forName etc. would be
Godmar> useless: in fact, they would probably test durability if the
Godmar> engine isn't shut down between tests.

Right now we run all the tests in a single harness invocation.
However, this isn't really fixed.  There is some info in README about
an idea to change this (grouping).  If you want, you could run a
separate invocation of the harness for each test.

Note that being able to run big batches of tests all at once is nice
for us gcj developers.  That's because we have to relink the test
harness when we compile.  If we put a main() into every test, then
we'd at least double the time it takes to run the test suite.

Tom

Re: Project Mauve: A Free Java Regression Test and Compatibility Package

[Godmar Back]

 I think mauve is a great idea.
I looked at the README file, and am somewhat concerned about this
section:

================================================================
You'll notice that each test has a public constructor.  This probably
seems strange -- why not have the `test' method be static, and justcall it?
The answer is that this approach would require java.lang.reflect.  The
test framework gets the class object for a test using Class.forName.
However, making a call to a static method on a class requires reflect.
By adding a constructor, we can just call Class.newInstance, and then
make a simple method call on the resulting object.
This requires a bit less machinery, and more importantly lets us use
it for libjava, where we haven't yet implemented reflect.
================================================================

That wouldn't allow me to test JNI compatibility, plus it would allow
me to run any of the tests that deal the complexities of the threading
and initialization parts.  (For instance: what happens if you get
an ExceptionInInitializerError in the class whose main() method is
invoked, etc.)

I would prefer something that would simply diff the expected against
the actual output, and voila.
It would also be nice if it didn't require anything but /bin/sh to run.

On the other hand, I would like it if I could use it to run a group of
tests or single tests quickly.

That is not to say that test using Class.forName etc. would be useless:
in fact, they would probably test durability if the engine isn't shut
down between tests.

	- Godmar