Library/VM tests

Library/VM tests

[Tim Wilkinson]

I have (finally) been looking into integrating our modest testsuite into
Mauve but have run up against a problem.  So far as I understand things
Mauve is designed to test *only* the library functionality.  Given this
what are we suppose to do about test which sit on the boundary -
ClassLoaders spring to mind but there's also such things as
Serialization, Reflection and bunch of other borderline stuff.  None of
this is VM specific of course but it relies on more heavy VM support
than some tests (some stuff uses threads for example).

So my question is - what's the policy on this?

Cheers
Tim

--
  Tim Wilkinson                         Tel:     +1 510 704 1660
  Transvirtual Technologies, Inc.,      Fax:     +1 510 704 1893
  Berkeley, CA, USA.                    Email:   tim@transvirtual.com

Re: Library/VM tests

[Tom Tromey]

>>>>> "Tim" == Tim Wilkinson <tim@transvirtual.com> writes:

Tim> I have (finally) been looking into integrating our modest
Tim> testsuite into Mauve but have run up against a problem.  So far
Tim> as I understand things Mauve is designed to test *only* the
Tim> library functionality.  Given this what are we suppose to do
Tim> about test which sit on the boundary - ClassLoaders spring to
Tim> mind but there's also such things as Serialization, Reflection
Tim> and bunch of other borderline stuff.  None of this is VM specific
Tim> of course but it relies on more heavy VM support than some tests
Tim> (some stuff uses threads for example).

Tim> So my question is - what's the policy on this?

Policy is really for us to decide as a group.  The original goal was
to write a test suite for the class libraries, but I'm beginning to
think we'd be better off expanding the scope to include VM/runtime
tests as well.  After all, why not?  This will only benefit everybody.

I'm not fully clear on all the issues in terms of implementation.
However, I'm pretty clear on what Cygnus' requirements are in terms of
functionality.  I'd be happy to discuss this with anybody
contemplating revising Mauve to include more kinds of tests (that is,
if the test framework needs changing.  I don't know whether it does).

Tom

Re: Library/VM tests

[Moses DeJong]

On 4 Mar 1999, Tom Tromey wrote:

> >>>>> "Tim" == Tim Wilkinson <tim@transvirtual.com> writes:
> 
> Tim> I have (finally) been looking into integrating our modest
> Tim> testsuite into Mauve but have run up against a problem.  So far
> Tim> as I understand things Mauve is designed to test *only* the
> Tim> library functionality.  Given this what are we suppose to do
> Tim> about test which sit on the boundary - ClassLoaders spring to
> Tim> mind but there's also such things as Serialization, Reflection
> Tim> and bunch of other borderline stuff.  None of this is VM specific
> Tim> of course but it relies on more heavy VM support than some tests
> Tim> (some stuff uses threads for example).
> 
> Tim> So my question is - what's the policy on this?
> 
> Policy is really for us to decide as a group.  The original goal was
> to write a test suite for the class libraries, but I'm beginning to
> think we'd be better off expanding the scope to include VM/runtime
> tests as well.  After all, why not?  This will only benefit everybody.
> 
> I'm not fully clear on all the issues in terms of implementation.
> However, I'm pretty clear on what Cygnus' requirements are in terms of
> functionality.  I'd be happy to discuss this with anybody
> contemplating revising Mauve to include more kinds of tests (that is,
> if the test framework needs changing.  I don't know whether it does).
> 
> Tom
> 


I have some tests that fall into this "other" catagory.
I developed some test test cases for the reflection API
for the Kaffe project but I think they really should but added
to the mauve tests. Is this the kind of thing that is not covered
under "libraries" tests?

Mo DeJong
dejong at cs.umn.edu

Re: Library/VM tests

[Godmar Back]

> 
> I have some tests that fall into this "other" catagory.
> I developed some test test cases for the reflection API
> for the Kaffe project but I think they really should but added
> to the mauve tests. Is this the kind of thing that is not covered
> under "libraries" tests?
> 

My take on this is that all tests that can be put in the existing
mauve framework are "library" tests, and all tests that cannot or for
which it would be unreasonably cumbersome to do so are "VM" tests.
(I admit that's a somewhat coarse and loose definition.)
For the latter kind, the VM tests, I think starting up a VM and diffing 
the output against the expected output is the most straightforward thing
to do.

	- Godmar

Re: Library/VM tests

[Godmar Back]

> 
> I have some tests that fall into this "other" catagory.
> I developed some test test cases for the reflection API
> for the Kaffe project but I think they really should but added
> to the mauve tests. Is this the kind of thing that is not covered
> under "libraries" tests?
> 

My take on this is that all tests that can be put in the existing
mauve framework are "library" tests, and all tests that cannot or for
which it would be unreasonably cumbersome to do so are "VM" tests.
(I admit that's a somewhat coarse and loose definition.)
For the latter kind, the VM tests, I think starting up a VM and diffing 
the output against the expected output is the most straightforward thing
to do.

	- Godmar

Re: Library/VM tests

[Moses DeJong]

On 4 Mar 1999, Tom Tromey wrote:

> >>>>> "Tim" == Tim Wilkinson <tim@transvirtual.com> writes:
> 
> Tim> I have (finally) been looking into integrating our modest
> Tim> testsuite into Mauve but have run up against a problem.  So far
> Tim> as I understand things Mauve is designed to test *only* the
> Tim> library functionality.  Given this what are we suppose to do
> Tim> about test which sit on the boundary - ClassLoaders spring to
> Tim> mind but there's also such things as Serialization, Reflection
> Tim> and bunch of other borderline stuff.  None of this is VM specific
> Tim> of course but it relies on more heavy VM support than some tests
> Tim> (some stuff uses threads for example).
> 
> Tim> So my question is - what's the policy on this?
> 
> Policy is really for us to decide as a group.  The original goal was
> to write a test suite for the class libraries, but I'm beginning to
> think we'd be better off expanding the scope to include VM/runtime
> tests as well.  After all, why not?  This will only benefit everybody.
> 
> I'm not fully clear on all the issues in terms of implementation.
> However, I'm pretty clear on what Cygnus' requirements are in terms of
> functionality.  I'd be happy to discuss this with anybody
> contemplating revising Mauve to include more kinds of tests (that is,
> if the test framework needs changing.  I don't know whether it does).
> 
> Tom
> 


I have some tests that fall into this "other" catagory.
I developed some test test cases for the reflection API
for the Kaffe project but I think they really should but added
to the mauve tests. Is this the kind of thing that is not covered
under "libraries" tests?

Mo DeJong
dejong at cs.umn.edu

Re: Library/VM tests

[Tom Tromey]

>>>>> "Tim" == Tim Wilkinson <tim@transvirtual.com> writes:

Tim> I have (finally) been looking into integrating our modest
Tim> testsuite into Mauve but have run up against a problem.  So far
Tim> as I understand things Mauve is designed to test *only* the
Tim> library functionality.  Given this what are we suppose to do
Tim> about test which sit on the boundary - ClassLoaders spring to
Tim> mind but there's also such things as Serialization, Reflection
Tim> and bunch of other borderline stuff.  None of this is VM specific
Tim> of course but it relies on more heavy VM support than some tests
Tim> (some stuff uses threads for example).

Tim> So my question is - what's the policy on this?

Policy is really for us to decide as a group.  The original goal was
to write a test suite for the class libraries, but I'm beginning to
think we'd be better off expanding the scope to include VM/runtime
tests as well.  After all, why not?  This will only benefit everybody.

I'm not fully clear on all the issues in terms of implementation.
However, I'm pretty clear on what Cygnus' requirements are in terms of
functionality.  I'd be happy to discuss this with anybody
contemplating revising Mauve to include more kinds of tests (that is,
if the test framework needs changing.  I don't know whether it does).

Tom

Re: Library/VM tests

[Artur Biesiadowski]

Tim Wilkinson wrote:
> 
> I have (finally) been looking into integrating our modest testsuite into
> Mauve but have run up against a problem.  So far as I understand things
> Mauve is designed to test *only* the library functionality.  Given this
> what are we suppose to do about test which sit on the boundary -
> ClassLoaders spring to mind but there's also such things as
> Serialization, Reflection and bunch of other borderline stuff.  None of
> this is VM specific of course but it relies on more heavy VM support
> than some tests (some stuff uses threads for example).
> 
> So my question is - what's the policy on this?

I think that as long as you test API, not VM it is ok. I mean that you
should not test if
array[-1]
throws ArrayIndexOutOfBounds exception, but if Vector.elementAt(-1)
throws such exception (even if it uses the same mechanism). In other
words, you can suppose that VM is perfect - only core lib can be faulty.

So, my personal opinion is that things that you mentioned fir into
mauve. Example fo thing that will not fit is StackOverflowException or
OutOfMemoryException testing - they are stritlty VM test, as there is
really not much to test in actual classes, and have a side effect of
possibly crashing entire test suite. I think that such tests (together
with bytecode tests etc), should be done separately - out of mauve.

Artur

Re: Library/VM tests

[Brian Jones]

Artur Biesiadowski <abies@pg.gda.pl> writes:

> Tim Wilkinson wrote:
> > 
> > I have (finally) been looking into integrating our modest testsuite into
> > Mauve but have run up against a problem.  So far as I understand things
> > Mauve is designed to test *only* the library functionality.  Given this
> > what are we suppose to do about test which sit on the boundary -
> > ClassLoaders spring to mind but there's also such things as
> > Serialization, Reflection and bunch of other borderline stuff.  None of
> > this is VM specific of course but it relies on more heavy VM support
> > than some tests (some stuff uses threads for example).
> > 
> > So my question is - what's the policy on this?
> 
> So, my personal opinion is that things that you mentioned fir into
> mauve. Example fo thing that will not fit is StackOverflowException or
> OutOfMemoryException testing - they are stritlty VM test, as there is
> really not much to test in actual classes, and have a side effect of
> possibly crashing entire test suite. I think that such tests (together
> with bytecode tests etc), should be done separately - out of mauve.

My personal opinion is that such a testing facility is needed for all
the free VMs including Kaffe, Japhar, etc.  If you want to look at the
Mauve project as containing a couple of modules, one for testing core
libraries and the other for testing vms/jit, then I think that would
work well.  What, if anything, has to be done to the testing framework
seems like a separate issue to me as well.

Brian
-- 
|-------------------------------|
|Brian Jones			|
|cbj@gnu.org			|
| http://www.classpath.org/      |

Re: Library/VM tests

[Brian Jones]

Artur Biesiadowski <abies@pg.gda.pl> writes:

> Tim Wilkinson wrote:
> > 
> > I have (finally) been looking into integrating our modest testsuite into
> > Mauve but have run up against a problem.  So far as I understand things
> > Mauve is designed to test *only* the library functionality.  Given this
> > what are we suppose to do about test which sit on the boundary -
> > ClassLoaders spring to mind but there's also such things as
> > Serialization, Reflection and bunch of other borderline stuff.  None of
> > this is VM specific of course but it relies on more heavy VM support
> > than some tests (some stuff uses threads for example).
> > 
> > So my question is - what's the policy on this?
> 
> So, my personal opinion is that things that you mentioned fir into
> mauve. Example fo thing that will not fit is StackOverflowException or
> OutOfMemoryException testing - they are stritlty VM test, as there is
> really not much to test in actual classes, and have a side effect of
> possibly crashing entire test suite. I think that such tests (together
> with bytecode tests etc), should be done separately - out of mauve.

My personal opinion is that such a testing facility is needed for all
the free VMs including Kaffe, Japhar, etc.  If you want to look at the
Mauve project as containing a couple of modules, one for testing core
libraries and the other for testing vms/jit, then I think that would
work well.  What, if anything, has to be done to the testing framework
seems like a separate issue to me as well.

Brian
-- 
|-------------------------------|
|Brian Jones			|
|cbj@gnu.org			|
| http://www.classpath.org/      |

Re: Library/VM tests

[Tim Wilkinson]

I guess I wasn't thinking of testing the basic operations of the VM -
but
when it comes down to it they should be tested too don't you think -
after
all they have to work correctly too right?

Perhaps the testsuite should be split into various sub-trees contains
test
designed to go after particular bits of the Java implementation - you
could
put array[-1] in the tests designed to check the VM in one area and
leave the
API tests in another.

For such things as serialization I'm not sure what the best structure
would
be - arguably you can serialize any class (even ones which don't
implement
Serializable have a behaviour that can be checked) - perhaps you just
put a
serializable check in each class test directory, or maybe you lump the
serialization together (I'd go for the first option).

Tim

Artur Biesiadowski wrote:

> I think that as long as you test API, not VM it is ok. I mean that you

> should not test if
> array[-1]
> throws ArrayIndexOutOfBounds exception, but if Vector.elementAt(-1)
> throws such exception (even if it uses the same mechanism). In other
> words, you can suppose that VM is perfect - only core lib can be
faulty.
>
> So, my personal opinion is that things that you mentioned fir into
> mauve. Example fo thing that will not fit is StackOverflowException or

> OutOfMemoryException testing - they are stritlty VM test, as there is
> really not much to test in actual classes, and have a side effect of
> possibly crashing entire test suite. I think that such tests (together

> with bytecode tests etc), should be done separately - out of mauve.
>
> Artur



--
  Tim Wilkinson                         Tel:     +1 510 704 1660
  Transvirtual Technologies, Inc.,      Fax:     +1 510 704 1893
  Berkeley, CA, USA.                    Email:   tim@transvirtual.com

Re: Library/VM tests

[Tom Tromey]

>>>>> "Tim" == Tim Wilkinson <tim@transvirtual.com> writes:

Tim> I guess I wasn't thinking of testing the basic operations of the
Tim> VM - but when it comes down to it they should be tested too don't
Tim> you think - after all they have to work correctly too right?

I agree.

Tim> Perhaps the testsuite should be split into various sub-trees
Tim> contains test designed to go after particular bits of the Java
Tim> implementation - you could put array[-1] in the tests designed to
Tim> check the VM in one area and leave the API tests in another.

This sounds good to me.

Tim> For such things as serialization I'm not sure what the best
Tim> structure would be - arguably you can serialize any class (even
Tim> ones which don't implement Serializable have a behaviour that can
Tim> be checked) - perhaps you just put a serializable check in each
Tim> class test directory, or maybe you lump the serialization
Tim> together (I'd go for the first option).

This sounds good too.  We just need to mark such tests with some kind
of serialization-specific tag.


Is there anybody who is *against* expanding Mauve to include VM tests?
If not, then we should go ahead and talk about requirements and
implementation.

Tom

VM tests [was Re: Library/VM tests]

[Artur Biesiadowski]

Tom Tromey wrote:

> Is there anybody who is *against* expanding Mauve to include VM tests?
> If not, then we should go ahead and talk about requirements and
> implementation.

This certainly would reuqire very different handling.
I think that it can be done by creating some informative directory
structure and then executing each class in separate VM by command line
(from makefile or perl or some other script).

A lot of tests should be written directly in assembler - you cannot be
sure how compilers will translate given code. With assembler we can test
verious verify error, some magic combinations of stack/jumps which would
be impossible from java. This requires java assembler - there are some
available on the net, with jasmin coming to mind, but it is a bit large
(450kb packed ?).

JNI tests would require C compiler and some knowledge about runtime libs
of given vm. Smart configure script should be enough.

Some way would have to be defined to kill infinite loops. Maybe second
process sleeping for few minutes ?

Artur

Re: VM tests [was Re: Library/VM tests]

[Godmar Back]

 For a start, we could just use the kaffe/test/regression/TestScript.in
Alexandre Oliva wrote.

It allows compile options and the expected output to be hidden in a
comment in the .java file, runs a test and (optionally) sorts and
diffs the output.

It does not support java assembler, JNI or native tests nor does it
support timeouts.  These are all very useful features.  It's using sh
at this point, but I wouldn't mind using perl.

	- Godmar

> 
> This certainly would reuqire very different handling.
> I think that it can be done by creating some informative directory
> structure and then executing each class in separate VM by command line
> (from makefile or perl or some other script).
> 
> A lot of tests should be written directly in assembler - you cannot be
> sure how compilers will translate given code. With assembler we can test
> verious verify error, some magic combinations of stack/jumps which would
> be impossible from java. This requires java assembler - there are some
> available on the net, with jasmin coming to mind, but it is a bit large
> (450kb packed ?).
> 
> JNI tests would require C compiler and some knowledge about runtime libs
> of given vm. Smart configure script should be enough.
> 
> Some way would have to be defined to kill infinite loops. Maybe second
> process sleeping for few minutes ?
> 
> Artur
> 

Re: VM tests [was Re: Library/VM tests]

[Godmar Back]

 For a start, we could just use the kaffe/test/regression/TestScript.in
Alexandre Oliva wrote.

It allows compile options and the expected output to be hidden in a
comment in the .java file, runs a test and (optionally) sorts and
diffs the output.

It does not support java assembler, JNI or native tests nor does it
support timeouts.  These are all very useful features.  It's using sh
at this point, but I wouldn't mind using perl.

	- Godmar

> 
> This certainly would reuqire very different handling.
> I think that it can be done by creating some informative directory
> structure and then executing each class in separate VM by command line
> (from makefile or perl or some other script).
> 
> A lot of tests should be written directly in assembler - you cannot be
> sure how compilers will translate given code. With assembler we can test
> verious verify error, some magic combinations of stack/jumps which would
> be impossible from java. This requires java assembler - there are some
> available on the net, with jasmin coming to mind, but it is a bit large
> (450kb packed ?).
> 
> JNI tests would require C compiler and some knowledge about runtime libs
> of given vm. Smart configure script should be enough.
> 
> Some way would have to be defined to kill infinite loops. Maybe second
> process sleeping for few minutes ?
> 
> Artur
> 

VM tests [was Re: Library/VM tests]

[Artur Biesiadowski]

Tom Tromey wrote:

> Is there anybody who is *against* expanding Mauve to include VM tests?
> If not, then we should go ahead and talk about requirements and
> implementation.

This certainly would reuqire very different handling.
I think that it can be done by creating some informative directory
structure and then executing each class in separate VM by command line
(from makefile or perl or some other script).

A lot of tests should be written directly in assembler - you cannot be
sure how compilers will translate given code. With assembler we can test
verious verify error, some magic combinations of stack/jumps which would
be impossible from java. This requires java assembler - there are some
available on the net, with jasmin coming to mind, but it is a bit large
(450kb packed ?).

JNI tests would require C compiler and some knowledge about runtime libs
of given vm. Smart configure script should be enough.

Some way would have to be defined to kill infinite loops. Maybe second
process sleeping for few minutes ?

Artur

Re: Library/VM tests

[Tom Tromey]

>>>>> "Tim" == Tim Wilkinson <tim@transvirtual.com> writes:

Tim> I guess I wasn't thinking of testing the basic operations of the
Tim> VM - but when it comes down to it they should be tested too don't
Tim> you think - after all they have to work correctly too right?

I agree.

Tim> Perhaps the testsuite should be split into various sub-trees
Tim> contains test designed to go after particular bits of the Java
Tim> implementation - you could put array[-1] in the tests designed to
Tim> check the VM in one area and leave the API tests in another.

This sounds good to me.

Tim> For such things as serialization I'm not sure what the best
Tim> structure would be - arguably you can serialize any class (even
Tim> ones which don't implement Serializable have a behaviour that can
Tim> be checked) - perhaps you just put a serializable check in each
Tim> class test directory, or maybe you lump the serialization
Tim> together (I'd go for the first option).

This sounds good too.  We just need to mark such tests with some kind
of serialization-specific tag.


Is there anybody who is *against* expanding Mauve to include VM tests?
If not, then we should go ahead and talk about requirements and
implementation.

Tom

Re: Library/VM tests

[Tim Wilkinson]

I guess I wasn't thinking of testing the basic operations of the VM -
but
when it comes down to it they should be tested too don't you think -
after
all they have to work correctly too right?

Perhaps the testsuite should be split into various sub-trees contains
test
designed to go after particular bits of the Java implementation - you
could
put array[-1] in the tests designed to check the VM in one area and
leave the
API tests in another.

For such things as serialization I'm not sure what the best structure
would
be - arguably you can serialize any class (even ones which don't
implement
Serializable have a behaviour that can be checked) - perhaps you just
put a
serializable check in each class test directory, or maybe you lump the
serialization together (I'd go for the first option).

Tim

Artur Biesiadowski wrote:

> I think that as long as you test API, not VM it is ok. I mean that you

> should not test if
> array[-1]
> throws ArrayIndexOutOfBounds exception, but if Vector.elementAt(-1)
> throws such exception (even if it uses the same mechanism). In other
> words, you can suppose that VM is perfect - only core lib can be
faulty.
>
> So, my personal opinion is that things that you mentioned fir into
> mauve. Example fo thing that will not fit is StackOverflowException or

> OutOfMemoryException testing - they are stritlty VM test, as there is
> really not much to test in actual classes, and have a side effect of
> possibly crashing entire test suite. I think that such tests (together

> with bytecode tests etc), should be done separately - out of mauve.
>
> Artur



--
  Tim Wilkinson                         Tel:     +1 510 704 1660
  Transvirtual Technologies, Inc.,      Fax:     +1 510 704 1893
  Berkeley, CA, USA.                    Email:   tim@transvirtual.com

Re: Library/VM tests

[Artur Biesiadowski]

Tim Wilkinson wrote:
> 
> I have (finally) been looking into integrating our modest testsuite into
> Mauve but have run up against a problem.  So far as I understand things
> Mauve is designed to test *only* the library functionality.  Given this
> what are we suppose to do about test which sit on the boundary -
> ClassLoaders spring to mind but there's also such things as
> Serialization, Reflection and bunch of other borderline stuff.  None of
> this is VM specific of course but it relies on more heavy VM support
> than some tests (some stuff uses threads for example).
> 
> So my question is - what's the policy on this?

I think that as long as you test API, not VM it is ok. I mean that you
should not test if
array[-1]
throws ArrayIndexOutOfBounds exception, but if Vector.elementAt(-1)
throws such exception (even if it uses the same mechanism). In other
words, you can suppose that VM is perfect - only core lib can be faulty.

So, my personal opinion is that things that you mentioned fir into
mauve. Example fo thing that will not fit is StackOverflowException or
OutOfMemoryException testing - they are stritlty VM test, as there is
really not much to test in actual classes, and have a side effect of
possibly crashing entire test suite. I think that such tests (together
with bytecode tests etc), should be done separately - out of mauve.

Artur

Library/VM tests

[Tim Wilkinson]

I have (finally) been looking into integrating our modest testsuite into
Mauve but have run up against a problem.  So far as I understand things
Mauve is designed to test *only* the library functionality.  Given this
what are we suppose to do about test which sit on the boundary -
ClassLoaders spring to mind but there's also such things as
Serialization, Reflection and bunch of other borderline stuff.  None of
this is VM specific of course but it relies on more heavy VM support
than some tests (some stuff uses threads for example).

So my question is - what's the policy on this?

Cheers
Tim

--
  Tim Wilkinson                         Tel:     +1 510 704 1660
  Transvirtual Technologies, Inc.,      Fax:     +1 510 704 1893
  Berkeley, CA, USA.                    Email:   tim@transvirtual.com