New runner

New runner

[Thomas Zander]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Hi,

I promised to continue work on a much much easier way for people to use 
mauve.  Many of the people I know are not able to run the tests due to the 
dependency on the autotools.
In short I plan to
a) provide a jar which anyone can run using their own preferred VM
b) provide simpler feedback in the form of html or even XML so everyone can 
put his or her results online.

I'd like to present an intermediate version for feedback and perhaps for 
inclusion in the mauve CVS (although I'm personally quite please with 
darcs :)

Please download and try on all your VMs;
http://members.home.nl/zander/alltests.jar

The resulting html of my installed vm is visible on:
http://members.home.nl/zander/results/

The sources of the runner are also provided on:
http://members.home.nl/zander/newRunner.tar.gz

I'm still going to work on an XML format, like discussed in an earlier post 
so better aggregate results can be posted by VM groups online for 
comparison.  Its not progressing as fast as I'd like; but I'm getting 
there :-)  Anyway, this is just a "I did not forget about this just yet" 
message.

I'd love to hear feedback on what is wrong/incomplete or great.
-- 
Thomas Zander

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: New runner

[Jeroen Frijters]

Thomas Zander wrote:
> The way the config settings are shipped is not the issue here;  the
> question where a socket can connect to that is reachable from 
> the testing machine so the test does not fail because of wrong
> reasons is a bigger problem.

I thought the tests only connect to an SMTP server? Presumably most
people have access to an SMTP server. You could ask a question in the
test configuration step (where you ask about awt tests).

> When we are actually running this test we already did a 'java 
> -jar' and we found several classes in that jar.  Does that not
> imply that the tests in that class will pass?

Not necessarily. With IKVM, for example, you can pre-compile the jar
into a .NET executable and run that directly, that way no Java I/O is
involved in starting up the tests. I'm sure a similar scenario exists
for gcj.

Regards,
Jeroen

Re: New runner

[Thomas Zander]

[-- Attachment #1: Type: text/plain, Size: 2446 bytes --]

On Wed, Feb 09, 2005 at 10:25:03AM +0100, Jeroen Frijters wrote:
> Thomas Zander wrote:
> > Hi, Jeroen. Thanks for your reply.
> > I've just looked into this; when I delete the config and the
> > SimpleTestHarness I get compile errors in (only) 9 tests.
> > The following are easy to fix and run correctly with some 
> > modifications
> > (read: removal of the cast):
> >     gnu/testlet/java/io/FileDescriptor/jdk11.java
> >     gnu/testlet/java/io/FileReader/jdk11.java
> >     gnu/testlet/java/io/FileWriter/jdk11.java
> >     gnu/testlet/java/io/RandomAccessFile/jdk11.java
> > 
> > These are a bit more difficult; but I believe them to be 
> > broken by design or not usable for a 'java -jar' kind of
> > test anyway.
> > I'll be more specific:
> > These 4 tests each test some socket stuff based completely on the
> > assumtion that there is someone (an smtp host) listening on a
> > pre-configured host/port.  This is quite an assumtion that is not very
> > usable in the proposed (portable) test-jar.
> >     gnu/testlet/java/net/Socket/SocketTest.java
> >     gnu/testlet/java/net/Socket/jdk12.java
> >     gnu/testlet/java/net/Socket/jdk13.java
> >     gnu/testlet/java/net/Socket/jdk14.java
> 
> I don't think they're inherently unusable from a jar, I'd like to see a
> gnu.testlet.config version that reads the settings from a properties
> file (or whatever) instead of hardcoding them in the auto dance.

The way the config settings are shipped is not the issue here;  the
question where a socket can connect to that is reachable from the testing
machine so the test does not fail because of wrong reasons is a bigger
problem. And the one I tried to list above.
When we solve that issue then we can think about how to config it.  I have
no idea how to do so; so I'm open to suggestions!

> > The following test tests things like being able to open a 
> > file we ship in mauve (in the same package) and other stuff thats
> > really really basic. (like File.seperator) Quite useless to test
> > in an environment that is shipped in a jar :-)
> >     gnu/testlet/java/io/File/jdk11.java
> 
> This is more tricky to handle, but hopefully we can figure out a way to
> get this test in as well.

When we are actually running this test we already did a 'java -jar' and we
found several classes in that jar.  Does that not imply that the tests in
that class will pass?

-- 
Thomas Zander

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: New runner

[Jeroen Frijters]

Thomas Zander wrote:
> Hi, Jeroen. Thanks for your reply.
> I've just looked into this; when I delete the config and the
> SimpleTestHarness I get compile errors in (only) 9 tests.
> The following are easy to fix and run correctly with some 
> modifications
> (read: removal of the cast):
>     gnu/testlet/java/io/FileDescriptor/jdk11.java
>     gnu/testlet/java/io/FileReader/jdk11.java
>     gnu/testlet/java/io/FileWriter/jdk11.java
>     gnu/testlet/java/io/RandomAccessFile/jdk11.java
> 
> These are a bit more difficult; but I believe them to be 
> broken by design or not usable for a 'java -jar' kind of
> test anyway.
> I'll be more specific:
> These 4 tests each test some socket stuff based completely on the
> assumtion that there is someone (an smtp host) listening on a
> pre-configured host/port.  This is quite an assumtion that is not very
> usable in the proposed (portable) test-jar.
>     gnu/testlet/java/net/Socket/SocketTest.java
>     gnu/testlet/java/net/Socket/jdk12.java
>     gnu/testlet/java/net/Socket/jdk13.java
>     gnu/testlet/java/net/Socket/jdk14.java

I don't think they're inherently unusable from a jar, I'd like to see a
gnu.testlet.config version that reads the settings from a properties
file (or whatever) instead of hardcoding them in the auto dance.

> The following test tests things like being able to open a 
> file we ship in mauve (in the same package) and other stuff thats
> really really basic. (like File.seperator) Quite useless to test
> in an environment that is shipped in a jar :-)
>     gnu/testlet/java/io/File/jdk11.java

This is more tricky to handle, but hopefully we can figure out a way to
get this test in as well.

Regards,
Jeroen

Re: New runner

[Thomas Zander]

[-- Attachment #1: Type: text/plain, Size: 2012 bytes --]

On Mon, Feb 07, 2005 at 09:54:54AM +0100, Jeroen Frijters wrote:
> Thomas Zander wrote:
> > Please download and try on all your VMs;
> > http://members.home.nl/zander/alltests.jar
> 
> On my (Windows) system a bunch of tests fail because the
> gnu.testlet.config settings don't make sense. Also, some of the socket
> tests fail because they do a downcast from TestHarness to
> SimpleTestHarness.

Hi, Jeroen. Thanks for your reply.
I've just looked into this; when I delete the config and the
SimpleTestHarness I get compile errors in (only) 9 tests.
The following are easy to fix and run correctly with some modifications
(read: removal of the cast):
    gnu/testlet/java/io/FileDescriptor/jdk11.java
    gnu/testlet/java/io/FileReader/jdk11.java
    gnu/testlet/java/io/FileWriter/jdk11.java
    gnu/testlet/java/io/RandomAccessFile/jdk11.java

These are a bit more difficult; but I believe them to be broken by design
or not usable for a 'java -jar' kind of test anyway.
I'll be more specific:
These 4 tests each test some socket stuff based completely on the
assumtion that there is someone (an smtp host) listening on a
pre-configured host/port.  This is quite an assumtion that is not very
usable in the proposed (portable) test-jar.
    gnu/testlet/java/net/Socket/SocketTest.java
    gnu/testlet/java/net/Socket/jdk12.java
    gnu/testlet/java/net/Socket/jdk13.java
    gnu/testlet/java/net/Socket/jdk14.java

The following test tests things like being able to open a file we ship in
mauve (in the same package) and other stuff thats really really basic.
(like File.seperator) Quite useless to test in an environment that is
shipped in a jar :-)
    gnu/testlet/java/io/File/jdk11.java

What do you think?  Should these last 5 tests be modified in any way? Or
should we keep the dependency on config (and SimpleTestHarness) for them
even though they have a different set of problems?

Whatever we decide, I'll stop shipping those in my jar.
-- 
Thomas Zander

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: New runner

[Jeroen Frijters]

Thomas Zander wrote:
> Please download and try on all your VMs;
> http://members.home.nl/zander/alltests.jar

On my (Windows) system a bunch of tests fail because the
gnu.testlet.config settings don't make sense. Also, some of the socket
tests fail because they do a downcast from TestHarness to
SimpleTestHarness.

Thanks,
Jeroen