Re: [Kissme-general] Re: Should I or not submit changes?

Re: [Kissme-general] Re: Should I or not submit changes?

[Stephen Crawley]

> I added your changes, but I can't figure out how to get the .in files to
> be turned into .sh files.

They would simply need to be added to the AC_CONFIG_FILES macro call in
Kissme's configure.ac file.

However ... I've looked at Alex's scripts, and concluded that it would be
better to use the 'mauve-kissme' key file that Mark Wielaard emailed last
week, especially if we can put that file into CVS.  

The problem is that if you run Mauve tests the way that the Mauve README
file says you should, the key file has to live in the Mauve directory.
This is real nuisance.  [I can arrange that the Kissme make rules for
running the Mauve tests copy the key file to the place where Mauve 
expects it to be ... but that's a HACK.]

Here are some other beefs that I have with Mauve:

  *  Mauve's "make clean" doesn't delete the .class files.

  *  Someone has checked some files generated by autoconf, etcetera
     (configure and Makefile IIRC).   These contain dependencies on
     the version of autoconf, etcetera used on said person's machine
     and break on my machine.  These generated files need to be 
     removed from CVS.

  *  Mauve needs a global cvsignore file to ignore *.class files and
     a .cvsignore file on the top directory to ignore the files created
     by autoconf and configure.

  *  The Mauve configure.ac metascript should have a --with option for
     specifying a directory or search path for finding key files.  
     
  *  The Mauve makefile should notice when I have edited the current
     key file, and rebuild the class list file.

     At the moment, I think that the class list only gets rebuilt 
     if you change the value of the KEYS environment variable.

-- Steve

Re: [Kissme-general] Re: Should I or not submit changes?

[Brian Jones]

Stephen Crawley <crawley@dstc.edu.au> writes:

> The problem is that if you run Mauve tests the way that the Mauve README
> file says you should, the key file has to live in the Mauve directory.
> This is real nuisance.  [I can arrange that the Kissme make rules for
> running the Mauve tests copy the key file to the place where Mauve 
> expects it to be ... but that's a HACK.]

I agree.

> HERE are some other beefs that I have with Mauve:
> 
>   *  Mauve's "make clean" doesn't delete the .class files.

Yes.

> 
>   *  Someone has checked some files generated by autoconf, etcetera
>      (configure and Makefile IIRC).   These contain dependencies on
>      the version of autoconf, etcetera used on said person's machine
>      and break on my machine.  These generated files need to be 
>      removed from CVS.

Has always been that way, but I don't know why.  I really dislike
seeing these files checked into CVS.  I even updated them once.

> 
>   *  Mauve needs a global cvsignore file to ignore *.class files and
>      a .cvsignore file on the top directory to ignore the files created
>      by autoconf and configure.

Yes, I'm sure these details need work.  Mauve needs dedicated contributors.

> 
>   *  The Mauve configure.ac metascript should have a --with option for
>      specifying a directory or search path for finding key files.  
>      
>   *  The Mauve makefile should notice when I have edited the current
>      key file, and rebuild the class list file.
> 
>      At the moment, I think that the class list only gets rebuilt 
>      if you change the value of the KEYS environment variable.

Yes, it really sucks to try to use Mauve in certain ways.  If anyone
has loads of time it would probably be more interesting to develop
junit based tests from the classpath source as a start and gradual
port of the Mauve tests to what is usually a more familiar test
framework to developers at large.

Brian
-- 
Brian Jones <cbj@gnu.org>

Re: [Kissme-general] Re: Should I or not submit changes?

[Dalibor Topic]

--- Brian Jones <cbj@gnu.org> wrote:
> Stephen Crawley <crawley@dstc.edu.au> writes:

> Yes, it really sucks to try to use Mauve in certain
> ways.  If anyone
> has loads of time it would probably be more
> interesting to develop
> junit based tests from the classpath source as a
> start and gradual
> port of the Mauve tests to what is usually a more
> familiar test
> framework to developers at large.

Junit has been lightly discussed on mauve this spring.
http://sources.redhat.com/ml/mauve-discuss/2002-q2/msg00037.html

It has also came up on the classpath X mailing list:
http://mail.gnu.org/pipermail/classpathx-discuss/2001-December/000095.html

There is a nice discussion on pros and cons of junit
on the xml-dev list:
http://lists.xml.org/archives/xml-dev/200107/msg00131.html

Porting tests over to Junit might not be too exciting,
though. I believe that (if the mauve hackers decide to
allow junit tests) it would be a better option to have
both frameworks in parallel for a while. As junit
relies on reflection, it wouldn't make much sense to
run it on an implementation with broken reflection
libraries. ;)

The major point junit has for it, in my opinion, is
the amount of documentation surrounding it: books,
tutorials etc. It seems to be easy to find answers.
That could lower the entry barrier for fresh mauve
contributors.

best regards,

dalibor topic

__________________________________________________
Do You Yahoo!?
Yahoo! Autos - Get free new car price quotes
http://autos.yahoo.com

Re: [Kissme-general] Re: Should I or not submit changes?

[Andrew Haley]

Brian Jones writes:
 > 
 > Yes, it really sucks to try to use Mauve in certain ways.  If anyone
 > has loads of time it would probably be more interesting to develop
 > junit based tests from the classpath source as a start and gradual
 > port of the Mauve tests to what is usually a more familiar test
 > framework to developers at large.

If mauve wasn't such a pain to use and hack on I certainly would have
contributed more to it.  Dejagnu, although horrible in its own special
way, at least allows individual tests to be developed and used without
needing the framework.

Andrew.

Re: [Kissme-general] Re: Should I or not submit changes?

[Alex Lau]

Stephen Crawley

>>I added your changes, but I can't figure out how to get the .in files to
>>be turned into .sh files.
>>    
>>
>
>They would simply need to be added to the AC_CONFIG_FILES macro call in
>Kissme's configure.ac file.
>
>However ... I've looked at Alex's scripts, and concluded that it would be
>better to use the 'mauve-kissme' key file that Mark Wielaard emailed last
>week, especially if we can put that file into CVS.  
>
I couldn't find the mauve-kissme file ...
I don't know how that key file look like, but I still have a hard time
using a key file to run that against Mauve... that's why I try to upload 
those file.
Yes I know it is not an officeal way to run their test.... but it at 
least able to
generate report for anything sub directory on any java and javax classes.

>
>The problem is that if you run Mauve tests the way that the Mauve README
>file says you should, the key file has to live in the Mauve directory.
>This is real nuisance.  [I can arrange that the Kissme make rules for
>running the Mauve tests copy the key file to the place where Mauve 
>expects it to be ... but that's a HACK.]
>
That's a question I like to know. If Mauve mean to be a testing tool...
should the script for specific VM or Classpath located at Mauve CVS
or the other way?
If you look at the buildtest.sh script it, it basically just look through
the directory and find all the tests then build some classes files for it.
I just through that will be easier for everyone to test a subset of the
classpath rather then all classes or one class.

>
>Here are some other beefs that I have with Mauve:
>
>  *  Mauve's "make clean" doesn't delete the .class files.
>  
>
Yup...

>  *  Someone has checked some files generated by autoconf, etcetera
>     (configure and Makefile IIRC).   These contain dependencies on
>     the version of autoconf, etcetera used on said person's machine
>     and break on my machine.  These generated files need to be 
>     removed from CVS.
>
>
>  *  Mauve needs a global cvsignore file to ignore *.class files and
>     a .cvsignore file on the top directory to ignore the files created
>     by autoconf and configure.
>
who is the gate keeper for the cvs...this shouldn't be that diffcult to 
put in.

>
>  *  The Mauve configure.ac metascript should have a --with option for
>     specifying a directory or search path for finding key files.  
>  
>
The key option always confused me....

>     
>  *  The Mauve makefile should notice when I have edited the current
>     key file, and rebuild the class list file.
>  
>
>     At the moment, I think that the class list only gets rebuilt 
>     if you change the value of the KEYS environment variable.
>
is that right? no wonder I couldn't do anything to the class list.....
The keything is not very "user friendly" .... once again that is why I
try to put in those script to help me generate keys... I really would
like to start writing test cases... but before that I think we need a
more complete mauve to do that. The whole system need
someone to upkeep and make it work the way it should be...
( as a matter of fact for more then a month I still not sure how
should it really work .... )



Dalibor Topic wrote:

>Porting tests over to Junit might not be too exciting,
>though. I believe that (if the mauve hackers decide to
>allow junit tests) it would be a better option to have
>both frameworks in parallel for a while. As junit
>relies on reflection, it wouldn't make much sense to
>run it on an implementation with broken reflection
>libraries. ;)

That's the trick I guess... if reflection is working :)
The question I always get is how much should the testing
tool depend on the VM and the Classpath?


Alex

Re: [Kissme-general] Re: Should I or not submit changes?

[Stephen Crawley]

Alex,

> I couldn't find the mauve-kissme file ...

It is there now.  Look in the kissme CVS for useful-scripts/mauve-kissme.  
Also look at the 'run-mauve-regression' rule in the latest version of the
Makefile.

[I'm sorry this took so long.  I trashed my home directory and had
to rebuild it.  :-(]

-- Steve

Re: [Kissme-general] Re: Should I or not submit changes?

[Dalibor Topic]

--- Alex Lau <alex@dentonlive.com> wrote:
> >  *  The Mauve makefile should notice when I have
> edited the current
> >     key file, and rebuild the class list file.
> >  
> >
> >     At the moment, I think that the class list
> only gets rebuilt 
> >     if you change the value of the KEYS
> environment variable.
> >
> is that right? no wonder I couldn't do anything to
> the class list.....
> The keything is not very "user friendly" .... once
> again that is why I
> try to put in those script to help me generate
> keys... I really would
> like to start writing test cases... but before that
> I think we need a
> more complete mauve to do that. The whole system
> need
> someone to upkeep and make it work the way it should
> be...
> ( as a matter of fact for more then a month I still
> not sure how
> should it really work .... )

I tried to use the key files when I started to play
around with mauve. Couldn't get it to work. Now I make
sure that I always pass KEYS to make check.

> Dalibor Topic wrote:
> 
> >Porting tests over to Junit might not be too
> exciting,
> >though. I believe that (if the mauve hackers decide
> to
> >allow junit tests) it would be a better option to
> have
> >both frameworks in parallel for a while. As junit
> >relies on reflection, it wouldn't make much sense
> to
> >run it on an implementation with broken reflection
> >libraries. ;)
> 
> That's the trick I guess... if reflection is working
> :)
> The question I always get is how much should the
> testing
> tool depend on the VM and the Classpath?

In my opinion, not at all, if we are still talking
about mauve. Mauve should just test the API spec and
avoid testing properties of a specific implementation.
  

Take java.lang.Character tests, for example. According
to the JLS second edition, Sun used Unicode 1.1.5 in
java 1.0, unicode 2.0 up to java 1.1.7 and unicode 2.1
from that up to 1.4, I think. Java 1.4 uses unicode
3.0. What do you test for?

best regards,

dalibor topic


__________________________________________________
Do You Yahoo!?
Yahoo! Autos - Get free new car price quotes
http://autos.yahoo.com

Re: [Kissme-general] Re: Should I or not submit changes?

[Alex Lau]

Stephen Crawley

>Alex,
>
>  
>
>>I couldn't find the mauve-kissme file ...
>>    
>>
>
>It is there now.  Look in the kissme CVS for useful-scripts/mauve-kissme.  
>Also look at the 'run-mauve-regression' rule in the latest version of the
>Makefile.
>
>[I'm sorry this took so long.  I trashed my home directory and had
>to rebuild it.  :-(]
>
>  
>
not a problem :) I'm still playing with all the script and hope to able 
to find a
way to make the testing and generating test and running test a lot easier
then right now.

I see how you use the mauve-kissme to run the full test ( regression ) but
I have a different reason to use the buildtest.sh script,  is isolating the
range of the test. Like in the mauve-kissme, it identifly some test is 
no good
to run. Unless who has a good understanding of mauve and kissme and
classpath code like you, or else it will very hard to identifly them. 
For me it is
not easy to sort it out. So far what I have been trying to do is take 
one sub
directory at a time to run the test on... and that is what buildtest.sh 
for.
I'm not trying to say my script is any good. But I really like to see 
some more
control to generate test and running test as need ( rather then all or 
one ).
If that some level of control is existing, please let me know.
e.g.   KEY file include java.io.* should only test on java.io.* classes 
right?
( I never get it to run the way I wanted )

Thanks
Alex

Re: [Kissme-general] Re: Should I or not submit changes?

[Mark Wielaard]

Hi,

On Wed, 2002-07-17 at 02:47, Stephen Crawley wrote:
> However ... I've looked at Alex's scripts, and concluded that it would be
> better to use the 'mauve-kissme' key file that Mark Wielaard emailed last
> week, especially if we can put that file into CVS.  

We should probably look at the choose script and Alex his scripts and
try to get the best of both worlds.

I think the main problem of Mauve is that the documentation is a bit out
of date and that there are no clear instructions on how to add new
keys/tags for test. There should also be a simple template/example file
that people can use when adding new tests. Some of the tests are really
a bit ugly and don't show the best possible way to use the framework.

> The problem is that if you run Mauve tests the way that the Mauve README
> file says you should, the key file has to live in the Mauve directory.
> This is real nuisance.  [I can arrange that the Kissme make rules for
> running the Mauve tests copy the key file to the place where Mauve 
> expects it to be ... but that's a HACK.]

Such a hack is also used in the libgcj testsuite.

> Here are some other beefs that I have with Mauve:

Thanks for the list. If nobody else does it I will try to work on these
items.

>   *  Mauve's "make clean" doesn't delete the .class files.
>
>   *  Someone has checked some files generated by autoconf, etcetera
>      (configure and Makefile IIRC).   These contain dependencies on
>      the version of autoconf, etcetera used on said person's machine
>      and break on my machine.  These generated files need to be 
>      removed from CVS.

Yes, having autogenerated files in CVS annoys me greatly. I am in favor
of just removing the Makefile.in, aclocal.m4 and configure files from
CVS and adding instructions on how to create them yourself. If nobody
objects I will do that soon.


>   *  Mauve needs a global cvsignore file to ignore *.class files and
>      a .cvsignore file on the top directory to ignore the files created
>      by autoconf and configure.

That would be nice. I don't know if I can update the CVSROOT/cvsignore
file for mauve though.

>   *  The Mauve configure.ac metascript should have a --with option for
>      specifying a directory or search path for finding key files.  

--with-keys-dir? Would be nice if that was also used to get the
appropriate xfails file. libgcj has some extra support for xfails that
we might be able to import into Mauve proper (although I think it
depends on dejagnu).

>   *  The Mauve makefile should notice when I have edited the current
>      key file, and rebuild the class list file.
> 
>      At the moment, I think that the class list only gets rebuilt 
>      if you change the value of the KEYS environment variable.

I always do a rm -f .save-keys choices before typing make.
Something clearly wants to do some optimization of what should/shouldn't
be recompiled/scanned, but I don't know what the intention was of the
mechanism.

Cheers,

Mark

Re: [Kissme-general] Re: Should I or not submit changes?

[Mark Wielaard]

Hi,

On Wed, 2002-07-17 at 11:32, Dalibor Topic wrote:
> Porting tests over to Junit might not be too exciting,
> though. I believe that (if the mauve hackers decide to
> allow junit tests) it would be a better option to have
> both frameworks in parallel for a while. As junit
> relies on reflection, it wouldn't make much sense to
> run it on an implementation with broken reflection
> libraries. ;)

I agree. We should keep the testframework as simple as possible because
it tests the core library implementation. I would like to avoid
depending on reflection.

> The major point junit has for it, in my opinion, is
> the amount of documentation surrounding it: books,
> tutorials etc. It seems to be easy to find answers.
> That could lower the entry barrier for fresh mauve
> contributors.

It would probably not that much work to have a compatibility API in
Mauve for junit tests. It isn't that different from the Mauve framework.
http://junit.sourceforge.net/javadoc/junit/framework/package-summary.html

It is nice that JUnit allows you to create a TestSuite by reflection (it
picks all methods starting with testXXX()) so you don't have to call
them directly but I don't think supporting that in Mauve is very
important.

Having a simple/short but good documentation and template/example file
for Mauve will probably be more important for people wanting to
contribute then JUnit support though.

Cheers,

Mark

Re: [Kissme-general] Re: Should I or not submit changes?

[John Leuner]

> I couldn't find the mauve-kissme file ...
> I don't know how that key file look like, but I still have a hard time
> using a key file to run that against Mauve... that's why I try to upload 
> those file.
> Yes I know it is not an officeal way to run their test.... but it at 
> least able to
> generate report for anything sub directory on any java and javax classes.

I agree with Alex, I have found mauve much easier to run by hand
(bypassing the KEYS mechanism).
 
> >
> who is the gate keeper for the cvs...this shouldn't be that diffcult to 
> put in.

There are several people who have CVS access. You can also get access if
you like.
 
John Leuner

Re: [Kissme-general] Re: Should I or not submit changes?

[Mark Wielaard]

Hi,

On Wed, 2002-07-17 at 16:13, Andrew Haley wrote:
> If mauve wasn't such a pain to use and hack on I certainly would have
> contributed more to it.  Dejagnu, although horrible in its own special
> way, at least allows individual tests to be developed and used without
> needing the framework.

I was thinking of having very simple support for "output comparison"
tests. You write a bla/test.java class with a normal static main method
that just prints to System.out.println and there is a bla/test.out file
that contains the correct output of the test. The only drawback is that
when people explicitly use Testlet.check() it is much more clear what it
is they are precisely testing.

What things in DejaGNU do you like that should be supported in Mauve? 

Cheers,

Mark

Re: [Kissme-general] Re: Should I or not submit changes?

[Brian Jones]

Dalibor Topic <robilad@yahoo.com> writes:

> Porting tests over to Junit might not be too exciting,
> though. I believe that (if the mauve hackers decide to
> allow junit tests) it would be a better option to have
> both frameworks in parallel for a while. As junit
> relies on reflection, it wouldn't make much sense to
> run it on an implementation with broken reflection
> libraries. ;)

Yes, it would be easier if for a time it were possible to run against
both the old tests and the new tests.  When I looked at junit last it
was to see what the base requirement for the JVM would be and I
thought reflection would be required but after reading the docs I had
the impression you could potentially get around not having reflection
by specifying the necessary information in a file instead.  No time to
dig this up right now though.

> The major point junit has for it, in my opinion, is
> the amount of documentation surrounding it: books,
> tutorials etc. It seems to be easy to find answers.
> That could lower the entry barrier for fresh mauve
> contributors.

Yes, I think that would be the win-win scenario.

Brian
-- 
Brian Jones <cbj@gnu.org>

Re: [Kissme-general] Re: Should I or not submit changes?

[Andrew Haley]

Mark Wielaard writes:
 > Hi,
 > 
 > On Wed, 2002-07-17 at 16:13, Andrew Haley wrote:
 > > If mauve wasn't such a pain to use and hack on I certainly would have
 > > contributed more to it.  Dejagnu, although horrible in its own special
 > > way, at least allows individual tests to be developed and used without
 > > needing the framework.
 > 
 > I was thinking of having very simple support for "output comparison"
 > tests. You write a bla/test.java class with a normal static main method
 > that just prints to System.out.println and there is a bla/test.out file
 > that contains the correct output of the test. The only drawback is that
 > when people explicitly use Testlet.check() it is much more clear what it
 > is they are precisely testing.
 > 
 > What things in DejaGNU do you like that should be supported in Mauve? 

Tests are stand-alone -- it must be possible to put an entire testcase
into an email to be sent to someone to fix a bug.  This is IMO more
important than any other feature.

DejaGNU allows tests to be written independently of the any test
framework to be included almost without any change.  The test usually
must print something, and the something gets recorded.  This is a good
design.

DejaGNU also has very versatile facilities for remote testing, where
the framework runs on one machine and the tests on another.  This is
good, but difficult to use.

Andrew.

Re: [Kissme-general] Re: Should I or not submit changes?

[Dalibor Topic]

--- Mark Wielaard <mark@klomp.org> wrote:
> Hi,
> 
> On Wed, 2002-07-17 at 16:13, Andrew Haley wrote:
> > If mauve wasn't such a pain to use and hack on I
> certainly would have
> > contributed more to it.  Dejagnu, although
> horrible in its own special
> > way, at least allows individual tests to be
> developed and used without
> > needing the framework.
> 
> I was thinking of having very simple support for
> "output comparison"
> tests. You write a bla/test.java class with a normal
> static main method
> that just prints to System.out.println and there is
> a bla/test.out file
> that contains the correct output of the test. The
> only drawback is that
> when people explicitly use Testlet.check() it is
> much more clear what it
> is they are precisely testing.

Kaffe's regression testing suite relies mostly on
output comparison. While it's a great way to create
tests in a minute, it's not that great in the end.

Sometimes the output depends on the locale you are in,
so people get erraneous failures. Sometimes people
will include toString() output in the expected
results, which is quite painful when you test for
exceptions. Unless you want to match Sun's messages
literally, of course.

In my opinion, explicit check methods that verify
single properties are more useful, as it is easier to
extract context information about failures, most
importantly which test failed. That is really hard if
something messes up your tests output beyound the
region matching capabilities of diff.

best regards,

dalibor topic

__________________________________________________
Do You Yahoo!?
Yahoo! Autos - Get free new car price quotes
http://autos.yahoo.com

Re: [Kissme-general] Re: Should I or not submit changes?

[Dalibor Topic]

Hi Mark,

--- Mark Wielaard <mark@klomp.org> wrote:
> On Wed, 2002-07-17 at 11:32, Dalibor Topic wrote:
> > The major point junit has for it, in my opinion,
> is
> > the amount of documentation surrounding it: books,
> > tutorials etc. It seems to be easy to find
> answers.
> > That could lower the entry barrier for fresh mauve
> > contributors.
> 
> Having a simple/short but good documentation and
> template/example file
> for Mauve will probably be more important for people
> wanting to
> contribute then JUnit support though.

I fully agree. The lack of a simple, step-by-step
guide to adding new tests to mauve, is the major
stumbling block for beginners. It seems to pop up
about once every three months, and I've even asked for
it myself ;) Documenting good testing practices should
also help quite a bit.

best regards,

dalibor topic

__________________________________________________
Do You Yahoo!?
Yahoo! Autos - Get free new car price quotes
http://autos.yahoo.com

Re: [Kissme-general] Re: Should I or not submit changes?

[Chris Gray]

On 18 Jul 2002, Mark Wielaard wrote:

> Hi,
> 
> On Wed, 2002-07-17 at 11:32, Dalibor Topic wrote:
> > Porting tests over to Junit might not be too exciting,
> > though. I believe that (if the mauve hackers decide to
> > allow junit tests) it would be a better option to have
> > both frameworks in parallel for a while. As junit
> > relies on reflection, it wouldn't make much sense to
> > run it on an implementation with broken reflection
> > libraries. ;)
> 
> I agree. We should keep the testframework as simple as possible because
> it tests the core library implementation. I would like to avoid
> depending on reflection.

That was indeed one of the reasons why we use the Mauve framework for 
testing Wonka, even the company had already Standardised On (tm) JUnit
before we began.

> > The major point junit has for it, in my opinion, is
> > the amount of documentation surrounding it: books,
> > tutorials etc. It seems to be easy to find answers.
> > That could lower the entry barrier for fresh mauve
> > contributors.
> 
> It would probably not that much work to have a compatibility API in
> Mauve for junit tests. It isn't that different from the Mauve framework.
> http://junit.sourceforge.net/javadoc/junit/framework/package-summary.html

Yep, we've ported JUnit tests to Mauve rather easily.

> It is nice that JUnit allows you to create a TestSuite by reflection (it
> picks all methods starting with testXXX()) so you don't have to call
> them directly but I don't think supporting that in Mauve is very
> important.
> 
> Having a simple/short but good documentation and template/example file
> for Mauve will probably be more important for people wanting to
> contribute then JUnit support though.
> 
> Cheers,
> 
> Mark
> 

Best wishes

-- 

Chris Gray

VM Architect, ACUNIA