request for comments (long)

request for comments (long)

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

hello there,

i'm trying to add Mauve regression tests to the GNU Crypto project.
currently regression tests are done with JUnit (framework and
implementation).  the plan is to port/convert those tests to Mauve.
this is an invitation for people who (a) have experienced the same
situation, and/or (b) are experienced users of Mauve, to help us with
this process.


looking at how Mauve works, it seems that the process (very roughly) is
as follows:

1. the tests are organised under a source tree rooted at gnu/testlet.
2. a 'choose' script uses a KEYS variable to ultimately work out, which
of the .java files, under java/ or javax/, are selected for processing;
in the process it creates a 'classes' file that is fed to
3. SimpleTestHarness, when invoked by the 'make check' command.
4. to tailor the Mauve environment to the desired java bytecode
compiler, bytecode interpreter or GCJ native code, the auto* machinery
must be used --i.e. aclocal, automake, autoconf, configure.

to add new Mauve tests, one would select either the java/ or javax/
hierarchy and place the .java files there as:

a. belonging to package gnu.testlet.java[x].foo; and
b. adding a line that MUST start with '// Tags: ' to encode the level
 of compliance of that code with some standard; eg. JDK, JLS, JDBC,
 etc... c. further refinement of what to [not] select can be done by
 adding special lines into a 'mauve-foo' spec file that would be passed
 as 'KEYS=foo' when 'make check'-ing.


I. GNU Crypto regression tests are all in packages with names that
 start with test.blah.  they mirror 1-for-1 gnu.crypto.blah packages.
 adopting a new name that starts with 'gnu.testlet.gnu.crypto' should
 not be a problem.

the 'choose' script has to be amended to look for .java files in a
gnu/ directory, in addition to the current java/ and javax/ ones.  i.e.

Index: choose
===================================================================
RCS file: /cvs/mauve/mauve/choose,v
retrieving revision 1.21
diff -u -r1.21 choose
- --- choose	12 Jul 2002 19:33:27 -0000	1.21
+++ choose	9 Dec 2002 17:57:12 -0000
@@ -61,7 +61,8 @@
        echo "Invalid tag: $i" 1>&2
        exit 1
        ;;
- -    !java.* | java.* | !javax.* | javax.*)
+#    !java.* | java.* | !javax.* | javax.*)
+    !java.* | java.* | !javax.* | javax.* | !gnu.* | gnu.*)
        file="`echo $i | tr . /`"
        file_specs="$file_specs $file"
        ;;
@@ -130,7 +131,7 @@
 usesfile=/tmp/uses-$$

 : > $usesfile

- -(cd gnu/testlet; find java javax -name '*.java' -print) |
+(cd gnu/testlet; find java javax gnu -name '*.java' -print) |
 while read file; do
    exact_match=no
    test -n "$verbose" && echo "Examining $file"


II. the 'choose' script seems to consider the value passed to KEYS;
 e.g. foo as a Tag to look for in the source files, _in addition_ to
 any other Tag that may be used in the '// Tags: ' line.  in other
 words, if in my 'mauve-gnu-crypto' file i have the following lines:

BAR
!gnu.crypto.cipher.BaseCipherTestCase

and in gnu/testlet/gnu/crypto/cipher i have a file, say TestOfSquare,
that has the following line:

// Tags: gnu-crypto

it will get chosen by 'choose'.

if this is a guarranteed feature, then does it make sense to use the
$(PACKAGE) --i.e. 'gnu-crypto'-- for these two tokens?


III. when i tried to use the exclusion mechanism --prefix the name of a
class with ! after chopping off the gnu.testlet prefix-- the class,
when it contains the '// Tags: xxx' line is always chosen.

the 'choose' script has to be amended (see above diff).


IV. the mechanics of configuring Mauve, and GNU Crypto, are
 complicated; the easy approach is to (a) add the gnu/crypto/ test
 classes under gnu/testlet, (b) have the user checkout both GNU Crypto
 and Mauve, (c) amend Makefile.am there as follows:

Index: Makefile.am
===================================================================
RCS file: /cvs/mauve/mauve/Makefile.am,v
retrieving revision 1.22
diff -u -r1.22 Makefile.am
- --- Makefile.am	13 Jun 2002 17:11:28 -0000	1.22
+++ Makefile.am	9 Dec 2002 18:04:00 -0000
@@ -66,10 +66,12 @@
 check_PROGRAMS = SimpleTestHarness

 AM_GCJFLAGS = -I. -I$(srcdir)
+package_so =

 SimpleTestHarness_SOURCES = gnu/testlet/config.java $(harness_files)
- -SimpleTestHarness_DEPENDENCIES = $(STAMP) $(javao_files)
- -SimpleTestHarness_LDADD = $(javao_files)
+SimpleTestHarness_DEPENDENCIES = $(STAMP) $(javao_files)
+#SimpleTestHarness_LDADD = $(javao_files)
+SimpleTestHarness_LDADD = $(javao_files) $(package_so)
 SimpleTestHarness_LDFLAGS = --main=gnu.testlet.SimpleTestHarness

 check-local: $(check_PROGRAMS) $(CHOICES)

and (d) in GNU Crypto Makefile.am add something like:

# Mauve targets
 ---------------------------------------------------------------

if FOUND_MAUVE_DIR

KEYS=$(PACKAGE)
TESTFLAGS="-verbose -debug"

configure-mauve:
	cd @MAUVE_DIR@ ; aclocal ; automake ; autoconf
	cd @MAUVE_DIR@ ; GCJFLAGS="$(GCJFLAGS) --classpath .:@SRCDIR@/source"
 ./configure --with-gcj

run-mauve:
	rm -f @MAUVE_DIR@/.save-keys choices
	LD_LIBRARY_PATH=$$LD_LIBRARY_PATH$(PATH_SEPARATOR)@SRCDIR@/source/.lib
s \ $(MAKE) -C @MAUVE_DIR@ check
 package_so="@SRCDIR@/source/.libs/lib-gnu-crypto.so" AM_GCJFLAGS="-I.
 -I@SRCDIR@/source" KEYS=$(KEYS) TESTFLAGS=$(TESTFLAGS)

endif # FOUND_MAUVE_DIR


the problem with this approach is that the test sources are in a
different tree than the project.

another approach is to import the Mauve base classes and scripts into
GNU Crypto, and handle the lot with the same toolchain.  the problem
with this approach is that every time an enhancement is made to Mauve,
the same has to be adopted for its twin in GNU Crypto.

is there a better way that we can use today?



TIA + cheers;
rsn

cc: GNU Crypto discussion list.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE99N/e+e1AKnsTRiERA5AfAJsFB62kLqzFr4o499RTKIrls/xx9gCfa1l8
AiEO41auVCKBFLpHZXiv//s=
=h/SA
-----END PGP SIGNATURE-----

Re: request for comments (long)

[Tom Tromey]

>>>>> "Raif" == Raif S Naffah <<raif@fl.net.au>(by way of Raif S. Naffah <raif@fl.net.au>)> writes:

Raif> i'm trying to add Mauve regression tests to the GNU Crypto project.
Raif> currently regression tests are done with JUnit (framework and
Raif> implementation).

Don't let me discourage you, but in the past there's been a fair
amount of discussion on this list along the lines of "let's convert
Mauve to use JUnit" and "Mauve's test framework is a pain".

Given that, I'm curious to know why you'd want to convert...

Raif> looking at how Mauve works, it seems that the process (very
Raif> roughly) is as follows:
Raif> [ ...]

Your understanding seems correct.

Raif> II. the 'choose' script seems to consider the value passed to KEYS;
Raif>  e.g. foo as a Tag to look for in the source files, _in addition_ to
Raif>  any other Tag that may be used in the '// Tags: ' line.  in other
Raif>  words, if in my 'mauve-gnu-crypto' file i have the following lines:
Raif> BAR
Raif> !gnu.crypto.cipher.BaseCipherTestCase
Raif> and in gnu/testlet/gnu/crypto/cipher i have a file, say TestOfSquare,
Raif> that has the following line:
Raif> // Tags: gnu-crypto
Raif> it will get chosen by 'choose'.

In this case you're running with `KEYS=gnu-crypto'?  That behavior
seems a bit unintentional to me.

Raif> III. when i tried to use the exclusion mechanism --prefix the name of a
Raif> class with ! after chopping off the gnu.testlet prefix-- the class,
Raif> when it contains the '// Tags: xxx' line is always chosen.

I don't follow :-(.

Raif> IV. the mechanics of configuring Mauve, and GNU Crypto, are
Raif>  complicated

From time to time I've considered simply remove the auto* machinery
and going with something simpler.  In this environment it would
probably be appropriate.  However, as things have always worked
adequately for my needs, I've never had a strong motivation to do
anything about it.

Raif> another approach is to import the Mauve base classes and scripts into
Raif> GNU Crypto, and handle the lot with the same toolchain.  the problem
Raif> with this approach is that every time an enhancement is made to Mauve,
Raif> the same has to be adopted for its twin in GNU Crypto.

Raif> is there a better way that we can use today?

Maybe not :-(.  We're definitely open to suggestions and patches
improving the situation.

For libgcj we just have our test harness run Mauve's own configure and
build.  However, as you point out this is a pain since the test code
is a different project which must somehow be pointed to.

Tom

Re: request for comments (long)

[Dalibor Topic]

--- Tom Tromey <tromey@redhat.com> wrote:
> >>>>> "Raif" == Raif S Naffah <<raif@fl.net.au>(by
> way of Raif S. Naffah <raif@fl.net.au>)> writes:
> 
> Raif> i'm trying to add Mauve regression tests to
> the GNU Crypto project.
> Raif> currently regression tests are done with JUnit
> (framework and
> Raif> implementation).
> 

I hope the regression tests, as far as they are
useable as API tests, could be contributed to mauve,
if the GNU Crypto developers are going to convert them
to use mauve's test framework.

best regards,

dalibor topic

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

Re: [GNU Crypto] Re: request for comments (long)

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

On Wednesday 11 December 2002 10:09, Dalibor Topic wrote:
> --- Tom Tromey <tromey@redhat.com> wrote:
> > >>>>> "Raif" == Raif S Naffah <<raif@fl.net.au>(by
> > way of Raif S. Naffah <raif@fl.net.au>)> writes:
> > Raif> i'm trying to add Mauve regression tests to
> > the GNU Crypto project...
>
> I hope the regression tests, as far as they are
> useable as API tests, could be contributed to mauve,
> if the GNU Crypto developers are going to convert them
> to use mauve's test framework.

GNU Crypto offers its own API in addition to the JCE API.  the 
regression tests cover both.

i see no problem in contributing the tests to mauve in principle.  in 
practice, where would these tests live in Mauve source tree?  (back to 
point IV of my post ;-)


cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE99vLG+e1AKnsTRiERA7vcAJ9ijiOxT72FdLfCZyrbgvO+oTY1lQCfefIS
wr8Xhb88huRPSmD2r2xCxeg=
=N06B
-----END PGP SIGNATURE-----

Re: request for comments (long)

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

hello Tom,

On Wednesday 11 December 2002 09:30, Tom Tromey wrote:
> Raif S. Naffah <raif@fl.net.au>)> writes:
>>
>> i'm trying to add Mauve regression tests to the GNU Crypto
>> project.  currently regression tests are done with JUnit
>> (framework and implementation).
>
> Don't let me discourage you, but in the past there's been a fair
> amount of discussion on this list along the lines of "let's convert
> Mauve to use JUnit" and "Mauve's test framework is a pain".
>
> Given that, I'm curious to know why you'd want to convert...

we already have worked out --thanks to Olivier-- how to build, and use, 
an .so from JUnit 3.7 and JUnit 3.8.1  which we currently use.  the 
problems with this are:

* building a JUnit .so is not straightforward --because of the lack of 
support for awt and swing in gcj.
* because of the state of GCJ, we end up debugging the compiler rather 
than testing our implementations.

another reason is, which would be moot, if we go with separate trees, is 
that to run the tests, one has to go and download/install YAP (yet 
another package).

also, because some developers asked for it; and it seems to be _the_ 
choice par excellence for testing GCJ-related work --e.g. GNU 
Classpath, Kissme.

finally because it's simpler to learn and use, considering how little 
(feature-wise) we need/use from JUnit. Mauve, in constrast, is few 
classes, 1 README and off you go.  i believe there's a place for such a 
light-weight testing framework.


>> II. the 'choose' script seems to consider the value passed to
>> KEYS; e.g. foo as a Tag to look for in the source files, _in
>> addition_ to any other Tag that may be used in the '// Tags: '
>> line.  in other Raif>  words, if in my 'mauve-gnu-crypto' file i have
>> the following lines:
>>
>>    BAR
>>    !gnu.crypto.cipher.BaseCipherTestCase
>>
>> and in gnu/testlet/gnu/crypto/cipher i have a file, say
>> TestOfSquare, that has the following line:
>>
>>    // Tags: gnu-crypto
>>
>> it will get chosen by 'choose'.
>
> In this case you're running with `KEYS=gnu-crypto'?

correct.

>...  That behavior seems a bit unintentional to me.

noted.


>> III. when i tried to use the exclusion mechanism --prefix the
>> name of a class with ! after chopping off the gnu.testlet
>> prefix-- the class, when it contains the '// Tags: xxx' line is
>> always chosen.
>
> I don't follow :-(.

if line (was) #64 in the 'choose' script:

    !java.* | java.* | !javax.* | javax.*)

is not amended to include the root directory to add to the exiting 
'java' and 'javax' ones (see the diff in previous mail), then any 
exclusion directive in the tags file --e.g.

   !gnu.blah

has no effect; and the gnu.blah class/file is included anyway.  
sometimes you dont want to include files that are for example abstract 
or base classes, and do not have a test() method, but live in the same 
folder hierarchy as the test classes.


>> IV. the mechanics of configuring Mauve, and GNU Crypto, are
>> complicated...
>> another approach is to import the Mauve base classes and
>> scripts into GNU Crypto, and handle the lot with the same
>> toolchain.  the problem with this approach is that every time
>> an enhancement is made to Mauve, the same has to be adopted for
>> its twin in GNU Crypto...
>> is there a better way that we can use today?
>
> Maybe not :-(.  We're definitely open to suggestions and patches
> improving the situation.
>
> For libgcj we just have our test harness run Mauve's own configure
> and build.  However, as you point out this is a pain since the test
> code is a different project which must somehow be pointed to.

understand.  i'm after something as simple as that too :-)  putting the 
gnu.crypto.xxx test classes under the gnu.testlet folder, amending the 
'choose' script, and Makefile.am to use a $(project_so) seem to do the 
job.  i'll wait and see if others suggest different alternatives/ideas 
before deciding which way to jump.

thanks + cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE99wT/+e1AKnsTRiERA0egAKCXypgpgpjGHW0F6owwjfbGacCzvQCghiOS
ZkdbDkjctR+Yl90RM9AD9pk=
=S6CK
-----END PGP SIGNATURE-----

Re: request for comments (long)

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

hello again,

On Wednesday 11 December 2002 20:27, i wrote:
> ...putting
> the gnu.crypto.xxx test classes under the gnu.testlet folder,
> amending the 'choose' script, and Makefile.am to use a $(project_so)
> seem to do the job.  i'll wait and see if others suggest different
> alternatives/ideas before deciding which way to jump.

if we decide to go this way, is it ok with the Mauve project to host the 
classes in the CVS tree, and amend the above mentioned files as 
suggested?


cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE99yWt+e1AKnsTRiERA+HtAJ9LpZkkdbA3oBgntcpAYk7+t69RLACfSFbd
m8vVsse8UK6tLeoPrpqvGX0=
=H4z6
-----END PGP SIGNATURE-----

Re: request for comments (long)

[Mark Wielaard]

Hi,

On Wed, 2002-12-11 at 10:27, Raif S. Naffah wrote:
> [...]
> finally because it's simpler to learn and use, considering how little 
> (feature-wise) we need/use from JUnit. Mauve, in constrast, is few 
> classes, 1 README and off you go.  i believe there's a place for such a 
> light-weight testing framework.

This is why I like the Mauve framework. It is small and simple and JUnit
sometimes uses trickery (reflection) to do its work.

> >> IV. the mechanics of configuring Mauve, and GNU Crypto, are
> >> complicated...
> >> another approach is to import the Mauve base classes and
> >> scripts into GNU Crypto, and handle the lot with the same
> >> toolchain.  the problem with this approach is that every time
> >> an enhancement is made to Mauve, the same has to be adopted for
> >> its twin in GNU Crypto...
> >> is there a better way that we can use today?
> >
> > Maybe not :-(.  We're definitely open to suggestions and patches
> > improving the situation.
> >
> > For libgcj we just have our test harness run Mauve's own configure
> > and build.  However, as you point out this is a pain since the test
> > code is a different project which must somehow be pointed to.
> 
> understand.  i'm after something as simple as that too :-)  putting the 
> gnu.crypto.xxx test classes under the gnu.testlet folder, amending the 
> 'choose' script, and Makefile.am to use a $(project_so) seem to do the 
> job.  i'll wait and see if others suggest different alternatives/ideas 
> before deciding which way to jump.

But I don't like Mauves configury/setup magic. Note that I once made
some scripts that make using Mauve simpler (when you have a GNU system
with bash).
See: http://sources.redhat.com/ml/mauve-discuss/2002-q3/msg00019.html

If these seem useful to you then I can import them into Mauve. You can
then just use Mauve as upstream provider of your testframework. I
wouldn't worry to much about having two copies of those files in
different CVS trees. Both Mauve and GNU Crypto see active development
but both groups of developers know each other, or are sometimes even the
same people :) And the basic framework doesn't change that much anyway.

It might also be time to make a real Mauve release. This can then be
used as a stable point to test core library implementations against
since now you have to remember which CVS checkout of Mauve you were
testing against last time.

Cheers,

Mark

Re: [GNU Crypto] Re: request for comments (long)

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

hello Mark,

On Saturday 14 December 2002 01:54, Mark Wielaard wrote:
> On Wed, 2002-12-11 at 10:27, Raif S. Naffah wrote:
> > ...putting
> > the gnu.crypto.xxx test classes under the gnu.testlet folder,
> > amending the 'choose' script, and Makefile.am to use a
> > $(project_so) seem to do the job.  i'll wait and see if others
> > suggest different alternatives/ideas before deciding which way to
> > jump.
>
> But I don't like Mauves configury/setup magic. Note that I once made
> some scripts that make using Mauve simpler (when you have a GNU
> system with bash).
> See: http://sources.redhat.com/ml/mauve-discuss/2002-q3/msg00019.html
>
> If these seem useful to you then I can import them into Mauve.

i havent looked very closely, but i like the way the process is broken 
into atomic steps.


>... You
> can then just use Mauve as upstream provider of your testframework. I
> wouldn't worry to much about having two copies of those files in
> different CVS trees. Both Mauve and GNU Crypto see active development
> but both groups of developers know each other, or are sometimes even
> the same people :) And the basic framework doesn't change that much
> anyway...

fine.  i'll add the Mauve classes and scripts to the GNU Crypto CVS and 
intergate locally.  make check in GNU Crypto should then run the mauve 
tests.


cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE9+lYJ+e1AKnsTRiERA1nEAJ9plVZ36BmkRbwP9o3qdtUxqpIDUQCdFS6q
oggM+t48IqT2K3JGqCdkCGw=
=kvn9
-----END PGP SIGNATURE-----

Mauve in GNU Crypto. was: [GNU Crypto] Re: request for comments (long)

[Raif S. Naffah]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

hello Mark,

On Saturday 14 December 2002 08:49, i wrote:
> On Saturday 14 December 2002 01:54, Mark Wielaard wrote:
> > ...
> >... You
> > can then just use Mauve as upstream provider of your testframework.
> > I wouldn't worry to much about having two copies of those files in
> > different CVS trees...
>
> fine.  i'll add the Mauve classes and scripts to the GNU Crypto CVS
> and intergate locally.  make check in GNU Crypto should then run the
> mauve tests.

this now is done.  i used the choose-classes script from 
<http://sources.redhat.com/ml/mauve-discuss/2002-q3/msg00019.html> 
(with similar modifications to those i did for the 'choose' script), 
the main gnu.testlet classes from Mauve, and the two m4 macros.  the 
rest was easy :-)

not all the JUnit tests have been ported, but i already can see the 
java.math.BigInteger.euclidInv bug there as well.  i'll try to write a 
specific mauve test for it unless somebody else beats me to it.


cheers;
rsn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Que du magnifique

iD8DBQE9+zS0+e1AKnsTRiERA/IYAJ9VgtlTdv+UhIYyGxgYO/hOBFRp2gCeLYSh
ChyEZp17PuPmUlrGFICLKNo=
=1o+d
-----END PGP SIGNATURE-----

Re: Mauve in GNU Crypto. was: [GNU Crypto] Re: request for comments (long)

[Mark Wielaard]

Hi,

On Sat, 2002-12-14 at 14:40, Raif S. Naffah wrote:
> this now is done.  i used the choose-classes script from 
> <http://sources.redhat.com/ml/mauve-discuss/2002-q3/msg00019.html> 
> (with similar modifications to those i did for the 'choose' script), 
> the main gnu.testlet classes from Mauve, and the two m4 macros.  the 
> rest was easy :-)

Good to hear that they were usefull. I tried out the GNU Crypto build
and make check works out of the box which is nice.

I will see if I can come up with a little documentation for the script
and that also make it part of the main Mauve tree.

Cheers,

Mark