Re: Updating Mauve tags

[Mark Wielaard <mark@klomp.org>]

[-- Attachment #1: Type: text/plain, Size: 3131 bytes --]

Hi Roman,

On Sat, 2006-01-14 at 22:33 +0100, Roman Kennke wrote:
> (BTW: This is a Mauve topic, I CC the classpath list because the
> mauve-discuss list seems so dead).

Once upon a time there were lots of standard class library projects
(kaffe, gcj, classpath) and we started cooperating by sharing a test
suite. When most of these libraries merged together the test suite
became a bit more the "gnu classpath" testsuite, but there are still
independent users out there (although most of them are indeed pretty
quiet - acunia used it for wonka, hp used it for chai, ibm used it for
j9 and of course aicas uses it for jamaica, there are probably some
others out there that never publicly announced their usage of the test
suite.).

> I see that we have a concept of tags in Mauve. That is a collection of
> keys at the top of each test class. This way we can filter the tests.
> ATM we have tags for the JDK versions like JDK1.4 JDK1.3 and so on and a
> couple of other tags. However, it seems that they are not maintained in
> a usable way, so most people simply include every tag that they can
> think of (that is what's done in batch_run for example) to run all
> tests.

Why do you feel they aren't maintained in a usable way?
A test should mention the minimum version that it should work against.
And can mention newer versions for which the tests isn't valid anymore
(although I don't know of many examples of that). The README explains as
follows:

        Tags must all appear on a single line beginning "// Tags: ".
        
        Many files test functionality that has existed since JDK1.0.
        The
        corresponding line in the source:
        
            // Tags: JDK1.0
        
        Here is how you would tag something that first appeared in
        JDK1.2:
        
            // Tags: JDK1.2
        
        Here is how you would tag something that was eliminated in
        JDK1.2:
        
            // Tags: JDK1.0 !JDK1.2
        
        The idea behind this scheme is that it is undesirable to update
        all
        the files whenever we add a tag.  So instead most tags are
        defined in terms of primitive tags, and then we note the
        exceptions.

The reason the batch_run script lists all the tags is simply because it
wants to run all the tests.

> I would like to fix the tagging of the tests with regard to the JDK
> versions. And since the current reference is JDK1.5, I would naturally
> start with this one. What I propose to do is run all the tests under
> JDK1.5 and set the JDK1.5 tag for all tests that pass there. For all
> tests that FAIL and have the JDK1.5 tag set, this tag would have to be
> removed. Later I would like to do the same for JDK1.4 and JDK1.3. (I
> have no JDK1.2 JDK1.1 or JDK1.0 available, otherwise I would probably do
> the same for these).

It might be interesting to know what tests fail against which version of
the reference implementation, but the reference implementation does have
bugs, some of which aren't present in other implementations.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]