Updating Mauve tags

Updating Mauve tags

[Roman Kennke]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

Hi there,

(BTW: This is a Mauve topic, I CC the classpath list because the
mauve-discuss list seems so dead).

I see that we have a concept of tags in Mauve. That is a collection of
keys at the top of each test class. This way we can filter the tests.
ATM we have tags for the JDK versions like JDK1.4 JDK1.3 and so on and a
couple of other tags. However, it seems that they are not maintained in
a usable way, so most people simply include every tag that they can
think of (that is what's done in batch_run for example) to run all
tests.

I would like to fix the tagging of the tests with regard to the JDK
versions. And since the current reference is JDK1.5, I would naturally
start with this one. What I propose to do is run all the tests under
JDK1.5 and set the JDK1.5 tag for all tests that pass there. For all
tests that FAIL and have the JDK1.5 tag set, this tag would have to be
removed. Later I would like to do the same for JDK1.4 and JDK1.3. (I
have no JDK1.2 JDK1.1 or JDK1.0 available, otherwise I would probably do
the same for these).

What do you think about this? Maybe I completely misunderstand the
concept and working of the tags?

Cheers,
/Roman

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Updating Mauve tags

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 3131 bytes --]

Hi Roman,

On Sat, 2006-01-14 at 22:33 +0100, Roman Kennke wrote:
> (BTW: This is a Mauve topic, I CC the classpath list because the
> mauve-discuss list seems so dead).

Once upon a time there were lots of standard class library projects
(kaffe, gcj, classpath) and we started cooperating by sharing a test
suite. When most of these libraries merged together the test suite
became a bit more the "gnu classpath" testsuite, but there are still
independent users out there (although most of them are indeed pretty
quiet - acunia used it for wonka, hp used it for chai, ibm used it for
j9 and of course aicas uses it for jamaica, there are probably some
others out there that never publicly announced their usage of the test
suite.).

> I see that we have a concept of tags in Mauve. That is a collection of
> keys at the top of each test class. This way we can filter the tests.
> ATM we have tags for the JDK versions like JDK1.4 JDK1.3 and so on and a
> couple of other tags. However, it seems that they are not maintained in
> a usable way, so most people simply include every tag that they can
> think of (that is what's done in batch_run for example) to run all
> tests.

Why do you feel they aren't maintained in a usable way?
A test should mention the minimum version that it should work against.
And can mention newer versions for which the tests isn't valid anymore
(although I don't know of many examples of that). The README explains as
follows:

        Tags must all appear on a single line beginning "// Tags: ".
        
        Many files test functionality that has existed since JDK1.0.
        The
        corresponding line in the source:
        
            // Tags: JDK1.0
        
        Here is how you would tag something that first appeared in
        JDK1.2:
        
            // Tags: JDK1.2
        
        Here is how you would tag something that was eliminated in
        JDK1.2:
        
            // Tags: JDK1.0 !JDK1.2
        
        The idea behind this scheme is that it is undesirable to update
        all
        the files whenever we add a tag.  So instead most tags are
        defined in terms of primitive tags, and then we note the
        exceptions.

The reason the batch_run script lists all the tags is simply because it
wants to run all the tests.

> I would like to fix the tagging of the tests with regard to the JDK
> versions. And since the current reference is JDK1.5, I would naturally
> start with this one. What I propose to do is run all the tests under
> JDK1.5 and set the JDK1.5 tag for all tests that pass there. For all
> tests that FAIL and have the JDK1.5 tag set, this tag would have to be
> removed. Later I would like to do the same for JDK1.4 and JDK1.3. (I
> have no JDK1.2 JDK1.1 or JDK1.0 available, otherwise I would probably do
> the same for these).

It might be interesting to know what tests fail against which version of
the reference implementation, but the reference implementation does have
bugs, some of which aren't present in other implementations.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Updating Mauve tags

[Roman Kennke]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

Hi Mark,

> > I see that we have a concept of tags in Mauve. That is a collection of
> > keys at the top of each test class. This way we can filter the tests.
> > ATM we have tags for the JDK versions like JDK1.4 JDK1.3 and so on and a
> > couple of other tags. However, it seems that they are not maintained in
> > a usable way, so most people simply include every tag that they can
> > think of (that is what's done in batch_run for example) to run all
> > tests.
> 
> Why do you feel they aren't maintained in a usable way?

This was caused by a misunderstanding of the usage/meaning of those
tags. I was thinking that when a test has the tag JDK1.x, that this test
is meant to PASS under a JDK1.x-ish JDK. As Michael and others have
pointed out on IRC this is not the case. If I want to test a JDK1.3-sh
(for example) environment I should include JDK1.0 JDK1.1 JDK1.2 and
JDK1.3 tags in my keys.

The problem that I am seeing is when a test that is written to PASS
under 1.4 fails under 1.5. There are lots of those tests in the
testsuite for the javax.swing package. So my plan would have been to tag
all tests that pass under JDK1.5 with the 1.5 tag and those that don't
only with JDK1.4 or whatever is ok. Since the tags are not meant to be
used that way, maybe we can do it different. Could we extend the
choose-classes script to detect !JDK1.x tags in the tag header of java
source files and don't include the test in a JDK1.x test run?

/Roman

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Updating Mauve tags

[Meskauskas Audrius]

The Sun's swing did have as many as 6514 bugs through the history, 
despite many of them are fixed
now. Together with the new features and improvements, each new major 
release inevitably brings
some regressions that are later fixed. J2SE 5.0 
<http://java.sun.com/j2se/1.5.0/download.jsp> already had six updates in 
the past. They do make some
mistakes, same as we do, same as any java development group inevitably 
does. This may be especially
true for tests that were succeeding in the past but fail with the newest 
version.

Because of that reason I would not suggest any automated removal or 
inactivation of the tests just
because they fail on Sun's or any other java implementation. If the 
implemented behavior clearly
mismatches the Sun's API specification, this is probably just a bug that 
is likely to be fixed - probably in the
next minor release.

Some tests can be invalid and can be inactivated or removed (probably 
better altered, making them valid).
However I think that this work (a difficult work) must be done manually. 
The typical reason can be if we find
the similar problem in the Sun's bug reports and the Sun states that 
this is not a bug or it will never be fixed,
or if we realize that the former author of the test clearly 
misinterprets the Sun's API standard.

If needed, we could probably simply have and use the list of tests that 
are known to pass with the final
releases of the JDK 1.2, 1.3 and 1.4 (for the 1.5, such list would need 
the regular updates).

Audrius.

Roman Kennke wrote:

>Hi Mark,
>
>  
>
>>>I see that we have a concept of tags in Mauve. That is a collection of
>>>keys at the top of each test class. This way we can filter the tests.
>>>ATM we have tags for the JDK versions like JDK1.4 JDK1.3 and so on and a
>>>couple of other tags. However, it seems that they are not maintained in
>>>a usable way, so most people simply include every tag that they can
>>>think of (that is what's done in batch_run for example) to run all
>>>tests.
>>>      
>>>
>>Why do you feel they aren't maintained in a usable way?
>>    
>>
>
>This was caused by a misunderstanding of the usage/meaning of those
>tags. I was thinking that when a test has the tag JDK1.x, that this test
>is meant to PASS under a JDK1.x-ish JDK. As Michael and others have
>pointed out on IRC this is not the case. If I want to test a JDK1.3-sh
>(for example) environment I should include JDK1.0 JDK1.1 JDK1.2 and
>JDK1.3 tags in my keys.
>
>The problem that I am seeing is when a test that is written to PASS
>under 1.4 fails under 1.5. There are lots of those tests in the
>testsuite for the javax.swing package. So my plan would have been to tag
>all tests that pass under JDK1.5 with the 1.5 tag and those that don't
>only with JDK1.4 or whatever is ok. Since the tags are not meant to be
>used that way, maybe we can do it different. Could we extend the
>choose-classes script to detect !JDK1.x tags in the tag header of java
>source files and don't include the test in a JDK1.x test run?
>
>/Roman
>  
>

Re: Updating Mauve tags

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

Hi Roman,

On Mon, 2006-01-16 at 00:59 +0100, Roman Kennke wrote:
> The problem that I am seeing is when a test that is written to PASS
> under 1.4 fails under 1.5. There are lots of those tests in the
> testsuite for the javax.swing package.

To be honest, I would just remove those tests and concentrate on the 1.5
ones.

> So my plan would have been to tag
> all tests that pass under JDK1.5 with the 1.5 tag and those that don't
> only with JDK1.4 or whatever is ok. Since the tags are not meant to be
> used that way, maybe we can do it different. Could we extend the
> choose-classes script to detect !JDK1.x tags in the tag header of java
> source files and don't include the test in a JDK1.x test run?

I always thought it already worked that way (see the README), so if it
doesn't and you can make it actually do that then that would be great.

Cheers,

Mark

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Updating Mauve tags

[David Gilbert]

Roman Kennke wrote:

>The problem that I am seeing is when a test that is written to PASS
>under 1.4 fails under 1.5. There are lots of those tests in the
>testsuite for the javax.swing package. 
>
Hi Roman,

Did I write those tests (or some of them)?   If so, send me the file 
names and I'll see if I can clean them up a bit.  When I wrote them, I 
checked the results only against Sun JDK 1.4.2 (*), but recently I have 
started to use JDK 1.5 more regularly so now I can work on bringing the 
tests up to date.

Regards,

Dave

(*)  One of the problems with Swing is that the API docs are very much 
underspecified (completely missing in many cases), so it becomes 
necessary to make educated guesses when writing the tests, then confirm 
these against a specific implementation (in my case Sun's 1.4.2).