Analysis of Mauve failures - Part 1

Analysis of Mauve failures - Part 1

[Mark Wielaard]

Hi,

Here are some notes on the Mauve failures I am currently seeing. I don't
have results for 3.0, so I don't know if we are doing better or worse.
But we are able to run a lot more tests now then with 3.0.

> 		=== libjava Summary ===
> 
> # of expected passes		12252
> # of unexpected failures	208
> # of unexpected successes	3
> # of expected failures	32
> # of untested testcases	14

The following are just notes to myself so I can either fix the issues or
make real bug reports later. But if you have any insights please let me
know.

> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 1)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 2)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 4)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 9)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 10)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 12)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 13)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 14)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 15)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 16)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 17)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 19)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 21)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 22)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 24)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 25)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 27)
> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 28)
All these failures seem to be because the test tries to initialize a
java.awt class which fails since java.awt.Toolkit cannot be initialized.
Some tests succeed for Object, BitSet and Applet. So it seems we must
implement more of java.awt first before seeing progress on this test.

> FAIL: gnu.testlet.java.io.BufferedByteOutputStream.interrupt: single-byte writes (number 3)
> FAIL: gnu.testlet.java.io.BufferedByteOutputStream.interrupt: single-byte writes (number 4)
Interaction between a BufferedByteOutputStream wrapped around an
OutputStream that throws an InterruptedIOException after a couple of
bytes. Would be nice to get right but using an OutputStream that is
interrupted seems dangerous anyway.

> FAIL: gnu.testlet.java.io.File.jdk11: getCanonicalPath () java.io.IOException: No such file or directory (number 1)
Don't know if this is actually wrong. getCanonicalPath() may throw an
IOException if the File does not exist since it might need help from the
file system to construct the path (which natFilePosix does since it uses
realpath() if it exists).

> FAIL: gnu.testlet.java.io.PipedStream.close (number 1)
Don't know why this one fails. The PipedOutputStream seems to notify the
PipedInputStream but somehow it keeps waiting till it gets the
InterruptedException.

Skipping analysis of Character failures (seem to be about 80) since Eric
has recently rewritten Character support for Unicode 3.0 and the tests
seem to be written for an older Unicode version. Probably needs a
rewrite of the Mauve tests.

> FAIL: gnu.testlet.java.lang.Class.ClassTest: Error: test_getClassLoader failed - 1 (number 1)
It seems to expect a core class like java.lang.String to have a 'null'
ClassLoader which seems reasonable.

> FAIL: gnu.testlet.java.lang.Class.ClassTest: Error: test_getClassLoader failed - 2 (number 1)
It expects the Class gnu.testlet.java.lang.Class.ClassTest to have a
'null' classloader probably because it is on the classpath. But this is
not true anymore for Java2 based systems. So the tests is now wrong.

> FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_toString failed - 8 (number 1)
> FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_equals failed CYGNUS: Double.equals - 8 (number 1)
> FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_intValue returned wrong results CYGNUS: Float to int conversions - 1 (number 1)
> FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_longValue returned wrong results CYGNUS: Float to int conversions - 1 (number 1)
Don't know what is going on here. If I run these by hand they succeed!?!

> FAIL: gnu.testlet.java.lang.Float.FloatTest: Error: test_Basics failed - 3 (number 1)
> FAIL: gnu.testlet.java.lang.Float.FloatTest: Error: test_Basics failed - 4 (number 1)
NumberFormatException thrown when constructing new Float("234.34f") and
new Float("1.4e-45f"). The f is legal here. Have to check Classpath
version to see if it fixes this bug.

> FAIL: gnu.testlet.java.lang.Float.FloatTest: Error: test_toString failed - 8 (number 1)
> FAIL: gnu.testlet.java.lang.Float.FloatTest: Error: test_equals failed - 8 (number 1)
> FAIL: gnu.testlet.java.lang.Float.FloatTest: Error: test_intValue returned wrong results - 1 (number 1)
Another Heisenbug. Running by hand makes them succeed!?!

> FAIL: gnu.testlet.java.lang.Float.FloatTest: test_valueOf 3.4e+32f (number 1)
> FAIL: gnu.testlet.java.lang.Float.FloatTest: test_valueOf " -23.45f    " (number 1)
Another case of 'f' at the end is legal, but now in Float.valueOf().
Check Classpath implementation.

> FAIL: gnu.testlet.java.lang.Integer.getInteger (number 12)
Fixed by recent patch on branch. Another fix is in Classpath which
should go to the trunk.

> FAIL: gnu.testlet.java.lang.Integer.getInteger (number 17)
Integer.getInteger(null) throws NullPointerException. Believed fixed in
Classpath.

> FAIL: gnu.testlet.java.lang.Integer.new_Integer: decode (number 6)
> FAIL: gnu.testlet.java.lang.Integer.new_Integer: uncaught exception at "decode" number 11: java.lang.NumberFormatException
> FAIL: gnu.testlet.java.lang.Long.LongTest: getLong (number 1)
> FAIL: gnu.testlet.java.lang.Long.LongTest: getLong (number 2)
> FAIL: gnu.testlet.java.lang.Long.LongTest: getLong (number 3)
> FAIL: gnu.testlet.java.lang.Long.LongTest: getLong (number 4)
> FAIL: gnu.testlet.java.lang.Long.LongTest: getLong (number 5)
> FAIL: gnu.testlet.java.lang.Long.LongTest: getLong (number 6)
> FAIL: gnu.testlet.java.lang.Long.getLong (number 1)
> FAIL: gnu.testlet.java.lang.Long.getLong (number 2)
> FAIL: gnu.testlet.java.lang.Long.getLong (number 4)
> FAIL: gnu.testlet.java.lang.Long.getLong (number 6)
> FAIL: gnu.testlet.java.lang.Long.getLong (number 7)
> FAIL: gnu.testlet.java.lang.Long.getLong (number 8)
All fixed by recent patches on branch. Believed to be also fixed in
Classpath. Wrapper classes should be merged for 3.2.

> FAIL: gnu.testlet.java.lang.Math.MathTest: Error : test_atan2 failed - 5 (number 1)
> FAIL: gnu.testlet.java.lang.Math.max (number 2)
> FAIL: gnu.testlet.java.lang.Math.max (number 18)
Heisenbugs!

> FAIL: Compile gnu/testlet/java/lang/Math/min.o for gnu.testlet.java.lang.Math.min
Strange. Seems to compile fine by hand. But then it gives:

FAIL: gnu.testlet.java.lang.Math.min: Float NaNs (number 1)
got 0.0 but expected NaN
FAIL: gnu.testlet.java.lang.Math.min: Float NaNs (number 2)
got 0.0 but expected NaN
2 of 41 tests failed

So these two tests fail
  harness.check (Float.toString (Math.min (2.0f, Float.NaN)), "NaN");
  harness.check (Math.min (2.0f, Float.NaN), Float.NaN);
But these two succeed???
  harness.check (Float.toString (Math.min (Float.NaN, 2.0f)), "NaN");
  harness.check (Math.min (Float.NaN, 2.0f), Float.NaN);
Strangeness since when making a test program it seems to always do the
right thing...

> FAIL: gnu.testlet.java.lang.String.CASE_INSENSITIVE_ORDER: unicode mappings (number 1)
> FAIL: gnu.testlet.java.lang.String.CASE_INSENSITIVE_ORDER: unicode mappings (number 2)
The source says: "These two tests are unspecified - see Sun bug 4425387"

> FAIL: gnu.testlet.java.lang.StringBuffer.plus (number 1)
Seems to be a compiler error. method_returning_null() + "" should result
in the String "null" not null.

> FAIL: gnu.testlet.java.lang.ref.PhantomReference.phantom: unreachable (number 1)
The test says:
// In this test we make some assumptions about how the GC operates
// that are probably not quite sound.  In particular we assume
// System.gc() will collect everything.
Lets assume that is the case for now.

> FAIL: gnu.testlet.java.lang.reflect.Method.invoke: takes_int (number 7)
Test expects a NullPointerException not an IllegalArgumentException when
it supplies a args[0] = null. Quick reading of API makes me think that
the test is wrong.

> FAIL: gnu.testlet.java.lang.reflect.Modifier.toString12 (number 1)
> FAIL: gnu.testlet.java.lang.reflect.Modifier.toString12 (number 2)
Test expects Modifier.STRICT to be represented by the String "strict",
we return "strictfp" which seems correct to me.

OK. That is it for now. Going to get some sleep. Analysis of the other
~60 failures will come later.

Cheers,

Mark

P.S. My machine seems to be unable to reach gcc.gnu.org although I do
get some messages from the mailinglist I seem to be unable to send them
at the moment. We will see if this reaches the list.

Re: Analysis of Mauve failures - Part 1

[Eric Blake]

Some comments on the classes I have touched (mainly on the Classpath
side) in the past two months:

Mark Wielaard wrote:
> 
> Skipping analysis of Character failures (seem to be about 80) since Eric
> has recently rewritten Character support for Unicode 3.0 and the tests
> seem to be written for an older Unicode version. Probably needs a
> rewrite of the Mauve tests.

Yes, I think Mauve needs to be updated here.  Unfortunately, I have not
yet taken the time to look into doing that.

> > FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_toString failed - 8 (number 1)
> > FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_equals failed CYGNUS: Double.equals - 8 (number 1)
> > FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_intValue returned wrong results CYGNUS: Float to int conversions - 1 (number 1)
> > FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_longValue returned wrong results CYGNUS: Float to int conversions - 1 (number 1)
> Don't know what is going on here. If I run these by hand they succeed!?!

I haven't touched string conversion in Classpath, so I doubt a merge
would solve this. I do know that the string->float conversion in
Classpath is wrong, since it uses string->double->float which can have
off-by-ulp errors (I know this, because I had to fix this very sort of
bug for the jikes compiler).

> 
> > FAIL: gnu.testlet.java.lang.Integer.getInteger (number 12)
> Fixed by recent patch on branch. Another fix is in Classpath which
> should go to the trunk.
> 
> > FAIL: gnu.testlet.java.lang.Integer.getInteger (number 17)
> Integer.getInteger(null) throws NullPointerException. Believed fixed in
> Classpath.
...
> All fixed by recent patches on branch. Believed to be also fixed in
> Classpath. Wrapper classes should be merged for 3.2.

Yes, I believe these are fixed in classpath. I'm happy to help with the
3.2 merge, when the time is right.

> 
> > FAIL: gnu.testlet.java.lang.String.CASE_INSENSITIVE_ORDER: unicode mappings (number 1)
> > FAIL: gnu.testlet.java.lang.String.CASE_INSENSITIVE_ORDER: unicode mappings (number 2)
> The source says: "These two tests are unspecified - see Sun bug 4425387"

Well, since I wrote that comment, Sun went ahead and specified the
behavior of CASE_INSENSITIVE_ORDER. I ought to update that test to match
my proposed patch for String (the one that Tom has put on the back
burner for a while).

> 
> > FAIL: gnu.testlet.java.lang.reflect.Modifier.toString12 (number 1)
> > FAIL: gnu.testlet.java.lang.reflect.Modifier.toString12 (number 2)
> Test expects Modifier.STRICT to be represented by the String "strict",
> we return "strictfp" which seems correct to me.

Yes, it looks like that test is wrong.

> 
> OK. That is it for now. Going to get some sleep. Analysis of the other
> ~60 failures will come later.

Thanks for looking into this!

> 
> Cheers,
> 
> Mark
> 
> P.S. My machine seems to be unable to reach gcc.gnu.org although I do
> get some messages from the mailinglist I seem to be unable to send them
> at the moment. We will see if this reaches the list.

-- 
This signature intentionally left boring.

Eric Blake             ebb9@email.byu.edu
  BYU student, free software programmer

Re: Analysis of Mauve failures - Part 1

[Tom Tromey]

>>>>> "Mark" == Mark Wielaard <mark@klomp.org> writes:

Mark> Here are some notes on the Mauve failures I am currently seeing.

Thanks a lot for doing this!  It is a big help for this release.

>> FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 28)

Mark> All these failures seem to be because the test tries to
Mark> initialize a java.awt class which fails since java.awt.Toolkit
Mark> cannot be initialized.  Some tests succeed for Object, BitSet
Mark> and Applet. So it seems we must implement more of java.awt first
Mark> before seeing progress on this test.

Meanwhile let's disable these tests in mauve-libgcj.  Feel free to
simply check in such a patch.

Mark> Interaction between a BufferedByteOutputStream wrapped around an
Mark> OutputStream that throws an InterruptedIOException after a
Mark> couple of bytes. Would be nice to get right but using an
Mark> OutputStream that is interrupted seems dangerous anyway.

I think I wrote this test.  It is pretty optimistic.  I seem to recall
general agreement that InterruptedIOException couldn't really be
relied on, since even Sun didn't fully implement it on all platforms.

Mark> Skipping analysis of Character failures (seem to be about 80)
Mark> since Eric has recently rewritten Character support for Unicode
Mark> 3.0 and the tests seem to be written for an older Unicode
Mark> version. Probably needs a rewrite of the Mauve tests.

The Character tests in Mauve have always been a bit flaky, too.  In
the earlier days, when I wrote this code, the specifications were not
as tight as they are now.  So these tests are a bit subjective.

>> FAIL: gnu.testlet.java.lang.Class.ClassTest: Error: test_getClassLoader failed - 1 (number 1)
Mark> It seems to expect a core class like java.lang.String to have a 'null'
Mark> ClassLoader which seems reasonable.

There's a few long threads on this subject.  Basically doing this is
hard, and arguably the current way is correct anyway.  (As I
understand it.)

>> FAIL: gnu.testlet.java.lang.Math.MathTest: Error : test_atan2 failed - 5 (number 1)
>> FAIL: gnu.testlet.java.lang.Math.max (number 2)
>> FAIL: gnu.testlet.java.lang.Math.max (number 18)
Mark> Heisenbugs!

These sure are curious.

Mark> FAIL: gnu.testlet.java.lang.Math.min: Float NaNs (number 1)
Mark> got 0.0 but expected NaN
Mark> FAIL: gnu.testlet.java.lang.Math.min: Float NaNs (number 2)
Mark> got 0.0 but expected NaN
Mark> 2 of 41 tests failed

Are you building with -O?  Math.min might be inlined; maybe the
inlining fails on your platform?  (What platform is it?)

>> FAIL: gnu.testlet.java.lang.StringBuffer.plus (number 1)
Mark> Seems to be a compiler error. method_returning_null() + ""
Mark> should result in the String "null" not null.

I think we have a PR for this already.  I seem to remember running
across this at some point.  I don't have the PR number handy though.

>> FAIL: gnu.testlet.java.lang.reflect.Method.invoke: takes_int (number 7)
Mark> Test expects a NullPointerException not an IllegalArgumentException when
Mark> it supplies a args[0] = null. Quick reading of API makes me think that
Mark> the test is wrong.

I agree.  In cases such as these running the test against the JDK can
also be helpful.

>> FAIL: gnu.testlet.java.lang.reflect.Modifier.toString12 (number 1)
>> FAIL: gnu.testlet.java.lang.reflect.Modifier.toString12 (number 2)
Mark> Test expects Modifier.STRICT to be represented by the String "strict",
Mark> we return "strictfp" which seems correct to me.

This is my fault :-(.  I made this same mistake in libgcj at one point.
The test is definitely in error.

Tom

Re: Analysis of Mauve failures - Part 1

[Tom Tromey]

>>>>> "Eric" == Eric Blake <ebb9@email.byu.edu> writes:

Eric> I haven't touched string conversion in Classpath, so I doubt a
Eric> merge would solve this. I do know that the string->float
Eric> conversion in Classpath is wrong, since it uses
Eric> string->double->float which can have off-by-ulp errors (I know
Eric> this, because I had to fix this very sort of bug for the jikes
Eric> compiler).

We use fdlibm to do string->double conversions.  I imagine that step
is fine (though of course only debugging will show the truth).  It is
possible that we introduce problems in the other steps, since gcc
doesn't fully respect Java semantics.  Failures like this, if that is
what they are, can just be ignored (disabled) for now; we can't fix
these problems in the 3.1 timeframe.

Tom

Re: Analysis of Mauve failures - Part 1

[Brian Jones]

Mark Wielaard <mark@klomp.org> writes:

> > FAIL: gnu.testlet.java.beans.Introspector.jdk11: getBeanInfo (number 1)
> > ...
> All these failures seem to be because the test tries to initialize a
> java.awt class which fails since java.awt.Toolkit cannot be initialized.
> Some tests succeed for Object, BitSet and Applet. So it seems we must
> implement more of java.awt first before seeing progress on this test.

These can probably be rewritten to avoid java.awt a bit more.

> > FAIL: gnu.testlet.java.io.File.jdk11: getCanonicalPath () java.io.IOException: No such file or directory (number 1)
> Don't know if this is actually wrong. getCanonicalPath() may throw an
> IOException if the File does not exist since it might need help from the
> file system to construct the path (which natFilePosix does since it uses
> realpath() if it exists).

This one should be working, works under ORP+Classpath and JDK last I checked.

Brian
-- 
Brian Jones <cbj@gnu.org>

Re: Analysis of Mauve failures - Part 1

[Bryce McKinlay]

Brian Jones wrote:

>>>FAIL: gnu.testlet.java.io.File.jdk11: getCanonicalPath () java.io.IOException: No such file or directory (number 1)
>>>
>>Don't know if this is actually wrong. getCanonicalPath() may throw an
>>IOException if the File does not exist since it might need help from the
>>file system to construct the path (which natFilePosix does since it uses
>>realpath() if it exists).
>>
>
>This one should be working, works under ORP+Classpath and JDK last I checked.
>

java.io.File stuff isn't merged...

regards

Bryce.

Analysis of Mauve failures - Part 2

[Mark Wielaard]

Hi,

Thanks for all the feedback on Part 1.
Since it it so popular here is Part 2!
Note again that the analysis is not very deep,
it is mainly a tool for me to get a list of issues to look into.

> FAIL: gnu.testlet.java.net.DatagramSocket.DatagramSocketTest: Error : test_echoWithTimeout failed - 5 Exception occured while sending/receiving  (number 1)
We get a NullPointerException after the DatagramSocket has been closed
and receive() is called. Don't know why since impl is not null.

> FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: Error : test_Basics failed - 10  Should not throw UnknownHostException here  (number 1)
> FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: test_Basics failed - 11 - NullPointerException (number 1)
> FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: test_Basics failed - 12 - NullPointerException (number 1)
Tests for host hpjavaux.cup.hp.com which does not exist. When changed to
lookup savannah.gnu.org all tests succeed.

> FAIL: gnu.testlet.java.net.ServerSocket.ServerSocketTest: Error : test_params failed - 4toString did not return proper values  (number 1)
ServerSocket.toString() returns:
  ServerSocket [addr=0.0.0.0/0.0.0.0,port=0,localport=30000]
the test expects:
  ServerSocket[addr=all-zeros-broadcast/0.0.0.0,port=0,localport=30000]
Since the API does not specify any format I think the test is wrong.

> FAIL: gnu.testlet.java.net.Socket.SocketTest: Error : test_Basics failed - 7 exception should not have been thrown here (number 1)
> FAIL: gnu.testlet.java.net.Socket.SocketTest: Error : test_params failed - 10 exception was thrown (number 1)
> FAIL: gnu.testlet.java.net.Socket.SocketTest: Error : test_BasicServer failed - 7 len returned is not correct.  1 (number 1)
> WARNING: program timed out.
This test does not run at all when run by hand. Have to figure out how
the setup is different. But since www.cygnus.com != 205.180.83.41
anymore some failures are expected.

> FAIL: gnu.testlet.java.net.URL.URLTest: openConnection (number 1)
> WARNING: program timed out.
Seems to test against
http://sourceware.cygnus.com:80/mauve/testarea/index.html
Which is now out of date since the failures read:
got 'Apache/1.3.23 (Unix)' but expected 'Apache/1.3.9 (Unix)'
got 'text/html; charset=iso-8859-1' but expected 'text/html'
got '<!DOCT' but expected '<HTML>'

> FAIL: Link for gnu.testlet.java.text.AttributedCharacterIterator.implement
Works fine when done by hand. Probably some setup lameness.

> FAIL: gnu.testlet.java.text.AttributedString.Test: Attribute key count (number 1)
Checks that getAllAttributeKeys() returns 1 when it has just added 2
Attributes. Is this test correct?

> FAIL: gnu.testlet.java.text.DateFormatSymbols.Test: invalid locale (number 1)
Our ResourceBundle.getBundle() returns something for Locale("yi", "yi").
Is that actually wrong?

> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: CDT (number 1)
> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: EDT (number 1)
> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: EST (number 1)
> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: PDT (number 1)
got 'Fri, 18 May 2001 12:18:06 CDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
got 'Fri, 18 May 2001 13:18:06 EDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
got 'Fri, 18 May 2001 13:18:06 EDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
got 'Fri, 18 May 2001 10:18:06 PDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
Are we missing a Calendar.setTimeZone() somewere?

> FAIL: Link for gnu.testlet.java.util.AbstractCollection.AcuniaAbstractCollectionTest
> FAIL: Link for gnu.testlet.java.util.AbstractList.AcuniaAbstractListTest
> FAIL: Link for gnu.testlet.java.util.AbstractMap.AcuniaAbstractMapTest
> FAIL: Link for gnu.testlet.java.util.AbstractMap.AcuniaExAbstractMap
> FAIL: Link for gnu.testlet.java.util.AbstractSequentialList.AcuniaAbstractSequentialListTest
> FAIL: Link for gnu.testlet.java.util.AbstractSet.AcuniaAbstractSetTest
The setup has problems with tests that use more than one class for the
Testlet. Will rewrite testlets to be self contained.

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw an IndexOutOfBoundsException -- 3 (number 1)
removeRange(3,2) called on an ArrayList of 14 elements.
Seems reasonable but spec does not explicitly specify this behaviour.

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: ArrayList must not be changed -- 3 (number 1)
Probably fallout from last failure.

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw an IndexOutOfBoundsException -- 5 (number 1)
> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: ArrayList must not be changed -- 5 (number 1)
Same as above removeRange(15,13).

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw a ConcurrentModificationException -- 7 (number 1)
set(int, Object) does not make the Iterator throw a
ConcurrentModificationException. Seems to not be required by the spec
though. Do people depend on this behaviour?

> FAIL: gnu.testlet.java.util.Arrays.sort (number 1)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 2)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 3)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 4)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 5)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 6)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 7)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 8)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 9)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 10)
Ugh. Our sort does not sort...
Some example failures:

- 79,84,94,42,84,75,11,52,84,18,69,76,14,69,32,28,49,87,98,39,.
+ 11,14,18,28,32,39,42,49,52,69,79,69,75,76,84,84,84,87,94,98

- 80,19,8,10,44,84,61,56,56,6,16,57,59,28,12,92,55,34,1,80,.
+ 1,6,8,10,12,16,19,28,34,44,55,56,56,57,59,61,80,80,92,84

- 68,82,69,40,68,44,11,89,87,92,62,71,52,86,51,23,92,42,13,65,.
+ 11,13,23,40,42,44,51,52,62,65,68,71,68,69,82,86,87,89,92,92

> FAIL: gnu.testlet.java.util.HashMap.AcuniaHashMapTest: checking value of loadFactor, got 1.5 (number 1)
Strange it creates a new HashMap(10,1.5f) and then expects to get a
loadFactor of 0.75f !?!

> FAIL: gnu.testlet.java.util.HashMap.AcuniaHashMapTest: it.hasNext should not throw ConcurrentModificationException (number 1)
Again a bogus test? It gets an iterator, removes an element and the
expects it.hasNext() not to throw and ConcurrentModificationException
!?!

> FAIL: gnu.testlet.java.util.HashMap.AcuniaHashMapTest: should throw an UnsupportedOperationException (number 1)
Seems fallout from the last bogus test.

> FAIL: gnu.testlet.java.util.Hashtable.AcuniaHashtableTest: should throw NullPointerException (number 1)
Our docs say: "This is the new API for the old contains(), except that
it is forgiving of null."
But the spec says "Note that this method is identical in functionality
to contains"

> FAIL: gnu.testlet.java.util.Hashtable.AcuniaHashtableTest: should throw NullPointerException -- 1 (number 1)
remove(null). Our docs say "Map.remove and Dictionary.remove disagree
whether null is a valid parameter; at the moment, this implementation
obeys Map.remove, and silently ignores null."

> FAIL: gnu.testlet.java.util.LinkedList.AcuniaLinkedListTest: should throw a ConcurrentModificationException -- 7 (number 1)
Same issue as with ArrayList.
set(int, Object) does not make the Iterator throw a
ConcurrentModificationException. Seems to not be required by the spec
though. Do people depend on this behaviour?

That was it for now.
Stay tuned for Part 3 in the never ending Mauve failures soap.

Cheers,

Mark

Re: Analysis of Mauve failures - Part 2

[Bryce McKinlay]

Mark Wielaard wrote:

>got 'Fri, 18 May 2001 10:18:06 PDT' but expected
>    'Fri, 18 May 2001 17:18:06 GMT'
>Are we missing a Calendar.setTimeZone() somewere?
>

TimeZone and Calendar stuff is known to be somewhat buggy in libgcj so 
it wouldn't surprise me if something really is wrong here.

>>FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw an IndexOutOfBoundsException -- 3 (number 1)
>>
>removeRange(3,2) called on an ArrayList of 14 elements.
>Seems reasonable but spec does not explicitly specify this behaviour.
>

Hehe, our javadoc says "You asked for it if you call this with invalid 
arguments", which isn't unreasonable given that its a protected method. 
But it doesn't seem expensive to change it to behave like the JDK 
implementation:

Index: ArrayList.java
===================================================================
RCS file: /cvs/gcc/gcc/libjava/java/util/ArrayList.java,v
retrieving revision 1.9
diff -u -r1.9 ArrayList.java
--- ArrayList.java      2002/01/22 22:40:38     1.9
+++ ArrayList.java      2002/04/04 01:37:55
@@ -444,12 +444,14 @@
    */
   protected void removeRange(int fromIndex, int toIndex)
   {
-    if (fromIndex != toIndex)
+    if (fromIndex <= toIndex)
       {
         modCount++;
         System.arraycopy(data, toIndex, data, fromIndex, size - toIndex);
         size -= toIndex - fromIndex;
       }
+    else
+      throw new ArrayIndexOutOfBoundsException();
   }
 
   /**

>>FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw a ConcurrentModificationException -- 7 (number 1)
>>
>set(int, Object) does not make the Iterator throw a
>ConcurrentModificationException. Seems to not be required by the spec
>though.
>

set() should not update modCount (and thus not cause iterators to throw 
ConcurrentModificationException), because it is not a structural change 
to the list. Only operations that could result in adding or removing 
entries should update modCount. So, the test is wrong.

>>FAIL: gnu.testlet.java.util.Arrays.sort (number 1)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 2)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 3)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 4)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 5)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 6)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 7)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 8)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 9)
>>FAIL: gnu.testlet.java.util.Arrays.sort (number 10)
>>
>Ugh. Our sort does not sort...
>Some example failures:
>
>- 79,84,94,42,84,75,11,52,84,18,69,76,14,69,32,28,49,87,98,39,.
>+ 11,14,18,28,32,39,42,49,52,69,79,69,75,76,84,84,84,87,94,98
>

Oh, crap ;-(. I thought I had fixed these last year. Either I was wrong 
or someone has broken them again.

>>FAIL: gnu.testlet.java.util.Hashtable.AcuniaHashtableTest: should throw NullPointerException (number 1)
>>
>Our docs say: "This is the new API for the old contains(), except that
>it is forgiving of null."
>But the spec says "Note that this method is identical in functionality
>to contains"
>

This is a bug in Hashtable. Probibly the wrong code has been 
accidentally pasted in from HashMap. I'll check in a patch soon.

>>FAIL: gnu.testlet.java.util.Hashtable.AcuniaHashtableTest: should throw NullPointerException -- 1 (number 1)
>>
>remove(null). Our docs say "Map.remove and Dictionary.remove disagree
>whether null is a valid parameter; at the moment, this implementation
>obeys Map.remove, and silently ignores null."
>

It should throw for a null argument. Another bug.

>>FAIL: gnu.testlet.java.util.LinkedList.AcuniaLinkedListTest: should throw a ConcurrentModificationException -- 7 (number 1)
>>
>Same issue as with ArrayList.
>set(int, Object) does not make the Iterator throw a
>ConcurrentModificationException. Seems to not be required by the spec
>though. Do people depend on this behaviour?
>

Not a bug, the test is wrong.

regards

Bryce.

Re: Analysis of Mauve failures - Part 2

[Mark Wielaard]

Hi,

On Thu, 2002-04-04 at 05:34, Bryce McKinlay wrote:
> Mark Wielaard wrote:
> >>FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw a ConcurrentModificationException -- 7 (number 1)
> >>
> >set(int, Object) does not make the Iterator throw a
> >ConcurrentModificationException. Seems to not be required by the spec
> >though.
> 
> set() should not update modCount (and thus not cause iterators to throw 
> ConcurrentModificationException), because it is not a structural change 
> to the list. Only operations that could result in adding or removing 
> entries should update modCount. So, the test is wrong.

Where does the definition of structural change come from? Although I
understand what you say I cannot find this definition in the spec. The
spec says that all standard Collection classes have to be fail-fast and
it says that the behavior of an iterator is unspecified if the
underlying collection is modified while the iteration is in progress in
any way other than by calling this method (Iterator.remove()).

So we are interpreting "modification of an collection" as modification
of the actual "structure" of the collection. That seems like a good
definition, but I cannot find it in the spec/api.

I quickly checked against kaffe and a Sun JDK and they seem to agree
since the following program does not throw a
ConcurrentModificationException:

import java.util.*;
public class FailFast
{
  public static void main(String[] args)
  {
    ArrayList al = new ArrayList();
    Object o = new Object();
    al.add(o);
    al.add(new Object());
    Iterator it = al.iterator();
    while (it.hasNext())
      {
        // al.add(new Object());
        // al.remove(o);
        // al.remove(new Object());
        al.set(0, new Object());
        it.next();
      }
  }
}

But uncommenting the al.add() or the al.remove(o) does throw, but the
al.remove(new Object()) does not throw.

Cheers,

Mark

Re: Analysis of Mauve failures - Part 2

[Bryce McKinlay]

Mark Wielaard wrote:

>Where does the definition of structural change come from? Although I
>understand what you say I cannot find this definition in the spec. 
>

It comes from the JCL, IIRC. I don't have my copy handy to check, however.

>So we are interpreting "modification of an collection" as modification
>of the actual "structure" of the collection. 
>

Right, and "structure" modification can be defined as any change to the 
size of the collection or order of the elements.

>But uncommenting the al.add() or the al.remove(o) does throw, but the
>al.remove(new Object()) does not throw.
>

It is interesting that list.remove(object_not_in_list) does not cause 
ConcurrentModificationException, because I also remember interpreting 
something in the JCL as "calling any method that _could_ result in a 
structural change should cause ConcurrentModificationException", but in 
this case it seems that the exception only happens if a change actually 
occurs. Of course, its not uncommon for the JCL to not match the actual 
implementation in current JDKs!

regards

Bryce.

Structural modification definition (Was: Analysis of Mauve failures - Part 2)

[Mark Wielaard]

Hi,

> >Where does the definition of structural change come from? Although I

For future reference. I found the definition:
http://java.sun.com/products/jdk/1.2/docs/api/java/util/AbstractList.html#modCount
"Structural modifications are those that change the size of the list, or
otherwise perturb it in such a fashion that iterations in progress may
yield incorrect results."

Cheers,

Mark

Re: Analysis of Mauve failures - Part 2

[Tom Tromey]

>>>>> "Bryce" == Bryce McKinlay <bryce@waitaki.otago.ac.nz> writes:

>> got 'Fri, 18 May 2001 10:18:06 PDT' but expected
>> 'Fri, 18 May 2001 17:18:06 GMT'
>> Are we missing a Calendar.setTimeZone() somewere?

Bryce> TimeZone and Calendar stuff is known to be somewhat buggy in
Bryce> libgcj so it wouldn't surprise me if something really is wrong
Bryce> here.

At one point most of the Mauve tests here worked for us.  Some
regressions were introduced at some point, but I never got around to
looking at them.  The Classpath calendar/time code is extremely
sensitive to change.  In my experience it is hard to fix a bug without
introducing a regression.

Tom

Re: Analysis of Mauve failures - Part 2

[Mark Wielaard]

Hi,

On Mon, 2002-04-08 at 22:45, Tom Tromey wrote:
> At one point most of the Mauve tests here worked for us.  Some
> regressions were introduced at some point, but I never got around to
> looking at them.  The Classpath calendar/time code is extremely
> sensitive to change.  In my experience it is hard to fix a bug without
> introducing a regression.

I would really like to go back to a state were we don't have any
unexpected failures for mauve (and Gnats reports for everything that we
expect to fail).

Would it it be OK for me to mark the remaining java.text mauve failures
as xfail and make bug reports for them? Or do you think you will
analyze/fix them before 3.1?

The failures that I am thinking of now would be:

FAIL: gnu.testlet.java.text.AttributedString.Test: Attribute key count
(number 1)
FAIL: gnu.testlet.java.text.DateFormatSymbols.Test: invalid locale
(number 1)
FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: CDT (number 1)
FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: EDT (number 1)
FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: EST (number 1)
FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: PDT (number 1)

But if you see others that you immediately know won't be further
analyzed/fixed for 3.1 please let me know.

Note my latest mauve results for i686-pc-linux-gnu can be found at
http://gcc.gnu.org/ml/gcc-testresults/2002-04/msg00259.html
(Only 42 unexpected failures to go)

There are also results for powerpc-unknown-linux-gnu, but they don't
look that good (110 unexpected failures).
http://gcc.gnu.org/ml/gcc-testresults/2002-04/msg00275.html

Cheers,

Mark

Re: Analysis of Mauve failures - Part 2

[Tom Tromey]

>>>>> "Mark" == Mark Wielaard <mark@klomp.org> writes:

Mark> I would really like to go back to a state were we don't have any
Mark> unexpected failures for mauve (and Gnats reports for everything
Mark> that we expect to fail).

That sounds good.  It's easier to use the test suite when you know
that any FAIL is a regression.

Mark> Would it it be OK for me to mark the remaining java.text mauve
Mark> failures as xfail and make bug reports for them? Or do you think
Mark> you will analyze/fix them before 3.1?

I doubt there's time to fix many of the problems, so xfailing is fine
by me.

Mark> FAIL: gnu.testlet.java.text.AttributedString.Test: Attribute key count
Mark> (number 1)

This isn't really a regression; we never had AttributedString before.
(Maybe it is a regression for Classpath though)

Mark> There are also results for powerpc-unknown-linux-gnu, but they don't
Mark> look that good (110 unexpected failures).
Mark> http://gcc.gnu.org/ml/gcc-testresults/2002-04/msg00275.html

Why is this platform so much worse?  I'm surprised by that.

Tom

Re: Analysis of Mauve failures - Part 2

[Bryce McKinlay]

Tom Tromey wrote:

>Mark> There are also results for powerpc-unknown-linux-gnu, but they don't
>Mark> look that good (110 unexpected failures).
>Mark> http://gcc.gnu.org/ml/gcc-testresults/2002-04/msg00275.html
>
>Why is this platform so much worse?  I'm surprised by that.
>

Are these compilation-from-bytecode failures by any chance? I've noticed 
problems with doubles and longs on PPC when doing that.

regards

Bryce.

Re: Analysis of Mauve failures - Part 2

[Mark Wielaard]

Hi,

On Wed, 2002-04-10 at 02:49, Bryce McKinlay wrote:
> Tom Tromey wrote:
> 
> >Mark> There are also results for powerpc-unknown-linux-gnu, but they don't
> >Mark> look that good (110 unexpected failures).
> >Mark> http://gcc.gnu.org/ml/gcc-testresults/2002-04/msg00275.html
> >
> >Why is this platform so much worse?  I'm surprised by that.
> 
> Are these compilation-from-bytecode failures by any chance? I've noticed 
> problems with doubles and longs on PPC when doing that.

Yes, they are. And most of the extra failures seem to be double and long
related.

Cheers,

Mark

P.S. These are the extra mauve failures on powerpc:

+FAIL: Array_3 execution from source compiled test
+FAIL: Array_3 execution from bytecode->native test
+FAIL: Array_3 -O execution from source compiled test
+FAIL: Array_3 -O execution from bytecode->native test
+FAIL: test_long output from bytecode->native test
+FAIL: test_long -O output from bytecode->native test
+FAIL: gnu.testlet.java.io.DataInputOutput.ReadReference: readDouble (number 1)
+FAIL: gnu.testlet.java.io.ObjectStreamClass.Test: getSerialVersionUID (number 1)
+FAIL: gnu.testlet.java.io.ObjectStreamClass.Test: getSerialVersionUID (number 2)
+FAIL: gnu.testlet.java.io.ObjectStreamClass.Test: getSerialVersionUID (number 3)
+FAIL: gnu.testlet.java.io.ObjectStreamClass.Test: getSerialVersionUID (number 5)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_Basics failed - 1b (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_Basics failed - 4 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_remainder failed 2 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_remainder failed 10 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_toString failed - 2 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_toString failed - 10 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_doubleToLongBits returned wrong results - 1 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_doubleToLongBits returned wrong results - 2 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_doubleToLongBits CYGNUS: NaN.doubleToLongBits (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_doubleToLongBits returned wrong results - 8 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_longBitsToDouble returned wrong results - 1 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_longBitsToDouble returned wrong results - 2 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_longBitsToDouble returned wrong results - 3 (number 1)
+FAIL: gnu.testlet.java.lang.Double.DoubleTest: Error: test_longBitsToDouble returned wrong results - 4 (number 1)
+FAIL: gnu.testlet.java.lang.Double.new_Double (number 6)
+FAIL: gnu.testlet.java.lang.Double.new_Double (number 7)
+FAIL: gnu.testlet.java.lang.Float.FloatTest: Error: test_floatValue returned wrong results - 2 (number 1)
+FAIL: gnu.testlet.java.lang.Integer.IntegerTest: Error: test_Basics failed - 1 (number 1)
+FAIL: gnu.testlet.java.lang.Long.LongTest: Error: test_Basics failed - 1 (number 1)
+FAIL: gnu.testlet.java.lang.Long.LongTest: Error: test_parseLong returned wrong results - 8 (number 1)
+FAIL: gnu.testlet.java.lang.Long.LongTest: Error: test_parseLong returned wrong results - 12 (number 1)
+FAIL: gnu.testlet.java.lang.Long.getLong (number 2)
+FAIL: gnu.testlet.java.lang.Long.getLong (number 4)
+FAIL: gnu.testlet.java.lang.Long.getLong (number 6)
+FAIL: gnu.testlet.java.lang.Long.getLong (number 7)
+FAIL: gnu.testlet.java.lang.Long.getLong (number 8)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 1)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 6)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 7)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 8)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 9)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 18)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 22)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 26)
+FAIL: gnu.testlet.java.lang.Long.new_Long (number 35)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toBinaryString (number 5)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toBinaryString (number 6)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toBinaryString (number 7)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toOctalString (number 5)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toOctalString (number 6)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toOctalString (number 7)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toHexString (number 5)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toHexString (number 6)
+FAIL: gnu.testlet.java.lang.Long.new_Long: toHexString (number 7)
+FAIL: gnu.testlet.java.lang.Long.new_Long: parseLong (number 6)
+FAIL: gnu.testlet.java.lang.Long.new_Long: parseLong (number 8)
+FAIL: gnu.testlet.java.lang.Long.new_Long: parseLong (number 10)
+FAIL: gnu.testlet.java.lang.Math.MathTest: Error : test_Basics failed - 1 (number 1)
+FAIL: gnu.testlet.java.lang.Math.MathTest: Error :  test_round failed - 7 (number 1)
+FAIL: gnu.testlet.java.lang.Math.MathTest: Error :  test_abs failed - 6 (number 1)
+FAIL: gnu.testlet.java.lang.Math.sin (number 1)
+FAIL: gnu.testlet.java.lang.StringBuffer.plus (number 6)
+FAIL: gnu.testlet.java.lang.ref.PhantomReference.phantom: object finalized (number 1)
+FAIL: gnu.testlet.java.lang.ref.WeakReference.weakref: unreachable (number 1)
+FAIL: gnu.testlet.java.lang.ref.WeakReference.weakref: contents of weak reference (number 1)
+FAIL: gnu.testlet.java.math.BigDecimal.construct (number 2)
+FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: Error : test_Basics failed - 8 the address array should of length 1 (number 1)