GCJ Mauve testing

GCJ Mauve testing

[Andrew John Hughes]

I've just run Mauve against both the 4.3 tree and our
Classpath 0.98 merge tree (effectively equivalent to Classpath
CVS + GCJ-specific changes).  Overall:

4.3:
281 of 3177 tests failed.  123926 total calls to harness.check() failed.

4.4:
239 of 3177 tests failed.  143832 total calls to harness.check() failed.

There will be some new tests that compile with 4.4 but not 4.3 (e.g.
java.util.Scanner tests).

Additional FAILs are as follows:

+FAIL: gnu.java.security.util.TestOfIntegerUtil
+FAIL: java.sql.Date.DateTest
+FAIL: java.lang.Character.Blocks15
+FAIL: java.lang.InheritableThreadLocal.simple
+FAIL: java.text.SimpleDateFormat.parse
+FAIL: java.util.Calendar.ampm
+FAIL: java.util.Scanner.LotsOfPMInts
+FAIL: java.util.Scanner.Inputs
+FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
+FAIL: javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground

The two Scanner tests we can discount as regressions as they simply won't
have run with 4.3.

The remaining failures:

FAIL: gnu.java.security.util.TestOfIntegerUtil
  line 71: IntegerUtil.valueOf(String) MUST be faster than Integer.valueOf(String) [1] -- boolean passed to check was fa\
lse
FAIL: java.sql.Date.DateTest
  uncaught exception:
   java.lang.OutOfMemoryError
FAIL: java.lang.Character.Blocks15
  uncaught exception:
   java.lang.OutOfMemoryError
FAIL: java.lang.InheritableThreadLocal.simple
  Test timed out.  Use -timeout [millis] option to change the timeout value.
FAIL: java.text.SimpleDateFormat.parse
  line 132: parse pattern yyyy-MM-dd HH:mm z [3] -- Objects were not equal.  Use -debug for more information.
  line 132: parse pattern yyyy-MM-dd HH:mm Z [3] -- Objects were not equal.  Use -debug for more information.
FAIL: java.util.Calendar.ampm
  line 72: 12:00 AM couldn't be parsed [1] -- forced fail
  line 72: 12:10 AM couldn't be parsed [1] -- forced fail
  line 57: 12:10 AM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
  line 72: 0:00 AM couldn't be parsed [1] -- forced fail
  line 57: 0:00 AM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
  line 72: 0:10 AM couldn't be parsed [1] -- forced fail
  line 72: 12:00 PM couldn't be parsed [1] -- forced fail
  line 72: 12:10 PM couldn't be parsed [1] -- forced fail
  line 57: 12:10 PM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
  line 72: 0:00 PM couldn't be parsed [1] -- forced fail
  line 57: 0:00 PM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
  line 72: 0:10 PM couldn't be parsed [1] -- forced fail
FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
  line 746: EmptyStackException must be thrown [1] -- forced fail
  line 759: EmptyStackException must be thrown [3] -- got 4 but expected 3
  line 765: EmptyStackException must be thrown [7] -- got 7 but expected 15
  line 767: EmptyStackException must be thrown [8] -- got 12 but expected 15
  line 768: EmptyStackException must be thrown [9] -- got 15 but expected 16
  line 771: EmptyStackException must be thrown [10] -- got javax.swing.text.AbstractDocument$DefaultDocumentEvent but ex\
pected null
FAIL: javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground
  line 54:  [2] -- got javax.swing.plaf.ColorUIResource but expected java.awt.Color

The first is odd, and CACAO+Classpath CVS fails this also. OpenJDK can't compile it.  CACAO + Classpath CVS
also fails with java.text.SimpleDateFormat.parse, java.util.Calendar.ampm and
javax.swing.text.DefaultStyledDocument.ElementBuffer.insert, which suggest a common Classpath issue.
java.sql.Date.DateTest, java.lang.Character.Blocks15, java.lang.InheritableThreadLocal.simple and
javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground are
all GCJ-specific failures.  OpenJDK passes all but the first, which is GNU Classpath specific.

Full logs are available at:

http://fuseyism.com/classpath/mauve-gcj-4.3.log
http://fuseyism.com/classpath/mauve-gcj-4.4.log
-- 
Andrew :)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net
PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint = F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

Re: GCJ Mauve testing

[Andrew John Hughes]

2008/9/27 Andrew John Hughes <gnu_andrew@member.fsf.org>:
> I've just run Mauve against both the 4.3 tree and our
> Classpath 0.98 merge tree (effectively equivalent to Classpath
> CVS + GCJ-specific changes).  Overall:
>
> 4.3:
> 281 of 3177 tests failed.  123926 total calls to harness.check() failed.
>
> 4.4:
> 239 of 3177 tests failed.  143832 total calls to harness.check() failed.
>
> There will be some new tests that compile with 4.4 but not 4.3 (e.g.
> java.util.Scanner tests).
>
> Additional FAILs are as follows:
>
> +FAIL: gnu.java.security.util.TestOfIntegerUtil
> +FAIL: java.sql.Date.DateTest
> +FAIL: java.lang.Character.Blocks15
> +FAIL: java.lang.InheritableThreadLocal.simple
> +FAIL: java.text.SimpleDateFormat.parse
> +FAIL: java.util.Calendar.ampm
> +FAIL: java.util.Scanner.LotsOfPMInts
> +FAIL: java.util.Scanner.Inputs
> +FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
> +FAIL: javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground
>
> The two Scanner tests we can discount as regressions as they simply won't
> have run with 4.3.
>
> The remaining failures:
>
> FAIL: gnu.java.security.util.TestOfIntegerUtil
>  line 71: IntegerUtil.valueOf(String) MUST be faster than Integer.valueOf(String) [1] -- boolean passed to check was fa\
> lse
> FAIL: java.sql.Date.DateTest
>  uncaught exception:
>   java.lang.OutOfMemoryError
> FAIL: java.lang.Character.Blocks15
>  uncaught exception:
>   java.lang.OutOfMemoryError
> FAIL: java.lang.InheritableThreadLocal.simple
>  Test timed out.  Use -timeout [millis] option to change the timeout value.
> FAIL: java.text.SimpleDateFormat.parse
>  line 132: parse pattern yyyy-MM-dd HH:mm z [3] -- Objects were not equal.  Use -debug for more information.
>  line 132: parse pattern yyyy-MM-dd HH:mm Z [3] -- Objects were not equal.  Use -debug for more information.
> FAIL: java.util.Calendar.ampm
>  line 72: 12:00 AM couldn't be parsed [1] -- forced fail
>  line 72: 12:10 AM couldn't be parsed [1] -- forced fail
>  line 57: 12:10 AM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
>  line 72: 0:00 AM couldn't be parsed [1] -- forced fail
>  line 57: 0:00 AM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
>  line 72: 0:10 AM couldn't be parsed [1] -- forced fail
>  line 72: 12:00 PM couldn't be parsed [1] -- forced fail
>  line 72: 12:10 PM couldn't be parsed [1] -- forced fail
>  line 57: 12:10 PM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
>  line 72: 0:00 PM couldn't be parsed [1] -- forced fail
>  line 57: 0:00 PM couldn't be parsed [2] -- Objects were not equal.  Use -debug for more information.
>  line 72: 0:10 PM couldn't be parsed [1] -- forced fail
> FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
>  line 746: EmptyStackException must be thrown [1] -- forced fail
>  line 759: EmptyStackException must be thrown [3] -- got 4 but expected 3
>  line 765: EmptyStackException must be thrown [7] -- got 7 but expected 15
>  line 767: EmptyStackException must be thrown [8] -- got 12 but expected 15
>  line 768: EmptyStackException must be thrown [9] -- got 15 but expected 16
>  line 771: EmptyStackException must be thrown [10] -- got javax.swing.text.AbstractDocument$DefaultDocumentEvent but ex\
> pected null
> FAIL: javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground
>  line 54:  [2] -- got javax.swing.plaf.ColorUIResource but expected java.awt.Color
>
> The first is odd, and CACAO+Classpath CVS fails this also. OpenJDK can't compile it.  CACAO + Classpath CVS
> also fails with java.text.SimpleDateFormat.parse, java.util.Calendar.ampm and
> javax.swing.text.DefaultStyledDocument.ElementBuffer.insert, which suggest a common Classpath issue.
> java.sql.Date.DateTest, java.lang.Character.Blocks15, java.lang.InheritableThreadLocal.simple and
> javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground are
> all GCJ-specific failures.  OpenJDK passes all but the first, which is GNU Classpath specific.
>
> Full logs are available at:
>
> http://fuseyism.com/classpath/mauve-gcj-4.3.log
> http://fuseyism.com/classpath/mauve-gcj-4.4.log
> --
> Andrew :)
>
> Support Free Java!
> Contribute to GNU Classpath and the OpenJDK
> http://www.gnu.org/software/classpath
> http://openjdk.java.net
> PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
> Fingerprint = F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8
>

Changed to:

http://fuseyism.com/classpath/mauve-gcj-4.3.log.gz
http://fuseyism.com/classpath/mauve-gcj-4.4.log.gz

due to size...
-- 
Andrew :-)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8

Re: GCJ Mauve testing

[Andrew Haley]

Andrew John Hughes wrote:

> FAIL: java.lang.InheritableThreadLocal.simple
>   Test timed out.  Use -timeout [millis] option to change the timeout value.

This is strange, given that gcj has its own InheritableThreadLocal that
hasn't changed.  I'll look.

Andrew.

Re: GCJ Mauve testing

[Andrew John Hughes]

2008/10/6 Andrew Haley <aph@redhat.com>:
> Andrew John Hughes wrote:
>
>> FAIL: java.lang.InheritableThreadLocal.simple
>>   Test timed out.  Use -timeout [millis] option to change the timeout value.
>
> This is strange, given that gcj has its own InheritableThreadLocal that
> hasn't changed.  I'll look.
>
> Andrew.
>

I did another Mauve run against the merge branch and gained some passes,
but also some new failures:

+PASS: gnu.java.security.util.TestOfIntegerUtil
+PASS: java.awt.event.MouseEvent.modifiersEx
+PASS: java.awt.event.MouseEvent.modifiers
+PASS: java.util.logging.Logger.global
+PASS: javax.swing.plaf.metal.MetalScrollBarUI.constructor
+PASS: javax.swing.plaf.metal.MetalScrollBarUI.installDefaults
+FAIL: java.util.logging.Logger.hierarchyChecks
+FAIL: java.util.logging.Logger.securityChecks
+FAIL: java.util.logging.SocketHandler.getFormatter
+FAIL: java.util.Scanner.DoubleFloat
+FAIL: java.util.Scanner.FishString
+FAIL: java.util.Scanner.Radix
+FAIL: javax.sound.sampled.AudioProperties
+FAIL: javax.swing.JTable.TableRobot

Odd to say the least.
-- 
Andrew :-)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

Re: GCJ Mauve testing

[Andrew Haley]

Andrew John Hughes wrote:
> 2008/10/6 Andrew Haley <aph@redhat.com>:
>> Andrew John Hughes wrote:
>>
>>> FAIL: java.lang.InheritableThreadLocal.simple
>>>   Test timed out.  Use -timeout [millis] option to change the timeout value.
>> This is strange, given that gcj has its own InheritableThreadLocal that
>> hasn't changed.  I'll look.
>>
>> Andrew.
>>
> 
> I did another Mauve run against the merge branch and gained some passes,
> but also some new failures:
> 
> +PASS: gnu.java.security.util.TestOfIntegerUtil
> +PASS: java.awt.event.MouseEvent.modifiersEx
> +PASS: java.awt.event.MouseEvent.modifiers
> +PASS: java.util.logging.Logger.global
> +PASS: javax.swing.plaf.metal.MetalScrollBarUI.constructor
> +PASS: javax.swing.plaf.metal.MetalScrollBarUI.installDefaults
> +FAIL: java.util.logging.Logger.hierarchyChecks
> +FAIL: java.util.logging.Logger.securityChecks
> +FAIL: java.util.logging.SocketHandler.getFormatter
> +FAIL: java.util.Scanner.DoubleFloat
> +FAIL: java.util.Scanner.FishString
> +FAIL: java.util.Scanner.Radix
> +FAIL: javax.sound.sampled.AudioProperties
> +FAIL: javax.swing.JTable.TableRobot
> 
> Odd to say the least.

Curious.  It's not immediately obvious how the change could have affected
those, but I'll have a look.

Andrew.

Re: GCJ Mauve testing

[Andrew John Hughes]

2008/10/7 Andrew Haley <aph@redhat.com>:
> Andrew John Hughes wrote:
>> 2008/10/6 Andrew Haley <aph@redhat.com>:
>>> Andrew John Hughes wrote:
>>>
>>>> FAIL: java.lang.InheritableThreadLocal.simple
>>>>   Test timed out.  Use -timeout [millis] option to change the timeout value.
>>> This is strange, given that gcj has its own InheritableThreadLocal that
>>> hasn't changed.  I'll look.
>>>
>>> Andrew.
>>>
>>
>> I did another Mauve run against the merge branch and gained some passes,
>> but also some new failures:
>>
>> +PASS: gnu.java.security.util.TestOfIntegerUtil
>> +PASS: java.awt.event.MouseEvent.modifiersEx
>> +PASS: java.awt.event.MouseEvent.modifiers
>> +PASS: java.util.logging.Logger.global
>> +PASS: javax.swing.plaf.metal.MetalScrollBarUI.constructor
>> +PASS: javax.swing.plaf.metal.MetalScrollBarUI.installDefaults
>> +FAIL: java.util.logging.Logger.hierarchyChecks
>> +FAIL: java.util.logging.Logger.securityChecks
>> +FAIL: java.util.logging.SocketHandler.getFormatter
>> +FAIL: java.util.Scanner.DoubleFloat
>> +FAIL: java.util.Scanner.FishString
>> +FAIL: java.util.Scanner.Radix
>> +FAIL: javax.sound.sampled.AudioProperties
>> +FAIL: javax.swing.JTable.TableRobot
>>
>> Odd to say the least.
>
> Curious.  It's not immediately obvious how the change could have affected
> those, but I'll have a look.
>
> Andrew.
>
>


I think some of them may be spurious.
gnu.java.security.util.TestOfIntegerUtil particularly is based on
timing.  I'll post the test log shortly.
-- 
Andrew :-)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

Re: GCJ Mauve testing

[Matthias Klose]

Andrew John Hughes schrieb:
> I've just run Mauve against both the 4.3 tree and our
> Classpath 0.98 merge tree (effectively equivalent to Classpath
> CVS + GCJ-specific changes).  Overall:
> 
> 4.3:
> 281 of 3177 tests failed.  123926 total calls to harness.check() failed.
> 
> 4.4:
> 239 of 3177 tests failed.  143832 total calls to harness.check() failed.
> 
> There will be some new tests that compile with 4.4 but not 4.3 (e.g.
> java.util.Scanner tests).

Here is another run, done with trunk and branch 20081017.

4.3:
277 of 3175 tests failed.  139350 total calls to harness.check() failed.

trunk:
232 of 3175 tests failed.  122871 total calls to harness.check() failed.

merge branch (as 4.4 above):
229 of 3175 tests failed.  122914 total calls to harness.check() failed.

> Additional FAILs are as follows:
> 
> +FAIL: gnu.java.security.util.TestOfIntegerUtil
> +FAIL: java.sql.Date.DateTest
> +FAIL: java.lang.Character.Blocks15
> +FAIL: java.lang.InheritableThreadLocal.simple
> +FAIL: java.text.SimpleDateFormat.parse
> +FAIL: java.util.Calendar.ampm
> +FAIL: java.util.Scanner.LotsOfPMInts
> +FAIL: java.util.Scanner.Inputs
> +FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
> +FAIL: javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground

comparing 4.3 with the merge branch:

56 progressions and

+FAIL: java.awt.TextField.PaintTest
+FAIL: java.lang.ref.PhantomReference.phantom
+FAIL: java.text.SimpleDateFormat.parse
+FAIL: java.util.Calendar.ampm
+FAIL: java.util.Scanner.FindWithinHorizon
+FAIL: java.util.logging.Logger.getParent
+FAIL: javax.swing.JButton.constructors
+FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert

The following three go away if the cacerts file is in place.

+FAIL: gnu.java.security.jce.TestOfHttps
+FAIL: gnu.java.security.util.TestOfIntegerUtil
+FAIL: java.net.URLConnection.getHeaderFields

comparing the trunk with the merge branch (0.97.2 vs. 0.98pre):

9 progressions and

+FAIL: gnu.java.security.util.TestOfIntegerUtil
+FAIL: java.awt.Component.clickModifiers
+FAIL: java.text.SimpleDateFormat.Localization
+FAIL: java.util.Calendar.ampm
+FAIL: java.util.Currency.US
+FAIL: java.util.Scanner.FindWithinHorizon
+FAIL: java.util.logging.Logger.getParent

Full logs at http://people.ubuntu.com/~doko/java/mauve-results/

libjava/testsuite/libjava.mauve doesn't work with current mauve, so all builds were installed, and
mauve was run on the installed build.

  Matthias

Re: GCJ Mauve testing

[Andrew John Hughes]

2008/10/18 Matthias Klose <doko@ubuntu.com>:
> Andrew John Hughes schrieb:
>> I've just run Mauve against both the 4.3 tree and our
>> Classpath 0.98 merge tree (effectively equivalent to Classpath
>> CVS + GCJ-specific changes).  Overall:
>>
>> 4.3:
>> 281 of 3177 tests failed.  123926 total calls to harness.check() failed.
>>
>> 4.4:
>> 239 of 3177 tests failed.  143832 total calls to harness.check() failed.
>>
>> There will be some new tests that compile with 4.4 but not 4.3 (e.g.
>> java.util.Scanner tests).
>
> Here is another run, done with trunk and branch 20081017.
>

Thanks for this.

> 4.3:
> 277 of 3175 tests failed.  139350 total calls to harness.check() failed.
>
> trunk:
> 232 of 3175 tests failed.  122871 total calls to harness.check() failed.
>
> merge branch (as 4.4 above):
> 229 of 3175 tests failed.  122914 total calls to harness.check() failed.
>
>> Additional FAILs are as follows:
>>
>> +FAIL: gnu.java.security.util.TestOfIntegerUtil
>> +FAIL: java.sql.Date.DateTest
>> +FAIL: java.lang.Character.Blocks15
>> +FAIL: java.lang.InheritableThreadLocal.simple
>> +FAIL: java.text.SimpleDateFormat.parse
>> +FAIL: java.util.Calendar.ampm
>> +FAIL: java.util.Scanner.LotsOfPMInts
>> +FAIL: java.util.Scanner.Inputs
>> +FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
>> +FAIL: javax.swing.table.JTableHeader.AccessibleJTableHeader.AccessibleJTableHeaderEntry.getForeground
>
> comparing 4.3 with the merge branch:
>
> 56 progressions and
>
> +FAIL: java.awt.TextField.PaintTest
> +FAIL: java.lang.ref.PhantomReference.phantom
> +FAIL: java.text.SimpleDateFormat.parse
> +FAIL: java.util.Calendar.ampm
> +FAIL: java.util.Scanner.FindWithinHorizon
> +FAIL: java.util.logging.Logger.getParent
> +FAIL: javax.swing.JButton.constructors
> +FAIL: javax.swing.text.DefaultStyledDocument.ElementBuffer.insert
>
> The following three go away if the cacerts file is in place.
>
> +FAIL: gnu.java.security.jce.TestOfHttps
> +FAIL: gnu.java.security.util.TestOfIntegerUtil
> +FAIL: java.net.URLConnection.getHeaderFields
>
> comparing the trunk with the merge branch (0.97.2 vs. 0.98pre):
>
> 9 progressions and
>
> +FAIL: gnu.java.security.util.TestOfIntegerUtil
> +FAIL: java.awt.Component.clickModifiers
> +FAIL: java.text.SimpleDateFormat.Localization
> +FAIL: java.util.Calendar.ampm
> +FAIL: java.util.Currency.US
> +FAIL: java.util.Scanner.FindWithinHorizon
> +FAIL: java.util.logging.Logger.getParent
>
> Full logs at http://people.ubuntu.com/~doko/java/mauve-results/
>
> libjava/testsuite/libjava.mauve doesn't work with current mauve, so all builds were installed, and
> mauve was run on the installed build.
>
>  Matthias
>
>

The Scanner tests can be written off, as they won't have compiled with
either 4.3 or trunk
(neither have java.util.Scanner).  The merge branch seems to be an
improvement, but there
are clearly some new localisation issues.

What is the plan for merging to trunk?
-- 
Andrew :-)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

Re: GCJ Mauve testing

[Andrew Haley]

Andrew John Hughes wrote:

> What is the plan for merging to trunk?

If we're sure it fixes bugs and doesn't introduce new ones,
once it's stable we can merge.  We're close, I think.

Andrew.