[Bug libgcj/49314] New: md5test, shatest output FAILs on Tru64 UNIX

[Bug libgcj/49314] New: md5test, shatest output FAILs on Tru64 UNIX

[ro at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

           Summary: md5test, shatest output FAILs on Tru64 UNIX
           Product: gcc
           Version: 4.7.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgcj
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: ro@gcc.gnu.org
              Host: alpha-dec-osf5.1b
            Target: alpha-dec-osf5.1b
             Build: alpha-dec-osf5.1b


Of the 12 remaining libjava failures on Tru64 UNIX, 8 are due to the same
reason:

FAIL: md5test output - source compiled test
FAIL: md5test -findirect-dispatch output - source compiled test
FAIL: md5test -O3 output - source compiled test
FAIL: md5test -O3 -findirect-dispatch output - source compiled test
FAIL: shatest execution - source compiled test
FAIL: shatest -findirect-dispatch execution - source compiled test
FAIL: shatest -O3 execution - source compiled test
FAIL: shatest -O3 -findirect-dispatch execution - source compiled test

Running md5test manually, I get 6 times this output:

java.lang.ArrayIndexOutOfBoundsException
   at java.lang.System.arraycopy(Unknown Source)
   at gnu.java.security.hash.BaseHash.update(Unknown Source)
   at gnu.java.security.jce.hash.MessageDigestAdapter.engineUpdate(Unknown
Source)
   at java.security.DummyMessageDigest.engineUpdate(Unknown Source)
   at java.security.MessageDigest.update(Unknown Source)
   at md5test.testString(Unknown Source)
   at md5test.main(Unknown Source)

Unfortunately, gdb 7.2.50 is no help:

Reading symbols from
/var/gcc/gcc-4.7.0-20110530/5.1b-gcc/alpha-dec-osf5.1b/libjava/testsuite/md5test.exe...done.
(gdb) run
Starting program:
/var/gcc/gcc-4.7.0-20110530/5.1b-gcc/alpha-dec-osf5.1b/libjava/testsuite/md5test.exe 
/vol/src/gnu/gdb/gdb-git/gdb/utils.c:1446: internal-error: virtual memory
exhausted: can't allocate 8375 bytes.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) y

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

Rainer Orth <ro at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aph at gcc dot gnu.org,
                   |                            |tromey at gcc dot gnu.org
   Target Milestone|---                         |4.7.0

--- Comment #1 from Rainer Orth <ro at gcc dot gnu.org> 2011-06-16 16:59:31 UTC ---
After some further debugging, I could reproduce the issue with the following
testcase:

class arraycopy {
    public static void main(String[] argv) {
        String src = "a";
        byte dst[] = new byte[64];

        System.arraycopy (src.getBytes(), 0, dst, 0, src.length());
    }
}

With some debug code in java/lang/natSystem.cc (java::lang::System::arraycopy),
I get

arraycopy: src_offset = 0 src_a->length = 0
           dst_offset = 0 dst_a->length = 64
           count = 1
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException
   at java.lang.System.arraycopy(Unknown Source)
   at arraycopy.main(Unknown Source)

I haven't yet found where src_a->length is supposed to be initialized.  Any
suggestions?

  Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[tromey at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #2 from Tom Tromey <tromey at gcc dot gnu.org> 2011-06-16 17:44:03 UTC ---
(In reply to comment #1)

> I haven't yet found where src_a->length is supposed to be initialized.  Any
> suggestions?

It is set when the array is created.
I would suggest stepping through String::getBytes.

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #3 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-16 18:18:59 UTC ---
> It is set when the array is created.
> I would suggest stepping through String::getBytes.

When I do this, I find that the count argument to _Jv_NewPrimArray is
0.  It is read in natString.cc:

jbyteArray java::lang::String::getBytes (jstring enc)

which calls String.java (public int length()).  This just return the
count member.  I couldn't find so far where this is initialized for the
static string src.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[tromey at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #4 from Tom Tromey <tromey at gcc dot gnu.org> 2011-06-16 18:57:35 UTC ---
(In reply to comment #3)
> I couldn't find so far where this is initialized for the
> static string src.

It has been a while, but I think either _Jv_NewStringUTF
or _Jv_NewStringUtf8Const.  IIRC one of these is run
during class initialization to turn string constants into
String objects.

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #5 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-16 20:51:00 UTC ---
> It has been a while, but I think either _Jv_NewStringUTF
> or _Jv_NewStringUtf8Const.  IIRC one of these is run
> during class initialization to turn string constants into
> String objects.

Thanks.  It turned out those were ok.

When I logged calls to JvNewByteArray, I found two:

JvNewByteArray: length = 1
JvNewByteArray: length = 0

The first one is as expected for "a", the second seems fishy: at the end
of java::lang::String::getBytes, bufpos was 0 and buflen 1, so a new
empty buffer was returned.

Looking further, this is a problem with iconv somehow: in
gnu::gcj::convert::Output_iconv::write, iconv_adapter returns -1 with
errno == EILSEQ.  Not understanding this iconv business at all, I have
no idea how this can happen.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #6 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-17 09:56:50 UTC ---
As a quick hack, I've just removed the definition of HAVE_ICONV and
rebuilt libgcj.so.  This way, testsuite results are clean.

I'll have to further investigate (preferably with a small C program)
what's going on here.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[dominiq at lps dot ens.fr]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #7 from Dominique d'Humieres <dominiq at lps dot ens.fr> 2011-06-17 13:04:17 UTC ---
Could this pr be related to pr49441?

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #8 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-17 13:09:06 UTC ---
> --- Comment #7 from Dominique d'Humieres <dominiq at lps dot ens.fr> 2011-06-17 13:04:17 UTC ---
> Could this pr be related to pr49441?

No: the iconv functions only live in libiconv (no stubs in libc), and
that is present in libgcj.spec and linked into the test programs.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[howarth at nitro dot med.uc.edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

Jack Howarth <howarth at nitro dot med.uc.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |howarth at nitro dot
                   |                            |med.uc.edu

--- Comment #9 from Jack Howarth <howarth at nitro dot med.uc.edu> 2011-06-17 14:01:23 UTC ---
(In reply to comment #8)
> > --- Comment #7 from Dominique d'Humieres <dominiq at lps dot ens.fr> 2011-06-17 13:04:17 UTC ---
> > Could this pr be related to pr49441?
> 
> No: the iconv functions only live in libiconv (no stubs in libc), and
> that is present in libgcj.spec and linked into the test programs.
> 
>     Rainer

 pr49441 is a totally different issue. That bug manifests itself as a runtime
error with unresolved symbols from libiconv. I still don't understand why we
just don't link libgcj directly against libiconv to avoid needing libgcj.spec
to pass anything for libiconv.

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #10 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-17 14:06:12 UTC ---
>  pr49441 is a totally different issue. That bug manifests itself as a runtime
> error with unresolved symbols from libiconv. I still don't understand why we
> just don't link libgcj directly against libiconv to avoid needing libgcj.spec
> to pass anything for libiconv.

Fully agreed: the gcc runtime libraries should just be self-contained.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #11 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-17 16:15:55 UTC ---
Further investigation revealed part of what's going on: the test in
gnu::gcj::convert::IOConverter::iconv_init fails with EINVAL on Tru64
UNIX, but the byte swapping is still needed.  If I hack the function to
return true, both the testcase and md5test/shatest succeed.

The question now is what's a clean fix for this issue, rather than
hardcoding the result?

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[howarth at nitro dot med.uc.edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #12 from Jack Howarth <howarth at nitro dot med.uc.edu> 2011-06-18 05:54:49 UTC ---
(In reply to comment #11)
> Further investigation revealed part of what's going on: the test in
> gnu::gcj::convert::IOConverter::iconv_init fails with EINVAL on Tru64
> UNIX, but the byte swapping is still needed.  If I hack the function to
> return true, both the testcase and md5test/shatest succeed.
> 
> The question now is what's a clean fix for this issue, rather than
> hardcoding the result?
> 
>     Rainer

Does http://gcc.gnu.org/ml/gcc-patches/2011-06/msg01375.html have any impact on
this bug?

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #13 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-20 14:30:11 UTC ---
> Does http://gcc.gnu.org/ml/gcc-patches/2011-06/msg01375.html have any impact on
> this bug?

No, as I said, the md5test and shatest executables are correctly linked
with libiconv.so and the system library is indeed present in the
program.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at CeBiTec dot Uni-Bielefeld.DE]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #14 from ro at CeBiTec dot Uni-Bielefeld.DE <ro at CeBiTec dot Uni-Bielefeld.DE> 2011-06-21 12:09:30 UTC ---
> The question now is what's a clean fix for this issue, rather than
> hardcoding the result?

I've now checked this with native and GNU iconv: feff.utf-8 is the input
sequence from iconv_init:

> od -x feff.utf-8
0000000 bbef 0abf

At first everything seemed to work with both tools, except that I need
byte swapping with native iconv, but not with GNU iconv:

> /usr/bin/iconv -f UTF-8 -t UCS-2 < feff.utf-8 | od -x
0000000 feff 000a
> /vol/gnu/bin/iconv -f UTF-8 -t UCS-2 < feff.utf-8 | od -x
0000000 fffe 0a00

But when I removed the newline from the input file, results differ:

> od -x feff.utf-8-nonl 
0000000 bbef 00bf

> /usr/bin/iconv -f UTF-8 -t UCS-2 < feff.utf-8-nonl | od -x
0000000 feff feff
> /vol/gnu/bin/iconv -f UTF-8 -t UCS-2 < feff.utf-8-nonl | od -x
0000000 fffe

As result, I've simply added a padding NUL byte to the input sequence.
This let the iconv succeed and my arraycopy testcase and both md5test
and shatest pass.

I'm running a full rebuild and check of libjava for good measure.

    Rainer

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

Rainer Orth <ro at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
                URL|                            |http://gcc.gnu.org/ml/gcc-p
                   |                            |atches/2011-06/msg01596.htm
                   |                            |l
   Last reconfirmed|                            |2011.06.21 13:58:15
         AssignedTo|unassigned at gcc dot       |ro at gcc dot gnu.org
                   |gnu.org                     |
     Ever Confirmed|0                           |1

--- Comment #15 from Rainer Orth <ro at gcc dot gnu.org> 2011-06-21 13:58:15 UTC ---
Patch posted.

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

--- Comment #16 from Rainer Orth <ro at gcc dot gnu.org> 2011-06-21 17:10:02 UTC ---
Author: ro
Date: Tue Jun 21 17:09:57 2011
New Revision: 175266

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=175266
Log:
    PR libgcj/49314
    * gnu/gcj/convert/natIconv.cc (iconv_init): Pad in[] with NUL byte.

Modified:
    trunk/libjava/ChangeLog
    trunk/libjava/gnu/gcj/convert/natIconv.cc

[Bug libgcj/49314] md5test, shatest output FAILs on Tru64 UNIX

[ro at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49314

Rainer Orth <ro at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #17 from Rainer Orth <ro at gcc dot gnu.org> 2011-06-21 17:12:26 UTC ---
Fixed for 4.7.0.