[PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[Dave Korn]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42811


    Hi lists,

  In this PR, ecj1 is failing (on Cygwin, presumably MinGW as well) owing to
unintended fallout from building libgcj as two separate DLLs.

  There are only static dependencies (i.e. unresolved symbols in the ecj.o
object) for ecjx to pull in the main libgcj at linktime, so only that DLL gets
loaded into memory at runtime.  But the class loader wants to load something
from the subsidiary (-noncore) DLL, and it's not present in memory, so it
fails.  (IIUC this is what the class definition data that is stored in .jcr
sections and made available to the class loader through a a .ctor-like
mechanism in the crtbegin code is all about.)

  In this patch, I add an arbitrary undef to the link spec, against a fairly
randomly chosen symbol (my only criterion was that the org.ietf.jgss namespace
is probably fairly stable) from the -noncore part of the library.  This
sufficed to build an ecjx that links against and hence loads both DLLs, and it
doesn't throw the exception shown in the PR anymore.

libjava/ChangeLog:

	PR java/42811
	* configure.ac (libgcj_spec_lgcj_override): Add undefined reference
	to arbitrary symbol in noncore library.
	(libgcj_spec_lgcj_bc_override): Likewise.

  I'm taking this for a bootstrap-and-test cycle on i686-pc-cygwin.  Does it
seem like a reasonable approach?

    cheers,
      DaveK


[-- Attachment #2: pr42811.diff --]
[-- Type: text/x-c, Size: 1104 bytes --]

Index: libjava/configure.ac
===================================================================
--- libjava/configure.ac	(revision 156467)
+++ libjava/configure.ac	(working copy)
@@ -244,9 +244,13 @@ if test "$enable_libgcj_sublibs" = yes ; then
   # tag "_bc" onto the end of it when we use it to generate the
   # spec, but that's an ugly thing to do when there are multiple
   # words in the string and you're relying on the ordering to
-  # append the correct one.
-  libgcj_spec_lgcj_override="-lgcj-noncore -lgcj"
-  libgcj_spec_lgcj_bc_override="-lgcj-noncore -lgcj_bc"
+  # append the correct one.  Note that we add an arbitrary undef
+  # from libgcj-noncore to ensure the library is actually linked
+  # and loaded, as there aren't static dependencies to tell us
+  # which classes will be loaded at runtime, so we need to ensure
+  # that all the class data (.jcr sections) are present and initialised.
+  libgcj_spec_lgcj_override="-lgcj-noncore -u__ZN3org4ietf4jgss10GSSManagerC1Ev -lgcj"
+  libgcj_spec_lgcj_bc_override="-lgcj-noncore -u__ZN3org4ietf4jgss10GSSManagerC1Ev -lgcj_bc"
 fi
 
 

Re: [PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[Dave Korn]

On 05/02/2010 22:26, Dave Korn wrote:

> libjava/ChangeLog:
> 
> 	PR java/42811
> 	* configure.ac (libgcj_spec_lgcj_override): Add undefined reference
> 	to arbitrary symbol in noncore library.
> 	(libgcj_spec_lgcj_bc_override): Likewise.
> 
>   I'm taking this for a bootstrap-and-test cycle on i686-pc-cygwin.  Does it
> seem like a reasonable approach?

  It caused a regression:

> FAIL: events output

  For some reason, a couple of extra jvm event callbacks appeared:

> $ diff -pu events.out .e.o
> --- events.out  2009-03-14 06:47:02.984375000 +0000
> +++ .e.o        2010-02-06 06:04:50.218750000 +0000
> @@ -4,9 +4,9 @@ created JVMTI environment #0
>  created JVMTI environment #1
>  created JVMTI environment #2
>  setting callbacks for envs
> -RequestedEvents:
> +RequestedEvents: VMInit,VMDeath,
>  enable VM_INIT for env0, env1, env2
> -RequestedEvents: VMInit,
> +RequestedEvents: VMInit,VMDeath,
>  enable VM_DEATH for env1,env2
>  RequestedEvents: VMInit,VMDeath,
>  enable THREAD_END for env2

  There must be something in the noncore library that registers callbacks in
addition to what the test was expecting.  But then I'm not sure why they
wouldn't also be registered if the full non-partitioned libgcj was in use.
Until I can figure this out, I guess I don't understand how dependencies work
in java well enough, so patch withdrawn/on hold.

    cheers,
      DaveK

[REVIVED: PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[Dave Korn]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

On 06/02/2010 06:29, Dave Korn wrote:
> On 05/02/2010 22:26, Dave Korn wrote:
> 
>> libjava/ChangeLog:
>>
>> 	PR java/42811
>> 	* configure.ac (libgcj_spec_lgcj_override): Add undefined reference
>> 	to arbitrary symbol in noncore library.
>> 	(libgcj_spec_lgcj_bc_override): Likewise.
>>
>>   I'm taking this for a bootstrap-and-test cycle on i686-pc-cygwin.  Does it
>> seem like a reasonable approach?
> 
>   It caused a regression:
> 
>> FAIL: events output

  I've now tracked this regression down to a use of uninitialised memory in
the core vm code(*), which appears to be entirely unrelated to (and presumably
entirely accidentally exposed by) this patch.  So, I'm reviving it.  I'm
running a full testsuite now for the combined patch, but I've already verified
this regression no longer appears.

libjava/ChangeLog:

	PR java/42811
	* configure.ac (libgcj_spec_lgcj_override): Add undefined reference
	to arbitrary symbol in noncore library.
	(libgcj_spec_lgcj_bc_override): Likewise.
	configure: Regenerate.
	* jvmti.cc (_Jv_GetJVMTIEnv): Fix use of uninitialised memory
	exposed by the above change.

  Bootstrapped on i686-pc-cygwin; currently running check-target-libjava only.
 Assuming no regressions this time, OK for head?

    cheers,
      DaveK
-- 
(*) - http://gcc.gnu.org/ml/java/2010-02/msg00002.html


[-- Attachment #2: pr42811-take2.diff --]
[-- Type: text/x-c, Size: 1607 bytes --]

Index: libjava/configure.ac
===================================================================
--- libjava/configure.ac	(revision 156730)
+++ libjava/configure.ac	(working copy)
@@ -244,9 +244,13 @@ if test "$enable_libgcj_sublibs" = yes ; then
   # tag "_bc" onto the end of it when we use it to generate the
   # spec, but that's an ugly thing to do when there are multiple
   # words in the string and you're relying on the ordering to
-  # append the correct one.
-  libgcj_spec_lgcj_override="-lgcj-noncore -lgcj"
-  libgcj_spec_lgcj_bc_override="-lgcj-noncore -lgcj_bc"
+  # append the correct one.  Note that we add an arbitrary undef
+  # from libgcj-noncore to ensure the library is actually linked
+  # and loaded, as there aren't static dependencies to tell us
+  # which classes will be loaded at runtime, so we need to ensure
+  # that all the class data (.jcr sections) are present and initialised.
+  libgcj_spec_lgcj_override="-lgcj-noncore -u__ZN3org4ietf4jgss10GSSManagerC1Ev -lgcj"
+  libgcj_spec_lgcj_bc_override="-lgcj-noncore -u__ZN3org4ietf4jgss10GSSManagerC1Ev -lgcj_bc"
 fi
 
 
Index: libjava/jvmti.cc
===================================================================
--- libjava/jvmti.cc	(revision 156730)
+++ libjava/jvmti.cc	(working copy)
@@ -2210,6 +2210,7 @@ _Jv_GetJVMTIEnv (void)
 {
   _Jv_JVMTIEnv *env
     = (_Jv_JVMTIEnv *) _Jv_MallocUnchecked (sizeof (_Jv_JVMTIEnv));
+  memset (env, 0, sizeof (_Jv_JVMTIEnv));
   env->p = &_Jv_JVMTI_Interface;
   struct jvmti_env_list *element
     = (struct jvmti_env_list *) _Jv_MallocUnchecked (sizeof (struct jvmti_env_list));

Re: [REVIVED: PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[Bryce McKinlay]

On Fri, Feb 5, 2010 at 10:26 PM, Dave Korn
<dave.korn.cygwin@googlemail.com> wrote:
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42811
>
>
>    Hi lists,
>
>  In this PR, ecj1 is failing (on Cygwin, presumably MinGW as well) owing to
> unintended fallout from building libgcj as two separate DLLs.
>
>  There are only static dependencies (i.e. unresolved symbols in the ecj.o
> object) for ecjx to pull in the main libgcj at linktime, so only that DLL gets
> loaded into memory at runtime.  But the class loader wants to load something
> from the subsidiary (-noncore) DLL, and it's not present in memory, so it
> fails.  (IIUC this is what the class definition data that is stored in .jcr
> sections and made available to the class loader through a a .ctor-like
> mechanism in the crtbegin code is all about.)
>
>  In this patch, I add an arbitrary undef to the link spec, against a fairly
> randomly chosen symbol (my only criterion was that the org.ietf.jgss namespace
> is probably fairly stable) from the -noncore part of the library.  This
> sufficed to build an ecjx that links against and hence loads both DLLs, and it
> doesn't throw the exception shown in the PR anymore.

> libjava/ChangeLog:
>
>        PR java/42811
>        * configure.ac (libgcj_spec_lgcj_override): Add undefined reference
>        to arbitrary symbol in noncore library.

Hi Dave

I'm not sure if this sounds like the right fix. Isn't it the point of
libgcj-noncore that applications which only want "core" classes can
link a smaller libgcj without having to distribute or load the
"noncore" classes? Doesn't adding a dependency from libgcj to
libgcj-noncore defeat this?

If ecj needs something from -noncore, shouldn't it be ecj that
explicitly links it?

>        * jvmti.cc (_Jv_GetJVMTIEnv): Fix use of uninitialised memory
>        exposed by the above change.

This part is fine.

Bryce

Re: [REVIVED: PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[Dave Korn]

On 15/02/2010 13:04, Bryce McKinlay wrote:
> On Fri, Feb 5, 2010 at 10:26 PM, Dave Korn wrote:
>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42811

>>  In this PR, ecj1 is failing (on Cygwin, presumably MinGW as well) owing to
>> unintended fallout from building libgcj as two separate DLLs.
>>
>>  There are only static dependencies (i.e. unresolved symbols in the ecj.o
>> object) for ecjx to pull in the main libgcj at linktime, so only that DLL gets
>> loaded into memory at runtime.  But the class loader wants to load something
>> from the subsidiary (-noncore) DLL, and it's not present in memory, so it
>> fails.  (IIUC this is what the class definition data that is stored in .jcr
>> sections and made available to the class loader through a a .ctor-like
>> mechanism in the crtbegin code is all about.)
>>
>>  In this patch, I add an arbitrary undef to the link spec, against a fairly
>> randomly chosen symbol (my only criterion was that the org.ietf.jgss namespace
>> is probably fairly stable) from the -noncore part of the library.  This
>> sufficed to build an ecjx that links against and hence loads both DLLs, and it
>> doesn't throw the exception shown in the PR anymore.
> 
>> libjava/ChangeLog:
>>
>>        PR java/42811
>>        * configure.ac (libgcj_spec_lgcj_override): Add undefined reference
>>        to arbitrary symbol in noncore library.
> 
> Hi Dave

  Hi Bryce, thanks for getting back to me on this one.

> I'm not sure if this sounds like the right fix. Isn't it the point of
> libgcj-noncore that applications which only want "core" classes can
> link a smaller libgcj without having to distribute or load the
> "noncore" classes? Doesn't adding a dependency from libgcj to
> libgcj-noncore defeat this?

  Yes, it does defeat that, but no, it's not the point.  The real driving
motivation behind breaking the library up into two parts is that it's simply
too big and exceeds system limits when compiled as a monolithic shared library
on some systems; specifically, it has ~82000 exported functions, and a windows
DLL can export no more than 65535.

  It would certainly be a nice benefit if it was possible to load just one or
the other individually, but it's only a nice-to-have; the subdivision itself
is necessary simply for correctness on the affected targets.

> If ecj needs something from -noncore, shouldn't it be ecj that
> explicitly links it?

  Apologies in advance if I'm about to say something dumb, I'm no java
specialist, but AIUI ecj (and other java apps) don't have to explicitly link
against "things they need" in general, they call the class loader and it gets
them either from a jar file on disk or by instantiating one of the built-in
classes that were recorded in the .jcr section, right?  Any app will assume
that it can classload all of the standard library classes without having to do
anything special, and I want it so that the subdivided library transparently
behaves the same as the monoloithic one.

  So, nice though it might be, I just can't see any way of avoiding the
necessity to load both libraries all the time everywhere (in order that all
the .jcr sections get loaded and so all the expected classes are registered
with the classloader), and that's what the effect of this patch should be.


>>        * jvmti.cc (_Jv_GetJVMTIEnv): Fix use of uninitialised memory
>>        exposed by the above change.
> 
> This part is fine.

  Well: it appears to have caused a regression (investigating which is the
reason why I took a while to respond to you).  I'm now seeing:

> Running /gnu/gcc/gcc/libjava/testsuite/libjava.loader/loader.exp ...
> FAIL: /gnu/gcc/obj-pr42811-3/i686-pc-cygwin/libjava/testsuite/TestEarlyGC.exe output - /gnu/gcc/obj-pr42811-3/i686-pc-cygwin/libjava/testsuite/TestEarlyGC.exe

*sigh*, and it turns out to be because some memory is getting released by the
garbage collector when its still in use.  I'm busy trying to learn how the GC
works real fast so I can figure out what's going wrong.  I was surprised to
discover that the test executable links its own entirely separate copy of the
GC library in addition to the one that's in the libjava DLL(s), is that
supposed to happen?

    cheers,
      DaveK

Re: [REVIVED: PATCH PR42811,4.5 regression] java.lang.ExceptionInInitializerError in ecj1

[Dave Korn]

On 18/02/2010 04:46, Dave Korn wrote:

>>>        * jvmti.cc (_Jv_GetJVMTIEnv): Fix use of uninitialised memory
>>>        exposed by the above change.
>> This part is fine.
> 
>   Well: it appears to have caused a regression (investigating which is the
> reason why I took a while to respond to you).  I'm now seeing:
> 
>> Running /gnu/gcc/gcc/libjava/testsuite/libjava.loader/loader.exp ...
>> FAIL: /gnu/gcc/obj-pr42811-3/i686-pc-cygwin/libjava/testsuite/TestEarlyGC.exe output - /gnu/gcc/obj-pr42811-3/i686-pc-cygwin/libjava/testsuite/TestEarlyGC.exe
> 
> *sigh*, and it turns out to be because some memory is getting released by the
> garbage collector when its still in use.  I'm busy trying to learn how the GC
> works real fast so I can figure out what's going wrong.

  I'm now very sure that the problem is a side-effect of the libgcj DLL being
rebased at runtime, and not related to the clearing of memory in
_Jv_GetJVMTIEnv (which isn't even called in this testcase).  So I'd like to go
ahead with this part of the patch after all.  I'll spend up to another 24
hours trying to fix the knock-on failure before I commit it, it would be nice
to have the fix for that ready to go in at the same time.  That still OK with
you?  (Also, did my explanation for the other part of the patch make sense, or
do I need to go into more detail about anything?)

>  I was surprised to
> discover that the test executable links its own entirely separate copy of the
> GC library in addition to the one that's in the libjava DLL(s), is that
> supposed to happen?

  I was wrong about this, it was a red herring caused by the fact that libgcj
was getting rebased down to a runtime address so low that I thought it was in
the EXE's text section; in fact it was not.  I've now proven that by rebasing
all the other DLLs out of the way I can make the test pass or fail just by
moving around where libgcj gets loaded in the process memory space, and it
fails in different ways according to where I move it to, which is symptomatic
of a problem with base relocations in the generated DLL.  I'm going to look at
how the GC determines the extent of the DLL's data area when it looks for
pointers during the mark phase, because if the static data region ranges
aren't being fixed up correctly when the DLL gets rebased to a different load
address at runtime, that could cause this kind of problem.

    cheers,
      DaveK