[GCJ-core] dynamically unloading native libraries

[GCJ-core] dynamically unloading native libraries

[Sylvain Marié]

Hi all,

I am working on a project involving GCJ on an embedded Linux busybox. 
We are particularly fond of the ability of GCJ to use java in "compiled mode".

In order to be able to use both native code (.so), compiled java (.so) and bytecode java 
(.jar, .class) together with a module loading system, we are running some tests to validate 
that dynamic code loading/unloading works fine.

Strangely enough this does not work on GCJ, whereas it works fine on sun's jvm (a native 
library is fully unloaded once its classloader is garbaged out).

see also : 
> openJDK, in jdk/src/share/classes/java/lang/ClassLoader.java: finalize method unloads native libraries


Is this feature planned to be inetgrated in next versions of the runtime ?
Thank you in advance for your support 
- congratulations for your amazing work so far ! 

Best regards


Sylvain

Sylvain MARIÉ
Schneider Electric DInnov, 
Software Engineering Technologies

Re: [GCJ-core] dynamically unloading native libraries

[Andrew Haley]

Sylvain Marié wrote:
> Hi all,
> 
> I am working on a project involving GCJ on an embedded Linux busybox. 
> We are particularly fond of the ability of GCJ to use java in "compiled mode".
> 
> In order to be able to use both native code (.so), compiled java (.so) and bytecode java 
> (.jar, .class) together with a module loading system, we are running some tests to validate 
> that dynamic code loading/unloading works fine.
> 
> Strangely enough this does not work on GCJ, whereas it works fine on sun's jvm (a native 
> library is fully unloaded once its classloader is garbaged out).
> 
> see also : 
>> openJDK, in jdk/src/share/classes/java/lang/ClassLoader.java: finalize method unloads native libraries
> 
> 
> Is this feature planned to be inetgrated in next versions of the runtime ?

No.  While this could be done, it would be quite tricky.

The core problem is that pointers to code (e.g. return addresses in
the stack) aren't recognized by the garbage collector as pointers to
the corresponding classes.  You'd have to teach the collector to
convert code addresses to classes, and then garbage collect the
classes.

Andrew.

Re: [GCJ-core] dynamically unloading native libraries

[Mark Wielaard]

Hi,

Responding to a pretty old email that I only saw just now for some
reason.

On Tue, 2008-10-07 at 13:41 +0100, Andrew Haley wrote:
> Sylvain Marié wrote:
> > I am working on a project involving GCJ on an embedded Linux busybox. 
> > We are particularly fond of the ability of GCJ to use java in "compiled mode".
> > 
> > In order to be able to use both native code (.so), compiled java (.so) and bytecode java 
> > (.jar, .class) together with a module loading system, we are running some tests to validate 
> > that dynamic code loading/unloading works fine.
> > 
> > Strangely enough this does not work on GCJ, whereas it works fine on sun's jvm (a native 
> > library is fully unloaded once its classloader is garbaged out).
> > 
> > see also : 
> >> openJDK, in jdk/src/share/classes/java/lang/ClassLoader.java: finalize method unloads native libraries
> > 
> > Is this feature planned to be inetgrated in next versions of the runtime ?
> 
> No.  While this could be done, it would be quite tricky.
> 
> The core problem is that pointers to code (e.g. return addresses in
> the stack) aren't recognized by the garbage collector as pointers to
> the corresponding classes.  You'd have to teach the collector to
> convert code addresses to classes, and then garbage collect the
> classes.

But couldn't/shouldn't we do it for JNI native code that was loaded
through the ClassLoader when that ClassLoader is garbage collected?
I believe that is what Sylvain is testing?
We seem to never even call JNI_OnUnload on the jni libraries. Which
seems to suggest we never ever try to unload them even when they are no
longer needed.

Cheers,

Mark

Re: [GCJ-core] dynamically unloading native libraries

[Andrew Haley]

Mark Wielaard wrote:
> 
> Responding to a pretty old email that I only saw just now for some
> reason.
> 
> On Tue, 2008-10-07 at 13:41 +0100, Andrew Haley wrote:
>> Sylvain Marié wrote:
>>> I am working on a project involving GCJ on an embedded Linux busybox. 
>>> We are particularly fond of the ability of GCJ to use java in "compiled mode".
>>>
>>> In order to be able to use both native code (.so), compiled java (.so) and bytecode java 
>>> (.jar, .class) together with a module loading system, we are running some tests to validate 
>>> that dynamic code loading/unloading works fine.
>>>
>>> Strangely enough this does not work on GCJ, whereas it works fine on sun's jvm (a native 
>>> library is fully unloaded once its classloader is garbaged out).
>>>
>>> see also : 
>>>> openJDK, in jdk/src/share/classes/java/lang/ClassLoader.java: finalize method unloads native libraries
>>> Is this feature planned to be inetgrated in next versions of the runtime ?
>> No.  While this could be done, it would be quite tricky.
>>
>> The core problem is that pointers to code (e.g. return addresses in
>> the stack) aren't recognized by the garbage collector as pointers to
>> the corresponding classes.  You'd have to teach the collector to
>> convert code addresses to classes, and then garbage collect the
>> classes.
> 
> But couldn't/shouldn't we do it for JNI native code that was loaded
> through the ClassLoader when that ClassLoader is garbage collected?

We could, but that isn't the subject of the question.  At least I didn't
think it was.

The answer to the original poster is that yes, it can be done, but it
would require some work.

> I believe that is what Sylvain is testing?

As far as I could see the question was about unloading of gcj-compiled
java code.  If the question is about JNI libraries, then yes, it still
requires some work but it should be easier.

> We seem to never even call JNI_OnUnload on the jni libraries. Which
> seems to suggest we never ever try to unload them even when they are no
> longer needed.

Sure, but making sure that in every case we can track all live references to
native code is not trivial.

Andrew.

Re: [GCJ-core] dynamically unloading native libraries

[Andrew Haley]

Sylvain Marié wrote:

> Thank you for putting the topic back on the table,
> 
>> We seem to never even call JNI_OnUnload on the jni libraries.
> 
> That's exactly my point - and also even libraries that do not contain JNI at all
> (simple .so without any relation to java - I know it is not a very useful case but it is
> very simple to test with a dummy .so).
> 
> But the root cause of the problem is a bit more complex: 
> 
> a) As you mentioned, ClassLoaders don't unload unused native libraries when they are finalized.
> 
> but also
> 
> b) ClassLoaders don't even seem to be finalized when they are not used any more.
> (no alive instances, etc.)
> 
> So even correcting a) would not help, we should first try to understand why 
> ClassLoaders are never gc'ed.

Yes.  I can't think of any immediate reason why they aren't: it's a matter
worth investigating.



Andrew.

Re: [GCJ-core] dynamically unloading native libraries

[Sylvain Marié]

Hi Mark,

Thank you for putting the topic back on the table,

> We seem to never even call JNI_OnUnload on the jni libraries.

That's exactly my point - and also even libraries that do not contain JNI at all
(simple .so without any relation to java - I know it is not a very useful case but it is
very simple to test with a dummy .so).

But the root cause of the problem is a bit more complex: 

a) As you mentioned, ClassLoaders don't unload unused native libraries when they are finalized.

but also

b) ClassLoaders don't even seem to be finalized when they are not used any more.
(no alive instances, etc.)

So even correcting a) would not help, we should first try to understand why 
ClassLoaders are never gc'ed.


Once again, this seems to be done in openJDK (at least item a, and maybe the b):

-------------------ClassLoader.java------------------------------------
protected void finalize() {
            synchronized (loadedLibraryNames) {
                if (fromClass.getClassLoader() != null && handle != 0) {
                    /* remove the native library name */
                    int size = loadedLibraryNames.size();
                    for (int i = 0; i < size; i++) {
                        if (name.equals(loadedLibraryNames.elementAt(i))) {
                            loadedLibraryNames.removeElementAt(i);
                            break;
                        }
                    }
                    /* unload the library. */
                    ClassLoader.nativeLibraryContext.push(this);
                    try {
                        unload();
                    } finally {
                        ClassLoader.nativeLibraryContext.pop();
                    }
                }
            }
        }
----------------------------------------------------------------------------------

Kind regards,


Sylvain


> ----- Message de Mark Wielaard <mark@klomp.org> sur Sun, 02 Nov 2008
> 18:43:12 +0100 -----
> 
> Pour:
> 
> Andrew Haley <aph@redhat.com>
> 
> cc:
> 
> Sylvain Marié <sylvainmarie2@yahoo.fr>, java@gcc.gnu.org
> 
> Objet:
> 
> Re: [GCJ-core] dynamically unloading native libraries
> 
> Hi,
> 
> Responding to a pretty old email that I only saw just now for some
> reason.
> 
> On Tue, 2008-10-07 at 13:41 +0100, Andrew Haley wrote:
> > Sylvain Marié wrote:
> > > I am working on a project involving GCJ on an embedded Linux busybox. 
> > > We are particularly fond of the ability of GCJ to use java in 
> "compiled mode".
> > > 
> > > In order to be able to use both native code (.so), compiled java
> (.so) and bytecode java 
> > > (.jar, .class) together with a module loading system, we are 
> running some tests to validate 
> > > that dynamic code loading/unloading works fine.
> > > 
> > > Strangely enough this does not work on GCJ, whereas it works 
> fine on sun's jvm (a native 
> > > library is fully unloaded once its classloader is garbaged out).
> > > 
> > > see also : 
> > >> openJDK, in jdk/src/share/classes/java/lang/ClassLoader.java: 
> finalize method unloads native libraries
> > > 
> > > Is this feature planned to be inetgrated in next versions of theruntime ?
> > 
> > No.  While this could be done, it would be quite tricky.
> > 
> > The core problem is that pointers to code (e.g. return addresses in
> > the stack) aren't recognized by the garbage collector as pointers to
> > the corresponding classes.  You'd have to teach the collector to
> > convert code addresses to classes, and then garbage collect the
> > classes.
> 
> But couldn't/shouldn't we do it for JNI native code that was loaded
> through the ClassLoader when that ClassLoader is garbage collected?
> I believe that is what Sylvain is testing?
> We seem to never even call JNI_OnUnload on the jni libraries. Which
> seems to suggest we never ever try to unload them even when they are no
> longer needed.
> 
> Cheers,
> 
> Mark
>