Dynamic Loading of Class Implementation at Runtime.

Dynamic Loading of Class Implementation at Runtime.

[Benjamin Scherrey]

    I've seen messages in the past that allude to this subject and I've
searched long and hard for a (somewhat) platform independent mechanism
to implement this feature but with no success. So... I'm in need of
information on how to load, at runtime, from a shared library, the
implementation of a class (or its subclass) and then be able to unload
it and load a new implementation.

    This requires the ability to resolve virtual and non-virtual member
functions and not just C-type functions which is what has been the big
problem. My platforms of interest are x86/Linux, SPARC/Solaris 2.6, and
HP9000/HP/Ux. This is using egcs1.1.1 G++ and later as they come
available. I understand that different OS's will likely have different
calls to get at the shared libraries but my main point of interest is
the resolution of these mangled symbols at runtime.

    Appreciate all pointers and information,

        Ben Scherrey

Re: Dynamic Loading of Class Implementation at Runtime.

[Alexandre Oliva]

On Dec 10, 1998, Benjamin Scherrey <scherrey@gte.net> wrote:

> information on how to load, at runtime, from a shared library, the
> implementation of a class (or its subclass) and then be able to unload
> it and load a new implementation.

I don't quite understand.  There's no important difference between the
implementation of C functions and C++ methods, from the point of view
of a shared library.  The only relevant difference is that C++
libraries may contain dynamic initialization, and destructors of
dynamically initialized objects won't be run when the library is
unloaded, only when the program terminates (and thus the program will
probably crash :-(

-- 
Alexandre Oliva  http://www.dcc.unicamp.br/~oliva  aoliva@{acm.org}
oliva@{dcc.unicamp.br,gnu.org,egcs.cygnus.com,samba.org}
Universidade Estadual de Campinas, SP, Brasil

Re: Dynamic Loading of Class Implementation at Runtime.

[Ruslan Shevchenko]

Alexandre Oliva wrote:

> On Dec 10, 1998, Benjamin Scherrey <scherrey@gte.net> wrote:
>
> > information on how to load, at runtime, from a shared library, the
> > implementation of a class (or its subclass) and then be able to unload
> > it and load a new implementation.
>
> I don't quite understand.  There's no important difference between the
> implementation of C functions and C++ methods, from the point of view
> of a shared library.  The only relevant difference is that C++
>

  mangling.

   in C I know, that object symbol name is '_' + C-source name.

 in C++ -- not.


> libraries may contain dynamic initialization, and destructors of
> dynamically initialized objects won't be run when the library is
> unloaded, only when the program terminates (and thus the program will
> probably crash :-(
>
> --
> Alexandre Oliva  http://www.dcc.unicamp.br/~oliva  aoliva@{acm.org}
> oliva@{dcc.unicamp.br,gnu.org,egcs.cygnus.com,samba.org}
> Universidade Estadual de Campinas, SP, Brasil

Re: Dynamic Loading of Class Implementation at Runtime.

[Alexandre Oliva]

On Dec 10, 1998, Ruslan Shevchenko <Ruslan@Shevchenko.Kiev.UA> wrote:

>> I don't quite understand.  There's no important difference between the
>> implementation of C functions and C++ methods, from the point of view
>> of a shared library.  The only relevant difference is that C++

>   mangling.

>    in C I know, that object symbol name is '_' + C-source name.

Do you?  Can you enumerate on which platforms every symbol starts with 
a `_'?

>  in C++ -- not.

I think the only portable solution is to define wrappers with "C"
linkage for interfacing with the library.  One such wrapper could be
used to access containers of Functors or pointers to member functions
with "C" linkage, and use them as starting points for searching C++
methods in the library.

Other than that, you might use the mangling/unmangling functions that
have been recently integrated into libiberty (have they?) to get
symbol names, but this will be restricted to GNU C++.

-- 
Alexandre Oliva  http://www.dcc.unicamp.br/~oliva  aoliva@{acm.org}
oliva@{dcc.unicamp.br,gnu.org,egcs.cygnus.com,samba.org}
Universidade Estadual de Campinas, SP, Brasil

Re: Dynamic Loading of Class Implementation at Runtime.

[Mumit Khan]

Alexandre Oliva <oliva@dcc.unicamp.br> writes:
> 
> Other than that, you might use the mangling/unmangling functions that
> have been recently integrated into libiberty (have they?) to get
> symbol names, but this will be restricted to GNU C++.

You should be able to get the mangling algorithm for most vendor
compilers, with the exception of perhaps the windows-based ones. I've
tried in vain to get this info out of Microsoft.

Currently, the mangling used in GNU C++ (obviously ;-), EDG and a few 
others are fairly well known (albeit perhaps not well-documented), so 
you're set if those are the compilers you support.  Since we currently 
only support EDG-based compilers and EGCS, the issue is quite simple 
for us, but not if you need to support all the other compilers.

I however recommend an object factory method (via an extern 'C'
interface) that avoids this whole issue. 

Regards,
Mumit

Re: Dynamic Loading of Class Implementation at Runtime.

[Marat Boshernitsan]

Well, I do it all the time :) My solution is to have an abstract base
class that is compiled together with the application and completely
describes the interface to the class to be loaded.  The shared library
subclasses that class, defining all those abstract methods and provides
a single global extern "C" function for creating instances of the
subclass.  When the shared library is dlopen()'ed, the creator function
is looked up and is used for producing instances of library class that
are then cast into the base class type and treated the usual way.

HTH,

Marat.


Benjamin Scherrey <scherrey@gte.net> writes:

>     I've seen messages in the past that allude to this subject and I've
> searched long and hard for a (somewhat) platform independent mechanism
> to implement this feature but with no success. So... I'm in need of
> information on how to load, at runtime, from a shared library, the
> implementation of a class (or its subclass) and then be able to unload
> it and load a new implementation.
> 
>     This requires the ability to resolve virtual and non-virtual member
> functions and not just C-type functions which is what has been the big
> problem. My platforms of interest are x86/Linux, SPARC/Solaris 2.6, and
> HP9000/HP/Ux. This is using egcs1.1.1 G++ and later as they come
> available. I understand that different OS's will likely have different
> calls to get at the shared libraries but my main point of interest is
> the resolution of these mangled symbols at runtime.
> 
>     Appreciate all pointers and information,
> 
>         Ben Scherrey

Re: Dynamic Loading of Class Implementation at Runtime.

[Joe Buck]

>     I've seen messages in the past that allude to this subject and I've
> searched long and hard for a (somewhat) platform independent mechanism
> to implement this feature but with no success. So... I'm in need of
> information on how to load, at runtime, from a shared library, the
> implementation of a class (or its subclass) and then be able to unload
> it and load a new implementation.

We did that in the Ptolemy project since 1991 or so
(URL:< http://ptolemy.eecs.berkeley.edu/ >).

>     This requires the ability to resolve virtual and non-virtual member
> functions and not just C-type functions which is what has been the big
> problem.

The scheme we used is roughly the object factory pattern (though since we
were doing it before the books were written the terminology might not
match up right).  You can load and unload new classes, provided that they
are derived from a class that is already in the executable and that this
pre-existing class has a clone method (which serves as a "virtual
constructor").  The trick works either with shared libraries or the old
"ld -A" stuff from Berkeley Unix.

The key is that when a shared library is loaded, the constructors for
global and static objects are called.  One or more of these registers
an instance of each new class in the registry (alternatively, you can
register an object that can create an object of the new class).  Then,
by saying something like Registry::makeNew(className) you construct
objects of the new class.

To determine when it is safe to unmap the shared library you'll need
some kind of reference counting or garbage collection, to know that
there are objects left that reference the code.

Note that using this scheme you can only call virtual methods that
override virtual methods of the base class that was pre-loaded.
But with a suitable class design that can be powerful enough.

Re: Dynamic Loading of Class Implementation at Runtime.

[Martin von Loewis]

> I don't quite understand.  There's no important difference between the
> implementation of C functions and C++ methods, from the point of view
> of a shared library.

There is another use of shared libraries, other than just sharing the
library. That other use is dynamic code loading. 

In C, you can very nicely have a long-running system, and replace the
implementation of some functions on-the-fly. You can add and remove
functions, and change the implementation of existing functions. It
will work as long as you don't change the signature of existing
functions.

In C++, there are much more hidden restrictions. You must not add
virtual functions, and you must not add members to classes, and you
better not have globals in your shared library. This essentially means
that you cannot modify your implementation in an interesting way
without breaking binary compatibility. Apparently, this is an issue
to some people.

Regards,
Martin

Re: Dynamic Loading of Class Implementation at Runtime

[Martin Reinecke]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Hi!

Some time ago, I have written a short piece of code which does exactly
what Joe Buck described. (I didn't know anything about
Ptolemy, though). The fine thing about the
method is that no call to dlsym() is needed. I have attached a short example
of this technique, which works on linux-i586-gnulibc1 with egcs-1.0.3
(and maybe later) and binutils 2.9.1. Alternatively, you can get the code
at http://www.mpa-garching.mpg.de/~martin/dlc.tar.gz .

I'd really like to know how portable that sort of code is. I suspect porting
should be fairly easy to other ELF platforms, but I don't know about other
platforms and have no means to test it.
If it *is* portable, this would be the easiest method to dynamically load
classes at runtime, IMHO.

Any feedback is greatly welcome.

Martin Reinecke (martin@mpa-garching.mpg.de)

P.S: Please CC to me, since I'm not on the list.
dlc.tar.gz


[-- Attachment #2: dlc.tar.gz --]
[-- Type: application/x-gzip, Size: 2068 bytes --]

RE: Dynamic Loading of Class Implementation at Runtime.

[Daniel Berlin]

> 
> You should be able to get the mangling algorithm for most vendor
> compilers, with the exception of perhaps the windows-based ones. I've
> tried in vain to get this info out of Microsoft.

I reverse engineered it, and never got around to sending you the doc i wrote
up on it (sorry bout that).
Then BeOS moved to ELF/DWARF2, so i had no need for demangling the stupid
names in the broken codeview i had to deal with originally to get gdb going
on BeOS.
I still have it around somewhere if you want it.
It's fairly easy to understand.
If you want official docs, well, can't. I wouldn't even know where to look.
Metrowerks uses the same algorithm for their windows compilers, but i don't
know if you'd have better luck with them.

--Dan
> 
> Regards,
> Mumit
> 
> 

Re: Dynamic Loading of Class Implementation at Runtime.

[Mike Stump]

> From: Alexandre Oliva <oliva@dcc.unicamp.br>
> Date: 10 Dec 1998 14:41:10 -0200

> I think the only portable solution is to define wrappers with "C"
> linkage for interfacing with the library.

Well, to be more constructive than this, why don't you design an
access system to get at the information you want and need, and
document it, and post it here.  In time, maybe someone will implement
it.  Without such a document, the chances are even lower it will ever
be done.