dlclose() doesn't unload any .so that uses Boost

dlclose() doesn't unload any .so that uses Boost

[J]

Hi.

First, I would like to apologize if this isn't in any way related to
GCC, however I don't know anywhere else where I can turn to. Please
consider the following code:

// host.cpp
// compile with: g++ -ldl host.cpp -o host
#include <stdio.h>
#include <dlfcn.h>

int main( int argc, char ** argv )
{
    printf( "host: Loading libchild.so...\n" );
    void * so = dlopen( "./libchild.so", RTLD_LOCAL | RTLD_NOW );
    printf( "host: so = %p\n", so );

    if( so == 0 ) return 1;

    printf( "host: Unloading libchild.so...\n" );
    if( dlclose( so ) != 0 )
    {
        printf( "host: Error: %s\n", dlerror() );
        return 1;
    }
    printf( "host: Unloaded.\n" );
    printf( "host: %p\n", dlopen( "./libchild.so", RTLD_NOLOAD ) );

    return 0;
}

// child.cpp
// compile with: g++ -lboost_signals child.cpp -shared -fPIC -o libchild.so
#include <stdio.h>
#include <boost/signals.hpp>

struct foobar_t
{
    boost :: signal< void () > signal;
};

void  __attribute__ ( ( constructor ) ) ctor()
{
    foobar_t foobar;

    printf( "child: Constructor\n" );
}

void  __attribute__ ( ( destructor ) ) dtor()
{
    printf( "child: Destructor\n" );
}


This is the output I get:

host: Loading libchild.so...
child: Constructor
host: so = 0x94b6020
host: Unloading libchild.so...
host: Unloaded.
host: (nil)
child: Destructor

As you can see, dlclose() doesn't properly unload the .so, even though
it claims that it has done so. If you'll remove this line:

boost :: signal< void () > signal;

it works properly:

host: Loading libchild.so...
child: Constructor
host: so = 0x8f6f020
host: Unloading libchild.so...
child: Destructor
host: Unloaded.
host: (nil)

It took me a while to track this down. Now, my question is - is this a
bug? If so, in what? In GCC? (I tried this with both 4.5 and 4.6) In
the dynamic linker? In Boost? If not - why? And how would I fix this?
Any insight in greatly appreciated.

Thanks.

Re: dlclose() doesn't unload any .so that uses Boost

[David Hagood]

On Sun, 2011-05-29 at 01:32 +0200, J wrote:
> // compile with: g++ -ldl host.cpp -o host
<snip>
> // compile with: g++ -lboost_signals child.cpp -shared -fPIC -o libchild.so

Just for grins, what happens if you link main against Boost?
g++ -ldl host.cpp -o host -lboost_signals

It may be that since Boost is being pulled in to satisfy the child,
somehow that is preventing the child from being unloaded.

Of course, this isn't really a GCC question, but a libc/binutils
question, so you may do better over on that list.

Re: dlclose() doesn't unload any .so that uses Boost

[J]

On Sun, May 29, 2011 at 1:51 PM, David Hagood <david.hagood@gmail.com> wrote:
> Just for grins, what happens if you link main against Boost?
> g++ -ldl host.cpp -o host -lboost_signals

Already tried that. It doesn't make any difference.

> Of course, this isn't really a GCC question, but a libc/binutils
> question, so you may do better over on that list.

Well, as this looks pretty much like voodoo to me (tracking it down
was pretty fun) so I though it *might* be an issue with the code GCC
generates, but if you say so then I'll try asking over there.

Thanks.

Re: dlclose() doesn't unload any .so that uses Boost

[Pavel Tolkachev]

--- On Sun, 5/29/11, J <j@jabster.pl> wrote:

> From: J <j@jabster.pl>
> Subject: dlclose() doesn't unload any .so that uses Boost
> To: gcc-help@gcc.gnu.org
> Date: Sunday, May 29, 2011, 3:32 AM
> Hi.
> 
> First, I would like to apologize if this isn't in any way
> related to
> GCC, however I don't know anywhere else where I can turn
> to. Please
> consider the following code:
> 
> // host.cpp
> // compile with: g++ -ldl host.cpp -o host
> #include <stdio.h>
> #include <dlfcn.h>
> 
> int main( int argc, char ** argv )
> {
>     printf( "host: Loading libchild.so...\n" );
>     void * so = dlopen( "./libchild.so",
> RTLD_LOCAL | RTLD_NOW );
>     printf( "host: so = %p\n", so );
> 
>     if( so == 0 ) return 1;
> 
>     printf( "host: Unloading libchild.so...\n"
> );
>     if( dlclose( so ) != 0 )
>     {
>         printf( "host: Error: %s\n",
> dlerror() );
>         return 1;
>     }
>     printf( "host: Unloaded.\n" );
>     printf( "host: %p\n", dlopen(
> "./libchild.so", RTLD_NOLOAD ) );
> 
>     return 0;
> }
> 
> // child.cpp
> // compile with: g++ -lboost_signals child.cpp -shared
> -fPIC -o libchild.so
> #include <stdio.h>
> #include <boost/signals.hpp>
> 
> struct foobar_t
> {
>     boost :: signal< void () > signal;
> };
> 
> void  __attribute__ ( ( constructor ) ) ctor()
> {
>     foobar_t foobar;
> 
>     printf( "child: Constructor\n" );
> }
> 
> void  __attribute__ ( ( destructor ) ) dtor()
> {
>     printf( "child: Destructor\n" );
> }
> 
> 
> This is the output I get:
> 
> host: Loading libchild.so...
> child: Constructor
> host: so = 0x94b6020
> host: Unloading libchild.so...
> host: Unloaded.
> host: (nil)
> child: Destructor
> 
> As you can see, dlclose() doesn't properly unload the .so,
> even though
> it claims that it has done so. 
It actually doesn't. It just confirms there were one or more references counts to the library and it decreased their count by 1 (see man dlclose). I think you can achieve same effect by defining any global object with non-trivial destructor in the library, not necessarily a boost::signal. 

On remediation side, if you are comfortable with the client code's explicitly calling dlclose() anyway, why don't you allocate your signal object in free storage via operator new and call something like MyDlclose() (or childClose()) instead of it where you would first delete this object (and all others of the kind) and then call dlclose()? This won't increase the degree of client code cooperation needed to use your library.


> If you'll remove this line:
> 
> boost :: signal< void () > signal;
> 
> it works properly:
> 
> host: Loading libchild.so...
> child: Constructor
> host: so = 0x8f6f020
> host: Unloading libchild.so...
> child: Destructor
> host: Unloaded.
> host: (nil)
> 
> It took me a while to track this down. Now, my question is
> - is this a
> bug? If so, in what? In GCC? (I tried this with both 4.5
> and 4.6) In
> the dynamic linker? In Boost? If not - why? And how would I
> fix this?
> Any insight in greatly appreciated.
> 
> Thanks.
> 

Hope this helps,
-Pavel

Re: dlclose() doesn't unload any .so that uses Boost

[J]

On Mon, May 30, 2011 at 9:12 PM, Pavel Tolkachev <paultolk@yahoo.com> wrote:
> It actually doesn't. It just confirms there were one or more references counts to the library and it decreased their count by 1 (see man dlclose).

Yes, I've read man dlclose:

"If the same library is loaded again with dlopen(), the same file
handle is returned. The dl library maintains reference counts for
library handles, so a dynamic library is not deallocated until
dlclose() has been called on it as many times as dlopen() has
succeeded on it."

I'm only calling dlopen() once and boost doesn't call dlopen() at all,
so following this manpage my dlclose() should close the library. If
the refcount can be increased by something else then maybe this
manpage needs updating?

Besides, RTLD_NOLOAD paragraph states this:

"This can be used to test if the library is already resident (dlopen()
returns NULL if it is not, or the library's handle if it is
resident)."

While library is clearly still resident dlopen( RTLD_NOLOAD ) returns
NULL; another bug either in the code or in the manpage.

> On remediation side, if you are comfortable with the client code's explicitly calling dlclose() anyway, why don't you allocate your signal object in free storage via operator new and call something like MyDlclose() (or childClose()) instead of it where you would first delete this object (and all others of the kind) and then call dlclose()? This won't increase the degree of client code cooperation needed to use your library.

Unfortunately, this doesn't work, e.g. this is enough to make a .so
impossible unload:

delete ( new boost :: signal< void () > );

Oh and, I've just noticed that this code doesn't really even have to
run. Consider:

// child.cpp
// compile with: g++ -lboost_signals child.cpp -shared -fPIC -o libchild.so
#include <stdio.h>
#include <boost/signals.hpp>

void  __attribute__ ( ( constructor ) ) ctor()
{
   printf( "child: Constructor\n" );
}

void  __attribute__ ( ( destructor ) ) dtor()
{
   printf( "child: Destructor\n" );
}

void a_function_that_does_not_get_called()
{
    new boost :: signal< void () >();
}

The result:

host: Loading libchild.so...
child: Constructor
host: so = 0x8938020
host: Unloading libchild.so...
host: Unloaded.
host: (nil)
child: Destructor

This pretty much looks like a bug to me.

Thanks.

Re: dlclose() doesn't unload any .so that uses Boost

[Pavel Tolkachev]

--- On Mon, 5/30/11, J <j@jabster.pl> wrote:

> From: J <j@jabster.pl>
> Subject: Re: dlclose() doesn't unload any .so that uses Boost
> To: paulTolk@yahoo.com
> Cc: gcc-help@gcc.gnu.org
> Date: Monday, May 30, 2011, 11:56 PM
> On Mon, May 30, 2011 at 9:12 PM,
> Pavel Tolkachev <paultolk@yahoo.com>
> wrote:
> > It actually doesn't. It just confirms there were one
> or more references counts to the library and it decreased
> their count by 1 (see man dlclose).
> 
> Yes, I've read man dlclose:
> 
> "If the same library is loaded again with dlopen(), the
> same file
> handle is returned. The dl library maintains reference
> counts for
> library handles, so a dynamic library is not deallocated
> until
> dlclose() has been called on it as many times as dlopen()
> has
> succeeded on it."

> 
> I'm only calling dlopen() once and boost doesn't call
> dlopen() at all,
> so following this manpage my dlclose() should close the
> library. If
> the refcount can be increased by something else then maybe
> this
> manpage needs updating?
> 
> Besides, RTLD_NOLOAD paragraph states this:
> 
> "This can be used to test if the library is already
> resident (dlopen()
> returns NULL if it is not, or the library's handle if it
> is
> resident)."
> 
> While library is clearly still resident dlopen( RTLD_NOLOAD
> ) returns
> NULL; another bug either in the code or in the manpage.
> 
> > On remediation side, if you are comfortable with the
> client code's explicitly calling dlclose() anyway, why don't
> you allocate your signal object in free storage via operator
> new and call something like MyDlclose() (or childClose())
> instead of it where you would first delete this object (and
> all others of the kind) and then call dlclose()? This won't
> increase the degree of client code cooperation needed to use
> your library.
> 
> Unfortunately, this doesn't work, e.g. this is enough to
> make a .so
> impossible unload:
> 
> delete ( new boost :: signal< void () > );
> 
> Oh and, I've just noticed that this code doesn't really
> even have to
> run. Consider:
> 
> // child.cpp
> // compile with: g++ -lboost_signals child.cpp -shared
> -fPIC -o libchild.so
> #include <stdio.h>
> #include <boost/signals.hpp>
> 
> void  __attribute__ ( ( constructor ) ) ctor()
> {
>    printf( "child: Constructor\n" );
> }
> 
> void  __attribute__ ( ( destructor ) ) dtor()
> {
>    printf( "child: Destructor\n" );
> }
> 
> void a_function_that_does_not_get_called()
> {
>     new boost :: signal< void () >();
> }
> 
> The result:
> 
> host: Loading libchild.so...
> child: Constructor
> host: so = 0x8938020
> host: Unloading libchild.so...
> host: Unloaded.
> host: (nil)
> child: Destructor
> 
> This pretty much looks like a bug to me.
I admit I spoke too soon, assumed before actually trying your code. After I have tried (both 1st and 2nd versions), the code gave different output to me:

host: Loading libchild.so...
child: Constructor
host: so = 0x8db4020
host: Unloading libchild.so...
child: Destructor
host: Unloaded.
host: (nil)

Which seems correct (dlclose() calls the dtor()). I used 
gcc version 4.4.5
libdl-2.12.1.so
boost 1.45.0
Linux kernel 2.6.35-28
libc-2.12.1.so

Hope this helps,
-Pavel


> 
> Thanks.
>

Re: dlclose() doesn't unload any .so that uses Boost

[J]

On Tue, May 31, 2011 at 1:10 AM, Pavel Tolkachev <paultolk@yahoo.com> wrote:
> I admit I spoke too soon, assumed before actually trying your code. After I have tried (both 1st and 2nd versions), the code gave different output to me:
>
> host: Loading libchild.so...
> child: Constructor
> host: so = 0x8db4020
> host: Unloading libchild.so...
> child: Destructor
> host: Unloaded.
> host: (nil)

Interesting. By "the code gave different output" do you mean the first
version or the second one? Or both?

> Which seems correct (dlclose() calls the dtor()). I used
> gcc version 4.4.5
> libdl-2.12.1.so
> boost 1.45.0
> Linux kernel 2.6.35-28
> libc-2.12.1.so
>

Mine are:
gcc version 4.6.0
libdl-2.13.so
boost 1.46.0
Linux kernel 2.6.38
libc-2.13.so

I'm basically running a more up-to-date system than you, so it's
either a) distribution induced bug (I'm on Debian) or b) a bug
introduced recently.

Thanks.

Re: dlclose() doesn't unload any .so that uses Boost

[Ian Lance Taylor]

David Hagood <david.hagood@gmail.com> writes:

> On Sun, 2011-05-29 at 01:32 +0200, J wrote:
>> // compile with: g++ -ldl host.cpp -o host
> <snip>
>> // compile with: g++ -lboost_signals child.cpp -shared -fPIC -o libchild.so
>
> Of course, this isn't really a GCC question, but a libc/binutils
> question, so you may do better over on that list.

To be precise, this is a libc question.  It is not a binutils question.
This is a question about the dynamic linker and libdl, both of which are
part of libc on a typical Unix or GNU/Linux system.

Ian

Re: dlclose() doesn't unload any .so that uses Boost

[J]

On Tue, May 31, 2011 at 7:44 AM, Ian Lance Taylor <iant@google.com> wrote:
> David Hagood <david.hagood@gmail.com> writes:
>
>> On Sun, 2011-05-29 at 01:32 +0200, J wrote:
>>> // compile with: g++ -ldl host.cpp -o host
>> <snip>
>>> // compile with: g++ -lboost_signals child.cpp -shared -fPIC -o libchild.so
>>
>> Of course, this isn't really a GCC question, but a libc/binutils
>> question, so you may do better over on that list.
>
> To be precise, this is a libc question.  It is not a binutils question.
> This is a question about the dynamic linker and libdl, both of which are
> part of libc on a typical Unix or GNU/Linux system.

I've managed to track this issue down. (Silly me, I should have tried
this sooner.) It seems this *is* a bug in GCC:

$ g++-4.4 -lboost_signals child.cc -shared -fPIC -o libchild.so
$ ./host
host: Loading libchild.so...
child: Constructor
host: so = 0x85ab020
host: Unloading libchild.so...
child: Destructor
host: Unloaded.
host: (nil)

$ g++-4.6 -lboost_signals child.cc -shared -fPIC -o libchild.so
$ ./host
host: Loading libchild.so...
child: Constructor
host: so = 0x8b34020
host: Unloading libchild.so...
host: Unloaded.
host: (nil)
child: Destructor

Should I file a bug report?

Re: dlclose() doesn't unload any .so that uses Boost

[Ian Lance Taylor]

J <j@jabster.pl> writes:

> I've managed to track this issue down. (Silly me, I should have tried
> this sooner.) It seems this *is* a bug in GCC:
>
> $ g++-4.4 -lboost_signals child.cc -shared -fPIC -o libchild.so
> $ ./host
> host: Loading libchild.so...
> child: Constructor
> host: so = 0x85ab020
> host: Unloading libchild.so...
> child: Destructor
> host: Unloaded.
> host: (nil)
>
> $ g++-4.6 -lboost_signals child.cc -shared -fPIC -o libchild.so
> $ ./host
> host: Loading libchild.so...
> child: Constructor
> host: so = 0x8b34020
> host: Unloading libchild.so...
> host: Unloaded.
> host: (nil)
> child: Destructor
>
> Should I file a bug report?

Over on binutils Jakub points out that this is because gcc 4.6 is
marking symbols that must be unique as STB_GNU_UNIQUE.  This is intended
to make dlopen with RTLD_LOCAL work correctly.  Apparently when a shared
library defines a STB_GNU_UNIQUE symbol the dynamic linker marks that
library so that it is not deleted by dlclose.

gcc's use of STB_GNU_UNIQUE seems correct.  If there is a bug here, I
think it must be in the way the dynamic linker (which is part of glibc)
refuses to permit closing a library which defines a STB_GNU_UNIQUE
symbol.  It seems to me that that is only required if there is in fact
some reference to the symbol from some other shared library.

Ian