Enable preloading in dlopen-ed shared libraries?

Enable preloading in dlopen-ed shared libraries?

[Fengkai Sun]

Hi list,

As I dive a little deeper into the source code, I found that
`_dl_map_object_deps' is called in both rtld.c and dl-open.c.

The semantics of two invocations are both clear: rtld.c makes a call like
this:
_dl_map_object_deps (main_map, preloads, npreloads, mode == trace, 0);
because ld.so needs to preload the libraries specified by LD_PRELOAD into
the global scope for interposing the symbols.

dl-open.c makes a call like this:
_dl_map_object_deps (new, NULL, 0, 0,
      mode & (__RTLD_DLOPEN | RTLD_DEEPBIND | __RTLD_AUDIT));
because no dlopen-ed library needs to preload anything.

However, I think it might be useful to allow users to preload some
libraries in the local scope after the map of the dlopen-ed library, just
like how ld.so treats preloaded libs in global scope.

By doing so, the user can easily provide a different definition of a symbol
from the one of the main executable, by enabling RTLD_DEEPBIND.
This is useful under some circumstances. For example, a dlopen-ed library
may want to use a separate heap from the main heap, and the user can
provide another malloc implementation for that library.

The auditing interface can do the similar thing, but after doing some
experiments, I found that `la_symbind64' cannot catch the bindings of
global variables, and it cannot hook all of the function bindings.

Would it be a good idea to add an interface to enable preloading in the
local scope of dlopen-ed shared libraries?

Thank you in advance.

--
Best,
Fengkai

Re: Enable preloading in dlopen-ed shared libraries?

[Adhemerval Zanella]

On 14/07/2021 05:44, Fengkai Sun via Libc-help wrote:
> Hi list,
> 
> As I dive a little deeper into the source code, I found that
> `_dl_map_object_deps' is called in both rtld.c and dl-open.c.
> 
> The semantics of two invocations are both clear: rtld.c makes a call like
> this:
> _dl_map_object_deps (main_map, preloads, npreloads, mode == trace, 0);
> because ld.so needs to preload the libraries specified by LD_PRELOAD into
> the global scope for interposing the symbols.
> 
> dl-open.c makes a call like this:
> _dl_map_object_deps (new, NULL, 0, 0,
>       mode & (__RTLD_DLOPEN | RTLD_DEEPBIND | __RTLD_AUDIT));
> because no dlopen-ed library needs to preload anything.
> 
> However, I think it might be useful to allow users to preload some
> libraries in the local scope after the map of the dlopen-ed library, just
> like how ld.so treats preloaded libs in global scope.

Do you mean by preloading the shared library list using dlmopen in a new
namespace? Or do you mean to use RTLD_DEEPBIND with the preload libraries?

> 
> By doing so, the user can easily provide a different definition of a symbol
> from the one of the main executable, by enabling RTLD_DEEPBIND.
> This is useful under some circumstances. For example, a dlopen-ed library
> may want to use a separate heap from the main heap, and the user can
> provide another malloc implementation for that library.

But how is this different than the malloc() interposition already supported
with LD_PRELOAD?

> 
> The auditing interface can do the similar thing, but after doing some
> experiments, I found that `la_symbind64' cannot catch the bindings of
> global variables, and it cannot hook all of the function bindings.

The rtld-audit currently only works for symbols which requires a PLT call,
the global variables either done with GOT access directly or through copy
relocations.  I am working on extending la_symbind() to work with bind-now
binaries, so it would be called at loading time in symbol resolution instead
on the lazy binding resolution.

> 
> Would it be a good idea to add an interface to enable preloading in the
> local scope of dlopen-ed shared libraries?

I am trying to understand better what you are trying to do here, because 
you are mixing two different usercases here.  The RTLD_DEEPBIND is usually
used for shared libraries to use its direct dependencies over the global
list, the rtld-audit interfaces are loaded in a different namespace.

It means that symbol interposition for heap functions you described does
not fit with the interfaces: LD_PRELOAD are already taking precedence
over the global scope and using a difference namespace meaning a different
scope.

Re: Enable preloading in dlopen-ed shared libraries?

[Fengkai Sun]

Hi Adhemerval,

Thanks so much for your reply! I will explain my idea in detail and sorry
for the unclearness.


Do you mean by preloading the shared library list using dlmopen in a new
> namespace? Or do you mean to use RTLD_DEEPBIND with the preload libraries?
>

I mean making users able to specify a list of preload libraries in a shared
library's local scope, so that when the library is loaded with
RTLD_DEEPBIND on, the preload libraries will take precedence over any other
library except the dlopen-ed one.


> By doing so, the user can easily provide a different definition of a
> symbol
> > from the one of the main executable, by enabling RTLD_DEEPBIND.
> > This is useful under some circumstances. For example, a dlopen-ed library
> > may want to use a separate heap from the main heap, and the user can
> > provide another malloc implementation for that library.
>
> But how is this different than the malloc() interposition already supported
> with LD_PRELOAD?
>

I found that LD_PRELOAD cannot provide a different definition for a
dlopen-ed library from the main executable. Let's say we are preloading
mymalloc.so in an executable:

scope0(global scope): ./main_exe  mymalloc.so  libc.so  libdl.so
So the reference in the main executable is binded with the definition in
mymalloc.so.

A dlopen-ed shared library will have such kind of scope, let's call it
lib1.so:
scope0(global scope): ./main_exe  mymalloc.so  libc.so  libdl.so
scope1(local scope): lib1.so  libc.so

If lib1.so is loaded without RTLD_DEEPBIND, its reference to malloc will be
binded to mymalloc.so too. That means shared libraries and the main
executable are using the same heap, sometimes the user may want to prevent
it.
My goal is to preload libraries inside local scope, and it will look like:
scope1(local scope): lib1.so  othermalloc.so  libc.so

In this way, the main executable will never see the definition inside
othermalloc.so, and lib1 can bind to it when RTLD_DEEPBIND is on.

> The auditing interface can do the similar thing, but after doing some
> > experiments, I found that `la_symbind64' cannot catch the bindings of
> > global variables, and it cannot hook all of the function bindings.
>
> The rtld-audit currently only works for symbols which requires a PLT call,
> the global variables either done with GOT access directly or through copy
> relocations.  I am working on extending la_symbind() to work with bind-now
> binaries, so it would be called at loading time in symbol resolution
> instead
> on the lazy binding resolution.
>
> >
> > Would it be a good idea to add an interface to enable preloading in the
> > local scope of dlopen-ed shared libraries?
>
> I am trying to understand better what you are trying to do here, because
> you are mixing two different usercases here.  The RTLD_DEEPBIND is usually
> used for shared libraries to use its direct dependencies over the global
> list, the rtld-audit interfaces are loaded in a different namespace.
>

I mentioned the rtld-audit interface here because it could provide
different definitions for the same symbol. In la_symbind(), the user can
return a different symbol address other than sym->st_value to achieve this:

void *mymalloc(...) {...}
void *la_symbind(...){
    if (refcook == lib1_cook && strcmp(symname, "malloc") == 0) return
mymalloc;
    else return sym->st_value;
}


However, la_symbind() currently happens in lazy binding resolution(thank
you for clarifying it!), so it cannot work on every symbol for now.

Again, I'm sorry for mixing RTLD_DEEPBIND, dlmopen and rtld-audit all
together in the first email. I hope I've made things clear now.

--
Best,
Fengkai

Re: Enable preloading in dlopen-ed shared libraries?

[Adhemerval Zanella]

On 23/07/2021 23:51, Fengkai Sun wrote:
> Hi Adhemerval,
> 
> Thanks so much for your reply! I will explain my idea in detail and sorry
> for the unclearness.
> 
> 
> Do you mean by preloading the shared library list using dlmopen in a new
>> namespace? Or do you mean to use RTLD_DEEPBIND with the preload libraries?
>>
> 
> I mean making users able to specify a list of preload libraries in a shared
> library's local scope, so that when the library is loaded with
> RTLD_DEEPBIND on, the preload libraries will take precedence over any other
> library except the dlopen-ed one.

This seems very specific to add such complexity and extra internal state
on loader and I want to understand better the issue you are trying to solve
and why you can't use the current tools instead.

> 
> 
>> By doing so, the user can easily provide a different definition of a
>> symbol
>>> from the one of the main executable, by enabling RTLD_DEEPBIND.
>>> This is useful under some circumstances. For example, a dlopen-ed library
>>> may want to use a separate heap from the main heap, and the user can
>>> provide another malloc implementation for that library.
>>
>> But how is this different than the malloc() interposition already supported
>> with LD_PRELOAD?
>>
> 
> I found that LD_PRELOAD cannot provide a different definition for a
> dlopen-ed library from the main executable. Let's say we are preloading
> mymalloc.so in an executable:

Yes, LD_PRELOAD act only for the default global scope.

> 
> scope0(global scope): ./main_exe  mymalloc.so  libc.so  libdl.so
> So the reference in the main executable is binded with the definition in
> mymalloc.so.
> 
> A dlopen-ed shared library will have such kind of scope, let's call it
> lib1.so:
> scope0(global scope): ./main_exe  mymalloc.so  libc.so  libdl.so
> scope1(local scope): lib1.so  libc.so
> 
> If lib1.so is loaded without RTLD_DEEPBIND, its reference to malloc will be
> binded to mymalloc.so too. That means shared libraries and the main
> executable are using the same heap, sometimes the user may want to prevent
> it.
> My goal is to preload libraries inside local scope, and it will look like:
> scope1(local scope): lib1.so  othermalloc.so  libc.so
> 
> In this way, the main executable will never see the definition inside
> othermalloc.so, and lib1 can bind to it when RTLD_DEEPBIND is on.

Right, now I have a better grasp of what you are trying to do.  I am not
sure if I really like this: besides adding more complexity on the loader for
a very specify user case through environment variables, I think you could 
implement it with in a different way with a new interface we are aiming 
to support on next 2.35 (RTLD_SHARED):

  void *h = dlmopen (LM_ID_NEWLM, "othermalloc.so", RTLD_NOW)
  Lmid_t id;
  dlinfo (h, RTLD_DI_LMID, &id);
  dlmopen (id, "lib1.so", RTLD_SHARED | RTLD_NOW);

The lib1.so will then bind its dependencies only on the newly created
namespace scope (LD_PRELOAD won't be added in this case).  So a subsequent
dlopen ("lib1.so", ...) will return the handler to the lib1.so already 
created in the extra namespace.

(Vivek can correct me if I am wrong here)

> 
>> The auditing interface can do the similar thing, but after doing some
>>> experiments, I found that `la_symbind64' cannot catch the bindings of
>>> global variables, and it cannot hook all of the function bindings.
>>
>> The rtld-audit currently only works for symbols which requires a PLT call,
>> the global variables either done with GOT access directly or through copy
>> relocations.  I am working on extending la_symbind() to work with bind-now
>> binaries, so it would be called at loading time in symbol resolution
>> instead
>> on the lazy binding resolution.
>>
>>>
>>> Would it be a good idea to add an interface to enable preloading in the
>>> local scope of dlopen-ed shared libraries?
>>
>> I am trying to understand better what you are trying to do here, because
>> you are mixing two different usercases here.  The RTLD_DEEPBIND is usually
>> used for shared libraries to use its direct dependencies over the global
>> list, the rtld-audit interfaces are loaded in a different namespace.
>>
> 
> I mentioned the rtld-audit interface here because it could provide
> different definitions for the same symbol. In la_symbind(), the user can
> return a different symbol address other than sym->st_value to achieve this:
> 
> void *mymalloc(...) {...}
> void *la_symbind(...){
>     if (refcook == lib1_cook && strcmp(symname, "malloc") == 0) return
> mymalloc;
>     else return sym->st_value;
> }
> 
> 
> However, la_symbind() currently happens in lazy binding resolution(thank
> you for clarifying it!), so it cannot work on every symbol for now.

Yeah, my plan is to fix it for next release.  The rtld-audit might indeed
be another way to implement it.

> 
> Again, I'm sorry for mixing RTLD_DEEPBIND, dlmopen and rtld-audit all
> together in the first email. I hope I've made things clear now.

No problem, this is quite complex due the multiple ways to change the loader
dynamic symbol resolution.

> 
> --
> Best,
> Fengkai
> 

Re: Enable preloading in dlopen-ed shared libraries?

[Vivek Das Mohapatra]

> to support on next 2.35 (RTLD_SHARED):
>
>  void *h = dlmopen (LM_ID_NEWLM, "othermalloc.so", RTLD_NOW)
>  Lmid_t id;
>  dlinfo (h, RTLD_DI_LMID, &id);
>  dlmopen (id, "lib1.so", RTLD_SHARED | RTLD_NOW);
>
> The lib1.so will then bind its dependencies only on the newly created
> namespace scope (LD_PRELOAD won't be added in this case).  So a subsequent
> dlopen ("lib1.so", ...) will return the handler to the lib1.so already
> created in the extra namespace.
>
> (Vivek can correct me if I am wrong here)

That's how it should work, yes. I should probably add a set of tests to
check this exact scenario to the test suite.

Re: Enable preloading in dlopen-ed shared libraries?

[Fengkai Sun]

Hi Adhemerval,

This seems very specific to add such complexity and extra internal state
> on loader and I want to understand better the issue you are trying to solve
> and why you can't use the current tools instead.
>
> Thanks for your reply! I now realize that changing the interface of the
dynamic linker needs reason and user cases that are solid and general
enough. But originally I planned to ask if this is a good idea and if there
is an existing interface to achieve this. Thank you for the information
provided!

I will explain the original problem I want to solve here:

We have an acadamic project which proposes a framework, and it should load
and isolate some external libraries completely, making them self-contained
and does not need to refer to anything outside their own namespace.
Ideally, dlmopen would suffice, but I found that dlmopen can only support
up to 16 namespaces, but we might use way more libraries than that.

So I came up with another solution to simulate multi-namespace inside one:
use dlopen and load the libraries all into the default namespace. To
achieve complete isolation, I recursively renamed the dependencies of the
external libraries to bypass the limitation of allowing only one instance
inside a namespace. This is done by ELF patching tools, and I used patchelf.
For example, let's say there is a dependency graph like this:
libexternal.so -> lib1.so; libexternal.so -> libc.so
lib1.so -> libc.so
libc.so -> ld-linux.so

After the renaming, the filename and dependency names of each library is
changed, so that ld.so will load another instance even if some libraries
already exist in the default namespace.
The dependency graph after renaming is like this:
libexternal.so -> lib1-1.so; libexternal.so -> libc-1.so
lib1-1.so -> libc-1.so
libc-1.so -> ld-linux-1.so

This is not a good solution because my understanding of glibc is quite
shallow. dlopen seems to have problems in initializing libc and libpthread
when the filename of them are changed. So I tried not to rename libc or
libpthread and made them shared(just like the RTLD_UNIQUE flag proposed by
Vivek).

And we also want to interpose the symbols inside the external libraries. So
I asked if there is a way to provide a different definition for symbols in
local scope, just like what LD_PRELOAD does to global scope.
Currently I use patchelf to prepend a dependency, let's call it
libpreload.so, in the dynamic section of libexternal.so, so that ld.so will
search the symbol for libexternal.so and its dependencies first in
libpreload.so. Things are going well so far, and I will be very grateful if
you and other people on the list have a look at this solution.
It seems that our use case is really rare, and a full-functioned rtld-audit
interface can solve the problem for it.

--
Best,
Fengkai

Re: Enable preloading in dlopen-ed shared libraries?

[Adhemerval Zanella]

On 28/07/2021 03:17, Fengkai Sun wrote:
> Hi Adhemerval,
> 
> This seems very specific to add such complexity and extra internal state
>> on loader and I want to understand better the issue you are trying to solve
>> and why you can't use the current tools instead.
>>
>> Thanks for your reply! I now realize that changing the interface of the
> dynamic linker needs reason and user cases that are solid and general
> enough. But originally I planned to ask if this is a good idea and if there
> is an existing interface to achieve this. Thank you for the information
> provided!
> 
> I will explain the original problem I want to solve here:
> 
> We have an acadamic project which proposes a framework, and it should load
> and isolate some external libraries completely, making them self-contained
> and does not need to refer to anything outside their own namespace.
> Ideally, dlmopen would suffice, but I found that dlmopen can only support
> up to 16 namespaces, but we might use way more libraries than that.

Do you need each library to be completely isolated from each other? Can't you
load all of them in a single namespace or at least work around specific sets
up to the 16 namespace limits?

> 
> So I came up with another solution to simulate multi-namespace inside one:
> use dlopen and load the libraries all into the default namespace. To
> achieve complete isolation, I recursively renamed the dependencies of the
> external libraries to bypass the limitation of allowing only one instance
> inside a namespace. This is done by ELF patching tools, and I used patchelf.
> For example, let's say there is a dependency graph like this:
> libexternal.so -> lib1.so; libexternal.so -> libc.so
> lib1.so -> libc.so
> libc.so -> ld-linux.so
> 
> After the renaming, the filename and dependency names of each library is
> changed, so that ld.so will load another instance even if some libraries
> already exist in the default namespace.
> The dependency graph after renaming is like this:
> libexternal.so -> lib1-1.so; libexternal.so -> libc-1.so
> lib1-1.so -> libc-1.so
> libc-1.so -> ld-linux-1.so
> 
> This is not a good solution because my understanding of glibc is quite
> shallow. dlopen seems to have problems in initializing libc and libpthread
> when the filename of them are changed. So I tried not to rename libc or
> libpthread and made them shared(just like the RTLD_UNIQUE flag proposed by
> Vivek).

You seems to want unlimited namespace, which is different than what Vivek 
is trying to fix with RTLD_SHARED / RTLD_UNIQUE.  This should be doable by 
adding an extra list on _rtld_global and handle all places that access the 
_dl_ns link_namespaces (dlopen, dlclose, etc.). This would require some work
on how we will handle static TLS (since this take in consideration the maximum
number of namespaces).

And you are correct, glibc internally check if DT_SONAME is the one expected
by the hard-coded LIBC_SO.  Trying to mess with it might broke in some places
and that's exactly that dlmopen intends to provide.

> 
> And we also want to interpose the symbols inside the external libraries. So
> I asked if there is a way to provide a different definition for symbols in
> local scope, just like what LD_PRELOAD does to global scope.
> Currently I use patchelf to prepend a dependency, let's call it
> libpreload.so, in the dynamic section of libexternal.so, so that ld.so will
> search the symbol for libexternal.so and its dependencies first in
> libpreload.so. Things are going well so far, and I will be very grateful if
> you and other people on the list have a look at this solution.
> It seems that our use case is really rare, and a full-functioned rtld-audit
> interface can solve the problem for it.

The dlmopen should work to provide the symbol resolution for a specific set of
libraries: you first load on the namespace and then load the set of libraries.

But without dlmopen I don't have a easy solution for you. If you are not bounded
to the system glibc, you might change the DL_NNS internally to expected number
of namespace.  It has some implication, specially in the static TLS requirement,
but I think it should work.

I am not really against the local scope preload environment variable, but this 
will be another *very* specific feature that we will need to support indefinitely
for an specific usercase that might not even used extensively (as you put this is
a research project). This will also have some semantic issues we will need to sort
out also, like how to represent an specific namespace (since glibc can return any
value and Lmid_t does not need to map to an specific number).
 

Re: Enable preloading in dlopen-ed shared libraries?

[Geoff T]

unsubscribe

On Wed, Jul 28, 2021 at 12:09 PM Adhemerval Zanella via Libc-help <
libc-help@sourceware.org> wrote:

>
>
> On 28/07/2021 03:17, Fengkai Sun wrote:
> > Hi Adhemerval,
> >
> > This seems very specific to add such complexity and extra internal state
> >> on loader and I want to understand better the issue you are trying to
> solve
> >> and why you can't use the current tools instead.
> >>
> >> Thanks for your reply! I now realize that changing the interface of the
> > dynamic linker needs reason and user cases that are solid and general
> > enough. But originally I planned to ask if this is a good idea and if
> there
> > is an existing interface to achieve this. Thank you for the information
> > provided!
> >
> > I will explain the original problem I want to solve here:
> >
> > We have an acadamic project which proposes a framework, and it should
> load
> > and isolate some external libraries completely, making them
> self-contained
> > and does not need to refer to anything outside their own namespace.
> > Ideally, dlmopen would suffice, but I found that dlmopen can only support
> > up to 16 namespaces, but we might use way more libraries than that.
>
> Do you need each library to be completely isolated from each other? Can't
> you
> load all of them in a single namespace or at least work around specific
> sets
> up to the 16 namespace limits?
>
> >
> > So I came up with another solution to simulate multi-namespace inside
> one:
> > use dlopen and load the libraries all into the default namespace. To
> > achieve complete isolation, I recursively renamed the dependencies of the
> > external libraries to bypass the limitation of allowing only one instance
> > inside a namespace. This is done by ELF patching tools, and I used
> patchelf.
> > For example, let's say there is a dependency graph like this:
> > libexternal.so -> lib1.so; libexternal.so -> libc.so
> > lib1.so -> libc.so
> > libc.so -> ld-linux.so
> >
> > After the renaming, the filename and dependency names of each library is
> > changed, so that ld.so will load another instance even if some libraries
> > already exist in the default namespace.
> > The dependency graph after renaming is like this:
> > libexternal.so -> lib1-1.so; libexternal.so -> libc-1.so
> > lib1-1.so -> libc-1.so
> > libc-1.so -> ld-linux-1.so
> >
> > This is not a good solution because my understanding of glibc is quite
> > shallow. dlopen seems to have problems in initializing libc and
> libpthread
> > when the filename of them are changed. So I tried not to rename libc or
> > libpthread and made them shared(just like the RTLD_UNIQUE flag proposed
> by
> > Vivek).
>
> You seems to want unlimited namespace, which is different than what Vivek
> is trying to fix with RTLD_SHARED / RTLD_UNIQUE.  This should be doable by
> adding an extra list on _rtld_global and handle all places that access the
> _dl_ns link_namespaces (dlopen, dlclose, etc.). This would require some
> work
> on how we will handle static TLS (since this take in consideration the
> maximum
> number of namespaces).
>
> And you are correct, glibc internally check if DT_SONAME is the one
> expected
> by the hard-coded LIBC_SO.  Trying to mess with it might broke in some
> places
> and that's exactly that dlmopen intends to provide.
>
> >
> > And we also want to interpose the symbols inside the external libraries.
> So
> > I asked if there is a way to provide a different definition for symbols
> in
> > local scope, just like what LD_PRELOAD does to global scope.
> > Currently I use patchelf to prepend a dependency, let's call it
> > libpreload.so, in the dynamic section of libexternal.so, so that ld.so
> will
> > search the symbol for libexternal.so and its dependencies first in
> > libpreload.so. Things are going well so far, and I will be very grateful
> if
> > you and other people on the list have a look at this solution.
> > It seems that our use case is really rare, and a full-functioned
> rtld-audit
> > interface can solve the problem for it.
>
> The dlmopen should work to provide the symbol resolution for a specific
> set of
> libraries: you first load on the namespace and then load the set of
> libraries.
>
> But without dlmopen I don't have a easy solution for you. If you are not
> bounded
> to the system glibc, you might change the DL_NNS internally to expected
> number
> of namespace.  It has some implication, specially in the static TLS
> requirement,
> but I think it should work.
>
> I am not really against the local scope preload environment variable, but
> this
> will be another *very* specific feature that we will need to support
> indefinitely
> for an specific usercase that might not even used extensively (as you put
> this is
> a research project). This will also have some semantic issues we will need
> to sort
> out also, like how to represent an specific namespace (since glibc can
> return any
> value and Lmid_t does not need to map to an specific number).
>
>