alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

I've got a situation where I need to hook a dlopen() made by VDDK, a 
proprietary library, where it passes a relative name expecting to 
resolve to a copy of several libraries, including libstdc++.so, that it 
installs alongside itself, and fails to load if that resolves to the 
system libstdc++.so.  The simplest solution of providing LD_LIBRARY_PATH 
is enough to load VDDK, but then poisons any child process which 
likewise fail to load if they pick up VDDK's libstdc++.so instead of the 
system one.  Up to now, we've documented throwing the burden on the end 
user who has to write convoluted:

LD_LIBRARY_PATH_save=$LD_LIBRARY_PATH \
  LD_LIBRARY_PATH=/path/to/vddklibs:$LD_LIBRARY_PATH \
  nbdkit vddk libdir=/path/to/vddklibs file --run \
    'LD_LIBRARY_PATH=$LD_LIBRARY_PATH_save; program args'

where we would rather the end-user could get away with a more concise:

nbdkit vddk libdir=/path/to/vddklibs file --run 'program args'

Sequentially, we have this scenario:

nbdkit vddk libdir=/path/to/libs file --run 'program args'
- nbdkit binary calls dlopen("/path/to/nbdkit-vddk-plugin.so")
   - nbdkit-vddk-plugin.so calls
     dlopen("/path/to/libs/libvixDiskLibs.so") using the libdir= argument
     to load vddk (rather than dlopen("libvixDiskLibs.so") relying on
     LD_LIBRARY_PATH)
   - vddk's initializer calls dlopen("libcrypto.so") expecting to
     open /path/to/libs/libcrypto.so, but either LD_LIBRARY_PATH
     made that possible (at which point we have to scrub it before
     a child process will be penalized), or we have to find a way to
     rewrite vddk's dlopen call from relative into absolute before
     passing it to the real dlopen
- nbdkit binary spawns a child process to exec 'program args'
   - program does not want /path/to/libs in its search path

Writing my own dlopen() wrapper directly in nbdkit seems like a 
non-starter (my override has to come from a shared library before it can 
replace the shared version that would be imported from -ldl, at least 
for all subsequent shared library loads that want to bind to the 
override).  And if I read 'man dlopen' correctly, since nbdkit used 
dlopen() to load nbdkit-vddk-plugin.so, then dlopen() is already bound 
in the main context, so unless I use RTLD_DEEPBIND from nbdkit, then 
nbdkit-vddk-plugin.so will also see dlopen() bound to -dl rather than 
anything it loads locally; but even with RTLD_DEEPBIND, it sounds like 
that higher precedence lasts only for nbdkit-vddk-plugin.so and does not 
extend to later bindings performed for libvixDiskLib.so (which means 
vddk is back to -ldl's dlopen, without my hook).  Thus, to hook dlopen 
within the same process, I need some way to create a scope where I can 
provide a shared dlopen() that will take precedence when resolving 
symbols during the load of libvixDiskLib.so, but where that hook code 
can still defer back to the real dlopen() from -ldl and does not 
penalize child processes.

I managed to create a solution that avoids the need to set 
LD_PRELOAD_PATH at all by installing a shared library that hooks 
dlopen(), then loading both my shim and vddk via dlmopen() without the 
use of RTLD_GLOBAL or RTLD_DEEPBIND.  More links on my solution:

https://www.redhat.com/archives/libguestfs/2020-February/msg00154.html
https://bugzilla.redhat.com/show_bug.cgi?id=1756307#c7
https://sourceware.org/bugzilla/show_bug.cgi?id=15971#c5

However, when Florian saw it, he suggested that my solution of dlmopen() 
for a shim library that overrides dlopen() is reinventing what 
la_objsearch() can already do.  This is in part because the moment you 
dlmopen() a library into a separate namespace, you can't debug it (both 
glibc and gdb need additional patches to expose alternative namespaces 
for debugging), but there may be other nasty surprises lurking.

But after spending more than an hour playing with la_objsearch() and 
reading 'man rtld-audit', it looks like an audit library cannot be 
triggered in glibc except by listing it in LD_AUDIT in the environment 
during exec - which is back to the same problem we have with needing 
LD_LIBRARY_PATH in the environment.  Furthermore, although I know that 
glibc's audit interface is slightly different from the Solaris version 
it copied from, the Solaris documentation states that an audit library 
has some rather tough restrictions (including that using 'malloc' is 
unsafe, 
https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc 
"Some system interfaces assume that the interfaces are the only instance 
of their implementation within a process. Examples of such 
implementations are signals and malloc(3C). Audit libraries should avoid 
using such interfaces, as doing so can inadvertently alter the behavior 
of the application.").  But Solaris also stated that a library could 
serve as an audit entry point without LD_AUDIT if it is registered 
locally, via -Wl,-paudit.so.1 when creating the shared library 
(https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc); 
it doesn't seem that this functionality exists with glibc 
(/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit).

Does anyone have any ideas on how to let a shared library implement an 
audit interface for just its own process, without having to edit 
LD_AUDIT or re-exec the process?  Or is there yet another way to hook a 
program to rewrite misbehaving dlopen() calls without relying on either 
dlmopen() or la_objsearch(), or requiring pre-set environment variables, 
or having to re-exec a process?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

On 2/14/20 1:02 PM, Eric Blake wrote:

> Writing my own dlopen() wrapper directly in nbdkit seems like a 
> non-starter (my override has to come from a shared library before it can 
> replace the shared version that would be imported from -ldl, at least 
> for all subsequent shared library loads that want to bind to the 
> override).

Maybe I spoke too soon. I've tried another approach that looks like it 
will do what I want: put my shim dlopen() in a shared library, but link 
nbdkit against that shared library PRIOR to -ldl (so that name lookup 
for dlopen resolves there first).  The shim library in turn depends on 
-ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real 
dlopen.  And by linking it directly into nbdkit, rather than into the 
nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen() 
in use for all subsequent loads is from my shim.  It's still a bit less 
clean than I'd like (it requires tighter coupling between nbdkit and 
nbdkit-vddk-plugin.so than what used to exist), but the fact that it 
works without dlmopen() or LD_LIBRARY_PATH is in its favor.  I'm now 
polishing up the experiment, and will post the patch when it's ready.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Carlos O'Donell]

On 2/14/20 4:29 PM, Eric Blake wrote:
> On 2/14/20 1:02 PM, Eric Blake wrote:
> 
>> Writing my own dlopen() wrapper directly in nbdkit seems like a
>> non-starter (my override has to come from a shared library before
>> it can replace the shared version that would be imported from -ldl,
>> at least for all subsequent shared library loads that want to bind
>> to the override).
> 
> Maybe I spoke too soon. I've tried another approach that looks like
> it will do what I want: put my shim dlopen() in a shared library, but
> link nbdkit against that shared library PRIOR to -ldl (so that name
> lookup for dlopen resolves there first).  The shim library in turn
> depends on -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get
> to the real dlopen.  And by linking it directly into nbdkit, rather
> than into the nbdkit-vddk-plugin.so that gets loaded later, the first
> bound dlopen() in use for all subsequent loads is from my shim.  It's
> still a bit less clean than I'd like (it requires tighter coupling
> between nbdkit and nbdkit-vddk-plugin.so than what used to exist),
> but the fact that it works without dlmopen() or LD_LIBRARY_PATH is in
> its favor.  I'm now polishing up the experiment, and will post the
> patch when it's ready.

I think that's the best solution you're going to get.

The alternatives (LD_LIBRARY_PATH, direct loader invocation, dlmopen)
all have limitations that aren't helpful to your particular design.

You have design strategies that look like this:

- Move the object higher in the search order at link time (interposition)
  - Investigate static link order.
  - Investigate dynamic loader search order

- Change what object is searched for
  - LD_LIBRARY_PATH, DT_RPATH, DT_RUNPATH, etc.
  - LD_AUDIT with la_objsearch()

Your "shim dlopen()" is a case of moving the static link order
around to ensure your shim is used first.

-- 
Cheers,
Carlos.

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

On 2/14/20 3:29 PM, Eric Blake wrote:
> On 2/14/20 1:02 PM, Eric Blake wrote:
> 
>> Writing my own dlopen() wrapper directly in nbdkit seems like a 
>> non-starter (my override has to come from a shared library before it 
>> can replace the shared version that would be imported from -ldl, at 
>> least for all subsequent shared library loads that want to bind to the 
>> override).
> 
> Maybe I spoke too soon. I've tried another approach that looks like it 
> will do what I want: put my shim dlopen() in a shared library, but link 
> nbdkit against that shared library PRIOR to -ldl (so that name lookup 
> for dlopen resolves there first).  The shim library in turn depends on 
> -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real 
> dlopen.  And by linking it directly into nbdkit, rather than into the 
> nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen() 
> in use for all subsequent loads is from my shim.  It's still a bit less 
> clean than I'd like (it requires tighter coupling between nbdkit and 
> nbdkit-vddk-plugin.so than what used to exist), but the fact that it 
> works without dlmopen() or LD_LIBRARY_PATH is in its favor.  I'm now 
> polishing up the experiment, and will post the patch when it's ready.

Progress report: I've posted a v4 series that relies on a shared library 
in the main executable; but I'm still trying to see if I can further 
reduce things (maybe with -rdynamic) so that the main binary itself 
provides the dlopen() override without needing an auxiliary shared library.
https://www.redhat.com/archives/libguestfs/2020-February/msg00162.html

> But after spending more than an hour playing with la_objsearch() and reading 'man rtld-audit', it looks like an audit library cannot be triggered in glibc except by listing it in LD_AUDIT in the environment during exec - which is back to the same problem we have with needing LD_LIBRARY_PATH in the environment.  Furthermore, although I know that glibc's audit interface is slightly different from the Solaris version it copied from, the Solaris documentation states that an audit library has some rather tough restrictions (including that using 'malloc' is unsafe, https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc "Some system interfaces assume that the interfaces are the only instance of their implementation within a process. Examples of such implementations are signals and malloc(3C). Audit libraries should avoid using such interfaces, as doing so can inadvertently alter the behavior of the application.").  But Solaris also stated that a library could serve as an audit entry point without LD_AUDIT if it is registered locally, via -Wl,-paudit.so.1 when creating the shared library (https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc); it doesn't seem that this functionality exists with glibc (/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit). 

I'm just now noticing that 'man ld' reports that you may pass '--audit 
LIB' during linking to add a DT_DEPAUDIT dependency on a library 
implementing the audit interface, which sounds like it might be an 
alternative to LD_AUDIT for getting a library with la_objsearch() to 
actually do something (but doesn't obviate the need for la_obsearch() to 
be in a separate library, rather than part of the main executable, 
unless a library can be reused as its own audit library...).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Florian Weimer]

* Eric Blake:

> I'm just now noticing that 'man ld' reports that you may pass '--audit
> LIB' during linking to add a DT_DEPAUDIT dependency on a library
> implementing the audit interface, which sounds like it might be an
> alternative to LD_AUDIT for getting a library with la_objsearch() to
> actually do something (but doesn't obviate the need for la_obsearch()
> to be in a separate library, rather than part of the main executable,
> unless a library can be reused as its own audit library...).

DT_AUDIT support has yet to be implemented in glibc:

  <https://sourceware.org/bugzilla/show_bug.cgi?id=24943>
  <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>

If you go on record saying that you need this, maybe someone will review
the patch.  Sorry. 8-(

Florian

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

On 2/17/20 9:12 AM, Florian Weimer wrote:
> * Eric Blake:
> 
>> I'm just now noticing that 'man ld' reports that you may pass '--audit
>> LIB' during linking to add a DT_DEPAUDIT dependency on a library
>> implementing the audit interface, which sounds like it might be an
>> alternative to LD_AUDIT for getting a library with la_objsearch() to
>> actually do something (but doesn't obviate the need for la_obsearch()
>> to be in a separate library, rather than part of the main executable,
>> unless a library can be reused as its own audit library...).
> 
> DT_AUDIT support has yet to be implemented in glibc:
> 
>    <https://sourceware.org/bugzilla/show_bug.cgi?id=24943>
>    <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>
> 
> If you go on record saying that you need this, maybe someone will review
> the patch.  Sorry. 8-(

Another followup: nbdkit-vddk-plugin.so is now using a re-exec setup 
[1], for several reasons:
1. all our other ideas tried so far (such as a dlopen() interposition in 
the main nbdkit binary) touched more files and required more exported 
APIs; we managed to get re-exec working with changes limited to just the 
plugin (other than a minor change to nbdkit to quit messing with argv[] 
so that /proc/self/cmdline would stay stable - but that did not require 
a new API).
2. it turns out that overriding dlopen() was insufficient to work around 
VDDK's setup [2].  It _did_ solve the initial dlopen() performed by 
VDDK, but that library in turn had DT_NEEDED entries for bare library 
names, which dlopen() does NOT affect but which la_objsearch() should.
3. for nbdkit, we want to minimize the number of binary files shipped; 
the re-exec solution works with just the nbdkit binary and the 
nbdkit-vddk-plugin.so.  Any solution that requires a third file to be 
shipped (be that a shared library providing dlopen, or a LD_AUDIT 
library, or otherwise) is less palatable than the 2-binary solution that 
our re-exec solution provides.

[1] https://github.com/libguestfs/nbdkit/commit/0c7ac4e655b
[2] https://www.redhat.com/archives/libguestfs/2020-February/msg00184.html

So with that said, here's a question I just thought of:

If your patch for glibc support for DT_AUDIT is incorporated, is it 
possible to mark a shared library as its own audit library via DT_AUDIT? 
  That is, if nbdkit-vddk-plugin.so can provide entry points for _both_ 
the nbdkit interface (which satisfies dlopen() from the nbdkit binary) 
and la_version/la_objsearch() (which satisfy the requirements for use 
from the audit code in ld.so), _and_ during the compilation of 
nbdkit-vddk-plugin.so, we marked the library as its own DT_AUDIT entry, 
would the mere act of dlopen("nbdkit-vddk-plugin.so") from nbdkit be 
sufficient to trigger audit actions such as la_objsearch() on all 
subsequent shared loads (whether by dlopen() or DT_NEEDED) performed by 
nbdkit-vddk-plugin.so and its descendant loaded libraries?  Because if 
so, we would have a use case where a single binary, set up to act as its 
own audit library, might be sufficient to hook the shared object search 
path without needing any of environment variable modification, a process 
re-exec, or a third shipping binary - in which case that would indeed be 
a nicer solution than the current re-exec solution we committed today 
(of course, nbdkit would not be able to rely on that solution except on 
systems with new enough glibc to support DT_AUDIT).

I guess even without DT_AUDIT support, I could at least answer the 
question of whether a single .so can be used to satisfy both dlopen() 
and LD_AUDIT interfaces at once by setting LD_AUDIT (where the only 
remaining gap is figuring out when glibc can let DT_AUDIT have the same 
effect).  During my earlier attempts to get a working dlopen() override, 
I didn't consider any solution that required setting LD_AUDIT, but now 
that I proved dlopen() override alone was not enough for the case at 
hand, having to re-exec to set LD_AUDIT is no worse than having to 
re-exec to set LD_LIBRARY_PATH as a fallback to systems where glibc does 
not support DT_AUDIT.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Florian Weimer]

* Eric Blake:

> So with that said, here's a question I just thought of:
>
> If your patch for glibc support for DT_AUDIT is incorporated, is it
> possible to mark a shared library as its own audit library via
> DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points
> for _both_ the nbdkit interface (which satisfies dlopen() from the
> nbdkit binary) and la_version/la_objsearch() (which satisfy the
> requirements for use from the audit code in ld.so), _and_ during the
> compilation of nbdkit-vddk-plugin.so, we marked the library as its own
> DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so")
> from nbdkit be sufficient to trigger audit actions such as
> la_objsearch() on all subsequent shared loads (whether by dlopen() or
> DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant
> loaded libraries?

So you want to dlopen nbdkit-vddk-plugin.so and launch a new auditor
even if the process so far hasn't used auditing?  And the main program
(which links agains a library which eventually makes this dlopen call)
would not know anything about the existence of this specific plugin and
auditing?

This isn't currently supported.  It's not just that the glibc
implementation cannot do it.  The audit API (as sketched in <link.h>) is
not a good fit for late loading where you have never observed open
events.  It pretty much assumes that auditors are loaded magically
*before* program start, so that they can observe all open calls and set
up their own data structures along the way.

I think what confuses me is that keep talking about a single binary, but
clearly there is this separate vddk DSO, and there is talk of plugins.
So it seems to me that multiple files are involved already?

Thanks,
Florian

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

On 2/21/20 6:19 AM, Florian Weimer wrote:
> * Eric Blake:
> 
>> So with that said, here's a question I just thought of:
>>
>> If your patch for glibc support for DT_AUDIT is incorporated, is it
>> possible to mark a shared library as its own audit library via
>> DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points
>> for _both_ the nbdkit interface (which satisfies dlopen() from the
>> nbdkit binary) and la_version/la_objsearch() (which satisfy the
>> requirements for use from the audit code in ld.so), _and_ during the
>> compilation of nbdkit-vddk-plugin.so, we marked the library as its own
>> DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so")
>> from nbdkit be sufficient to trigger audit actions such as
>> la_objsearch() on all subsequent shared loads (whether by dlopen() or
>> DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant
>> loaded libraries?
> 
> So you want to dlopen nbdkit-vddk-plugin.so and launch a new auditor
> even if the process so far hasn't used auditing?  And the main program
> (which links agains a library which eventually makes this dlopen call)
> would not know anything about the existence of this specific plugin and
> auditing?

Yes, you interpreted my question correctly.

> 
> This isn't currently supported.  It's not just that the glibc
> implementation cannot do it.  The audit API (as sketched in <link.h>) is
> not a good fit for late loading where you have never observed open
> events.  It pretty much assumes that auditors are loaded magically
> *before* program start, so that they can observe all open calls and set
> up their own data structures along the way.

The concern is not about nbdkit loading nbdkit-vddk-plugin.so, but 
nbdkit-vddk-plugin.so doing subsequent loads of libvixDiskLib.so and its 
bare dependencies on libstdc++.so and such that were incorrectly built 
without DT_RUNPATH, but where we can't rewrite libvixDiskLib.so because 
it is proprietary, so the best we can do is hook the loading environment 
(either by la_objsearch or by re-exec with LD_LIBRARY_PATH).

> 
> I think what confuses me is that keep talking about a single binary, but
> clearly there is this separate vddk DSO, and there is talk of plugins.
> So it seems to me that multiple files are involved already?

You are correct that there are multiple files involved:

The nbdkit project currently has 2 relevant files: 'nbdkit' and 
'nbdkit-vddk-plugin.so' (and various other plugins, but those are not 
relevant to the VDDK use case)

The VDDK project from VMware: multiple files: libvixDiskLib.so (primary 
interface), which dlopen()s libdiskLibPlugin.so, which in turn has 
DT_NEEDED on libstdc++.so and several other recompiled system libraries. 
'find vmware-vix-disklib-distrib/lib64/ -type f | wc' shows 23 libraries 
total, but the end user installs it as a single tarball from VMware.

We can't change what VDDK ships, but we want to avoid making the nbdkit 
portion change from 2 files into 3, as every additional file required 
beyond what VMware ships is that much more burden for a user to choose 
to use nbdkit for accessing their VMware disks.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Richard W.M. Jones]

On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
> I think what confuses me is that keep talking about a single binary, but
> clearly there is this separate vddk DSO, and there is talk of plugins.
> So it seems to me that multiple files are involved already?

nbdkit is a standalone binary that happens to be able to load plugins
from a well-known path, eg nbdkit-vddk-plugin.so.  nbdkit knows the
path for plugins, and there's a wrapper allowing it to get local
plugins even when it's still in the build directory.  Adding another
file would mean another path (or overloading the meaning of the plugin
path) and just makes the whole thing more fragile and complex.

Having said all that, what would also solve this is either an API for
updating LD_LIBRARY_PATH after the program has started; or making
setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
which takes a library path as an extra parameter.

Rich.

* “Why does setenv ("LD_LIBRARY_PATH") not work?” has several
stackoverflow answers.  Apparently even the JDK has to work around
this by re-execing.
https://www.google.com/search?q=setenv+%22LD_LIBRARY_PATH%22+site:stackoverflow.com

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Florian Weimer]

* Richard W. M. Jones:

> On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
>> I think what confuses me is that keep talking about a single binary, but
>> clearly there is this separate vddk DSO, and there is talk of plugins.
>> So it seems to me that multiple files are involved already?
>
> nbdkit is a standalone binary that happens to be able to load plugins
> from a well-known path, eg nbdkit-vddk-plugin.so.  nbdkit knows the
> path for plugins, and there's a wrapper allowing it to get local
> plugins even when it's still in the build directory.  Adding another
> file would mean another path (or overloading the meaning of the plugin
> path) and just makes the whole thing more fragile and complex.
>
> Having said all that, what would also solve this is either an API for
> updating LD_LIBRARY_PATH after the program has started; or making
> setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
> which takes a library path as an extra parameter.

Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
Or does the path have to be chosen dynamically?

If you merely want to prevent loading of libstdc++.so or libcrypto.so by
vddk, it may be possible to explicitly dlopen DSOs of that name before
loading vddk.  But there is an existing bug where we do not duplicate
properly on soname alone, so we may have to fix that first.

Thanks,
Florian

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Richard W.M. Jones]

On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
> * Richard W. M. Jones:
> 
> > On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
> >> I think what confuses me is that keep talking about a single binary, but
> >> clearly there is this separate vddk DSO, and there is talk of plugins.
> >> So it seems to me that multiple files are involved already?
> >
> > nbdkit is a standalone binary that happens to be able to load plugins
> > from a well-known path, eg nbdkit-vddk-plugin.so.  nbdkit knows the
> > path for plugins, and there's a wrapper allowing it to get local
> > plugins even when it's still in the build directory.  Adding another
> > file would mean another path (or overloading the meaning of the plugin
> > path) and just makes the whole thing more fragile and complex.
> >
> > Having said all that, what would also solve this is either an API for
> > updating LD_LIBRARY_PATH after the program has started; or making
> > setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
> > which takes a library path as an extra parameter.
> 
> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
> Or does the path have to be chosen dynamically?

To be clear, the situation is:

  nbdkit (free)
   -> dlopens nbdkit-vddk-plugin.so (free)
     -> dlopens libvixDiskLib.so (proprietary)
       -> dlopens other proprietary plugins
       -> both libvixDiskLib.so and its plugins have DT_NEEDED
          "libstdc++.so.X" and other objects that have odd/old
          compiled versions in its own directory

It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
which has trouble opening its own plugins.  I guess you mean adding
DT_* to the proprietary library?

We don't distribute the proprietary library, end users have to choose
to accept the terms of the (very poisonous) license on that library
and so they download and install it themselves and just give us a path
where they installed it.

With all this low quality proprietary software involved I don't expect
there's going to be a clean way to solve this, was really only
wondering if there's a better way than re-execing the whole program.
But we do at least have the re-exec working now.

> If you merely want to prevent loading of libstdc++.so or libcrypto.so by
> vddk, it may be possible to explicitly dlopen DSOs of that name before
> loading vddk.  But there is an existing bug where we do not duplicate
> properly on soname alone, so we may have to fix that first.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Florian Weimer]

* Richard W. M. Jones:

> On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
>> * Richard W. M. Jones:
>> 
>> > On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
>> >> I think what confuses me is that keep talking about a single binary, but
>> >> clearly there is this separate vddk DSO, and there is talk of plugins.
>> >> So it seems to me that multiple files are involved already?
>> >
>> > nbdkit is a standalone binary that happens to be able to load plugins
>> > from a well-known path, eg nbdkit-vddk-plugin.so.  nbdkit knows the
>> > path for plugins, and there's a wrapper allowing it to get local
>> > plugins even when it's still in the build directory.  Adding another
>> > file would mean another path (or overloading the meaning of the plugin
>> > path) and just makes the whole thing more fragile and complex.
>> >
>> > Having said all that, what would also solve this is either an API for
>> > updating LD_LIBRARY_PATH after the program has started; or making
>> > setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
>> > which takes a library path as an extra parameter.
>> 
>> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
>> Or does the path have to be chosen dynamically?
>
> To be clear, the situation is:
>
>   nbdkit (free)
>    -> dlopens nbdkit-vddk-plugin.so (free)
>      -> dlopens libvixDiskLib.so (proprietary)
>        -> dlopens other proprietary plugins
>        -> both libvixDiskLib.so and its plugins have DT_NEEDED
>           "libstdc++.so.X" and other objects that have odd/old
>           compiled versions in its own directory
>
> It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
> which has trouble opening its own plugins.  I guess you mean adding
> DT_* to the proprietary library?

No, to nbdkit-vddk-plugin.so.

But it looks like have misunderstood the request.

Do you want to inhibit loading the libstdc++.so.X from vddk, or the
opposite—ensure that it is loaded?  The latter obviously taints the
process.  But maybe that's what you want?

Thanks,
Florian

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Richard W.M. Jones]

On Fri, Feb 21, 2020 at 05:02:12PM +0100, Florian Weimer wrote:
> * Richard W. M. Jones:
> 
> > On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
> >> * Richard W. M. Jones:
> >> 
> >> > On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
> >> >> I think what confuses me is that keep talking about a single binary, but
> >> >> clearly there is this separate vddk DSO, and there is talk of plugins.
> >> >> So it seems to me that multiple files are involved already?
> >> >
> >> > nbdkit is a standalone binary that happens to be able to load plugins
> >> > from a well-known path, eg nbdkit-vddk-plugin.so.  nbdkit knows the
> >> > path for plugins, and there's a wrapper allowing it to get local
> >> > plugins even when it's still in the build directory.  Adding another
> >> > file would mean another path (or overloading the meaning of the plugin
> >> > path) and just makes the whole thing more fragile and complex.
> >> >
> >> > Having said all that, what would also solve this is either an API for
> >> > updating LD_LIBRARY_PATH after the program has started; or making
> >> > setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
> >> > which takes a library path as an extra parameter.
> >> 
> >> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
> >> Or does the path have to be chosen dynamically?
> >
> > To be clear, the situation is:
> >
> >   nbdkit (free)
> >    -> dlopens nbdkit-vddk-plugin.so (free)
> >      -> dlopens libvixDiskLib.so (proprietary)
> >        -> dlopens other proprietary plugins
> >        -> both libvixDiskLib.so and its plugins have DT_NEEDED
> >           "libstdc++.so.X" and other objects that have odd/old
> >           compiled versions in its own directory
> >
> > It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
> > which has trouble opening its own plugins.  I guess you mean adding
> > DT_* to the proprietary library?
> 
> No, to nbdkit-vddk-plugin.so.
> 
> But it looks like have misunderstood the request.
> 
> Do you want to inhibit loading the libstdc++.so.X from vddk, or the
> opposite—ensure that it is loaded?  The latter obviously taints the
> process.  But maybe that's what you want?

Ideally load it but only make it available to VDDK.  I think Eric was
working on something along these lines, using dlmopen.

We're sort of lucky that none of the libraries that VDDK tries to load
overlaps with libraries that nbdkit uses (currently).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

On 2/21/20 9:00 AM, Florian Weimer wrote:
> * Richard W. M. Jones:
> 
>> On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
>>> I think what confuses me is that keep talking about a single binary, but
>>> clearly there is this separate vddk DSO, and there is talk of plugins.
>>> So it seems to me that multiple files are involved already?
>>
>> nbdkit is a standalone binary that happens to be able to load plugins
>> from a well-known path, eg nbdkit-vddk-plugin.so.  nbdkit knows the
>> path for plugins, and there's a wrapper allowing it to get local
>> plugins even when it's still in the build directory.  Adding another
>> file would mean another path (or overloading the meaning of the plugin
>> path) and just makes the whole thing more fragile and complex.
>>
>> Having said all that, what would also solve this is either an API for
>> updating LD_LIBRARY_PATH after the program has started; or making
>> setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
>> which takes a library path as an extra parameter.
> 
> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so

Post-processing an existing closed-source .so shipped from an external 
vendor might have negative consequences - while it may be possible to 
modify the ELF image to add a DT_RUNPATH entry or modify the DT_NEEDED 
entries to use anchored names based on $ORIGIN rather than bare names, 
there's no telling if such modification would be in violation of 
agreements or even cause failure to load if the proprietary code is 
using shenanigans like validating a checksum of shipped binaries to 
detect tampering.

> Or does the path have to be chosen dynamically?

So, since we cannot fix the existing product, and have no idea if/when 
the vendor will release an updated version that has saner libraries, a 
dynamic search path is the only option to getting their product to load 
(but whether that is done by LD_LIBRARY_PATH, LD_AUDIT, pre-loading 
libraries, or something else, is where we have a bit more control).

> 
> If you merely want to prevent loading of libstdc++.so or libcrypto.so by
> vddk, it may be possible to explicitly dlopen DSOs of that name before
> loading vddk.  But there is an existing bug where we do not duplicate
> properly on soname alone, so we may have to fix that first.

The problem then becomes "given an arbitrary libvixDiskLib.so, how do we 
determine the dependency of bare libraries it would want to load so that 
we can pre-emptively load those libraries by direct path name first". 
My re-exec solution was nice - it works for all versions of VDDK. 
Whereas with scraping the binary to see what DT_NEEDED entries it has is 
a bit more image-specific: for example, VDDK 5.5.5 loads 
"libcrypto.so.1.0.0", VDDK-6.5 loads "libcrypto.so.1.0.2".  I don't want 
to hard-code any of those library names into nbdkit-vddk-plugin.so, 
because that would needlessly tie the open source program into a 
specific VDDK release, instead of working with all of them.  However, if 
it is easy enough to compute our own topological sort of DT_NEEDED for a 
given VDDK .so, and then pre-load in leaf-first order, so that by the 
time we finally get around to dlopen()ing libvixDiskLib.so and calling 
its init function, then all of the subsequent dlopen()s performed by 
VDDK code will succeed right off the bat because the library has already 
been loaded in memory.  Or at least, that's what I'm hoping your 
suggesting.  And if we do that, it would avoid the need to re-exec with 
LD_LIBRARY_PATH set.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?

[Eric Blake]

On 2/21/20 11:42 AM, Richard W.M. Jones wrote:

>>> To be clear, the situation is:
>>>
>>>    nbdkit (free)
>>>     -> dlopens nbdkit-vddk-plugin.so (free)
>>>       -> dlopens libvixDiskLib.so (proprietary)
>>>         -> dlopens other proprietary plugins
>>>         -> both libvixDiskLib.so and its plugins have DT_NEEDED
>>>            "libstdc++.so.X" and other objects that have odd/old
>>>            compiled versions in its own directory
>>>
>>> It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
>>> which has trouble opening its own plugins.  I guess you mean adding
>>> DT_* to the proprietary library?
>>
>> No, to nbdkit-vddk-plugin.so.
>>
>> But it looks like have misunderstood the request.
>>
>> Do you want to inhibit loading the libstdc++.so.X from vddk, or the
>> opposite—ensure that it is loaded?  The latter obviously taints the
>> process.  But maybe that's what you want?
> 
> Ideally load it but only make it available to VDDK.  I think Eric was
> working on something along these lines, using dlmopen.

The nbdkit process having its namespace tainted by VDDK code is (so far) 
a non-issue, as long as the libraries that VDDK ships that override 
system libraries do not intersect with the set of libraries used by 
nbdkit itself.  (For example, VDDK overrides libstdc++.so, but since 
nbdkit is written in C rather than C++, we don't care what C++ stuff 
VDDK drags into the process).  But you are right that if, down the road, 
a future VDDK release poisons even more system libraries, we'd NEED to 
use dlmopen() to isolate the real system library interfaces used by 
nbdkit from the out-of-date override library interfaces pulled in by 
VDDK's DT_NEEDED.

However, it was my annoying discovery that dlmopen() is still not a 
first-class citizen in glibc/gdb that spawned this whole thread in the 
first place.  While I did have a working example using dlmopen() where I 
was able to hook dlopen() to my heart's content, that hook was not alone 
sufficient to overcome the DT_NEEDED problems, and it made life a lot 
tougher since anything in the dlmopen'd space was undebuggable under gdb.

Loading the leaf dependencies first before libvixDiskLib.so has no 
difference on the amount of tainting to the overall nbdkit process 
compared to letting the leaf dependencies get pulled in via 
LD_LIBRARY_PATH under re-exec.

> 
> We're sort of lucky that none of the libraries that VDDK tries to load
> overlaps with libraries that nbdkit uses (currently).
> 
> Rich.
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org