* alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
@ 2020-02-14 19:02 Eric Blake
2020-02-14 21:29 ` [Libguestfs] " Eric Blake
0 siblings, 1 reply; 15+ messages in thread
From: Eric Blake @ 2020-02-14 19:02 UTC (permalink / raw)
To: libc-help; +Cc: libguestfs
I've got a situation where I need to hook a dlopen() made by VDDK, a
proprietary library, where it passes a relative name expecting to
resolve to a copy of several libraries, including libstdc++.so, that it
installs alongside itself, and fails to load if that resolves to the
system libstdc++.so. The simplest solution of providing LD_LIBRARY_PATH
is enough to load VDDK, but then poisons any child process which
likewise fail to load if they pick up VDDK's libstdc++.so instead of the
system one. Up to now, we've documented throwing the burden on the end
user who has to write convoluted:
LD_LIBRARY_PATH_save=$LD_LIBRARY_PATH \
LD_LIBRARY_PATH=/path/to/vddklibs:$LD_LIBRARY_PATH \
nbdkit vddk libdir=/path/to/vddklibs file --run \
'LD_LIBRARY_PATH=$LD_LIBRARY_PATH_save; program args'
where we would rather the end-user could get away with a more concise:
nbdkit vddk libdir=/path/to/vddklibs file --run 'program args'
Sequentially, we have this scenario:
nbdkit vddk libdir=/path/to/libs file --run 'program args'
- nbdkit binary calls dlopen("/path/to/nbdkit-vddk-plugin.so")
- nbdkit-vddk-plugin.so calls
dlopen("/path/to/libs/libvixDiskLibs.so") using the libdir= argument
to load vddk (rather than dlopen("libvixDiskLibs.so") relying on
LD_LIBRARY_PATH)
- vddk's initializer calls dlopen("libcrypto.so") expecting to
open /path/to/libs/libcrypto.so, but either LD_LIBRARY_PATH
made that possible (at which point we have to scrub it before
a child process will be penalized), or we have to find a way to
rewrite vddk's dlopen call from relative into absolute before
passing it to the real dlopen
- nbdkit binary spawns a child process to exec 'program args'
- program does not want /path/to/libs in its search path
Writing my own dlopen() wrapper directly in nbdkit seems like a
non-starter (my override has to come from a shared library before it can
replace the shared version that would be imported from -ldl, at least
for all subsequent shared library loads that want to bind to the
override). And if I read 'man dlopen' correctly, since nbdkit used
dlopen() to load nbdkit-vddk-plugin.so, then dlopen() is already bound
in the main context, so unless I use RTLD_DEEPBIND from nbdkit, then
nbdkit-vddk-plugin.so will also see dlopen() bound to -dl rather than
anything it loads locally; but even with RTLD_DEEPBIND, it sounds like
that higher precedence lasts only for nbdkit-vddk-plugin.so and does not
extend to later bindings performed for libvixDiskLib.so (which means
vddk is back to -ldl's dlopen, without my hook). Thus, to hook dlopen
within the same process, I need some way to create a scope where I can
provide a shared dlopen() that will take precedence when resolving
symbols during the load of libvixDiskLib.so, but where that hook code
can still defer back to the real dlopen() from -ldl and does not
penalize child processes.
I managed to create a solution that avoids the need to set
LD_PRELOAD_PATH at all by installing a shared library that hooks
dlopen(), then loading both my shim and vddk via dlmopen() without the
use of RTLD_GLOBAL or RTLD_DEEPBIND. More links on my solution:
https://www.redhat.com/archives/libguestfs/2020-February/msg00154.html
https://bugzilla.redhat.com/show_bug.cgi?id=1756307#c7
https://sourceware.org/bugzilla/show_bug.cgi?id=15971#c5
However, when Florian saw it, he suggested that my solution of dlmopen()
for a shim library that overrides dlopen() is reinventing what
la_objsearch() can already do. This is in part because the moment you
dlmopen() a library into a separate namespace, you can't debug it (both
glibc and gdb need additional patches to expose alternative namespaces
for debugging), but there may be other nasty surprises lurking.
But after spending more than an hour playing with la_objsearch() and
reading 'man rtld-audit', it looks like an audit library cannot be
triggered in glibc except by listing it in LD_AUDIT in the environment
during exec - which is back to the same problem we have with needing
LD_LIBRARY_PATH in the environment. Furthermore, although I know that
glibc's audit interface is slightly different from the Solaris version
it copied from, the Solaris documentation states that an audit library
has some rather tough restrictions (including that using 'malloc' is
unsafe,
https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc
"Some system interfaces assume that the interfaces are the only instance
of their implementation within a process. Examples of such
implementations are signals and malloc(3C). Audit libraries should avoid
using such interfaces, as doing so can inadvertently alter the behavior
of the application."). But Solaris also stated that a library could
serve as an audit entry point without LD_AUDIT if it is registered
locally, via -Wl,-paudit.so.1 when creating the shared library
(https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc);
it doesn't seem that this functionality exists with glibc
(/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit).
Does anyone have any ideas on how to let a shared library implement an
audit interface for just its own process, without having to edit
LD_AUDIT or re-exec the process? Or is there yet another way to hook a
program to rewrite misbehaving dlopen() calls without relying on either
dlmopen() or la_objsearch(), or requiring pre-set environment variables,
or having to re-exec a process?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-14 19:02 alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT? Eric Blake
@ 2020-02-14 21:29 ` Eric Blake
2020-02-14 22:20 ` Carlos O'Donell
2020-02-17 15:04 ` Eric Blake
0 siblings, 2 replies; 15+ messages in thread
From: Eric Blake @ 2020-02-14 21:29 UTC (permalink / raw)
To: libc-help; +Cc: libguestfs
On 2/14/20 1:02 PM, Eric Blake wrote:
> Writing my own dlopen() wrapper directly in nbdkit seems like a
> non-starter (my override has to come from a shared library before it can
> replace the shared version that would be imported from -ldl, at least
> for all subsequent shared library loads that want to bind to the
> override).
Maybe I spoke too soon. I've tried another approach that looks like it
will do what I want: put my shim dlopen() in a shared library, but link
nbdkit against that shared library PRIOR to -ldl (so that name lookup
for dlopen resolves there first). The shim library in turn depends on
-ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real
dlopen. And by linking it directly into nbdkit, rather than into the
nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen()
in use for all subsequent loads is from my shim. It's still a bit less
clean than I'd like (it requires tighter coupling between nbdkit and
nbdkit-vddk-plugin.so than what used to exist), but the fact that it
works without dlmopen() or LD_LIBRARY_PATH is in its favor. I'm now
polishing up the experiment, and will post the patch when it's ready.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-14 21:29 ` [Libguestfs] " Eric Blake
@ 2020-02-14 22:20 ` Carlos O'Donell
2020-02-17 15:04 ` Eric Blake
1 sibling, 0 replies; 15+ messages in thread
From: Carlos O'Donell @ 2020-02-14 22:20 UTC (permalink / raw)
To: Eric Blake, libc-help; +Cc: libguestfs
On 2/14/20 4:29 PM, Eric Blake wrote:
> On 2/14/20 1:02 PM, Eric Blake wrote:
>
>> Writing my own dlopen() wrapper directly in nbdkit seems like a
>> non-starter (my override has to come from a shared library before
>> it can replace the shared version that would be imported from -ldl,
>> at least for all subsequent shared library loads that want to bind
>> to the override).
>
> Maybe I spoke too soon. I've tried another approach that looks like
> it will do what I want: put my shim dlopen() in a shared library, but
> link nbdkit against that shared library PRIOR to -ldl (so that name
> lookup for dlopen resolves there first). The shim library in turn
> depends on -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get
> to the real dlopen. And by linking it directly into nbdkit, rather
> than into the nbdkit-vddk-plugin.so that gets loaded later, the first
> bound dlopen() in use for all subsequent loads is from my shim. It's
> still a bit less clean than I'd like (it requires tighter coupling
> between nbdkit and nbdkit-vddk-plugin.so than what used to exist),
> but the fact that it works without dlmopen() or LD_LIBRARY_PATH is in
> its favor. I'm now polishing up the experiment, and will post the
> patch when it's ready.
I think that's the best solution you're going to get.
The alternatives (LD_LIBRARY_PATH, direct loader invocation, dlmopen)
all have limitations that aren't helpful to your particular design.
You have design strategies that look like this:
- Move the object higher in the search order at link time (interposition)
- Investigate static link order.
- Investigate dynamic loader search order
- Change what object is searched for
- LD_LIBRARY_PATH, DT_RPATH, DT_RUNPATH, etc.
- LD_AUDIT with la_objsearch()
Your "shim dlopen()" is a case of moving the static link order
around to ensure your shim is used first.
--
Cheers,
Carlos.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-14 21:29 ` [Libguestfs] " Eric Blake
2020-02-14 22:20 ` Carlos O'Donell
@ 2020-02-17 15:04 ` Eric Blake
2020-02-17 15:12 ` Florian Weimer
1 sibling, 1 reply; 15+ messages in thread
From: Eric Blake @ 2020-02-17 15:04 UTC (permalink / raw)
To: libc-help; +Cc: libguestfs
On 2/14/20 3:29 PM, Eric Blake wrote:
> On 2/14/20 1:02 PM, Eric Blake wrote:
>
>> Writing my own dlopen() wrapper directly in nbdkit seems like a
>> non-starter (my override has to come from a shared library before it
>> can replace the shared version that would be imported from -ldl, at
>> least for all subsequent shared library loads that want to bind to the
>> override).
>
> Maybe I spoke too soon. I've tried another approach that looks like it
> will do what I want: put my shim dlopen() in a shared library, but link
> nbdkit against that shared library PRIOR to -ldl (so that name lookup
> for dlopen resolves there first). The shim library in turn depends on
> -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real
> dlopen. And by linking it directly into nbdkit, rather than into the
> nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen()
> in use for all subsequent loads is from my shim. It's still a bit less
> clean than I'd like (it requires tighter coupling between nbdkit and
> nbdkit-vddk-plugin.so than what used to exist), but the fact that it
> works without dlmopen() or LD_LIBRARY_PATH is in its favor. I'm now
> polishing up the experiment, and will post the patch when it's ready.
Progress report: I've posted a v4 series that relies on a shared library
in the main executable; but I'm still trying to see if I can further
reduce things (maybe with -rdynamic) so that the main binary itself
provides the dlopen() override without needing an auxiliary shared library.
https://www.redhat.com/archives/libguestfs/2020-February/msg00162.html
> But after spending more than an hour playing with la_objsearch() and reading 'man rtld-audit', it looks like an audit library cannot be triggered in glibc except by listing it in LD_AUDIT in the environment during exec - which is back to the same problem we have with needing LD_LIBRARY_PATH in the environment. Furthermore, although I know that glibc's audit interface is slightly different from the Solaris version it copied from, the Solaris documentation states that an audit library has some rather tough restrictions (including that using 'malloc' is unsafe, https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc "Some system interfaces assume that the interfaces are the only instance of their implementation within a process. Examples of such implementations are signals and malloc(3C). Audit libraries should avoid using such interfaces, as doing so can inadvertently alter the behavior of the application."). But Solaris also stated that a library could serve as an audit entry point without LD_AUDIT if it is registered locally, via -Wl,-paudit.so.1 when creating the shared library (https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc); it doesn't seem that this functionality exists with glibc (/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit).
I'm just now noticing that 'man ld' reports that you may pass '--audit
LIB' during linking to add a DT_DEPAUDIT dependency on a library
implementing the audit interface, which sounds like it might be an
alternative to LD_AUDIT for getting a library with la_objsearch() to
actually do something (but doesn't obviate the need for la_obsearch() to
be in a separate library, rather than part of the main executable,
unless a library can be reused as its own audit library...).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-17 15:04 ` Eric Blake
@ 2020-02-17 15:12 ` Florian Weimer
2020-02-18 21:32 ` Eric Blake
0 siblings, 1 reply; 15+ messages in thread
From: Florian Weimer @ 2020-02-17 15:12 UTC (permalink / raw)
To: Eric Blake; +Cc: libc-help, libguestfs
* Eric Blake:
> I'm just now noticing that 'man ld' reports that you may pass '--audit
> LIB' during linking to add a DT_DEPAUDIT dependency on a library
> implementing the audit interface, which sounds like it might be an
> alternative to LD_AUDIT for getting a library with la_objsearch() to
> actually do something (but doesn't obviate the need for la_obsearch()
> to be in a separate library, rather than part of the main executable,
> unless a library can be reused as its own audit library...).
DT_AUDIT support has yet to be implemented in glibc:
<https://sourceware.org/bugzilla/show_bug.cgi?id=24943>
<https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>
If you go on record saying that you need this, maybe someone will review
the patch. Sorry. 8-(
Florian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-17 15:12 ` Florian Weimer
@ 2020-02-18 21:32 ` Eric Blake
2020-02-21 12:19 ` Florian Weimer
0 siblings, 1 reply; 15+ messages in thread
From: Eric Blake @ 2020-02-18 21:32 UTC (permalink / raw)
To: Florian Weimer; +Cc: libc-help, libguestfs
On 2/17/20 9:12 AM, Florian Weimer wrote:
> * Eric Blake:
>
>> I'm just now noticing that 'man ld' reports that you may pass '--audit
>> LIB' during linking to add a DT_DEPAUDIT dependency on a library
>> implementing the audit interface, which sounds like it might be an
>> alternative to LD_AUDIT for getting a library with la_objsearch() to
>> actually do something (but doesn't obviate the need for la_obsearch()
>> to be in a separate library, rather than part of the main executable,
>> unless a library can be reused as its own audit library...).
>
> DT_AUDIT support has yet to be implemented in glibc:
>
> <https://sourceware.org/bugzilla/show_bug.cgi?id=24943>
> <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>
>
> If you go on record saying that you need this, maybe someone will review
> the patch. Sorry. 8-(
Another followup: nbdkit-vddk-plugin.so is now using a re-exec setup
[1], for several reasons:
1. all our other ideas tried so far (such as a dlopen() interposition in
the main nbdkit binary) touched more files and required more exported
APIs; we managed to get re-exec working with changes limited to just the
plugin (other than a minor change to nbdkit to quit messing with argv[]
so that /proc/self/cmdline would stay stable - but that did not require
a new API).
2. it turns out that overriding dlopen() was insufficient to work around
VDDK's setup [2]. It _did_ solve the initial dlopen() performed by
VDDK, but that library in turn had DT_NEEDED entries for bare library
names, which dlopen() does NOT affect but which la_objsearch() should.
3. for nbdkit, we want to minimize the number of binary files shipped;
the re-exec solution works with just the nbdkit binary and the
nbdkit-vddk-plugin.so. Any solution that requires a third file to be
shipped (be that a shared library providing dlopen, or a LD_AUDIT
library, or otherwise) is less palatable than the 2-binary solution that
our re-exec solution provides.
[1] https://github.com/libguestfs/nbdkit/commit/0c7ac4e655b
[2] https://www.redhat.com/archives/libguestfs/2020-February/msg00184.html
So with that said, here's a question I just thought of:
If your patch for glibc support for DT_AUDIT is incorporated, is it
possible to mark a shared library as its own audit library via DT_AUDIT?
That is, if nbdkit-vddk-plugin.so can provide entry points for _both_
the nbdkit interface (which satisfies dlopen() from the nbdkit binary)
and la_version/la_objsearch() (which satisfy the requirements for use
from the audit code in ld.so), _and_ during the compilation of
nbdkit-vddk-plugin.so, we marked the library as its own DT_AUDIT entry,
would the mere act of dlopen("nbdkit-vddk-plugin.so") from nbdkit be
sufficient to trigger audit actions such as la_objsearch() on all
subsequent shared loads (whether by dlopen() or DT_NEEDED) performed by
nbdkit-vddk-plugin.so and its descendant loaded libraries? Because if
so, we would have a use case where a single binary, set up to act as its
own audit library, might be sufficient to hook the shared object search
path without needing any of environment variable modification, a process
re-exec, or a third shipping binary - in which case that would indeed be
a nicer solution than the current re-exec solution we committed today
(of course, nbdkit would not be able to rely on that solution except on
systems with new enough glibc to support DT_AUDIT).
I guess even without DT_AUDIT support, I could at least answer the
question of whether a single .so can be used to satisfy both dlopen()
and LD_AUDIT interfaces at once by setting LD_AUDIT (where the only
remaining gap is figuring out when glibc can let DT_AUDIT have the same
effect). During my earlier attempts to get a working dlopen() override,
I didn't consider any solution that required setting LD_AUDIT, but now
that I proved dlopen() override alone was not enough for the case at
hand, having to re-exec to set LD_AUDIT is no worse than having to
re-exec to set LD_LIBRARY_PATH as a fallback to systems where glibc does
not support DT_AUDIT.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-18 21:32 ` Eric Blake
@ 2020-02-21 12:19 ` Florian Weimer
2020-02-21 13:26 ` Eric Blake
2020-02-21 14:51 ` Richard W.M. Jones
0 siblings, 2 replies; 15+ messages in thread
From: Florian Weimer @ 2020-02-21 12:19 UTC (permalink / raw)
To: Eric Blake; +Cc: libc-help, libguestfs
* Eric Blake:
> So with that said, here's a question I just thought of:
>
> If your patch for glibc support for DT_AUDIT is incorporated, is it
> possible to mark a shared library as its own audit library via
> DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points
> for _both_ the nbdkit interface (which satisfies dlopen() from the
> nbdkit binary) and la_version/la_objsearch() (which satisfy the
> requirements for use from the audit code in ld.so), _and_ during the
> compilation of nbdkit-vddk-plugin.so, we marked the library as its own
> DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so")
> from nbdkit be sufficient to trigger audit actions such as
> la_objsearch() on all subsequent shared loads (whether by dlopen() or
> DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant
> loaded libraries?
So you want to dlopen nbdkit-vddk-plugin.so and launch a new auditor
even if the process so far hasn't used auditing? And the main program
(which links agains a library which eventually makes this dlopen call)
would not know anything about the existence of this specific plugin and
auditing?
This isn't currently supported. It's not just that the glibc
implementation cannot do it. The audit API (as sketched in <link.h>) is
not a good fit for late loading where you have never observed open
events. It pretty much assumes that auditors are loaded magically
*before* program start, so that they can observe all open calls and set
up their own data structures along the way.
I think what confuses me is that keep talking about a single binary, but
clearly there is this separate vddk DSO, and there is talk of plugins.
So it seems to me that multiple files are involved already?
Thanks,
Florian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 12:19 ` Florian Weimer
@ 2020-02-21 13:26 ` Eric Blake
2020-02-21 14:51 ` Richard W.M. Jones
1 sibling, 0 replies; 15+ messages in thread
From: Eric Blake @ 2020-02-21 13:26 UTC (permalink / raw)
To: Florian Weimer; +Cc: libc-help, libguestfs
On 2/21/20 6:19 AM, Florian Weimer wrote:
> * Eric Blake:
>
>> So with that said, here's a question I just thought of:
>>
>> If your patch for glibc support for DT_AUDIT is incorporated, is it
>> possible to mark a shared library as its own audit library via
>> DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points
>> for _both_ the nbdkit interface (which satisfies dlopen() from the
>> nbdkit binary) and la_version/la_objsearch() (which satisfy the
>> requirements for use from the audit code in ld.so), _and_ during the
>> compilation of nbdkit-vddk-plugin.so, we marked the library as its own
>> DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so")
>> from nbdkit be sufficient to trigger audit actions such as
>> la_objsearch() on all subsequent shared loads (whether by dlopen() or
>> DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant
>> loaded libraries?
>
> So you want to dlopen nbdkit-vddk-plugin.so and launch a new auditor
> even if the process so far hasn't used auditing? And the main program
> (which links agains a library which eventually makes this dlopen call)
> would not know anything about the existence of this specific plugin and
> auditing?
Yes, you interpreted my question correctly.
>
> This isn't currently supported. It's not just that the glibc
> implementation cannot do it. The audit API (as sketched in <link.h>) is
> not a good fit for late loading where you have never observed open
> events. It pretty much assumes that auditors are loaded magically
> *before* program start, so that they can observe all open calls and set
> up their own data structures along the way.
The concern is not about nbdkit loading nbdkit-vddk-plugin.so, but
nbdkit-vddk-plugin.so doing subsequent loads of libvixDiskLib.so and its
bare dependencies on libstdc++.so and such that were incorrectly built
without DT_RUNPATH, but where we can't rewrite libvixDiskLib.so because
it is proprietary, so the best we can do is hook the loading environment
(either by la_objsearch or by re-exec with LD_LIBRARY_PATH).
>
> I think what confuses me is that keep talking about a single binary, but
> clearly there is this separate vddk DSO, and there is talk of plugins.
> So it seems to me that multiple files are involved already?
You are correct that there are multiple files involved:
The nbdkit project currently has 2 relevant files: 'nbdkit' and
'nbdkit-vddk-plugin.so' (and various other plugins, but those are not
relevant to the VDDK use case)
The VDDK project from VMware: multiple files: libvixDiskLib.so (primary
interface), which dlopen()s libdiskLibPlugin.so, which in turn has
DT_NEEDED on libstdc++.so and several other recompiled system libraries.
'find vmware-vix-disklib-distrib/lib64/ -type f | wc' shows 23 libraries
total, but the end user installs it as a single tarball from VMware.
We can't change what VDDK ships, but we want to avoid making the nbdkit
portion change from 2 files into 3, as every additional file required
beyond what VMware ships is that much more burden for a user to choose
to use nbdkit for accessing their VMware disks.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 12:19 ` Florian Weimer
2020-02-21 13:26 ` Eric Blake
@ 2020-02-21 14:51 ` Richard W.M. Jones
2020-02-21 15:00 ` Florian Weimer
1 sibling, 1 reply; 15+ messages in thread
From: Richard W.M. Jones @ 2020-02-21 14:51 UTC (permalink / raw)
To: Florian Weimer; +Cc: Eric Blake, libc-help, libguestfs
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
> I think what confuses me is that keep talking about a single binary, but
> clearly there is this separate vddk DSO, and there is talk of plugins.
> So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins
from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the
path for plugins, and there's a wrapper allowing it to get local
plugins even when it's still in the build directory. Adding another
file would mean another path (or overloading the meaning of the plugin
path) and just makes the whole thing more fragile and complex.
Having said all that, what would also solve this is either an API for
updating LD_LIBRARY_PATH after the program has started; or making
setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
which takes a library path as an extra parameter.
Rich.
* “Why does setenv ("LD_LIBRARY_PATH") not work?” has several
stackoverflow answers. Apparently even the JDK has to work around
this by re-execing.
https://www.google.com/search?q=setenv+%22LD_LIBRARY_PATH%22+site:stackoverflow.com
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 14:51 ` Richard W.M. Jones
@ 2020-02-21 15:00 ` Florian Weimer
2020-02-21 15:40 ` Richard W.M. Jones
2020-02-21 20:21 ` Eric Blake
0 siblings, 2 replies; 15+ messages in thread
From: Florian Weimer @ 2020-02-21 15:00 UTC (permalink / raw)
To: Richard W.M. Jones; +Cc: Eric Blake, libc-help, libguestfs
* Richard W. M. Jones:
> On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
>> I think what confuses me is that keep talking about a single binary, but
>> clearly there is this separate vddk DSO, and there is talk of plugins.
>> So it seems to me that multiple files are involved already?
>
> nbdkit is a standalone binary that happens to be able to load plugins
> from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the
> path for plugins, and there's a wrapper allowing it to get local
> plugins even when it's still in the build directory. Adding another
> file would mean another path (or overloading the meaning of the plugin
> path) and just makes the whole thing more fragile and complex.
>
> Having said all that, what would also solve this is either an API for
> updating LD_LIBRARY_PATH after the program has started; or making
> setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
> which takes a library path as an extra parameter.
Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
Or does the path have to be chosen dynamically?
If you merely want to prevent loading of libstdc++.so or libcrypto.so by
vddk, it may be possible to explicitly dlopen DSOs of that name before
loading vddk. But there is an existing bug where we do not duplicate
properly on soname alone, so we may have to fix that first.
Thanks,
Florian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 15:00 ` Florian Weimer
@ 2020-02-21 15:40 ` Richard W.M. Jones
2020-02-21 16:02 ` Florian Weimer
2020-02-21 20:21 ` Eric Blake
1 sibling, 1 reply; 15+ messages in thread
From: Richard W.M. Jones @ 2020-02-21 15:40 UTC (permalink / raw)
To: Florian Weimer; +Cc: Eric Blake, libc-help, libguestfs
On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
> * Richard W. M. Jones:
>
> > On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
> >> I think what confuses me is that keep talking about a single binary, but
> >> clearly there is this separate vddk DSO, and there is talk of plugins.
> >> So it seems to me that multiple files are involved already?
> >
> > nbdkit is a standalone binary that happens to be able to load plugins
> > from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the
> > path for plugins, and there's a wrapper allowing it to get local
> > plugins even when it's still in the build directory. Adding another
> > file would mean another path (or overloading the meaning of the plugin
> > path) and just makes the whole thing more fragile and complex.
> >
> > Having said all that, what would also solve this is either an API for
> > updating LD_LIBRARY_PATH after the program has started; or making
> > setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
> > which takes a library path as an extra parameter.
>
> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
> Or does the path have to be chosen dynamically?
To be clear, the situation is:
nbdkit (free)
-> dlopens nbdkit-vddk-plugin.so (free)
-> dlopens libvixDiskLib.so (proprietary)
-> dlopens other proprietary plugins
-> both libvixDiskLib.so and its plugins have DT_NEEDED
"libstdc++.so.X" and other objects that have odd/old
compiled versions in its own directory
It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
which has trouble opening its own plugins. I guess you mean adding
DT_* to the proprietary library?
We don't distribute the proprietary library, end users have to choose
to accept the terms of the (very poisonous) license on that library
and so they download and install it themselves and just give us a path
where they installed it.
With all this low quality proprietary software involved I don't expect
there's going to be a clean way to solve this, was really only
wondering if there's a better way than re-execing the whole program.
But we do at least have the re-exec working now.
> If you merely want to prevent loading of libstdc++.so or libcrypto.so by
> vddk, it may be possible to explicitly dlopen DSOs of that name before
> loading vddk. But there is an existing bug where we do not duplicate
> properly on soname alone, so we may have to fix that first.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 15:40 ` Richard W.M. Jones
@ 2020-02-21 16:02 ` Florian Weimer
2020-02-21 17:42 ` Richard W.M. Jones
0 siblings, 1 reply; 15+ messages in thread
From: Florian Weimer @ 2020-02-21 16:02 UTC (permalink / raw)
To: Richard W.M. Jones; +Cc: Eric Blake, libc-help, libguestfs
* Richard W. M. Jones:
> On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
>> * Richard W. M. Jones:
>>
>> > On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
>> >> I think what confuses me is that keep talking about a single binary, but
>> >> clearly there is this separate vddk DSO, and there is talk of plugins.
>> >> So it seems to me that multiple files are involved already?
>> >
>> > nbdkit is a standalone binary that happens to be able to load plugins
>> > from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the
>> > path for plugins, and there's a wrapper allowing it to get local
>> > plugins even when it's still in the build directory. Adding another
>> > file would mean another path (or overloading the meaning of the plugin
>> > path) and just makes the whole thing more fragile and complex.
>> >
>> > Having said all that, what would also solve this is either an API for
>> > updating LD_LIBRARY_PATH after the program has started; or making
>> > setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
>> > which takes a library path as an extra parameter.
>>
>> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
>> Or does the path have to be chosen dynamically?
>
> To be clear, the situation is:
>
> nbdkit (free)
> -> dlopens nbdkit-vddk-plugin.so (free)
> -> dlopens libvixDiskLib.so (proprietary)
> -> dlopens other proprietary plugins
> -> both libvixDiskLib.so and its plugins have DT_NEEDED
> "libstdc++.so.X" and other objects that have odd/old
> compiled versions in its own directory
>
> It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
> which has trouble opening its own plugins. I guess you mean adding
> DT_* to the proprietary library?
No, to nbdkit-vddk-plugin.so.
But it looks like have misunderstood the request.
Do you want to inhibit loading the libstdc++.so.X from vddk, or the
opposite—ensure that it is loaded? The latter obviously taints the
process. But maybe that's what you want?
Thanks,
Florian
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 16:02 ` Florian Weimer
@ 2020-02-21 17:42 ` Richard W.M. Jones
2020-02-21 20:28 ` Eric Blake
0 siblings, 1 reply; 15+ messages in thread
From: Richard W.M. Jones @ 2020-02-21 17:42 UTC (permalink / raw)
To: Florian Weimer; +Cc: Eric Blake, libc-help, libguestfs
On Fri, Feb 21, 2020 at 05:02:12PM +0100, Florian Weimer wrote:
> * Richard W. M. Jones:
>
> > On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
> >> * Richard W. M. Jones:
> >>
> >> > On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
> >> >> I think what confuses me is that keep talking about a single binary, but
> >> >> clearly there is this separate vddk DSO, and there is talk of plugins.
> >> >> So it seems to me that multiple files are involved already?
> >> >
> >> > nbdkit is a standalone binary that happens to be able to load plugins
> >> > from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the
> >> > path for plugins, and there's a wrapper allowing it to get local
> >> > plugins even when it's still in the build directory. Adding another
> >> > file would mean another path (or overloading the meaning of the plugin
> >> > path) and just makes the whole thing more fragile and complex.
> >> >
> >> > Having said all that, what would also solve this is either an API for
> >> > updating LD_LIBRARY_PATH after the program has started; or making
> >> > setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
> >> > which takes a library path as an extra parameter.
> >>
> >> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
> >> Or does the path have to be chosen dynamically?
> >
> > To be clear, the situation is:
> >
> > nbdkit (free)
> > -> dlopens nbdkit-vddk-plugin.so (free)
> > -> dlopens libvixDiskLib.so (proprietary)
> > -> dlopens other proprietary plugins
> > -> both libvixDiskLib.so and its plugins have DT_NEEDED
> > "libstdc++.so.X" and other objects that have odd/old
> > compiled versions in its own directory
> >
> > It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
> > which has trouble opening its own plugins. I guess you mean adding
> > DT_* to the proprietary library?
>
> No, to nbdkit-vddk-plugin.so.
>
> But it looks like have misunderstood the request.
>
> Do you want to inhibit loading the libstdc++.so.X from vddk, or the
> opposite—ensure that it is loaded? The latter obviously taints the
> process. But maybe that's what you want?
Ideally load it but only make it available to VDDK. I think Eric was
working on something along these lines, using dlmopen.
We're sort of lucky that none of the libraries that VDDK tries to load
overlaps with libraries that nbdkit uses (currently).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 17:42 ` Richard W.M. Jones
@ 2020-02-21 20:28 ` Eric Blake
0 siblings, 0 replies; 15+ messages in thread
From: Eric Blake @ 2020-02-21 20:28 UTC (permalink / raw)
To: Richard W.M. Jones, Florian Weimer; +Cc: libc-help, libguestfs
On 2/21/20 11:42 AM, Richard W.M. Jones wrote:
>>> To be clear, the situation is:
>>>
>>> nbdkit (free)
>>> -> dlopens nbdkit-vddk-plugin.so (free)
>>> -> dlopens libvixDiskLib.so (proprietary)
>>> -> dlopens other proprietary plugins
>>> -> both libvixDiskLib.so and its plugins have DT_NEEDED
>>> "libstdc++.so.X" and other objects that have odd/old
>>> compiled versions in its own directory
>>>
>>> It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK")
>>> which has trouble opening its own plugins. I guess you mean adding
>>> DT_* to the proprietary library?
>>
>> No, to nbdkit-vddk-plugin.so.
>>
>> But it looks like have misunderstood the request.
>>
>> Do you want to inhibit loading the libstdc++.so.X from vddk, or the
>> opposite—ensure that it is loaded? The latter obviously taints the
>> process. But maybe that's what you want?
>
> Ideally load it but only make it available to VDDK. I think Eric was
> working on something along these lines, using dlmopen.
The nbdkit process having its namespace tainted by VDDK code is (so far)
a non-issue, as long as the libraries that VDDK ships that override
system libraries do not intersect with the set of libraries used by
nbdkit itself. (For example, VDDK overrides libstdc++.so, but since
nbdkit is written in C rather than C++, we don't care what C++ stuff
VDDK drags into the process). But you are right that if, down the road,
a future VDDK release poisons even more system libraries, we'd NEED to
use dlmopen() to isolate the real system library interfaces used by
nbdkit from the out-of-date override library interfaces pulled in by
VDDK's DT_NEEDED.
However, it was my annoying discovery that dlmopen() is still not a
first-class citizen in glibc/gdb that spawned this whole thread in the
first place. While I did have a working example using dlmopen() where I
was able to hook dlopen() to my heart's content, that hook was not alone
sufficient to overcome the DT_NEEDED problems, and it made life a lot
tougher since anything in the dlmopen'd space was undebuggable under gdb.
Loading the leaf dependencies first before libvixDiskLib.so has no
difference on the amount of tainting to the overall nbdkit process
compared to letting the leaf dependencies get pulled in via
LD_LIBRARY_PATH under re-exec.
>
> We're sort of lucky that none of the libraries that VDDK tries to load
> overlaps with libraries that nbdkit uses (currently).
>
> Rich.
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Libguestfs] alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
2020-02-21 15:00 ` Florian Weimer
2020-02-21 15:40 ` Richard W.M. Jones
@ 2020-02-21 20:21 ` Eric Blake
1 sibling, 0 replies; 15+ messages in thread
From: Eric Blake @ 2020-02-21 20:21 UTC (permalink / raw)
To: Florian Weimer, Richard W.M. Jones; +Cc: libc-help, libguestfs
On 2/21/20 9:00 AM, Florian Weimer wrote:
> * Richard W. M. Jones:
>
>> On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
>>> I think what confuses me is that keep talking about a single binary, but
>>> clearly there is this separate vddk DSO, and there is talk of plugins.
>>> So it seems to me that multiple files are involved already?
>>
>> nbdkit is a standalone binary that happens to be able to load plugins
>> from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the
>> path for plugins, and there's a wrapper allowing it to get local
>> plugins even when it's still in the build directory. Adding another
>> file would mean another path (or overloading the meaning of the plugin
>> path) and just makes the whole thing more fragile and complex.
>>
>> Having said all that, what would also solve this is either an API for
>> updating LD_LIBRARY_PATH after the program has started; or making
>> setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant
>> which takes a library path as an extra parameter.
>
> Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so
Post-processing an existing closed-source .so shipped from an external
vendor might have negative consequences - while it may be possible to
modify the ELF image to add a DT_RUNPATH entry or modify the DT_NEEDED
entries to use anchored names based on $ORIGIN rather than bare names,
there's no telling if such modification would be in violation of
agreements or even cause failure to load if the proprietary code is
using shenanigans like validating a checksum of shipped binaries to
detect tampering.
> Or does the path have to be chosen dynamically?
So, since we cannot fix the existing product, and have no idea if/when
the vendor will release an updated version that has saner libraries, a
dynamic search path is the only option to getting their product to load
(but whether that is done by LD_LIBRARY_PATH, LD_AUDIT, pre-loading
libraries, or something else, is where we have a bit more control).
>
> If you merely want to prevent loading of libstdc++.so or libcrypto.so by
> vddk, it may be possible to explicitly dlopen DSOs of that name before
> loading vddk. But there is an existing bug where we do not duplicate
> properly on soname alone, so we may have to fix that first.
The problem then becomes "given an arbitrary libvixDiskLib.so, how do we
determine the dependency of bare libraries it would want to load so that
we can pre-emptively load those libraries by direct path name first".
My re-exec solution was nice - it works for all versions of VDDK.
Whereas with scraping the binary to see what DT_NEEDED entries it has is
a bit more image-specific: for example, VDDK 5.5.5 loads
"libcrypto.so.1.0.0", VDDK-6.5 loads "libcrypto.so.1.0.2". I don't want
to hard-code any of those library names into nbdkit-vddk-plugin.so,
because that would needlessly tie the open source program into a
specific VDDK release, instead of working with all of them. However, if
it is easy enough to compute our own topological sort of DT_NEEDED for a
given VDDK .so, and then pre-load in leaf-first order, so that by the
time we finally get around to dlopen()ing libvixDiskLib.so and calling
its init function, then all of the subsequent dlopen()s performed by
VDDK code will succeed right off the bat because the library has already
been loaded in memory. Or at least, that's what I'm hoping your
suggesting. And if we do that, it would avoid the need to re-exec with
LD_LIBRARY_PATH set.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2020-02-21 20:28 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-14 19:02 alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT? Eric Blake
2020-02-14 21:29 ` [Libguestfs] " Eric Blake
2020-02-14 22:20 ` Carlos O'Donell
2020-02-17 15:04 ` Eric Blake
2020-02-17 15:12 ` Florian Weimer
2020-02-18 21:32 ` Eric Blake
2020-02-21 12:19 ` Florian Weimer
2020-02-21 13:26 ` Eric Blake
2020-02-21 14:51 ` Richard W.M. Jones
2020-02-21 15:00 ` Florian Weimer
2020-02-21 15:40 ` Richard W.M. Jones
2020-02-21 16:02 ` Florian Weimer
2020-02-21 17:42 ` Richard W.M. Jones
2020-02-21 20:28 ` Eric Blake
2020-02-21 20:21 ` Eric Blake
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).