dlmopen in LD_PRELOAD

dlmopen in LD_PRELOAD

[Baojun Wang]

Hi libc,

Can `dlmopen` be called in a DSO being `LD_PRELOAD`-ed? The idea is to
create a minimal DSO used for `LD_PRELOAD`, then inside the DSO
(.init_array), call `dlmopen` to open the DSO that I'm really interested
in. hence the DSO being `LD_PRELOAD` acts like a mini loader only.

I did exactly above, but ran into issue (segfault) with stack overflow:

```
       ... 47539 frames omitted...

    frame #47540: 0x00007ffff73cd1b0
libc.so.6`__GI___libc_malloc(bytes=160) at malloc.c:3042

    frame #47541: 0x00007ffff7de7b90 ld-linux-x86-64.so.2`tls_get_addr_tail
at dl-tls.c:594

    frame #47542: 0x00007ffff7de7b6c ld-linux-x86-64.so.2`tls_get_addr_tail
at dl-tls.c:607

    frame #47543: 0x00007ffff7de7b5e
ld-linux-x86-64.so.2`tls_get_addr_tail(ti=0x00007ffff6f14940,
dtv=0x0000000000608330, the_map=0x0000000000602330) at dl-tls.c:787

    frame #47544: 0x00007ffff7deea28 ld-linux-x86-64.so.2`__tls_get_addr at
tls_get_addr.S:55

    frame #47545: 0x00007ffff6ce800c

    frame #47546: 0x00007ffff73c790a libc.so.6`arena_get2(size=576,
avoid_arena=0x00007ffff609e0d0) at arena.c:888

    frame #47547: 0x00007ffff73cc54d libc.so.6`tcache_init at arena.c:879

    frame #47548: 0x00007ffff73cc530 libc.so.6`tcache_init at malloc.c:2986

    frame #47549: 0x00007ffff73cd1cb libc.so.6`__GI___libc_malloc at
malloc.c:2983

    frame #47550: 0x00007ffff73cd1b0
libc.so.6`__GI___libc_malloc(bytes=160) at malloc.c:3042

    frame #47551: 0x00007ffff7de7b90 ld-linux-x86-64.so.2`tls_get_addr_tail
at dl-tls.c:594

    frame #47552: 0x00007ffff7de7b6c ld-linux-x86-64.so.2`tls_get_addr_tail
at dl-tls.c:607

    frame #47553: 0x00007ffff7de7b5e
ld-linux-x86-64.so.2`tls_get_addr_tail(ti=0x00007ffff6f14940,
dtv=0x0000000000608330, the_map=0x0000000000602330) at dl-tls.c:787

    frame #47554: 0x00007ffff7deea28 ld-linux-x86-64.so.2`__tls_get_addr at
tls_get_addr.S:55
```

Is this an ideal use case for `dlmopen`? What (went wrong) caused the stack
overflow?

Thanks
Baojun

Re: dlmopen in LD_PRELOAD

[Florian Weimer]

* Baojun Wang:

> Can `dlmopen` be called in a DSO being `LD_PRELOAD`-ed? The idea is to
> create a minimal DSO used for `LD_PRELOAD`, then inside the DSO
> (.init_array), call `dlmopen` to open the DSO that I'm really interested
> in. hence the DSO being `LD_PRELOAD` acts like a mini loader only.
>
> I did exactly above, but ran into issue (segfault) with stack overflow:

Does the library you load via dlmopen contain its own definition of
malloc, perhaps indirectly?

I expect what happens that with regular LD_PRELOAD, its TLS usage gets
promoted to the static TLS allocation, which is allocated by the dynamic
loader by the bootstrap/minimal malloc.  With the indirection through
dlmopen, dynamic TLS will be allocated with malloc, which does not work
if malloc itself uses dynamic TLS.  You can try to build that malloc
with initial-exec TLS, but dlmopen'ing that could then fail because the
static TLS reserve could be exhausted.

Thanks,
Florian

Re: dlmopen in LD_PRELOAD

[Baojun Wang]

> Does the library you load via dlmopen contain its own definition of
malloc, perhaps indirectly?

I don't think so, though the dynamic library does depends on libc.so. To
elaborate, the dynamic library is a rust `cdylib` (`.so`),
rust use system malloc by default, however, rust does use TLS quite
extensively. glibc's malloc uses TLS by default, seems even
malloc-experimental disabled during configure phase.

Maybe `LD_PRELOAD` a customized `libmalloc.so` (without using TLS) could be
an option, though I wouldn't like the idea because malloc itself exports
quite a lot symbols, and the
additional efforts of having to maintain the hacky `libmalloc.so`. Using
exactly the same glibc is a huge plus and I'd really like to stick will
this route.

Also worth mentioning is if I manually patch `__get_nprocs`, which is
called `area_get2`, then I can see the stack overflow any longer. the
pseudo assembly used to patch `__get_nprocs`:

/*

__libc_get_nprocs:

   0:   b8 02 00 00 00          mov    $0x2,%eax

   5:   c3                      retq

*/

Thanks
Baojun


On Mon, Jun 17, 2019 at 2:40 PM Florian Weimer <fweimer@redhat.com> wrote:

> * Baojun Wang:
>
> > Can `dlmopen` be called in a DSO being `LD_PRELOAD`-ed? The idea is to
> > create a minimal DSO used for `LD_PRELOAD`, then inside the DSO
> > (.init_array), call `dlmopen` to open the DSO that I'm really interested
> > in. hence the DSO being `LD_PRELOAD` acts like a mini loader only.
> >
> > I did exactly above, but ran into issue (segfault) with stack overflow:
>
> Does the library you load via dlmopen contain its own definition of
> malloc, perhaps indirectly?
>
> I expect what happens that with regular LD_PRELOAD, its TLS usage gets
> promoted to the static TLS allocation, which is allocated by the dynamic
> loader by the bootstrap/minimal malloc.  With the indirection through
> dlmopen, dynamic TLS will be allocated with malloc, which does not work
> if malloc itself uses dynamic TLS.  You can try to build that malloc
> with initial-exec TLS, but dlmopen'ing that could then fail because the
> static TLS reserve could be exhausted.
>
> Thanks,
> Florian
>

Re: dlmopen in LD_PRELOAD

[Baojun Wang]

 I can  => I cannot

On Mon, Jun 17, 2019 at 2:55 PM Baojun Wang <wangbj@gmail.com> wrote:

> > Does the library you load via dlmopen contain its own definition of
> malloc, perhaps indirectly?
>
> I don't think so, though the dynamic library does depends on libc.so. To
> elaborate, the dynamic library is a rust `cdylib` (`.so`),
> rust use system malloc by default, however, rust does use TLS quite
> extensively. glibc's malloc uses TLS by default, seems even
> malloc-experimental disabled during configure phase.
>
> Maybe `LD_PRELOAD` a customized `libmalloc.so` (without using TLS) could
> be an option, though I wouldn't like the idea because malloc itself exports
> quite a lot symbols, and the
> additional efforts of having to maintain the hacky `libmalloc.so`. Using
> exactly the same glibc is a huge plus and I'd really like to stick will
> this route.
>
> Also worth mentioning is if I manually patch `__get_nprocs`, which is
> called `area_get2`, then I can see the stack overflow any longer. the
> pseudo assembly used to patch `__get_nprocs`:
>
> /*
>
> __libc_get_nprocs:
>
>    0:   b8 02 00 00 00          mov    $0x2,%eax
>
>    5:   c3                      retq
>
> */
>
> Thanks
> Baojun
>
>
> On Mon, Jun 17, 2019 at 2:40 PM Florian Weimer <fweimer@redhat.com> wrote:
>
>> * Baojun Wang:
>>
>> > Can `dlmopen` be called in a DSO being `LD_PRELOAD`-ed? The idea is to
>> > create a minimal DSO used for `LD_PRELOAD`, then inside the DSO
>> > (.init_array), call `dlmopen` to open the DSO that I'm really interested
>> > in. hence the DSO being `LD_PRELOAD` acts like a mini loader only.
>> >
>> > I did exactly above, but ran into issue (segfault) with stack overflow:
>>
>> Does the library you load via dlmopen contain its own definition of
>> malloc, perhaps indirectly?
>>
>> I expect what happens that with regular LD_PRELOAD, its TLS usage gets
>> promoted to the static TLS allocation, which is allocated by the dynamic
>> loader by the bootstrap/minimal malloc.  With the indirection through
>> dlmopen, dynamic TLS will be allocated with malloc, which does not work
>> if malloc itself uses dynamic TLS.  You can try to build that malloc
>> with initial-exec TLS, but dlmopen'ing that could then fail because the
>> static TLS reserve could be exhausted.
>>
>> Thanks,
>> Florian
>>
>

Re: dlmopen in LD_PRELOAD

[Florian Weimer]

* Baojun Wang:

>> Does the library you load via dlmopen contain its own definition of
>> malloc, perhaps indirectly?
>
> I don't think so, though the dynamic library does depends on libc.so.

The malloc loaded from glibc's libc.so.6 should work in this scenario,
and we have tests for basic dlmopen usage to prove this.

> Also worth mentioning is if I manually patch `__get_nprocs`, which is
> called `area_get2`, then I can see the stack overflow any longer. the
> pseudo assembly used to patch `__get_nprocs`:

Hmm.  Do you have a small reproducer which only uses C libraries?

Thanks,
Florian

Re: dlmopen in LD_PRELOAD

[Baojun Wang]

Hi Florian,

(re-send because previous email failed to send due to attachment)

I made a tiny project this issue, it didn't show the same failure, however
it did segfaults inside the library being `dlmopen`-ed.

link: `git clone https://github.com/wangbj/dlns-test.git`
(sorry attachment seems to cause failure to send message)

run.c: an launcher to inject DSOs.
preload.c: a mini loader using LD_PRELOAD -> dlmopen
dlns.c: DSO being `dlmopen`-ed
app.c: app being injected with DSOs.

running `make test` should reproduce the issue. (need to run `ulimit -c
unlimited` to get core dump).

In app.c, it used manual symbol lookups to find symbols in libdlns.so
(built from dlns.c), so it might have broken the total isolation
assumption, maybe this isn't something suppose to happen?

Thanks

On Mon, Jun 17, 2019 at 3:02 PM Florian Weimer <fweimer@redhat.com> wrote:

> * Baojun Wang:
>
> >> Does the library you load via dlmopen contain its own definition of
> >> malloc, perhaps indirectly?
> >
> > I don't think so, though the dynamic library does depends on libc.so.
>
> The malloc loaded from glibc's libc.so.6 should work in this scenario,
> and we have tests for basic dlmopen usage to prove this.
>
> > Also worth mentioning is if I manually patch `__get_nprocs`, which is
> > called `area_get2`, then I can see the stack overflow any longer. the
> > pseudo assembly used to patch `__get_nprocs`:
>
> Hmm.  Do you have a small reproducer which only uses C libraries?
>
> Thanks,
> Florian
>

Re: dlmopen in LD_PRELOAD

[Florian Weimer]

* Baojun Wang:

> Hi Florian,
>
> (re-send because previous email failed to send due to attachment)
>
> I made a tiny project this issue, it didn't show the same failure, however it did segfaults inside the library being `dlmopen`-ed.
>
> link: `git clone https://github.com/wangbj/dlns-test.git`
> (sorry attachment seems to cause failure to send message)
>
> run.c: an launcher to inject DSOs.
> preload.c: a mini loader using LD_PRELOAD -> dlmopen
> dlns.c: DSO being `dlmopen`-ed
> app.c: app being injected with DSOs.
>
> running `make test` should reproduce the issue. (need to run `ulimit
> -c unlimited` to get core dump).

Sorry, for me, it does not.

This is the output I get:

./run libpreload.so libdlns.so ./app
open /tmp/dlns-test/libdlns.so
dlns_init
[link_map] dso: /tmp/dlns-test/libdlns.so, loaded at: 0x7f0ca0d53000, _DYNAMIC: 0x7f0ca0d56de8
[link_map] dso: /lib64/libdl.so.2, loaded at: 0x7f0ca0d4d000, _DYNAMIC: 0x7f0ca0d51d70
[link_map] dso: /lib64/libpthread.so.0, loaded at: 0x7f0ca0b12000, _DYNAMIC: 0x7f0ca0b2dd60
[link_map] dso: /lib64/libc.so.6, loaded at: 0x7f0ca094c000, _DYNAMIC: 0x7f0ca0b0bb60
[link_map] dso: /lib64/ld-linux-x86-64.so.2, loaded at: (nil), _DYNAMIC: (nil)
#1 local_getpid addr = 0x401610
#0 local_getpid addr = 0x401610
#2 local_getpid addr = 0x401610
#3 local_getpid addr = 0x401610
#1 local_getpid: 0
#1 local_getpid: 0
#1 my own tls: 1234
#2 local_getpid: 0
#2 local_getpid: 0
#3 local_getpid: 0
#3 local_getpid: 0
#3 my own tls: 1234
#2 my own tls: 1234
#0 local_getpid: 0
#0 local_getpid: 0
#0 my own tls: 1234

Has your glibc been built with --enable-bind-now?

Thanks,
Florian

Re: dlmopen in LD_PRELOAD

[Baojun Wang]

I believe so, was using glibc-2.27 from ubuntu 18.04 LTS.
Would you try with commit df58c5f3f328110c3e64ebbe667764185909d180 (or git
pull)? I made some mistake in some early commit.

Thanks
Baojun

On Fri, Jun 21, 2019 at 7:53 AM Florian Weimer <fweimer@redhat.com> wrote:

> * Baojun Wang:
>
> > Hi Florian,
> >
> > (re-send because previous email failed to send due to attachment)
> >
> > I made a tiny project this issue, it didn't show the same failure,
> however it did segfaults inside the library being `dlmopen`-ed.
> >
> > link: `git clone https://github.com/wangbj/dlns-test.git`
> <https://github.com/wangbj/dlns-test.git>
> > (sorry attachment seems to cause failure to send message)
> >
> > run.c: an launcher to inject DSOs.
> > preload.c: a mini loader using LD_PRELOAD -> dlmopen
> > dlns.c: DSO being `dlmopen`-ed
> > app.c: app being injected with DSOs.
> >
> > running `make test` should reproduce the issue. (need to run `ulimit
> > -c unlimited` to get core dump).
>
> Sorry, for me, it does not.
>
> This is the output I get:
>
> ./run libpreload.so libdlns.so ./app
> open /tmp/dlns-test/libdlns.so
> dlns_init
> [link_map] dso: /tmp/dlns-test/libdlns.so, loaded at: 0x7f0ca0d53000,
> _DYNAMIC: 0x7f0ca0d56de8
> [link_map] dso: /lib64/libdl.so.2, loaded at: 0x7f0ca0d4d000, _DYNAMIC:
> 0x7f0ca0d51d70
> [link_map] dso: /lib64/libpthread.so.0, loaded at: 0x7f0ca0b12000,
> _DYNAMIC: 0x7f0ca0b2dd60
> [link_map] dso: /lib64/libc.so.6, loaded at: 0x7f0ca094c000, _DYNAMIC:
> 0x7f0ca0b0bb60
> [link_map] dso: /lib64/ld-linux-x86-64.so.2, loaded at: (nil), _DYNAMIC:
> (nil)
> #1 local_getpid addr = 0x401610
> #0 local_getpid addr = 0x401610
> #2 local_getpid addr = 0x401610
> #3 local_getpid addr = 0x401610
> #1 local_getpid: 0
> #1 local_getpid: 0
> #1 my own tls: 1234
> #2 local_getpid: 0
> #2 local_getpid: 0
> #3 local_getpid: 0
> #3 local_getpid: 0
> #3 my own tls: 1234
> #2 my own tls: 1234
> #0 local_getpid: 0
> #0 local_getpid: 0
> #0 my own tls: 1234
>
> Has your glibc been built with --enable-bind-now?
>
> Thanks,
> Florian
>

Re: dlmopen in LD_PRELOAD

[Florian Weimer]

* Baojun Wang:

> I believe so, was using glibc-2.27 from ubuntu 18.04 LTS.  Would you
> try with commit df58c5f3f328110c3e64ebbe667764185909d180 (or git
> pull)? I made some mistake in some early commit.

I already tested with that commit (admittedly on glibc 2.28).  Can you
try something newer?  Maybe it's a Ubuntu-specific patch that causes
this?

Have you set LD_PRELOAD?

Thanks,
Florian

Re: dlmopen in LD_PRELOAD

[Baojun Wang]

Yes LD_PRELOAD was set in execvpe. I tried on a newer platform (Debian
buster) and also couldn't reproduce any crash, will update to you later.

Thanks
Baojun

On Fri, Jun 21, 2019 at 12:11 PM Florian Weimer <fweimer@redhat.com> wrote:

> * Baojun Wang:
>
> > I believe so, was using glibc-2.27 from ubuntu 18.04 LTS.  Would you
> > try with commit df58c5f3f328110c3e64ebbe667764185909d180 (or git
> > pull)? I made some mistake in some early commit.
>
> I already tested with that commit (admittedly on glibc 2.28).  Can you
> try something newer?  Maybe it's a Ubuntu-specific patch that causes
> this?
>
> Have you set LD_PRELOAD?
>
> Thanks,
> Florian
>

Re: dlmopen in LD_PRELOAD

[Florian Weimer]

* Baojun Wang:

> Yes LD_PRELOAD was set in execvpe. I tried on a newer platform (Debian
> buster) and also couldn't reproduce any crash, will update to you later.

I see.  It turns out that Fedora's glibc has a downstream-only fix for
that:

<https://src.fedoraproject.org/rpms/glibc/blob/f30/f/glibc-fedora-__libc_multiple_libcs.patch>

I ended up at this patch for a completely different reason yesterday.
Maybe I can finally upstream a different version of it.

Thanks,
Florian

Re: dlmopen in LD_PRELOAD

[Florian Weimer]

* Baojun Wang:

> Hi Florian,
>
> (re-send because previous email failed to send due to attachment)
>
> I made a tiny project this issue, it didn't show the same failure, however
> it did segfaults inside the library being `dlmopen`-ed.
>
> link: `git clone https://github.com/wangbj/dlns-test.git`
> (sorry attachment seems to cause failure to send message)

May I put this into a tarball and upload it to our Bugzilla?

I fixed a bunch of things (yet to be submitted/committed), and together
with Fedora's __libc_multiple_libcs patch, I'm now stuck at this bug:

  <https://sourceware.org/bugzilla/show_bug.cgi?id=24775>

You can like work around this if you use __thread variables with
__attribute__ ((tls_model ("initial-exec"))) instead
pthread_setspecific, although you won't get a destructor function this
way.

Our implementation of thread_local for C++ has a similar issue, so it's
not a way out unfortunately.

Thanks,
Florian

Re: dlmopen in LD_PRELOAD

[Baojun Wang]

Hi Florian,

No problem to upload it to bugzilla, and thanks a lot for the suggestions.

Best Regards
Baojun

On Fri, Jul 5, 2019 at 2:21 PM Florian Weimer <fweimer@redhat.com> wrote:

> * Baojun Wang:
>
> > Hi Florian,
> >
> > (re-send because previous email failed to send due to attachment)
> >
> > I made a tiny project this issue, it didn't show the same failure,
> however
> > it did segfaults inside the library being `dlmopen`-ed.
> >
> > link: `git clone https://github.com/wangbj/dlns-test.git`
> <https://github.com/wangbj/dlns-test.git>
> > (sorry attachment seems to cause failure to send message)
>
> May I put this into a tarball and upload it to our Bugzilla?
>
> I fixed a bunch of things (yet to be submitted/committed), and together
> with Fedora's __libc_multiple_libcs patch, I'm now stuck at this bug:
>
>   <https://sourceware.org/bugzilla/show_bug.cgi?id=24775>
>
> You can like work around this if you use __thread variables with
> __attribute__ ((tls_model ("initial-exec"))) instead
> pthread_setspecific, although you won't get a destructor function this
> way.
>
> Our implementation of thread_local for C++ has a similar issue, so it's
> not a way out unfortunately.
>
> Thanks,
> Florian
>