[Bug dynamic-link/31991] New: Using LD_AUDIT with LD_PRELOAD and TLS segfaults

[Bug dynamic-link/31991] New: Using LD_AUDIT with LD_PRELOAD and TLS segfaults

[matthewkenigsberg at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

            Bug ID: 31991
           Summary: Using LD_AUDIT with LD_PRELOAD and TLS segfaults
           Product: glibc
           Version: 2.31
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: dynamic-link
          Assignee: unassigned at sourceware dot org
          Reporter: matthewkenigsberg at gmail dot com
  Target Milestone: ---

When a library loaded with LD_PRELOAD uses sufficient TLS and an audit library
is also loaded with LD_AUDIT, the program being run segfaults with:
./main: error while loading shared libraries: ./preload.so: cannot allocate
memory in static TLS block

This was originally reported in a jemalloc issue:
https://github.com/jemalloc/jemalloc/issues/2472

The case reported there is LD_PRELOAD=$MY_SCRATCH_DIR/mylibs/lib/libjemalloc.so
LD_AUDIT=libsimple.so ls

libjemalloc uses a large enough amount of TLS, so ls segfaults

When I tried to extract a more minimal reproducer, it seems jemalloc does not
have to be involved at all, so I'm wondering if this is actually a bug in
glibc.

I don't see the bug if the amount of TLS is small. If I change 465 to 464
below, I don't see the segfault.

Here's a minimal reproducer:

cat <<EOF > preload.c
__thread int preload_data[465] __attribute__((tls_model("initial-exec")));

int preload_function() {
    return preload_data[0];
}
EOF

gcc -shared preload.c -o preload.so

cat <<EOF > audit.c
unsigned int
la_version( unsigned int version )
{
  return version;
}
EOF

gcc -shared audit.c -o audit.so

cat <<EOF > main.c
#include <stdio.h>

int main()
{
    printf("Hello, World!\n");
    return 0;
}
EOF

gcc main.c -o main

./main # succeeds
LD_PRELOAD=./preload.so ./main # succeeds
LD_AUDIT=./audit.so ./main # succeeds
LD_PRELOAD=./preload.so LD_AUDIT=./audit.so ./main # segfaults

I'm running on aarch64 Debian 5.10 with glibc 2.31. gcc --version gives:
gcc (Debian 10.2.1-6) 10.2.1 20210110

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Using LD_AUDIT with LD_PRELOAD and TLS segfaults

[akostadinov at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

Aleksandar Kostadinov <akostadinov at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |akostadinov at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-
            Summary|Using LD_AUDIT with         |Insufficient static TLS
                   |LD_PRELOAD and TLS          |reservation when auditing
                   |segfaults                   |leads to process start
                   |                            |failure
                 CC|                            |fweimer at redhat dot com

--- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
Does your actual auditor define la_objsearch?

This is not a segmentation fault, ld.so simply exists. In auditing mode, we
must allocate the TCB early because the auditor needs it. We do not know the
static TLS requirements of the application at this point. If there are auditors
with la_objsearch defined, the requirements are in fact impossible to know at
this point.

It's possible to increase the static TLS allocation by setting a tunable:

export GLIBC_TUNABLES=glibc.rtld.optional_static_tls=10000

Given the la_objsearch problem, the only thing we might be able to do here is
to reserve plenty of address space for the TCB of the main thread, so that we
can grow it in place if need be.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[matthewkenigsberg at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #2 from Matthew Kenigsberg <matthewkenigsberg at gmail dot com> ---
Thanks for the response!

Our actual auditor does define la_objsearch.

Oh right, the jemalloc issue does describe a segfault when a different
tls_model is used, but that's not the case in the example I gave.

Looks like setting GLIBC_TUNABLES does fix the issue for the minimal
reproducer! I can look into whether it would be possible to compile jemalloc
with that set.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[matthewkenigsberg at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #3 from Matthew Kenigsberg <matthewkenigsberg at gmail dot com> ---
We would like to set LD_AUDIT for all programs we're running, so we can't
really know the TLS size of everything we're running. So we don't know what to
set GLIBC_TUNABLES to. Since you're saying TCB has to be allocated before
la_obsearch, does that just mean auditing something with unknown TLS size isn't
something that is supported?

Also, once I set GLIBC_TUNABLES, I see segfaults when using Debian's
libjemalloc (which is used by Debian's redis-server). That may just be a
jemalloc issue though.

$ LD_AUDIT=./audit.so redis-server
redis-server: error while loading shared libraries:
/lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS
block

$ GLIBC_TUNABLES=glibc.rtld.optional_static_tls=10000 LD_AUDIT=./audit.so
redis-server
Segmentation fault (core dumped)


Cutting redis out of the equation (but still using libjemalloc), I see both the
exit and then the segfault:

cat <<EOF > main.c
#include <stdio.h>

int main()
{
    xallocx(0, 0, 0, 0);
    printf("Hello, World!\n");
    return 0;
}
EOF

gcc main.c -ljemalloc -o main


$ GLIBC_TUNABLES= LD_AUDIT=./audit.so ./main
./main: error while loading shared libraries:
/lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS
block
$ GLIBC_TUNABLES=glibc.rtld.optional_static_tls=10000 LD_AUDIT=./audit.so
./main
Segmentation fault (core dumped)

The stack trace is:
                Stack trace of thread 18000:
                #0  0x0000ffff8ed79160 __GI___get_nprocs (libc.so.6 + 0xd1160)
                #1  0x0000ffff8ed4dfa4 posix_sysconf (libc.so.6 + 0xa5fa4)
                #2  0x0000ffff8ee25810 n/a (libjemalloc.so.2 + 0x9810)
                #3  0x0000ffff8ee29650 calloc (libjemalloc.so.2 + 0xd650)
                #4  0x0000ffff8ef69770 n/a (n/a + 0x0)
                #5  0x0000ffff8ef69770 n/a (n/a + 0x0)
                #6  0x0000ffff8ef61604 n/a (n/a + 0x0)
                #7  0x0000ffff8ef72f44 n/a (n/a + 0x0)
                #8  0x0000ffff8ef5ec5c n/a (n/a + 0x0)
                #9  0x0000ffff8ef5ee9c n/a (n/a + 0x0)
                #10 0x0000ffff8ef5e148 n/a (n/a + 0x0)


If I get rid of libjemalloc, I don't see the segfault. So I'm not really sure,
but maybe there are two different issues? Maybe the segfault has something to
do with calloc being redefined? And there's the separate issue of the size of
the static TLS allocation?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #4 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Matthew Kenigsberg from comment #3)
> We would like to set LD_AUDIT for all programs we're running, so we can't
> really know the TLS size of everything we're running. So we don't know what
> to set GLIBC_TUNABLES to. Since you're saying TCB has to be allocated before
> la_obsearch, does that just mean auditing something with unknown TLS size
> isn't something that is supported?

It's supposed to work, but right now it's not. As I said before, we should
probably throw address space at the problem, and create a generous reservation
for the main thread (initially not even backed by memory). It's much simpler
than an alternative that involves a protocol for unloading TLS underneath
auditors.

> Also, once I set GLIBC_TUNABLES, I see segfaults when using Debian's
> libjemalloc (which is used by Debian's redis-server). That may just be a
> jemalloc issue though.

It's a different issue. Try getting a more complete backtrace. It could be
caused by sysconf using malloc, and jemalloc calling sysconf. Not sure why it
only happens with auditing enabled.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[matthewkenigsberg at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #5 from Matthew Kenigsberg <matthewkenigsberg at gmail dot com> ---
(In reply to Florian Weimer from comment #4)

> It's supposed to work, but right now it's not. As I said before, we should
> probably throw address space at the problem, and create a generous
> reservation for the main thread (initially not even backed by memory). It's
> much simpler than an alternative that involves a protocol for unloading TLS
> underneath auditors.

Is doing that something you could point me in the direction of patching, or
would that be pretty involved?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #6 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Matthew Kenigsberg from comment #5)
> (In reply to Florian Weimer from comment #4)
> 
> > It's supposed to work, but right now it's not. As I said before, we should
> > probably throw address space at the problem, and create a generous
> > reservation for the main thread (initially not even backed by memory). It's
> > much simpler than an alternative that involves a protocol for unloading TLS
> > underneath auditors.
> 
> Is doing that something you could point me in the direction of patching, or
> would that be pretty involved?

I expect the changes to be concentrated in elf/dl-tls.c, touching code in the
vicinity of:

[PATCH v12 00/17] Add rseq extensible ABI support
<https://inbox.sourceware.org/libc-alpha/20240801-rseq-abi-v11-split-v12-0-0e87479dddc0@efficios.com/T/#m9836dd9bea43ff814adb6a0c6af05af64de2c3bc>

While the change itself is fairly localized, the thread descriptor layout is
not really transparent (at least to me). I think you'd have to figure out in
which direction the static TLS area grows (depending on architecture), and then
find the mmap call that allocates TLS (perhaps with strace -k or a strategic
breakpoint), and add padding at the right end (map first with MAP_NONE, then
change the to-be-used part to writable with mprotect). Then before the static
TLS exhaustion message is printed, get more memory with mprotect and update the
static TLS bookkeeping information, unless we are already multi-threaded. In
this case we cannot resize the non-main thread TLS areas, and we still have to
fail.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #7 from Florian Weimer <fweimer at redhat dot com> ---
This seems to be the  TLS allocation for the main thread:

mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f
30ce41a000
 > /usr/lib64/ld-linux-x86-64.so.2(__mmap+0x2c) [0x2357c]
 > /usr/lib64/ld-linux-x86-64.so.2(__minimal_malloc+0xb0) [0xb8f0]
 > /usr/lib64/ld-linux-x86-64.so.2(_dl_allocate_tls_storage+0x29) [0x12729]
 > /usr/lib64/ld-linux-x86-64.so.2(init_tls+0xef) [0x1dcbf]
 > /usr/lib64/ld-linux-x86-64.so.2(dl_main+0x2953) [0x20fb3]
 > /usr/lib64/ld-linux-x86-64.so.2(_dl_sysdep_start+0x85) [0x1c9f5]
 > /usr/lib64/ld-linux-x86-64.so.2(_dl_start+0x5dd) [0x1e35d]
 > /usr/lib64/ld-linux-x86-64.so.2(_start+0x7) [0x1d047]
 > no matching address range

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/31991] Insufficient static TLS reservation when auditing leads to process start failure

[matthewkenigsberg at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=31991

--- Comment #8 from Matthew Kenigsberg <matthewkenigsberg at gmail dot com> ---
Thanks for the explanation!

-- 
You are receiving this mail because:
You are on the CC list for the bug.