[Bug runtime/23876] New: User backtraces get truncated at frame 0x7fffffffe000 with function probes

[Bug runtime/23876] New: User backtraces get truncated at frame 0x7fffffffe000 with function probes

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23876

            Bug ID: 23876
           Summary: User backtraces get truncated at frame 0x7fffffffe000
                    with function probes
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com
  Target Milestone: ---

The userspace backtraces often get truncated at a fixed frame 0x7fffffffe000
when a lot of function probes are enabled.

To reproduce this issue. We first prepare a minimal C program like below:

```C
int foo(int a) {
    a++;
    return a + 1;
}

int bar(int b) {
    b++;
    return foo(b + 1);
}

int main(void) {
    bar(1);
    return 0;
}
```

Then compile it with gcc:

```
gcc -g a.c
```

And then prepare the following stap script:

```stap
probe process.function("bar"), process.function("foo") {
    printf("\nenter %s\n", probefunc());
    print_ubacktrace();
}

probe process.function("bar").return, process.function("foo").return {
    printf("\nreturn %s\n", probefunc());
    print_ubacktrace();
}
```

Run the stap script with the C program:

```
$ stap/bin/stap a.stp -c ./a.out -d /lib64/libc.so.6

enter bar
 0x4004a5 : bar+0xb/0x1e [/mnt/home/agentzh/git/ylang/a.out]
 0x7fffffffe000

enter foo
 0x40048e : foo+0x7/0x13 [/mnt/home/agentzh/git/ylang/a.out]
 0x7fffffffe000

return bar
 0x4004b6 : bar+0x1c/0x1e [/mnt/home/agentzh/git/ylang/a.out]
 0x7fffffffe000

return main
 0x4004c6 : main+0xe/0x18 [/mnt/home/agentzh/git/ylang/a.out]
 0x7fc6e8f8cfea : __libc_start_main+0xea/0x1c0 [/usr/lib64/libc-2.26.so]
 0x4003da : _start+0x2a/0x30 [/mnt/home/agentzh/git/ylang/a.out]
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry
with -DDEBUG_UNWIND)'
```

If we remove the return probes from the script:

```stap
probe process.function("bar"), process.function("foo") {
    printf("\nenter %s\n", probefunc());
    print_ubacktrace();
}

```

Then the backtraces in the output are perfect:

```
$ stap b.stp -c ./a.out -d /lib64/libc.so.6

enter bar
 0x4004a5 : bar+0xb/0x1e [/mnt/home/agentzh/git/ylang/a.out]
 0x4004c6 : main+0xe/0x18 [/mnt/home/agentzh/git/ylang/a.out]
 0x7fba984d2fea : __libc_start_main+0xea/0x1c0 [/usr/lib64/libc-2.26.so]
 0x4003da : _start+0x2a/0x30 [/mnt/home/agentzh/git/ylang/a.out]

enter foo
 0x40048e : foo+0x7/0x13 [/mnt/home/agentzh/git/ylang/a.out]
 0x4004b6 : bar+0x1c/0x1e [/mnt/home/agentzh/git/ylang/a.out]
 0x4004c6 : main+0xe/0x18 [/mnt/home/agentzh/git/ylang/a.out]
 0x7fba984d2fea : __libc_start_main+0xea/0x1c0 [/usr/lib64/libc-2.26.so]
 0x4003da : _start+0x2a/0x30 [/mnt/home/agentzh/git/ylang/a.out]
```

I'm using the latest git master branch as of this writing (commit 28cf23f8e).

I'm on Fedora 27 x86_64 (kernel 4.16.16-200.fc27.x86_64).

More info:

```
$ stap -V
Systemtap translator/driver (version 4.1/0.173, commit
release-4.0-56-g28cf23f8e593 + changes)
Copyright (C) 2005-2018 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.19-rc7
enabled features: AVAHI BOOST_STRING_REF DYNINST BPF PYTHON3 NLS NSS

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap
--enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr
--mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared
--enable-threads=posix --enable-checking=release --enable-multilib
--with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-linker-build-id
--with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin
--enable-initfini-array --with-isl --enable-libmpx
--enable-offload-targets=nvptx-none --without-cuda-driver
--enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Thread model: posix
gcc version 7.3.1 20180712 (Red Hat 7.3.1-6) (GCC)

$ uname -a
Linux work2 4.16.16-200.fc27.x86_64 #1 SMP Sun Jun 17 03:06:00 UTC 2018 x86_64
x86_64 x86_64 GNU/Linux
```

I tried setting breakpoints on all these C functions' entries and return points
and gdb can successfully get the full backtraces without issues.

Any hints on how to fix this will be highly appreciated. Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/23876] User backtraces get truncated at frame 0x7fffffffe000 with function probes

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23876

--- Comment #1 from agentzh <agentzh at gmail dot com> ---
I'm also seeing this in target C programs compiled with gcc optimization
enabled (like -Og, -O1, and -O2). Though it is harder for me to create a
minimal C program that won't be optimized to nothing for demonstrating this
stap stack unwinding issue.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/23876] User backtraces get truncated at frame 0x7fffffffe000 with function probes

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23876

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
                 CC|                            |fche at redhat dot com
         Resolution|---                         |DUPLICATE

--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
unfortunately, an old known problem

*** This bug has been marked as a duplicate of bug 6436 ***

-- 
You are receiving this mail because:
You are the assignee for the bug.