[Bug runtime/25853] New: Stap's own dwarf-based stack unwinder does not give very accurate kernel backtraces

[Bug runtime/25853] New: Stap's own dwarf-based stack unwinder does not give very accurate kernel backtraces

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25853

            Bug ID: 25853
           Summary: Stap's own dwarf-based stack unwinder does not give
                    very accurate kernel backtraces
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com
  Target Milestone: ---

Intuitively, stap's own dwarf-based kernel stack unwinder should behave better
than kernel's `dump_stack()` function, but it seems that the opposite is true:

Using the current git master of stap on Fedora 30 x86_64:

```
$ /opt/stap/bin/stap -g -v -e 'probe oneshot { print_backtrace() dump_stack()
}'
Pass 1: parsed user script and 486 library scripts using
378940virt/154300res/8128shr/146396data kb, in 1190usr/40sys/1238real ms.
Pass 2: analyzed script: 1 probe, 3 functions, 0 embeds, 0 globals using
383824virt/159440res/8264shr/151280data kb, in 70usr/0sys/66real ms.
Pass 3: translated to C into
"/tmp/stapiqhPvJ/stap_15a72d9462813416bf28386f595c0677_1270_src.c" using
383824virt/159440res/8264shr/151280data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_15a72d9462813416bf28386f595c0677_1270.ko" in
10840usr/2210sys/2457real ms.
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d kernel'
 0xffffffffc0ace13b [stap_15a72d9462813416bf28386f595c06_108017+0x813b/0x0]
 0xffffffffc0acf68f [stap_15a72d9462813416bf28386f595c06_108017+0x968f/0x0]
 0xffffffffc0acf92e [stap_15a72d9462813416bf28386f595c06_108017+0x992e/0x0]
 0xffffffffc0ad0f13 [stap_15a72d9462813416bf28386f595c06_108017+0xaf13/0x0]
 0xffffffffa042f803
 0xffffffffa042f803 (inexact)
 0xffffffffa030d2c6 (inexact)
 0xffffffffa030d56f (inexact)
 0xffffffffa000438b (inexact)
 0xffffffffa0c0008c (inexact)
Pass 5: run completed in 10usr/30sys/421real ms.

$ dmesg|tail -n13
[4739989.625769] stap_15a72d9462813416bf28386f595c06_108017 (<input>):
systemtap: 4.3/0.177, base: 0000000090fcb81d, memory:
160data/48text/268ctx/2063net/914alloc kb, probes: 1
[4739989.625812] CPU: 55 PID: 108017 Comm: stapio Tainted: G      D    OE    
5.4.19-100.fc30.x86_64 #1
[4739989.625813] Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS
MASTER/TRX40 AORUS MASTER, BIOS F4 01/20/2020
[4739989.625813] Call Trace:
[4739989.625817]  dump_stack+0x66/0x90
[4739989.625821]  probe_19154.cold+0x5/0x7e
[stap_15a72d9462813416bf28386f595c06_108017]
[4739989.625824]  ? enter_be_probe.isra.0.constprop.0+0x11e/0x230
[stap_15a72d9462813416bf28386f595c06_108017]
[4739989.625826]  ? _stp_ctl_write_cmd.cold+0x10b/0x11f8
[stap_15a72d9462813416bf28386f595c06_108017]
[4739989.625828]  ? full_proxy_write+0x53/0x80
[4739989.625830]  ? vfs_write+0xb6/0x1a0
[4739989.625831]  ? ksys_write+0x5f/0xe0
[4739989.625832]  ? do_syscall_64+0x5b/0x1a0
[4739989.625835]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
```

More details:

```
$ /opt/stap/bin/stap -V
Systemtap translator/driver (version 4.3/0.177, commit
release-4.2-65-g6de9136a2794)
Copyright (C) 2005-2019 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.32 ... 5.4-rc6
enabled features: BPF PYTHON3 LIBSQLITE3 NLS READLINE

$ uname -r
5.4.19-100.fc30.x86_64
```

Also tried on an older system, Fedora 28 x86_64. It's the same thing:

```
$ /opt/stap/bin/stap -g -v -e 'probe oneshot { print_backtrace() dump_stack()
}'
Pass 1: parsed user script and 480 library scripts using
332760virt/89532res/7364shr/82644data kb, in 390usr/30sys/420real ms.
Pass 2: analyzed script: 1 probe, 3 functions, 0 embeds, 0 globals using
334344virt/91864res/7884shr/84228data kb, in 10usr/0sys/12real ms.
Pass 3: translated to C into
"/tmp/stap1aI32F/stap_4add63f3ab8162b5a22e27aee890f56b_1270_src.c" using
334344virt/91864res/7884shr/84228data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_4add63f3ab8162b5a22e27aee890f56b_1270.ko" in
11110usr/1740sys/3196real ms.
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d kernel'
 0xffffffffc0ded3ab [stap_4add63f3ab8162b5a22e27aee890f56_22901+0x83ab/0x0]
 0xffffffffc0dee68a [stap_4add63f3ab8162b5a22e27aee890f56_22901+0x968a/0x0]
 0xffffffffc0dee95e [stap_4add63f3ab8162b5a22e27aee890f56_22901+0x995e/0x0]
 0xffffffffc0deff16 [stap_4add63f3ab8162b5a22e27aee890f56_22901+0xaf16/0x0]
 0xffffffff813edbf3
 0xffffffff813edbf3 (inexact)
 0xffffffff812e3b86 (inexact)
 0xffffffff812e3e95 (inexact)
 0xffffffff812e4137 (inexact)
 0xffffffff810041ab (inexact)
 0xffffffff81a0008c (inexact)
Pass 5: run completed in 10usr/20sys/371real ms.

$ dmesg|tail -n15
[66680.072659] CPU: 7 PID: 22901 Comm: stapio Tainted: G           OE    
5.0.16-100.fc28.x86_64 #1
[66680.072659] Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS
MASTER/Z390 AORUS MASTER-CF, BIOS F6 11/08/2018
[66680.072660] Call Trace:
[66680.072662]  dump_stack+0x5c/0x80
[66680.072666]  probe_6406.cold.78+0x5/0x7e
[stap_4add63f3ab8162b5a22e27aee890f56_22901]
[66680.072667]  ? enter_be_probe.isra.35.constprop.67+0x11e/0x230
[stap_4add63f3ab8162b5a22e27aee890f56_22901]
[66680.072669]  ? _stp_ctl_write_cmd.cold.79+0x5a/0x1144
[stap_4add63f3ab8162b5a22e27aee890f56_22901]
[66680.072671]  ? full_proxy_write+0x53/0x80
[66680.072672]  ? __vfs_write+0x36/0x1b0
[66680.072673]  ? selinux_file_permission+0xf0/0x130
[66680.072674]  ? security_file_permission+0x2e/0xf0
[66680.072675]  ? vfs_write+0xa5/0x1a0
[66680.072676]  ? ksys_write+0x57/0xd0
[66680.072677]  ? do_syscall_64+0x5b/0x160
[66680.072678]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
```

More details:

```
$ /opt/stap/bin/stap -V
Systemtap translator/driver (version 4.3/0.177, commit
release-4.2-65-g6de9136a2794)
Copyright (C) 2005-2019 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.32 ... 5.4-rc6
enabled features: BPF LIBSQLITE3 NLS

$ uname -r
5.0.16-100.fc28.x86_64
```

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/25853] Stap's own dwarf-based stack unwinder does not give very accurate kernel backtraces

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25853

--- Comment #1 from agentzh <agentzh at gmail dot com> ---
Adding the '-d kernel' option is not very helpful when the kernel debuginfo is
missing in the system:

```
$ /opt/stap/bin/stap -d kernel -e 'probe oneshot { print_backtrace() }'
 0xffffffffc123c8cb [stap_e4014a9c5a0decc7f01b4bb37a893a9_29973+0x88cb/0x0]
 0xffffffffc123c84a [stap_e4014a9c5a0decc7f01b4bb37a893a9_29973+0x884a/0x0]
(inexact)
```

-- 
You are receiving this mail because:
You are the assignee for the bug.