From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id BDED3385B835; Sat, 18 Apr 2020 19:51:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BDED3385B835 From: "agentzh at gmail dot com" To: systemtap@sourceware.org Subject: [Bug runtime/25853] New: Stap's own dwarf-based stack unwinder does not give very accurate kernel backtraces Date: Sat, 18 Apr 2020 19:51:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: systemtap X-Bugzilla-Component: runtime X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: agentzh at gmail dot com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: systemtap at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: systemtap@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Systemtap mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Apr 2020 19:51:25 -0000 https://sourceware.org/bugzilla/show_bug.cgi?id=3D25853 Bug ID: 25853 Summary: Stap's own dwarf-based stack unwinder does not give very accurate kernel backtraces Product: systemtap Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: runtime Assignee: systemtap at sourceware dot org Reporter: agentzh at gmail dot com Target Milestone: --- Intuitively, stap's own dwarf-based kernel stack unwinder should behave bet= ter than kernel's `dump_stack()` function, but it seems that the opposite is tr= ue: Using the current git master of stap on Fedora 30 x86_64: ``` $ /opt/stap/bin/stap -g -v -e 'probe oneshot { print_backtrace() dump_stack= () }' Pass 1: parsed user script and 486 library scripts using 378940virt/154300res/8128shr/146396data kb, in 1190usr/40sys/1238real ms. Pass 2: analyzed script: 1 probe, 3 functions, 0 embeds, 0 globals using 383824virt/159440res/8264shr/151280data kb, in 70usr/0sys/66real ms. Pass 3: translated to C into "/tmp/stapiqhPvJ/stap_15a72d9462813416bf28386f595c0677_1270_src.c" using 383824virt/159440res/8264shr/151280data kb, in 0usr/0sys/0real ms. Pass 4: compiled C into "stap_15a72d9462813416bf28386f595c0677_1270.ko" in 10840usr/2210sys/2457real ms. Pass 5: starting run. WARNING: Missing unwind data for a module, rerun with 'stap -d kernel' 0xffffffffc0ace13b [stap_15a72d9462813416bf28386f595c06_108017+0x813b/0x0] 0xffffffffc0acf68f [stap_15a72d9462813416bf28386f595c06_108017+0x968f/0x0] 0xffffffffc0acf92e [stap_15a72d9462813416bf28386f595c06_108017+0x992e/0x0] 0xffffffffc0ad0f13 [stap_15a72d9462813416bf28386f595c06_108017+0xaf13/0x0] 0xffffffffa042f803 0xffffffffa042f803 (inexact) 0xffffffffa030d2c6 (inexact) 0xffffffffa030d56f (inexact) 0xffffffffa000438b (inexact) 0xffffffffa0c0008c (inexact) Pass 5: run completed in 10usr/30sys/421real ms. $ dmesg|tail -n13 [4739989.625769] stap_15a72d9462813416bf28386f595c06_108017 (): systemtap: 4.3/0.177, base: 0000000090fcb81d, memory: 160data/48text/268ctx/2063net/914alloc kb, probes: 1 [4739989.625812] CPU: 55 PID: 108017 Comm: stapio Tainted: G D OE= =20=20=20=20 5.4.19-100.fc30.x86_64 #1 [4739989.625813] Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS MASTER/TRX40 AORUS MASTER, BIOS F4 01/20/2020 [4739989.625813] Call Trace: [4739989.625817] dump_stack+0x66/0x90 [4739989.625821] probe_19154.cold+0x5/0x7e [stap_15a72d9462813416bf28386f595c06_108017] [4739989.625824] ? enter_be_probe.isra.0.constprop.0+0x11e/0x230 [stap_15a72d9462813416bf28386f595c06_108017] [4739989.625826] ? _stp_ctl_write_cmd.cold+0x10b/0x11f8 [stap_15a72d9462813416bf28386f595c06_108017] [4739989.625828] ? full_proxy_write+0x53/0x80 [4739989.625830] ? vfs_write+0xb6/0x1a0 [4739989.625831] ? ksys_write+0x5f/0xe0 [4739989.625832] ? do_syscall_64+0x5b/0x1a0 [4739989.625835] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 ``` More details: ``` $ /opt/stap/bin/stap -V Systemtap translator/driver (version 4.3/0.177, commit release-4.2-65-g6de9136a2794) Copyright (C) 2005-2019 Red Hat, Inc. and others This is free software; see the source for copying conditions. tested kernel versions: 2.6.32 ... 5.4-rc6 enabled features: BPF PYTHON3 LIBSQLITE3 NLS READLINE $ uname -r 5.4.19-100.fc30.x86_64 ``` Also tried on an older system, Fedora 28 x86_64. It's the same thing: ``` $ /opt/stap/bin/stap -g -v -e 'probe oneshot { print_backtrace() dump_stack= () }' Pass 1: parsed user script and 480 library scripts using 332760virt/89532res/7364shr/82644data kb, in 390usr/30sys/420real ms. Pass 2: analyzed script: 1 probe, 3 functions, 0 embeds, 0 globals using 334344virt/91864res/7884shr/84228data kb, in 10usr/0sys/12real ms. Pass 3: translated to C into "/tmp/stap1aI32F/stap_4add63f3ab8162b5a22e27aee890f56b_1270_src.c" using 334344virt/91864res/7884shr/84228data kb, in 0usr/0sys/0real ms. Pass 4: compiled C into "stap_4add63f3ab8162b5a22e27aee890f56b_1270.ko" in 11110usr/1740sys/3196real ms. Pass 5: starting run. WARNING: Missing unwind data for a module, rerun with 'stap -d kernel' 0xffffffffc0ded3ab [stap_4add63f3ab8162b5a22e27aee890f56_22901+0x83ab/0x0] 0xffffffffc0dee68a [stap_4add63f3ab8162b5a22e27aee890f56_22901+0x968a/0x0] 0xffffffffc0dee95e [stap_4add63f3ab8162b5a22e27aee890f56_22901+0x995e/0x0] 0xffffffffc0deff16 [stap_4add63f3ab8162b5a22e27aee890f56_22901+0xaf16/0x0] 0xffffffff813edbf3 0xffffffff813edbf3 (inexact) 0xffffffff812e3b86 (inexact) 0xffffffff812e3e95 (inexact) 0xffffffff812e4137 (inexact) 0xffffffff810041ab (inexact) 0xffffffff81a0008c (inexact) Pass 5: run completed in 10usr/20sys/371real ms. $ dmesg|tail -n15 [66680.072659] CPU: 7 PID: 22901 Comm: stapio Tainted: G OE=20=20= =20=20 5.0.16-100.fc28.x86_64 #1 [66680.072659] Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS MASTER/Z390 AORUS MASTER-CF, BIOS F6 11/08/2018 [66680.072660] Call Trace: [66680.072662] dump_stack+0x5c/0x80 [66680.072666] probe_6406.cold.78+0x5/0x7e [stap_4add63f3ab8162b5a22e27aee890f56_22901] [66680.072667] ? enter_be_probe.isra.35.constprop.67+0x11e/0x230 [stap_4add63f3ab8162b5a22e27aee890f56_22901] [66680.072669] ? _stp_ctl_write_cmd.cold.79+0x5a/0x1144 [stap_4add63f3ab8162b5a22e27aee890f56_22901] [66680.072671] ? full_proxy_write+0x53/0x80 [66680.072672] ? __vfs_write+0x36/0x1b0 [66680.072673] ? selinux_file_permission+0xf0/0x130 [66680.072674] ? security_file_permission+0x2e/0xf0 [66680.072675] ? vfs_write+0xa5/0x1a0 [66680.072676] ? ksys_write+0x57/0xd0 [66680.072677] ? do_syscall_64+0x5b/0x160 [66680.072678] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 ``` More details: ``` $ /opt/stap/bin/stap -V Systemtap translator/driver (version 4.3/0.177, commit release-4.2-65-g6de9136a2794) Copyright (C) 2005-2019 Red Hat, Inc. and others This is free software; see the source for copying conditions. tested kernel versions: 2.6.32 ... 5.4-rc6 enabled features: BPF LIBSQLITE3 NLS $ uname -r 5.0.16-100.fc28.x86_64 ``` --=20 You are receiving this mail because: You are the assignee for the bug.=