[Bug runtime/15664] New: Kernel crash when using ubacktrace/print_ubacktrace in timer.profile on kernel 3.9.4

[Bug runtime/15664] New: Kernel crash when using ubacktrace/print_ubacktrace in timer.profile on kernel 3.9.4

[agentzh at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15664

            Bug ID: 15664
           Summary: Kernel crash when using ubacktrace/print_ubacktrace in
                    timer.profile on kernel 3.9.4
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com

We've seen a kernel crash on an x86_64 box with kernel 3.9.4 installed when
running a stap script using ubacktrace/print_ubacktrace in probe timer.profile.

The output of our stap script before the kernel crash:

  WARNING: Tracing 29685 (/usr/local/bin/load-logs) in user-space only...
  WARNING: Missing unwind data for module, rerun with 'stap -d
/usr/local/lib/libmsgpackc.so.2.0.0'
  WARNING: Missing unwind data for module, rerun with 'stap -d
/usr/lib/libpolarssl.so'
  WARNING: Missing unwind data for module, rerun with 'stap -d
/usr/local/lib/line_handler.so'
  WARNING: Missing unwind data for module, rerun with 'stap -d
stap_144a2a5f843564742581d523725a704f_2987'

And the kernel backtrace:

  [112186.086052] INFO: rcu_sched self-detected stall on CPU { 10}  (t=6000
jiffies g=1519966 c=1519965 q=3)
  [112186.096687] Pid: 47, comm: migration/10 Tainted: G           O 3.9.4 #1
  [112186.105319] Call Trace:
  [112186.106073] INFO: rcu_sched detected stalls on CPUs/tasks: { 10 11}
(detected by 9, t=6002 jiffies, g=1519966, c=1519965, q=3)
  [112186.106074] Task dump for CPU 10:
  [112186.106077] migration/10    R  running task        0    47      2
0x00000000
  [112186.106080]  0000000000000000 ffffffff814f5c4a 0000000000013c40
ffff880c240bbfd8
  [112186.106082]  000000002767820c ffff880c2403b080 ffffffff81a39ba0
ffff880c240b1c50
  [112186.106084]  ffff880c240b1c50 ffff880c240b1c50 0000000000000000
ffffffff8105a96f
  [112186.106085] Call Trace:
  [112186.106095]  [<ffffffff814f5c4a>] ? __schedule+0x41f/0x4e6
  [112186.106100]  [<ffffffff8105a96f>] ? smpboot_thread_fn+0x13b/0x157
  [112186.106102]  [<ffffffff8105a834>] ? lg_global_lock+0x65/0x65
  [112186.106105]  [<ffffffff81055436>] ? kthread+0x7d/0x85
  [112186.106107]  [<ffffffff810553b9>] ? __kthread_parkme+0x59/0x59
  [112186.106112]  [<ffffffff814fc73c>] ? ret_from_fork+0x7c/0xb0
  [112186.106114]  [<ffffffff810553b9>] ? __kthread_parkme+0x59/0x59
  [112186.106115] Task dump for CPU 11:
  [112186.106117] load-logs       R  running task        0 29704  29446
0x10000008
  [112186.106120]  ffffffff8107956e ffffffff814f5c4a 0000000000013c40
ffff880c174efd90
  [112186.106122]  0000000000b54fe0 ffff880c174efd50 ffffffff81078bbb
ffff880c174efd90
  [112186.106124]  ffff880c174efd90 ffffffff8107986a fffffffffffffff5
0000000081079e36
  [112186.106124] Call Trace:
  [112186.106130]  [<ffffffff8107956e>] ? get_futex_key+0x7a/0x1c8
  [112186.106132]  [<ffffffff814f5c4a>] ? __schedule+0x41f/0x4e6
  [112186.106134]  [<ffffffff81078bbb>] ? get_futex_value_locked+0x24/0x31
  [112186.106137]  [<ffffffff8107986a>] ? futex_wait_setup+0xbf/0xd1
  [112186.106139]  [<ffffffff8107a2bb>] ? futex_wait+0xe2/0x211
  [112186.106141]  [<ffffffff8107956e>] ? get_futex_key+0x7a/0x1c8
  [112186.106143]  [<ffffffff81079791>] ? futex_wake+0xd5/0xef
  [112186.106146]  [<ffffffff8107ab80>] ? do_futex+0xc5/0x77f
  [112186.106149]  [<ffffffff8107c87d>] ? arch_local_irq_save+0x11/0x17
  [112186.106152]  [<ffffffff814f6706>] ? _raw_read_lock_irqsave+0x1c/0x25
  [112186.106154]  [<ffffffff814f6731>] ? _raw_read_unlock_irqrestore+0xc/0xd
  [112186.106167]  [<ffffffffa0c91147>] ? __stp_tf_get_map_entry+0x9b/0xa0
[stap_144a2a5f843564742581d523725a704f_2987]
  [112186.106178]  [<ffffffffa0c9d4dc>] ?
__stp_utrace_task_finder_target_syscall_exit+0x3d/0x1e1
[stap_144a2a5f843564742581d523725a704f_2987]
  [112186.106187]  [<ffffffffa0c93e99>] ? utrace_report_syscall_exit+0xcb/0xda
[stap_144a2a5f843564742581d523725a704f_2987]
  [112186.106190]  [<ffffffff810175a4>] ? syscall_trace_leave+0xbb/0xe7
  [112186.106193]  [<ffffffff814fca87>] ? int_check_syscall_exit_work+0x34/0x3d
  [112186.371501]  <IRQ>  [<ffffffff8109e355>] ?
rcu_check_callbacks+0x1b5/0x53b
  [112186.387395]  [<ffffffff81077cc5>] ? tick_sched_do_timer+0x25/0x25
  [112186.394347]  [<ffffffff81047e6a>] ? update_process_times+0x31/0x5c
  [112186.401396]  [<ffffffff81077be2>] ? tick_sched_handle+0x31/0x3b
  [112186.408153]  [<ffffffff81077cf5>] ? tick_sched_timer+0x30/0x4c
  [112186.414813]  [<ffffffff81057934>] ? __run_hrtimer+0x87/0x132
  [112186.421277]  [<ffffffff810580ad>] ? hrtimer_interrupt+0xcf/0x1a9
  [112186.428124]  [<ffffffff8108df44>] ? cpu_stopper_thread+0x119/0x119
  [112186.435174]  [<ffffffff81027f1f>] ? smp_apic_timer_interrupt+0x6d/0x7e
  [112186.442613]  [<ffffffff814fd3dd>] ? apic_timer_interrupt+0x6d/0x80
  [112186.449658]  <EOI>  [<ffffffff81066888>] ? load_balance+0x401/0x624
  [112186.456829]  [<ffffffff8108df95>] ? stop_machine_cpu_stop+0x51/0xaf
  [112186.463975]  [<ffffffff8108deb6>] ? cpu_stopper_thread+0x8b/0x119
  [112186.470928]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
  [112186.476613]  [<ffffffff8100d5a7>] ? __switch_to+0x136/0x3c0
  [112186.482981]  [<ffffffff8105ecbd>] ? finish_task_switch+0x40/0xb3
  [112186.489836]  [<ffffffff814f5c4a>] ? __schedule+0x41f/0x4e6
  [112186.496106]  [<ffffffff8105a96f>] ? smpboot_thread_fn+0x13b/0x157
  [112186.503050]  [<ffffffff8105a834>] ? lg_global_lock+0x65/0x65
  [112186.509514]  [<ffffffff81055436>] ? kthread+0x7d/0x85
  [112186.515299]  [<ffffffff810553b9>] ? __kthread_parkme+0x59/0x59
  [112186.521949]  [<ffffffff814fc73c>] ? ret_from_fork+0x7c/0xb0
  [112186.528316]  [<ffffffff810553b9>] ? __kthread_parkme+0x59/0x59

Any help will be appreciated! Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15664] Kernel crash when using ubacktrace/print_ubacktrace in timer.profile on kernel 3.9.4

[agentzh at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15664

--- Comment #1 from agentzh <agentzh at gmail dot com> ---
Sorry. I forgot to mention that the systemtap version should be a recent
snapshot marked with 2.3 blah blah.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15664] Kernel crash when using ubacktrace/print_ubacktrace in timer.profile on kernel 3.9.4

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15664

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #2 from David Smith <dsmith at redhat dot com> ---
Would it be possible to see the script you were using?

That last warning (pasted below) is a bit odd - were you probing a different
systemtap module or was stap trying to probe its own module?

>  WARNING: Missing unwind data for module, rerun with 'stap -d
> stap_144a2a5f843564742581d523725a704f_2987'

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15664] Kernel crash when using ubacktrace/print_ubacktrace in timer.profile on kernel 3.9.4

[agentzh at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15664

--- Comment #3 from agentzh <agentzh at gmail dot com> ---
We were using the following script:

https://github.com/agentzh/nginx-systemtap-toolkit/blob/master/ngx-sample-bt

And we were specifying the -u option, that is, using ubacktrace() and
print_ubacktrace() only.

We're seeing the warning "WARNING: Missing unwind data for module, rerun with
'stap -d stap_144a2a5f843564742581d523725a704f_2987'" just because we were
using probe timer.profile in the script, I guess :)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15664] Kernel crash when using ubacktrace/print_ubacktrace in timer.profile on kernel 3.9.4

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=15664

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |fche at redhat dot com
         Resolution|---                         |WORKSFORME

--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> ---
Doesn't appear to appear on kernel 4.0+ stap 2.7+

-- 
You are receiving this mail because:
You are the assignee for the bug.