* [Bug runtime/26537] New: stapio might get stuck in Dl state forever
@ 2020-08-26 2:02 agentzh at gmail dot com
0 siblings, 0 replies; only message in thread
From: agentzh at gmail dot com @ 2020-08-26 2:02 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=26537
Bug ID: 26537
Summary: stapio might get stuck in Dl state forever
Product: systemtap
Version: unspecified
Status: UNCONFIRMED
Severity: normal
Priority: P2
Component: runtime
Assignee: systemtap at sourceware dot org
Reporter: agentzh at gmail dot com
Target Milestone: ---
We've noted that sometimes especially under load, the stapio process may get
stuck in the Dl state forever:
```
# ps aux | grep stapio | grep -v grep
root 47763 0.2 0.0 23000 1636 ? Dl 10:00 1:23 stapio -R -v
-o c-fgraph.bt stap_c_fgraph_X_47763 var1_total_frames=100
var1_sample_frames=100 -F3
```
The kernel backtrace looks interesting:
```
# sudo cat /proc/47763/stack
[<ffffffffc0a7c923>] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763]
[<ffffffffc0a8fcb5>] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763]
[<ffffffffc0a8ff36>] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763]
[<ffffffffc0a90a98>] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_47763]
[<ffffffffc0a90d66>] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_47763]
[<ffffffffc0a90e5c>] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763]
[<ffffffff9c324f04>] full_proxy_write+0x64/0x9c
[<ffffffff9c2848ba>] __vfs_write+0x3a/0x16d
[<ffffffff9c284b92>] vfs_write+0xb2/0x1a1
[<ffffffff9c284df5>] SyS_write+0x55/0xb9
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
```
And below are all the backtraces for all the OS threads of this stapio process:
```
# sudo cat /proc/47763/task/*/stack
[<ffffffffc0a7c923>] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763]
[<ffffffffc0a8fcb5>] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763]
[<ffffffffc0a8ff36>] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763]
[<ffffffffc0a90a98>] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_47763]
[<ffffffffc0a90d66>] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_47763]
[<ffffffffc0a90e5c>] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763]
[<ffffffff9c324f04>] full_proxy_write+0x64/0x9c
[<ffffffff9c2848ba>] __vfs_write+0x3a/0x16d
[<ffffffff9c284b92>] vfs_write+0xb2/0x1a1
[<ffffffff9c284df5>] SyS_write+0x55/0xb9
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff9c0a5687>] do_sigtimedwait+0x1a7/0x241
[<ffffffff9c0a8ef5>] SYSC_rt_sigtimedwait+0x85/0xe1
[<ffffffff9c0a8f6e>] SyS_rt_sigtimedwait+0xe/0x10
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff9c29b28a>] poll_schedule_timeout+0x4a/0x67
[<ffffffff9c29c648>] do_sys_poll+0x3d8/0x54d
[<ffffffff9c29d0be>] SyS_ppoll+0xce/0x189
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
```
Seems like only the first thread is interesting.
The `stp_task_work_exit` function is very short:
```
static void
stp_task_work_exit(void)
{
while (atomic_read(&stp_task_work_callbacks))
schedule_timeout_uninterruptible(1);
return;
}
```
Seems like it is stuck in the loop here.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2020-08-26 2:02 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-26 2:02 [Bug runtime/26537] New: stapio might get stuck in Dl state forever agentzh at gmail dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).