From: "agentzh at gmail dot com" <sourceware-bugzilla@sourceware.org>
To: systemtap@sourceware.org
Subject: [Bug runtime/26537] New: stapio might get stuck in Dl state forever
Date: Wed, 26 Aug 2020 02:02:47 +0000 [thread overview]
Message-ID: <bug-26537-6586@http.sourceware.org/bugzilla/> (raw)
https://sourceware.org/bugzilla/show_bug.cgi?id=26537
Bug ID: 26537
Summary: stapio might get stuck in Dl state forever
Product: systemtap
Version: unspecified
Status: UNCONFIRMED
Severity: normal
Priority: P2
Component: runtime
Assignee: systemtap at sourceware dot org
Reporter: agentzh at gmail dot com
Target Milestone: ---
We've noted that sometimes especially under load, the stapio process may get
stuck in the Dl state forever:
```
# ps aux | grep stapio | grep -v grep
root 47763 0.2 0.0 23000 1636 ? Dl 10:00 1:23 stapio -R -v
-o c-fgraph.bt stap_c_fgraph_X_47763 var1_total_frames=100
var1_sample_frames=100 -F3
```
The kernel backtrace looks interesting:
```
# sudo cat /proc/47763/stack
[<ffffffffc0a7c923>] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763]
[<ffffffffc0a8fcb5>] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763]
[<ffffffffc0a8ff36>] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763]
[<ffffffffc0a90a98>] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_47763]
[<ffffffffc0a90d66>] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_47763]
[<ffffffffc0a90e5c>] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763]
[<ffffffff9c324f04>] full_proxy_write+0x64/0x9c
[<ffffffff9c2848ba>] __vfs_write+0x3a/0x16d
[<ffffffff9c284b92>] vfs_write+0xb2/0x1a1
[<ffffffff9c284df5>] SyS_write+0x55/0xb9
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
```
And below are all the backtraces for all the OS threads of this stapio process:
```
# sudo cat /proc/47763/task/*/stack
[<ffffffffc0a7c923>] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763]
[<ffffffffc0a8fcb5>] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763]
[<ffffffffc0a8ff36>] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763]
[<ffffffffc0a90a98>] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_47763]
[<ffffffffc0a90d66>] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_47763]
[<ffffffffc0a90e5c>] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763]
[<ffffffff9c324f04>] full_proxy_write+0x64/0x9c
[<ffffffff9c2848ba>] __vfs_write+0x3a/0x16d
[<ffffffff9c284b92>] vfs_write+0xb2/0x1a1
[<ffffffff9c284df5>] SyS_write+0x55/0xb9
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff9c0a5687>] do_sigtimedwait+0x1a7/0x241
[<ffffffff9c0a8ef5>] SYSC_rt_sigtimedwait+0x85/0xe1
[<ffffffff9c0a8f6e>] SyS_rt_sigtimedwait+0xe/0x10
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff9c29b28a>] poll_schedule_timeout+0x4a/0x67
[<ffffffff9c29c648>] do_sys_poll+0x3d8/0x54d
[<ffffffff9c29d0be>] SyS_ppoll+0xce/0x189
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
```
Seems like only the first thread is interesting.
The `stp_task_work_exit` function is very short:
```
static void
stp_task_work_exit(void)
{
while (atomic_read(&stp_task_work_callbacks))
schedule_timeout_uninterruptible(1);
return;
}
```
Seems like it is stuck in the loop here.
--
You are receiving this mail because:
You are the assignee for the bug.
reply other threads:[~2020-08-26 2:02 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bug-26537-6586@http.sourceware.org/bugzilla/ \
--to=sourceware-bugzilla@sourceware.org \
--cc=systemtap@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).