From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 48) id 567153857C5C; Wed, 26 Aug 2020 02:02:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 567153857C5C From: "agentzh at gmail dot com" To: systemtap@sourceware.org Subject: [Bug runtime/26537] New: stapio might get stuck in Dl state forever Date: Wed, 26 Aug 2020 02:02:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: systemtap X-Bugzilla-Component: runtime X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: agentzh at gmail dot com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: systemtap at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: systemtap@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Systemtap mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2020 02:02:47 -0000 https://sourceware.org/bugzilla/show_bug.cgi?id=3D26537 Bug ID: 26537 Summary: stapio might get stuck in Dl state forever Product: systemtap Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: runtime Assignee: systemtap at sourceware dot org Reporter: agentzh at gmail dot com Target Milestone: --- We've noted that sometimes especially under load, the stapio process may get stuck in the Dl state forever: ``` # ps aux | grep stapio | grep -v grep root 47763 0.2 0.0 23000 1636 ? Dl 10:00 1:23 stapio -R = -v -o c-fgraph.bt stap_c_fgraph_X_47763 var1_total_frames=3D100 var1_sample_frames=3D100 -F3 ``` The kernel backtrace looks interesting: ``` # sudo cat /proc/47763/stack [] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763] [] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763] [] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763] [] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_4776= 3] [] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_4776= 3] [] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763] [] full_proxy_write+0x64/0x9c [] __vfs_write+0x3a/0x16d [] vfs_write+0xb2/0x1a1 [] SyS_write+0x55/0xb9 [] do_syscall_64+0x79/0x1b9 [] entry_SYSCALL_64_after_hwframe+0x169/0x0 [] 0xffffffffffffffff ``` And below are all the backtraces for all the OS threads of this stapio proc= ess: ``` # sudo cat /proc/47763/task/*/stack [] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763] [] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763] [] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763] [] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_4776= 3] [] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_4776= 3] [] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763] [] full_proxy_write+0x64/0x9c [] __vfs_write+0x3a/0x16d [] vfs_write+0xb2/0x1a1 [] SyS_write+0x55/0xb9 [] do_syscall_64+0x79/0x1b9 [] entry_SYSCALL_64_after_hwframe+0x169/0x0 [] 0xffffffffffffffff [] do_sigtimedwait+0x1a7/0x241 [] SYSC_rt_sigtimedwait+0x85/0xe1 [] SyS_rt_sigtimedwait+0xe/0x10 [] do_syscall_64+0x79/0x1b9 [] entry_SYSCALL_64_after_hwframe+0x169/0x0 [] 0xffffffffffffffff [] poll_schedule_timeout+0x4a/0x67 [] do_sys_poll+0x3d8/0x54d [] SyS_ppoll+0xce/0x189 [] do_syscall_64+0x79/0x1b9 [] entry_SYSCALL_64_after_hwframe+0x169/0x0 [] 0xffffffffffffffff ``` Seems like only the first thread is interesting. The `stp_task_work_exit` function is very short: ``` static void stp_task_work_exit(void) { while (atomic_read(&stp_task_work_callbacks)) schedule_timeout_uninterruptible(1); return; } ``` Seems like it is stuck in the loop here. --=20 You are receiving this mail because: You are the assignee for the bug.=