From: fche@redhat.com (Frank Ch. Eigler)
To: liang xie <xieliang007@gmail.com>
Cc: systemtap@sourceware.org
Subject: Re: How to get the detailed kernel stack trace if specified fuction takes too long to finish?
Date: Fri, 28 Mar 2014 16:56:00 -0000 [thread overview]
Message-ID: <y0m4n2imfcg.fsf@fche.csb> (raw)
In-Reply-To: <CADu=CFo=AtrBJLacPTa_6EfQA7vao=bSuT2Yns45x-hWYfmWZA@mail.gmail.com> (liang xie's message of "Fri, 28 Mar 2014 12:22:24 +0800")
Hi -
xieliang007 wrote:
> Probably it's a newbie question, to be honest, i am new to SystemTap,
> currently i want to capture the kernel stack trace if "sys_write" runs
> too long, if that i could know where it sucks.
OK.
> [...] probe kernel.function("sys_write").return [...]
> print_backtrace();
> Returning from: 0xffffffff81176f90 : sys_write+0x0/0x90 [kernel]
> Returning to : 0xffffffff8100b0f2 : system_call_fastpath+0x16/0x1b [kernel]
>
> It doesn't show my "expected" kernel stack trace,e.g. file system or
> block layer related stack trace.
The expectation is mistaken. By the time that sys_write returns, all
that kernel activity is in the past, and will have left no trace on
the stack.
> How to archive that? Any pointers will be highly appreciated!
Not easy.
One way could be to sample the kernel backtrace (from probe
timer.profile) while a sys_write is in effect. Attributing the
backtraces to the particular thread suffering from a slow sys_write is
going to be tricky, but maybe this could work:
global backtraces%
global active
probe syscall.write {
active[tid()]=1
}
probe timer.profile {
if (! (tid() in active)) next;
backtraces[tid(), backtrace()] <<< 1
}
probe syscall.write.return {
delete active[tid()]
elapsed = gettimeofday_ms() - @entry(gettimeofday_ms())
if (elapsed <= 4)
next; // oops, leaves behind backtraces[] records
printf("%s[%d] sleepy for %d ms:\n", execname(), tid(), elapsed)
foreach ([t,bt] in backtraces- limit /* top */ 10)
if (t == tid()) {
printf("\n%d hits:\n", @count(backtraces[t,bt]))
print_stack(bt)
}
}
# stap --all-modules FOO.stp
- FChE
next prev parent reply other threads:[~2014-03-28 16:56 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-28 4:22 liang xie
2014-03-28 16:56 ` Frank Ch. Eigler [this message]
2014-03-28 17:06 ` Josh Stone
2014-04-02 10:25 ` liang xie
2014-04-02 16:32 ` Josh Stone
2014-04-03 2:47 ` liang xie
2014-04-03 16:24 ` Josh Stone
2014-04-08 2:15 ` liang xie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=y0m4n2imfcg.fsf@fche.csb \
--to=fche@redhat.com \
--cc=systemtap@sourceware.org \
--cc=xieliang007@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).