From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
To: Ingo Molnar <mingo@kernel.org>
Cc: "Frank Ch. Eigler" <fche@redhat.com>,
linux-arch@vger.kernel.org,
Ananth N Mavinakayanahalli <ananth@in.ibm.com>,
Sandeepa Prabhu <sandeepa.prabhu@linaro.org>,
x86@kernel.org, lkml <linux-kernel@vger.kernel.org>,
"Steven Rostedt (Red Hat)" <rostedt@goodmis.org>,
systemtap@sourceware.org, "David S. Miller" <davem@davemloft.net>
Subject: Re: [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs
Date: Mon, 10 Feb 2014 04:02:00 -0000 [thread overview]
Message-ID: <52F84F61.7060103@hitachi.com> (raw)
In-Reply-To: <20140209143709.GA31608@gmail.com>
(2014/02/09 23:37), Ingo Molnar wrote:
>
> * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>
>> I guess the second reason is why the stap takes so long time to set
>> probes. stap tries to register kprobes without disabled flag, that
>> means we enables thousands of probes (overheads).
>>
>> So the similar thing happens when we enables events as below;
>>
>> # for i in /sys/kernel/debug/tracing/events/kprobes/* ; do date; echo 1 > $i; done
>> Wed Jan 29 10:44:50 UTC 2014
>> ...
>>
>> I tried it and canceled after 4 min passed. It enabled about 17k
>> events and slowed down my system very much(I almost got hang check
>> timer).
>
> Ok, I guess that's the slowdown bug that Frank reported.
>
>> I think we should have some performance statistics (hit count?) and
>> if it goes over a threshold, we should stop enabling other events.
>
> That really feels like a hack. How about fixing the root cause? Does
> the enabling of all probes have to be so slow?
When I tried to use perf top, most of the time was consumed in
kprobe_ftrace_handler and optimized_callback, both of them
are the handler of kprobes. Since I just tried on a VM guest and it
didn't support NMI nor PMU, thus I have to use a bare metal machine
for deeper investigation (and I'll do).
And yes, it seems that the performance problem comes from probing
and tracing itself.
Thank you,
--
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
next prev parent reply other threads:[~2014-02-10 4:02 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-12-19 9:09 Masami Hiramatsu
2013-12-19 9:09 ` [PATCH -tip v6 02/22] kprobes: Introduce NOKPROBE_SYMBOL() macro for blacklist Masami Hiramatsu
2013-12-19 9:09 ` [PATCH -tip v6 06/22] [BUGFIX] x86: Prohibit probing on memcpy/memset Masami Hiramatsu
2013-12-19 9:37 ` Jovi Zhangwei
2013-12-20 2:38 ` Masami Hiramatsu
2013-12-20 3:07 ` Jovi Zhangwei
2013-12-20 4:42 ` Masami Hiramatsu
2013-12-20 8:31 ` Jovi Zhangwei
2013-12-20 9:21 ` Masami Hiramatsu
2013-12-23 4:51 ` Jovi Zhangwei
2013-12-23 10:59 ` Masami Hiramatsu
2013-12-24 6:39 ` Jovi Zhangwei
2013-12-24 8:32 ` Masami Hiramatsu
2013-12-24 9:53 ` Jovi Zhangwei
2013-12-24 15:58 ` Masami Hiramatsu
2013-12-25 14:44 ` Jovi Zhangwei
2013-12-19 9:09 ` [PATCH -tip v6 01/22] kprobes: Prohibit probing on .entry.text code Masami Hiramatsu
2013-12-19 9:09 ` [PATCH -tip v6 05/22] [BUGFIX] x86: Prohibit probing on thunk functions and restore Masami Hiramatsu
2013-12-19 9:09 ` [PATCH -tip v6 03/22] [BUGFIX] kprobes/x86: Prohibit probing on debug_stack_* Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 10/22] ftrace/kprobes: Allow probing on some preparation functions Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 08/22] kprobes/x86: Allow probe on some kprobe " Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 16/22] sched: Use NOKPROBE_SYMBOL macro in sched Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 09/22] kprobes: Allow probe on some kprobe functions Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 15/22] notifier: Use NOKPROBE_SYMBOL macro in notifier Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 20/22] kprobes/x86: Use kprobe_blacklist for .kprobes.text and .entry.text Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 22/22] [RFC] kprobes/x86: Add emergency recovery process for bad kprobes Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 19/22] kprobes: Use NOKPROBE_SYMBOL() in sample modules Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 21/22] kprobes/x86: Remove unneeded preempt_disable/enable in interrupt handlers Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 14/22] ftrace/kprobes: Use NOKPROBE_SYMBOL macro in ftrace Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 07/22] kprobes/x86: Call exception handlers directly from do_int3/do_debug Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 18/22] kprobes: Support blacklist functions in module Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 17/22] kprobes: Show blacklist entries via debugfs Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 13/22] kprobes: Use NOKPROBE_SYMBOL macro instead of __kprobes Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 04/22] [BUGFIX] x86: Prohibit probing on native_set_debugreg/load_idt Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 11/22] x86: Allow kprobes on text_poke/hw_breakpoint Masami Hiramatsu
2013-12-19 9:10 ` [PATCH -tip v6 12/22] x86: Use NOKPROBE_SYMBOL() instead of __kprobes annotation Masami Hiramatsu
2013-12-19 20:46 ` [PATCH -tip v6 00/22] kprobes: introduce NOKPROBE_SYMBOL(), cleanup and fixes crash bugs Frank Ch. Eigler
2013-12-20 4:22 ` Masami Hiramatsu
2013-12-20 8:21 ` Ingo Molnar
2013-12-20 9:31 ` Masami Hiramatsu
2013-12-20 10:46 ` Ingo Molnar
2013-12-22 21:10 ` Masami Hiramatsu
2013-12-23 13:04 ` Ingo Molnar
2013-12-24 1:19 ` Masami Hiramatsu
2014-01-29 11:22 ` Masami Hiramatsu
2014-02-09 14:37 ` Ingo Molnar
2014-02-09 21:27 ` Frank Ch. Eigler
2014-02-10 4:02 ` Masami Hiramatsu [this message]
2014-02-10 11:32 ` Masami Hiramatsu
2013-12-20 13:41 ` Frank Ch. Eigler
2013-12-22 21:32 ` Masami Hiramatsu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=52F84F61.7060103@hitachi.com \
--to=masami.hiramatsu.pt@hitachi.com \
--cc=ananth@in.ibm.com \
--cc=davem@davemloft.net \
--cc=fche@redhat.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=rostedt@goodmis.org \
--cc=sandeepa.prabhu@linaro.org \
--cc=systemtap@sourceware.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).