From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
To: Ingo Molnar <mingo@kernel.org>
Cc: linux-kernel@vger.kernel.org, Andi Kleen <andi@firstfloor.org>,
Ananth N Mavinakayanahalli <ananth@in.ibm.com>,
Sandeepa Prabhu <sandeepa.prabhu@linaro.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
x86@kernel.org, Steven Rostedt <rostedt@goodmis.org>,
fche@redhat.com, mingo@redhat.com, systemtap@sourceware.org,
"H. Peter Anvin" <hpa@zytor.com>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce cache misshits
Date: Thu, 24 Apr 2014 11:38:00 -0000 [thread overview]
Message-ID: <5358F7A3.1000108@hitachi.com> (raw)
In-Reply-To: <20140424090134.GC7768@gmail.com>
(2014/04/24 18:01), Ingo Molnar wrote:
>
> * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>
>> Introduce kprobe cache to reduce cache misshits for
>> massive multiple kprobes.
>> For stress testing kprobes, we need to activate kprobes
>> as many as possible. This situation causes cache miss
>> hit storm on kprobe hash-list. kprobe hashlist is already
>> enlarged to 4k entries and this is still small for 40k
>> kprobes.
>>
>> For example, when registering 40k probes on the hlist and
>> enabling 20k probes, perf tools shows still a lot of
>> cache-misses are on the get_kprobe.
>> ----
>> Samples: 633 of event 'cache-misses', Event count (approx.): 3414776
>> + 68.13% [k] get_kprobe
>> + 4.38% [k] ftrace_lookup_ip
>> + 2.54% [k] kprobe_ftrace_handler
>> ----
>>
>> Also, I found that the most of the kprobes are not hit.
>> In that case, to reduce cache-misses, we can reduce the
>> random memory access by introducing a per-cpu cache which
>> caches the address of frequently used kprobe data structure
>> and its probe address.
>>
>> With kpcache enabled, the get_kprobe_cached goes down to
>> around 4-5% of cache-misses with 20k probes.
>> ----
>> Samples: 729 of event 'cache-misses', Event count (approx.): 690125
>> + 14.49% [k] ftrace_lookup_ip
>> + 5.61% [k] kprobe_trace_func
>> + 5.17% [k] kprobe_ftrace_handler
>> + 4.62% [k] get_kprobe_cached
>> ----
>>
>> Of course this reduces the enabling time too.
>>
>> Without this fix (just enlarge hash table):
>> (2934 sec, 1 min intervals for each 2000 probes enabled)
>>
>> ----
>> Enabling trace events: start at 1393921862
>> 0 1393921864 a2mp_chan_alloc_skb_cb_38581
>> ...
>> 19999 1393924928 nfs4_open_confirm_done_11785
>> ----
>>
>> With this fix:
>> (2025 sec, 1 min intervals for each 2000 probes enabled)
>
> That's a nice speedup.
Thanks :)
>
> So I don't think this should be a Kconfig entry, just enable it
> unconditionally. That will further simplify the code.
Hmm, it consumes some amount of memory (36KB/core) just for the
case of several thousand of kprobes. On enterprise servers and desktop
it's OK, no problem. But I think, some embedded systems with small
resources will not want that. So, how about enabling Kconfig by default?
Thank you,
--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
next prev parent reply other threads:[~2014-04-24 11:38 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-17 8:16 [PATCH -tip v9 00/26] kprobes: introduce NOKPROBE_SYMBOL, bugfixes and scalbility efforts Masami Hiramatsu
2014-04-17 8:16 ` [PATCH -tip v9 01/26] [BUGFIX]kprobes/x86: Fix page-fault handling logic Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 09/26] x86: Call exception_enter after kprobes handled Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 07/26] [BUGFIX] x86: Prohibit probing on thunk functions and restore Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 06/26] [BUGFIX] x86: Prohibit probing on native_set_debugreg/load_idt Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 03/26] kprobes: Prohibit probing on .entry.text code Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 10/26] kprobes/x86: Allow probe on some kprobe preparation functions Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 08/26] kprobes/x86: Call exception handlers directly from do_int3/do_debug Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 05/26] [BUGFIX] kprobes/x86: Prohibit probing on debug_stack_* Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 02/26] kprobes/x86: Allow to handle reentered kprobe on singlestepping Masami Hiramatsu
2014-04-17 8:17 ` [PATCH -tip v9 04/26] kprobes: Introduce NOKPROBE_SYMBOL() macro for blacklist Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 11/26] kprobes: Allow probe on some kprobe functions Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 14/26] x86: Use NOKPROBE_SYMBOL() instead of __kprobes annotation Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 18/26] sched: Use NOKPROBE_SYMBOL macro in sched Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 13/26] x86: Allow kprobes on text_poke/hw_breakpoint Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 17/26] notifier: Use NOKPROBE_SYMBOL macro in notifier Masami Hiramatsu
2014-04-17 14:40 ` Josh Triplett
2014-04-17 8:18 ` [PATCH -tip v9 15/26] kprobes: Use NOKPROBE_SYMBOL macro instead of __kprobes Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 12/26] ftrace/*probes: Allow probing on some functions Masami Hiramatsu
2014-04-17 8:18 ` [PATCH -tip v9 16/26] ftrace/kprobes: Use NOKPROBE_SYMBOL macro in ftrace Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 21/26] kprobes: Use NOKPROBE_SYMBOL() in sample modules Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce cache misshits Masami Hiramatsu
2014-04-24 9:01 ` Ingo Molnar
2014-04-24 11:38 ` Masami Hiramatsu [this message]
2014-04-25 8:21 ` Ingo Molnar
2014-04-25 9:43 ` Masami Hiramatsu
2014-04-26 7:12 ` Ingo Molnar
2014-04-27 12:49 ` Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 19/26] kprobes: Show blacklist entries via debugfs Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 26/26] ftrace: Introduce FTRACE_OPS_FL_SELF_FILTER for ftrace-kprobe Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 20/26] kprobes: Support blacklist functions in module Masami Hiramatsu
2014-04-24 8:56 ` Ingo Molnar
2014-04-24 11:24 ` Masami Hiramatsu
2014-04-25 8:20 ` Ingo Molnar
2014-04-25 10:12 ` Masami Hiramatsu
2014-04-25 10:56 ` Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 24/26] kprobes: Enlarge hash table to 512 entries Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 22/26] kprobes/x86: Use kprobe_blacklist for .kprobes.text and .entry.text Masami Hiramatsu
2014-04-24 8:58 ` Ingo Molnar
2014-04-24 11:22 ` Masami Hiramatsu
2014-04-17 8:19 ` [PATCH -tip v9 23/26] kprobes/x86: Remove unneeded preempt_disable/enable in interrupt handlers Masami Hiramatsu
2014-04-17 8:37 ` [PATCH -tip v9 00/26] kprobes: introduce NOKPROBE_SYMBOL, bugfixes and scalbility efforts Ingo Molnar
2014-04-17 8:53 ` Masami Hiramatsu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5358F7A3.1000108@hitachi.com \
--to=masami.hiramatsu.pt@hitachi.com \
--cc=ananth@in.ibm.com \
--cc=andi@firstfloor.org \
--cc=fche@redhat.com \
--cc=fweisbec@gmail.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
--cc=sandeepa.prabhu@linaro.org \
--cc=systemtap@sourceware.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).