From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 28344 invoked by alias); 24 Apr 2014 09:01:43 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 28332 invoked by uid 89); 24 Apr 2014 09:01:43 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-ee0-f54.google.com Received: from mail-ee0-f54.google.com (HELO mail-ee0-f54.google.com) (74.125.83.54) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-SHA encrypted) ESMTPS; Thu, 24 Apr 2014 09:01:42 +0000 Received: by mail-ee0-f54.google.com with SMTP id d49so1572099eek.41 for ; Thu, 24 Apr 2014 02:01:38 -0700 (PDT) X-Received: by 10.15.50.136 with SMTP id l8mr739260eew.73.1398330098407; Thu, 24 Apr 2014 02:01:38 -0700 (PDT) Received: from gmail.com (54033002.catv.pool.telekom.hu. [84.3.48.2]) by mx.google.com with ESMTPSA id y7sm13905872eev.5.2014.04.24.02.01.36 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 24 Apr 2014 02:01:37 -0700 (PDT) Date: Thu, 24 Apr 2014 09:01:00 -0000 From: Ingo Molnar To: Masami Hiramatsu Cc: linux-kernel@vger.kernel.org, Andi Kleen , Ananth N Mavinakayanahalli , Sandeepa Prabhu , Frederic Weisbecker , x86@kernel.org, Steven Rostedt , fche@redhat.com, mingo@redhat.com, systemtap@sourceware.org, "H. Peter Anvin" , Thomas Gleixner Subject: Re: [PATCH -tip v9 25/26] kprobes: Introduce kprobe cache to reduce cache misshits Message-ID: <20140424090134.GC7768@gmail.com> References: <20140417081636.26341.87858.stgit@ltc230.yrl.intra.hitachi.co.jp> <20140417081931.26341.47154.stgit@ltc230.yrl.intra.hitachi.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140417081931.26341.47154.stgit@ltc230.yrl.intra.hitachi.co.jp> User-Agent: Mutt/1.5.21 (2010-09-15) X-SW-Source: 2014-q2/txt/msg00087.txt.bz2 * Masami Hiramatsu wrote: > Introduce kprobe cache to reduce cache misshits for > massive multiple kprobes. > For stress testing kprobes, we need to activate kprobes > as many as possible. This situation causes cache miss > hit storm on kprobe hash-list. kprobe hashlist is already > enlarged to 4k entries and this is still small for 40k > kprobes. > > For example, when registering 40k probes on the hlist and > enabling 20k probes, perf tools shows still a lot of > cache-misses are on the get_kprobe. > ---- > Samples: 633 of event 'cache-misses', Event count (approx.): 3414776 > + 68.13% [k] get_kprobe > + 4.38% [k] ftrace_lookup_ip > + 2.54% [k] kprobe_ftrace_handler > ---- > > Also, I found that the most of the kprobes are not hit. > In that case, to reduce cache-misses, we can reduce the > random memory access by introducing a per-cpu cache which > caches the address of frequently used kprobe data structure > and its probe address. > > With kpcache enabled, the get_kprobe_cached goes down to > around 4-5% of cache-misses with 20k probes. > ---- > Samples: 729 of event 'cache-misses', Event count (approx.): 690125 > + 14.49% [k] ftrace_lookup_ip > + 5.61% [k] kprobe_trace_func > + 5.17% [k] kprobe_ftrace_handler > + 4.62% [k] get_kprobe_cached > ---- > > Of course this reduces the enabling time too. > > Without this fix (just enlarge hash table): > (2934 sec, 1 min intervals for each 2000 probes enabled) > > ---- > Enabling trace events: start at 1393921862 > 0 1393921864 a2mp_chan_alloc_skb_cb_38581 > ... > 19999 1393924928 nfs4_open_confirm_done_11785 > ---- > > With this fix: > (2025 sec, 1 min intervals for each 2000 probes enabled) That's a nice speedup. So I don't think this should be a Kconfig entry, just enable it unconditionally. That will further simplify the code. Thanks, Ingo