From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 58889 invoked by alias); 16 Jul 2015 02:44:57 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 58880 invoked by uid 89); 16 Jul 2015 02:44:56 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.2 required=5.0 tests=AWL,BAYES_50,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail7.hitachi.co.jp Received: from mail7.hitachi.co.jp (HELO mail7.hitachi.co.jp) (133.145.228.42) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 16 Jul 2015 02:44:54 +0000 Received: from mlsv3.hitachi.co.jp (unknown [133.144.234.166]) by mail7.hitachi.co.jp (Postfix) with ESMTP id D678FB1D38B for ; Thu, 16 Jul 2015 11:44:51 +0900 (JST) Received: from mfilter03.hitachi.co.jp by mlsv3.hitachi.co.jp (8.13.1/8.13.1) id t6G2ipwY012699; Thu, 16 Jul 2015 11:44:51 +0900 Received: from vshuts01.hitachi.co.jp (vshuts01.hitachi.co.jp [10.201.6.83]) by mfilter03.hitachi.co.jp (Switch-3.3.4/Switch-3.3.4) with ESMTP id t6G2ip7w020584 for ; Thu, 16 Jul 2015 11:44:51 +0900 Received: from gxml20a.ad.clb.hitachi.co.jp (unknown [158.213.157.160]) by vshuts01.hitachi.co.jp (Postfix) with ESMTP id 96C9C2F0043; Thu, 16 Jul 2015 11:44:50 +0900 (JST) Received: from [10.198.220.34] by gxml20a.ad.clb.hitachi.co.jp (Switch-3.1.10/Switch-3.1.9) id 76G238EXE0000F148; Thu, 16 Jul 2015 11:44:50 +0900 Message-ID: <55A71A9D.10603@hitachi.com> Date: Thu, 16 Jul 2015 02:44:00 -0000 From: Masami Hiramatsu User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: systemtap@sourceware.org Subject: Re: [PATCH -tip v11 0/7] kprobes: NOKPROBE_SYMBOL for modules, and scalbility efforts References: <20140514082034.5791.38607.stgit@ltc230.yrl.intra.hitachi.co.jp> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2015-q3/txt/msg00039.txt.bz2 On 2015/07/09 20:40, Pratyush Anand wrote: > Hi Masami, > > Masami Hiramatsu hitachi.com> writes: > >> >> Hi, >> Here is the version 11 of NOKPROBE_SYMBOL/scalability series. >> This fixes some issues. > > If I have not missed then v11 seems the last version posted for this series. > However, I do not see any ACK/NACK for the series. So, I am just curious > that why didn't it got merged into upstream. Yeah, that is still not merged yet. > It seems a useful series and without these patches system becomes extremely > slow with large number of kprobes enabled. Yes, I should revise it for the latest kernel... > > When I enabled massive kprobes [1] on my ARM64 system without this patch > series, I see "NMI watchdog: BUG: soft lockup" for all CPUs even before all > kprobes are enabled. However, If I use these patches then system is able to > survive and I can see following output for script [1]. > > real 1m49.734s > user 0m0.000s > sys 0m53.950s > 31364 kprobes enabled :-) Wow! really nice result!! > > I was able to run `usex -e` to load the system, although I see that system > is mostly occupied by kernel (97%). > > * Unix System EXerciser * USER SYSTEM IDLE LOADAVG TASKS/RUN TEST TIME * > * USEX Version 1.9-36 * 3% 97% 0% 24.58 232/27 000:02:49 * I see, since every function calls are interrupted by kprobes... BTW, have you ever tried to analyze hot spot by using perf? Thank you! > > ~Pratyush > > [1] > #!/bin/sh > grep ' [tT] ' /proc/kallsyms | fgrep -v '[' | awk '{print $3}' > syms.list > echo 0 > /sys/kernel/debug/tracing/events/enable > echo 0 > /sys/kernel/debug/tracing/events/kprobes/enable > echo > /sys/kernel/debug/tracing/kprobe_events > for i in `cat syms.list`; > do > echo "p $i" >> /sys/kernel/debug/tracing/kprobe_events ; > done > time echo 1 > /sys/kernel/debug/tracing/events/kprobes/enable > num_kprobe=`cat /sys/kernel/debug/tracing/kprobe_events | wc -l` > echo "$num_kprobe kprobes enabled :-)" > > > > > -- Masami HIRAMATSU Linux Technology Research Center, System Productivity Research Dept. Center for Technology Innovation - Systems Engineering Hitachi, Ltd., Research & Development Group E-mail: masami.hiramatsu.pt@hitachi.com