Re: [PATCH -tip v11 0/7] kprobes: NOKPROBE_SYMBOL for modules, and scalbility efforts

[Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>]

On 2015/07/09 20:40, Pratyush Anand wrote:
> Hi Masami,
> 
> Masami Hiramatsu <masami.hiramatsu.pt <at> hitachi.com> writes:
> 
>>
>> Hi,
>> Here is the version 11 of NOKPROBE_SYMBOL/scalability series.
>> This fixes some issues.
> 
> If I have not missed then v11 seems the last version posted for this series.
> However, I do not see any ACK/NACK for the series. So, I am just curious
> that why didn't it got merged into upstream.

Yeah, that is still not merged yet.

> It seems a useful series and without these patches system becomes extremely
> slow with large number of kprobes enabled.

Yes, I should revise it for the latest kernel...

> 
> When I enabled massive kprobes [1] on my ARM64 system without this patch
> series, I see "NMI watchdog: BUG: soft lockup" for all CPUs even before all
> kprobes are enabled. However, If I use these patches then system is able to
> survive and I can see following output for script [1].
> 
> real    1m49.734s
> user    0m0.000s
> sys     0m53.950s
> 31364 kprobes enabled :-)

Wow! really nice result!!

> 
> I was able to run `usex -e` to load the system, although I see that system
> is mostly occupied by kernel (97%).
> 
> *  Unix System EXerciser  *  USER SYSTEM IDLE  LOADAVG  TASKS/RUN   TEST TIME  *
> *   USEX Version 1.9-36   *    3%   97%    0%   24.58     232/27    000:02:49  *

I see, since every function calls are interrupted by kprobes...

BTW, have you ever tried to analyze hot spot by using perf?


Thank you!

> 
> ~Pratyush
> 
> [1]
> #!/bin/sh
> grep ' [tT] ' /proc/kallsyms | fgrep -v '[' | awk '{print $3}' > syms.list
> echo 0 > /sys/kernel/debug/tracing/events/enable
> echo 0 > /sys/kernel/debug/tracing/events/kprobes/enable
> echo > /sys/kernel/debug/tracing/kprobe_events
> for i in `cat syms.list`;
> do
> 	echo "p $i" >> /sys/kernel/debug/tracing/kprobe_events ;
> done
> time echo 1 > /sys/kernel/debug/tracing/events/kprobes/enable
> num_kprobe=`cat /sys/kernel/debug/tracing/kprobe_events | wc -l`
> echo "$num_kprobe kprobes enabled :-)"
> 
> 
> 
> 
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com