Kprobes stress test prototype

Kprobes stress test prototype

[Masami Hiramatsu]

[-- Attachment #1: Type: text/plain, Size: 2212 bytes --]

Hi,

Here is a script of kprobes stress test which uses
kprobes-ftrace engine. Currently, this doesn't use any
benchmarks for making a load.
Any updates, comments, and reports are welcome!

On KVM/x86-64, I've found some symbols must be marked
as __kprobes or blacklisted by using this script.

<always system hang up>
p irq_return
p start_critical_timing
p time_hardirqs_off
p bad_address
p trace_hardirqs_off_thunk
p restore

<not always, but potentially dangerous>
p start_critical_timings
p stop_critical_timing
p trace_hardirqs_on_thunk
p restore_norax

(I've encountered other odd memory corruptions too.
 However, I couldn't identify the reason, it might be a bug
 in kprobes or kvm.)

Basic usage is very simple.
Of course, this needs kprobes-ftrace engine patchset which
I posted to lkml (http://lkml.org/lkml/2009/6/1/461).

1. Make a working area and put the script;
 $ mkdir stest/
 $ cp kprobestest stest/

2. Run it.
 $ cd stest/
 $ ./kprobestest run

3. If kernel gets trouble, reboot it and run again.
 $ cd stest/
 $ ./kprobestest run

4. Repeat it, until all symbols are tested.

First, the test lists up all symbols in ".text" section, except
".text.kprobes"(*) and breaks it into sub-lists. Initially, each
sub-lists has 64 symbols and are stored into "gray" dir.
The test passes each sub-list to kprobes-ftrace engine and makes
a load(currently, just sleep 2secs).
If a sub-list hasn't made any trouble, it is moved to "white" dir.
If it causes a kernel panic etc., it is moved to "black" dir.

If "gray" dir becomes empty, it collects sub-lists in "black" dir
and breaks it into smaller sub-lists which has 16 symbols.

The test repeats it until each sub-list has just 1 symbol.

(*) You can also pass limited symbols which you are interested in
    instead of all symbols. In the step 2,
   $ cd stest/
   $ ./kprobestest run SYMBOL_LIST_FILE

  The SYMBOL_LIST_FILE must be written in the format of kprobes_events.
  This means you have to add a "p " prefix for each symbol. So,
  $ cat SYMBOL_LIST_FILE
  p symbol1
  p symbol2
  ...


Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com


[-- Attachment #2: kprobestest --]
[-- Type: text/plain, Size: 2262 bytes --]

#!/bin/bash
#
#  kprobestest: Kprobes stress test tool
#  Written by Masami Hiramatsu <mhiramat@redhat.com>
#
#  Usage:
#     $ kprobestest run [SYMLIST]
#        Run stress test. If SYMLIST file is specified, use it as 
#        an initial symbol list (This is useful for verifying white list
#        after diagnosing all symbols).
#
#     $ kprobestest cleanup
#        Cleanup all lists

if [ "$1" = "cleanup" ]; then
  rm -rf symlist kptext black white gray
  echo > /debug/tracing/kprobe_events
  exit 0
fi

if [ "$1" != "run" ]; then
  echo "Usage: kprobestest run|cleanup [SYMLIST]"
  exit 0
fi

if [ -f "$2" ]; then
SYMLIST=$2
fi

NR=64

function nomoreblack () {
  echo "no more kprobe blacklisted functions"
  exit 0
}

function allfound () {
  echo "found all blacklisted functions"
  cat black/* | tee blacklist
  rm black/*
  exit 0
}

function preparing () {
local i=0
local n=0
local outfile=symlist
if [ ! -f symlist ]; then
  if [ -f $SYMLIST ]; then
    cat $SYMLIST > symlist
  else
    for sym in `sort /proc/kallsyms | egrep '[0-9a-f]+ [Tt] [^[]*$' | cut -d\  -f 3 ` ;do
      [ $sym  = "__kprobes_text_start" ] && outfile=kptext && continue
      [ $sym  = "__kprobes_text_end" ] && outfile=symlist && continue
      [ $sym  = "_etext" ] && break
      echo p $sym >> $outfile
    done
  fi
  [ -d black ] || mkdir black
  [ -d white ] || mkdir white
else
  [ -z "`ls black`" ] && nomoreblack
  for b in black/list-* ; do
    NR=`wc -l $b| cut -f1 -d\ `
    [ $NR -eq 1 ] && allfound
    NR=$((NR/4))
    [ $NR -eq 0 ] && NR=1
    break
  done
  cat black/* > symlist
  rm black/*
fi
cat symlist | while read ln; do
  echo "$ln" >> gray/list-$NR-$i
  n=$((n+1))
  if [ $n -eq $NR ]; then
    n=0
    i=$((i+1))
  fi
done
sync
}

[ -d gray ] || mkdir gray
[ -z "`ls gray/`" ] && preparing
echo > /debug/tracing/kprobe_events

function execload () {
  # do some benchmark
  sleep 2
}

for list in `cd gray/; ls`; do
echo -n $list
mv gray/$list black/
sync;sync
echo -n "..sync"
cat black/$list > /debug/tracing/kprobe_events
if [ $? -ne 0 ]; then
  echo > /debug/tracing/kprobe_events
  echo "..error"
  sync;sync
else
  execload
  echo > /debug/tracing/kprobe_events
  sync;sync
  echo "..done"
  mv black/$list white/
  sync;sync
fi
done

Re: Kprobes stress test prototype

[Ananth N Mavinakayanahalli]

On Fri, Jun 26, 2009 at 02:33:04PM -0400, Masami Hiramatsu wrote:
> Hi,
> 
> Here is a script of kprobes stress test which uses
> kprobes-ftrace engine. Currently, this doesn't use any
> benchmarks for making a load.
> Any updates, comments, and reports are welcome!
> 
> On KVM/x86-64, I've found some symbols must be marked
> as __kprobes or blacklisted by using this script.
> 
> <always system hang up>
> p irq_return
> p start_critical_timing
> p time_hardirqs_off
> p bad_address
> p trace_hardirqs_off_thunk
> p restore
> 
> <not always, but potentially dangerous>
> p start_critical_timings
> p stop_critical_timing
> p trace_hardirqs_on_thunk
> p restore_norax
> 
> (I've encountered other odd memory corruptions too.
>  However, I couldn't identify the reason, it might be a bug
>  in kprobes or kvm.)
> 
> Basic usage is very simple.
> Of course, this needs kprobes-ftrace engine patchset which
> I posted to lkml (http://lkml.org/lkml/2009/6/1/461).
> 
> 1. Make a working area and put the script;
>  $ mkdir stest/
>  $ cp kprobestest stest/
> 
> 2. Run it.
>  $ cd stest/
>  $ ./kprobestest run
> 
> 3. If kernel gets trouble, reboot it and run again.
>  $ cd stest/
>  $ ./kprobestest run
> 
> 4. Repeat it, until all symbols are tested.

Excellent!

> First, the test lists up all symbols in ".text" section, except
> ".text.kprobes"(*) and breaks it into sub-lists. Initially, each
> sub-lists has 64 symbols and are stored into "gray" dir.
> The test passes each sub-list to kprobes-ftrace engine and makes
> a load(currently, just sleep 2secs).

Wouldn't this be a very simplistic scenario? There could be cases where
the probes inserted were never hit during the 2sec interval, but may
potentially still be dangerous.

Ananth

Re: Kprobes stress test prototype

[Masami Hiramatsu]

Ananth N Mavinakayanahalli wrote:
> On Fri, Jun 26, 2009 at 02:33:04PM -0400, Masami Hiramatsu wrote:
>> Hi,
>>
>> Here is a script of kprobes stress test which uses
>> kprobes-ftrace engine. Currently, this doesn't use any
>> benchmarks for making a load.
>> Any updates, comments, and reports are welcome!
>>
>> On KVM/x86-64, I've found some symbols must be marked
>> as __kprobes or blacklisted by using this script.
>>
>> <always system hang up>
>> p irq_return
>> p start_critical_timing
>> p time_hardirqs_off
>> p bad_address
>> p trace_hardirqs_off_thunk
>> p restore
>>
>> <not always, but potentially dangerous>
>> p start_critical_timings
>> p stop_critical_timing
>> p trace_hardirqs_on_thunk
>> p restore_norax
>>
>> (I've encountered other odd memory corruptions too.
>>  However, I couldn't identify the reason, it might be a bug
>>  in kprobes or kvm.)
>>
>> Basic usage is very simple.
>> Of course, this needs kprobes-ftrace engine patchset which
>> I posted to lkml (http://lkml.org/lkml/2009/6/1/461).
>>
>> 1. Make a working area and put the script;
>>  $ mkdir stest/
>>  $ cp kprobestest stest/
>>
>> 2. Run it.
>>  $ cd stest/
>>  $ ./kprobestest run
>>
>> 3. If kernel gets trouble, reboot it and run again.
>>  $ cd stest/
>>  $ ./kprobestest run
>>
>> 4. Repeat it, until all symbols are tested.
> 
> Excellent!
> 
>> First, the test lists up all symbols in ".text" section, except
>> ".text.kprobes"(*) and breaks it into sub-lists. Initially, each
>> sub-lists has 64 symbols and are stored into "gray" dir.
>> The test passes each sub-list to kprobes-ftrace engine and makes
>> a load(currently, just sleep 2secs).
> 
> Wouldn't this be a very simplistic scenario? There could be cases where
> the probes inserted were never hit during the 2sec interval, but may
> potentially still be dangerous.

Yeah, right.
I think we can easily expand kprobes-ftrace engine not only for
tracing events, but also for making a statistics result. Then,
we can check whether each event has been hit via debugfs files.

Thanks,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com