Performance Impact of Systemtap

Performance Impact of Systemtap

[ydev]

Hi,

I want to measure the overhead introduced by systemtap scripts. 
For example, say I were to introduce a SCSI fault during a read
or write op and I want to measure the overhead by systemtap in doing
this.

What would be the best way to do it.

Thanks,
Yathi

Re: Performance Impact of Systemtap

[Frank Ch. Eigler]

ydev wrote:

> I want to measure the overhead introduced by systemtap scripts.  For
> example, say I were to introduce a SCSI fault during a read or write
> op and I want to measure the overhead by systemtap in doing this. [...]

General time costs come in two pieces: the costs of dispatching the
underlying event to the systemtap module, and the execution time of
the systemtap probe handler itself.  The former depends on what kind
of probe point you attach to (and may vary with kernel version,
architecture, tracepoints cheaper than kprobes, etc.); the latter
depends on your script contents (and may very with systemtap version).

The former may be (under)estimated with microbenchmarks that
repeatedly trigger the particular kind of probe for a null kind of
handler (as in a -DSTP_ALIBI mode run).  The latter may be
(under)estimated with stap -t (timing mode), which prints cycles taken
up by the handlers themselves.

The best way is to measure the impact of your particular workload.  It
may be immeasurably small there.

- FChE

Re: Performance Impact of Systemtap

[Frank Ch. Eigler]

On Wed, Sep 07, 2011 at 08:42:23PM -0600, ydev wrote:
> Thank you Frank. I'm very new to SystemTap so could you please
> tell me what is the -DSTP_ALIBI mode run.

-DSTP_ALIBI compiles out the probe handlers from the systemtap module,
leaving only the parts that hook into the kernel, and just keep hit
counts/time.  So it's as if the probe handlers were all { } empty.

> [...] I started systemtap in flight mode and triggered many
> reads/write and used the 'time' command to measure how long it takes
> to execute cat.

OK.

> I did this without systemtap running in the background. The figures
> were almost the same. I'm not sure if using 'time' command gives me
> accurate results.

Overall elapsed times should be accurate with /bin/time.
Instrumenting relatively few & slow calls like disk I/O should indeed
have negligible overhead when instrumented with systemtap.

- FChE