Slides demonstrating our systemtap usage in production

Slides demonstrating our systemtap usage in production

[Yichun Zhang (agentzh)]

Hello folks!

I'd like to share some of my slides that demonstrate our usage of
systemtap in the production environment:

* "The Way of Optimizing and Troubleshooting Our Lua Waf"
http://agentzh.org/misc/slides/beer-meeting-2013-04-19.pdf

* "Flame Graphs for online performance profiling"
http://agentzh.org/misc/slides/yapc-na-2013-flame-graphs.pdf

* "Introduction to off-CPU Time Flame Graphs"
http://agentzh.org/misc/slides/off-cpu-flame-graphs.pdf

Basically we've been using systemtap to do online performance tuning
and trouble shooting on various levels (often at the same time): code
in scripting languages, user-space C code (for scripting language VM,
nginx, and other system software), and the OS kernel.

To make stap scripting easier and more modular (especially for
userspace probing), I've introduced a simple macro-style language
named stap++ atop systemtap:

    https://github.com/agentzh/stapxx

This tool is mentioned in the 3rd slide deck given above. And quite a
few real-world examples using stap++ are also provided there.

I must say I've been getting a lot of inspiration from the dtrace
community, especially from the dtrace.org blog site:

    http://dtrace.org/blogs/

Also I'm getting a lot of help and support from the systemtap
community of course :) I always appreciate that ;)

We're still exploring more new possibilities with the dynamic tracing
technology while systemtap already keeps saving our days right now :)

I haven't found enough time to write proper articles on these things,
so I hope you can just enjoy the slides for now :)

If you find these slides interesting, you may also want to check out
some of my other slides listed on my site here:

    http://agentzh.org/#Presentations

Enjoy!
-agentzh

Re: Slides demonstrating our systemtap usage in production

[Frank Ch. Eigler]

Hi, agentzh -

> I'd like to share some of my slides that demonstrate our usage of
> systemtap in the production environment:[...]
> http://agentzh.org/misc/slides/beer-meeting-2013-04-19.pdf
> http://agentzh.org/misc/slides/yapc-na-2013-flame-graphs.pdf
> http://agentzh.org/misc/slides/off-cpu-flame-graphs.pdf

Thanks, interesting work!


> [...]
> To make stap scripting easier and more modular (especially for
> userspace probing), I've introduced a simple macro-style language
> named stap++ atop systemtap:
>     https://github.com/agentzh/stapxx

It turns out that a lot of what you are doing in this package is
supported or almost supported in the base.  Would you be interested in
porting over the unique bits?


$^exec_path: ISTR there was a PR for this, but can't find it now.
             This functionality could go into core stap, implementing
             the exact same heuristic.  See how 
                 stap -e 'probe process.FOO {}' -c 'CMD ARGS...'
             is implemented.  One complication with this is compile-server
             mode (where -x-expansion cannot be relied upon to match -x on
             the invoking host).  We could just punt on this for now.


$^libFOO_path: Could be a combined with the above:
                 stap -e 'probe process.library("FOO").BAR { }' -c 'CMD ARGS...'


$^arg_NAME: Have you looked at the
                 stap ... -G name=value
            mechanism for setting integer/string globals?  If you desire
            named parameters in a context other than for global variables,
            perhaps stap could grow a -M macro='value' option, which would
            work like a
                 @define macro %( value %) 
            in an stpm file.


default values: in the context of globals, could be done thusly:
            stap -e '
               global foo = 5 /* default */
               probe begin { println(foo) }' -G foo=10


macros: since stap 2.0, see smakarov's @define facility:
            @define foo(x) %( 2, 4, @x %)
            probe begin { println ( @foo(6) ) }


tapset modules: we hadn't seen a requirement for module/namespacing constructs
                in the tapset, so things work more automagically:

                % cat tapset/socketfd-lookup.stp
                function BAR ()  { }
                % stap -Itapset -e 'probe begin { BAR() }'  # just works


@pfunc(): likely implementable as a single stap macro:
          @define pfunc(x) %( process.function(@x) %)


- FChE

Re: Slides demonstrating our systemtap usage in production

[Yichun Zhang (agentzh)]

Hello Frank!

On Mon, Sep 9, 2013 at 4:01 PM, Frank Ch. Eigler wrote:
>
> It turns out that a lot of what you are doing in this package is
> supported or almost supported in the base.  Would you be interested in
> porting over the unique bits?
>

Yeah sure. Actually that was the plan :) stap++ is good for playing
with various features and notations :)

>
> $^exec_path: ISTR there was a PR for this, but can't find it now.
>              This functionality could go into core stap, implementing
>              the exact same heuristic.  See how
>                  stap -e 'probe process.FOO {}' -c 'CMD ARGS...'
>              is implemented.  One complication with this is compile-server
>              mode (where -x-expansion cannot be relied upon to match -x on
>              the invoking host).  We could just punt on this for now.
>

Great! :)

>
> $^libFOO_path: Could be a combined with the above:
>                  stap -e 'probe process.library("FOO").BAR { }' -c 'CMD ARGS...'
>

Nice :)

>
> $^arg_NAME: Have you looked at the
>                  stap ... -G name=value
>             mechanism for setting integer/string globals?  If you desire
>             named parameters in a context other than for global variables,
>             perhaps stap could grow a -M macro='value' option, which would
>             work like a
>                  @define macro %( value %)
>             in an stpm file.
>

I like the idea of the -M macro=value option because I often use this
for contexts where global variables cannot be used, as in probe
declarators, for instance, the func-latency-distr tool written in
stap++:

https://github.com/agentzh/stapxx/blob/master/samples/func-latency-distr.sxx

For default values, maybe we can make -M override existing macros
defined in the .stp/.stpm files?

>
> macros: since stap 2.0, see smakarov's @define facility:
>             @define foo(x) %( 2, 4, @x %)
>             probe begin { println ( @foo(6) ) }
>

Oh yeah, this is very useful :)

>
> tapset modules: we hadn't seen a requirement for module/namespacing constructs
>                 in the tapset, so things work more automagically:
>

I prefer proper module systems because

1. it can reduce the startup time of systemtap when there are a huge
number of tapset functions ready for use.

2. it's easy to specify dependency relationships among different sets
of tapsets.

>
> @pfunc(): likely implementable as a single stap macro:
>           @define pfunc(x) %( process.function(@x) %)
>

Yes, you are right :) The @pfunc thing can be implemented just as a
macro in a predefined tapset file :)

Thanks!
-agentzh