Systemtap userspace probing: status and roadmap?

Systemtap userspace probing: status and roadmap?

[Stefano Doni]

Hi guys,
    first of all let me thank you for the efforts you give on this
wonderful tool! I'm a great fan of this project and of this kind of
tools in general, since I have a passion for performance analysis,
testing and tuning. I often read that the you are willing to hear some
feedback about what user wants, so I thought it would be nice to ask
you some question and share some whishes that a user can have about
such a good tool.

I would like to know what is the current status of Systemtap user
space probing and what is the future roadmap (if such a thing exists).
For example, man stapprobes in my Ubuntu Intrepid tells me that only
"early prototype support" for user space probing is available.

IMHO a great part of value of this kind of tools is to be able to
observe the whole system, in particular the user space layer, which is
often the culprit of real life performance issues. This meas to be
able to measure how things go into the application, for example doing
all the kind of measurements that a profiler can perform, such as:

- inclusive and exclusive method latencies
- object allocation
- lock contention
- "hot" application areas (CPU and IO resources)

Having *one* tool such as Systemtap and being able to perform *all*
the performance analysis on a *live* system without acquiring external
(and intrusive) profilers to me is the *greatest* value of a tool like
that.

I read that DTrace already do this for a bunch of programming
languages (most importantly Java but also Ruby, Python and even
Javascript). Having Systemtap being able to do this kind of
measurements (for example with Java, usually the language found in
enterprise installations) would be definitely my best dream!

Thanks again and keep up with the hard work!

Re: Systemtap userspace probing: status and roadmap?

[Jim Keniston]

On Tue, 2008-11-04 at 19:31 +0100, Stefano Doni wrote:
> Hi guys,
>     first of all let me thank you for the efforts you give on this
> wonderful tool! I'm a great fan of this project and of this kind of
> tools in general, since I have a passion for performance analysis,
> testing and tuning. I often read that the you are willing to hear some
> feedback about what user wants, so I thought it would be nice to ask
> you some question and share some whishes that a user can have about
> such a good tool.
> 
> I would like to know what is the current status of Systemtap user
> space probing and what is the future roadmap (if such a thing exists).
> For example, man stapprobes in my Ubuntu Intrepid tells me that only
> "early prototype support" for user space probing is available.

SystemTap does user-space probing the same way it does kernel probing:
by generating and inserting a kernel module that implements the desired
implementation.  Just as subsystems like kprobes and markers provide the
infrastructure for kernel probing, utrace and uprobes provide the
infrastructure for user-space probing.  A SystemTap script can include
both kernel and user-space probes.

Utrace-based probes are described under the heading "USER-SPACE" in
stapprobes(5).  To probe a particular function or statement in a user
program (i.e., using uprobes), you can do stuff like
	probe process("/path/to/executable").function("funcname") ...
You can also probe function returns by adding ".return" to the probe
specification.

Uprobes depends on utrace.  Neither utrace nor uprobes are in Linus's
kernel.  Utrace is included with RHEL 5, or you can get a utrace patch
for your kernel.org kernel at http://people.redhat.com/roland/utrace/ .
Uprobes is packaged with SystemTap, in runtime/uprobes*/; stap will
automatically build and insert uprobes.ko if your script uses uprobes
(but again, you must be running a utrace-enabled kernel).

Our intention is certainly for utrace and uprobes to be accepted
upstream eventually, but I won't try to predict how/when that will
happen.  Uprobes in particular is about to undergo some rework.

Stap has been exploiting uprobes for only a few months, and we're still
shaking out the bugs.  In fact, the 2.6.27-compatible version of uprobes
was added to SystemTap just this week.  You're welcome to give it a try.

Jim Keniston
IBM Linux Technology Center
uprobes maintainer

Re: Systemtap userspace probing: status and roadmap?

[Jim Keniston]

On Tue, 2008-11-04 at 13:16 -0800, Jim Keniston wrote:
...
> 
> SystemTap does user-space probing the same way it does kernel probing:
> by generating and inserting a kernel module that implements the desired
> implementation.

Oops.  s/implementation/instrumentation/ :-}

Jim

Re: Systemtap userspace probing: status and roadmap?

[Stefano Doni]

Hi Jim,
   thanks for the reply.

Now the whole situation is more clear to me. I didn't know that
utrace/uprobe wasn't part of mainline kernel, I really look forward
for this to happen.

Is there any linux distro easy to try with that patches included?
Fedora 10? This way systemtap testing would be much easier in a VM,
for example.

Are there any plan to start the inclusion into mainline in the next
kernel releases?

Thanks

2008/11/4 Jim Keniston <jkenisto@us.ibm.com>:
> On Tue, 2008-11-04 at 19:31 +0100, Stefano Doni wrote:
>> Hi guys,
>>     first of all let me thank you for the efforts you give on this
>> wonderful tool! I'm a great fan of this project and of this kind of
>> tools in general, since I have a passion for performance analysis,
>> testing and tuning. I often read that the you are willing to hear some
>> feedback about what user wants, so I thought it would be nice to ask
>> you some question and share some whishes that a user can have about
>> such a good tool.
>>
>> I would like to know what is the current status of Systemtap user
>> space probing and what is the future roadmap (if such a thing exists).
>> For example, man stapprobes in my Ubuntu Intrepid tells me that only
>> "early prototype support" for user space probing is available.
>
> SystemTap does user-space probing the same way it does kernel probing:
> by generating and inserting a kernel module that implements the desired
> implementation.  Just as subsystems like kprobes and markers provide the
> infrastructure for kernel probing, utrace and uprobes provide the
> infrastructure for user-space probing.  A SystemTap script can include
> both kernel and user-space probes.
>
> Utrace-based probes are described under the heading "USER-SPACE" in
> stapprobes(5).  To probe a particular function or statement in a user
> program (i.e., using uprobes), you can do stuff like
>        probe process("/path/to/executable").function("funcname") ...
> You can also probe function returns by adding ".return" to the probe
> specification.
>
> Uprobes depends on utrace.  Neither utrace nor uprobes are in Linus's
> kernel.  Utrace is included with RHEL 5, or you can get a utrace patch
> for your kernel.org kernel at http://people.redhat.com/roland/utrace/ .
> Uprobes is packaged with SystemTap, in runtime/uprobes*/; stap will
> automatically build and insert uprobes.ko if your script uses uprobes
> (but again, you must be running a utrace-enabled kernel).
>
> Our intention is certainly for utrace and uprobes to be accepted
> upstream eventually, but I won't try to predict how/when that will
> happen.  Uprobes in particular is about to undergo some rework.
>
> Stap has been exploiting uprobes for only a few months, and we're still
> shaking out the bugs.  In fact, the 2.6.27-compatible version of uprobes
> was added to SystemTap just this week.  You're welcome to give it a try.
>
> Jim Keniston
> IBM Linux Technology Center
> uprobes maintainer
>
>

Re: Systemtap userspace probing: status and roadmap?

[Mark Wielaard]

Hi Stefano,

On Wed, 2008-11-05 at 12:33 +0100, Stefano Doni wrote:
> Now the whole situation is more clear to me. I didn't know that
> utrace/uprobe wasn't part of mainline kernel, I really look forward
> for this to happen.

Parts of utrace are now trickling into the upstream kernel.

> Is there any linux distro easy to try with that patches included?
> Fedora 10? This way systemtap testing would be much easier in a VM,
> for example.

Systemtap contains the relevant uprobes sources now. CentOS 5 comes with
a utrace included in the kernel that is compatible with the uprobes
version 1, which is pretty stable (but based on an older 2.6.18 kernel).

Fedora 10 will come (current prereleases/rawhide already have) a utrace
compatible with the uprobes version 2 that Jim just landed in systemtap
git (the code picks up the right version depending on the utrace version
in the kernel it is running on). I just tested that against the rawhide
kernel 2.6.27.4-68.fc10.i686 and it seems to work for some quick and
dirty test cases. Hurray for Jim!

Note, that Fedora 9 kernels (2.6.26 based) contain an alternative utrace
version that doesn't seem very stable combined with the new uprobes
version 2 (and doesn't work at all against uprobes version 1). So you
really want either a CentOS or Fedora 10/rawhide kernel for now to play
with the user space probes.

Cheers,

Mark

Re: Systemtap userspace probing: status and roadmap?

[Mark Wielaard]

Hi,

On Wed, 2008-11-05 at 14:00 +0100, Mark Wielaard wrote:
> Note, that Fedora 9 kernels (2.6.26 based) contain an alternative
> utrace version that doesn't seem very stable combined with the new
> uprobes version 2 (and doesn't work at all against uprobes version 1).
> So you really want either a CentOS or Fedora 10/rawhide kernel for now
> to play with the user space probes.

This seems untrue. It must have been my local changes that made it not
work. There is an issue currently with utrace as in the fedora kernel
(any version) as described in bug #6964. The workaround for that
(currently committed in git 1b1d63) makes that -c doesn't work reliably.
I had that workaround disabled which seems to have caused my
instability. So the latest Fedora 9 kernel should be enough.

With current trunk systemtap and 2.6.26.6-79.fc9.x86_64
things like stap -v -e 'probe process("/bin/ls").function("*").call
{printf("%s=>%s(%s)\n", thread_indent(1),probefunc(), $$parms)} probe
process("/bin/ls").function("*").return {printf("%s<=%s %s\n",
thread_indent(-1), probefunc(), $$return)}' work just fine (given that
you have core-utils-debuginfo installed). And look pretty spectacular :)

Cheers,

Mark