Newb: Probe My Application

Newb: Probe My Application

[Hassan Radwan]

Hello,

I recently found out about systemtap and am still getting the hang of
things. I'm interested in trying to use systemtap as an alternative to a
debugger to probe my own program that I wrote. To test things out I
wrote a simple program and comiled it to an executable. But after this
I'm stumped as to what to do to get systemtap to find the probe points
in my program.

So let's say I have an executable "foo" compiled with g++ using the -g
flag. How do I tell systemtap that I'm interested in all methods within
that one executable? All the examples I've seen so far point towards
kernel related probes. Is this something that can be done using
systemtap?

I've tried:

probe module("foo").*

probe process("./foo").*

And run the scripts using the following command:

> stap foo.stp -c foo

The first didn't find any probe points the second gave me a compilation
error on "pass 4". Any advice to a helpless newb?

Thanks!
Hassan

Re: Newb: Probe My Application

[Frank Ch. Eigler]

"Hassan Radwan" <HassanR@avi.com> writes:

> [...]  I recently found out about systemtap and am still getting the
> hang of things. I'm interested in trying to use systemtap as an
> alternative to a debugger to probe my own program that I wrote. [...]

Great.

> [...] How do I tell systemtap that I'm interested in all methods within
> that one executable? [...]

If by "all methods" you mean "all functions", this would work:

  probe process("a.out").function("*") { log ($$parms) }

as would .statement("foo@file.c:333") {} and so on.  In time, we hope
to have special syntax for C++ apps.

The big proviso is that this "user-space probing" support currently
only works against a version of the kernel-side utrace API that is
somewhat obsolete and is only present in RHEL5 / CentOS / older Fedora
type kernels.  This should be fixed soon.


- FChE

RE: Newb: Probe My Application

[Hassan Radwan]

Frank, thanks for your help! I setup CentOS 5.2 and compiled and
installed systemtap on it. I'm getting further along now but still being
stopped by an error at Pass 5:

# more aprobe.stp
probe process("a.out").function("*") {
  log($$parms)
}

# stap aprobe.stp -c ./a.out
Pass 1: parsed user script and 45 library script(s) in
100usr/200sys/338real ms.
Pass 2: analyzed script: 9 probe(s), 4 function(s), 0 embed(s), 0
global(s) in 10usr/10sys/18real ms.
Pass 3: using cached
/home/radwan/.systemtap/cache/ad/stap_ad195bef414096e2124bb78599d13e1f_3
407.c
Pass 4: using cached
/home/radwan/.systemtap/cache/ad/stap_ad195bef414096e2124bb78599d13e1f_3
407.ko
Pass 5: starting run.
ERROR: module release mismatch (2.6.18-92.1.13.el5 vs 2.6.18-92.el5)

Waiting for processes to exit
Pass 5: run completed in 0usr/60sys/96real ms.
# uname -r
2.6.18-92.el5

I had to update the uprobes runtime by running make in
/usr/local/share/systemtap/runtime/uprobes/ after getting an error
message from stap about uprobes being out of date.

Any help about how to resolve the "module release mismatch" error would
be greatly appreciated!

-----Original Message-----
From: Frank Ch. Eigler [mailto:fche@redhat.com] 
Sent: Thursday, October 09, 2008 11:08 AM
To: Hassan Radwan
Cc: systemtap@sourceware.org
Subject: Re: Newb: Probe My Application


"Hassan Radwan" <HassanR@avi.com> writes:

> [...]  I recently found out about systemtap and am still getting the
> hang of things. I'm interested in trying to use systemtap as an
> alternative to a debugger to probe my own program that I wrote. [...]

Great.

> [...] How do I tell systemtap that I'm interested in all methods
within
> that one executable? [...]

If by "all methods" you mean "all functions", this would work:

  probe process("a.out").function("*") { log ($$parms) }

as would .statement("foo@file.c:333") {} and so on.  In time, we hope
to have special syntax for C++ apps.

The big proviso is that this "user-space probing" support currently
only works against a version of the kernel-side utrace API that is
somewhat obsolete and is only present in RHEL5 / CentOS / older Fedora
type kernels.  This should be fixed soon.


- FChE

Re: Newb: Probe My Application

[Frank Ch. Eigler]

Hi -

On Tue, Oct 14, 2008 at 02:36:34PM -0400, Hassan Radwan wrote:
> # stap aprobe.stp -c ./a.out
> [...]
> ERROR: module release mismatch (2.6.18-92.1.13.el5 vs 2.6.18-92.el5)
> # uname -r
> 2.6.18-92.el5
> [...]

I wonder if this is a collision in our cache hashing.  It seems like
the module found was built for a different kernel, which is not
supposed to happen.  Have you up- or downgraded your kernel recently?

Clean out the cache (rm -rf $HOME/.systemtap/cache) and try again.

- FChE

Re: Newb: Probe My Application

[Masami Hiramatsu]

Hi,

Hassan Radwan wrote:
> Pass 5: starting run.
> ERROR: module release mismatch (2.6.18-92.1.13.el5 vs 2.6.18-92.el5)

Your kernel-devel(and kernel) package might have been updated.
Could you check it by executing "rpm -q kernel-devel" ?

If the version was different from the result of "uname -r",
you would have to ;
1. make sure you have installed new kernel package(2.6.18-92.1.13.el5).
 $ rpm -q kernel-2.6.18-92.1.13.el5
2. update your kernel-debuginfo package by debuginfo-install as below.
 $ sudo debuginfo-install kernel
3. reboot the system and boot with the newer kernel(2.6.18-92.1.13.el5)
4. retry to run stap! :-)

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

RE: Newb: Probe My Application

[Hassan Radwan]

Frank, Masami, thanks a lot!

My problem was indeed a slight mismatch between the kernel-devel and the
actual kernel versions installed. I'm now happily probing away at my
application :)

Thanks!

-----Original Message-----
From: Masami Hiramatsu [mailto:mhiramat@redhat.com] 
Sent: Tuesday, October 14, 2008 3:47 PM
To: Hassan Radwan
Cc: systemtap@sourceware.org
Subject: Re: Newb: Probe My Application

Hi,

Hassan Radwan wrote:
> Pass 5: starting run.
> ERROR: module release mismatch (2.6.18-92.1.13.el5 vs 2.6.18-92.el5)

Your kernel-devel(and kernel) package might have been updated.
Could you check it by executing "rpm -q kernel-devel" ?

If the version was different from the result of "uname -r",
you would have to ;
1. make sure you have installed new kernel package(2.6.18-92.1.13.el5).
 $ rpm -q kernel-2.6.18-92.1.13.el5
2. update your kernel-debuginfo package by debuginfo-install as below.
 $ sudo debuginfo-install kernel
3. reboot the system and boot with the newer kernel(2.6.18-92.1.13.el5)
4. retry to run stap! :-)

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com