help with running systemtap

help with running systemtap

[p v]

Hello,

I've read most of the FAQ's on the web (and more) but I still can't figure out what's wrong with my setup. I've installed fresh FC9 on top of which I did install a debug kernel rpm. But stap is still unhappy and returning the "semantic error: libdwfl failure". Stracing it it shows that it's looking for the debug version of the vmlinux. The debug kernel package does have it but it's compressed - "vmlinuz" and puts this compressed version under /boot. What other steps do I need to perform in order to get stap working? Do I need additional packages (I am sorry  don't have the output of strace on hand with me right now but if requested I'll post it). Simply what I am asking is - what are the steps needed to be able to run stap after installing fresh FC9. It's surprising that stap packages are installed although they are useless without some additional steps. Is there any other linux distribution which would have stap running out of the box? Also how stable
 is stap? Is it stable enough to be run on production machines?

I apologize if this is the wrong forum for these questions in which case plese can you send me a pointer to the right forum?

thx

--pv
pvlogin@yahoo.com



      

Re: help with running systemtap

[Frank Ch. Eigler]

p v <pvlogin@yahoo.com> writes:

> I've read most of the FAQ's on the web (and more) but I still can't
> figure out what's wrong with my setup. I've installed fresh FC9 on
> top of which I did install a debug kernel rpm.

The confusion here is that fedora includes a fully separate "debug"
variant of the kernel that enables lockdep and other functions,
whereas systemtap needs the "-debuginfo" associated with any
particular kernel variant.  So, install "kernel-debug-debuginfo" and
"kernel-debug-devel".

> It's surprising that stap packages are installed although they are
> useless without some additional steps. [...]

One reason for this is that there is no convenient RPM syntax for
systemtap to declare a dependency upon a matching pair of
kernel-*-debuginfo and kernel-*-devel.

> Also how stable is stap? Is it stable enough to be run on production
> machines?

A number of organizations are using it in that capacity.  Most of them
carefully test their scripts first on a development machine.  The
nature of our failures is that things rarely fail on the former if
they worked on the latter.

- FChE

Re: help with running systemtap

[Frank Ch. Eigler]

Hi -

On Mon, Sep 08, 2008 at 11:45:54PM -0700, p v wrote:
> [...]
> [root@robot ~]# stap -v -k hello.stp
> [...]
> Pass 3: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.c
> Pass 4: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko
> Pass 5: starting run.
> Error inserting module '/tmp/stapvQz24y/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko': Invalid module format
> [...]

Here's the key hint.  Systemtap reused a prior compiled version of the
same script, which must have been built back when you had the .14-108
kernels set up.  It should not have done that - that's a bug.  You can
work around it by "rm -rf /root/.systemtap/cache".

- FChE

Re: help with running systemtap

[p v]

I think I am running the correct kernel -

[root@robot ~]# uname -a
Linux robot 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686 i686 i386 GNU/Linux

however I did install wrong debuginfo packages at first (14-108) but I did removed them and installed the correct ones. Maybe rpm -e left something around? I  guess I should just reinstall all at this point ...

--pv




----- Original Message ----
From: Wenji Huang <wenji.huang@oracle.com>
To: p v <pvlogin@yahoo.com>
Cc: Frank Ch. Eigler <fche@redhat.com>; systemtap@sourceware.org
Sent: Monday, September 8, 2008 11:52:18 PM
Subject: Re: help with running systemtap

p v wrote:
> Thanks for the reply. I got little bit further but stap is still not working for me. These are the packages installed -
> 
> [root@robot ~]# rpm -qa | grep kernel
> kernel-devel-2.6.25-14.fc9.i686
> kernel-debuginfo-2.6.25-14.fc9.i686
> kernel-headers-2.6.25-14.fc9.i386
> kerneloops-0.10-11.fc9.i386
> kernel-debuginfo-common-2.6.25-14.fc9.i686
> kernel-2.6.25-14.fc9.i686
> 
> stap does find the debug info it needs to compile but it fails to insmod with "Invalid module format" -
> 
> [root@robot ~]# stap -v -k hello.stp
> Pass 1: parsed user script and 38 library script(s) in 250usr/10sys/268real ms.
> Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 10usr/0sys/3real ms.
> Pass 3: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.c
> Pass 4: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko
> Pass 5: starting run.
> Error inserting module '/tmp/stapvQz24y/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko': Invalid module format
> Pass 5: run completed in 0usr/0sys/4real ms.
> Pass 5: run failed.  Try again with more '-v' (verbose) options.
> Keeping temporary directory "/tmp/stapvQz24y"
> 
> and dmesg spits out -
> 
> stap_2b11f1a63be8d5e9310e1f646ee698de_264: version magic '2.6.25.14-108.fc9.i686 SMP mod_unload 686 4KSTACKS ' should be '2.6.25-14.fc9.i686 SMP mod_unload 686 4KSTACKS '
> 
> where is the wrong version coming from?
I guess your running kernel is 2.6.25.14-108.fc9.i686, see uname -a.
But your kernel-debuginfo is 2.6.25-14.fc9.i686. Those two don't match 
as stap requires.

Regards,
Wenji


      

Re: help with running systemtap

[Wenji Huang]

p v wrote:
> Thanks for the reply. I got little bit further but stap is still not working for me. These are the packages installed -
> 
> [root@robot ~]# rpm -qa | grep kernel
> kernel-devel-2.6.25-14.fc9.i686
> kernel-debuginfo-2.6.25-14.fc9.i686
> kernel-headers-2.6.25-14.fc9.i386
> kerneloops-0.10-11.fc9.i386
> kernel-debuginfo-common-2.6.25-14.fc9.i686
> kernel-2.6.25-14.fc9.i686
> 
> stap does find the debug info it needs to compile but it fails to insmod with "Invalid module format" -
> 
> [root@robot ~]# stap -v -k hello.stp
> Pass 1: parsed user script and 38 library script(s) in 250usr/10sys/268real ms.
> Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 10usr/0sys/3real ms.
> Pass 3: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.c
> Pass 4: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko
> Pass 5: starting run.
> Error inserting module '/tmp/stapvQz24y/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko': Invalid module format
> Pass 5: run completed in 0usr/0sys/4real ms.
> Pass 5: run failed.  Try again with more '-v' (verbose) options.
> Keeping temporary directory "/tmp/stapvQz24y"
> 
> and dmesg spits out -
> 
> stap_2b11f1a63be8d5e9310e1f646ee698de_264: version magic '2.6.25.14-108.fc9.i686 SMP mod_unload 686 4KSTACKS ' should be '2.6.25-14.fc9.i686 SMP mod_unload 686 4KSTACKS '
> 
> where is the wrong version coming from?
I guess your running kernel is 2.6.25.14-108.fc9.i686, see uname -a.
But your kernel-debuginfo is 2.6.25-14.fc9.i686. Those two don't match 
as stap requires.

Regards,
Wenji

Re: help with running systemtap

[p v]

Thanks for the reply. I got little bit further but stap is still not working for me. These are the packages installed -

[root@robot ~]# rpm -qa | grep kernel
kernel-devel-2.6.25-14.fc9.i686
kernel-debuginfo-2.6.25-14.fc9.i686
kernel-headers-2.6.25-14.fc9.i386
kerneloops-0.10-11.fc9.i386
kernel-debuginfo-common-2.6.25-14.fc9.i686
kernel-2.6.25-14.fc9.i686

stap does find the debug info it needs to compile but it fails to insmod with "Invalid module format" -

[root@robot ~]# stap -v -k hello.stp
Pass 1: parsed user script and 38 library script(s) in 250usr/10sys/268real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 10usr/0sys/3real ms.
Pass 3: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.c
Pass 4: using cached /root/.systemtap/cache/2b/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko
Pass 5: starting run.
Error inserting module '/tmp/stapvQz24y/stap_2b11f1a63be8d5e9310e1f646ee698de_264.ko': Invalid module format
Pass 5: run completed in 0usr/0sys/4real ms.
Pass 5: run failed.  Try again with more '-v' (verbose) options.
Keeping temporary directory "/tmp/stapvQz24y"

and dmesg spits out -

stap_2b11f1a63be8d5e9310e1f646ee698de_264: version magic '2.6.25.14-108.fc9.i686 SMP mod_unload 686 4KSTACKS ' should be '2.6.25-14.fc9.i686 SMP mod_unload 686 4KSTACKS '

where is the wrong version coming from?

thanks

--pv




----- Original Message ----
From: Frank Ch. Eigler <fche@redhat.com>
To: p v <pvlogin@yahoo.com>
Cc: systemtap@sourceware.org
Sent: Friday, September 5, 2008 11:37:16 AM
Subject: Re: help with running systemtap


p v <pvlogin@yahoo.com> writes:

> I've read most of the FAQ's on the web (and more) but I still can't
> figure out what's wrong with my setup. I've installed fresh FC9 on
> top of which I did install a debug kernel rpm.

The confusion here is that fedora includes a fully separate "debug"
variant of the kernel that enables lockdep and other functions,
whereas systemtap needs the "-debuginfo" associated with any
particular kernel variant.  So, install "kernel-debug-debuginfo" and
"kernel-debug-devel".

> It's surprising that stap packages are installed although they are
> useless without some additional steps. [...]

One reason for this is that there is no convenient RPM syntax for
systemtap to declare a dependency upon a matching pair of
kernel-*-debuginfo and kernel-*-devel.

> Also how stable is stap? Is it stable enough to be run on production
> machines?

A number of organizations are using it in that capacity.  Most of them
carefully test their scripts first on a development machine.  The
nature of our failures is that things rarely fail on the former if
they worked on the latter.

- FChE