Fwd: staprun-only system

Fwd: staprun-only system

[Jim Keniston]

Here's email from Mike Anderson, an IBMer who has been using SystemTap
for some time.  As indicated, the staprun-only build is not working for
him as desired, and he has taken considerable pains to work around the
failures he sees.

I'm forwarding this with his permission in hopes that someone can give
him some guidance.

Thanks.
Jim
-------- Forwarded Message --------
From: Mike Anderson <andmike@linux.vnet.ibm.com>
To: jkenisto@linux.vnet.ibm.com
Subject: Re: staprun-only system
Date: Fri, 17 Apr 2009 13:50:20 -0700

jkenisto@linux.vnet.ibm.com wrote:
>    Hi, Mike.  We talked about your desire for a stripped-down staprun-only
>    build.  This is supposed to be supported.  The simplest approach,
>    according to Frank Eigler, is to do:
>    $ ./configure     # Ignore messages about missing components.
>    $ make staprun
> 
>    I haven't tried this.
> 
>    There's also documentation about this in Section 2.2 in the SystemTap
>    Beginner's Guide (see
>    [1]http://sourceware.org/systemtap/documentation.html).
> 

Thanks for the email.

The -g (guru) mode solved my blacklist issue.
"... guru mode enabled - ignoring blacklist ..."

On the stripped down staprun for the target system here is what I
observed using the current systemtap git tree (systemtap-0.9.5-25620).

In the "SystemTap Beginner's Guide" section 2.2 the target system setup
uses systemtap-runtime not the configure / build steps (which makes sense
for a beginner's guide). In my test cases this prebuilt runtime will not
always work.

1.) In early distro bring up sometimes the systemtap-runtime is not
available or the distro we are testing does not provide it.

2.) In some cases I want to run the latest systemtap version for the git
tree. In the future I assume this need will be reduced.

I tried the ./configure on the target system and it did not generate a
Makefile as the elfutils where not installed, I could install them, but
since staprun does not need them I wanted to simplify my target system
default install by not requiring extra rpms for debug. 

I hacked the configure script to allow a Makefile to be created. I was
then able to do a make staprun, but I could not do a make install just for
staprun. To run staprun in wants stapio in the prefix location. I can
leave the --prefix off my configure operation, which generates a not
supported message from configure but still I receive the
"/usr/local/libexec/systemtap/stapio: No such file or directory" message.

Since make install did not work. I copied the files by hand. I am still
cleaning up a issue where staprun will not load the previously built
scsi_test systemtap module.
"# staprun ./scsi_test.ko
ERROR: scsi_test: inconsistent scsi_mod build-id byte #0 (0x6e [actual]
vs. 0x83 [debuginfo])"

In essence what I was looking for is close to working, but my point
was that it would be nice if a few things where cleaned up to eliminate
the workarounds to make a cleaner build / install staprun only process
that I think a number of developers would use.

Here is my current process.

#On the Build system.
1.) Pull latest systemtap git tree.
2.) Generate tar file from git tree # Not needed for build system, but keeps target and build in sync.
3.) Untar systemtap tar file to a directory on the build system.
4.) ./configure  --prefix=/home/andmike/systemtap-0.9.5-25620 --disable-server --disable-refdocs --disable-docs --disable-sqlite
5.) make; make install
6.) Add /home/andmike/systemtap-0.9.5-25620/bin to path
7.) stap -g -r ${BUILD_TREE} -m scsi_test scsi_test.stp
/lib/modules/2.6.30-rc1-v2.6.30-rc1-dirty/systemtap/scsi_test.ko':
8.) rsync scsi_test.ko module to target system.

# On Target system.
1.) Untar systemtap tar file to a directory on the target system.
2.) Install libdw-devel-0.137-8.19.x86_64.rpm libebl-devel-0.137-8.19.x86_64.rpm libelf-devel-0.137-8.19.x86_64.rpm
or
2.) Hack configure script for the elfutils checks.
3.) ./configure  --prefix=/root/systemtap-0.9.5-25620 --disable-server --disable-refdocs --disable-docs --disable-sqlite
5.) make staprun stapio
6.) Hack the install to /root/systemtap-0.9.5-25620 so staprun will work.
	- mkdir -p ~/systemtap-0.9.5-25620/libexec/systemtap
	- mkdir -p ~/systemtap-0.9.5-25620/bin
	- cp staprun ~/systemtap-0.9.5-25620/bin
	- cp stapio ~/systemtap-0.9.5-25620/libexec/systemtap
4.) ./staprun ~/scsi_test.ko

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

Re: Fwd: staprun-only system

[David Smith]

Jim Keniston wrote:
> Here's email from Mike Anderson, an IBMer who has been using SystemTap
> for some time.  As indicated, the staprun-only build is not working for
> him as desired, and he has taken considerable pains to work around the
> failures he sees.
> 
> I'm forwarding this with his permission in hopes that someone can give
> him some guidance.
> 
> Thanks.
> Jim
> -------- Forwarded Message --------
> From: Mike Anderson <andmike@linux.vnet.ibm.com>
> To: jkenisto@linux.vnet.ibm.com
> Subject: Re: staprun-only system
> Date: Fri, 17 Apr 2009 13:50:20 -0700
> 
> jkenisto@linux.vnet.ibm.com wrote:
>>    Hi, Mike.  We talked about your desire for a stripped-down staprun-only
>>    build.  This is supposed to be supported.  The simplest approach,
>>    according to Frank Eigler, is to do:
>>    $ ./configure     # Ignore messages about missing components.
>>    $ make staprun
>>
>>    I haven't tried this.
>>
>>    There's also documentation about this in Section 2.2 in the SystemTap
>>    Beginner's Guide (see
>>    [1]http://sourceware.org/systemtap/documentation.html).
>>
> 
> Thanks for the email.
> 
> The -g (guru) mode solved my blacklist issue.
> "... guru mode enabled - ignoring blacklist ..."
> 
> On the stripped down staprun for the target system here is what I
> observed using the current systemtap git tree (systemtap-0.9.5-25620).

... steps deleted ...

Assuming you are using an rpm-based system, here's what you should do.
Install your distro-supplied 'systemtap-runtime' rpm on the target system.

If you want the latest .git version, you would do something like the
following on your build system:

1.) Pull latest systemtap git tree.
2.) Generate tar file from git tree
3.) Run rpmbuild on the tar file and spec file.  Something like:
    # rpmbuild --define "_topdir `pwd`" --define "_sourcedir `pwd`" -bb
systemtap.spec
4.) Copy RPMS/{ARCH}/systemtap-runtime-*.rpm to the target system
5.) Install the systemtap-runtime rpm on the target system.
6.) Install RPMS/{ARCH}/systemtap-* rpms on the build system (minus the
debuginfo rpm and probably the testsuite rpm)

Now you have matching systemtap versions on both your build and target
systems.

If you aren't using an rpm-based system, I'd basically mimic what the
spec file does.  Configure and build, then copy the files mentioned in
the '%files runtime' section over to the target system.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Fwd: staprun-only system

[Frank Ch. Eigler]

Jim Keniston <jkenisto@us.ibm.com> forwarded:

> [...]
> I tried the ./configure on the target system and it did not generate a
> Makefile as the elfutils where not installed, I could install them, but
> since staprun does not need them I wanted to simplify my target system
> default install by not requiring extra rpms for debug. 

We could add a configure option to disable building the translator,
and thus only build staprun/stapio.


> Since make install did not work. I copied the files by hand. I am still
> cleaning up a issue where staprun will not load the previously built
> scsi_test systemtap module.
> "# staprun ./scsi_test.ko
> ERROR: scsi_test: inconsistent scsi_mod build-id byte #0 (0x6e [actual]
> vs. 0x83 [debuginfo])"

This usually indicates a mismatch between the kernel build you've
targeted for the instrumentation and the kernel build you're trying to
run it on.

> Here is my current process.
>
> #On the Build system.
> [...]
> #On Target system.
> [...]

If you're already building the whole systemtap on your build machine,
was there some reason not to just copy the staprun/stapio executables
over to the target?


- FChE

Re: Fwd: staprun-only system

[Mike Anderson]

David Smith <dsmith@redhat.com> wrote:
> Assuming you are using an rpm-based system, here's what you should do.
> Install your distro-supplied 'systemtap-runtime' rpm on the target system.
> 
> If you want the latest .git version, you would do something like the
> following on your build system:
> 
> 1.) Pull latest systemtap git tree.
> 2.) Generate tar file from git tree
> 3.) Run rpmbuild on the tar file and spec file.  Something like:
>     # rpmbuild --define "_topdir `pwd`" --define "_sourcedir `pwd`" -bb
> systemtap.spec
> 4.) Copy RPMS/{ARCH}/systemtap-runtime-*.rpm to the target system
> 5.) Install the systemtap-runtime rpm on the target system.
> 6.) Install RPMS/{ARCH}/systemtap-* rpms on the build system (minus the
> debuginfo rpm and probably the testsuite rpm)
> 
> Now you have matching systemtap versions on both your build and target
> systems.
> 

Thanks for the response. While this is an option I was trying to avoid
setting up build environments matching all the test machines that I have.
The test systems may be of more than one distro type and library versions.

> If you aren't using an rpm-based system, I'd basically mimic what the
> spec file does.  Configure and build, then copy the files mentioned in
> the '%files runtime' section over to the target system.

In my previous example my configure and make steps where pretty close to
the what the spec file is doing. Copying the runtime files over to the
target system assumes that they are like enough that the commands will
run.

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

Re: Fwd: staprun-only system

[Mike Anderson]

Frank Ch. Eigler <fche@redhat.com> wrote:
> 
> Jim Keniston <jkenisto@us.ibm.com> forwarded:
> 
> > [...]
> > I tried the ./configure on the target system and it did not generate a
> > Makefile as the elfutils where not installed, I could install them, but
> > since staprun does not need them I wanted to simplify my target system
> > default install by not requiring extra rpms for debug. 
> 
> We could add a configure option to disable building the translator,
> and thus only build staprun/stapio.
> 
> 
> > Since make install did not work. I copied the files by hand. I am still
> > cleaning up a issue where staprun will not load the previously built
> > scsi_test systemtap module.
> > "# staprun ./scsi_test.ko
> > ERROR: scsi_test: inconsistent scsi_mod build-id byte #0 (0x6e [actual]
> > vs. 0x83 [debuginfo])"
> 
> This usually indicates a mismatch between the kernel build you've
> targeted for the instrumentation and the kernel build you're trying to
> run it on.
> 

The current running kernel and the module contain the same vermagic
string. I will spend some time on trying to locate why my build-ids
differ.


> > Here is my current process.
> >
> > #On the Build system.
> > [...]
> > #On Target system.
> > [...]
> 
> If you're already building the whole systemtap on your build machine,
> was there some reason not to just copy the staprun/stapio executables
> over to the target?

My build system is currently a Fedora release 9 system and the target
system is running SUSE Linux Enterprise Server 10. I thought the
difference in distros would lead to issues in just copying the files. I
previous had tried it during some testing and had issues.

I just tried copying the files over from build to target system. The
staprun failed to run. I assume this is because of the glibc 2.8 vs the
2.4 version on the target system.

# staprun ~/scsi_test.ko 
Floating point exception

# ldd stapio
 ./stapio: /lib64/libc.so.6: version `GLIBC_2.8' not found (required by
 ./stapio)

# readelf -s stapio | grep -i GLIBC_2.8
    33: 0000000000000000   139 FUNC    GLOBAL DEFAULT  UND __asprintf_chk@GLIBC_2.8 (6)
    173: 0000000000000000   139 FUNC    GLOBAL DEFAULT  UND __asprintf_chk@@GLIBC_2.8

 	

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

Re: Fwd: staprun-only system

[David Smith]

Mike Anderson wrote:
> David Smith <dsmith@redhat.com> wrote:
>> Assuming you are using an rpm-based system, here's what you should do.
>> Install your distro-supplied 'systemtap-runtime' rpm on the target system.
>>
>> If you want the latest .git version, you would do something like the
>> following on your build system:
>>
>> 1.) Pull latest systemtap git tree.
>> 2.) Generate tar file from git tree
>> 3.) Run rpmbuild on the tar file and spec file.  Something like:
>>     # rpmbuild --define "_topdir `pwd`" --define "_sourcedir `pwd`" -bb
>> systemtap.spec
>> 4.) Copy RPMS/{ARCH}/systemtap-runtime-*.rpm to the target system
>> 5.) Install the systemtap-runtime rpm on the target system.
>> 6.) Install RPMS/{ARCH}/systemtap-* rpms on the build system (minus the
>> debuginfo rpm and probably the testsuite rpm)
>>
>> Now you have matching systemtap versions on both your build and target
>> systems.
>>
> 
> Thanks for the response. While this is an option I was trying to avoid
> setting up build environments matching all the test machines that I have.
> The test systems may be of more than one distro type and library versions.

You might be making more work for yourself in trying to avoid setting up
build environments.  Besides setting up actual machines to match all
your environments, there are other options of mock or virtual machines.

>> If you aren't using an rpm-based system, I'd basically mimic what the
>> spec file does.  Configure and build, then copy the files mentioned in
>> the '%files runtime' section over to the target system.
> 
> In my previous example my configure and make steps where pretty close to
> the what the spec file is doing. Copying the runtime files over to the
> target system assumes that they are like enough that the commands will
> run.

I'm confused.  Are you only building once and then deploying on multiple
target platforms?

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Fwd: staprun-only system

[Mike Anderson]

David Smith <dsmith@redhat.com> wrote:
> Mike Anderson wrote:
> > David Smith <dsmith@redhat.com> wrote:
> >> Assuming you are using an rpm-based system, here's what you should do.
> >> Install your distro-supplied 'systemtap-runtime' rpm on the target system.
> >>
> >> If you want the latest .git version, you would do something like the
> >> following on your build system:
> >>
> >> 1.) Pull latest systemtap git tree.
> >> 2.) Generate tar file from git tree
> >> 3.) Run rpmbuild on the tar file and spec file.  Something like:
> >>     # rpmbuild --define "_topdir `pwd`" --define "_sourcedir `pwd`" -bb
> >> systemtap.spec
> >> 4.) Copy RPMS/{ARCH}/systemtap-runtime-*.rpm to the target system
> >> 5.) Install the systemtap-runtime rpm on the target system.
> >> 6.) Install RPMS/{ARCH}/systemtap-* rpms on the build system (minus the
> >> debuginfo rpm and probably the testsuite rpm)
> >>
> >> Now you have matching systemtap versions on both your build and target
> >> systems.
> >>
> > 
> > Thanks for the response. While this is an option I was trying to avoid
> > setting up build environments matching all the test machines that I have.
> > The test systems may be of more than one distro type and library versions.
> 
> You might be making more work for yourself in trying to avoid setting up
> build environments.  Besides setting up actual machines to match all
> your environments, there are other options of mock or virtual machines.
> 

I have not used mock personally, but will look into this. I currently
setup up virtual machines (recent fan of thincrust), but mainly for
testing kernels using pseudo storage devices.

> >> If you aren't using an rpm-based system, I'd basically mimic what the
> >> spec file does.  Configure and build, then copy the files mentioned in
> >> the '%files runtime' section over to the target system.
> > 
> > In my previous example my configure and make steps where pretty close to
> > the what the spec file is doing. Copying the runtime files over to the
> > target system assumes that they are like enough that the commands will
> > run.
> 
> I'm confused.  Are you only building once and then deploying on multiple
> target platforms?

Currently I will build a kernel for one arch (in this example the arch is
x86_64) then I will rsync the kernel to a number of target systems. As
indicted some target systems will contain different distros. Since I am
only dealing with a kernel this process there is normally no issue with
this method.

My goal with these steps was to build up tapsets by using them during
kernel development work (vs. coding debugfs interfaces or utilizing other
kernel debug / trace infrastructure). Then in future situations where I
may need a system tape script for a customer issue I can possible draw
from this group of tapsets or pull fragments from these tapsets.

I know my usage model may not be the common usage model for systemtap. I
was just trying to see if I can increase the utility of systemtap for me
and maybe obtain info on how best to use it in a kernel development
environment.

Thanks,

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

Re: Fwd: staprun-only system

[David Smith]

Mike Anderson wrote:
> David Smith <dsmith@redhat.com> wrote:
>> Mike Anderson wrote:
>>> David Smith <dsmith@redhat.com> wrote:
>>>> Assuming you are using an rpm-based system, here's what you should do.
>>>> Install your distro-supplied 'systemtap-runtime' rpm on the target system.
>>>>
>>>> If you want the latest .git version, you would do something like the
>>>> following on your build system:
>>>>
>>>> 1.) Pull latest systemtap git tree.
>>>> 2.) Generate tar file from git tree
>>>> 3.) Run rpmbuild on the tar file and spec file.  Something like:
>>>>     # rpmbuild --define "_topdir `pwd`" --define "_sourcedir `pwd`" -bb
>>>> systemtap.spec
>>>> 4.) Copy RPMS/{ARCH}/systemtap-runtime-*.rpm to the target system
>>>> 5.) Install the systemtap-runtime rpm on the target system.
>>>> 6.) Install RPMS/{ARCH}/systemtap-* rpms on the build system (minus the
>>>> debuginfo rpm and probably the testsuite rpm)
>>>>
>>>> Now you have matching systemtap versions on both your build and target
>>>> systems.
>>>>
>>> Thanks for the response. While this is an option I was trying to avoid
>>> setting up build environments matching all the test machines that I have.
>>> The test systems may be of more than one distro type and library versions.
>> You might be making more work for yourself in trying to avoid setting up
>> build environments.  Besides setting up actual machines to match all
>> your environments, there are other options of mock or virtual machines.
>>
> 
> I have not used mock personally, but will look into this. I currently
> setup up virtual machines (recent fan of thincrust), but mainly for
> testing kernels using pseudo storage devices.
> 
>>>> If you aren't using an rpm-based system, I'd basically mimic what the
>>>> spec file does.  Configure and build, then copy the files mentioned in
>>>> the '%files runtime' section over to the target system.
>>> In my previous example my configure and make steps where pretty close to
>>> the what the spec file is doing. Copying the runtime files over to the
>>> target system assumes that they are like enough that the commands will
>>> run.
>> I'm confused.  Are you only building once and then deploying on multiple
>> target platforms?
> 
> Currently I will build a kernel for one arch (in this example the arch is
> x86_64) then I will rsync the kernel to a number of target systems. As
> indicted some target systems will contain different distros. Since I am
> only dealing with a kernel this process there is normally no issue with
> this method.
> 
> My goal with these steps was to build up tapsets by using them during
> kernel development work (vs. coding debugfs interfaces or utilizing other
> kernel debug / trace infrastructure). Then in future situations where I
> may need a system tape script for a customer issue I can possible draw
> from this group of tapsets or pull fragments from these tapsets.
> 
> I know my usage model may not be the common usage model for systemtap. I
> was just trying to see if I can increase the utility of systemtap for me
> and maybe obtain info on how best to use it in a kernel development
> environment.

Systemtap is a user-space exe, that builds and runs kernel modules.
Because it is a user-space exe, you are going to have trouble building
it on one distro and copying it to a different distro (just as you would
any other user-space exe).  If you want to do this, your best bet is to
build it on a system with the lowest version of glibc (and other
libraries) and install any needed compat libraries on the other distros.

Building on F9 and deploying on F10 should probably work.  Building on
F9 and trying to deploy on SLES - you might get lucky and you might not.
 This is certainly not a "supported" or recommended procedure.

Your tapset scripts should work across different distros with no
problems (except for differences in architecture) when run against the
same kernel.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)