Anyone tried SystemTap with the latest RHEL5 Beta refresh

Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Vara Prasad]

One of my colleague tried SystemTap on x86_64 machine got the following 
error
stap -g ioblock.stp
while: registering probe alias ioblock.submit = 
kernel.function("submit_bio")
semantic error: duplicate probe point pattern
while: registering probe alias ioblock.end = kernel.function("bio_endio")
semantic error: duplicate probe point pattern
Pass 2: analysis failed.  Try again with more '-v' (verbose) options.


I dont have the setup to debug his problem, was wondering if anyone has 
experienced these problems.
He is using stock RHEL 5 setup. I thought we now bundle everything 
needed in RHEL5, correct.
Does he needs to download any additional packages?

Thanks for your help,
Vara Prasad

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Mike Mason]

Looks like he's trying to run the ioblock tapset file directly?  That won't work.  Stap ends up loading ioblock.stp twice which results in the duplicate probe errors. He needs to write a script that uses the ioblock tapset probes.

- Mike

Vara Prasad wrote:
> One of my colleague tried SystemTap on x86_64 machine got the following 
> error
> stap -g ioblock.stp
> while: registering probe alias ioblock.submit = 
> kernel.function("submit_bio")
> semantic error: duplicate probe point pattern
> while: registering probe alias ioblock.end = kernel.function("bio_endio")
> semantic error: duplicate probe point pattern
> Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
> 
> 
> I dont have the setup to debug his problem, was wondering if anyone has 
> experienced these problems.
> He is using stock RHEL 5 setup. I thought we now bundle everything 
> needed in RHEL5, correct.
> Does he needs to download any additional packages?
> 
> Thanks for your help,
> Vara Prasad
> 

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

Vara Prasad <prasadav@us.ibm.com> writes:

> One of my colleague tried SystemTap on x86_64 machine got the
> following error
> stap -g ioblock.stp
> while: registering probe alias ioblock.submit =
> kernel.function("submit_bio")
> semantic error: duplicate probe point pattern
> while: registering probe alias ioblock.end = kernel.function("bio_endio")
> [...]

This tends to happen if for some reason duplicate tapsets are found by
systemtap (as might happen if two different versions get installed on
top of each other), or if something is wrong with the x86-64 tapset
code, or if one tries to run a tapset script directly (which would
duplicate the copy later found during tapset search).

My guess is that the latter is the case.  One should not in general
run a file under tapset/.

- FChE

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

Hi -

On Thu, Nov 02, 2006 at 03:53:06PM -0800, Badari Pulavarty wrote:
> [...]
> Okay, I installed the debug-info rpm packages and now running into 
> new issue ..

Great.

> # stap -vv xx.stp
> Created temporary directory "/tmp/stapDnsZZn"
> Searched '/usr/share/systemtap/tapset/2.6.18-1.2714.el5/x86_64/*.stp',
> match count 0
> [...]
> /tmp/stapDnsZZn/.tmp_stap_9870.o /tmp/stapDnsZZn/stap_9870.c
> In file included from /usr/share/systemtap/runtime/print.c:14,
>                  from /usr/share/systemtap/runtime/runtime.h:65,
>                  from /tmp/stapDnsZZn/stap_9870.c:31:
> include/linux/config.h:6:2: error: #warning Including config.h is
> deprecated.
> [...]

Your copy of systemtap must come from somewhere other than the RHEL5
snapshot whose kernel you are testing.  This problem has been fixed
for some time.  (Perhaps stap -vv should imply -V (version string
printing)).

- FChE

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Boreham]

>
>
>
>
>Okay, I installed the debug-info rpm packages and now running into 
>new issue ..
>  
>
Did you need kernel-devel too ?

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Badari Pulavarty]

On Thu, 2006-11-02 at 14:09 -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Nov 02, 2006 at 10:51:20AM -0800, Badari Pulavarty wrote:
> > Okay, just to make some progress and verify systemtap ..
> > 
> > Where can I get -debuginfo package from ? I looked at all
> > 6 RHEL5 Beta CD images and didn't find it. What am I missing ?
> 
> I believe that the debuginfo packages are at this time not placed on
> CDs at all.  However, they are online in the same place where the .iso
> files were.


Okay, I installed the debug-info rpm packages and now running into 
new issue ..

# stap -vv xx.stp
Created temporary directory "/tmp/stapDnsZZn"
Searched '/usr/share/systemtap/tapset/2.6.18-1.2714.el5/x86_64/*.stp',
match count 0
Searched '/usr/share/systemtap/tapset/2.6.18-1.2714.el5/*.stp', match
count 0
Searched '/usr/share/systemtap/tapset/2.6.18/x86_64/*.stp', match count
0
Searched '/usr/share/systemtap/tapset/2.6.18/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/x86_64/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/x86_64/*.stp', match count 1
Searched '/usr/share/systemtap/tapset/*.stp', match count 26
Searched
'/usr/share/systemtap/tapset/LKET/2.6.18-1.2714.el5/x86_64/*.stp', match
count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.18-1.2714.el5/*.stp',
match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.18/x86_64/*.stp', match
count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6.18/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6/x86_64/*.stp', match
count 0
Searched '/usr/share/systemtap/tapset/LKET/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/x86_64/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/LKET/*.stp', match count 13
Pass 1: parsed user script and 40 library script(s) in
120usr/10sys/136real ms.
probe submit_bio@block/ll_rw_blk.c:3112 pc=0xffffffff80035c09
probe bio_endio@fs/bio.c:1032 pc=0xffffffff8003cd09
Pass 2: analyzed script: 3 probe(s), 10 function(s), 0 global(s) in
200usr/50sys/243real ms.
probe_734 locks nothing
probe_736 locks nothing
probe_738 locks nothing
Running grep " [tT] " /proc/kallsyms | sort -k 1,16 -s -
o /tmp/stapDnsZZn/symbols.sorted
Pass 3: translated to C into "/tmp/stapDnsZZn/stap_9870.c" in
80usr/70sys/161real ms.
Running make -C "/lib/modules/2.6.18-1.2714.el5/build"
M="/tmp/stapDnsZZn" modules V=1
make: Entering directory `/usr/src/kernels/2.6.18-1.2714.el5-x86_64'
test -e include/linux/autoconf.h -a -e include/config/auto.conf || (
\
        echo;
\
        echo "  ERROR: Kernel configuration is invalid.";
\
        echo "         include/linux/autoconf.h or
include/config/auto.conf are missing.";      \
        echo "         Run 'make oldconfig && make prepare' on kernel
src to fix it.";  \
        echo;
\
        /bin/false)
mkdir -p /tmp/stapDnsZZn/.tmp_versions
rm -f /tmp/stapDnsZZn/.tmp_versions/*
make -f scripts/Makefile.build obj=/tmp/stapDnsZZn
  gcc -Wp,-MD,/tmp/stapDnsZZn/.stap_9870.o.d  -nostdinc -
isystem /usr/lib/gcc/x86_64-redhat-linux/4.1.1/include -D__KERNEL__ -
Iinclude  -include include/linux/autoconf.h  -Wall -Wundef -Wstrict-
prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-
prototypes -Wundef -Werror-implicit-function-declaration -Os  -
mtune=generic -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-
blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -
mno-3dnow -fno-omit-frame-pointer -fno-optimize-sibling-calls -
fasynchronous-unwind-tables -g  -fno-stack-protector -Wdeclaration-
after-statement -Wno-pointer-sign -freorder-blocks -Wno-unused -Werror -
I"/usr/share/systemtap/runtime"   -DMODULE -D"KBUILD_STR(s)=#s" -
D"KBUILD_BASENAME=KBUILD_STR(stap_9870)"  -D"KBUILD_MODNAME=KBUILD_STR
(stap_9870)" -c -
o /tmp/stapDnsZZn/.tmp_stap_9870.o /tmp/stapDnsZZn/stap_9870.c
In file included from /usr/share/systemtap/runtime/print.c:14,
                 from /usr/share/systemtap/runtime/runtime.h:65,
                 from /tmp/stapDnsZZn/stap_9870.c:31:
include/linux/config.h:6:2: error: #warning Including config.h is
deprecated.
make[1]: *** [/tmp/stapDnsZZn/stap_9870.o] Error 1
make: *** [_module_/tmp/stapDnsZZn] Error 2
make: Leaving directory `/usr/src/kernels/2.6.18-1.2714.el5-x86_64'
Pass 4: compiled C into "stap_9870.ko" in 1020usr/360sys/1386real ms.
Pass 4: compilation failed.  Try again with more '-v' (verbose) options.
Running rm -rf /tmp/stapDnsZZn

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Boreham]

Frank Ch. Eigler wrote:

>
>I believe that the debuginfo packages are at this time not placed on
>CDs at all.  However, they are online in the same place where the .iso
>files were.
>
>  
>
The kernel-debuginfo RPM wouldn't fix on a CD.

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

Hi -

On Thu, Nov 02, 2006 at 10:51:20AM -0800, Badari Pulavarty wrote:
> Okay, just to make some progress and verify systemtap ..
> 
> Where can I get -debuginfo package from ? I looked at all
> 6 RHEL5 Beta CD images and didn't find it. What am I missing ?

I believe that the debuginfo packages are at this time not placed on
CDs at all.  However, they are online in the same place where the .iso
files were.


- FChE

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Badari Pulavarty]

Okay, just to make some progress and verify systemtap ..

Where can I get -debuginfo package from ? I looked at all
6 RHEL5 Beta CD images and didn't find it. What am I missing ?

Thanks,
Badari

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[William Cohen]

Vara Prasad wrote:
> Frank Ch. Eigler wrote:
> 
>> Hi -
>>
>> On Mon, Oct 30, 2006 at 10:27:47AM -0800, Vara Prasad wrote:
>>
>>  
>>
>>> [...]  Badari is playing the role of a customer. His point of view
>>> is, i have stock RHEL 5 setup, i have not made any changes to the
>>> setup and trying to use systemtap and i expect it to have all the
>>> pieces needed [...]
>>>   
>>
>>
>> Unfortunately, other customers prefer that systemtap's loose
>> dependencies (particularly, the kernel-debuginfo) not be installed by
>> default, even if they choose an "install everything" option at the
>> anaconda screens.  Can you think of a way of satisfying both groups?
>>
>> - FChE
>>  
>>
> O.k, before i think of a way to satisfy both the needs i need to 
> understand the objection of the other group.
> The objection of other customers to not to install debuginfo package is 
> it because of wasted disk space due to large size of debuginfo package 
> or time to install or something else.
> I am assuming kernel debuginfo is considered a dependency for SystemTap, 
> am i right.
> What is the point of installing a package that doesn't work due to lack 
> of dependencies? In other words are we not breaking the semantics or 
> meaning of full package install if we don't install required 
> dependencies when the customer chooses a package to install.
> 
> bye,
> Vara Prasad
> 

There are cases where one could be using systemtap on a locally-built kernel. 
For this case the kernel-debuginfo dependency does not make sense. This is a 
common use case for kernel engineers.

There are may be more than one kernel installed on the machine. Forcing a 
kernel-debuginfo to be installed doesn't guarantee that it matches with the 
currently running kernel. RPM dependencies only make sense if there is only one 
possible package installed on the system. Kernel rpms violate that rule. For 
Fedora 6 (and RHEL5) there are multiple kernel-debuginfo. Having dependencies on 
all the possible debuginfo because the might be used isn't going to work.

The translator error reporting about the lack of debug information could be 
clearer about what is going on. However, putting in a questional RPM dependency 
in the systemtap package is not going to solve the problem.

-Will

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Badari Pulavarty]

On Wed, 2006-11-01 at 12:46 -0800, Keshavamurthy, Anil S wrote:

> Some of the difficulties that I see people face are
> 1) Debuginfo RPM is not part of the standard CD's or the partner released ISO's. If it is not part of the 
> standard CD, people are not going to google for debuginfo and install it. Also in the lab's where
> people debug the RHEL releases, accessing outside the corporate firewall is restricted and hence
> people can not google for the debuginfo rpm even if they want to.
>
> 2) When the kernel is updated via RHN, there is no automated way to update the 
> corresponding debuginfo RPM's as this is not available via RHN.
> 
> 3) Also for people who are willing to spare 300MB for installing debuginfo rpm we need the
> installer support which will install this while installing the OS.

Its a good summary of the discussion/issues.

4) Even if sysadmin/service people wants to install bunch of packages
that are *not* part of the default install - they may need to jump
through hoops (manager approvals, special permissions etc..)

IMHO - Enterprise customers can live with the additional space
requirements, if we can provide out-of-box experience. I am sure
my default RH install has 50% more than what I need for regular
use. Yes. There will be always people complaining about how bloated
are the distro images - well, we do it for a reason (most of the
times :)

Bottomline is, can we use systemtap as its intended to use without
these extra packages/info ? If not, we need to take a hard look
at it. But if its serving 90% of its intended use - may be we
are okay.

To be honest, I don't use/test default systemtap support which
comes out by the distro releases -- just because, i have to find
out where all these bits and pieces are. Instead, I build my own
kernels, get latest CVS snapshots of systemtap to make it work. 
I have this luxury since I use it for my development :)

Thanks,
Badari 

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Keshavamurthy, Anil S]

On Wed, Nov 01, 2006 at 11:54:04AM -0800, Vara Prasad wrote:
> Frank Ch. Eigler wrote:
> 
> >Hi -
> >
> > 
> >
> >>fche wrote:
> >>[...]
> >>   
> >>
> >>>Unfortunately, other customers prefer that systemtap's loose
> >>>dependencies (particularly, the kernel-debuginfo) not be installed by
> >>>default, even if they choose an "install everything" option at the
> >>>anaconda screens.  [..]
> >>>     
> >>>
> >
> >varap wrote:
> >
> > 
> >
> >>O.k, before i think of a way to satisfy both the needs i need to 
> >>understand the objection of the other group.
> >>   
> >>
> >
> > 
> >
> >>The objection of other customers to not to install debuginfo package is 
> >>it because of wasted disk space due to large size of debuginfo package 
> >>or time to install or something else.
> >>   
> >>
> >
> >All those, plus the manual way in which RHEL debuginfo packages are at
> >present published.
> > 
> >
> Well if my memory serves well Elena's plan for RHEL 5 is not to go 
> through this manual step. She tried her best to eliminate for RHEL 4 but 
> it didn't work due to some reasons of packaging i am not aware but the 
> assurance she had from the packaging team was in RHEL 5 it will be 
> available as part of the install media. If you want I can reopen the 
> issue tracker for this.
> 
> I am fine to install only kernel debuginfo package as a compromise and 
> module debuginfo package doesn't need to be installed although we should 
> also make it available on the install media.
> 
> > 
> >
> >>I am assuming kernel debuginfo is considered a dependency for
> >>SystemTap, am I right?
> >>   
> >>
> >
> >It is a loose dependency, in that the RPM does not list it as such
> >(any more), and that it is theoretically possible to run some scripts
> >without kprobes, and thus without dwarf data.
> > 
> >
> How many of our example scripts and tapset probe points i can use 
> without debug info package, my guess not much, so how a customer can 
> even experiment if they can't use the examples system comes with.
> 
> > 
> >
> >>What is the point of installing a package that doesn't work due to
> >>lack of dependencies?
> >>   
> >>
> >
> >Among other reasons, it simplifies the task of the person overseeing
> >an initial installation.  
> >
> Well if a person wants a simplified install of limited packages you can 
> go through custom install and choose the packages you like, that is what 
> normally i do if i don't care for the default install.
> 
> >They may not want systemtap as such, merely
> >use the "everything that is on this CD" option as a time-saver.
> > 
> >
> 
> As mentioned above for RHEL 5 we need to make debuginfo package readily 
> available as part of the install media, if we have any chance of getting 
> feedback from real end users (not just developers). I personally think 
> the default install should have SystemTap package and required 
> dependencies installed but for those who don't want that we should let 
> them opt out.  Competition is already installing the full package as 
> part of the default install, if we make it difficult to use we will loose.
> 
> I agree for FC installation we could avoid installing debuginfo package 
> as the space for those small setups is significant.

If we want people to use systemtap, then we need to make it easy for them
and get it running right out of the box, i.e right after the system is installed.
If we want systemtap to be successful, we need to listen to the users of Systemap.
I am sure no body disagrees with this point which is being debated over here.

Some of the difficulties that I see people face are
1) Debuginfo RPM is not part of the standard CD's or the partner released ISO's. If it is not part of the 
standard CD, people are not going to google for debuginfo and install it. Also in the lab's where
people debug the RHEL releases, accessing outside the corporate firewall is restricted and hence
people can not google for the debuginfo rpm even if they want to.

2) When the kernel is updated via RHN, there is no automated way to update the 
corresponding debuginfo RPM's as this is not available via RHN.

3) Also for people who are willing to spare 300MB for installing debuginfo rpm we need the
installer support which will install this while installing the OS.

-Anil

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Michael K. Dolan Jr.]

Hate plain-text only... another try.

This note isn't specific to David's response, I'm just responding to 
this entire thread with my view... and from my what I see on other 
platforms...

I suggest ignoring the developer user set. The value of SystemTap is 
that it can be used in a production environment to observe a running 
kernel and diagnose performance issues or areas for opportunity to 
increase performance. Yes, there are dev values but ignore them for the 
moment - too often these discussion turn to a developer PoV - but think 
of an admin situation. For every 1 development server/wkstn, there are 
probably 1,000 production servers out there with admins watching them.

It's absurd to think that an admin who's watching a troubled system will 
go and install a debuginfo package for the server he's (or she's - to be 
balanced) watching. Not to mention that installing packages like 
debuginfo may require approvals beyond just that admins' manager. Maybe 
in an SMB shop with little formal process it's possible but certainly 
not in enterprise accounts - heck most don't give their admins root 
privileges.

It's also absurd to assume he/she will cross compile these scripts 
locally against a target system. First, there's always the possibility 
that the cross-compile won't match exactly - how do you probe drivers on 
a server but not on your workstation? Second, for admins this would be a 
process of testing different scripts, testing different probe points - 
an iterative problem analysis situation. Do you really think they're 
going to compile one script to check A, copy the compiled one to the 
server, run it on the server, go back to their workstation, 
re-cross-compile a script to check for B, and copy back to server, 
repeat for C,D,E,F... It's maddening.

What I think SystemTap needs to be is a defacto tool that is running out 
of the box with RHEL5.0 (yes, I said 5.0 b/c that's when every admin 
will get trained on "what's new" in RHEL5). I'm not sure what all the 
issues are, but it needs to be there and running for every RHEL5.0 user 
to experience on day 1 without having to install dependencies they don't 
know about. What happens if stap it is in RHEL5 without debuginfo and 
the user can't compile their scripts - the manual says it's there, but I 
have to install more to get what I really need to use? Will they 
automagically know to install debuginfo? Will they easily find the right 
one to match their kernel?

I'll give an example - have you used htop? It's cool, in some ways much 
better than top, but how many people are using it compared to top? 
Everyone knows every Linux server has top and by default when you open a 
term on a server you know top is there. That's what stap needs to be.

Just my thoughts - if you were wondering... ;)

miked

David Smith wrote:
> David Wilder wrote:
>> Frank Ch. Eigler wrote:
>>
>>> "Ken Robson" <ken@odtv.com> writes:
>>>
>>>  
>>>
>>>> [...]  To me it is valid to install minus the debuginfo files on
>>>> almost all Production hosts.  I am experimenting with developing my
>>>> scripts off box with my cache directory set to an exported read-only
>>>> NFS share which is then mounted as the module cache directory on my
>>>> Production boxes [...]
>>>>   
>>>
>>> More than that - on such production boxes, you will need to install
>>> only the "staprun" (formerly "stapd") binary, now separated into a
>>> systemtap-runtime RPM.  For the moment though please be careful with
>>> building probes for mismatching machines: the module address tables
>>> are not yet fully adaptive.
>>>
>>> - FChE
>>>  
>>>
>> The cached debuginfo is a really cool concept.
>
> You got a bit confused here.  The debuginfo isn't cached, the 
> systemtap compiled modules are cached.
>
>> But it wont solve the problem of simplifying the use of systemtap for 
>> the customers. From a support standpoint if a customer system is 
>> missing a debug tool (or some dependent component) the tool may as 
>> well not exist! If it comes down to fix the debug tool or find 
>> another approach to solve the customers problem the later will 
>> generally win. To make stap successful we need to get people using it 
>> and providing feedback, let's make it as easy as possible to use. All 
>> dependencies must be installed when selecting a product for install.
>
> In general, I certainly agree with you that all dependencies must be 
> installed.
>
> However, systemtap (and any other program that would like to use 
> debuginfo) is a special case.  From my understanding, there is a 
> policy (perhaps unwritten) that no rpm can require a debuginfo rpm.  
> Plus, even if we did require the debuginfo rpm, it still wouldn't get 
> installed automatically.  For FC[56], the debuginfo yum repositories 
> are disabled by default.  For RHEL[34], the debuginfo RPMs aren't 
> available from a RHN channel, they have to be downloaded separately 
> (from my vague understanding which could be wrong).  In addition, 
> debuginfo RPMs are not present on RHEL/FC install media.  So, from a 
> current logistical point of view, if the systemtap RPM required the 
> kernel debuginfo RPM, systemtap itself could never be installed 
> because of missing dependencies that could never be met.
>
> Currently, using systemtap isn't much different than using gdb.  Let's 
> assume that /bin/ls keeps crashing on you for some strange reason that 
> you'd like to debug.  You are going to need to download/install the 
> coreutils debuginfo RPM, then use gdb to debug your problem.
>

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Michael K. Dolan Jr.]

This note isn't specific to David's response, I'm just responding to 
this entire thread with my view... and from my what I see on other 
platforms...

I suggest ignoring the developer user set. The value of SystemTap is 
that it can be used in a production environment to observe a running 
kernel and diagnose performance issues or areas for opportunity to 
increase performance. Yes, there are dev values but ignore them for the 
moment - too often these discussion turn to a developer PoV - but think 
of an admin situation. For every 1 development server/wkstn, there are 
probably 1,000 production servers out there with admins watching them.

It's absurd to think that an admin who's watching a troubled system will 
go and install a debuginfo package for the server he's (or she's - to be 
balanced) watching. Not to mention that installing packages like 
debuginfo may require approvals beyond just that admins' manager. Maybe 
in an SMB shop with little formal process it's possible but certainly 
not in enterprise accounts - heck most don't give their admins root 
privileges.

It's also absurd to assume he/she will cross compile these scripts 
locally against a target system. First, there's always the possibility 
that the cross-compile won't match exactly - how do you probe drivers on 
a server but not on your workstation? Second, for admins this would be a 
process of testing different scripts, testing different probe points - 
an iterative problem analysis situation. Do you really think they're 
going to compile one script to check A, copy the compiled one to the 
server, run it on the server, go back to their workstation, 
re-cross-compile a script to check for B, and copy back to server, 
repeat for C,D,E,F... It's maddening.

What I think SystemTap needs to be is a defacto tool that is running out 
of the box with RHEL5.0 (yes, I said 5.0 b/c that's when every admin 
will get trained on "what's new" in RHEL5). I'm not sure what all the 
issues are, but it needs to be there and running for every RHEL5.0 user 
to experience on day 1 without having to install dependencies they don't 
know about. What happens if stap it is in RHEL5 without debuginfo and 
the user can't compile their scripts - the manual says it's there, but I 
have to install more to get what I really need to use? Will they 
automagically know to install debuginfo? Will they easily find the right 
one to match their kernel?

I'll give an example - have you used htop? It's cool, in some ways much 
better than top, but how many people are using it compared to top? 
Everyone knows every Linux server has top and by default when you open a 
term on a server you know top is there. That's what stap needs to be.

Just my thoughts - if you were wondering... ;)

miked



David Smith wrote:
> David Wilder wrote:
>> Frank Ch. Eigler wrote:
>>
>>> "Ken Robson" <ken@odtv.com> writes:
>>>
>>>  
>>>
>>>> [...]  To me it is valid to install minus the debuginfo files on
>>>> almost all Production hosts.  I am experimenting with developing my
>>>> scripts off box with my cache directory set to an exported read-only
>>>> NFS share which is then mounted as the module cache directory on my
>>>> Production boxes [...]
>>>>   
>>>
>>> More than that - on such production boxes, you will need to install
>>> only the "staprun" (formerly "stapd") binary, now separated into a
>>> systemtap-runtime RPM.  For the moment though please be careful with
>>> building probes for mismatching machines: the module address tables
>>> are not yet fully adaptive.
>>>
>>> - FChE
>>>  
>>>
>> The cached debuginfo is a really cool concept.
>
> You got a bit confused here.  The debuginfo isn't cached, the 
> systemtap compiled modules are cached.
>
>> But it wont solve the problem of simplifying the use of systemtap for 
>> the customers. From a support standpoint if a customer system is 
>> missing a debug tool (or some dependent component) the tool may as 
>> well not exist! If it comes down to fix the debug tool or find 
>> another approach to solve the customers problem the later will 
>> generally win. To make stap successful we need to get people using it 
>> and providing feedback, let's make it as easy as possible to use. All 
>> dependencies must be installed when selecting a product for install.
>
> In general, I certainly agree with you that all dependencies must be 
> installed.
>
> However, systemtap (and any other program that would like to use 
> debuginfo) is a special case.  From my understanding, there is a 
> policy (perhaps unwritten) that no rpm can require a debuginfo rpm.  
> Plus, even if we did require the debuginfo rpm, it still wouldn't get 
> installed automatically.  For FC[56], the debuginfo yum repositories 
> are disabled by default.  For RHEL[34], the debuginfo RPMs aren't 
> available from a RHN channel, they have to be downloaded separately 
> (from my vague understanding which could be wrong).  In addition, 
> debuginfo RPMs are not present on RHEL/FC install media.  So, from a 
> current logistical point of view, if the systemtap RPM required the 
> kernel debuginfo RPM, systemtap itself could never be installed 
> because of missing dependencies that could never be met.
>
> Currently, using systemtap isn't much different than using gdb.  Let's 
> assume that /bin/ls keeps crashing on you for some strange reason that 
> you'd like to debug.  You are going to need to download/install the 
> coreutils debuginfo RPM, then use gdb to debug your problem.
>

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Vara Prasad]

Frank Ch. Eigler wrote:

>Hi -
>
>  
>
>>fche wrote:
>>[...]
>>    
>>
>>>Unfortunately, other customers prefer that systemtap's loose
>>>dependencies (particularly, the kernel-debuginfo) not be installed by
>>>default, even if they choose an "install everything" option at the
>>>anaconda screens.  [..]
>>>      
>>>
>
>varap wrote:
>
>  
>
>>O.k, before i think of a way to satisfy both the needs i need to 
>>understand the objection of the other group.
>>    
>>
>
>  
>
>>The objection of other customers to not to install debuginfo package is 
>>it because of wasted disk space due to large size of debuginfo package 
>>or time to install or something else.
>>    
>>
>
>All those, plus the manual way in which RHEL debuginfo packages are at
>present published.
>  
>
Well if my memory serves well Elena's plan for RHEL 5 is not to go 
through this manual step. She tried her best to eliminate for RHEL 4 but 
it didn't work due to some reasons of packaging i am not aware but the 
assurance she had from the packaging team was in RHEL 5 it will be 
available as part of the install media. If you want I can reopen the 
issue tracker for this.

I am fine to install only kernel debuginfo package as a compromise and 
module debuginfo package doesn't need to be installed although we should 
also make it available on the install media.

>  
>
>>I am assuming kernel debuginfo is considered a dependency for
>>SystemTap, am I right?
>>    
>>
>
>It is a loose dependency, in that the RPM does not list it as such
>(any more), and that it is theoretically possible to run some scripts
>without kprobes, and thus without dwarf data.
>  
>
How many of our example scripts and tapset probe points i can use 
without debug info package, my guess not much, so how a customer can 
even experiment if they can't use the examples system comes with.

>  
>
>>What is the point of installing a package that doesn't work due to
>>lack of dependencies?
>>    
>>
>
>Among other reasons, it simplifies the task of the person overseeing
>an initial installation.  
>
Well if a person wants a simplified install of limited packages you can 
go through custom install and choose the packages you like, that is what 
normally i do if i don't care for the default install.

>They may not want systemtap as such, merely
>use the "everything that is on this CD" option as a time-saver.
>  
>

As mentioned above for RHEL 5 we need to make debuginfo package readily 
available as part of the install media, if we have any chance of getting 
feedback from real end users (not just developers). I personally think 
the default install should have SystemTap package and required 
dependencies installed but for those who don't want that we should let 
them opt out.  Competition is already installing the full package as 
part of the default install, if we make it difficult to use we will loose.

I agree for FC installation we could avoid installing debuginfo package 
as the space for those small setups is significant.

>- FChE
>  
>

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Vara Prasad]

David Smith wrote:

> David Wilder wrote:
>
>> David Smith wrote:
>>
>>> David Wilder wrote:
>>>
>>>> Frank Ch. Eigler wrote:
>>>>
>>>>> "Ken Robson" <ken@odtv.com> writes:
>>>>>
>>>>>  
>>>>>
>>>>> [snip]
>>>>
>>>> But it wont solve the problem of simplifying the use of systemtap 
>>>> for the customers. From a support standpoint if a customer system 
>>>> is missing a debug tool (or some dependent component) the tool may 
>>>> as well not exist! If it comes down to fix the debug tool or find 
>>>> another approach to solve the customers problem the later will 
>>>> generally win. To make stap successful we need to get people using 
>>>> it and providing feedback, let's make it as easy as possible to 
>>>> use. All dependencies must be installed when selecting a product 
>>>> for install.
>>>
>>>
>>>
>>> In general, I certainly agree with you that all dependencies must be 
>>> installed.
>>>
>>> However, systemtap (and any other program that would like to use 
>>> debuginfo) is a special case.  From my understanding, there is a 
>>> policy (perhaps unwritten) that no rpm can require a debuginfo rpm.  
>>> Plus, even if we did require the debuginfo rpm, it still wouldn't 
>>> get installed automatically.  For FC[56], the debuginfo yum 
>>> repositories are disabled by default.  For RHEL[34], the debuginfo 
>>> RPMs aren't available from a RHN channel, they have to be downloaded 
>>> separately (from my vague understanding which could be wrong).  In 
>>> addition, debuginfo RPMs are not present on RHEL/FC install media.  
>>> So, from a current logistical point of view, if the systemtap RPM 
>>> required the kernel debuginfo RPM, systemtap itself could never be 
>>> installed because of missing dependencies that could never be met.
>>>
>>> Currently, using systemtap isn't much different than using gdb.  
>>> Let's assume that /bin/ls keeps crashing on you for some strange 
>>> reason that you'd like to debug.  You are going to need to 
>>> download/install the coreutils debuginfo RPM, then use gdb to debug 
>>> your problem.
>>>
>> Yea but gdb has other uses then debugging coreutils.  SystemTap is 
>> only used to instrument the kernel.
>
>
> gdb is used to debug user apps.  If you compiled the app yourself, 
> you've got the debuginfo (assuming you compiled with '-g').  If you 
> are debugging a vendor compiled app, you'll need to download the 
> associated debuginfo RPM.

gdb is a debugger and SystemTap is just not a debugger and it is much 
more than that.

>
> systemtap is used to instrument the kernel.  

No, it is much more. SystemTap is used instrument entire software stack 
all the way from interpreted applications to device driver interface 
level. Agree we don't have support for user space apps right now but it 
is coming soon so we need to design/package with end goal in mind.

The second major difference is the audience of SystemTap is just not 
developers it is useful all the way from end user (sys admins), to 
service team, app. developers and kernel developers.

> If you compiled your own kernel, you've got the debuginfo (assuming 
> you compiled with '-g').  If you are instrumenting a vendor compiled 
> kernel, you'll need to download the associated debuginfo RPM.
>
Developers have the luxury to debug in their own environment where they 
can reboot at their will most other class of users of system don't have 
such luxury and i think we need to make life easy for others more than 
developers.

If i am a sysadmin and i want to use some scripts that i have developed 
or got it through some trusted source i should have the system fully 
ready to execute my scripts, i don't have to go through hoops to run my 
script.

If i am a service person when i am working under tight deadlines to fix 
the problem if the tool i would like to use is not there it is not 
productive to go through setup of the tool instead other methods will be 
used, which means SystemTap is not used to its potential.

bye,
Vara Prasad

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

Hi -

On Wed, Nov 01, 2006 at 10:59:14AM -0800, David Wilder wrote:
> >>[...]  ure this will increase the size of the kernel package by
> >>about 40Meg, but the full debug info package is closer to 150Meg.
>
> >Don't forget about debug information for the modules.  For a random
> >FC5 kernel, that adds another 300MB (uncompressed).

> Frank,  do you agree that stap is still useful without the debug info 
> modules?

It is more useful than with no debuginfo at all, and less useful than
with full debuginfo.  Whether that intermediate state of utility is
good enough to justify changing the kernel packaging is unknown.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Wilder]

David Smith wrote:

> David Wilder wrote:
>
>> David Smith wrote:
>>
>>> David Wilder wrote:
>>>
>>>> Frank Ch. Eigler wrote:
>>>>
>>>>> "Ken Robson" <ken@odtv.com> writes:
>>>>>
>>>>>  
>>>>>
>>>>>> [...]  To me it is valid to install minus the debuginfo files on
>>>>>> almost all Production hosts.  I am experimenting with developing my
>>>>>> scripts off box with my cache directory set to an exported read-only
>>>>>> NFS share which is then mounted as the module cache directory on my
>>>>>> Production boxes [...]
>>>>>>   
>>>>>
>>>>>
>>>>>
>>>>> More than that - on such production boxes, you will need to install
>>>>> only the "staprun" (formerly "stapd") binary, now separated into a
>>>>> systemtap-runtime RPM.  For the moment though please be careful with
>>>>> building probes for mismatching machines: the module address tables
>>>>> are not yet fully adaptive.
>>>>>
>>>>> - FChE
>>>>>  
>>>>>
>>>> The cached debuginfo is a really cool concept.
>>>
>>>
>>>
>>> You got a bit confused here.  The debuginfo isn't cached, the 
>>> systemtap compiled modules are cached.
>>
>>
>>
>> Thanks for the clarification.
>>
>>>
>>>> But it wont solve the problem of simplifying the use of systemtap 
>>>> for the customers. From a support standpoint if a customer system 
>>>> is missing a debug tool (or some dependent component) the tool may 
>>>> as well not exist! If it comes down to fix the debug tool or find 
>>>> another approach to solve the customers problem the later will 
>>>> generally win. To make stap successful we need to get people using 
>>>> it and providing feedback, let's make it as easy as possible to 
>>>> use. All dependencies must be installed when selecting a product 
>>>> for install.
>>>
>>>
>>>
>>> In general, I certainly agree with you that all dependencies must be 
>>> installed.
>>>
>>> However, systemtap (and any other program that would like to use 
>>> debuginfo) is a special case.  From my understanding, there is a 
>>> policy (perhaps unwritten) that no rpm can require a debuginfo rpm.  
>>> Plus, even if we did require the debuginfo rpm, it still wouldn't 
>>> get installed automatically.  For FC[56], the debuginfo yum 
>>> repositories are disabled by default.  For RHEL[34], the debuginfo 
>>> RPMs aren't available from a RHN channel, they have to be downloaded 
>>> separately (from my vague understanding which could be wrong).  In 
>>> addition, debuginfo RPMs are not present on RHEL/FC install media.  
>>> So, from a current logistical point of view, if the systemtap RPM 
>>> required the kernel debuginfo RPM, systemtap itself could never be 
>>> installed because of missing dependencies that could never be met.
>>>
>>> Currently, using systemtap isn't much different than using gdb.  
>>> Let's assume that /bin/ls keeps crashing on you for some strange 
>>> reason that you'd like to debug.  You are going to need to 
>>> download/install the coreutils debuginfo RPM, then use gdb to debug 
>>> your problem.
>>>
>> Yea but gdb has other uses then debugging coreutils.  SystemTap is 
>> only used to instrument the kernel.
>
>
> gdb is used to debug user apps.  If you compiled the app yourself, 
> you've got the debuginfo (assuming you compiled with '-g').  If you 
> are debugging a vendor compiled app, you'll need to download the 
> associated debuginfo RPM.
>
> systemtap is used to instrument the kernel.  If you compiled your own 
> kernel, you've got the debuginfo (assuming you compiled with '-g').  
> If you are instrumenting a vendor compiled kernel, you'll need to 
> download the associated debuginfo RPM.
>
Yes very true.  But customers typically don't build their own kernels if 
they want to stay supportable.  I see the issue is: how do we make stap 
useful to the support engineer?  If the tool is not working on the 
customers system then it will not be used.  So how do we insure that 
stap is working out of the box for all systems?

-- 
David Wilder
IBM Linux Technology Center
Beaverton, Oregon, USA 
dwilder@us.ibm.com
(503)578-3789

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Smith]

David Wilder wrote:
> David Smith wrote:
> 
>> David Wilder wrote:
>>
>>> Frank Ch. Eigler wrote:
>>>
>>>> "Ken Robson" <ken@odtv.com> writes:
>>>>
>>>>  
>>>>
>>>>> [...]  To me it is valid to install minus the debuginfo files on
>>>>> almost all Production hosts.  I am experimenting with developing my
>>>>> scripts off box with my cache directory set to an exported read-only
>>>>> NFS share which is then mounted as the module cache directory on my
>>>>> Production boxes [...]
>>>>>   
>>>>
>>>>
>>>> More than that - on such production boxes, you will need to install
>>>> only the "staprun" (formerly "stapd") binary, now separated into a
>>>> systemtap-runtime RPM.  For the moment though please be careful with
>>>> building probes for mismatching machines: the module address tables
>>>> are not yet fully adaptive.
>>>>
>>>> - FChE
>>>>  
>>>>
>>> The cached debuginfo is a really cool concept.
>>
>>
>> You got a bit confused here.  The debuginfo isn't cached, the 
>> systemtap compiled modules are cached.
> 
> 
> Thanks for the clarification.
> 
>>
>>> But it wont solve the problem of simplifying the use of systemtap for 
>>> the customers. From a support standpoint if a customer system is 
>>> missing a debug tool (or some dependent component) the tool may as 
>>> well not exist! If it comes down to fix the debug tool or find 
>>> another approach to solve the customers problem the later will 
>>> generally win. To make stap successful we need to get people using it 
>>> and providing feedback, let's make it as easy as possible to use. All 
>>> dependencies must be installed when selecting a product for install.
>>
>>
>> In general, I certainly agree with you that all dependencies must be 
>> installed.
>>
>> However, systemtap (and any other program that would like to use 
>> debuginfo) is a special case.  From my understanding, there is a 
>> policy (perhaps unwritten) that no rpm can require a debuginfo rpm.  
>> Plus, even if we did require the debuginfo rpm, it still wouldn't get 
>> installed automatically.  For FC[56], the debuginfo yum repositories 
>> are disabled by default.  For RHEL[34], the debuginfo RPMs aren't 
>> available from a RHN channel, they have to be downloaded separately 
>> (from my vague understanding which could be wrong).  In addition, 
>> debuginfo RPMs are not present on RHEL/FC install media.  So, from a 
>> current logistical point of view, if the systemtap RPM required the 
>> kernel debuginfo RPM, systemtap itself could never be installed 
>> because of missing dependencies that could never be met.
>>
>> Currently, using systemtap isn't much different than using gdb.  Let's 
>> assume that /bin/ls keeps crashing on you for some strange reason that 
>> you'd like to debug.  You are going to need to download/install the 
>> coreutils debuginfo RPM, then use gdb to debug your problem.
>>
> Yea but gdb has other uses then debugging coreutils.  SystemTap is only 
> used to instrument the kernel.

gdb is used to debug user apps.  If you compiled the app yourself, 
you've got the debuginfo (assuming you compiled with '-g').  If you are 
debugging a vendor compiled app, you'll need to download the associated 
debuginfo RPM.

systemtap is used to instrument the kernel.  If you compiled your own 
kernel, you've got the debuginfo (assuming you compiled with '-g').  If 
you are instrumenting a vendor compiled kernel, you'll need to download 
the associated debuginfo RPM.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

RE: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Stone, Joshua I]

On Wednesday, November 01, 2006 10:13 AM, Frank Ch. Eigler wrote:
> Hi -
> 
> On Wed, Nov 01, 2006 at 10:41:56AM -0800, David Wilder wrote:
> 
>> [...]  Why not package the debuginfo "vmlinux" with the kernel rpm?
>> [...]  ure this will increase the size of the kernel package by
>> about 40Meg, but the full debug info package is closer to 150Meg.
> 
> Don't forget about debug information for the modules.  For a random
> FC5 kernel, that adds another 300MB (uncompressed).
> 
> - FChE

I wonder -- could we have some sort of simplified debuginfo shipped for
systemtap?  I'm thinking just an rpm package that contains the debuginfo
necessary to resolve all of the shipped tapsets.  This
'systemtap-tapset-debuginfo' package would be version-matched to the
kernel AND systemtap though, which I suppose might be problematic.

It doesn't matter much what the format of this cache is, but the tapsets
touch such a small subset of the full debuginfo that we could probably
even get away with XML files.

Hopefully the tapsets provide enough framework that you could do some
reasonable debugging from this alone.  Only when you need to go above &
beyond this would you need to install the full kernel-debuginfo.


Josh

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Wilder]

Frank Ch. Eigler wrote:

>Hi -
>
>On Wed, Nov 01, 2006 at 10:41:56AM -0800, David Wilder wrote:
>
>  
>
>>[...]  Why not package the debuginfo "vmlinux" with the kernel rpm?
>>[...]  ure this will increase the size of the kernel package by
>>about 40Meg, but the full debug info package is closer to 150Meg.
>>    
>>
>
>Don't forget about debug information for the modules.  For a random
>FC5 kernel, that adds another 300MB (uncompressed).
>
>- FChE
>  
>
Frank,  do you agree that stap is still useful without the debug info 
modules?

-- 
David Wilder
IBM Linux Technology Center
Beaverton, Oregon, USA 
dwilder@us.ibm.com
(503)578-3789

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

Hi -

On Wed, Nov 01, 2006 at 10:41:56AM -0800, David Wilder wrote:

> [...]  Why not package the debuginfo "vmlinux" with the kernel rpm?
> [...]  ure this will increase the size of the kernel package by
> about 40Meg, but the full debug info package is closer to 150Meg.

Don't forget about debug information for the modules.  For a random
FC5 kernel, that adds another 300MB (uncompressed).

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Wilder]

David Smith wrote:

> David Wilder wrote:
>
>> Frank Ch. Eigler wrote:
>>
>>> "Ken Robson" <ken@odtv.com> writes:
>>>
>>>  
>>>
>>>> [...]  To me it is valid to install minus the debuginfo files on
>>>> almost all Production hosts.  I am experimenting with developing my
>>>> scripts off box with my cache directory set to an exported read-only
>>>> NFS share which is then mounted as the module cache directory on my
>>>> Production boxes [...]
>>>>   
>>>
>>>
>>> More than that - on such production boxes, you will need to install
>>> only the "staprun" (formerly "stapd") binary, now separated into a
>>> systemtap-runtime RPM.  For the moment though please be careful with
>>> building probes for mismatching machines: the module address tables
>>> are not yet fully adaptive.
>>>
>>> - FChE
>>>  
>>>
>> The cached debuginfo is a really cool concept.
>
>
> You got a bit confused here.  The debuginfo isn't cached, the 
> systemtap compiled modules are cached.


Thanks for the clarification.

>
>> But it wont solve the problem of simplifying the use of systemtap for 
>> the customers. From a support standpoint if a customer system is 
>> missing a debug tool (or some dependent component) the tool may as 
>> well not exist! If it comes down to fix the debug tool or find 
>> another approach to solve the customers problem the later will 
>> generally win. To make stap successful we need to get people using it 
>> and providing feedback, let's make it as easy as possible to use. All 
>> dependencies must be installed when selecting a product for install.
>
>
> In general, I certainly agree with you that all dependencies must be 
> installed.
>
> However, systemtap (and any other program that would like to use 
> debuginfo) is a special case.  From my understanding, there is a 
> policy (perhaps unwritten) that no rpm can require a debuginfo rpm.  
> Plus, even if we did require the debuginfo rpm, it still wouldn't get 
> installed automatically.  For FC[56], the debuginfo yum repositories 
> are disabled by default.  For RHEL[34], the debuginfo RPMs aren't 
> available from a RHN channel, they have to be downloaded separately 
> (from my vague understanding which could be wrong).  In addition, 
> debuginfo RPMs are not present on RHEL/FC install media.  So, from a 
> current logistical point of view, if the systemtap RPM required the 
> kernel debuginfo RPM, systemtap itself could never be installed 
> because of missing dependencies that could never be met.
>
> Currently, using systemtap isn't much different than using gdb.  Let's 
> assume that /bin/ls keeps crashing on you for some strange reason that 
> you'd like to debug.  You are going to need to download/install the 
> coreutils debuginfo RPM, then use gdb to debug your problem.
>
Yea but gdb has other uses then debugging coreutils.  SystemTap is only 
used to instrument the kernel.

The more I think about it,  installing the debug info package at install 
time(ignoring the packaging issue for now) solves only half the 
problem.  The other half is,  when the kernel is updated and the 
debuginfo package is not, then stap breaks.

I bet I'll get a lot of objections to this suggestion....   But here 
goes..   Why not package the debuginfo "vmlinux" with the kernel rpm?  
Just the vmlinux, keep all the debug modules in the debuginfo package.   
I  bet that most of the users of  stap are debugging the kernel proper, 
and vmlinux is all that is needed.  Sure this will increase the size of 
the kernel package by about 40Meg, but the full debug info package is 
closer to 150Meg.

This would also solve the same issue for kdump/crash..

-- 
David Wilder
IBM Linux Technology Center
Beaverton, Oregon, USA 
dwilder@us.ibm.com
(503)578-3789

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Smith]

David Wilder wrote:
> Frank Ch. Eigler wrote:
> 
>> "Ken Robson" <ken@odtv.com> writes:
>>
>>  
>>
>>> [...]  To me it is valid to install minus the debuginfo files on
>>> almost all Production hosts.  I am experimenting with developing my
>>> scripts off box with my cache directory set to an exported read-only
>>> NFS share which is then mounted as the module cache directory on my
>>> Production boxes [...]
>>>   
>>
>> More than that - on such production boxes, you will need to install
>> only the "staprun" (formerly "stapd") binary, now separated into a
>> systemtap-runtime RPM.  For the moment though please be careful with
>> building probes for mismatching machines: the module address tables
>> are not yet fully adaptive.
>>
>> - FChE
>>  
>>
> The cached debuginfo is a really cool concept.

You got a bit confused here.  The debuginfo isn't cached, the systemtap 
compiled modules are cached.

> But it wont solve the 
> problem of simplifying the use of systemtap for the customers. From a 
> support standpoint if a customer system is missing a debug tool (or some 
> dependent component) the tool may as well not exist! If it comes down to 
> fix the debug tool or find another approach to solve the customers 
> problem the later will generally win. To make stap successful we need to 
> get people using it and providing feedback, let's make it as easy as 
> possible to use. All dependencies must be installed when selecting a 
> product for install.

In general, I certainly agree with you that all dependencies must be 
installed.

However, systemtap (and any other program that would like to use 
debuginfo) is a special case.  From my understanding, there is a policy 
(perhaps unwritten) that no rpm can require a debuginfo rpm.  Plus, even 
if we did require the debuginfo rpm, it still wouldn't get installed 
automatically.  For FC[56], the debuginfo yum repositories are disabled 
by default.  For RHEL[34], the debuginfo RPMs aren't available from a 
RHN channel, they have to be downloaded separately (from my vague 
understanding which could be wrong).  In addition, debuginfo RPMs are 
not present on RHEL/FC install media.  So, from a current logistical 
point of view, if the systemtap RPM required the kernel debuginfo RPM, 
systemtap itself could never be installed because of missing 
dependencies that could never be met.

Currently, using systemtap isn't much different than using gdb.  Let's 
assume that /bin/ls keeps crashing on you for some strange reason that 
you'd like to debug.  You are going to need to download/install the 
coreutils debuginfo RPM, then use gdb to debug your problem.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[David Wilder]

Frank Ch. Eigler wrote:

>"Ken Robson" <ken@odtv.com> writes:
>
>  
>
>>[...]  To me it is valid to install minus the debuginfo files on
>>almost all Production hosts.  I am experimenting with developing my
>>scripts off box with my cache directory set to an exported read-only
>>NFS share which is then mounted as the module cache directory on my
>>Production boxes [...]
>>    
>>
>
>More than that - on such production boxes, you will need to install
>only the "staprun" (formerly "stapd") binary, now separated into a
>systemtap-runtime RPM.  For the moment though please be careful with
>building probes for mismatching machines: the module address tables
>are not yet fully adaptive.
>
>- FChE
>  
>
The cached debuginfo is a really cool concept. But it wont solve the 
problem of simplifying the use of systemtap for the customers. From a 
support standpoint if a customer system is missing a debug tool (or some 
dependent component) the tool may as well not exist! If it comes down to 
fix the debug tool or find another approach to solve the customers 
problem the later will generally win. To make stap successful we need to 
get people using it and providing feedback, let's make it as easy as 
possible to use. All dependencies must be installed when selecting a 
product for install.

-- 
David Wilder
IBM Linux Technology Center
Beaverton, Oregon, USA 
dwilder@us.ibm.com
(503)578-3789

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

"Ken Robson" <ken@odtv.com> writes:

> [...]  To me it is valid to install minus the debuginfo files on
> almost all Production hosts.  I am experimenting with developing my
> scripts off box with my cache directory set to an exported read-only
> NFS share which is then mounted as the module cache directory on my
> Production boxes [...]

More than that - on such production boxes, you will need to install
only the "staprun" (formerly "stapd") binary, now separated into a
systemtap-runtime RPM.  For the moment though please be careful with
building probes for mismatching machines: the module address tables
are not yet fully adaptive.

- FChE

RE: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Ken Robson]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

> varap wrote:
> > fche wrote:
> > [...]
> > >Unfortunately, other customers prefer that systemtap's loose
> > >dependencies (particularly, the kernel-debuginfo) not be 
> installed by
> > >default, even if they choose an "install everything" option at the
> > >anaconda screens.  [..]
> 
> > O.k, before i think of a way to satisfy both the needs i need to 
> > understand the objection of the other group.
> 
> > The objection of other customers to not to install 
> debuginfo package is 
> > it because of wasted disk space due to large size of 
> debuginfo package 
> > or time to install or something else.
> 
> All those, plus the manual way in which RHEL debuginfo packages are at
> present published.
> 
> > I am assuming kernel debuginfo is considered a dependency for
> > SystemTap, am I right?
> 
> It is a loose dependency, in that the RPM does not list it as such
> (any more), and that it is theoretically possible to run some scripts
> without kprobes, and thus without dwarf data.
> 
> > What is the point of installing a package that doesn't work due to
> > lack of dependencies?
> 
> Among other reasons, it simplifies the task of the person overseeing
> an initial installation.  They may not want systemtap as such, merely
> use the "everything that is on this CD" option as a time-saver.

Hi Folks,

To me it is valid to install minus the debuginfo files on almost all
Production hosts.  I am experimenting with developing my scripts off box
with my cache directory set to an exported read-only NFS share which is then
mounted as the module cache directory on my Production boxes, allowing me to
do off box Development and then run scripts on Production hosts without
elaborating, recompiling, etc.

Roll on the day of the 'debuginfo/cache' server but this works for me for
now.

Kind reagrds,

Ken. 

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3181 bytes --]

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Gerrit Huizenga]

On Tue, 31 Oct 2006 20:37:34 EST, "Frank Ch. Eigler" wrote:
> Hi -
> 
> > fche wrote:
> > [...]
> > >Unfortunately, other customers prefer that systemtap's loose
> > >dependencies (particularly, the kernel-debuginfo) not be installed by
> > >default, even if they choose an "install everything" option at the
> > >anaconda screens.  [..]
> 
> varap wrote:
> 
> > O.k, before i think of a way to satisfy both the needs i need to
> > understand the objection of the other group.
> 
> > The objection of other customers to not to install debuginfo package is
> > it because of wasted disk space due to large size of debuginfo package
> > or time to install or something else.
> 
> All those, plus the manual way in which RHEL debuginfo packages are at
> present published.
> 
> > I am assuming kernel debuginfo is considered a dependency for
> > SystemTap, am I right?
> 
> It is a loose dependency, in that the RPM does not list it as such
> (any more), and that it is theoretically possible to run some scripts
> without kprobes, and thus without dwarf data.
> 
> > What is the point of installing a package that doesn't work due to
> > lack of dependencies?
> 
> Among other reasons, it simplifies the task of the person overseeing
> an initial installation.  They may not want systemtap as such, merely
> use the "everything that is on this CD" option as a time-saver.

Um, okay, when I'm dealing with a customer issue, the *last* thing I
can get permission for is to install additional packages to debug
a problem.  If systemtap isn't usable on enterprise kernels right
out of the box at all installations, it is going to be *seriously*
hampered in terms of real marketplace adoption.

It seems like this should be more of an opt-out installation than an
opt-in - otherwise no one will opt-in until they've had to suffer
through some debug situations.

I know these things are tricky but what percentage of the overall
install footprint is the kernel debuginfo?  Aren't we talking mostly
about Enterprise configurations here?  People that often have a
Terrabyte of storge on a single blade?  The major value of SystemTap
is to save time in debugging customer environments.  If finding, installing,
?configuring? SystemTap are added to the complexity of deployment, we
are not going to save near as much time.  :(

gerrit

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

Hi -

> fche wrote:
> [...]
> >Unfortunately, other customers prefer that systemtap's loose
> >dependencies (particularly, the kernel-debuginfo) not be installed by
> >default, even if they choose an "install everything" option at the
> >anaconda screens.  [..]

varap wrote:

> O.k, before i think of a way to satisfy both the needs i need to 
> understand the objection of the other group.

> The objection of other customers to not to install debuginfo package is 
> it because of wasted disk space due to large size of debuginfo package 
> or time to install or something else.

All those, plus the manual way in which RHEL debuginfo packages are at
present published.

> I am assuming kernel debuginfo is considered a dependency for
> SystemTap, am I right?

It is a loose dependency, in that the RPM does not list it as such
(any more), and that it is theoretically possible to run some scripts
without kprobes, and thus without dwarf data.

> What is the point of installing a package that doesn't work due to
> lack of dependencies?

Among other reasons, it simplifies the task of the person overseeing
an initial installation.  They may not want systemtap as such, merely
use the "everything that is on this CD" option as a time-saver.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Tim Bird]

Vara Prasad wrote:
>> Unfortunately, other customers prefer that systemtap's loose
>> dependencies (particularly, the kernel-debuginfo) not be installed by
>> default, even if they choose an "install everything" option at the
>> anaconda screens.  Can you think of a way of satisfying both groups?
>>
>> - FChE
>>  
>>
> O.k, before i think of a way to satisfy both the needs i need to
> understand the objection of the other group.
> The objection of other customers to not to install debuginfo package is
> it because of wasted disk space due to large size of debuginfo package
> or time to install or something else.
> I am assuming kernel debuginfo is considered a dependency for SystemTap,
> am i right.
> What is the point of installing a package that doesn't work due to lack
> of dependencies? In other words are we not breaking the semantics or
> meaning of full package install if we don't install required
> dependencies when the customer chooses a package to install.

I won't answer the last question ("what's the point of installing
a non-functional package?").  But how about creating a stap wrapper
that checks for required dependencies.  If present, the wrapper removes itself
from the invocation chain (e.g. by renaming itself and real stap) and then
calls the real stap.  If the dependencies are not present, the wrapper could
tell the user to install the required packages (or do it itself).

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Vara Prasad]

Frank Ch. Eigler wrote:

>Hi -
>
>On Mon, Oct 30, 2006 at 10:27:47AM -0800, Vara Prasad wrote:
>
>  
>
>>[...]  Badari is playing the role of a customer. His point of view
>>is, i have stock RHEL 5 setup, i have not made any changes to the
>>setup and trying to use systemtap and i expect it to have all the
>>pieces needed [...]
>>    
>>
>
>Unfortunately, other customers prefer that systemtap's loose
>dependencies (particularly, the kernel-debuginfo) not be installed by
>default, even if they choose an "install everything" option at the
>anaconda screens.  Can you think of a way of satisfying both groups?
>
>- FChE
>  
>
O.k, before i think of a way to satisfy both the needs i need to 
understand the objection of the other group.
The objection of other customers to not to install debuginfo package is 
it because of wasted disk space due to large size of debuginfo package 
or time to install or something else.
I am assuming kernel debuginfo is considered a dependency for SystemTap, 
am i right.
What is the point of installing a package that doesn't work due to lack 
of dependencies? In other words are we not breaking the semantics or 
meaning of full package install if we don't install required 
dependencies when the customer chooses a package to install.

bye,
Vara Prasad

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

Hi -

On Mon, Oct 30, 2006 at 10:27:47AM -0800, Vara Prasad wrote:

> [...]  Badari is playing the role of a customer. His point of view
> is, i have stock RHEL 5 setup, i have not made any changes to the
> setup and trying to use systemtap and i expect it to have all the
> pieces needed [...]

Unfortunately, other customers prefer that systemtap's loose
dependencies (particularly, the kernel-debuginfo) not be installed by
default, even if they choose an "install everything" option at the
anaconda screens.  Can you think of a way of satisfying both groups?

- FChE

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Vara Prasad]

Frank Ch. Eigler wrote:

>Badari Pulavarty <pbadari@gmail.com> writes:
>
>  
>
>>[...]
>>Okay, I just tried it on my x86_64 RHEL5 Beta machine
>>(2.6.18-1.2714.el5) ..
>>
>># stap iostp.stp
>>semantic error: probe point mismatch at position 1 (alternatives: end
>>submit)
>>while: resolving probe point ioblock.request
>>semantic error: libdwfl failure (dwfl_linux_kernel_report_kernel): No
>>such file or directory
>>Ensure kernel debuginfo is installed
>>[...]
>>    
>>
>
>The key line is right there.
>  
>
Frank,

Badari is playing the role of a customer. His point of view is, i have 
stock RHEL 5 setup, i have not made any changes to the setup and trying 
to use systemtap and i expect it to have all the pieces needed as part 
of the default install and it should work just out of the box and it 
doesn't work as reported by him. I agree with his point of view that 
customers shouldn't be asked to jump through hoops for systemtap to 
work, any additional hurdle or setup they have to go through less 
interest they have in using our product.

bye,
Vara Prasad

>- FChE
>  
>

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Frank Ch. Eigler]

Badari Pulavarty <pbadari@gmail.com> writes:

> [...]
> Okay, I just tried it on my x86_64 RHEL5 Beta machine
> (2.6.18-1.2714.el5) ..
> 
> # stap iostp.stp
> semantic error: probe point mismatch at position 1 (alternatives: end
> submit)
> while: resolving probe point ioblock.request
> semantic error: libdwfl failure (dwfl_linux_kernel_report_kernel): No
> such file or directory
> Ensure kernel debuginfo is installed
> [...]

The key line is right there.

- FChE

RE: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Nguyen, Thang P]

Can you check your tapset/ioblock.stp to see if you have this probe
ioblock.request() ?

If you don't, and you have ioblock.submit() instead, it means you have
an earlier version of tapset. In that case, you can try this modified
script.

#! stap

probe begin {
   printf("DEVNAME\tSECTOR\tRW\n")
}

probe ioblock.submit {
   printf("%s\t%d\t%d\n", devname, sector, rw)
}

probe ioblock.end {
    printf("%s\t%d\t%d\n", devname, sector, rw)
}


Thang


>-----Original Message-----
>From: Badari Pulavarty [mailto:pbadari@gmail.com]
>Sent: Friday, October 27, 2006 1:25 PM
>To: Nguyen, Thang P
>Cc: Mike Mason; Vara Prasad; William Cohen; SystemTAP
>Subject: RE: Anyone tried SystemTap with the latest RHEL5 Beta refresh
>
>On Thu, 2006-10-26 at 13:43 -0700, Nguyen, Thang P wrote:
>> For example:
>>
>> > stap ioblock_script.stp
>>
>> ------ ioblock_script.stp ----
>> #! stap
>>
>> probe begin {
>>   printf("DEVNAME\tSECTOR\tRW\n")
>> }
>> probe ioblock.request {
>>    printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
>> }
>> probe ioblock.end {
>>    printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
>> }
>>
>> Thang
>
>Okay, I just tried it on my x86_64 RHEL5 Beta machine
>(2.6.18-1.2714.el5) ..
>
># stap iostp.stp
>semantic error: probe point mismatch at position 1 (alternatives: end
>submit)
>while: resolving probe point ioblock.request
>semantic error: libdwfl failure (dwfl_linux_kernel_report_kernel): No
>such file or directory
>Ensure kernel debuginfo is installed
>while: resolving probe point kernel.function("bio_endio")
>semantic error: no match for probe point
>while: resolving probe point ioblock.end
>Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
>
>Thanks,
>Badari

RE: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Badari Pulavarty]

On Thu, 2006-10-26 at 13:43 -0700, Nguyen, Thang P wrote:
> For example:
> 
> > stap ioblock_script.stp
> 
> ------ ioblock_script.stp ----
> #! stap
> 
> probe begin {
>   printf("DEVNAME\tSECTOR\tRW\n")
> }
> probe ioblock.request {
>    printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
> }
> probe ioblock.end {
>    printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
> }
> 
> Thang

Okay, I just tried it on my x86_64 RHEL5 Beta machine
(2.6.18-1.2714.el5) ..

# stap iostp.stp
semantic error: probe point mismatch at position 1 (alternatives: end
submit)
while: resolving probe point ioblock.request
semantic error: libdwfl failure (dwfl_linux_kernel_report_kernel): No
such file or directory
Ensure kernel debuginfo is installed
while: resolving probe point kernel.function("bio_endio")
semantic error: no match for probe point
while: resolving probe point ioblock.end
Pass 2: analysis failed.  Try again with more '-v' (verbose) options.

Thanks,
Badari

Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Li Guanglei]

Nguyen, Thang P wrote:

> For example:
> 
>> stap ioblock_script.stp
> 
> ------ ioblock_script.stp ----
> #! stap
> 
> probe begin {
>   printf("DEVNAME\tSECTOR\tRW\n")
> }
> probe ioblock.request {
>    printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
> }
> probe ioblock.end {
>    printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
> }
> 
> Thang

This script works for me on x86_64 with latest CVS codes on RHEL5.

RE: Anyone tried SystemTap with the latest RHEL5 Beta refresh

[Nguyen, Thang P]

For example:

> stap ioblock_script.stp

------ ioblock_script.stp ----
#! stap

probe begin {
  printf("DEVNAME\tSECTOR\tRW\n")
}
probe ioblock.request {
   printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
}
probe ioblock.end {
   printf("%s\t%d\t%s\n", devname, sector, bio_rw_str(rw))
}

Thang

>-----Original Message-----
>From: systemtap-owner@sourceware.org [mailto:systemtap-
>owner@sourceware.org] On Behalf Of Mike Mason
>Sent: Thursday, October 26, 2006 1:39 PM
>To: Vara Prasad
>Cc: William Cohen; SystemTAP
>Subject: Re: Anyone tried SystemTap with the latest RHEL5 Beta refresh
>
>Looks like he's trying to run the ioblock tapset file directly?  That
won't
>work.  Stap ends up loading ioblock.stp twice which results in the
>duplicate probe errors. He needs to write a script that uses the
ioblock
>tapset probes.
>
>- Mike
>
>Vara Prasad wrote:
>> One of my colleague tried SystemTap on x86_64 machine got the
following
>> error
>> stap -g ioblock.stp
>> while: registering probe alias ioblock.submit =
>> kernel.function("submit_bio")
>> semantic error: duplicate probe point pattern
>> while: registering probe alias ioblock.end =
kernel.function("bio_endio")
>> semantic error: duplicate probe point pattern
>> Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
>>
>>
>> I dont have the setup to debug his problem, was wondering if anyone
has
>> experienced these problems.
>> He is using stock RHEL 5 setup. I thought we now bundle everything
>> needed in RHEL5, correct.
>> Does he needs to download any additional packages?
>>
>> Thanks for your help,
>> Vara Prasad
>>