Symbol resolution problem in stap on FC5

Symbol resolution problem in stap on FC5

[srinivasa]

Hi
 I have installed systemtap(systemtap-20060422) along with elfutils  
patch(elfutils-0.120.tar.gz,elfutils-portability.patch) on FC5. But when 
I wrote a simple script to probe sys_open(),sys_open didn't get probed. 
So I retained temprory files and checked it.
============================================
probe kernel.statement("sys_open") {
                printf("hi\n");
              }
==============================================

Found that in stap_9697.c address to be probed is not valid.
=====================================================
static struct kprobe dwarf_kprobe_probe_0[1]= {
  {.addr= (void *) 0xc0151313}   <<<<this adress to be probed>>>
};

char const * dwarf_kprobe_probe_0_location_names[1] = {
  "kernel.function(\"sys_open@fs/open.c:1089\")"
};
========================================================
[root@localhost stap]# cat /proc/kallsyms | grep sys_open
c0152074 T do_sys_open
c015211e T sys_openat
c0152139 T sys_open
=========================================
But when I specified the address explicitly it works cleraly.
=================================
probe kernel.statement(0xc0152139) {
                printf("hi\n");
              }
====================================
These are the kernel packages I have
===============================
rpm -qa | grep kernel
kernel-debuginfo-2.6.15-1.2054_FC5
kernel-smp-devel-2.6.15-1.2054_FC5
kernel-devel-2.6.15-1.2054_FC5
kernel-smp-2.6.15-1.2054_FC5
kernel-2.6.15-1.2054_FC5
===================================
Is my analysis is correct? or this is a bug. Please clarify.

Re: Symbol resolution problem in stap on FC5

[Frank Ch. Eigler]

srinivasa@in.ibm.com wrote:

> [...]  I have installed systemtap(systemtap-20060422) along with
> elfutils patch(elfutils-0.120.tar.gz,elfutils-portability.patch) on
> FC5. But when I wrote a simple script to probe sys_open(),sys_open
> didn't get probed. [...]

I can't tell which platform this occurred on.  In any case, running
stap with great verbosity (-vvvv) will produce extra information about
why systemtap picks any particular address to probe.


- FChE

Re: Symbol resolution problem in stap on FC5

[Ananth N Mavinakayanahalli]

On Tue, Apr 25, 2006 at 10:05:38AM -0400, Frank Ch. Eigler wrote:
> 
> srinivasa@in.ibm.com wrote:
> 
> > [...]  I have installed systemtap(systemtap-20060422) along with
> > elfutils patch(elfutils-0.120.tar.gz,elfutils-portability.patch) on
> > FC5. But when I wrote a simple script to probe sys_open(),sys_open
> > didn't get probed. [...]
> 
> I can't tell which platform this occurred on.  In any case, running
> stap with great verbosity (-vvvv) will produce extra information about
> why systemtap picks any particular address to probe.

Looking at the size/value of the probed address suggests this is
plain i386.

Ananth

Re: Symbol resolution problem in stap on FC5

[srinivasa]

Ananth N Mavinakayanahalli wrote:
> On Tue, Apr 25, 2006 at 10:05:38AM -0400, Frank Ch. Eigler wrote:
>   
>> srinivasa@in.ibm.com wrote:
>>
>>     
>>> [...]  I have installed systemtap(systemtap-20060422) along with
>>> elfutils patch(elfutils-0.120.tar.gz,elfutils-portability.patch) on
>>> FC5. But when I wrote a simple script to probe sys_open(),sys_open
>>> didn't get probed. [...]
>>>       
>> I can't tell which platform this occurred on.  In any case, running
>> stap with great verbosity (-vvvv) will produce extra information about
>> why systemtap picks any particular address to probe.
>>     
>
> Looking at the size/value of the probed address suggests this is
> plain i386.
>
> Ananth
>   
Yes its a i386 machine
==============================================================================
[root@localhost stap]# uname -a
Linux localhost.localdomain 2.6.15-1.2054_FC5 #1 Tue Mar 14 15:48:33 EST 
2006 i686 i686 i386 GNU/Linux
================================================================================
On running stap -vvvv open.stp ,I get this
============================================================
root@localhost stap]# stap -vvvv open.stp
Created temporary directory "/tmp/stapRDfHTo"
Searched 
'/usr/local/share/systemtap/tapset/2.6.15-1.2054_FC5/i686/*.stp', match 
count 0
Searched '/usr/local/share/systemtap/tapset/2.6.15-1.2054_FC5/*.stp', 
match count 0
Searched '/usr/local/share/systemtap/tapset/2.6.15/i686/*.stp', match 
count 0
Searched '/usr/local/share/systemtap/tapset/2.6.15/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/2.6/i686/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/i686/*.stp', match count 1
Searched '/usr/local/share/systemtap/tapset/*.stp', match count 15
Pass 1: parsed user script and 16 library script(s) in 
100usr/0sys/138real ms.
parsed 'sys_open' -> func 'sys_open'
pattern 'kernel' matches module 'kernel'
focused on module 'kernel' = [c0100000-c041efdc, bias 0]
pattern 'sys_open' matches function 'sys_open'
selected function sys_open
prologue searching function 'sys_open' 0xc0151313-0xc015132d@fs/open.c:1089
checking line record 0xc0151313@fs/open.c:1093
prologue found function 'sys_open' (naked) = 0xc0151313
probe sys_open@fs/open.c:1089 pc=0xc0151313
pattern 'kernel' matches module 'kernel'
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 global(s) in 
170usr/40sys/246real ms.
probe_0 locks nothing
Running grep " [tT] " /proc/kallsyms | sort -k 1,8 -s -o 
/tmp/stapRDfHTo/symbols.sorted
==================================================================================

Re: Symbol resolution problem in stap on FC5

[Frank Ch. Eigler]

srinivasa@in.ibm.com wrote (in two separate messages):

> probe kernel.statement("sys_open") {
>                 printf("hi\n");
> [...]
> char const * dwarf_kprobe_probe_0_location_names[1] = {
>   "kernel.function(\"sys_open@fs/open.c:1089\")"
> };

Note that there is already a separate bug there.  When you specify
"kernel.statement()", systemtap should probe the address nearest the
statement, not the logical function entry.  So, in this case, it
should not look for a prologue at all (new bug #2608).

> Searched '/usr/local/share/systemtap/tapset/2.6.15-1.2054_FC5/i686/*.stp',
> [...]
> parsed 'sys_open' -> func 'sys_open'
> focused on module 'kernel' = [c0100000-c041efdc, bias 0]
> [...]
> prologue searching function 'sys_open' 0xc0151313-0xc015132d@fs/open.c:1089
> checking line record 0xc0151313@fs/open.c:1093
> prologue found function 'sys_open' (naked) = 0xc0151313
> probe sys_open@fs/open.c:1089 pc=0xc0151313

The only thing that occurs to me is that perhaps your running kernel
and your debuginfo have a slight version mismatch.  Is your "uname -r"
exactly "2.6.15-1.2054_FC5"?

At this time, systemtap does not emit any additional version-related
checking logic, so if modprobe lets slight drifts through, we don't
detect it.  If this is really happening, we need to fix it.

- FChE

Re: Symbol resolution problem in stap on FC5

[srinivasa]

Frank Ch. Eigler wrote:
> srinivasa@in.ibm.com wrote (in two separate messages):
>
>   
>> probe kernel.statement("sys_open") {
>>                 printf("hi\n");
>> [...]
>> char const * dwarf_kprobe_probe_0_location_names[1] = {
>>   "kernel.function(\"sys_open@fs/open.c:1089\")"
>> };
>>     
>
> Note that there is already a separate bug there.  When you specify
> "kernel.statement()", systemtap should probe the address nearest the
> statement, not the logical function entry.  So, in this case, it
> should not look for a prologue at all (new bug #2608).
>   
No,script fails even if I specify "kernel.function()".
=================================
[root@localhost stap]# cat open.stp
probe kernel.function("sys_open") {
printf("hi\n");
}
=====================================
[root@localhost stap]# stap -vvvv open.stp
Created temporary directory "/tmp/stap78fhtA"
Searched 
'/usr/local/share/systemtap/tapset/2.6.15-1.2054_FC5/i686/*.stp', match 
count 0
Searched '/usr/local/share/systemtap/tapset/2.6.15-1.2054_FC5/*.stp', 
match count 0
Searched '/usr/local/share/systemtap/tapset/2.6.15/i686/*.stp', match 
count 0
Searched '/usr/local/share/systemtap/tapset/2.6.15/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/2.6/i686/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/i686/*.stp', match count 1
Searched '/usr/local/share/systemtap/tapset/*.stp', match count 15
Pass 1: parsed user script and 16 library script(s) in 
100usr/0sys/138real ms.
parsed 'sys_open' -> func 'sys_open'
pattern 'kernel' matches module 'kernel'
focused on module 'kernel' = [c0100000-c041efdc, bias 0]
pattern 'sys_open' matches function 'sys_open'
selected function sys_open
prologue searching function 'sys_open' 0xc0151313-0xc015132d@fs/open.c:1089
checking line record 0xc0151313@fs/open.c:1093
prologue found function 'sys_open' (naked) = 0xc0151313
probe sys_open@fs/open.c:1089 pc=0xc0151313
=================================================
static struct kprobe dwarf_kprobe_probe_0[1]= {
{.addr= (void *) 0xc0151313}
};

char const * dwarf_kprobe_probe_0_location_names[1] = {
"kernel.function(\"sys_open@fs/open.c:1089\")"
============================================
[root@localhost stap]# cat /proc/kallsyms | grep sys_open
c0152074 T do_sys_open
c015211e T sys_openat
c0152139 T sys_open
=================================================


>   
>> Searched '/usr/local/share/systemtap/tapset/2.6.15-1.2054_FC5/i686/*.stp',
>> [...]
>> parsed 'sys_open' -> func 'sys_open'
>> focused on module 'kernel' = [c0100000-c041efdc, bias 0]
>> [...]
>> prologue searching function 'sys_open' 0xc0151313-0xc015132d@fs/open.c:1089
>> checking line record 0xc0151313@fs/open.c:1093
>> prologue found function 'sys_open' (naked) = 0xc0151313
>> probe sys_open@fs/open.c:1089 pc=0xc0151313
>>     
>
> The only thing that occurs to me is that perhaps your running kernel
> and your debuginfo have a slight version mismatch.  Is your "uname -r"
> exactly "2.6.15-1.2054_FC5"?
>   
I also doubt the samething,but it looks both are having same version.
===============
[root@localhost stap]# uname -r
2.6.15-1.2054_FC5
====================
[root@localhost stap]# rpm -qa | grep kernel
kernel-debuginfo-2.6.15-1.2054_FC5
kernel-smp-devel-2.6.15-1.2054_FC5
kernel-devel-2.6.15-1.2054_FC5
kernel-smp-2.6.15-1.2054_FC5
kernel-2.6.15-1.2054_FC5
===================================

> At this time, systemtap does not emit any additional version-related
> checking logic, so if modprobe lets slight drifts through, we don't
> detect it.  If this is really happening, we need to fix it.
>
> - FChE
>   

Re: Symbol resolution problem in stap on FC5

[srinivasa]

Frank Ch. Eigler wrote:
> Hi -
>
> On Tue, May 02, 2006 at 07:46:47PM +0530, srinivasa wrote:
>   
>> [...]
>> prologue searching function 'sys_open' 0xc0151313-0xc015132d@fs/open.c:1089
>> [...]
>>     
>>> The only thing that occurs to me is that perhaps your running kernel
>>> and your debuginfo have a slight version mismatch.  Is your "uname -r"
>>> exactly "2.6.15-1.2054_FC5"?
>>>       
>> [...]
>> [root@localhost stap]# uname -r
>> 2.6.15-1.2054_FC5
>>     
>
> How about "uname -p" versus
> % rpm -q --queryformat='%{name}-%{version}-%{release}.%{arch}\n' kernel
>
> (That is, I suspect an i586 vs i686 difference.)
>   
Yes,It was problem with i586 kernel-debuginfo rpm and i686 kernel rpm .
Now I installed i686 kernel-debuginfo rpm. Script is working correctly.
Thanks for your help.

> - FChE
>