systemtap.base/kmodule.exp crashes on RHEL4-U2 x86-64 kernel

systemtap.base/kmodule.exp crashes on RHEL4-U2 x86-64 kernel

[William Cohen]

Is anyone else exercising SystemTap on RHEL4-U2 x86-64 machines, 
2.6.9-22.ELsmp. This morning when I built the the latest snapshot of 
systemtap (20051019) and ran some tests with it I mangaged to get the 
kernel to crash on the systemtap.base/kmodule.exp.

Additional characterization of the problem

The fault happens with both the smp and up kernel in 
arch_copy_kprobes+17, the insruction that attempts to copy the probed 
instruction. Something wrong with the probe location.

Apparently this latest SystemTap snapshot is able to get address in the 
module from the debuginfo.

$ stap -v -p4 -k systemtap.base/kmodule.stp
Created temporary directory "/tmp/stapL43JdQ"
Searched '/usr/share/systemtap/tapset/2.6.9-22.ELsmp/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6.9/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/share/systemtap/tapset/*.stp', match count 9
Pass 1: parsed user script and 9 library script(s).
parsed 'ext3_sync_file' -> func 'ext3_sync_file'
pattern 'ext3' matches module 'ext3'
focused on module 'ext3' = [7fffffffffffffff-80000000000219d0, bias 0]
pattern 'ext3_sync_file' matches function 'ext3_sync_file'
selected function ext3_sync_file
querying prologue-end of function 'ext3_sync_file'
Pass 2: analyzed user script.  3 probe(s), 10 function(s), 1 global(s).
Running grep " [tT] " /proc/kallsyms | sort -k 1,16 -s -o 
/tmp/stapL43JdQ/symbols.sorted
Pass 3: translated to C into "/tmp/stapL43JdQ/stap_3784.c"
Running make -C "/lib/modules/2.6.9-22.ELsmp/build" M="/tmp/stapL43JdQ" 
modules
make: Entering directory `/usr/src/kernels/2.6.9-22.EL-smp-x86_64'
   CC [M]  /tmp/stapL43JdQ/stap_3784.o
   Building modules, stage 2.
   MODPOST
   CC      /tmp/stapL43JdQ/stap_3784.mod.o
   LD [M]  /tmp/stapL43JdQ/stap_3784.ko
make: Leaving directory `/usr/src/kernels/2.6.9-22.EL-smp-x86_64'
Pass 4: compiled into "stap_3784.ko"
Keeping temporary directory "/tmp/stapL43JdQ"


The address for the probe in the generated code looks odd.

static struct kprobe dwarf_kprobe_1[1]= {
   {.addr= (void *) 0x8000000000001f54}
};

Is this address fixed before it is used?  Looking through /proc/modules.

$ more /proc/modules |grep ext3
ext3 137681 3 - Live 0xffffffffa0051000
jbd 68849 1 ext3, Live 0xffffffffa003f000

How is the address in the struct kprobe converted into the address 
usable by kprobes?

Looking through the generated code there is a big table at the end of 
the generated code, stap_symbols. Is this used for something? I didn't 
see it referenced in the generated code.

-Will

Re: systemtap.base/kmodule.exp crashes on RHEL4-U2 x86-64 kernel

[Martin Hunt]

On Wed, 2005-10-19 at 12:09 -0400, William Cohen wrote:
> Is anyone else exercising SystemTap on RHEL4-U2 x86-64 machines, 
> 2.6.9-22.ELsmp. This morning when I built the the latest snapshot of 
> systemtap (20051019) and ran some tests with it I mangaged to get the 
> kernel to crash on the systemtap.base/kmodule.exp.

Happens to me too. Works fine on i686.

> Looking through the generated code there is a big table at the end of 
> the generated code, stap_symbols. Is this used for something? I didn't 
> see it referenced in the generated code.

The runtime uses it to lookup symbols.

2005-10-06  Frank Ch. Eigler  <fche@elastic.org>

        PR 1332.
        * translate.cxx (emit_symbol_data): New function to transcribe
        a processed address->symbol lookup table, based
upon /proc/kallsyms.

Re: systemtap.base/kmodule.exp crashes on RHEL4-U2 x86-64 kernel

[Hien Nguyen]

Martin Hunt wrote:

>The runtime uses it to lookup symbols.
>
>2005-10-06  Frank Ch. Eigler  <fche@elastic.org>
>
>        PR 1332.
>        * translate.cxx (emit_symbol_data): New function to transcribe
>        a processed address->symbol lookup table, based
>upon /proc/kallsyms.
>
>
>  
>
Martin, does that mean we don't need the dumpstack_to_buffer in the 
kernel anymore? I and Anil are working on the dumpstack_to_buffer() api. 
Perhaps, we don't need to that anymore.

Hien.