run time error - how to debug?

run time error - how to debug?

[Peter Teoh]

After an earlier patch for symbols.c in previous email, I run several
stap probes and all encounter the same errors - compilation comples,
insmod ok, and lsmod verified ok, but nothing runs (command was stap
-vvvvvk used for compilation).

Pass 5: starting run.
Running /usr/local/bin/staprun -v -v
/tmp/stapS35EY1/stap_ad3df1525135daf3d6a9f5e4e8dcda2c_14196.ko
staprun:main:237
modpath="/tmp/stapS35EY1/stap_ad3df1525135daf3d6a9f5e4e8dcda2c_14196.ko",
modname="stap_ad3df1525135daf3d6a9f5e4e8dcda2c_14196"
staprun:init_staprun:194 init_staprun
staprun:insert_module:47 inserting module
staprun:insert_module:66 module options: _stp_bufsize=0
Pass 5: run completed in 0usr/0sys/35real ms.
Pass 5: run failed.  Try again with more '-v' (verbose) options.
Keeping temporary directory "/tmp/stapS35EY1"

lsmod|grep stap_ad
stap_ad3df1525135daf3d6a9f5e4e8dcda2c_14196   453132  1

Case 2:

Pass 5: starting run.
Running /usr/local/bin/staprun -v -v
/tmp/stap2mffUi/stap_c0c666510da0e3ba8e16cde92214621f_14763.ko
staprun:main:237
modpath="/tmp/stap2mffUi/stap_c0c666510da0e3ba8e16cde92214621f_14763.ko",
modname="stap_c0c666510da0e3ba8e16cde92214621f_14763"
staprun:init_staprun:194 init_staprun
staprun:insert_module:47 inserting module
staprun:insert_module:66 module options: _stp_bufsize=0
Pass 5: run completed in 0usr/0sys/20real ms.
Pass 5: run failed.  Try again with more '-v' (verbose) options.
Keeping temporary directory "/tmp/stap2mffUi"

lsmod |grep stap_c0c
stap_c0c666510da0e3ba8e16cde92214621f_14763   448436  1

How do I debug this?
-- 
Regards,
Peter Teoh

Re: run time error - how to debug?

[Peter Teoh]

Let me try again (after rebooting the system):

a.   Run the stap probe:

probe kernel.function("*@mm/swapfile.c").call {
        printf ("%s -> %s\n", thread_indent(1), probefunc())
        print_backtrace();
        printf("\n");
}
probe kernel.function("*@mm/swapfile.c").return {
        printf ("%s <- %s\n", thread_indent(-1), probefunc())
        printf("\n");
}

b.   It got through pass5, and loaded as module into kernel, and lsmod gives:

stap_861c1a9009233ff719572d129f564e06_12247   451808  1

but still failed as:

Pass 5: run failed.  Try again with more '-v' (verbose) options.

c.   dmesg gives:

BUG: unable to handle kernel paging request at 40000000
IP: [<40000000>]
*pde = 00000000
Oops: 0000 [#1] SMP

Pid: 4516, comm: staprun Not tainted (2.6.26-rc2 #8)
EIP: 0060:[<40000000>] EFLAGS: 00010246 CPU: 0
EIP is at 0x40000000
EAX: 00000000 EBX: f7917b00 ECX: 00000000 EDX: f3669e70
ESI: c122a8c0 EDI: f8e551dc EBP: f8ef2800 ESP: f3669d64
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process staprun (pid: 4516, ti=f3669000 task=f69bab50 task.ti=f3669000)
Stack: f8e8d3f6 00000001 f3e84e44 74737973 61746d65 c2000070 00000286 c0434c61
       51518c1a 0000014e 00000286 ffffffff c2012780 f780c6c0 00000001 c04d8ce2
       f3669df0 00000009 00000001 70617473 3136385f 39613163 32393030 66663333
Call Trace:
 [<f8e8d3f6>] _stp_transport_init+0x3a3/0x4cd
[stap_861c1a9009233ff719572d129f564e06_12247]
 [<c0434c61>] hrtimer_start+0xf4/0x10d
 [<c04d8ce2>] idr_get_empty_slot+0x11e/0x1ef
 [<c04a63a7>] sysfs_ilookup_test+0x0/0xd
 [<c0411b1b>] native_smp_call_function_mask+0xab/0xe8
 [<c0411f43>] do_flush_tlb_all+0x0/0x40
 [<c0411f43>] do_flush_tlb_all+0x0/0x40
 [<c04533a4>] free_pages_bulk+0x129/0x18e
 [<c043fb4c>] sys_init_module+0x1844/0x19b4
 [<c04b92b1>] avc_has_perm+0x3c/0x46
 [<c04dc01e>] simple_strtoull+0x0/0xdc
 [<c04b611a>] security_capset_set+0xe/0x11
 [<c040380d>] sysenter_past_esp+0x6a/0x91
 [<c0610000>] yenta_probe+0x201/0x567
 =======================
Code:  Bad EIP value.
EIP: [<40000000>] 0x40000000 SS:ESP 0068:f3669d64
---[ end trace 145d18014bdbe738 ]---

What I don't understand is why EIP is at 0x40000000? as for
_stp_transport_init().....

Re: run time error - how to debug?

[Frank Ch. Eigler]

"Peter Teoh" <htmldeveloper@gmail.com> writes:

> After an earlier patch for symbols.c in previous email [...]

I don't think that patch is correct.  If you must use a 2.6.25 or
2.6.26-track kernel, please wait or perhaps use an older snapshot of
systemtap.  Does one come for your distribution and does it work for
you?

- FChE

Re: run time error - how to debug?

[Peter Teoh]

no problems.....just throwing ideas for consideration....

Have a nice day!!

On 5/18/08, Frank Ch. Eigler <fche@redhat.com> wrote:
> "Peter Teoh" <htmldeveloper@gmail.com> writes:
>
> > After an earlier patch for symbols.c in previous email [...]
>
> I don't think that patch is correct.  If you must use a 2.6.25 or
> 2.6.26-track kernel, please wait or perhaps use an older snapshot of
> systemtap.  Does one come for your distribution and does it work for
> you?
>
> - FChE
>


-- 
Regards,
Peter Teoh