Kernel crash when using systemtap

Kernel crash when using systemtap

[Padmanabhan S N]

While using systemtap to do function call tracing (.call, .inline. .return probes) on one of our proprietary kernel modules, the system crashes. The function call tracing however works perfectly fine on kernel functions and other modules (for ex. ext3, nfsd, etc.).


I've tried downgrading (to versions as low as 0.8) and upgrading to the latest version, but that doesnt help. I've even tried blacklisting some of our module functions, but that doesnt work either. 

Any pointers ? 


The environment is RHEL 5.5 64-bit. I've tried to look at the crash 
dump, and it does point to the generated systemtap module. But I 
couldn't proceed beyond that. Here is a bt from one of the crash dumps. 

 #0 [ffff8100956f17a0] crash_kexec at ffffffff800aeb6b
 #1 [ffff8100956f1860] __die at ffffffff80066157
 #2 [ffff8100956f18a0] do_page_fault at ffffffff80067dd7
 #3 [ffff8100956f1990] error_exit at ffffffff8005ede9
 #4 [ffff8100956f1af0] add_to_page_cache at ffffffff8000c722
 #5 [ffff8100956f1b48] _stp_map_get_ii at ffffffff889d8459 [stap_44a2f6019f0e29a88c9470c55950ae1a_14041197]
 #6 [ffff8100956f1c20] __generic_file_aio_read at ffffffff8000c5a8
 #7 [ffff8100956f1c90] generic_file_aio_read at ffffffff80016dd4
 #8 [ffff8100956f1e10] kretprobe_trampoline_holder at ffffffff80066b82
 #9 [ffff8100956f1f10] vfs_read at ffffffff8000b6b0
#10 [ffff8100956f1f40] sys_read at ffffffff80011c01


Thanks,
Padhu.

Re: Kernel crash when using systemtap

[Frank Ch. Eigler]

Padmanabhan S N <paddusn@yahoo.com> writes:

> While using systemtap to do function call tracing (.call,
> .inline. .return probes) on one of our proprietary kernel modules,
> the system crashes. [...]

OK.

> The environment is RHEL 5.5 64-bit. I've tried to look at the crash
> dump, and it does point to the generated systemtap module. But I
> couldn't proceed beyond that. Here is a bt from one of the crash dumps.
>
> [...]
> #4 [ffff8100956f1af0] add_to_page_cache at ffffffff8000c722
> #5 [ffff8100956f1b48] _stp_map_get_ii at ffffffff889d8459 [stap_44a2f6019f0e29a88c9470c55950ae1a_14041197]
> [...]

How big is this module?  That _14041197 string makes it sound large.
Can you share the script source?  Consider removing the .inline
probes, or narrow the wildcards and/or pretty-printing.  Consider
testing out an updated kernel; IIRC there have been kprobes robustness
improvements.

- FChE

Re: Kernel crash when using systemtap

[Padmanabhan S N]

The stap .c file is about 16M in size and the generated .ko is about 13M.


The script is very simple, but traces all functions in our module :


probe module("_our_module_").function("*").call {
        printf("%s %s %s %s\n", thread_indent(1), "->",  probefunc(), $$parms);
}

probe module("_our_module_").function("*").inline {
        printf("%s %s %s %s\n", thread_indent(1), "->",  probefunc(), $$parms);
        thread_indent(-1);
}

probe module("_our_module_").function("*").return {
        printf("%s %s %s %s\n", thread_indent(-1), "<-",  probefunc(), $$return);
}


I did try removing the inline probes. But that doesnt help. 

Narrowing down the wildcards will limit the scope of tracing, right ?. And we are pretty much tied to RHEL 5.5 at this moment. 

Is there anything else that I can try ?


~Padhu.



----- Original Message -----
From: Frank Ch. Eigler <fche@redhat.com>
To: Padmanabhan S N <paddusn@yahoo.com>
Cc: "systemtap@sourceware.org" <systemtap@sourceware.org>
Sent: Monday, July 18, 2011 12:37 PM
Subject: Re: Kernel crash when using systemtap


Padmanabhan S N <paddusn@yahoo.com> writes:

> While using systemtap to do function call tracing (.call,
> .inline. .return probes) on one of our proprietary kernel modules,
> the system crashes. [...]

OK.

> The environment is RHEL 5.5 64-bit. I've tried to look at the crash
> dump, and it does point to the generated systemtap module. But I
> couldn't proceed beyond that. Here is a bt from one of the crash dumps.
>
> [...]
> #4 [ffff8100956f1af0] add_to_page_cache at ffffffff8000c722
> #5 [ffff8100956f1b48] _stp_map_get_ii at ffffffff889d8459 [stap_44a2f6019f0e29a88c9470c55950ae1a_14041197]
> [...]

How big is this module?  That _14041197 string makes it sound large.
Can you share the script source?  Consider removing the .inline
probes, or narrow the wildcards and/or pretty-printing.  Consider
testing out an updated kernel; IIRC there have been kprobes robustness
improvements.

- FChE

Re: Kernel crash when using systemtap

[Frank Ch. Eigler]

Hi -

On Mon, Jul 18, 2011 at 02:08:27PM -0700, Padmanabhan S N wrote:
> The stap .c file is about 16M in size and the generated .ko is about 13M.

Wow.

> [...]
> The script is very simple, but traces all functions in our module :
> [...]
> I did try removing the inline probes. But that doesnt help. 

OK.  (Remember that the .inline probes don't constitute a
nesting-trackable function entry, so thread_indent(1) is wrong.)

> Narrowing down the wildcards will limit the scope of tracing, right?

Yes.

> Is there anything else that I can try ?

Taking out the $$parms printing - or restricting it to some key
functions only - should dramatically shrink things.

- FChE