Profiling Kernel function blocks

Profiling Kernel function blocks

[Domenico Di Leo]

Hi,
I was wondering if anybody has experienced a
hang of the entire OS when profiling
the functions of the kernel module.
In other words, when I run the following script
probe kernel.function("*@kernel/*.c").return{
     printf("%s  %s", threadindent(-1),probefunc());
}
the entire OS gets stuck.
I also tried with
probe kernel.function("*@kernel/*sched.c").return{
     printf("%s  %s", threadindent(-1),probefunc());
}
and after some printing the OS gets stuck again.

I'm using systemtap v1.4 and kernel 2.6.24 on Gentoo.
Thank

-- 
Domenico Di Leo, PhD student, Università degli Studi di Napoli Federico II
Ph:     +39 081 676770
Fax:    +39 081 676574
Web: http://wpage.unina.it/domenico.dileo (WP)

Re: Profiling Kernel function blocks

[Domenico Di Leo]

Just I typo, I meant thread_indent

On 8/2/2012 4:17 PM, Domenico Di Leo wrote:
> Hi,
> I was wondering if anybody has experienced a
> hang of the entire OS when profiling
> the functions of the kernel module.
> In other words, when I run the following script
> probe kernel.function("*@kernel/*.c").return{
>     printf("%s  %s", threadindent(-1),probefunc());
> }
> the entire OS gets stuck.
> I also tried with
> probe kernel.function("*@kernel/*sched.c").return{
>     printf("%s  %s", threadindent(-1),probefunc());
> }
> and after some printing the OS gets stuck again.
>
> I'm using systemtap v1.4 and kernel 2.6.24 on Gentoo.
> Thank
>


-- 
Domenico Di Leo, PhD student, Università degli Studi di Napoli Federico II
Ph:     +39 081 676770
Fax:    +39 081 676574
Web: http://wpage.unina.it/domenico.dileo (WP)

Re: Profiling Kernel function blocks

[Mark Wielaard]

On Thu, 2012-08-02 at 16:17 +0200, Domenico Di Leo wrote:
> I was wondering if anybody has experienced a
> hang of the entire OS when profiling
> the functions of the kernel module.
> In other words, when I run the following script
> probe kernel.function("*@kernel/*.c").return{
>      printf("%s  %s", threadindent(-1),probefunc());
> }
> the entire OS gets stuck.
> I also tried with
> probe kernel.function("*@kernel/*sched.c").return{
>      printf("%s  %s", threadindent(-1),probefunc());
> }
> and after some printing the OS gets stuck again.

This sounds like:
Bug 2725 - function("*") probes sometimes crash & burn
http://sourceware.org/bugzilla/show_bug.cgi?id=2725

> I'm using systemtap v1.4 and kernel 2.6.24 on Gentoo.

Those are pretty old. You might want to try systemtap 1.8.

Cheers,

Mark

Re: Profiling Kernel function blocks

[Frank Ch. Eigler]

Domenico Di Leo <domenico.dileo@unina.it> writes:

> [...]
> In other words, when I run the following script
> probe kernel.function("*@kernel/*.c").return{
>     printf("%s  %s", threadindent(-1),probefunc());
> }
> the entire OS gets stuck.
> [...]

Several parts of the kernel are not capable of tolerating kprobes in
various places.  While mjw is right to point to our PR2725, as far as
we know, these are long-standing kernel (rather than systemtap) bugs
that LKML has not been able (or find time?) to solve.

Our normal suggestion is to try later rather than earlier kernel and
systemtap versions, and limit wildcarded probes like that to smaller
and less-sensitive sections of the kernel.

- FChE

Re: Profiling Kernel function blocks

[Domenico Di Leo]

I tried with systemtap 1.8, I can't change Kernel version.
and now I get the error message
Unknown symbol system_lookup_name during Pass 4.
The kernel is configured with CONFIG_KALLSUYMS_ALL=y.

On 8/2/2012 5:07 PM, Mark Wielaard wrote:
> On Thu, 2012-08-02 at 16:17 +0200, Domenico Di Leo wrote:
>> I was wondering if anybody has experienced a
>> hang of the entire OS when profiling
>> the functions of the kernel module.
>> In other words, when I run the following script
>> probe kernel.function("*@kernel/*.c").return{
>>       printf("%s  %s", threadindent(-1),probefunc());
>> }
>> the entire OS gets stuck.
>> I also tried with
>> probe kernel.function("*@kernel/*sched.c").return{
>>       printf("%s  %s", threadindent(-1),probefunc());
>> }
>> and after some printing the OS gets stuck again.
> This sounds like:
> Bug 2725 - function("*") probes sometimes crash&  burn
> http://sourceware.org/bugzilla/show_bug.cgi?id=2725
>
>> I'm using systemtap v1.4 and kernel 2.6.24 on Gentoo.
> Those are pretty old. You might want to try systemtap 1.8.
>
> Cheers,
>
> Mark
>


-- 
Domenico Di Leo, PhD student, Università degli Studi di Napoli Federico II
Ph:     +39 081 676770
Fax:    +39 081 676574
Web: http://wpage.unina.it/domenico.dileo (WP)

Re: Profiling Kernel function blocks

[Mark Wielaard]

On Fri, 2012-08-03 at 15:28 +0200, Domenico Di Leo wrote:
> I tried with systemtap 1.8, I can't change Kernel version.
> and now I get the error message
> Unknown symbol system_lookup_name during Pass 4.

Strange, I don't know why that symbol would be used.
Could you provide a bit more info?
For example the actual command/script you are trying to run and the -v
output? And/Or the stap-report output, etc.

Thanks,

Mark