kernel panic when kretprobe all system calls

kernel panic when kretprobe all system calls

[lglyahoo-news]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb2312, Size: 1083 bytes --]

Hi,
  I wrote a very simple stp file:

#! stap -p4

probe begin
{
      log("systemtap starting probe")
}

probe kernel.function("sys_*")
{
	log("enter")
}

probe kernel.function("sys_*").return
{
	log("leave")
}

probe end
{
	log("Now end log")
}

  After I ran this script, it will print out lots of
"enter" and "leave" as expected. But when I switched
to another console and tried to run any command(such
as ls), the kernel will panic.
  
  I am using RHEL 4 Update 2. kernel is 2.6.9-17.EL

  I tried to comment out the probe:
kernel.function("sys_*").return, it worked well, no
kernel panic again in spite of I was running some
commands in another console at the same time.

  But if I commented out the probe:
kernel.function("sys_*"), still kernel panic.

  So it seems that there is something wrong with
kretprobe.

  I will try the latest kernel(2.6.13-2) tomorrow to
see if kretprobe still has this error.



		
___________________________________________________________ 
ÑÅ»¢Ãâ·ÑGÓÊÏ䣭No.1µÄ·À¶¾·ÀÀ¬»ø³¬´óÓÊÏä 
http://cn.mail.yahoo.com

Re: kernel panic when kretprobe all system calls

[Frank Ch. Eigler]

Hi -

> [... kretprobes on syscalls ...]

This has been seen before, and is being tracked as bug #1345 in
bugzilla.  It has apparently been reproduced by the kretprobes
developers and is being debugged.  More RAM seems to trigger the
bug less often.

- FChE

Re: kernel panic when kretprobe all system calls

[Hien Nguyen]

Frank Ch. Eigler wrote:

>Hi -
>
>  
>
>>[... kretprobes on syscalls ...]
>>    
>>
>
>This has been seen before, and is being tracked as bug #1345 in
>bugzilla.  It has apparently been reproduced by the kretprobes
>developers and is being debugged.  More RAM seems to trigger the
>bug less often.
>
>- FChE
>
>  
>
It helps if  we don't insert return probes in sys_calls such as 
sys_execve, sys_exit, sys_groupexit. Frank, can we tempory put an 
retprobe embargo policy on those system calls?

Hien.

Re: kernel panic when kretprobe all system calls

[Frank Ch. Eigler]

Hi -

On Mon, Sep 26, 2005 at 08:55:08AM -0700, Hien Nguyen wrote:
> [...]
> It helps if  we don't insert return probes in sys_calls such as 
> sys_execve, sys_exit, sys_groupexit. 

Does that solve the problem completely?  We would need to treat
similarly some of their subsidiary "noreturn" functions.

> Frank, can we tempory put an retprobe embargo policy on those system
> calls?

By policy, you mean code that blacklists certain functions from
probing?  I was holding off until results from jkenisto's bug #1235
were collected.

- FChE

Re: kernel panic when kretprobe all system calls

[Hien Nguyen]

Frank Ch. Eigler wrote:

>Hi -
>
>On Mon, Sep 26, 2005 at 08:55:08AM -0700, Hien Nguyen wrote:
>  
>
>>[...]
>>Does that solve the problem completely?  We would need to treat
>>similarly some of their subsidiary "noreturn" functions.
>>
>>    
>>
It does seem solve a bulk of the problem (particularly excluding 
sys_execve). So far those are the "noreturn" system calls (they 're also 
tearing down task) that we know of.

>By policy, you mean code that blacklists certain functions from
>probing?  I was holding off until results from jkenisto's bug #1235
>were collected.
>
>  
>
Yes, put them on the blacklist.

Hien.

Re: kernel panic when kretprobe all system calls

[Guang Lei Li]

systemtap-owner@sources.redhat.com wrote on 2005-09-26 23:55:08:

> Frank Ch. Eigler wrote:
> 
> >Hi -
> >
> >>[... kretprobes on syscalls ...]
> >
> >This has been seen before, and is being tracked as bug #1345 in
> >bugzilla.  It has apparently been reproduced by the kretprobes
> >developers and is being debugged.  More RAM seems to trigger the
> >bug less often.
> >
> >- FChE
> >
> It helps if  we don't insert return probes in sys_calls such as 
> sys_execve, sys_exit, sys_groupexit. Frank, can we tempory put an 
> retprobe embargo policy on those system calls?
> 
> Hien.
> 
Hi, how is the bug #1345 going now? I ran the latest systemTap to probe
all returns of syscalls on my x86, and still got kernel panic. Although
I didn't meet the same problem on my Power5 system, I think it is due to
the large RAM of it(15G)

I looked into the comments in the bugzilla, and found Hien has worked 
out a fix for i386. But he abandoned his idea because of not portable.
So at present, will I avoid this problem only by not probing the return
of those syscalls in the blacklist? Is there a better solution now?

Thanks.

Re: kernel panic when kretprobe all system calls

[Jim Keniston]

On Mon, 2005-11-07 at 18:30, Guang Lei Li wrote:
> systemtap-owner@sources.redhat.com wrote on 2005-09-26 23:55:08:
> 
> > Frank Ch. Eigler wrote:
> > 
> > >Hi -
> > >
> > >>[... kretprobes on syscalls ...]
> > >
> > >This has been seen before, and is being tracked as bug #1345 in
> > >bugzilla.  It has apparently been reproduced by the kretprobes
> > >developers and is being debugged.  More RAM seems to trigger the
> > >bug less often.
> > >
> > >- FChE
> > >
> > It helps if  we don't insert return probes in sys_calls such as 
> > sys_execve, sys_exit, sys_groupexit. Frank, can we tempory put an 
> > retprobe embargo policy on those system calls?
> > 
> > Hien.
> > 
> Hi, how is the bug #1345 going now? I ran the latest systemTap to probe
> all returns of syscalls on my x86, and still got kernel panic. Although
> I didn't meet the same problem on my Power5 system, I think it is due to
> the large RAM of it(15G)
> 
> I looked into the comments in the bugzilla, and found Hien has worked 
> out a fix for i386. But he abandoned his idea because of not portable.
> So at present, will I avoid this problem only by not probing the return
> of those syscalls in the blacklist? Is there a better solution now?
> 
> Thanks.

Hien and I have discussed an architecture-independent fix to kprobes
that would allow you to set return probes on "problem" functions such as
sys_execve without encountering the problems described in #1345.  I've
updated the bugzilla entry.

Jim