[Bug uprobes/7092] New: uprobes probes causing OOM

[Bug uprobes/7092] New: uprobes probes causing OOM

[dsmith at redhat dot com]

Using the test cases from bug 7082 modified to use uprobes instead of
uretprobes.  Here are the testcases:

[1]
# stap -ve 'probe process("bash").function("*") {}' -w -c bash
# exit

[2]
# stap -ve 'probe process("bash").function("*") {}' -w -c bash
# bash
# exit
# exit

[3]
# stap -ve 'probe process("bash").function("main") {}' -w -c bash
# exit

[4]
# stap -ve 'probe process("bash").function("main") {}' -w -c bash
# bash
# exit
# exit

On a 16-way i386 system running kernel 2.6.18-125.el5PAE (beta RHEL5.3), with
systemtap head, here are the test results:

[1] OOM.  When run, the machine gets an OOM in pass 5.  The bash prompt is never
seen.  On the console, I see:

agetty invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
 [<c0457b31>] out_of_memory+0x72/0x1a5
 [<c0459006>] __alloc_pages+0x216/0x297
 [<c046db80>] cache_alloc_refill+0x26d/0x450
 [<c046d909>] kmem_cache_alloc+0x41/0x4b
 [<c047e134>] getname+0x1a/0xb0
 [<c046fe81>] do_sys_open+0x12/0xae
 [<c046ff4a>] sys_open+0x16/0x18
 [<c0404f17>] syscall_call+0x7/0xb

[2] same as [1]
[3] works
[4] works

-- 
           Summary: uprobes probes causing OOM
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: uprobes
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: dsmith at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug uprobes/7092] New: uprobes probes causing OOM

[Jim Keniston]

On Wed, 2008-12-10 at 19:08 +0000, dsmith at redhat dot com wrote:
...
> [1] OOM.  When run, the machine gets an OOM in pass 5.  The bash prompt is never
> seen.  On the console, I see:
> 
> agetty invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
>  ...
> 
> [2] same as [1]
> [3] works
> [4] works
> 

[1], [3], and [4] work for me, but as mentioned in 7082, I'm using odd
bits with at least a partial fix for 7082.

[2] yields a Trace/breakpoint trap, as with 7082, so I'm investigating
both 7082 and 7092 in that light.

[Bug uprobes/7092] uprobes probes causing OOM

[jkenisto at us dot ibm dot com]

------- Additional Comments From jkenisto at us dot ibm dot com  2008-12-10 19:53 -------
Subject: Re:  New: uprobes probes causing OOM

On Wed, 2008-12-10 at 19:08 +0000, dsmith at redhat dot com wrote:
...
> [1] OOM.  When run, the machine gets an OOM in pass 5.  The bash prompt is never
> seen.  On the console, I see:
> 
> agetty invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
>  ...
> 
> [2] same as [1]
> [3] works
> [4] works
> 

[1], [3], and [4] work for me, but as mentioned in 7082, I'm using odd
bits with at least a partial fix for 7082.

[2] yields a Trace/breakpoint trap, as with 7082, so I'm investigating
both 7082 and 7092 in that light.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/7092] uprobes probes causing OOM

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2008-12-11 20:50 -------
I also ran this on an (real) x86_64 system, running the 2.6.27.5-117.fc0.x86_64
kernel (and head systemtap).  Here I get:

[1] works
[2] 'Trace/breakpoint trap'
[3] works
[4] works

So, the OOM may be an i386 only (or RHEL5.3 only) problem.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/7092] uprobes probes causing OOM

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2008-12-15 19:25 -------
With the uprobes 1 patch from bug #7082, I'm still seeing the OOM for testcases
[1] and [2].

System details: kernel 2.6.18-125.el5, i686 (dual p4), systemtap head with the
patch.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/7092] uprobes probes causing OOM

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-12-15 19:39 -------
(In reply to comment #0)
> Using the test cases from bug 7082 modified to use uprobes instead of
> uretprobes.  Here are the testcases:
> 
> [1]
> # stap -ve 'probe process("bash").function("*") {}' -w -c bash
> # exit [...]
> On a 16-way i386 system running kernel 2.6.18-125.el5PAE (beta RHEL5.3), with
> systemtap head, here are the test results:
> 
> [1] OOM.  When run, the machine gets an OOM in pass 5.  The bash prompt is never
> seen.  On the console, I see:
> [...]

Can you run a systemtap probe against this kernel to count the number of
utrace engine allocations?  It could be that we are able to suck them up
at such a rate that we can exhaust free kernel memory.

See bug #7016 for some self-throttling logic we use elsewhere to limit
systemtap's impact on free kernel memory.  We could have something similar
for uprobes attachment.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/7092] uprobes probes causing OOM

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2009-03-06 17:21 -------
Please check if this is still happening.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/7092] uprobes probes causing OOM

[jkenisto at us dot ibm dot com]

------- Additional Comments From jkenisto at us dot ibm dot com  2009-03-09 20:31 -------
Given the utrace fix described in #7082, this works for me.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/7092] uprobes probes causing OOM

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2009-03-09 20:55 -------
Fix confirmed, thanks.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=7092

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.