[Bug runtime/17461] New: probing process.end crashes on busy systems

[Bug runtime/17461] New: probing process.end crashes on busy systems

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17461

            Bug ID: 17461
           Summary: probing process.end crashes on busy systems
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: jlebon at redhat dot com

Created attachment 7816
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7816&action=edit
crash_testcase.exp

Running the following simple script on a busy system (where many processes are
created/destroyed quickly) eventually causes the system to lock up. It takes a
while sometimes to occur (e.g. 1-2 hours), but it always does. I haven't been
able so far to determine the cause of the issue, although the backtraces might
implicate utrace.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17461] probing process.end crashes on busy systems

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17461

--- Comment #1 from Jonathan Lebon <jlebon at redhat dot com> ---
Created attachment 7817
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7817&action=edit
dmesg.log

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17461] probing process.end crashes on busy systems

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17461

--- Comment #2 from Jonathan Lebon <jlebon at redhat dot com> ---
Forgot to add, this happened on f20 3.16.2-200 on git stap at least as of
commit 3525152, but also earlier (including prior to the rt patches). Will try
to do a bisect.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17461] probing process.end crashes on busy systems

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17461

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #3 from David Smith <dsmith at redhat dot com> ---
I'd certainly suspect utrace, especially since I see utrace_free() in your
dmesg output. However, I also see _raw_spin_lock, and that's got me confused.
We added some patches recently to add support for realtime kernels, but we
shouldn't be using raw spinlocks anywhere but realtime kernels.

The only real utrace change lately was the following:

====
commit d9d07e99777c6e7aaaa8db0049c5fd5e5a2f01b0                                 
Author: David Smith <dsmith@redhat.com>                                         
Date:   Fri Jul 18 15:49:39 2014 -0500                                          

    Fixed PR17181 by making utrace handle interrupting processes better.        
====

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17461] probing process.end crashes on busy systems

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17461

--- Comment #4 from Jonathan Lebon <jlebon at redhat dot com> ---
Created attachment 8312
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8312&action=edit
dmesg.log

This is still an issue on the latest f20 3.19.5 with the latest git stap.
Interestingly, adding debug statements in utrace_free() confirms that the crash
does not happen there, but the rest of the stack is still very similar (showing
a backtrace coming from exit() related calls).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/17461] probing process.end crashes on busy systems

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17461

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |fche at redhat dot com
         Resolution|---                         |WORKSFORME

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> ---
running this test on a rawhide (5.13-rc0 kernel, 4.5-rc stap), it's solid.

-- 
You are receiving this mail because:
You are the assignee for the bug.