[Bug uprobes/10185] New: stap uprobe script on rawhide causes system crash

[Bug uprobes/10185] New: stap uprobe script on rawhide causes system crash

[ananth at in dot ibm dot com]

[root@... ~]# uname -a
Linux ... 2.6.29.3-140.fc11.i586 #1 SMP Tue May 12 10:30:21 EDT 2009 i686 i686
i386 GNU/Linux

SystemTap snapshot from last week (15May)

[root@... ~]# cat /home/ananth/scripts/user_bt.stp
probe process("/home/ananth/temp/new-hwrld").function("main")
{
	log("main");
	print_ubacktrace()
}

probe process("/lib/libc.so.6").function("raise")
{
	log("raise")
	print_ubacktrace()
}

...

main
 0x080484a5 : main+0x1/0x5c [/home/ananth/temp/new-hwrld]
raise
 0x433109d1 : raise+0x1/0x90 [/lib/libc-2.10.1.so]
^C

kernel BUG at /usr/local/share/systemtap/runtime/uprobes/../uprobes2/uprobes.c:534
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/module/video/sections/__param
Modules linked in: stap_b18709e7777f5130d8ad9fd893df16a6_1453 uprobes ...

Pid: 1817, comm: stapio Not Tainted (2.6.29.3-140.fc11.i586 #1) 6824IAA
EIP: 0060:[<e1c681cc>] EFLAGS: 00010246 CPU: 0
EIP is at uprobe_free_task+0x3b/0xdc [uprobes]
EAX: ffffff8d EBX: de858a80 ECX: 00000005 EDX: 00000006
ESI: de858a20 EDI: 00000000 EBP: de9c9e24 ESP: de9c9e14
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process stapio (pid: 1817, ti=de9c8000 task=dd3225e0 task.ti=de9c8000)
Stack:
 de9c9e30 de98e400 de858a20 00000000 de9c9e3c e1c682e1 d398e49c de98e400
 00000001 00000000 de9c9e50 e1c68c09 de997880 de98e400 e1d6bdc4 de9c9e88
 e1c69eb8 deba5a60 019c9e74 e1d4b39b 00000000 e1d60f34 dd326500 e1d60f34
Call Trace:
[<e1c682e1>] ? uprobe_free_process+0x74/0xa3 [uprobes]
[<e1c68c09>] ? uprobe_put_process+0x54/0x78 [uprobes]
[<e1c69eb8>] ? __unregister_uprobe+0x1cf/0x1d7 [uprobes]
[<e1d4b39b>] ? stap_utrace_detach+0x3a/0xdd
[stap_b18709e7777f5130d8ad9fd893df16a6_1453]
[<e1c69ef6>] ? unregister_uprobe+0x12/0x14 [uprobes]
[<e1d4cab9>] ? _stp_cleanup_and_exit+0x87/0x178
[stap_b18709e7777f5130d8ad9fd893df16a6_1453]
[<e1d4d1fd>] ? _stp_ctl_write_cmd+0x620/0x768
[stap_b18709e7777f5130d8ad9fd893df16a6_1453]
[<c045e500>] ? finish_resume_report+0x86/0x8b
[<c045e5cb>] ? utrace_resume+0xc6/0xce
[<c04038c6>] ? do_notify_resume+0x60c/0x62d
[<c043a2b2>] ? recalc_sigpending+0x1e/0x90
[<c043ad92>] ? sys_rt_sigtimedwait+0x1d7/0x235
[<c052e0bf>] ? security_file_permission+0x14/0x16
[<c04a08ab>] ? rw_verify_area+0x9a/0xbc
[<e1d4cbdd>] ? _stp_ctl_write_cmd+0x0/0x768
[stap_b18709e7777f5130d8ad9fd893df16a6_1453]
[<c04a0f7c>] ? vfs_write+0x95/0xf4
[<c04a1097>] ? sys_write+0x4c/0x70
[<c0403f72>] ? syscall_call+0x7/0xb
Code: 88 d0 8b 53 18 85 d2 74 26 64 8b 0d 00 f0 94 c0 39 4b 10 75 04 84 c0 75 16
8b 43 14 b9 06 00 00 00 e8 97 ff ff ff 83 f8 8d 75 04 <0f> 0b eb fe 8b 43 14 e8
d1 65 7d de b8 c4 ce c6 e1 e8 db f9 a9 
EIP: [<e1c681cc>] uprobe_free_task+0x3b/0xdc [uprobes] SS:ESP 0068:de9c9e14
Kernel panic - not syncing: Fatal exception

-- 
           Summary: stap uprobe script on rawhide causes system crash
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: uprobes
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: ananth at in dot ibm dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10185

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/10185] stap uprobe script on rawhide causes system crash

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2009-05-27 20:12 -------
The uprobes2 code in question reads thusly:

static void uprobe_free_task(struct uprobe_task *utask, bool in_callback)
{
        [...]
        if (utask->engine && (utask->tsk != current || !in_callback)) {
                /*
                 * No other tasks in this process should be running
                 * uprobe_report_* callbacks.  (If they are, utrace_barrier()
                 * here could deadlock.)
                 */
                int result = utrace_control_pid(utask->pid, utask->engine,
                                                                UTRACE_DETACH);
                        BUG_ON(result == -EINPROGRESS);
        }
        [...]



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10185

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/10185] stap uprobe script on rawhide causes system crash

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2009-05-28 01:10 -------
Here is a simple reproducer on 2.6.29.3-140.fc11:
  stap -ve 'probe process("/lib/libc.so.6").function("raise"), begin { exit() }'

That crashes every time on i686.PAE.  With s|/lib/|/lib64/| it also crashes on
x86_64.

I tried it quickly on 2.6.27.21-170.2.56.fc10.i686 and it was fine there.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10185

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/10185] stap uprobe script on rawhide causes system crash

[jkenisto at us dot ibm dot com]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1974 bytes --]


------- Additional Comments From jkenisto at us dot ibm dot com  2009-05-29 00:04 -------
(In reply to comment #1)
> The uprobes2 code in question reads thusly:
> 
> static void uprobe_free_task(struct uprobe_task *utask, bool in_callback)
> {
>         [...]
>         if (utask->engine && (utask->tsk != current || !in_callback)) {
>                 /*
>                  * No other tasks in this process should be running
>                  * uprobe_report_* callbacks.  (If they are, utrace_barrier()
>                  * here could deadlock.)
>                  */
>                 int result = utrace_control_pid(utask->pid, utask->engine,
>                                                                 UTRACE_DETACH);
>                         BUG_ON(result == -EINPROGRESS);
>         }
>         [...]
> 
> 

As I mentioned in 5/13 email to Srikar (which covered several topics), I think
that this is one place where we really do want to wait for all the other threads
to finish running their callbacks.  As mentioned, calling utrace_barrier() while
holding uproc->rwsem could deadlock.  But I think that something like the
following would work:
- In uproc_free_process():
	- While holding uproc->rwsem, set a flag in uproc to indicate that we're
shutting down uproc (because it has no more probes – that's the only reason a
task would call uprobe_free_task() on a different task).  The shutting-down flag
means that no callbacks can make changes to uproc.
	- Unlock uproc->rwsem.
- In uprobe_free_task(), call utrace_control_pid(..., UTRACE_DETACH) and (if
necessary) utrace_barrier_pid() for the thread.
- Each uprobe_report_* callback would need to check uproc's shutting-down flag
after locking uproc->rwsem, and if uproc is shutting down, just unlock
uproc->rwsem and return.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10185

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/10185] stap uprobe script on rawhide causes system crash

[srikar at linux dot vnet dot ibm dot com]

------- Additional Comments From srikar at linux dot vnet dot ibm dot com  2009-05-30 16:42 -------
(In reply to comment #2)
> Here is a simple reproducer on 2.6.29.3-140.fc11:
>   stap -ve 'probe process("/lib/libc.so.6").function("raise"), begin { exit() }'
> 
> That crashes every time on i686.PAE.  With s|/lib/|/lib64/| it also crashes on
> x86_64.
> 
> I tried it quickly on 2.6.27.21-170.2.56.fc10.i686 and it was fine there.

I tried roland's tree with commit id ea301763ee4b3ddbe853cb17c5cf4127babe603e
(corresponds to 2.6.29-rc2 with latest systemtap and that doesn't exhibit the
problem. 


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|systemtap at sources dot    |srikar at linux dot vnet dot
                   |redhat dot com              |ibm dot com
             Status|NEW                         |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=10185

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.