[Bug runtime/6964] New: process probes cause kernel crash on f9

[Bug runtime/6964] New: process probes cause kernel crash on f9

[wcohen at redhat dot com]

The current snapshot of systemtap on f9 x86_64 machine running 2.6.26.5-45.fc9
x86_64 kernel can be crashed with the following short script:

stap -e 'probe process.syscall, process.end \
{printf("%s %d %s\n", execname(), pid(), pp())}' -c ls

This was collected a qemu-kvm instance. The following is the oops message:

kernel BUG at kernel/utrace.c:1999!
invalid opcode: 0000 [1] SMP 
CPU 0 
Modules linked in: stap_1b6ca5628c7bafdf5c3aaa4738cb75df_621 bridge bnep rfcomm
l2cap bluetooth fuse sunrpc ipt_REJECT nf_conntrack_ipv4 iptable_filter
ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack
ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath ppdev parport_pc
parport floppy pcspkr 8139too 8139cp i2c_piix4 mii i2c_core sr_mod cdrom sg
dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_generic ata_piix pata_acpi
libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last
unloaded: freq_table]
Pid: 2844, comm: ls Not tainted 2.6.26.5-45.fc9.x86_64 #1
RIP: 0010:[<ffffffff8106a14e>]  [<ffffffff8106a14e>] utrace_get_signal+0x470/0x4f2
RSP: 0018:ffff81002c6e1ce8  EFLAGS: 00010093
RAX: 0000000000000000 RBX: 0000000000000010 RCX: ffff81003850a300
RDX: ffff81002c6e1fd8 RSI: ffff81002c6e1d48 RDI: ffff81003850a328
RBP: ffff81002c6e1d88 R08: ffff81002c6e0000 R09: 00000000ffffffff
R10: 000000064d4111f0 R11: ffff81002c6e1b58 R12: 0000000000000000
R13: 0000000000001000 R14: ffff81003f9f70f0 R15: ffff8100375fad40
FS:  00007f72ed5c06f0(0000) GS:ffffffff81417000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000626450 CR3: 000000002c58d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ls (pid: 2844, threadinfo ffff81002c6e0000, task ffff8100375fad40)
Stack:  ffff81002c6e1ef8 ffff8100375fad40 ffff81002c6e1d38 0000000000001001
 ffff81002c6e1ef8 ffff81002c6e1e78 ffff81002c6e1f58 ffff81003f9f7100
 ffff8100385a3a48 ffff8100374256c0 0000000000000000 ffffffff81010000
Call Trace:
 [<ffffffff81010000>] ? disable_8259A_irq+0x14/0x50
 [<ffffffff810e4863>] ? load_elf_binary+0x1583/0x16f8
 [<ffffffff81043625>] get_signal_to_deliver+0x119/0x2cf
 [<ffffffff8104212f>] ? send_sig_info+0x69/0x78
 [<ffffffff8100b363>] do_notify_resume+0x90/0x90c
 [<ffffffff81071f22>] ? audit_syscall_exit+0x331/0x353
 [<ffffffff8100c367>] int_signal+0x12/0x17


Code: c7 45 c0 ff fe ff ff 48 81 c7 08 08 00 00 e8 91 27 23 00 48 8b 55 88 48 8d
75 c0 4c 89 ff e8 ac 92 fd ff 83 f8 09 41 89 c4 74 04 <0f> 0b eb fe 49 8b 87 60
05 00 00 b9 08 00 00 00 48 8b 7d 80 48 
RIP  [<ffffffff8106a14e>] utrace_get_signal+0x470/0x4f2
 RSP <ffff81002c6e1ce8>
---[ end trace 7ff14a49de87c977 ]---
BUG: sleeping function called from invalid context at kernel/rwsem.c:21
in_atomic():0, irqs_disabled():1
Pid: 2844, comm: ls Tainted: G      D   2.6.26.5-45.fc9.x86_64 #1

Call Trace:
 [<ffffffff8102dce1>] __might_sleep+0xd4/0xd8
 [<ffffffff8129b9ee>] down_read+0x1d/0x2e
 [<ffffffff8105faec>] acct_collect+0x42/0x1a6
 [<ffffffff8103a487>] do_exit+0x215/0x84c
 [<ffffffff8106a123>] ? utrace_get_signal+0x445/0x4f2
 [<ffffffff811a250a>] ? do_unblank_screen+0x2e/0x12a
 [<ffffffff8129d2cc>] oops_begin+0x0/0xa0
 [<ffffffff8100dde6>] die+0x5d/0x66
 [<ffffffff8129d82a>] do_trap+0x110/0x11f
 [<ffffffff8100e521>] do_invalid_op+0xa0/0xa9
 [<ffffffff8106a14e>] ? utrace_get_signal+0x470/0x4f2
 [<ffffffff8129aeed>] ? thread_return+0x6a/0xc2
 [<ffffffff8129cc79>] error_exit+0x0/0x60
 [<ffffffff8106a14e>] ? utrace_get_signal+0x470/0x4f2
 [<ffffffff8106a146>] ? utrace_get_signal+0x468/0x4f2
 [<ffffffff81010000>] ? disable_8259A_irq+0x14/0x50
 [<ffffffff810e4863>] ? load_elf_binary+0x1583/0x16f8
 [<ffffffff81043625>] ? get_signal_to_deliver+0x119/0x2cf
 [<ffffffff8104212f>] ? send_sig_info+0x69/0x78
 [<ffffffff8100b363>] ? do_notify_resume+0x90/0x90c
 [<ffffffff81071f22>] ? audit_syscall_exit+0x331/0x353
 [<ffffffff8100c367>] ? int_signal+0x12/0x17

-- 
           Summary: process probes cause kernel crash on f9
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: runtime
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: wcohen at redhat dot com
                CC: eteo at redhat dot com
  GCC host triplet: x86_64


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[eteo at redhat dot com]

------- Additional Comments From eteo at redhat dot com  2008-10-16 17:42 -------
(In reply to comment #0)
> The current snapshot of systemtap on f9 x86_64 machine running 2.6.26.5-45.fc9
> x86_64 kernel can be crashed with the following short script:

I'm able to reproduce this on a x86 host as well.

Eugene

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2008-10-16 19:20 -------
Removing the "-c ls" will allow this to work:

stap -e 'probe process.syscall, process.end \
{printf("%s %d %s\n", execname(), pid(), pp())}'


The problem is still present on the 2.6.26.6-75.fc9 kernel.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2008-10-16 21:51 -------
Did some additional experiments to gather more information about what is going on.

Suggested that things run with "strace -f". The machine did not crash when the
command was prepended with this command.

Found that crashes still occurred with "-c ls" for:

probe process.syscall {}

and:

probe process.end {}



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[roland at gnu dot org]

------- Additional Comments From roland at gnu dot org  2008-10-17 20:05 -------
Please show the crash details for a current kernel (f9 or f10).
Ideal would be to reduce this to an example utrace module that demonstrates the
bug without using stap.  If it's triggered by ptrace, then a simple test program
using ptrace alone (for the ptrace-tests suite).

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |roland at redhat dot com
             Status|NEW                         |WAITING


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2008-10-17 20:58 -------
Created an attachment (id=3004)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=3004&action=view)
Kernel oops message for 2.6.26.6-75.fc9.x86_64 kernel


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2008-10-17 21:33 -------
Compiled the instrumentation one line 'probe process.end {}' separately into
module then ran with:

staprun -vv 6964f.ko -c ls

This caused a crash right after the following line:

stapio:stp_main_loop:372 detaching pid 30260

Looks like things are going wrong after when the following line from
systemtap/runtime/staprun/mainloop.c is attempted:

          int rc = ptrace (PTRACE_DETACH, target_pid, 0, 0);


Is there a ptrace tests that register utrace engines similar to what systemtap
generates in the instrumentation module?



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[fche at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW
   Last reconfirmed|0000-00-00 00:00:00         |2008-10-18 05:57:44
               date|                            |


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-10-18 18:28 -------
Reported "upstream": https://bugzilla.redhat.com/show_bug.cgi?id=467568


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |SUSPENDED


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wenji dot huang at oracle dot com]

------- Additional Comments From wenji dot huang at oracle dot com  2008-10-20 02:05 -------
(In reply to comment #1)
Reproduced it on 2.6.27+latest utrace patch, both x86 and x86_64.




-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wenji dot huang at oracle dot com]

------- Additional Comments From wenji dot huang at oracle dot com  2008-10-20 06:51 -------
Related code in utrace_get_signal:

  if (unlikely(report.killed)) {
                /* COMMENT */
		sigset_t sigkill_only;
		siginitsetinv(&sigkill_only, sigmask(SIGKILL));
		spin_lock_irq(&task->sighand->siglock);
		signr = dequeue_signal(task, &sigkill_only, info);
		BUG_ON(signr != SIGKILL);
		*return_ka = task->sighand->action[SIGKILL - 1];
		return signr;
  }

The BUG_ON is triggered and I debug the signr value which is equal to 0, not
SIGKILL.




-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wenji dot huang at oracle dot com]

------- Additional Comments From wenji dot huang at oracle dot com  2008-10-21 03:09 -------
Did some tests to narrow the scope.
It works fine for:
   stap -vve 'probe kernel.function("sys_open"){}' -c ls
   stap -vve 'probe process("./test").syscall {}' -c ls

only got crashed when
   stap -vve 'probe process.syscall {}' -c ls

I guess the reason is the spawn process by stap is being hitted. Some signals
are delayed/messed so that BUG_ON in utrace_get_signal is triggered.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[jan dot kratochvil at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jan dot kratochvil at redhat
                   |                            |dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[wenji dot huang at oracle dot com]

------- Additional Comments From wenji dot huang at oracle dot com  2008-10-22 09:08 -------
Created an attachment (id=3015)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=3015&action=view)
workaround for the bug

This patch temporarily disables the PTRACE_TRACEME/DETACH when executing
command. The command will be running once child forked.  No kernel panic any
more.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-10-29 02:52 -------
> This patch temporarily disables the PTRACE_TRACEME/DETACH when executing
> command.

Thanks, committed, for I hope not too long :-(


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[mjw at redhat dot com]

------- Additional Comments From mjw at redhat dot com  2008-11-03 08:52 -------
This workaround is causing some confusion for people (me included) who use
systemtap git trunk since -c doesn't work reliably now. Is there a timeline to
get this fixed for real?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[mjw at redhat dot com]

------- Additional Comments From mjw at redhat dot com  2008-11-11 21:35 -------
Currently make installcheck will not really work because the systemtap.syscall
tests all use -c. It often just hangs with this workaround applied.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug runtime/6964] process probes cause kernel crash on f9

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-11-11 22:05 -------
I'll fix the -c synchronization logic.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|systemtap at sources dot    |fche at redhat dot com
                   |redhat dot com              |
             Status|SUSPENDED                   |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=6964

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.