* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
@ 2005-12-20 9:40 ` anil dot s dot keshavamurthy at intel dot com
2005-12-21 18:17 ` hien at us dot ibm dot com
` (24 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: anil dot s dot keshavamurthy at intel dot com @ 2005-12-20 9:40 UTC (permalink / raw)
To: systemtap
------- Additional Comments From anil dot s dot keshavamurthy at intel dot com 2005-12-20 07:34 -------
Created an attachment (id=808)
--> (http://sourceware.org/bugzilla/attachment.cgi?id=808&action=view)
test modules
Attaching a test case.
1) Untar
2)cd reent_test; make
3) ./please_load_me
4) do some make -jx
Should see a system crash in few minutes.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
2005-12-20 9:40 ` [Bug kprobes/2071] " anil dot s dot keshavamurthy at intel dot com
@ 2005-12-21 18:17 ` hien at us dot ibm dot com
2005-12-22 2:26 ` hien at us dot ibm dot com
` (23 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: hien at us dot ibm dot com @ 2005-12-21 18:17 UTC (permalink / raw)
To: systemtap
------- Additional Comments From hien at us dot ibm dot com 2005-12-21 18:11 -------
Anil, I run your test on ppc64, the system gave an oops after building the
kernel with make -j8 for a while. Here's the trace
kernel BUG in do_exit at kernel/exit.c:880!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=128 NUMA PSERIES LPAR
Modules linked in: mon_sched mon_sched_4 mon_sched_3 mon_sched_2 mon_sched_1
mon_irq mon_irq_4 mon_irq_3 mon_irq_2 mon_irq_1 mon_reent mon_dummy ipv6
parport_pc lp parport sg autofs4 binfmt_misc dm_multipath dm_mod pdc202xx_new
e1000 ipr firmware_class sd_mod scsi_mod
NIP: C000000000059694 LR: C00000000002C9F8 CTR: C00000000004CFCC
REGS: c000000038847a40 TRAP: 0700 Not tainted (2.6.15-rc5cel)
MSR: 8000000000029032 <EE,ME,IR,DR> CR: 24004422 XER: 00000010
TASK = c000000041c447e0[32713] 'as' THREAD: c000000038844000 CPU: 1
GPR00: 0000000000000000 C000000038847CC0 C0000000005BF790 0000000000000000
GPR04: 8000000000001032 C0000000417FDC80 00000000283B9E5D C000000001F34DA0
GPR08: 0000000000000000 0000000000000004 C000000038847D30 C0000000005BF790
GPR12: 0000000024004482 C00000000048D400 00000000100F39F8 0000000010030000
GPR16: 0000000010030000 0000000010020000 00000000FFFF9008 0000000010050000
GPR20: 0000000010050000 0000000000000002 C000000041C44908 C000000038847D30
GPR24: C000000041C44970 C000000041C44890 C000000041C44890 C0000000028C47E0
GPR28: 0000000000000010 C000000041C447E0 C0000000004FCBB0 C000000038847D30
NIP [C000000000059694] .do_exit+0xa9c/0xda4
LR [C00000000002C9F8] kretprobe_trampoline+0x0/0x8
Call Trace:
[C000000038847CC0] [C00000000002C9F8] kretprobe_trampoline+0x0/0x8 (unreliable)
[C000000038847D90] [C000000000059A2C] .do_group_exit+0x50/0xe4
[C000000038847E30] [C000000000008600] syscall_exit+0x0/0x18
Instruction dump:
480586a1 60000000 e81d0018 39200000 f93d0788 70000008 0b000000 e93d0018
61290008 f93d0018 48348ea1 60000000 <0fe00000> 48000000 39200001 4bfffda8
<1>Fixing recursive fault but reboot is needed!
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
2005-12-20 9:40 ` [Bug kprobes/2071] " anil dot s dot keshavamurthy at intel dot com
2005-12-21 18:17 ` hien at us dot ibm dot com
@ 2005-12-22 2:26 ` hien at us dot ibm dot com
2005-12-27 3:41 ` bibo dot mao at intel dot com
` (22 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: hien at us dot ibm dot com @ 2005-12-22 2:26 UTC (permalink / raw)
To: systemtap
------- Additional Comments From hien at us dot ibm dot com 2005-12-22 01:17 -------
On ppc64, I tried to disable the interrupt in the kprobe handler in the case of
reentry and re-enable interrupt when it came out of the handler and it seems to
*WORK*. I was able to complete my kernel build (make -j8), where it gave an oops
before.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (2 preceding siblings ...)
2005-12-22 2:26 ` hien at us dot ibm dot com
@ 2005-12-27 3:41 ` bibo dot mao at intel dot com
2005-12-28 6:13 ` bibo dot mao at intel dot com
` (21 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: bibo dot mao at intel dot com @ 2005-12-27 3:41 UTC (permalink / raw)
To: systemtap
------- Additional Comments From bibo dot mao at intel dot com 2005-12-26 09:15 -------
I test in EM64T in linux 2.6.9 with RCU patch, it does not crash. But when I
test it in linux 2.6.15-RC5-mm3, it crashed.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (3 preceding siblings ...)
2005-12-27 3:41 ` bibo dot mao at intel dot com
@ 2005-12-28 6:13 ` bibo dot mao at intel dot com
2005-12-30 9:44 ` bibo dot mao at intel dot com
` (20 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: bibo dot mao at intel dot com @ 2005-12-28 6:13 UTC (permalink / raw)
To: systemtap
------- Additional Comments From bibo dot mao at intel dot com 2005-12-27 03:41 -------
In linux 2.6.15-RC5-mm3, it does not crash in IA32, when running in EM64T, I
find that when there is int3 instruction in the first int3 hanlder function,
the first int 3 handler function can continue to execute, but when this
funciton return, system will crash.
For example, when kp_pre() in mon_sched.c calls
my_test_reentrant_export_function function which has been probed,
my_test_reentrant_export_function can continue to execute,but when it returned
system crashed.
And I think maybe it is the problem of trap stack in EM64T, I do not know how
trap stack is established when trap happens in use/kenrel mode or trap
executing mode.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (4 preceding siblings ...)
2005-12-28 6:13 ` bibo dot mao at intel dot com
@ 2005-12-30 9:44 ` bibo dot mao at intel dot com
2006-01-13 2:08 ` anil dot s dot keshavamurthy at intel dot com
` (19 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: bibo dot mao at intel dot com @ 2005-12-30 9:44 UTC (permalink / raw)
To: systemtap
------- Additional Comments From bibo dot mao at intel dot com 2005-12-29 08:53 -------
I think this problem only happens in 2.6.15-rc5-mm3, there is one patch
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.15-
rc5/2.6.15-rc5-mm3/broken-out/x86_64-debug-stack.patch, this patch changed int3
handler stack, debug and int3 share the same stack(DEBUG_STACK), this stack is
saved in TSS structure, (64-Bit Extension Technology Software Developer's Guide
Volume 1 of 2, section 1.6.10.5).
With this patch, every time debug/int3 exception happens, it will changed to
DEBUG_STACK, so when reentrancy int3 happens, the later int3 handler will
overwrite pervious DEBUG_STACK. So system crashed.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (5 preceding siblings ...)
2005-12-30 9:44 ` bibo dot mao at intel dot com
@ 2006-01-13 2:08 ` anil dot s dot keshavamurthy at intel dot com
2006-01-13 2:14 ` anil dot s dot keshavamurthy at intel dot com
` (18 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: anil dot s dot keshavamurthy at intel dot com @ 2006-01-13 2:08 UTC (permalink / raw)
To: systemtap
------- Additional Comments From anil dot s dot keshavamurthy at intel dot com 2006-01-13 02:08 -------
(In reply to comment #3)
> On ppc64, I tried to disable the interrupt in the kprobe handler in the case
of
> reentry and re-enable interrupt when it came out of the handler and it seems
to
> *WORK*. I was able to complete my kernel build (make -j8), where it gave an
oops
> before.
Hien, I tried your patch (porting onto IA64) and it did not work for me. Also
I see the solution you have mentioned might now work for x86_64 too.
Especially on x86_64 even when you disable interrupt, NMI's can still happen
and any probes on that path will cause the problem again.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (6 preceding siblings ...)
2006-01-13 2:08 ` anil dot s dot keshavamurthy at intel dot com
@ 2006-01-13 2:14 ` anil dot s dot keshavamurthy at intel dot com
2006-01-13 2:15 ` anil dot s dot keshavamurthy at intel dot com
` (17 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: anil dot s dot keshavamurthy at intel dot com @ 2006-01-13 2:14 UTC (permalink / raw)
To: systemtap
------- Additional Comments From anil dot s dot keshavamurthy at intel dot com 2006-01-13 02:14 -------
Created an attachment (id=827)
--> (http://sourceware.org/bugzilla/attachment.cgi?id=827&action=view)
[PATCH] please review and provide comments
I am attaching a patch which has worked for me on IA64. Can someone port the
same onto PPC64 & x86_64 and give it a try. Porting this onto other
architecture should be very easy. Also I am looking for review comments on the
patch itself.
Thanks,
Anil
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (7 preceding siblings ...)
2006-01-13 2:14 ` anil dot s dot keshavamurthy at intel dot com
@ 2006-01-13 2:15 ` anil dot s dot keshavamurthy at intel dot com
2006-01-13 4:42 ` bibo dot mao at intel dot com
` (16 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: anil dot s dot keshavamurthy at intel dot com @ 2006-01-13 2:15 UTC (permalink / raw)
To: systemtap
--
What |Removed |Added
----------------------------------------------------------------------------
CC| |jkenisto at us dot ibm dot
| |com
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (8 preceding siblings ...)
2006-01-13 2:15 ` anil dot s dot keshavamurthy at intel dot com
@ 2006-01-13 4:42 ` bibo dot mao at intel dot com
2006-01-13 17:37 ` hien at us dot ibm dot com
` (15 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: bibo dot mao at intel dot com @ 2006-01-13 4:42 UTC (permalink / raw)
To: systemtap
------- Additional Comments From bibo dot mao at intel dot com 2006-01-13 04:42 -------
(In reply to comment #7)
> Hien, I tried your patch (porting onto IA64) and it did not work for me. Also
> I see the solution you have mentioned might now work for x86_64 too.
> Especially on x86_64 even when you disable interrupt, NMI's can still happen
> and any probes on that path will cause the problem again.
In x86_86 linux 2.6.15-git8, when INT3 trap happens recursively system will
crash, but in x86_64 Linux v2.6.15, system will not crash. If you change
arch/x86_64/kernel/traps.c:969 :set_system_gate_ist(3,&int3,DEBUG_STACK) as
set_system_gate(3,&int3), system will not crash.
Current in IA32 and X86_64, INT3 vector makes use of GATE_INTERRUPT, when trap
happens hardware will clear interrupt flag automatically. So I think this bug
is architecture relative, in IA64 it actually crashed.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (9 preceding siblings ...)
2006-01-13 4:42 ` bibo dot mao at intel dot com
@ 2006-01-13 17:37 ` hien at us dot ibm dot com
2006-01-13 19:23 ` anil dot s dot keshavamurthy at intel dot com
` (14 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: hien at us dot ibm dot com @ 2006-01-13 17:37 UTC (permalink / raw)
To: systemtap
------- Additional Comments From hien at us dot ibm dot com 2006-01-13 17:37 -------
(In reply to comment #8)
> Created an attachment (id=827)
--> (http://sourceware.org/bugzilla/attachment.cgi?id=827&action=view)
> [PATCH] please review and provide comments
>
> I am attaching a patch which has worked for me on IA64. Can someone port the
> same onto PPC64 & x86_64 and give it a try. Porting this onto other
> architecture should be very easy. Also I am looking for review comments on the
> patch itself.
>
Anil,
Do you still disable IRQ in the handler in the case of re-entrance with this patch?
I could test this patch with ppc64 after you verify this. Let me know.
Thanks, Hien.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (10 preceding siblings ...)
2006-01-13 17:37 ` hien at us dot ibm dot com
@ 2006-01-13 19:23 ` anil dot s dot keshavamurthy at intel dot com
2006-01-13 23:00 ` hien at us dot ibm dot com
` (13 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: anil dot s dot keshavamurthy at intel dot com @ 2006-01-13 19:23 UTC (permalink / raw)
To: systemtap
------- Additional Comments From anil dot s dot keshavamurthy at intel dot com 2006-01-13 19:23 -------
> Anil,
> Do you still disable IRQ in the handler in the case of re-entrance with this
patch?
No, you don;t have to disable IRQ with my new approach which has worked on
IA64.
> I could test this patch with ppc64 after you verify this. Let me know.
You have to port the patch to ppc64 and test it.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (11 preceding siblings ...)
2006-01-13 19:23 ` anil dot s dot keshavamurthy at intel dot com
@ 2006-01-13 23:00 ` hien at us dot ibm dot com
2006-01-14 0:55 ` hien at us dot ibm dot com
` (12 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: hien at us dot ibm dot com @ 2006-01-13 23:00 UTC (permalink / raw)
To: systemtap
------- Additional Comments From hien at us dot ibm dot com 2006-01-13 23:00 -------
Anil,
It works for ppc64. I ported your patch to ppc64 and run the test and build the
kernel (make -j8). Kernel build completed no crash.
I am going to look for a x86_64 box and try the same patch on that platform.
Hien.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (12 preceding siblings ...)
2006-01-13 23:00 ` hien at us dot ibm dot com
@ 2006-01-14 0:55 ` hien at us dot ibm dot com
2006-01-16 21:14 ` jkenisto at us dot ibm dot com
` (11 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: hien at us dot ibm dot com @ 2006-01-14 0:55 UTC (permalink / raw)
To: systemtap
------- Additional Comments From hien at us dot ibm dot com 2006-01-14 00:55 -------
Anil,
It works on x86_64 too. Ported and tested with kernel v2.6.15. I've just done my
kernel build (make -j4) with the test running on x86_64. Great job.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (13 preceding siblings ...)
2006-01-14 0:55 ` hien at us dot ibm dot com
@ 2006-01-16 21:14 ` jkenisto at us dot ibm dot com
2006-12-11 19:34 ` fche at redhat dot com
` (10 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: jkenisto at us dot ibm dot com @ 2006-01-16 21:14 UTC (permalink / raw)
To: systemtap
------- Additional Comments From jkenisto at us dot ibm dot com 2006-01-16 21:14 -------
Assigning this to Anil, since he's been coordinating the work on this.
--
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|systemtap at sources dot |anil dot s dot keshavamurthy
|redhat dot com |at intel dot com
Status|NEW |ASSIGNED
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (14 preceding siblings ...)
2006-01-16 21:14 ` jkenisto at us dot ibm dot com
@ 2006-12-11 19:34 ` fche at redhat dot com
2007-06-11 12:07 ` srikar at linux dot vnet dot ibm dot com
` (9 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: fche at redhat dot com @ 2006-12-11 19:34 UTC (permalink / raw)
To: systemtap
------- Additional Comments From fche at redhat dot com 2006-12-11 18:30 -------
This patch has *amazing* results. All my private kernel.function("*") tests are
now passing (on i686; testing on x86-64 in progress). Please let's push it
upstream immediately.
--
What |Removed |Added
----------------------------------------------------------------------------
CC| |systemtap at sources dot
| |redhat dot com
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (15 preceding siblings ...)
2006-12-11 19:34 ` fche at redhat dot com
@ 2007-06-11 12:07 ` srikar at linux dot vnet dot ibm dot com
2007-06-11 16:14 ` jkenisto at us dot ibm dot com
` (8 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: srikar at linux dot vnet dot ibm dot com @ 2007-06-11 12:07 UTC (permalink / raw)
To: systemtap
------- Additional Comments From srikar at linux dot vnet dot ibm dot com 2007-06-11 12:07 -------
Anil,
I ran your test on ppc64, on 2.6.21-rc6-mm1 for around 3 days. However I
couldn't reproduce the problem. Hence I don't think that the problem affects ppc64.
Please do push your patch and close the bug. If ever the problem is seen on
ppc64 later we can reopen this bug.
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (16 preceding siblings ...)
2007-06-11 12:07 ` srikar at linux dot vnet dot ibm dot com
@ 2007-06-11 16:14 ` jkenisto at us dot ibm dot com
2007-06-11 16:43 ` anil dot s dot keshavamurthy at intel dot com
` (7 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: jkenisto at us dot ibm dot com @ 2007-06-11 16:14 UTC (permalink / raw)
To: systemtap
------- Additional Comments From jkenisto at us dot ibm dot com 2007-06-11 16:14 -------
Subject: Re: Probes on ISR with probes on task thread's
prehandler crash the system
On Mon, 2007-06-11 at 12:07 +0000, srikar at linux dot vnet dot ibm dot
com wrote:
> ------- Additional Comments From srikar at linux dot vnet dot ibm dot com 2007-06-11 12:07 -------
> Anil,
>
> I ran your test on ppc64, on 2.6.21-rc6-mm1 for around 3 days. However I
> couldn't reproduce the problem. Hence I don't think that the problem affects ppc64.
> Please do push your patch and close the bug. If ever the problem is seen on
> ppc64 later we can reopen this bug.
>
Thanks, Srikar.
Jim
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (17 preceding siblings ...)
2007-06-11 16:14 ` jkenisto at us dot ibm dot com
@ 2007-06-11 16:43 ` anil dot s dot keshavamurthy at intel dot com
2008-01-17 14:54 ` mhiramat at redhat dot com
` (6 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: anil dot s dot keshavamurthy at intel dot com @ 2007-06-11 16:43 UTC (permalink / raw)
To: systemtap
------- Additional Comments From anil dot s dot keshavamurthy at intel dot com 2007-06-11 16:43 -------
(In reply to comment #21)
> Subject: Re: Probes on ISR with probes on task thread's
> prehandler crash the system
> On Mon, 2007-06-11 at 12:07 +0000, srikar at linux dot vnet dot ibm dot
> com wrote:
> > ------- Additional Comments From srikar at linux dot vnet dot ibm dot com
2007-06-11 12:07 -------
> > Anil,
> >
> > I ran your test on ppc64, on 2.6.21-rc6-mm1 for around 3 days. However I
> > couldn't reproduce the problem. Hence I don't think that the problem
affects ppc64.
> > Please do push your patch and close the bug. If ever the problem is seen on
> > ppc64 later we can reopen this bug.
> >
Already pushed this for Ia64 and it is has made it to Linus's kernel.
Will close this bug as we have covered for all architecture.
-Anil
--
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution| |FIXED
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (18 preceding siblings ...)
2007-06-11 16:43 ` anil dot s dot keshavamurthy at intel dot com
@ 2008-01-17 14:54 ` mhiramat at redhat dot com
2008-01-17 21:56 ` mhiramat at redhat dot com
` (5 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: mhiramat at redhat dot com @ 2008-01-17 14:54 UTC (permalink / raw)
To: systemtap
------- Additional Comments From mhiramat at redhat dot com 2008-01-17 14:53 -------
I found this bug still exist in the latest kernel.
I'm investing this bug.
Here is the kernel bug message:
------
probes registered
probes registered
probes registered
probes registered
probes registered
probes registered
probes registered
probes registered
probes registered
probes registered
probes registered
kernel BUG at /home/mhiramat/ksrc/linux-2.6.24-rc7/kernel/exit.c:1050!
mv[32200]: bugcheck! 0 [1]
Modules linked in: mon_sched mon_sched_4 mon_sched_3 mon_sched_2 mon_sched_1
mon_irq mon_irq_4 mon_irq_3 mon_irq_2 mon_irq_1 mon_reent mon_dummy sunrpc
binfmt_misc dm_multipath fan sg thermal processor button container dm_snapshot
dm_zero dm_mirror dm_mod usb_storage megaraid_mbox megaraid_mm ehci_hcd ohci_hcd
uhci_hcd usbcore
Pid: 32200, CPU 1, comm: mv
psr : 0000101008526030 ifs : 800000000000040c ip : [<a00000010009def0>] Not
tainted (2.6.24-rc7)
ip is at do_exit+0x11b0/0x11c0
unat: 0000000000000000 pfs : 000000000000040c rsc : 0000000000000003
rnat: 0000000000000400 bsps: 0000000000000400 pr : 0000000000556959
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70033f
csd : 0000000000000000 ssd : 0000000000000000
b0 : a00000010009def0 b6 : a00000010008d300 b7 : a00000010000f9f0
f6 : 1003e000000000001c000 f7 : 1003e0000000000000400
f8 : 1003e0000000000000070 f9 : 0ffff8000000000000000
f10 : 10008fffffffff0000000 f11 : 1003e0000000000000400
r1 : a000000100e60c60 r2 : e00000071eab0024 r3 : a000000100c05348
r8 : 000000000000004a r9 : a000000100c05348 r10 : 0000000000000007
r11 : 0000000000004000 r12 : e0000005a639fe20 r13 : e0000005a6390000
r14 : 0000000000004000 r15 : a000000100c05348 r16 : a000000100c05330
r17 : 0000000000004000 r18 : 0000001516f675a7 r19 : e000000001129460
r20 : e000000001129460 r21 : e0000000011257d8 r22 : e000000707dea6d8
r23 : 0000001513fb8527 r24 : e000000707dea6c0 r25 : e0000000011257c0
r26 : a000000100c79e4c r27 : 0000000000000400 r28 : 0000000000000400
r29 : 0000000000001000 r30 : 0000000000000070 r31 : e00000071eab0048
Call Trace:
[<a000000100015340>] show_stack+0x40/0xa0
sp=e0000005a639f9f0 bsp=e0000005a6390e78
[<a000000100015c50>] show_regs+0x850/0x8a0
sp=e0000005a639fbc0 bsp=e0000005a6390e20
[<a000000100038d60>] die+0x1a0/0x2a0
sp=e0000005a639fbc0 bsp=e0000005a6390dd0
[<a000000100038eb0>] die_if_kernel+0x50/0x80
sp=e0000005a639fbc0 bsp=e0000005a6390da0
[<a0000001007627c0>] ia64_bad_break+0x240/0x440
sp=e0000005a639fbc0 bsp=e0000005a6390d78
[<a00000010000b9a0>] ia64_leave_kernel+0x0/0x270
sp=e0000005a639fc50 bsp=e0000005a6390d78
[<a00000010009def0>] do_exit+0x11b0/0x11c0
sp=e0000005a639fe20 bsp=e0000005a6390d18
[<a00000010009e050>] do_group_exit+0x150/0x160
sp=e0000005a639fe30 bsp=e0000005a6390ce0
[<a00000010009e080>] sys_exit_group+0x20/0x40
sp=e0000005a639fe30 bsp=e0000005a6390c88
[<a00000010000b800>] ia64_ret_from_syscall+0x0/0x20
sp=e0000005a639fe30 bsp=e0000005a6390c88
[<a000000000010720>] __kernel_syscall_via_break+0x0/0x20
sp=e0000005a63a0000 bsp=e0000005a6390c88
Fixing recursive fault but reboot is needed!
------
--
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|FIXED |
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (19 preceding siblings ...)
2008-01-17 14:54 ` mhiramat at redhat dot com
@ 2008-01-17 21:56 ` mhiramat at redhat dot com
2008-01-17 21:57 ` mhiramat at redhat dot com
` (4 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: mhiramat at redhat dot com @ 2008-01-17 21:56 UTC (permalink / raw)
To: systemtap
------- Additional Comments From mhiramat at redhat dot com 2008-01-17 21:56 -------
As the result of investigation, I found a bug in restore_previous_kprobe().
This function and save_previous_kprobe() do FILO(stack) operation.
These functions work as like below;
save_previous_kprobe() // this pushes a value to stack
{
i = ++index;
stack[i-1] = val;
}
restore_previous_kprobe() // this pops a value from stack
{
i = --index; // (a)
val = stack[i]; // (b)
}
However, if an interrupt occurs between (a) and (b), and a kprobe
is hit in that interrupt, this overwrites previous stack[] entry.
restore_previous_kprobe() // this pops a value from stack
{
i = --index; // (a) (i == 0, index == 0)
--(interrupt)
save_previous_kprobe() // this pushes a value to stack
{
i = ++index; (i == 1, index == 1)
stack[i-1] = val2; (!!overwrite stack[0]!!)
}
restore_previous_kprobe() // this pops a value from stack
{
i = --index; (i == 0, index == 0)
val2 = stack[i]; (stack[0] == val2)
}
--
val = stack[i]; // (b) (val = val2)
}
Thus, the index must be decremented AFTER reading the value.
restore_previous_kprobe() // this pops a value from stack
{
i = index;
val = stack[i-1];
--index;
}
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (20 preceding siblings ...)
2008-01-17 21:56 ` mhiramat at redhat dot com
@ 2008-01-17 21:57 ` mhiramat at redhat dot com
2008-01-17 21:58 ` mhiramat at redhat dot com
` (3 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: mhiramat at redhat dot com @ 2008-01-17 21:57 UTC (permalink / raw)
To: systemtap
--
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|anil dot s dot keshavamurthy|mhiramat at redhat dot com
|at intel dot com |
Status|REOPENED |ASSIGNED
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (21 preceding siblings ...)
2008-01-17 21:57 ` mhiramat at redhat dot com
@ 2008-01-17 21:58 ` mhiramat at redhat dot com
2008-01-18 3:30 ` shaohua dot li at intel dot com
` (2 subsequent siblings)
25 siblings, 0 replies; 27+ messages in thread
From: mhiramat at redhat dot com @ 2008-01-17 21:58 UTC (permalink / raw)
To: systemtap
------- Additional Comments From mhiramat at redhat dot com 2008-01-17 21:58 -------
Created an attachment (id=2203)
--> (http://sourceware.org/bugzilla/attachment.cgi?id=2203&action=view)
Fix the order of atomic operations in restore_previous_kprobe() on ia64
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (22 preceding siblings ...)
2008-01-17 21:58 ` mhiramat at redhat dot com
@ 2008-01-18 3:30 ` shaohua dot li at intel dot com
2008-01-21 22:47 ` mhiramat at redhat dot com
2008-03-13 15:00 ` mhiramat at redhat dot com
25 siblings, 0 replies; 27+ messages in thread
From: shaohua dot li at intel dot com @ 2008-01-18 3:30 UTC (permalink / raw)
To: systemtap
------- Additional Comments From shaohua dot li at intel dot com 2008-01-18 03:29 -------
re comments #25: patch looks good, please send to ia64 maillist.
--
What |Removed |Added
----------------------------------------------------------------------------
CC| |shaohua dot li at intel dot
| |com
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (23 preceding siblings ...)
2008-01-18 3:30 ` shaohua dot li at intel dot com
@ 2008-01-21 22:47 ` mhiramat at redhat dot com
2008-03-13 15:00 ` mhiramat at redhat dot com
25 siblings, 0 replies; 27+ messages in thread
From: mhiramat at redhat dot com @ 2008-01-21 22:47 UTC (permalink / raw)
To: systemtap
------- Additional Comments From mhiramat at redhat dot com 2008-01-21 22:46 -------
(In reply to comment #26)
> re comments #25: patch looks good, please send to ia64 maillist.
Thank you for review,
I sent this patch to linux-ia64 ml.
Here is the title:
[PATCH]Fix the order of atomic operations in restore_previous_kprobes on ia64
Could you give me your Ack on the ml?
--
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [Bug kprobes/2071] Probes on ISR with probes on task thread's prehandler crash the system
2005-12-20 7:34 [Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system anil dot s dot keshavamurthy at intel dot com
` (24 preceding siblings ...)
2008-01-21 22:47 ` mhiramat at redhat dot com
@ 2008-03-13 15:00 ` mhiramat at redhat dot com
25 siblings, 0 replies; 27+ messages in thread
From: mhiramat at redhat dot com @ 2008-03-13 15:00 UTC (permalink / raw)
To: systemtap
------- Additional Comments From mhiramat at redhat dot com 2008-03-13 14:59 -------
The patch was merged into linus tree(2.6.25-rc1).
--
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution| |FIXED
http://sourceware.org/bugzilla/show_bug.cgi?id=2071
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply [flat|nested] 27+ messages in thread