[Bug kprobes/5963] New: testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[Bug kprobes/5963] New: testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[wcohen at redhat dot com]

I have a vmware image on a uniprocessor rhel4 i686 machine set up with the
Fedora 9 rawhide image. When running the "make installcheck" on a cvs nightly
checkout following test causes the kernel to oops:

testsuite/systemtap.maps/pmap_agg_overflow.stp

Able to crash the machine with the following command line:

./stap -v  -DMAXERRORS=1 -g \
../src/testsuite/systemtap.maps/pmap_agg_overflow.stp

The test crashes very frequently, but it doesn't crash every time. Below is the
back trace from /var/log/messages:

BUG: unable to handle kernel paging request at e0ac61c0
IP: [<c063b7b4>] get_kprobe+0x2d/0x3c
Oops: 0000 [#1] SMP 
Modules linked in: rfcomm l2cap bluetooth autofs4 sunrpc ipt_REJECT xt_tcpudp
nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables ip6table_filter
ip6_tables x_tables ipv6 loop dm_multipath snd_ens1371 gameport snd_rawmidi
snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq
snd_seq_device parport_pc snd_pcm_oss parport snd_mixer_oss floppy snd_pcm
snd_timer pcspkr snd soundcore snd_page_alloc pcnet32 mii BusLogic i2c_piix4
i2c_core ac button sr_mod sg cdrom dm_snapshot dm_zero dm_mirror dm_mod ata_piix
ata_generic pata_acpi libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd
ehci_hcd [last unloaded: stap_968413d115d4668ca6effc95d048ae96_1280]

Pid: 2532, comm: staprun Not tainted (2.6.25-0.121.rc5.git4.fc9 #1)
EIP: 0060:[<c063b7b4>] EFLAGS: 00010086 CPU: 0
EIP is at get_kprobe+0x2d/0x3c
EAX: e0ac61c0 EBX: deb70c68 ECX: c0425bd0 EDX: 00f5d000
ESI: deb70d24 EDI: c1709164 EBP: deb70c70 ESP: deb70c68
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process staprun (pid: 2532, ti=deb70000 task=deb90000 task.ti=deb70000)
Stack: e0ac61c0 00000002 deb70c8c c063aa7f c0730714 00000097 c0731de4 ffffffff 
       00000000 deb70cac c063b700 deb70cec 00000002 00000000 c073032c deb70cec 
       00000002 deb70cd4 c063b756 ffffffff 00000000 00000002 00000001 c063b71f 
Call Trace:
 [<c063aa7f>] ? kprobe_exceptions_notify+0x70/0x442
 [<c063b700>] ? notifier_call_chain+0x2b/0x4a
 [<c063b756>] ? __atomic_notifier_call_chain+0x37/0x5a
 [<c063b71f>] ? __atomic_notifier_call_chain+0x0/0x5a
 [<c063b785>] ? atomic_notifier_call_chain+0xc/0xe
 [<c043d16b>] ? notify_die+0x2d/0x2f
 [<c063a290>] ? do_int3+0x3a/0x7d
 [<c0639cf3>] ? int3+0x27/0x2c
 [<c0425bd1>] ? scheduler_tick+0x1/0x284
 [<c043063a>] ? update_process_times+0x3d/0x49
 [<c04418fe>] ? tick_sched_timer+0x6d/0xa5
 [<c0441891>] ? tick_sched_timer+0x0/0xa5
 [<c043bc7e>] ? __run_hrtimer+0x51/0x88
 [<c043c681>] ? hrtimer_interrupt+0xf8/0x163
 [<c0414c87>] ? smp_apic_timer_interrupt+0x69/0x7c
 [<c040684b>] ? apic_timer_interrupt+0x33/0x38
 [<c044596a>] ? lock_acquire+0x7e/0x90
 [<c063b2a1>] ? do_page_fault+0x289/0x6bd
 [<c043c7da>] ? down_read_trylock+0x37/0x41
 [<c063b2a1>] ? do_page_fault+0x289/0x6bd
 [<c063b2a1>] ? do_page_fault+0x289/0x6bd
 [<c04d621d>] ? avc_has_perm_noaudit+0x3a1/0x3dc
 [<c04d623a>] ? avc_has_perm_noaudit+0x3be/0x3dc
 [<c040a2f4>] ? native_sched_clock+0xb5/0xd1
 [<c04d6b9c>] ? avc_has_perm+0x39/0x43
 [<c040a2f4>] ? native_sched_clock+0xb5/0xd1
 [<c040a026>] ? sched_clock+0x8/0xb
 [<c0442ffd>] ? lock_release_holdtime+0x1a/0x115
 [<c042f364>] ? sys_capset+0x2a1/0x2b2
 [<c06394d5>] ? _spin_unlock+0x1d/0x20
 [<c045eacb>] ? audit_syscall_exit+0x2b1/0x2cc
 [<c063b018>] ? do_page_fault+0x0/0x6bd
 [<c0639b5a>] ? error_code+0x72/0x78
 =======================
Code: 69 c0 01 00 37 9e 55 89 e5 53 c1 e8 1a 83 ec 04 8b 04 85 90 82 a9 c0 8d 5d
f8 89 45 f8 eb 03 89 55 f8 8b 03 85 c0 74 0e 8b 45 f8 <8b> 10 0f 18 02 90 39 48
18 75 e9 5a 5b 5d c3 55 89 e5 57 89 d7 
EIP: [<c063b7b4>] get_kprobe+0x2d/0x3c SS:ESP 0068:deb70c68
Kernel panic - not syncing: Fatal exception in interrupt


The disassembly of the function that EIP points to:

c063b787 <get_kprobe>:
c063b787:	89 c1                	mov    %eax,%ecx
c063b789:	69 c0 01 00 37 9e    	imul   $0x9e370001,%eax,%eax
c063b78f:	55                   	push   %ebp
c063b790:	89 e5                	mov    %esp,%ebp
c063b792:	53                   	push   %ebx
c063b793:	c1 e8 1a             	shr    $0x1a,%eax
c063b796:	83 ec 04             	sub    $0x4,%esp
c063b799:	8b 04 85 90 82 a9 c0 	mov    -0x3f567d70(,%eax,4),%eax
c063b7a0:	8d 5d f8             	lea    -0x8(%ebp),%ebx
c063b7a3:	89 45 f8             	mov    %eax,-0x8(%ebp)
c063b7a6:	eb 03                	jmp    c063b7ab <get_kprobe+0x24>
c063b7a8:	89 55 f8             	mov    %edx,-0x8(%ebp)
c063b7ab:	8b 03                	mov    (%ebx),%eax
c063b7ad:	85 c0                	test   %eax,%eax
c063b7af:	74 0e                	je     c063b7bf <get_kprobe+0x38>
c063b7b1:	8b 45 f8             	mov    -0x8(%ebp),%eax
c063b7b4:	8b 10                	mov    (%eax),%edx
c063b7b6:	8d 74 26 00          	lea    0x0(%esi,%eiz,1),%esi
c063b7ba:	39 48 18             	cmp    %ecx,0x18(%eax)
c063b7bd:	75 e9                	jne    c063b7a8 <get_kprobe+0x21>
c063b7bf:	5a                   	pop    %edx
c063b7c0:	5b                   	pop    %ebx
c063b7c1:	5d                   	pop    %ebp
c063b7c2:	c3                   	ret

-- 
           Summary: testsuite/systemtap.maps/pmap_agg_overflow.stp crashes
                    on  2.6.25-0.121.rc5.git4.fc9
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: kprobes
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: wcohen at redhat dot com
GCC target triplet: i686


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-03-18 22:27 -------
Note that according to the oops message, the systemtap probe module
has already been unloaded.  This int3 in schedule_tick appears to have
been left behind.  Could this be associated with the batch-unregister
kprobe code - if that's in that kernel?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2008-03-19 01:56 -------
(In reply to comment #1)
>  Could this be associated with the batch-unregister
> kprobe code - if that's in that kernel?

I think it is not in that kernel yet... could you check it in /proc/kallsyms?
$ grep register_kprobes /proc/kallsyms

And have you ever gotten this panic on bare hardware(not on vmware)?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2008-03-19 02:09 -------
(In reply to comment #2)
> (In reply to comment #1)
> >  Could this be associated with the batch-unregister
> > kprobe code - if that's in that kernel?
> 
> I think it is not in that kernel yet... could you check it in /proc/kallsyms?

I checked that batch-unregister patch were not in 2.6.25-0.121.rc5.git4.fc9.
(from src.rpm)


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[ananth at in dot ibm dot com]

------- Additional Comments From ananth at in dot ibm dot com  2008-03-19 05:55 -------
Will,
Was your test on an smp system?

I've seen this crash once on my F9-Alpha... its a uni-processor system. The test
says its applicable only for SMPs. This test crashes when run during make
installcheck. But, stap -vvv -g pmap_agg_overflow.stp aborts the run with a warning.

I looked at the stap script... it probes scheduler_tick. To just verify I tried
a kprobe module that had a probe on scheduler_tick and it worked without problems.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[ananth at in dot ibm dot com]

------- Additional Comments From ananth at in dot ibm dot com  2008-03-19 09:32 -------
From the objdump, it looks like the OOPS is at hlist_for_each_entry_rcu() in
get_kprobe(). Code surrounding this hasn't changed since July 2005!

However, a custom built 2.6.25-rc6 on the F9-Alpha works fine. Strange!

[ananth@... linux-2.6.25-rc6]$ uname -a
Linux ....in.ibm.com 2.6.25-rc6-lean #1 Wed Mar 19 13:53:38 IST 2008 i686 i686
i386 GNU/Linux

[ananth@... linux-2.6.25-rc6]$ stap -V
SystemTap translator/driver (version 0.6.2/0.133 built 2008-03-18)
Copyright (C) 2005-2008 Red Hat, Inc. and others
This is free software; see the source for copying conditions.

Objdump of the same portion on the working kernel is:

00000000 <get_kprobe>:
   0:   89 c1                   mov    %eax,%ecx
   2:   69 c0 01 00 37 9e       imul   $0x9e370001,%eax,%eax
   8:   53                      push   %ebx
   9:   83 ec 04                sub    $0x4,%esp
   c:   89 e3                   mov    %esp,%ebx
   e:   c1 e8 1a                shr    $0x1a,%eax
  11:   8b 04 85 08 00 00 00    mov    0x8(,%eax,4),%eax
  18:   89 04 24                mov    %eax,(%esp)
  1b:   eb 03                   jmp    20 <get_kprobe+0x20>
  1d:   89 14 24                mov    %edx,(%esp)
  20:   8b 03                   mov    (%ebx),%eax
  22:   85 c0                   test   %eax,%eax
  24:   74 0e                   je     34 <get_kprobe+0x34>
  26:   8b 04 24                mov    (%esp),%eax
  29:   8b 10                   mov    (%eax),%edx
  2b:   8d 74 26 00             lea    0x0(%esi,%eiz,1),%esi
  2f:   39 48 18                cmp    %ecx,0x18(%eax)
  32:   75 e9                   jne    1d <get_kprobe+0x1d>
  34:   5a                      pop    %edx
  35:   5b                      pop    %ebx
  36:   c3                      ret


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2008-03-19 12:55 -------
I have only tested this on a vmware machine running F-9. I have not tested this
on bare hardware. This is a uniprocessor machine. Maybe this is something that
vmware is not handling properly. I have a F-8 image on the same vmware machine.
I will check to whether the test runs correctly there.

The test correct prints the message that the test is only applies to
uniprocessor machines and then crashes. The test appears to be exiting. Could
there be a race where the handler for the probe is removed before. The problem
is elsewhere just manifested in that particular place because the data
structures are incorrect?

Are there other changes in the kprobes between the F-8 (2.6.24.3-34.fc8)and F-9
(2.6.25-0.121.rc5.git4.fc9) kernels)?

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[wcohen at redhat dot com]

------- Additional Comments From wcohen at redhat dot com  2008-03-19 14:21 -------
Trying the test on F-8 vmware machine cause it to crash in the same manner. I am
beginning to think this is an issue with vmware. I have an F-8 i686 machine
running the kernel and checkout of systemtap. It is a dual processor machine.
I set it to be uniprocessor and ran the test repeatedly and it ran fine.



-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[ananth at in dot ibm dot com]

------- Additional Comments From ananth at in dot ibm dot com  2008-03-19 14:30 -------
That an upstream kernel runs fine leads me to suspect the Fedora kernel. That
said, there have been updates to kprobes between 2.6.24 and 2.6.25-rc6 upstream
kernels, but none of them are in the general vicinity that'd cause such a crash.
(They are mostly related to the kretprobe entry handler, Masami's kretprobe
bugfix, and my CONFIG_KRETPROBE addition).

I don't think this is a handler issue. The problem seems to be at get_kprobe()
time, which lends suspicion of kp.hlist being corrupt. kp.hlist is the first
element of the structure, and *maybe* some pointer/data manipulation is leading
to this getting corrupt.

Also, probing scheduler_tick() directly using a plain C kprobe module on a
kernel that exhibits the crash, works fine. Even stap invoked on just the script
also puts out a warning and doesn't crash the system.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-04-17 14:00 -------
I've seen similar crashes lately on my KVM virtual machines.
I wonder if this is a systemic problem with these emulators.
Even if so, it is probably worth working around it in systemtap
if possible, by perhaps adding a bunch of redundant cache-flush,
schedule() type calls into the shutdown sequence.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug kprobes/5963] testsuite/systemtap.maps/pmap_agg_overflow.stp crashes on 2.6.25-0.121.rc5.git4.fc9

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2008-07-09 17:12 -------
I found a race condition in the runtime w.r.t. probes that
exit() during their begin probes.  Testing a patch.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|systemtap at sources dot    |fche at redhat dot com
                   |redhat dot com              |
             Status|NEW                         |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=5963

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.