[Bug kprobes/13108] New: kprobing some paravirt stuff seems unsafe

[Bug kprobes/13108] New: kprobing some paravirt stuff seems unsafe

[mjw at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13108

             Bug #: 13108
           Summary: kprobing some paravirt stuff seems unsafe
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: kprobes
        AssignedTo: systemtap@sourceware.org
        ReportedBy: mjw@redhat.com
    Classification: Unclassified


The following, run inside a kvm guest will often (but not always) crash the kvm
guest:

$ /usr/local/install/systemtap/bin/stap -m clts -e "global c; probe
kernel.function(\"clts\") { if(c++ < 3) log(pp()) else exit() }" -c 'sleep 1;
ls -laR /dev /proc > /tmp/garbage.out 2>&1; sync'

The crashes aren't consistent though:

exhibit 1)

clts: systemtap: 1.7/0.152, base: ffffffffa06f5000, memory:
48data/18text/10ctx/10net/33alloc kb, probes: 7
BUG: unable to handle kernel paging request at fffffffffffffff0
IP: [<ffffffff810155e7>] restore_i387_xstate+0xc7/0x1c0
PGD 1a27067 PUD 1a28067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/module/xt_state/sections/__mcount_loc
CPU 3 
Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: cls_destroy]

Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: cls_destroy]
Pid: 13482, comm: stapio Tainted: G           ---------------- T
2.6.32-131.6.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810155e7>]  [<ffffffff810155e7>]
restore_i387_xstate+0xc7/0x1c0
RSP: 0018:ffff8800061b7ea8  EFLAGS: 00010346
RAX: ffff8800061b6000 RBX: 00007fff5d53c6c0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037a76800
RBP: ffff8800061b7ef8 R08: 0000000000000000 R09: ffff880037a76600
R10: 00007fff5d53c710 R11: 0000000000000246 R12: ffff880099747540
R13: ffff880099747540 R14: ffff8800061b7fd8 R15: 00007fff5d53c500
FS:  00007f080b574700(0000) GS:ffff880002180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: fffffffffffffff0 CR3: 00000000061ff000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
Process stapio (pid: 13482, threadinfo ffff8800061b6000, task ffff880099747540)
Stack:
 ffff8800061b7f48 00007fffffffeffd ffff8800061b7f08 0000000101e1dd39
<0> ffff880099747af8 ffff8800061b7fd8 0000000000402080 00007fff5d53c4f8
<0> ffff8800061b7f58 0000000000000200 ffff8800061b7f48 ffffffff8100adc0
Call Trace:
 [<ffffffff8100adc0>] sys_rt_sigreturn+0x200/0x280
 [<ffffffff8100b68c>] stub_rt_sigreturn+0x6c/0xa0
Code: e0 ff ff 48 83 de 00 48 85 f6 0f 85 ff 00 00 00 41 f6 44 24 15 20 74 7c
65 4c 8b 2c 25 00 cc 00 00 49 8b 45 08 f6 40 14 01 75 0f <cc> 06 0f 1f 44 00 00
49 8b 45 08 83 48 14 01 b0 00 84 c0 74 44 
RIP  [<ffffffff810155e7>] restore_i387_xstate+0xc7/0x1c0
 RSP <ffff8800061b7ea8>
CR2: fffffffffffffff0
---[ end trace d2747920f0b64285 ]---
Kernel panic - not syncing: Fatal exception
Pid: 13482, comm: stapio Tainted: G      D    ---------------- T
2.6.32-131.6.1.el6.x86_64 #1
Call Trace:
 [<ffffffff814da518>] ? panic+0x78/0x143
 [<ffffffff814de564>] ? oops_end+0xe4/0x100
 [<ffffffff81040c9b>] ? no_context+0xfb/0x260
 [<ffffffff81040f25>] ? __bad_area_nosemaphore+0x125/0x1e0
 [<ffffffff81040ff3>] ? bad_area_nosemaphore+0x13/0x20
 [<ffffffff810416cd>] ? __do_page_fault+0x31d/0x480
 [<ffffffff8107d8ed>] ? __sigqueue_free+0x3d/0x50
 [<ffffffff8108120f>] ? __dequeue_signal+0xdf/0x1f0
 [<ffffffff810813fa>] ? dequeue_signal+0xda/0x170
 [<ffffffff814e054e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814dd8d5>] ? page_fault+0x25/0x30
 [<ffffffff810155e7>] ? restore_i387_xstate+0xc7/0x1c0
 [<ffffffff81015658>] ? restore_i387_xstate+0x138/0x1c0
 [<ffffffff8100adc0>] ? sys_rt_sigreturn+0x200/0x280
 [<ffffffff8100b68c>] ? stub_rt_sigreturn+0x6c/0xa0

exhibit 2) [note it ran the same probe first without trouble]

clts: systemtap: 1.7/0.152, base: ffffffffa00f8000, memory:
48data/18text/10ctx/10net/33alloc kb, probes: 7
clts: systemtap: 1.7/0.152, base: ffffffffa02c6000, memory:
48data/18text/10ctx/10net/33alloc kb, probes: 7
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/module/xt_state/sections/__mcount_loc
CPU 3 
Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts]

Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts]
Pid: 0, comm: swapper Tainted: G           ---------------- T
2.6.32-131.6.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffffa001a002>]  [<ffffffffa001a002>] 0xffffffffa001a002
RSP: 0018:ffff880099eb7ad8  EFLAGS: 00010102
RAX: ffff88009c046000 RBX: ffff88009b327580 RCX: ffff88009c01eb00
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88009b327c20
RBP: ffff880099eb7b28 R08: 0000000000000000 R09: 0000000000000001
R10: 0000002f818d8aa9 R11: 0000000000000001 R12: ffff88009c01eb00
R13: 0000000000000000 R14: 0000000000000003 R15: ffff88009b327c20
FS:  0000000000000000(0000) GS:ffff880002180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000020ba110 CR3: 000000009a062000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88009c046000, task ffff88009c01eb00)
Stack:
 0000000000000000 0000000000000000 0000000001000000 ffff880002193b40
<0> 0000000000000001 ffff880002195f80 ffff88009acb5a00 0000000000000003
<0> ffff88009b16a440 00000000ffffffff ffff88009b327580 ffffffff814dabd9
Call Trace:
Code: 
BUG: unable to handle kernel paging request at ffffffffa0019fd7
IP: [<ffffffff81009757>] __switch_to+0x157/0x320
PGD 1a27067 PUD 1a2b063 PMD 37b49067 PTE 0
Oops: 0000 [#2] SMP 
last sysfs file: /sys/module/xt_state/sections/__mcount_loc
CPU 3 
Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts]

Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts]
Pid: 0, comm: swapper Tainted: G           ---------------- T
2.6.32-131.6.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff81009757>]  [<ffffffff81009757>] __switch_to+0x157/0x320
RSP: 0018:ffff880099eb7850  EFLAGS: 00010097
RAX: ffff880099eb7887 RBX: ffff880099eb7a28 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffffffa0019fd7 RDI: ffff880099eb7887
RBP: ffff880099eb78b8 R08: ffffffff81b9e300 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: ffffffffa0019fd7
R13: ffff88009c047fd8 R14: ffff88009c046000 R15: 000000000000002b
FS:  0000000000000000(0000) GS:ffff880002180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffa0019fd7 CR3: 000000009a062000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88009c046000, task ffff88009c01eb00)
Stack:
 ffffffff8100e3cf ffffffff81773d83 ffffffffffffffff 0000000000000000
<0> 0000004000000006 ffff880099eb7888 ffffffff814e066a ffff880099eb78b8
<0> 0000000000000000 ffffffff81773d83 ffff880099eb7a28 0000000000000000
Call Trace:
Code: cb 02 00 66 90 48 89 c7 48 83 cf 08 e8 83 cb 02 00 66 90 eb 10 0f 1f 80
00 00 00 00 41 c6 84 24 10 02 00 00 00 80 7d c3 00 74 07 <0f> 06 0f 1f 44 00 00
48 89 df 0f 1f 80 00 00 00 00 45 85 ed 0f 
RIP  [<ffffffff81009757>] __switch_to+0x157/0x320
 RSP <ffff880099eb7850>
CR2: ffffffffa0019fd7
---[ end trace 07cc9d4c6df5c545 ]---
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D    ---------------- T
2.6.32-131.6.1.el6.x86_64 #1
Call Trace:


A lot of the paravirt stuff (at least that inside
arch/x86/include/asm/paravirt.h and arch/x86/kernel/paravirt*.c) looks somewhat
problematic/tricky to handle through kprobes.

Trying the following patch:

diff --git a/dwflpp.cxx b/dwflpp.cxx
index 7da8a72..36a4a3c 100644
--- a/dwflpp.cxx
+++ b/dwflpp.cxx
@@ -2963,6 +2963,9 @@ dwflpp::build_blacklist()
   blfile += "|arch/.*/include/asm/io\\.h";
   blfile += "|arch/.*/include/asm/bitops\\.h";
   blfile += "|drivers/ide/ide-iops\\.c";
+  // paravirt ops
+  blfile += "|arch/.*/kernel/paravirt.*c";
+  blfile += "|arch/.*/include/asm/paravirt\\.h";

   // XXX: it would be nice if these blacklisted functions were pulled
   // in dynamically, instead of being statically defined here.

Might be overkill?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13108] kprobing some paravirt stuff seems unsafe

[mjw at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13108

--- Comment #1 from Mark Wielaard <mjw at redhat dot com> 2011-08-18 20:15:30 UTC ---
The patch in comment #1 doesn't seem to prevent setting a probe on
kernel.function("clts"), so it is wrong, haven't figured out why yet though.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13108] kprobing some paravirt stuff seems unsafe

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13108

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #2 from Josh Stone <jistone at redhat dot com> 2011-08-18 21:31:50 UTC ---
(In reply to comment #1)
> The patch in comment #1 doesn't seem to prevent setting a probe on
> kernel.function("clts"), so it is wrong, haven't figured out why yet though.

Ugh -- I think you've uncovered another bug, that blfile probably shouldn't be
^-anchored at the start.  I get:

> $ stap -l 'kernel.function("*@paravirt*.c")' | wc -l
> 0
> $ stap -l 'kernel.function("*@paravirt*.h")' | wc -l
> 93
> $ stap -l 'kernel.function("clts")'
> kernel.function("clts@/usr/src/debug/kernel-2.6.39.fc15/linux-2.6.39.x86_64/arch/x86/include/asm/paravirt.h:47")

It looks like all #include <asm/*> files are showing a full path, thus we
shouldn't expect to match the leading portions at all.  Either those specific
asm paths need a ".*" in front, or we can remove the "^" from blfile
altogether.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13108] kprobing some paravirt stuff seems unsafe

[mjw at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13108

Mark Wielaard <mjw at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #3 from Mark Wielaard <mjw at redhat dot com> 2011-09-05 22:07:56 UTC ---
(In reply to comment #2)
> (In reply to comment #1)
> > The patch in comment #1 doesn't seem to prevent setting a probe on
> > kernel.function("clts"), so it is wrong, haven't figured out why yet though.
> 
> Ugh -- I think you've uncovered another bug, that blfile probably shouldn't be
> ^-anchored at the start.

Wow, you are right! After some experimentation it seems only include/asm/*.h
files that get selected by "full path". So I fixed it with:

commit 44a7e76ab8cd9b9942b8d8a72d065269cd637c4a
Author: Mark Wielaard <mjw@redhat.com>
Date:   Mon Sep 5 23:37:11 2011 +0200

    PR13112 (and PR13108) blacklist probing function from include/asm .h files.

    dwflpp.cxx (build_blacklist): all include/asm .h blfile patterns might
    need "full path" so prefix those with '.*'
    Add new XFAIL semok.exp inb_blacklisted.stp testcase.

Plus a patch to explicitly list the paravirt ops functions with:

commit 1b43894364dec075d13636b3c78c786da73fc8ad
Author: Mark Wielaard <mjw@redhat.com>
Date:   Mon Sep 5 23:44:53 2011 +0200

    PR13108 blacklist probing paravirt ops from paravirt.c or paravirt.h.

See also the follow up commit 4a507d for PR13112.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.