From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 11129 invoked by alias); 18 Aug 2011 20:08:32 -0000 Received: (qmail 11118 invoked by uid 22791); 18 Aug 2011 20:08:25 -0000 X-SWARE-Spam-Status: No, hits=-2.6 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00,TW_CV,TW_II,TW_JB,TW_PC,TW_SN X-Spam-Check-By: sourceware.org Received: from localhost (HELO sourceware.org) (127.0.0.1) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 18 Aug 2011 20:08:11 +0000 From: "mjw at redhat dot com" To: systemtap@sourceware.org Subject: [Bug kprobes/13108] New: kprobing some paravirt stuff seems unsafe Date: Thu, 18 Aug 2011 20:08:00 -0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: systemtap X-Bugzilla-Component: kprobes X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: mjw at redhat dot com X-Bugzilla-Status: NEW X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: systemtap at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Changed-Fields: Message-ID: X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q3/txt/msg00178.txt.bz2 http://sourceware.org/bugzilla/show_bug.cgi?id=13108 Bug #: 13108 Summary: kprobing some paravirt stuff seems unsafe Product: systemtap Version: unspecified Status: NEW Severity: normal Priority: P2 Component: kprobes AssignedTo: systemtap@sourceware.org ReportedBy: mjw@redhat.com Classification: Unclassified The following, run inside a kvm guest will often (but not always) crash the kvm guest: $ /usr/local/install/systemtap/bin/stap -m clts -e "global c; probe kernel.function(\"clts\") { if(c++ < 3) log(pp()) else exit() }" -c 'sleep 1; ls -laR /dev /proc > /tmp/garbage.out 2>&1; sync' The crashes aren't consistent though: exhibit 1) clts: systemtap: 1.7/0.152, base: ffffffffa06f5000, memory: 48data/18text/10ctx/10net/33alloc kb, probes: 7 BUG: unable to handle kernel paging request at fffffffffffffff0 IP: [] restore_i387_xstate+0xc7/0x1c0 PGD 1a27067 PUD 1a28067 PMD 0 Oops: 0002 [#1] SMP last sysfs file: /sys/module/xt_state/sections/__mcount_loc CPU 3 Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: cls_destroy] Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: cls_destroy] Pid: 13482, comm: stapio Tainted: G ---------------- T 2.6.32-131.6.1.el6.x86_64 #1 Bochs RIP: 0010:[] [] restore_i387_xstate+0xc7/0x1c0 RSP: 0018:ffff8800061b7ea8 EFLAGS: 00010346 RAX: ffff8800061b6000 RBX: 00007fff5d53c6c0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037a76800 RBP: ffff8800061b7ef8 R08: 0000000000000000 R09: ffff880037a76600 R10: 00007fff5d53c710 R11: 0000000000000246 R12: ffff880099747540 R13: ffff880099747540 R14: ffff8800061b7fd8 R15: 00007fff5d53c500 FS: 00007f080b574700(0000) GS:ffff880002180000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffffffffffff0 CR3: 00000000061ff000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Process stapio (pid: 13482, threadinfo ffff8800061b6000, task ffff880099747540) Stack: ffff8800061b7f48 00007fffffffeffd ffff8800061b7f08 0000000101e1dd39 <0> ffff880099747af8 ffff8800061b7fd8 0000000000402080 00007fff5d53c4f8 <0> ffff8800061b7f58 0000000000000200 ffff8800061b7f48 ffffffff8100adc0 Call Trace: [] sys_rt_sigreturn+0x200/0x280 [] stub_rt_sigreturn+0x6c/0xa0 Code: e0 ff ff 48 83 de 00 48 85 f6 0f 85 ff 00 00 00 41 f6 44 24 15 20 74 7c 65 4c 8b 2c 25 00 cc 00 00 49 8b 45 08 f6 40 14 01 75 0f 06 0f 1f 44 00 00 49 8b 45 08 83 48 14 01 b0 00 84 c0 74 44 RIP [] restore_i387_xstate+0xc7/0x1c0 RSP CR2: fffffffffffffff0 ---[ end trace d2747920f0b64285 ]--- Kernel panic - not syncing: Fatal exception Pid: 13482, comm: stapio Tainted: G D ---------------- T 2.6.32-131.6.1.el6.x86_64 #1 Call Trace: [] ? panic+0x78/0x143 [] ? oops_end+0xe4/0x100 [] ? no_context+0xfb/0x260 [] ? __bad_area_nosemaphore+0x125/0x1e0 [] ? bad_area_nosemaphore+0x13/0x20 [] ? __do_page_fault+0x31d/0x480 [] ? __sigqueue_free+0x3d/0x50 [] ? __dequeue_signal+0xdf/0x1f0 [] ? dequeue_signal+0xda/0x170 [] ? do_page_fault+0x3e/0xa0 [] ? page_fault+0x25/0x30 [] ? restore_i387_xstate+0xc7/0x1c0 [] ? restore_i387_xstate+0x138/0x1c0 [] ? sys_rt_sigreturn+0x200/0x280 [] ? stub_rt_sigreturn+0x6c/0xa0 exhibit 2) [note it ran the same probe first without trouble] clts: systemtap: 1.7/0.152, base: ffffffffa00f8000, memory: 48data/18text/10ctx/10net/33alloc kb, probes: 7 clts: systemtap: 1.7/0.152, base: ffffffffa02c6000, memory: 48data/18text/10ctx/10net/33alloc kb, probes: 7 invalid opcode: 0000 [#1] SMP last sysfs file: /sys/module/xt_state/sections/__mcount_loc CPU 3 Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts] Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts] Pid: 0, comm: swapper Tainted: G ---------------- T 2.6.32-131.6.1.el6.x86_64 #1 Bochs RIP: 0010:[] [] 0xffffffffa001a002 RSP: 0018:ffff880099eb7ad8 EFLAGS: 00010102 RAX: ffff88009c046000 RBX: ffff88009b327580 RCX: ffff88009c01eb00 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88009b327c20 RBP: ffff880099eb7b28 R08: 0000000000000000 R09: 0000000000000001 R10: 0000002f818d8aa9 R11: 0000000000000001 R12: ffff88009c01eb00 R13: 0000000000000000 R14: 0000000000000003 R15: ffff88009b327c20 FS: 0000000000000000(0000) GS:ffff880002180000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000020ba110 CR3: 000000009a062000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffff88009c046000, task ffff88009c01eb00) Stack: 0000000000000000 0000000000000000 0000000001000000 ffff880002193b40 <0> 0000000000000001 ffff880002195f80 ffff88009acb5a00 0000000000000003 <0> ffff88009b16a440 00000000ffffffff ffff88009b327580 ffffffff814dabd9 Call Trace: Code: BUG: unable to handle kernel paging request at ffffffffa0019fd7 IP: [] __switch_to+0x157/0x320 PGD 1a27067 PUD 1a2b063 PMD 37b49067 PTE 0 Oops: 0000 [#2] SMP last sysfs file: /sys/module/xt_state/sections/__mcount_loc CPU 3 Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts] Modules linked in: clts(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: clts] Pid: 0, comm: swapper Tainted: G ---------------- T 2.6.32-131.6.1.el6.x86_64 #1 Bochs RIP: 0010:[] [] __switch_to+0x157/0x320 RSP: 0018:ffff880099eb7850 EFLAGS: 00010097 RAX: ffff880099eb7887 RBX: ffff880099eb7a28 RCX: 0000000000000001 RDX: 0000000000000001 RSI: ffffffffa0019fd7 RDI: ffff880099eb7887 RBP: ffff880099eb78b8 R08: ffffffff81b9e300 R09: 0000000000000000 R10: 000000000000000f R11: 0000000000000000 R12: ffffffffa0019fd7 R13: ffff88009c047fd8 R14: ffff88009c046000 R15: 000000000000002b FS: 0000000000000000(0000) GS:ffff880002180000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffa0019fd7 CR3: 000000009a062000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffff88009c046000, task ffff88009c01eb00) Stack: ffffffff8100e3cf ffffffff81773d83 ffffffffffffffff 0000000000000000 <0> 0000004000000006 ffff880099eb7888 ffffffff814e066a ffff880099eb78b8 <0> 0000000000000000 ffffffff81773d83 ffff880099eb7a28 0000000000000000 Call Trace: Code: cb 02 00 66 90 48 89 c7 48 83 cf 08 e8 83 cb 02 00 66 90 eb 10 0f 1f 80 00 00 00 00 41 c6 84 24 10 02 00 00 00 80 7d c3 00 74 07 <0f> 06 0f 1f 44 00 00 48 89 df 0f 1f 80 00 00 00 00 45 85 ed 0f RIP [] __switch_to+0x157/0x320 RSP CR2: ffffffffa0019fd7 ---[ end trace 07cc9d4c6df5c545 ]--- Kernel panic - not syncing: Fatal exception Pid: 0, comm: swapper Tainted: G D ---------------- T 2.6.32-131.6.1.el6.x86_64 #1 Call Trace: A lot of the paravirt stuff (at least that inside arch/x86/include/asm/paravirt.h and arch/x86/kernel/paravirt*.c) looks somewhat problematic/tricky to handle through kprobes. Trying the following patch: diff --git a/dwflpp.cxx b/dwflpp.cxx index 7da8a72..36a4a3c 100644 --- a/dwflpp.cxx +++ b/dwflpp.cxx @@ -2963,6 +2963,9 @@ dwflpp::build_blacklist() blfile += "|arch/.*/include/asm/io\\.h"; blfile += "|arch/.*/include/asm/bitops\\.h"; blfile += "|drivers/ide/ide-iops\\.c"; + // paravirt ops + blfile += "|arch/.*/kernel/paravirt.*c"; + blfile += "|arch/.*/include/asm/paravirt\\.h"; // XXX: it would be nice if these blacklisted functions were pulled // in dynamically, instead of being statically defined here. Might be overkill? -- Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.