* [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine
@ 2011-09-16 13:38 mjw at redhat dot com
2011-09-23 11:53 ` [Bug testsuite/13193] " mjw at redhat dot com
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: mjw at redhat dot com @ 2011-09-16 13:38 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=13193
Bug #: 13193
Summary: Running bz6503.exp results in unresponsive/crashed
machine
Product: systemtap
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: testsuite
AssignedTo: systemtap@sourceware.org
ReportedBy: mjw@redhat.com
Classification: Unclassified
This happens on both "real hardware" with x86_64 f14 (2.6.35.14-96.fc14.x86_64)
and under kvm with x86_64 rhel6 (2.6.32-131.12.1.el6.x86_64).
The following is the result of running make installcheck
RUNTESTFLAGS=bz6503.exp on the rhel6 kvm instance, with echo 9 >
/proc/sysrq-trigger and adding -DDEBUG_SYMBOLS to the testcase:
SysRq : Changing Loglevel
Loglevel set to 9
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646: systemtap: 1.7/0.152, base:
ffffffffa0507000, memory: 221data/20text/2ctx/10net/106alloc kb, probes: 373
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5]
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat
BUG: soft lockup - CPU#1 stuck for 67s! [migration/1:8]
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
CPU 1:
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
Pid: 8, comm: migration/1 Tainted: G ---------------- T
2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f4>] [<ffffffff810c36f4>]
stop_machine_cpu_stop+0x74/0xe0
RSP: 0000:ffff88009cbd5dd0 EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbd5df0 RCX: ffff8800022911e8
RDX: 0000000000000000 RSI: ffff880099994100 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbd4000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbd5d50
R13: ffffffff814dad57 R14: ffff88009cbd5df0 R15: ffff88009ac86f00
FS: 0000000000000000(0000) GS:ffff880002280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031ae41b397 CR3: 0000000037335000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
[<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
[<ffffffff814dad57>] ? thread_return+0x4e/0x777
[<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
[<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
[<ffffffff8108de16>] ? kthread+0x96/0xa0
[<ffffffff8100c1ca>] ? child_rip+0xa/0x20
[<ffffffff8108dd80>] ? kthread+0x0/0xa0
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#2 stuck for 67s! [modprobe:2652]
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
CPU 2:
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
Pid: 2652, comm: modprobe Tainted: G ---------------- T
2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810a4536>] [<ffffffff810a4536>]
smp_call_function_many+0x1b6/0x210
RSP: 0018:ffff880098ccbcc8 EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff880098ccbd08 RCX: 0000000000000004
RDX: 0000000000000010 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffffffff8100bc8e R08: 0000000000000000 R09: ffff880037bc0c00
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
R13: ffff880098ccbc78 R14: 0000000000000000 R15: 0000000000000020
FS: 00007f5f3eebf700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000024ef0a8 CR3: 0000000099006000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810a451a>] ? smp_call_function_many+0x19a/0x210
[<ffffffff81159560>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a45b2>] ? smp_call_function+0x22/0x30
[<ffffffff8106f2f4>] ? on_each_cpu+0x24/0x50
[<ffffffff8115c53f>] ? do_tune_cpucache+0x12f/0x630
[<ffffffff8115cc1b>] ? enable_cpucache+0x3b/0xf0
[<ffffffff814c575f>] ? setup_cpu_cache+0x22f/0x340
[<ffffffff8115b1b2>] ? kmem_cache_alloc+0x182/0x190
[<ffffffff8115d8aa>] ? kmem_cache_create+0x3fa/0x580
[<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
[<ffffffffa0362510>] ? jffs2_i_init_once+0x0/0x40 [jffs2]
[<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
[<ffffffffa008c1ba>] ? init_jffs2_fs+0x36/0xc7 [jffs2]
[<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0
[<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250
[<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T)
nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror
dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode
virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core
ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 0:
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
Pid: 5, comm: migration/0 Tainted: G ---------------- T
2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f8>] [<ffffffff810c36f8>]
stop_machine_cpu_stop+0x78/0xe0
RSP: 0018:ffff88009cbcbdd0 EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbcbdf0 RCX: ffff8800022111e8
RDX: 0000000000000000 RSI: ffff8800953d4ac0 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbca000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbcbd50
R13: ffffffff814dad57 R14: ffff88009cbcbdf0 R15: ffff880037763200
FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031af203088 CR3: 0000000098cc3000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
[<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
[<ffffffff814dad57>] ? thread_return+0x4e/0x777
[<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
[<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
[<ffffffff8108de16>] ? kthread+0x96/0xa0
[<ffffffff8100c1ca>] ? child_rip+0xa/0x20
[<ffffffff8108dd80>] ? kthread+0x0/0xa0
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5]
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat
BUG: soft lockup - CPU#1 stuck for 67s! [migration/1:8]
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
CPU 1:
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
Pid: 8, comm: migration/1 Tainted: G ---------------- T
2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f4>] [<ffffffff810c36f4>]
stop_machine_cpu_stop+0x74/0xe0
RSP: 0000:ffff88009cbd5dd0 EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbd5df0 RCX: ffff8800022911e8
RDX: 0000000000000000 RSI: ffff880099994100 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbd4000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbd5d50
R13: ffffffff814dad57 R14: ffff88009cbd5df0 R15: ffff88009ac86f00
FS: 0000000000000000(0000) GS:ffff880002280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031ae41b397 CR3: 0000000037335000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
[<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
[<ffffffff814dad57>] ? thread_return+0x4e/0x777
[<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
[<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
[<ffffffff8108de16>] ? kthread+0x96/0xa0
[<ffffffff8100c1ca>] ? child_rip+0xa/0x20
[<ffffffff8108dd80>] ? kthread+0x0/0xa0
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#2 stuck for 67s! [modprobe:2652]
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
CPU 2:
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
Pid: 2652, comm: modprobe Tainted: G ---------------- T
2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810a4536>] [<ffffffff810a4536>]
smp_call_function_many+0x1b6/0x210
RSP: 0018:ffff880098ccbcc8 EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff880098ccbd08 RCX: 0000000000000004
RDX: 0000000000000010 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffffffff8100bc8e R08: 0000000000000000 R09: ffff880037bc0c00
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
R13: ffff880098ccbc78 R14: 0000000000000000 R15: 0000000000000020
FS: 00007f5f3eebf700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000024ef0a8 CR3: 0000000099006000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810a451a>] ? smp_call_function_many+0x19a/0x210
[<ffffffff81159560>] ? do_ccupdate_local+0x0/0x40
[<ffffffff810a45b2>] ? smp_call_function+0x22/0x30
[<ffffffff8106f2f4>] ? on_each_cpu+0x24/0x50
[<ffffffff8115c53f>] ? do_tune_cpucache+0x12f/0x630
[<ffffffff8115cc1b>] ? enable_cpucache+0x3b/0xf0
[<ffffffff814c575f>] ? setup_cpu_cache+0x22f/0x340
[<ffffffff8115b1b2>] ? kmem_cache_alloc+0x182/0x190
[<ffffffff8115d8aa>] ? kmem_cache_create+0x3fa/0x580
[<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
[<ffffffffa0362510>] ? jffs2_i_init_once+0x0/0x40 [jffs2]
[<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
[<ffffffffa008c1ba>] ? init_jffs2_fs+0x36/0xc7 [jffs2]
[<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0
[<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250
[<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl
auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log
vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0
snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore
snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk
sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix
dm_mod [last unloaded: speedstep_lib]
CPU 0:
Modules linked in: jffs2(+) ext2(+) zlib_deflate
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc
autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun
uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net
i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci
virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded:
speedstep_lib]
Pid: 5, comm: migration/0 Tainted: G ---------------- T
2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f4>] [<ffffffff810c36f4>]
stop_machine_cpu_stop+0x74/0xe0
RSP: 0018:ffff88009cbcbdd0 EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbcbdf0 RCX: ffff8800022111e8
RDX: 0000000000000000 RSI: ffff8800953d4ac0 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbca000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbcbd50
R13: ffffffff814dad57 R14: ffff88009cbcbdf0 R15: ffff880037763200
FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031af203088 CR3: 0000000098cc3000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
[<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
[<ffffffff814dad57>] ? thread_return+0x4e/0x777
[<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
[<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
[<ffffffff8108de16>] ? kthread+0x96/0xa0
[<ffffffff8100c1ca>] ? child_rip+0xa/0x20
[<ffffffff8108dd80>] ? kthread+0x0/0xa0
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Bug testsuite/13193] Running bz6503.exp results in unresponsive/crashed machine
2011-09-16 13:38 [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine mjw at redhat dot com
@ 2011-09-23 11:53 ` mjw at redhat dot com
2011-11-06 16:09 ` fche at redhat dot com
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: mjw at redhat dot com @ 2011-09-23 11:53 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=13193
--- Comment #2 from Mark Wielaard <mjw at redhat dot com> 2011-09-23 11:52:17 UTC ---
There was a workaround committed which looked like it was perfect:
commit 01cfbeb43559e5a93eda648fc81d4aa102e6cb13
Author: Frank Ch. Eigler <fche@redhat.com>
Date: Fri Sep 16 17:34:14 2011 -0400
PR13193 wallpaper
The reported stuck process bug (whether it's in the kernel or
in stap), is apparently triggered by concurrent modprobes. Change
the testsuite to avoid this practice, to wallpaper over the problem
temporarily.
Unfortunately I just got another issue with this testcase, this time on i386
f15:
[157004.483057] stap_6ab51e385550b0fd6d843037b0d7d38c_9609: systemtap:
1.7/0.148.90, base: f8aca000, memory: 34data/20text/2ctx/10net/16alloc kb,
probes: 2
[157008.989239] stap_a40026234a1787c83a1dafb3aae558ca__9841: systemtap:
1.7/0.148.90, base: f8b24000, memory: 61data/24text/15ctx/10net/163alloc kb,
probes: 10
[157013.346702] stap_190d377c71fa6a45f476ce480c4912a1_10067: systemtap:
1.7/0.148.90, base: f8ad9000, memory: 51data/24text/15ctx/10net/163alloc kb,
probes: 6
[157018.205175] stap_1aa19d6453909e68c4758c58d1d67725_10296: systemtap:
1.7/0.148.90, base: f8b61000, memory: 143data/15text/0ctx/10net/64alloc kb,
probes: 396
[157018.227386] Warning: found a stray unused aggrprobe@f8942011
[157018.227398] ------------[ cut here ]------------
[157018.227453] kernel BUG at kernel/kprobes.c:707!
[157018.227498] invalid opcode: 0000 [#1] SMP
[157018.227543] Modules linked in: jffs2(+)
stap_1aa19d6453909e68c4758c58d1d67725_10296 uprobes nf_conntrack_ipv6
nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4
xt_state nf_conntrack stap_e9739cce4a6f43356403eefe748313cc_565
stap_06d4707db0c0eb2d86b639a84f948296_9487
stap_e0d0211e1b7bbf8f7df399c1665c8493_565 mtd zlib_deflate tcp_lp tun fuse
ebtable_nat ebtables xt_CHECKSUM bridge stp llc ppdev parport_pc lp parport
sunrpc cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth xt_physdev
ip6t_REJECT snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_seq
snd_seq_device iTCO_wdt snd_pcm arc4 iwl3945 snd_timer iwl_legacy mac80211
iTCO_vendor_support thinkpad_acpi snd soundcore cfg80211 e1000e i2c_i801 joydev
microcode snd_page_alloc rfkill virtio_net kvm_intel kvm uinput ipv6 xts
gf128mul dm_crypt yenta_socket i915 drm_kms_helper drm i2c_algo_bit i2c_core
video [last unloaded: stap_190d377c71fa6a45f476ce480c4912a1_10067]
[157018.228133]
[157018.228133] Pid: 10302, comm: modprobe Not tainted 2.6.40.4-5.fc15.i686 #1
LENOVO 195144G/195144G
[157018.228133] EIP: 0060:[<c07ee524>] EFLAGS: 00010246 CPU: 0
[157018.228133] EIP is at register_kprobe+0x1ca/0x3a5
[157018.228133] EAX: 00000047 EBX: f8b827fc ECX: 00000046 EDX: 00000000
[157018.228133] ESI: e9856420 EDI: 00000000 EBP: d1693eb0 ESP: d1693e90
[157018.228133] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[157018.228133] Process modprobe (pid: 10302, ti=d1692000 task=dba09940
task.ti=d1692000)
[157018.228133] Stack:
[157018.228133] c0939f72 f8942011 f8b6f1fe f8b82804 f8bfa5c4 f8b827fc f8b6f1f0
00002100
[157018.228133] d1693ed4 f8b6370e f6173a40 00000246 d1693ee4 000000c0 f8b6d0e4
00000000
[157018.228133] 00000001 d1693ef0 c07ed017 f8bfa5c4 c0a3b7ec c0a398c0 ffffffff
00000460
[157018.228133] Call Trace:
[157018.228133] [<f8942011>] ? jffs2_compressors_init+0x11/0x11 [jffs2]
[157018.228133] [<f8b6370e>] _stp_module_notifier+0x1a0/0x26e
[stap_1aa19d6453909e68c4758c58d1d67725_10296]
[157018.228133] [<c07ed017>] notifier_call_chain+0x2b/0x4d
[157018.228133] [<c04582e6>] __blocking_notifier_call_chain+0x40/0x52
[157018.228133] [<c0458317>] blocking_notifier_call_chain+0x1f/0x21
[157018.228133] [<c0469ea0>] sys_init_module+0x1384/0x15af
[157018.228133] [<c07e9f54>] syscall_call+0x7/0xb
[157018.228133] Code: 39 46 40 75 13 8b 46 14 c7 04 24 72 9f 93 c0 89 44 24 04
e8 08 41 ff ff 83 66 3c fd 81 7e 20 9c d1 7e c0 75 06 83 7e 50 00 75 02 <0f> 0b
89 f0 e8 1c f5 ff ff f6 46 3c 01 74 22 89 f0 e8 e6 da ff
[157018.228133] EIP: [<c07ee524>] register_kprobe+0x1ca/0x3a5 SS:ESP
0068:d1693e90
[157018.293049] ---[ end trace f0f649cc63eb195a ]---
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Bug testsuite/13193] Running bz6503.exp results in unresponsive/crashed machine
2011-09-16 13:38 [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine mjw at redhat dot com
2011-09-23 11:53 ` [Bug testsuite/13193] " mjw at redhat dot com
@ 2011-11-06 16:09 ` fche at redhat dot com
2011-11-06 17:39 ` fche at redhat dot com
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: fche at redhat dot com @ 2011-11-06 16:09 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=13193
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |fche at redhat dot com,
| |masami.hiramatsu.pt at
| |hitachi dot com
--- Comment #3 from Frank Ch. Eigler <fche at redhat dot com> 2011-11-06 16:09:12 UTC ---
cc:ing Masami because of the optimized-kprobes kernel warnings.
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Bug testsuite/13193] Running bz6503.exp results in unresponsive/crashed machine
2011-09-16 13:38 [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine mjw at redhat dot com
2011-09-23 11:53 ` [Bug testsuite/13193] " mjw at redhat dot com
2011-11-06 16:09 ` fche at redhat dot com
@ 2011-11-06 17:39 ` fche at redhat dot com
2011-11-10 14:28 ` masami.hiramatsu.pt at hitachi dot com
2012-02-21 15:12 ` fche at redhat dot com
4 siblings, 0 replies; 6+ messages in thread
From: fche at redhat dot com @ 2011-11-06 17:39 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=13193
--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> 2011-11-06 17:22:37 UTC ---
With echo 0 > /proc/sys/debug/kprobes-optimization, the test, executed
in a tight loop with even the "sleep 3;" part removed from bz6503.exp,
runs fine for hours.
# while true; do make installcheck RUNTESTFLAGS=bz6503.exp; done
With echo 1, the test loop fails within a minute or two, on this VM:
Linux vm-f15-64 2.6.40.6-0.fc15.x86_64 #1 SMP Tue Oct 4 00:39:50 UTC 2011
x86_64 x86_64 x86_64 GNU/Linux
2 VCPU, 2GB RAM.
I strongly suspect this is a kernel bug that we'll just have to work
around somehow, perhaps by teaching staprun to turn off kprobes optimization.
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Bug testsuite/13193] Running bz6503.exp results in unresponsive/crashed machine
2011-09-16 13:38 [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine mjw at redhat dot com
` (2 preceding siblings ...)
2011-11-06 17:39 ` fche at redhat dot com
@ 2011-11-10 14:28 ` masami.hiramatsu.pt at hitachi dot com
2012-02-21 15:12 ` fche at redhat dot com
4 siblings, 0 replies; 6+ messages in thread
From: masami.hiramatsu.pt at hitachi dot com @ 2011-11-10 14:28 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=13193
--- Comment #5 from Masami Hiramatsu <masami.hiramatsu.pt at hitachi dot com> 2011-11-10 14:28:08 UTC ---
(In reply to comment #4)
> I strongly suspect this is a kernel bug that we'll just have to work
> around somehow, perhaps by teaching staprun to turn off kprobes optimization.
Hmm, it seems that an aggr probe(which wasn't optimized) was gone and child
probes were also gone, but the aggr probe was not released and on the kprobe's
hash table.
BTW, I'd like to get more information:
- does this happen on x86-64 too?
- does this happen on upstream kernel too?
- what the actual version of systemtap did you use?
- is there any C code of stap code?
I'm not sure why systemtap tries to put a probe on the "TAIL" of a function in
.init.text...
[157018.227386] Warning: found a stray unused aggrprobe@f8942011
[157018.228133] [<f8942011>] ? jffs2_compressors_init+0x11/0x11 [jffs2]
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Bug testsuite/13193] Running bz6503.exp results in unresponsive/crashed machine
2011-09-16 13:38 [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine mjw at redhat dot com
` (3 preceding siblings ...)
2011-11-10 14:28 ` masami.hiramatsu.pt at hitachi dot com
@ 2012-02-21 15:12 ` fche at redhat dot com
4 siblings, 0 replies; 6+ messages in thread
From: fche at redhat dot com @ 2012-02-21 15:12 UTC (permalink / raw)
To: systemtap
http://sourceware.org/bugzilla/show_bug.cgi?id=13193
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> 2012-02-21 15:11:41 UTC ---
commit cd125e94 included kprobes-optimization disabling.
Once lkml upstream fixes the associated problem, and we find
a way of identifying the fixed releases, it can be removed,
or made further conditional. (Note that one can test stap 1.7
against possible kernel fixes with env STAP_PR13193_OVERRIDE=1.)
--
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-02-21 15:12 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-09-16 13:38 [Bug testsuite/13193] New: Running bz6503.exp results in unresponsive/crashed machine mjw at redhat dot com
2011-09-23 11:53 ` [Bug testsuite/13193] " mjw at redhat dot com
2011-11-06 16:09 ` fche at redhat dot com
2011-11-06 17:39 ` fche at redhat dot com
2011-11-10 14:28 ` masami.hiramatsu.pt at hitachi dot com
2012-02-21 15:12 ` fche at redhat dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).