[Bug testsuite/15803] New: systemtap 2.3 testsuite crashes linux 3.10 sometimes at netdev/enter_netfilter_probe_0

[Bug testsuite/15803] New: systemtap 2.3 testsuite crashes linux 3.10 sometimes at netdev/enter_netfilter_probe_0

[timo.lindfors at iki dot fi]

http://sourceware.org/bugzilla/show_bug.cgi?id=15803

            Bug ID: 15803
           Summary: systemtap 2.3 testsuite crashes linux 3.10 sometimes
                    at netdev/enter_netfilter_probe_0
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: testsuite
          Assignee: systemtap at sourceware dot org
          Reporter: timo.lindfors at iki dot fi

Running systemtap 2.3 testsuite as root in a xen domU seems to sometimes crash
the system. Last lines from the testsuite point at netdev:

Running ./systemtap.clone/dtrace_clone.exp ...
Running ./systemtap.clone/dtrace_fork_exec.exp ...
Running ./systemtap.clone/dtrace_vfork_exec.exp ...
Running ./systemtap.clone/main_quiesce.exp ...
Running ./systemtap.clone/probe_by_pid.exp ...
Running ./systemtap.context/context.exp ...
Running ./systemtap.context/fib.exp ...
Running ./systemtap.context/symbols.exp ...
Running ./systemtap.context/uprobe_stmt_num.exp ...
Running ./systemtap.context/uprobe_uaddr.exp ...
Running ./systemtap.context/uprobe_uaddr_mark.exp ...
Running ./systemtap.context/usymbols.exp ...
Running ./systemtap.examples/check.exp ...
FAIL: ./systemtap.examples/general/badname build
FAIL: ./systemtap.examples/io/deviceseeks build
FAIL: ./systemtap.examples/io/ioblktime build
FAIL: ./systemtap.examples/memory/vm.tracepoints build
FAIL: ./systemtap.examples/network/autofs4 build
FAIL: ./systemtap.examples/network/netdev build


Here's also full output from xen's "xm console":

[   75.743192] Kprobes globally unoptimized
[ 7660.976883] jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
[ 7934.136840] NFS: v4 server nfs does not accept raw uid/gids. Reenabling the
idmapper.
[ 7975.750953] Systemtap Error at _stp_mempool_init:50 Memory allocation
failed.
[ 7975.750973] Systemtap Error at _stp_register_ctl_channel:602 Error creating
systemtap control channel.
[ 8346.593943] stap[13589]: segfault at 7fff39881fda ip 00002aaaaacdc7f5 sp
00007fff39881fc0 error 4 in libdw-0.153.so[2aaaaaccf000+34000]
[ 8567.550314] nr_pdflush_threads exported in /proc is scheduled for removal
[ 8567.555212] cat: The scan_unevictable_pages sysctl/node-interface has been
disabled for lack of a legitimate use case.  If you have one, please send an
email to linux-mm@kvack.org.
[ 8769.796054] BUG: unable to handle kernel NULL pointer dereference at
0000000000000280
[ 8769.796067] IP: [<ffffffffa0248c95>] enter_netfilter_probe_0+0x1ce/0x1dc
[stap_f52e2bbf38e566092418a5b32fa1dfc_28280]
[ 8769.796078] PGD 3c542067 PUD 3c06f067 PMD 0 
[ 8769.796083] Oops: 0000 [#1] SMP 
[ 8769.796087] Modules linked in: stap_f52e2bbf38e566092418a5b32fa1dfc_28280(O)
zlib_deflate mtd parport_pc ppdev lp parport cuse binfmt_misc fuse
rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss oid_registry nfs_acl nfs lockd
dns_resolver fscache sunrpc loop evdev snd_pcm_oss snd_mixer_oss snd_pcm
acpi_cpufreq snd_page_alloc snd_timer mperf snd processor thermal_sys soundcore
pcspkr ext3 mbcache jbd virtio_rng rng_core virtio_net virtio_blk
virtio_balloon virtio_pci virtio_ring virtio xen_blkfront xen_netfront [last
unloaded: stap_29eaf70d8ee6f01449c59c1929eff6f_28061]
[ 8769.796138] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O
3.10-1-amd64 #1 Debian 3.10.3-1
[ 8769.796144] task: ffffffff81613400 ti: ffffffff81600000 task.ti:
ffffffff81600000
[ 8769.796149] RIP: e030:[<ffffffffa0248c95>]  [<ffffffffa0248c95>]
enter_netfilter_probe_0+0x1ce/0x1dc
[stap_f52e2bbf38e566092418a5b32fa1dfc_28280]
[ 8769.796158] RSP: e02b:ffff88003f803cd8  EFLAGS: 00010297
[ 8769.796166] RAX: 0000000000000000 RBX: ffffffff816994a0 RCX:
0000000000000000
[ 8769.796174] RDX: ffff880003aa0000 RSI: ffff88003d8f8280 RDI:
0000000000000001
[ 8769.796177] RBP: ffff88003d8f8280 R08: ffffffff812eea8c R09:
ffff88003f803d60
[ 8769.796181] R10: ffffffff8167bf40 R11: 0000000000000001 R12:
0000000000000001
[ 8769.796184] R13: ffff880003aa0000 R14: 0000000000000000 R15:
ffffffff812eea8c
[ 8769.796190] FS:  00002aaaaaae4ec0(0000) GS:ffff88003f800000(0000)
knlGS:0000000000000000
[ 8769.796194] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8769.796198] CR2: 0000000000000280 CR3: 000000003d69a000 CR4:
0000000000000660
[ 8769.796202] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 8769.796205] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 8769.796208] Stack:
[ 8769.796211]  ffffffff8167bf40 0000000000000000 ffffffff816994a0
ffff88003d8f8280
[ 8769.796217]  ffffffff812e9ce2 ffff88003f803d60 ffffffffa024d2b0
ffff88003d8f8280
[ 8769.796223]  ffff88003d8f8280 0000000000000001 ffff880003aa0000
0000000000000000
[ 8769.796229] Call Trace:
[ 8769.796231]  <IRQ> 
[ 8769.796233]  [<ffffffff812e9ce2>] ? nf_iterate+0x41/0x7e
[ 8769.796242]  [<ffffffff812e9d88>] ? nf_hook_slow+0x69/0xfd
[ 8769.796247]  [<ffffffff812eea8c>] ?
__xfrm_policy_check2.constprop.8+0x4c/0x4c
[ 8769.796252]  [<ffffffff812eed67>] ? ip_local_deliver+0x68/0x74
[ 8769.796257]  [<ffffffff812c6526>] ? __netif_receive_skb_core+0x426/0x49c
[ 8769.796262]  [<ffffffff812c6775>] ? netif_receive_skb+0x4c/0x7d
[ 8769.796267]  [<ffffffffa0002749>] ? xennet_poll+0x9d2/0xa07 [xen_netfront]
[ 8769.796272]  [<ffffffff812c6c77>] ? net_rx_action+0xa7/0x1df
[ 8769.796278]  [<ffffffff810421ba>] ? __do_softirq+0xea/0x205
[ 8769.796282]  [<ffffffff8104239f>] ? irq_exit+0x3e/0x81
[ 8769.796287]  [<ffffffff8123d582>] ? xen_evtchn_do_upcall+0x27/0x32
[ 8769.796293]  [<ffffffff8138e77e>] ? xen_do_hypervisor_callback+0x1e/0x30
[ 8769.796296]  <EOI> 
[ 8769.796298]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[ 8769.796305]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[ 8769.796310]  [<ffffffff81006a16>] ? xen_safe_halt+0xc/0x13
[ 8769.796315]  [<ffffffff810133f6>] ? default_idle+0x17/0x3f
[ 8769.796320]  [<ffffffff81072597>] ? cpu_startup_entry+0x10d/0x187
[ 8769.796327]  [<ffffffff816b3d3d>] ? start_kernel+0x3e8/0x3f3
[ 8769.796331]  [<ffffffff816b3777>] ? repair_env_string+0x54/0x54
[ 8769.796335]  [<ffffffff816b59f7>] ? xen_start_kernel+0x4a3/0x4ac
[ 8769.796339] Code: 94 c3 75 0a c7 05 30 5c 00 00 02 00 00 00 e8 94 15 00 00
48 85 db 74 0e 48 c7 c7 c2 bd 24 a0 31 c0 e8 dc fd ff ff 48 8b 44 24 08 <48> 8b
80 80 02 00 00 48 83 c4 10 5b 5d c3 53 48 83 ec 10 48 c7 
[ 8769.796377] RIP  [<ffffffffa0248c95>] enter_netfilter_probe_0+0x1ce/0x1dc
[stap_f52e2bbf38e566092418a5b32fa1dfc_28280]
[ 8769.796384]  RSP <ffff88003f803cd8>
[ 8769.796386] CR2: 0000000000000280
[ 8769.796402] ---[ end trace 0bacda9529c62d93 ]---
[ 8769.796406] Kernel panic - not syncing: Fatal exception in interrupt
[ 8769.796445] Stap trace buffer for processor 0 sub-buffer 0:
[ 8769.796455] Dropping packets! Ctrl-C to exit.
[ 8769.796460] Stap trace buffer for processor 1 sub-buffer 0:
[ 8769.796463] Dropping packets! Ctrl-C to exit.
[ 8769.796466] Stap trace buffer for processor 2 sub-buffer 0:
[ 8769.796470] Dropping packets! Ctrl-C to exit.
[ 8769.796473] Stap trace buffer for processor 3 sub-buffer 0:
[ 8769.796476] Dropping packets! Ctrl-C to exit.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/15803] systemtap 2.3 testsuite crashes linux 3.10 sometimes at netdev/enter_netfilter_probe_0

[timo.lindfors at iki dot fi]

http://sourceware.org/bugzilla/show_bug.cgi?id=15803

--- Comment #1 from Timo Juhani Lindfors <timo.lindfors at iki dot fi> ---
I can reproduce this if I use

ssh lindi3 cat /dev/zero > /dev/null

and then ssh to lindi3 to run

sudo stap -g ./testsuite/systemtap.examples/network/netfilter_drop.stp TCP 1 -c
"sleep 5"

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/15803] systemtap 2.3 testsuite crashes linux 3.10 sometimes at netdev/enter_netfilter_probe_0

[timo.lindfors at iki dot fi]

http://sourceware.org/bugzilla/show_bug.cgi?id=15803

--- Comment #2 from Timo Juhani Lindfors <timo.lindfors at iki dot fi> ---
With

sudo stap -m testcase -B CONFIG_DEBUG_INFO=y -g
./testsuite/systemtap.examples/network/netfilter_drop.stp TCP 1 -c "sleep 5"

I see

[  433.252967] BUG: unable to handle kernel NULL pointer dereference at
0000000000000280
[  433.252987] IP: [<ffffffffa0219c95>] enter_netfilter_probe_0+0x1ce/0x1dc
[testcase]
[  433.253000] PGD 0 
[  433.253003] Oops: 0000 [#1] SMP 
[  433.253007] Modules linked in: testcase(O) parport_pc ppdev lp parport cuse
binfmt_misc fuse rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss oid_registry nfs_acl
nfs lockd dns_resolver fscache sunrpc loop evdev snd_pcm_oss snd_mixer_oss
snd_pcm snd_page_alloc acpi_cpufreq snd_timer mperf processor snd thermal_sys
soundcore pcspkr ext3 mbcache jbd virtio_rng rng_core virtio_net virtio_blk
virtio_balloon virtio_pci virtio_ring virtio xen_blkfront xen_netfront [last
unloaded: testcase]
[  433.253056] CPU: 0 PID: 4186 Comm: sshd Tainted: G           O 3.10-1-amd64
#1 Debian 3.10.3-1
[  433.253062] task: ffff88003c73f100 ti: ffff88003c000000 task.ti:
ffff88003c000000
[  433.253067] RIP: e030:[<ffffffffa0219c95>]  [<ffffffffa0219c95>]
enter_netfilter_probe_0+0x1ce/0x1dc [testcase]
[  433.253078] RSP: e02b:ffff88003f803cd8  EFLAGS: 00010297
[  433.253082] RAX: 0000000000000000 RBX: ffffffff816994a0 RCX:
0000000000000000
[  433.253086] RDX: ffff880003b10000 RSI: ffff88003e38d080 RDI:
0000000000000001
[  433.253092] RBP: ffff88003e38d080 R08: ffffffff812eea8c R09:
ffff88003f803d60
[  433.253097] R10: ffffffff8167bf40 R11: 0000000000000001 R12:
0000000000000001
[  433.253101] R13: ffff880003b10000 R14: 0000000000000000 R15:
ffffffff812eea8c
[  433.253112] FS:  00007f2ac2850800(0000) GS:ffff88003f800000(0000)
knlGS:0000000000000000
[  433.253119] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  433.253123] CR2: 0000000000000280 CR3: 000000003c0a5000 CR4:
0000000000000660
[  433.253129] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  433.253133] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[  433.253137] Stack:
[  433.253140]  ffffffff8167bf40 0000000000000000 ffffffff816994a0
ffff88003e38d080
[  433.253147]  ffffffff812e9ce2 ffff88003f803d60 ffffffffa021e2b0
ffff88003e38d080
[  433.253155]  ffff88003e38d080 0000000000000001 ffff880003b10000
0000000000000000
[  433.253164] Call Trace:
[  433.253167]  <IRQ> 
[  433.253169]  [<ffffffff812e9ce2>] ? nf_iterate+0x41/0x7e
[  433.253179]  [<ffffffff812e9d88>] ? nf_hook_slow+0x69/0xfd
[  433.253186]  [<ffffffff812eea8c>] ?
__xfrm_policy_check2.constprop.8+0x4c/0x4c
[  433.253193]  [<ffffffff812eed67>] ? ip_local_deliver+0x68/0x74
[  433.253201]  [<ffffffff812c6526>] ? __netif_receive_skb_core+0x426/0x49c
[  433.253206]  [<ffffffff812c6775>] ? netif_receive_skb+0x4c/0x7d
[  433.253213]  [<ffffffffa0002749>] ? xennet_poll+0x9d2/0xa07 [xen_netfront]
[  433.253219]  [<ffffffff812c6c77>] ? net_rx_action+0xa7/0x1df
[  433.253225]  [<ffffffff810421ba>] ? __do_softirq+0xea/0x205
[  433.253230]  [<ffffffff8104239f>] ? irq_exit+0x3e/0x81
[  433.253235]  [<ffffffff8123d582>] ? xen_evtchn_do_upcall+0x27/0x32
[  433.253241]  [<ffffffff8138e77e>] ? xen_do_hypervisor_callback+0x1e/0x30
[  433.253245]  <EOI> 
[  433.253247] Code: 94 c3 75 0a c7 05 30 5c 00 00 02 00 00 00 e8 94 15 00 00
48 85 db 74 0e 48 c7 c7 e4 cf 21 a0 31 c0 e8 dc fd ff ff 48 8b 44 24 08 <48> 8b
80 80 02 00 00 48 83 c4 10 5b 5d c3 53 48 83 ec 10 48 c7 
[  433.253293] RIP  [<ffffffffa0219c95>] enter_netfilter_probe_0+0x1ce/0x1dc
[testcase]
[  433.253300]  RSP <ffff88003f803cd8>
[  433.253303] CR2: 0000000000000280
[  433.253312] ---[ end trace 892021ee0971cf5e ]---

"objdump -drS testcase.ko" shows

...
static inline void atomic_set(atomic_t *v, int i)
{
        v->counter = i;
    3c6e:       c7 05 00 00 00 00 02    movl   $0x2,0x0(%rip)        # 3c78
<enter_netfilter_probe_0+0x1b1>
    3c75:       00 00 00 
                        3c70: R_X86_64_PC32     .bss+0x1360
    3c78:       e8 00 00 00 00          callq  3c7d
<enter_netfilter_probe_0+0x1b6>
                        3c79: R_X86_64_PC32     .text.unlikely+0x4
    3c7d:       48 85 db                test   %rbx,%rbx
    3c80:       74 0e                   je     3c90
<enter_netfilter_probe_0+0x1c9>
    3c82:       48 c7 c7 00 00 00 00    mov    $0x0,%rdi
                        3c85: R_X86_64_32S      .rodata.str1.1+0xc5c
    3c89:       31 c0                   xor    %eax,%eax
    3c8b:       e8 dc fd ff ff          callq  3a6c <_stp_error>
    3c90:       48 8b 44 24 08          mov    0x8(%rsp),%rax
    3c95:       48 8b 80 80 02 00 00    mov    0x280(%rax),%rax
    3c9c:       48 83 c4 10             add    $0x10,%rsp
    3ca0:       5b                      pop    %rbx
    3ca1:       5d                      pop    %rbp
    3ca2:       c3                      retq   
...

and the faulting instruction is at 0x3c95.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/15803] systemtap 2.3 testsuite crashes linux 3.10 sometimes at netdev/enter_netfilter_probe_0

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15803

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |fche at redhat dot com
          Component|testsuite                   |translator
         Resolution|---                         |FIXED

--- Comment #3 from Frank Ch. Eigler <fche at redhat dot com> ---
commit 6b5871b

-- 
You are receiving this mail because:
You are the assignee for the bug.