* systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
@ 2013-05-17 11:11 Timo Juhani Lindfors
2013-05-17 12:21 ` Frank Ch. Eigler
0 siblings, 1 reply; 12+ messages in thread
From: Timo Juhani Lindfors @ 2013-05-17 11:11 UTC (permalink / raw)
To: systemtap
Hi,
systemtap 2.2.1 installcheck gets stuck on my debian amd64 unstable
system with linux 3.8 as xen domU.
Here's full dmesg and ps axuf:
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.8-1-amd64 (debian-kernel@lists.debian.org) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Debian 3.8.12-1
[ 0.000000] Command line: root=UUID=ee66b917-5151-4a4b-944a-99385b7d4d62 ro
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x00000000407fffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI not present or invalid.
[ 0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] No AGP bridge found
[ 0.000000] e820: last_pfn = 0x40800 max_arch_pfn = 0x400000000
[ 0.000000] initial memory mapped: [mem 0x00000000-0x03bc9fff]
[ 0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x407fffff]
[ 0.000000] [mem 0x00000000-0x407fffff] page 4k
[ 0.000000] kernel direct mapping tables up to 0x407fffff @ [mem 0x01a26000-0x01c2cfff]
[ 0.000000] xen: setting RW the range 1c0b000 - 1c2d000
[ 0.000000] RAMDISK: [mem 0x01c2d000-0x03bc9fff]
[ 0.000000] NUMA turned off
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000000407fffff]
[ 0.000000] Initmem setup node 0 [mem 0x00000000-0x407fffff]
[ 0.000000] NODE_DATA [mem 0x3fffc000-0x3fffffff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00010000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00010000-0x0009ffff]
[ 0.000000] node 0: [mem 0x00100000-0x407fffff]
[ 0.000000] On node 0 totalpages: 264080
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 6 pages reserved
[ 0.000000] DMA zone: 3922 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 3556 pages used for memmap
[ 0.000000] DMA32 zone: 256540 pages, LIFO batch:31
[ 0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[ 0.000000] No local APIC present
[ 0.000000] APIC: disable apic facility
[ 0.000000] APIC: switched to apic NOOP
[ 0.000000] nr_irqs_gsi: 16
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[ 0.000000] e820: [mem 0x40800000-0xffffffff] available for PCI devices
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 4.1.3 (preserve-AD)
[ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88003f800000 s84096 r8192 d22400 u524288
[ 0.000000] pcpu-alloc: s84096 r8192 d22400 u524288 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 260462
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: root=UUID=ee66b917-5151-4a4b-944a-99385b7d4d62 ro
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] __ex_table already sorted, skipping sort
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] Calgary: detecting Calgary via BIOS EBDA area
[ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[ 0.000000] Memory: 986408k/1056768k available (3632k kernel code, 448k absent, 69912k reserved, 3106k data, 616k init)
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=4.
[ 0.000000] NR_IRQS:33024 nr_irqs:304 16
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] Xen: using vcpuop timer interface
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] tsc: Detected 3210.882 MHz processor
[ 0.000000] tsc: Marking TSC unstable due to TSCs unsynchronized
[ 0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 6421.76 BogoMIPS (lpj=12843528)
[ 0.004000] pid_max: default: 32768 minimum: 301
[ 0.004000] Security Framework initialized
[ 0.004000] AppArmor: AppArmor disabled by boot time parameter
[ 0.004000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.004000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.004000] Mount-cache hash table entries: 256
[ 0.004000] Initializing cgroup subsys cpuacct
[ 0.004000] Initializing cgroup subsys memory
[ 0.004000] Initializing cgroup subsys devices
[ 0.004000] Initializing cgroup subsys freezer
[ 0.004000] Initializing cgroup subsys net_cls
[ 0.004000] Initializing cgroup subsys blkio
[ 0.004000] Initializing cgroup subsys perf_event
[ 0.004000] tseg: 0000000000
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 4
[ 0.004000] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8
[ 0.004000] Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64
[ 0.004000] tlb_flushall_shift: 4
[ 0.008220] Performance Events:
[ 0.008230] no APIC, boot with the "lapic" boot parameter to force-enable it.
[ 0.008235] no hardware sampling interrupt available.
[ 0.008248] Broken PMU hardware detected, using software events only.
[ 0.008252] Failed to access perfctr msr (MSR c0010004 is 35159cd74955)
[ 0.008447] NMI watchdog: disabled (cpu0): hardware events not enabled
[ 0.008539] installing Xen timer for CPU 1
[ 0.008583] SMP alternatives: switching to SMP code
[ 0.015700] installing Xen timer for CPU 2
[ 0.015946] installing Xen timer for CPU 3
[ 0.016008] Brought up 4 CPUs
[ 0.016097] devtmpfs: initialized
[ 0.018041] Grant tables using version 2 layout.
[ 0.018041] Grant table initialized
[ 0.018041] regulator-dummy: no parameters
[ 0.018041] NET: Registered protocol family 16
[ 0.018041] PCI: setting up Xen PCI frontend stub
[ 0.018041] PCI: pci_cache_line_size set to 64 bytes
[ 0.020118] bio: create slab <bio-0> at 0
[ 0.020175] ACPI: Interpreter disabled.
[ 0.020175] xen/balloon: Initialising balloon driver.
[ 0.020175] xen-balloon: Initialising balloon driver.
[ 0.020175] vgaarb: loaded
[ 0.020175] PCI: System does not support PCI
[ 0.020175] PCI: System does not support PCI
[ 0.020175] Switching to clocksource xen
[ 0.021602] pnp: PnP ACPI: disabled
[ 0.024520] NET: Registered protocol family 2
[ 0.024716] TCP established hash table entries: 8192 (order: 5, 131072 bytes)
[ 0.024783] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[ 0.024816] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.024852] TCP: reno registered
[ 0.024860] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.024870] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.024953] NET: Registered protocol family 1
[ 0.024962] PCI: CLS 0 bytes, default 64
[ 0.024997] Unpacking initramfs...
[ 0.053191] Freeing initrd memory: 32372k freed
[ 0.061525] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 0.061895] audit: initializing netlink socket (disabled)
[ 0.061911] type=2000 audit(1368740817.784:1): initialized
[ 0.072002] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.072339] VFS: Disk quotas dquot_6.5.2
[ 0.072375] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.072461] msgmni has been set to 1989
[ 0.072795] alg: No test for stdrng (krng)
[ 0.072845] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 0.072897] io scheduler noop registered
[ 0.072901] io scheduler deadline registered
[ 0.072927] io scheduler cfq registered (default)
[ 0.072981] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 0.072996] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 0.073000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.198093] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.198465] Linux agpgart interface v0.103
[ 0.198558] i8042: PNP: No PS/2 controller found. Probing ports directly.
[ 1.196728] i8042: No controller found
[ 1.196932] mousedev: PS/2 mouse device common for all mice
[ 1.256633] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[ 1.256688] rtc_cmos: probe of rtc_cmos failed with error -38
[ 1.256754] drop_monitor: Initializing network drop monitor service
[ 1.256815] TCP: cubic registered
[ 1.256886] NET: Registered protocol family 10
[ 1.257052] mip6: Mobile IPv6
[ 1.257057] NET: Registered protocol family 17
[ 1.257230] PM: Hibernation image not present or could not be loaded.
[ 1.257240] registered taskstats version 1
[ 1.257257] XENBUS: Device with no driver: device/vbd/51712
[ 1.257261] XENBUS: Device with no driver: device/vif/0
[ 1.257421] /build/buildd-linux_3.8.12-1-amd64-RaG_7r/linux-3.8.12/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[ 1.257675] Freeing unused kernel memory: 616k freed
[ 1.257799] Write protecting the kernel read-only data: 6144k
[ 1.259424] Freeing unused kernel memory: 452k freed
[ 1.259673] Freeing unused kernel memory: 524k freed
[ 1.283770] udevd[71]: starting version 175
[ 1.302231] Initialising Xen virtual ethernet driver.
[ 1.343497] blkfront: xvda: barrier: enabled
[ 1.344889] xvda: unknown partition table
[ 1.470542] kjournald starting. Commit interval 5 seconds
[ 1.470561] EXT3-fs (xvda): mounted filesystem with ordered data mode
[ 2.149376] udevd[269]: starting version 175
[ 2.379090] input: PC Speaker as /devices/platform/pcspkr/input/input0
[ 2.451016] Error: Driver 'pcspkr' is already registered, aborting...
[ 3.191955] EXT3-fs (xvda): using internal journal
[ 3.302417] loop: module loaded
[ 4.774422] RPC: Registered named UNIX socket transport module.
[ 4.774438] RPC: Registered udp transport module.
[ 4.774442] RPC: Registered tcp transport module.
[ 4.774445] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 4.793494] FS-Cache: Loaded
[ 4.793911] Key type dns_resolver registered
[ 4.804770] FS-Cache: Netfs 'nfs' registered for caching
[ 4.836789] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 4.910960] NFS: Registering the id_resolver key type
[ 4.910980] Key type id_resolver registered
[ 4.910983] Key type id_legacy registered
[ 5.355311] fuse init (API version 7.20)
[ 49.942648] stap_5cf49ce5b5dbed1c6d5e2523af0293b4__2758: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 19data/24text/33ctx/2058net/33alloc kb, probes: 1
[34559.494867] NFS: v4 server nfs does not accept raw uid/gids. Reenabling the idmapper.
[34677.152392] stap_c0ee9842da2dfdbd2d657ccaffe7681_6594: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34679.621815] stap_79710b96b9ad6d6755a8b11068559e6f_6775: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 4
[34681.888789] stap_655b4e599ec4f3d548ede9329a3d51c_6956: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34684.329124] stap_36be3eba8deb5b5a549e1ed9c64e257d_7137: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 4
[34686.774823] stap_c980f19b5b8d0db65c622b9399456d4_7318: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34689.210169] stap_26807663cf3bb71f370274e7d04e7b0_7499: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34691.814735] stap_3a22e75c69126633448d5dcd84245b53_7680: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[34696.793452] stap_4d76d8a7898934ae3bb8a41ff6f4f743_7873: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 19data/24text/142ctx/2058net/33alloc kb, probes: 3
[34700.179229] stap_47ffb2d5d65e0cffbb5a4d372c40fbbf_8052: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 19data/24text/79ctx/2058net/33alloc kb, probes: 2
[34702.597293] stap_c4f4358ecd20ef4d8ce70bc410e90078_8231: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/20text/2ctx/2058net/38alloc kb, probes: 4
[34705.174421] stap_f65071e57c788a0e566765ea37406667_8412: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 23data/20text/16ctx/2058net/66alloc kb, probes: 2
[34707.729237] stap_3ff5408eaed74b2d28097f54aa66fda0_8599: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34709.983403] stap_e5b13df3371671705ce8e85168201dc8_8779: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34712.230449] stap_68f8f19740ebf97271f5e5e48d4d939d_8959: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34714.476328] stap_e88c6335d330e5e0510f20642be57c02_9139: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34716.680010] stap_45372b967a9a452d27b042162a7f1208_9319: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34718.901108] stap_688b2d7cd28fa6debfc0c75666e9326a_9499: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34721.566070] stap_16bcfe9497a1ce315b256f0b98687f7f_9684: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34723.802281] stap_82ef7bd982449db05807cf3d0cbf17c6_9864: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34726.624843] stap_08f8f1b523d1cbf0c067f6080b023db_10044: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 24data/32text/22ctx/2058net/33alloc kb, probes: 5
[34734.509905] stap_a69da0c58a6009fb3f4ced6c6417426_10223: systemtap: 2.2.1/0.153, base: ffffffffa0217000, memory: 3716data/36text/22ctx/2058net/33alloc kb, probes: 5
[34742.820894] stap_e3b9d4bd3f738b0875ee2bd35cb93f5_10574: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 19data/20text/2ctx/2058net/33alloc kb, probes: 4
[34745.503681] stap_bdf12d855375397aec423bb33da5894_10753: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/20text/2ctx/2058net/33alloc kb, probes: 4
[34748.164616] stap_e977d6f690fda6f255120c82ef1269b_10932: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 19data/20text/2ctx/2058net/33alloc kb, probes: 4
[34750.965384] stap_366c39f8510b3d42ccfebe81a926642_11111: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/20text/2ctx/2058net/33alloc kb, probes: 4
[34753.495520] stap_77bff416bc09a25505acd904d2293c3_11290: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 23data/16text/6ctx/2058net/33alloc kb, probes: 14
[34755.983832] stap_ed56a0a22ea3c2f4665c1e0227b6117_11471: systemtap: 2.2.1/0.153, base: ffffffffa01e5000, memory: 15data/16text/10ctx/2058net/33alloc kb, probes: 8
[34758.718446] stap_d67c15b40f97a7b5842c9fcba993dcd7__11650: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 104data/20text/14ctx/2058net/2416alloc kb, probes: 564
[34762.282578] stap_77773d42edcc6cd6fbf77d51c5b6bbb_11829: systemtap: 2.2.1/0.153, base: ffffffffa01ae000, memory: 19data/20text/4ctx/2058net/33alloc kb, probes: 1
[34766.612617] stap_f8fee0b58272a433efc05108079cce21__12031: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 299data/20text/2ctx/2058net/143alloc kb, probes: 562
[34766.631540] jffs2: version 2.2. (NAND) (SUMMARY) \xffffffc2\xffffffa9\xffffffa9 2001-2006 Red Hat, Inc.
[34818.795804] stap_5c65f103d32bbbeaf8b47e418a6f42cc_15284: systemtap: 2.2.1/0.153, base: ffffffffa0258000, memory: 19data/20text/2ctx/2058net/33alloc kb, probes: 1
[34822.294725] stap_f5d2de97f52dab4ce641df1abe5ec9d_15462: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/20text/16ctx/2058net/33alloc kb, probes: 1
[34824.532260] stap_73aa1a70351481fe9bdfc79c9392ff_15641: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34825.109825] stap_73aa1a70351481fe9bdfc79c9392ff_15652: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34825.656950] stap_73aa1a70351481fe9bdfc79c9392ff_15663: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34826.200690] stap_73aa1a70351481fe9bdfc79c9392ff_15675: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34826.746166] stap_73aa1a70351481fe9bdfc79c9392ff_15687: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34827.285628] stap_73aa1a70351481fe9bdfc79c9392ff_15699: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34827.819538] stap_73aa1a70351481fe9bdfc79c9392ff_15710: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34831.474951] stap_7d0382035f8d414499bea1e3a2742c_15895: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/6ctx/2058net/66alloc kb, probes: 2
[34833.878605] stap_17667f37462b23eeb11063994bdf8a0_16074: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/6ctx/2058net/68alloc kb, probes: 2
[34836.229872] stap_b97f13ff6ba446a3f3109d4bc0d2c75_16253: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/6ctx/2058net/68alloc kb, probes: 2
[34838.641526] stap_105f91fdcad9604e3098c6efee2a73b_16432: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/6ctx/2058net/66alloc kb, probes: 2
[34855.987170] stap_c4d1282fc8793dde4d731d0d3948639_17338: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/16text/54ctx/2058net/33alloc kb, probes: 4
[34858.460639] stap_b42352b97aca714bb3d9ccdc6a32428_17517: systemtap: 2.2.1/0.153, base: ffffffffa0211000, memory: 19data/16text/118ctx/2058net/33alloc kb, probes: 4
[34860.780634] stap_35d8f5cb73642ca3b990d81ef275771_17700: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/16text/11ctx/2058net/33alloc kb, probes: 4
[34863.066730] stap_84e53eb7f48cb4d96763fc388df2c90_17879: systemtap: 2.2.1/0.153, base: ffffffffa0211000, memory: 19data/16text/115ctx/2058net/33alloc kb, probes: 4
[34865.437686] stap_523cccd3acac5444b2def4131e77f38_18056: systemtap: 2.2.1/0.153, base: ffffffffa021b000, memory: 19data/20text/16ctx/2058net/33alloc kb, probes: 1
[34870.347152] stap_cc37b6f66eb67c2717d1b39901ac5ee_18364: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/24text/10ctx/2058net/33alloc kb, probes: 3
[34872.998573] stap_2c675783be205c5b1cbe9e7d81d3ef_18545: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34876.315270] stap_00583de2099d90f421e4077b82db817_18764: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/2ctx/2058net/33alloc kb, probes: 3
[34878.884407] stap_15a8389e88b936065c07b3f02b2b6d2_18945: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/10ctx/2058net/33alloc kb, probes: 2
[34883.279531] stap_6122d71231f1dcd824eb2215b09afe6_19228: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/2ctx/2058net/33alloc kb, probes: 3
[34884.070769] stap_73aa1a70351481fe9bdfc79c9392ff_19241: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34890.087501] stap_fbbe1baa4d07b1dabd1feb9d185d3a_19432: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[34900.746349] stap_d600a8393587dd41c98c9d0e8f36914_19631: systemtap: 2.2.1/0.153, base: ffffffffa0210000, memory: 19data/20text/2ctx/8202net/33alloc kb, probes: 1
[34910.162989] stap_73aa1a70351481fe9bdfc79c9392ff_19663: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/2ctx/2058net/33alloc kb, probes: 1
[34920.208971] stap_d12aa61185eb135a1676c7f8c10630_19859: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 14data/16text/2ctx/8202net/33alloc kb, probes: 1
[34932.714627] stap_b6a695604d624e4c12033ac708a7931_20051: systemtap: 2.2.1/0.153, base: ffffffffa0210000, memory: 20data/28text/3ctx/2058net/918alloc kb, probes: 1
[34935.364385] stap_c48b0fa03286b4cf45bf164e09fff67_20228: systemtap: 2.2.1/0.153, base: ffffffffa021e000, memory: 20data/24text/12ctx/2058net/12427alloc kb, probes: 1
[34938.166320] stap_6219f8e8a1ef89ac47d94686582e11c_20409: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 24data/28text/10ctx/2058net/10486alloc kb, probes: 7
[34940.772688] stap_e6e4ac245c37d9b8b8c7980aeb39219_20588: systemtap: 2.2.1/0.153, base: ffffffffa0216000, memory: 19data/24text/43ctx/2058net/10259alloc kb, probes: 1
[34943.496332] stap_0f8f3c8116ef378d7d671a2b09a786e_20767: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/20text/6ctx/2058net/2306alloc kb, probes: 8
[34945.825496] stap_c0d50e4b140288f056aca9b8e5c16f3_20946: systemtap: 2.2.1/0.153, base: ffffffffa0212000, memory: 23data/16text/6ctx/2058net/33alloc kb, probes: 4
[34948.324581] stap_9ccfa427f8e748ab6fedd344170c18_21127: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[34950.654776] stap_35bdc2dfcead9832f1bb8e09a68ccec_21306: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/20text/3ctx/2058net/34alloc kb, probes: 3
[34955.043946] stap_d838da31e69172e9e9e3d4cac3011fb2_21623: systemtap: 2.2.1/0.153, base: ffffffffa0211000, memory: 23data/20text/16ctx/2058net/33alloc kb, probes: 2
[34957.535068] stap_6d3200bffc69b294eaedd6e22ff074f_21805: systemtap: 2.2.1/0.153, base: ffffffffa021d000, memory: 23data/20text/10ctx/2058net/66alloc kb, probes: 4
[34960.997756] stap_e09054acfdc18f448f8ac5d8cffa591_22001: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/2ctx/2058net/34alloc kb, probes: 1
[34962.259769] stap_e09054acfdc18f448f8ac5d8cffa591_22014: systemtap: 2.2.1/0.153, base: ffffffffa0213000, memory: 23data/20text/2ctx/2058net/34alloc kb, probes: 1
[34964.477741] stap_e09054acfdc18f448f8ac5d8cffa591_22027: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/2ctx/2058net/34alloc kb, probes: 1
[34977.059111] stap_8bae191ee3999227d19cef207fd91d_22212: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34979.622713] stap_e2585f68284e9fffe0565a5982924b7_22400: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[34989.061344] stap_58e9b2c7e6d7562e8193d814641cedc5_23035: systemtap: 2.2.1/0.153, base: ffffffffa0210000, memory: 24data/28text/32ctx/2058net/1170alloc kb, probes: 3
[34994.092590] stap_74a7f2a625d3fe5c6077cc9bd81634e0_23220: systemtap: 2.2.1/0.153, base: ffffffffa021f000, memory: 24data/32text/32ctx/2058net/1170alloc kb, probes: 4
[34997.754320] stap_c907289e1ce45352e7984eaca7f724f_23419: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/2ctx/2058net/33alloc kb, probes: 1
[35007.827122] stap_8c77763f9ddf5aa0e660f993e6341e1_23946: systemtap: 2.2.1/0.153, base: ffffffffa0213000, memory: 19data/20text/2ctx/2058net/33alloc kb, probes: 3
[35011.617064] stap_8c1a185686061670c1e237c4eeeff6c_24127: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/8ctx/2058net/33alloc kb, probes: 4
[35014.222036] stap_4d061ae9de7516406eaa47fab7eb60c_24311: systemtap: 2.2.1/0.153, base: ffffffffa0213000, memory: 19data/20text/4ctx/2058net/33alloc kb, probes: 3
[35017.555726] stap_a238c55ee7c3f19ba3d228a7d41e236_24492: systemtap: 2.2.1/0.153, base: ffffffffa021e000, memory: 24data/24text/34ctx/2058net/33alloc kb, probes: 4
[35019.934395] stap_933538ad86ef7df52da61d934e5c6d2_24676: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/20text/2ctx/2058net/33alloc kb, probes: 1
[35027.398473] stap_f55a34593347de1f78133f94c31b036_24912: systemtap: 2.2.1/0.153, base: ffffffffa0211000, memory: 19data/16text/2ctx/2058net/33alloc kb, probes: 2
[35030.815982] stap_acb4ca43332768100ca0f21ab76922b_25098: systemtap: 2.2.1/0.153, base: ffffffffa021b000, memory: 23data/20text/6ctx/2058net/33alloc kb, probes: 5
[35034.912411] stap_5604e5ab0b43797ff9790c999acfb37_25279: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/6ctx/2058net/33alloc kb, probes: 5
[35038.996421] stap_06fe46edc563fd6c475511abbac97b4_25460: systemtap: 2.2.1/0.153, base: ffffffffa0213000, memory: 19data/20text/6ctx/2058net/1170alloc kb, probes: 5
[35042.098250] Systemtap Error at _stp_mempool_init:50 Memory allocation failed.
[35042.098270] Systemtap Error at _stp_register_ctl_channel:597 Error creating systemtap control channel.
[35044.164026] stap_2145e946ed485a9e1969713321533d33_1072: systemtap: 2.2.1/0.153, base: ffffffffa0212000, memory: 19data/16text/6ctx/2058net/33alloc kb, probes: 1
[35049.660235] stap_2211af27f1e4ed21a49c8495b7f20b0_26170: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35053.252979] stap_dbedcec505dad33ee8d62c663ce89cf_26349: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35056.801489] stap_7ef08219f50a47c20eb5e4400ba8441_26528: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35060.388648] stap_fce7f242e70ed1a6890ac6a4e38cccf_26707: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35063.900019] stap_31daf5081e663c01b70def2fb9b4a3e_26886: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35067.479678] stap_1abf5f9af81c471deca09a292aa9131_27065: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35070.957082] stap_bb9e3e0376cc5cc3a4860f68bc24126_27244: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35074.465397] stap_a84dabaa3938f82e7ef904b8ecf2aea_27423: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35077.982421] stap_97e1e43d6cec284111baafac2723e19_27602: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35081.542933] stap_4aea277b76906e908429e241b966937_27782: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35085.145487] stap_ceac30a2c7bdf10e8baa03eb6421f9a_27965: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35088.686897] stap_6e64557e409183b293c9a69f4507dcf_28144: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35092.245546] stap_b9bc862a01659ad374a40cf9fdd099b_28323: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35095.827785] stap_aed78c6f08bbed1c46c5d7ba3274f17_28502: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35099.394668] stap_3e7cb873d91b6e847bb7d9449569a2c_28681: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 3
[35101.764487] stap_e4949dc8e4452c26ea00332b0d8aa1_28864: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35105.377405] stap_d615c222af0e0f66b810a3fc0d7d665_29045: systemtap: 2.2.1/0.153, base: ffffffffa0210000, memory: 24data/28text/8ctx/2058net/33alloc kb, probes: 10
[35109.449892] stap_f4b3adb95c0201d526e4342b3da11ba_29240: systemtap: 2.2.1/0.153, base: ffffffffa021f000, memory: 15data/20text/2ctx/2058net/162alloc kb, probes: 2
[35112.170986] stap_9abfefbbef7e45cd318aa677d4dfa0a_29421: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/20text/10ctx/2058net/33alloc kb, probes: 3
[35115.833347] stap_0fb8820f58dac1b7262328bde5ec2a0_29604: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35118.460123] stap_f60380f08b289ac5c6f76a00ac4ba82_29783: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 20data/24text/3ctx/2058net/232alloc kb, probes: 4
[35119.672303] stap_06fe46edc563fd6c475511abbac97b4_29793: systemtap: 2.2.1/0.153, base: ffffffffa0214000, memory: 19data/20text/6ctx/2058net/1170alloc kb, probes: 5
[35122.804555] stap_03d7c5e341c164ab4f29bc23e183dbb_29973: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/20text/6ctx/2058net/1170alloc kb, probes: 5
[35125.888602] stap_672459c5d50b435ca000f17a7cfd770_30153: systemtap: 2.2.1/0.153, base: ffffffffa0212000, memory: 19data/20text/6ctx/2058net/1170alloc kb, probes: 5
[35130.798122] stap_b44a58357c7870acd2f2aea7a3c902b_30383: systemtap: 2.2.1/0.153, base: ffffffffa021d000, memory: 30data/32text/2ctx/2058net/33alloc kb, probes: 4
[35133.964647] stap_5dadf30f85fb7be43bea69971e36a2_30563: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 30data/32text/2ctx/2058net/33alloc kb, probes: 2
[35138.820902] stap_9ea0ee85f68f8577c5d0192c342c5aad_30772: systemtap: 2.2.1/0.153, base: ffffffffa0218000, memory: 19data/20text/16ctx/2058net/33alloc kb, probes: 1
[35144.401571] stap_7483f794e35a6a4b2bc4c64a3dadbcd3_30954: systemtap: 2.2.1/0.153, base: ffffffffa0223000, memory: 2063data/40text/18ctx/2058net/2194alloc kb, probes: 46
[35150.689807] stap_6296cc9d7f272cd65e57d60236d50b02_31137: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 24data/24text/10ctx/2058net/5958alloc kb, probes: 4
[35154.087488] stap_344ca976162de99a2db247f0c4ee2711_31319: systemtap: 2.2.1/0.153, base: ffffffffa0215000, memory: 24data/24text/10ctx/2058net/5958alloc kb, probes: 4
[35160.317462] stap_41c463231bf83630c7fe8d417c67f8f8_20771: systemtap: 2.2.1/0.153, base: ffffffffa0223000, memory: 2079data/44text/62ctx/2058net/418alloc kb, probes: 86
[35175.049366] stap_39038c374dec05d79756e8b338bb48c_31910: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/20text/6ctx/2058net/33alloc kb, probes: 3
[35177.515710] stap_3ff7322557fcbeaf7263f1ae2ee1c48_32089: systemtap: 2.2.1/0.153, base: ffffffffa0211000, memory: 19data/20text/6ctx/2058net/34alloc kb, probes: 3
[35179.906462] stap_187d61a98be13b522f2e252d6fd40e5_32268: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/20text/6ctx/2058net/33alloc kb, probes: 3
[35186.355131] stap_5d21913090ef5f8a237cfd20ca6fe83_32447: systemtap: 2.2.1/0.153, base: ffffffffa0211000, memory: 19data/20text/6ctx/2058net/34alloc kb, probes: 3
[35196.993843] stap_4bbf1df165293631787b8ffab8bb879b_32626: systemtap: 2.2.1/0.153, base: ffffffffa021c000, memory: 1744data/32text/22ctx/2058net/33alloc kb, probes: 120
[35203.516258] stap_68f654da6df6898f9758550c280e330a_5_338: systemtap: 2.2.1/0.153, base: ffffffffa03da000, memory: 1744data/32text/22ctx/2058net/33alloc kb, probes: 120
[35210.115353] stap_29dfbf0460640a21f34aa228219419af_6_521: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 1744data/32text/22ctx/2058net/33alloc kb, probes: 120
[35213.688356] stap_767bdb7a1df84754a58a6929b04d6aed_710: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35216.336243] stap_0b64ea30bfda9fdcf67f324c9cd1465b__897: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/20text/16ctx/2058net/33alloc kb, probes: 2
[35221.740569] stap_67659cc158f61533e123e37dd36b7d68_1082: systemtap: 2.2.1/0.153, base: ffffffffa0212000, memory: 2047data/36text/12ctx/2058net/33alloc kb, probes: 3
[35228.288947] stap_2133a5650485081200141046ce814d5a_1263: systemtap: 2.2.1/0.153, base: ffffffffa041c000, memory: 2047data/36text/12ctx/2058net/33alloc kb, probes: 3
[35233.979282] stap_a528b8b93ae13378535e6dcac6690d70_1448: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 2031data/20text/10ctx/2058net/33alloc kb, probes: 3
[35238.127653] PROCFS: systemtap: 2.2.1/0.153, base: ffffffffa0409000, memory: 23data/20text/12ctx/2058net/33alloc kb, probes: 4
[35240.868691] PROCFS_BUFFER1: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 20data/24text/1ctx/2058net/66alloc kb, probes: 4
[35243.622806] PROCFS_BUFFER2: systemtap: 2.2.1/0.153, base: ffffffffa0214000, memory: 20data/24text/1ctx/2058net/66alloc kb, probes: 4
[35246.347794] PROCFS_BUFFER3: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 20data/24text/1ctx/2058net/66alloc kb, probes: 4
[35249.057610] PROCFS_BUFFER4: systemtap: 2.2.1/0.153, base: ffffffffa0214000, memory: 20data/24text/1ctx/2058net/66alloc kb, probes: 4
[35251.755368] PROCFS_BUFFER5: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 20data/24text/1ctx/2058net/66alloc kb, probes: 4
[35254.656068] PROCFS_BUFFER6: systemtap: 2.2.1/0.153, base: ffffffffa0214000, memory: 20data/24text/1ctx/2058net/66alloc kb, probes: 5
[35257.323539] PROCFS_BUFFER7: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/22ctx/2058net/71alloc kb, probes: 3
[35260.013732] PROCFS_UMASK: systemtap: 2.2.1/0.153, base: ffffffffa0213000, memory: 28data/20text/12ctx/2058net/33alloc kb, probes: 15
[35262.734077] PROCFS_WRITE: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/10ctx/2058net/76alloc kb, probes: 3
[35266.065224] stap_d85a2953fa0576be902364a5a0dcb0ae_3525: systemtap: 2.2.1/0.153, base: ffffffffa0213000, memory: 23data/24text/26ctx/2058net/33alloc kb, probes: 3
[35269.353537] stap_b488968842132b6f9da440e61364b3fa_3704: systemtap: 2.2.1/0.153, base: ffffffffa0220000, memory: 23data/24text/26ctx/2058net/33alloc kb, probes: 3
[35272.195326] stap_83b5aef2f9b2c78b6a713e6788f7e41e_3888: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/20text/2ctx/2058net/34alloc kb, probes: 3
[35278.621976] stap_bcc0638638ba71836c202ced66981444_4162: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35280.946487] stap_6f013e541142ce3b5df62fdb2a81b7c_4341: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35283.238530] stap_a4714d1ca32f71954a57a33d920bf93_4520: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35285.507704] stap_71c6262ddbfe480eee08d2d061763fe_4699: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35287.898262] stap_7122dafaf07d24aff4ec6afbfaf0f74_4878: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35290.163404] stap_53808e68fa55a9761040b8fdf18d185_5057: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35292.513613] stap_8e327a5a55f613e3c582720bd2f0375_5240: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35294.876689] stap_6405b8c197e3ee6a4dd2b90a5a93981_5419: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35297.416562] stap_c05915a29de1c8cb158b0975da3ebce_5598: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35299.768596] stap_cbeec011ca3a069dce16df96be00a1b_5777: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35302.131702] stap_6e0fd66f51ed8ef5416d458aac49f5b_5964: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35304.483101] stap_c6f6461271120ce99e985c390cfc3e5_6143: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35306.805936] stap_eed7b915687fbfe78185c1ac0d9c28e1_6322: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35309.181913] stap_81f042e53087e297b455aa5ed74259b7_6501: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35311.489675] stap_e2ddbc38406987536701947c0ee0ea5d_6680: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35313.740344] stap_66f2a1d8446b8a32dc4882a5291ef6f6_6859: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35316.071141] stap_7ecfd33cb0f80cf7838c461a75dc331d_7038: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35318.469052] stap_2bb5b9c405bdd1fb6117117ee3e85b8a_7217: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35320.848684] stap_f707f447dc45830765a9b14c6bb1e8e9_7396: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35323.222659] stap_993e81e31692814ffc7a2394d8c939f_7576: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35325.567454] stap_7576453a0e481b69c747f1fd8f30dbe_7755: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35327.851241] stap_0bac4bf60f9a3fc429bacb92700c5da_7934: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35330.667446] stap_d8c8c2310b12d1dc58ddbba98cd265cf_8122: systemtap: 2.2.1/0.153, base: ffffffffa0210000, memory: 15data/20text/10ctx/2058net/33alloc kb, probes: 3
[35331.464256] stap_aa91f8a443f26d340ff3e7083054388_8141: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35331.518783] stap_aa91f8a443f26d340ff3e7083054388_8142: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35342.608497] stap_aa91f8a443f26d340ff3e70830543887_742: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35353.165118] stap_aa91f8a443f26d340ff3e7083054388_8172: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35353.179423] stap_aa91f8a443f26d340ff3e7083054388_8175: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35419.884589] stap[8431]: segfault at 7fffb5441ff8 ip 00002aaaaacdc7f9 sp 00007fffb5441ff0 error 6 in libdw-0.153.so[2aaaaaccf000+34000]
[35423.597222] stap[8437]: segfault at 7fff18800fea ip 00002aaaaacdc7f5 sp 00007fff18800fd0 error 4 in libdw-0.153.so[2aaaaaccf000+34000]
[35434.100132] stap_d7c5ea89bbeab842e164904400c0dcc6_8979: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/28text/40ctx/2058net/33alloc kb, probes: 3
[35438.360791] stap_234d26d7ba3dc9a8d0b6bf7eeb05d9d_9215: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35454.054468] stap_55f668b466eac978a02f44913d2f253_9510: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35454.636305] stap_55f668b466eac978a02f44913d2f253_9519: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 1
[35456.938114] stap_e9822e739976d6638eec0492d477dc85_9700: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 4
[35459.304056] stap_ccabed5f9fad6c687d7ca56846d2c913_9879: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 4
[35462.404960] stap_23eb524146d77c4dfbda6b0479c6e6a_10061: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/20text/10ctx/2058net/33alloc kb, probes: 5
[35464.862906] stap_f56d0b0e1c7271dd8e2650150a429a_10248: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35467.174114] stap_10b801c8cb1fbe410c2378b5ca11a7_10426: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35469.524301] stap_151a638fd070f12883580732bdc12b_10604: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35473.773085] stap_9d2a40611825f4541eff5eda373f34a7_10784: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 24data/28text/3ctx/2058net/162alloc kb, probes: 7
[35487.419558] stap_e7889144b772bccf5179052a928a6b8_10976: systemtap: 2.2.1/0.153, base: ffffffffa0216000, memory: 23data/20text/8ctx/2058net/34alloc kb, probes: 3
[35491.224573] stap_aab76a37508801ea7dfbdece092ff5e_11161: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 23data/24text/2ctx/2058net/33alloc kb, probes: 9
[35504.444849] stap_334209c3beb46c551d00d879611ad0_11347: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35507.098870] stap_c408cf0c9ba7a536da428eb471191a5_11530: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 15data/16text/2ctx/2058net/33alloc kb, probes: 2
[35509.699621] stap_f65e1446861c877ae73f8d2194daefc_11711: systemtap: 2.2.1/0.153, base: ffffffffa0210000, memory: 19data/20text/20ctx/2058net/33alloc kb, probes: 1
[35512.022553] stap_180ee6d653d0c203cb00bda94fc08d3_11890: systemtap: 2.2.1/0.153, base: ffffffffa021b000, memory: 19data/20text/20ctx/2058net/33alloc kb, probes: 1
[35514.699683] stap_4d70fc457771edbe01b6ad74615aba4_12087: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 19data/20text/8ctx/2058net/33alloc kb, probes: 1
[35516.986151] stap_223f38b304ba3e3b0a3cc2d63f5d058_12264: systemtap: 2.2.1/0.153, base: ffffffffa0212000, memory: 19data/20text/0ctx/2058net/33alloc kb, probes: 1
[35525.380958] stap_4f1aa30c00fde96e4e9dab0607b1c91c_12560: systemtap: 2.2.1/0.153, base: ffffffffa021d000, memory: 31data/36text/16ctx/2058net/33alloc kb, probes: 3
[35526.685699] stap_4f1aa30c00fde96e4e9dab0607b1c91c_12576: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 31data/36text/16ctx/2058net/33alloc kb, probes: 3
[35530.277601] stap_f4f89679241db67ce6ee989bf280b88_12798: systemtap: 2.2.1/0.153, base: ffffffffa0219000, memory: 23data/20text/6ctx/2058net/33alloc kb, probes: 1
[35539.061629] stap_d9178f619f973728a003a4a978d1e51e__12989: systemtap: 2.2.1/0.153, base: ffffffffa0225000, memory: 194data/160text/79ctx/2058net/33alloc kb, probes: 394
[35552.666837] stap_fbc5c2f5d796af75f6c72be0ee0a7d_13373: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/6ctx/2058net/33alloc kb, probes: 1
[35553.308210] stap_fbc5c2f5d796af75f6c72be0ee0a7d_13383: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 14data/16text/6ctx/2058net/33alloc kb, probes: 1
[35553.871329] stap_fbc5c2f5d796af75f6c72be0ee0a7d_13393: systemtap: 2.2.1/0.153, base: ffffffffa01ed000, memory: 14data/16text/6ctx/2058net/33alloc kb, probes: 1
[35564.489578] stap_4ed9fdb2cd6708e2efa0c51aeb4b64f_13962: systemtap: 2.2.1/0.153, base: ffffffffa0207000, memory: 3720data/36text/67ctx/2058net/33alloc kb, probes: 6
[35567.567939] stap_ea692c11a3d17766a0d577ba42aeeaaa_14143: systemtap: 2.2.1/0.153, base: ffffffffa05b4000, memory: 24data/28text/12ctx/2058net/33alloc kb, probes: 13
[35567.567946] Warning: found a stray unused aggrprobe@ffffffffa01b6000
[35567.567963] ------------[ cut here ]------------
[35567.567967] kernel BUG at /build/buildd-linux_3.8.12-1-amd64-RaG_7r/linux-3.8.12/kernel/kprobes.c:707!
[35567.567972] invalid opcode: 0000 [#1] SMP
[35567.567976] Modules linked in: stap_ea692c11a3d17766a0d577ba42aeeaaa_14143 systemtap_test_module1(O) systemtap_test_module2(O) zlib_deflate mtd binfmt_misc fuse nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd dns_resolver fscache sunrpc loop evdev snd_pcm_oss snd_mixer_oss snd_pcm acpi_cpufreq snd_page_alloc snd_timer mperf snd processor soundcore thermal_sys pcspkr ext3 mbcache jbd virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront [last unloaded: stap_4ed9fdb2cd6708e2efa0c51aeb4b64f_13962]
[35567.568039] CPU 1
[35567.568043] Pid: 14143, comm: stapio Tainted: G O 3.8-1-amd64 #1 Debian 3.8.12-1
[35567.568049] RIP: e030:[<ffffffff81387fb6>] [<ffffffff81387fb6>] register_kprobe+0x1c8/0x418
[35567.568060] RSP: e02b:ffff88003d791e48 EFLAGS: 00010246
[35567.568063] RAX: 0000000000000038 RBX: ffffffffa05bee70 RCX: 0000000003a603a6
[35567.568068] RDX: 0000000000000cfb RSI: 0000000000000000 RDI: ffff88003d790208
[35567.568072] RBP: ffff88003c2f9cc0 R08: 0000000000000000 R09: 0720073007300730
[35567.568076] R10: 0720073007300730 R11: 0736076207310730 R12: 0000000000000000
[35567.568080] R13: ffffffffa01b8000 R14: ffffffffa05bee80 R15: ffffffffa05bee70
[35567.568088] FS: 00002aaaaaadc0c0(0000) GS:ffff88003f880000(0000) knlGS:0000000000000000
[35567.568093] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[35567.568097] CR2: 00002aaaab9d5ff8 CR3: 000000003b861000 CR4: 0000000000000660
[35567.568101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[35567.568106] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[35567.568110] Process stapio (pid: 14143, threadinfo ffff88003d790000, task ffff88003e2ad180)
[35567.568115] Stack:
[35567.568117] 000000000000000c 0000000000000000 0000000000000000 0000000000000000
[35567.568124] ffffffffa01b6000 000000000000000c ffffffffa05bee70 ffffffffa05b93a3
[35567.568130] ffffffff0000000c ffff88000000080a ffffffff00000021 ffffffff0000000d
[35567.568137] Call Trace:
[35567.568142] [<ffffffffa01b6000>] ? 0xffffffffa01b5fff
[35567.568148] [<ffffffffa05b93a3>] ? _stp_ctl_write_cmd+0x3d8/0x7f9 [stap_ea692c11a3d17766a0d577ba42aeeaaa_14143]
[35567.568157] [<ffffffff8106429e>] ? wake_up_new_task+0xaf/0xc8
[35567.568164] [<ffffffff8110b74a>] ? vfs_write+0x96/0xf8
[35567.568168] [<ffffffff8110b972>] ? sys_write+0x51/0x80
[35567.568173] [<ffffffff81388c89>] ? stub_clone+0x69/0x90
[35567.568181] [<ffffffff81388929>] ? system_call_fastpath+0x16/0x1b
[35567.568185] Code: 75 12 48 8b 75 28 48 c7 c7 76 17 4f 81 31 c0 e8 ab 5f ff ff 83 65 78 fd 48 81 7d 40 39 69 38 81 75 09 83 bd a0 00 00 00 00 75 02 <0f> 0b 48 89 ef e8 48 f6 ff ff f6 45 78 01 74 27 48 89 ef e8 e6
[35567.568233] RIP [<ffffffff81387fb6>] register_kprobe+0x1c8/0x418
[35567.568238] RSP <ffff88003d791e48>
[35567.568261] ---[ end trace 66e9937400424719 ]---
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S 00:46 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 00:46 0:00 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kworker/u:0]
root 7 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kworker/u:0H]
root 8 0.0 0.0 0 0 ? S 00:46 0:00 \_ [migration/0]
root 9 0.0 0.0 0 0 ? S 00:46 0:00 \_ [rcu_bh]
root 10 0.0 0.0 0 0 ? S 00:46 0:02 \_ [rcu_sched]
root 11 0.0 0.0 0 0 ? S 00:46 0:00 \_ [watchdog/0]
root 12 0.0 0.0 0 0 ? S 00:46 0:00 \_ [watchdog/1]
root 13 0.0 0.0 0 0 ? S 00:46 0:00 \_ [ksoftirqd/1]
root 14 0.0 0.0 0 0 ? S 00:46 0:00 \_ [migration/1]
root 16 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kworker/1:0H]
root 17 0.0 0.0 0 0 ? S 00:46 0:00 \_ [watchdog/2]
root 18 0.0 0.0 0 0 ? S 00:46 0:00 \_ [ksoftirqd/2]
root 19 0.0 0.0 0 0 ? S 00:46 0:00 \_ [migration/2]
root 21 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kworker/2:0H]
root 22 0.0 0.0 0 0 ? S 00:46 0:00 \_ [watchdog/3]
root 23 0.0 0.0 0 0 ? S 00:46 0:00 \_ [ksoftirqd/3]
root 24 0.0 0.0 0 0 ? S 00:46 0:00 \_ [migration/3]
root 26 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kworker/3:0H]
root 27 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [cpuset]
root 28 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [khelper]
root 29 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kdevtmpfs]
root 30 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [netns]
root 31 0.0 0.0 0 0 ? S 00:46 0:00 \_ [xenwatch]
root 32 0.0 0.0 0 0 ? S 00:46 0:00 \_ [xenbus]
root 33 0.0 0.0 0 0 ? S 00:46 0:00 \_ [bdi-default]
root 34 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kintegrityd]
root 35 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kblockd]
root 37 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kworker/1:1]
root 38 0.0 0.0 0 0 ? S 00:46 0:00 \_ [khungtaskd]
root 39 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kswapd0]
root 40 0.0 0.0 0 0 ? SN 00:46 0:00 \_ [ksmd]
root 41 0.0 0.0 0 0 ? S 00:46 0:00 \_ [fsnotify_mark]
root 42 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [crypto]
root 46 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kthrotld]
root 47 0.0 0.0 0 0 ? S 00:46 0:00 \_ [khvcd]
root 50 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kworker/3:1]
root 51 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [deferwq]
root 52 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kworker/u:1]
root 99 0.0 0.0 0 0 ? S< 00:46 0:00 \_ [kworker/0:1H]
root 124 0.0 0.0 0 0 ? S 00:46 0:00 \_ [kjournald]
root 1542 0.0 0.0 0 0 ? S< 00:47 0:00 \_ [rpciod]
root 1544 0.0 0.0 0 0 ? S< 00:47 0:00 \_ [nfsiod]
root 1556 0.0 0.0 0 0 ? S 00:47 0:00 \_ [nfsv4.0-svc]
root 2138 0.0 0.0 0 0 ? S< 00:47 0:00 \_ [kworker/1:1H]
root 3166 0.0 0.0 0 0 ? S 10:22 0:00 \_ [kworker/2:0]
root 16454 0.0 0.0 0 0 ? S 10:27 0:00 \_ [kworker/0:2]
root 30926 0.0 0.0 0 0 ? S 10:32 0:00 \_ [kworker/0:0]
root 8116 0.0 0.0 0 0 ? S 10:35 0:00 \_ [kworker/3:0]
root 8389 0.0 0.0 0 0 ? S 10:37 0:00 \_ [kworker/1:2]
root 12557 0.0 0.0 0 0 ? S 10:39 0:00 \_ [kworker/2:2]
root 14206 0.0 0.0 0 0 ? S 14:05 0:00 \_ [kworker/2:1]
root 14265 0.0 0.0 0 0 ? S 14:05 0:00 \_ [flush-202:0]
root 14267 0.0 0.0 0 0 ? S 14:07 0:00 \_ [kworker/0:1]
root 14272 0.0 0.0 0 0 ? S 14:07 0:00 \_ [kworker/3:2]
root 1 0.0 0.0 15272 892 ? Ss 00:46 0:01 init [2]
root 269 0.0 0.1 23696 1172 ? Ss 00:47 0:00 udevd --daemon
root 12038 0.0 0.0 23692 796 ? S 10:26 0:00 \_ udevd --daemon
root 12039 0.0 0.0 23692 744 ? S 10:26 0:00 \_ udevd --daemon
root 1500 0.0 0.2 10116 2848 ? Ss 00:47 0:00 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
root 1528 0.0 0.0 19120 892 ? Ss 00:47 0:00 /sbin/rpcbind -w
statd 1537 0.0 0.1 23492 1312 ? Ss 00:47 0:00 /sbin/rpc.statd
root 1550 0.0 0.0 29652 956 ? Ss 00:47 0:00 /usr/sbin/rpc.idmapd
root 1961 0.0 0.1 267160 1260 ? Sl 00:47 0:00 /usr/sbin/rsyslogd
root 2050 0.0 0.0 4260 640 ? Ss 00:47 0:00 /usr/sbin/acpid
root 2087 0.0 0.0 26812 948 ? Ss 00:47 0:00 /usr/sbin/cron
message+ 2216 0.0 0.0 32256 524 ? Ss 00:47 0:00 /usr/bin/dbus-daemon --system
root 2377 0.0 0.1 52348 1228 ? Ss 00:47 0:00 /usr/sbin/sshd
root 3085 0.0 0.3 88448 3640 ? Ss 10:22 0:00 \_ sshd: lindi [priv]
lindi 3092 0.0 0.1 88448 1588 ? S 10:22 0:00 | \_ sshd: lindi@pts/0
lindi 3093 0.0 0.3 23100 3896 pts/0 Ss 10:22 0:00 | \_ -bash
root 5802 0.0 0.1 58828 1908 pts/0 S+ 10:24 0:00 | \_ sudo make installcheck
root 5803 0.0 0.1 12352 1408 pts/0 S+ 10:24 0:00 | \_ make installcheck
root 5805 0.0 0.0 11928 968 pts/0 S+ 10:24 0:00 | \_ make -C testsuite installcheck RUNTESTFLAGS=
root 5808 0.0 0.0 11928 964 pts/0 S+ 10:24 0:00 | \_ make check-DEJAGNU RUNTESTFLAGS= --tool_opts \'install \'
root 5809 0.0 0.1 12728 1388 pts/0 S+ 10:24 0:00 | \_ /bin/bash -c srcdir='.'; export srcdir; \ EXPECT=expect; export EXPECT; \ runtest="env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib/systemtap CRASH_LIBDIR=/usr/lib/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest"; \ if /bin/bash -c "$runtest --version" > /dev/null 2>&1; then \ exit_status=0; l='systemtap'; for tool in $l; do \ if $runtest --tool $tool --tool_opts \'\' --srcdir $srcdir --tool_opts \'install \'; \ then :; else exit_status=1; fi; \ done; \ else echo "WARNING: could not find 'runtest'" 1>&2; :;\ fi; \ exit $exit_status
root 5841 0.0 0.0 4320 596 pts/0 S+ 10:24 0:00 | \_ /bin/sh ./execrc runtest --tool systemtap --tool_opts '' --srcdir . --tool_opts 'install '
root 5842 0.0 1.7 52716 17700 pts/0 Sl+ 10:24 0:05 | \_ expect -- /usr/share/dejagnu/runtest.exp --tool systemtap --tool_opts --srcdir . --tool_opts install
root 721 0.0 0.0 0 0 ? Zs 10:33 0:00 | \_ [stap] <defunct>
root 13417 0.0 0.0 0 0 ? Zs 10:39 0:00 | \_ [main_quiesce_ex] <defunct>
root 13971 0.0 2.7 201996 27596 ? Ss 10:39 0:00 | \_ stap ./systemtap.context/args.stp
root 14143 0.0 0.0 0 0 ? Zl 10:39 0:01 | \_ [stapio] <defunct>
root 14200 0.0 0.3 88448 3636 ? Ss 14:05 0:00 \_ sshd: lindi [priv]
lindi 14207 0.0 0.1 88448 1588 ? S 14:05 0:00 \_ sshd: lindi@pts/7
lindi 14208 0.0 0.3 23100 3868 pts/7 Ss 14:05 0:00 \_ -bash
lindi 14297 0.0 0.1 18948 1272 pts/7 R+ 14:08 0:00 \_ ps axuf
Debian-+ 2407 0.0 0.0 47468 1004 ? Ss 00:47 0:00 /usr/sbin/exim4 -bd -q30m
root 2477 0.0 0.0 14724 840 hvc0 Ss+ 00:47 0:00 /sbin/getty 38400 hvc0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 11:11 systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707 Timo Juhani Lindfors
@ 2013-05-17 12:21 ` Frank Ch. Eigler
2013-05-17 14:20 ` Timo Juhani Lindfors
2013-05-21 2:45 ` Masami Hiramatsu
0 siblings, 2 replies; 12+ messages in thread
From: Frank Ch. Eigler @ 2013-05-17 12:21 UTC (permalink / raw)
To: Timo Juhani Lindfors; +Cc: systemtap, srostedt, masami.hiramatsu.pt
timo.lindfors@iki.fi wrote:
> [35567.567939] stap_ea692c11a3d17766a0d577ba42aeeaaa_14143: systemtap: 2.2.1/0.153, base: ffffffffa05b4000, memory: 24data/28text/12ctx/2058net/33alloc kb, probes: 13
> [35567.567946] Warning: found a stray unused aggrprobe@ffffffffa01b6000
> [35567.567963] ------------[ cut here ]------------
> [35567.567967] kernel BUG at /build/buildd-linux_3.8.12-1-amd64-RaG_7r/linux-3.8.12/kernel/kprobes.c:707!
> [35567.567972] invalid opcode: 0000 [#1] SMP
> [35567.567976] Modules linked in: stap_ea692c11a3d17766a0d577ba42aeeaaa_14143 systemtap_test_module1(O) systemtap_test_module2(O) zlib_deflate mtd binfmt_misc fuse nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd dns_resolver fscache sunrpc loop evdev snd_pcm_oss snd_mixer_oss snd_pcm acpi_cpufreq snd_page_alloc snd_timer mperf snd processor soundcore thermal_sys pcspkr ext3 mbcache jbd virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront [last unloaded: stap_4ed9fdb2cd6708e2efa0c51aeb4b64f_13962]
> [35567.568137] Call Trace:
> [35567.568142] [<ffffffffa01b6000>] ? 0xffffffffa01b5fff
> [35567.568148] [<ffffffffa05b93a3>] ? _stp_ctl_write_cmd+0x3d8/0x7f9 [stap_ea692c11a3d17766a0d577ba42aeeaaa_14143]
> [...]
> [35567.568185] Code: 75 12 48 8b 75 28 48 c7 c7 76 17 4f 81 31 c0 e8 ab 5f ff ff 83 65 78 fd 48 81 7d 40 39 69 38 81 75 09 83 bd a0 00 00 00 00 75 02 <0f> 0b 48 89 ef e8 48 f6 ff ff f6 45 78 01 74 27 48 89 ef e8 e6
> [35567.568233] RIP [<ffffffff81387fb6>] register_kprobe+0x1c8/0x418
> [35567.568238] RSP <ffff88003d791e48>
> [35567.568261] ---[ end trace 66e9937400424719 ]---
This seems like a recurrence of an old kernel bug related to
optimized-kprobes, see <http://sourceware.org/bugzilla/show_bug.cgi?id=13193>.
It could also be something related to ftrace/kprobes perhaps.
- FChE
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 12:21 ` Frank Ch. Eigler
@ 2013-05-17 14:20 ` Timo Juhani Lindfors
2013-05-17 18:00 ` Timo Juhani Lindfors
2013-05-17 18:02 ` Timo Juhani Lindfors
2013-05-21 2:45 ` Masami Hiramatsu
1 sibling, 2 replies; 12+ messages in thread
From: Timo Juhani Lindfors @ 2013-05-17 14:20 UTC (permalink / raw)
To: systemtap
[-- Attachment #1: Type: text/plain, Size: 463 bytes --]
fche@redhat.com (Frank Ch. Eigler) writes:
> This seems like a recurrence of an old kernel bug related to
> optimized-kprobes, see <http://sourceware.org/bugzilla/show_bug.cgi?id=13193>.
> It could also be something related to ftrace/kprobes perhaps.
Thanks! After "echo 0 > /proc/sys/debug/kprobes-optimization" the kernel
does not crash anymore and the testsuite completes. I see however a few
stap segfaults and OOM killer hits.
See the attached log files.
[-- Attachment #2: systemtap.log.gz --]
[-- Type: application/octet-stream, Size: 435192 bytes --]
[-- Attachment #3: systemtap.sum.gz --]
[-- Type: application/octet-stream, Size: 14281 bytes --]
[-- Attachment #4: dmesg.txt.gz --]
[-- Type: application/octet-stream, Size: 24619 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 14:20 ` Timo Juhani Lindfors
@ 2013-05-17 18:00 ` Timo Juhani Lindfors
2013-05-17 18:18 ` Mark Wielaard
2013-05-17 18:02 ` Timo Juhani Lindfors
1 sibling, 1 reply; 12+ messages in thread
From: Timo Juhani Lindfors @ 2013-05-17 18:00 UTC (permalink / raw)
To: systemtap
Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:
> Thanks! After "echo 0 > /proc/sys/debug/kprobes-optimization" the kernel
> does not crash anymore and the testsuite completes. I see however a few
> stap segfaults and OOM killer hits.
First segfault:
lindi3:~/tmp/systemtap-2.2.1/testsuite$ gdb --args stap --rlimit-stack=1 --rlimit-stack=999999999999 -p4 ./systemtap.base/rlimit.stp
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/stap...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/bin/stap --rlimit-stack=1 --rlimit-stack=999999999999 -p4 ./systemtap.base/rlimit.stp
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Unable to set resource limits for rlimit_stack : Operation not permitted
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bb17f9 in dwarf_getsrclines () from /usr/lib/x86_64-linux-gnu/libdw.so.1
(gdb) bt
#0 0x00007ffff7bb17f9 in dwarf_getsrclines () from /usr/lib/x86_64-linux-gnu/libdw.so.1
#1 0x00007ffff7bb5ce1 in dwarf_decl_file () from /usr/lib/x86_64-linux-gnu/libdw.so.1
#2 0x0000000000541d1b in ?? ()
#3 0x00000000004b9782 in ?? ()
#4 0x0000000000545bac in ?? ()
#5 0x00000000004ccebf in ?? ()
#6 0x0000000000546a42 in ?? ()
#7 0x00000000004ce287 in ?? ()
#8 0x00000000004ce438 in ?? ()
#9 0x00000000004b246d in ?? ()
#10 0x00007ffff7bbf3d2 in dwfl_getmodules () from /usr/lib/x86_64-linux-gnu/libdw.so.1
#11 0x00000000004caa91 in ?? ()
#12 0x000000000045b27b in ?? ()
#13 0x000000000045b2fc in ?? ()
#14 0x000000000045b2fc in ?? ()
#15 0x000000000045b2fc in ?? ()
#16 0x000000000045c2bf in ?? ()
#17 0x000000000045e2f3 in ?? ()
#18 0x000000000046af1c in ?? ()
#19 0x0000000000414892 in ?? ()
#20 0x000000000040ff1a in ?? ()
#21 0x00007ffff54d5a55 in __libc_start_main (main=0x40f650, argc=5, ubp_av=0x7fffffffe978, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe968)
at libc-start.c:260
#22 0x0000000000412895 in ?? ()
(gdb) info register
rax 0x7ffffffdcff0 140737488211952
rbx 0xffffffff816bba78 -2123646344
rcx 0x0 0
rdx 0x0 0
rsi 0x7ffff2bad3bc 140737265718204
rdi 0x0 0
rbp 0x7fffffffb780 0x7fffffffb780
rsp 0x7ffffffdcff0 0x7ffffffdcff0
r8 0x7ffff2bad3bb 140737265718203
r9 0x0 0
r10 0x0 0
r11 0x7ffffffdd030 140737488212016
r12 0x7fffffffb7a0 140737488336800
r13 0x0 0
r14 0xa99 2713
r15 0x257f088 39317640
rip 0x7ffff7bb17f9 0x7ffff7bb17f9 <dwarf_getsrclines+1897>
eflags 0x10256 [ PF AF ZF IF RF ]
cs 0xe033 57395
ss 0xe02b 57387
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb) x/4i $rip
=> 0x7ffff7bb17f9 <dwarf_getsrclines+1897>: mov %rbx,0x8(%rax)
0x7ffff7bb17fd <dwarf_getsrclines+1901>: mov %r14d,0x14(%rax)
0x7ffff7bb1801 <dwarf_getsrclines+1905>: mov %r9w,0x18(%rax)
0x7ffff7bb1806 <dwarf_getsrclines+1910>: and $0xe01f,%dx
The version of libdw1 is 0.153-2. I rebuilt it with -O0 -g and now I see bit more:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7ba24ce in add_new_line (new_line=0x7ffffffdd020, end_sequence=false) at dwarf_getsrclines.c:361
361 {
(gdb) bt
#0 0x00007ffff7ba24ce in add_new_line (new_line=0x7ffffffdd020, end_sequence=false) at dwarf_getsrclines.c:361
#1 0x00007ffff7ba1144 in dwarf_getsrclines (cudie=0x7fffffffb7b0, lines=0x7fffffffb778, nlines=0x7fffffffb780) at dwarf_getsrclines.c:421
#2 0x00007ffff7ba7b14 in dwarf_decl_file (die=0x2e5cd78) at dwarf_decl_file.c:87
#3 0x0000000000541d1b in ?? ()
#4 0x00000000004b9782 in ?? ()
#5 0x0000000000545bac in ?? ()
#6 0x00000000004ccebf in ?? ()
#7 0x0000000000546a42 in ?? ()
#8 0x00000000004ce287 in ?? ()
#9 0x00000000004ce438 in ?? ()
#10 0x00000000004b246d in ?? ()
#11 0x00007ffff7bb6ab5 in dwfl_getmodules (dwfl=0x1e028b0, callback=0x4b2310, arg=0x7fffffffc3b0, offset=0) at dwfl_getmodules.c:103
#12 0x00000000004caa91 in ?? ()
#13 0x000000000045b27b in ?? ()
#14 0x000000000045b2fc in ?? ()
#15 0x000000000045b2fc in ?? ()
#16 0x000000000045b2fc in ?? ()
#17 0x000000000045c2bf in ?? ()
#18 0x000000000045e2f3 in ?? ()
#19 0x000000000046af1c in ?? ()
#20 0x0000000000414892 in ?? ()
#21 0x000000000040ff1a in ?? ()
#22 0x00007ffff54c2a55 in __libc_start_main (main=0x40f650, argc=5, ubp_av=0x7fffffffe978, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe968)
at libc-start.c:260
#23 0x0000000000412895 in ?? ()
(gdb) l
356 end_seq))) \
357 goto invalid_data; \
358 } while (0)
359
360 inline bool add_new_line (struct linelist *new_line, bool end_sequence)
361 {
362 /* Set the line information. For some fields we use bitfields,
363 so we would lose information if the encoded values are too large.
364 Check just for paranoia, and call the data "invalid" if it
365 violates our assumptions on reasonable limits for the values. */
(gdb) p *new_line
$2 = {line = {files = 0x0, addr = 0, file = 0, line = 0, column = 0, is_stmt = 0, basic_block = 0, end_sequence = 0, prologue_end = 0, epilogue_begin = 0, op_index = 0, isa = 0, discriminator = 0},
next = 0x7ffffffdd060}
(gdb) p *new_line->next
$3 = {line = {files = 0x0, addr = 18446744071585905164, file = 4, line = 2702, column = 0, is_stmt = 1, basic_block = 0, end_sequence = 0, prologue_end = 0, epilogue_begin = 0, op_index = 0, isa = 0,
discriminator = 0}, next = 0x7ffffffdd0a0}
(gdb) up
#1 0x00007ffff7ba1144 in dwarf_getsrclines (cudie=0x7fffffffb7b0, lines=0x7fffffffb778, nlines=0x7fffffffb780) at dwarf_getsrclines.c:421
421 NEW_LINE (0);
(gdb) l
416 /* Perform the increments. */
417 line += line_increment;
418 advance_pc ((opcode - opcode_base) / line_range);
419
420 /* Add a new line with the current state machine values. */
421 NEW_LINE (0);
422
423 /* Reset the flags. */
424 basic_block = false;
425 prologue_end = false;
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 14:20 ` Timo Juhani Lindfors
2013-05-17 18:00 ` Timo Juhani Lindfors
@ 2013-05-17 18:02 ` Timo Juhani Lindfors
2013-05-17 18:20 ` Mark Wielaard
1 sibling, 1 reply; 12+ messages in thread
From: Timo Juhani Lindfors @ 2013-05-17 18:02 UTC (permalink / raw)
To: systemtap
Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:
> Thanks! After "echo 0 > /proc/sys/debug/kprobes-optimization" the kernel
> does not crash anymore and the testsuite completes. I see however a few
> stap segfaults and OOM killer hits.
Second segfault:
lindi3:~/tmp/systemtap-2.2.1/testsuite$ gdb --args stap -p2 ./systemtap.pass1-4/debugtypes.stp
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/stap...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/bin/stap -p2 ./systemtap.pass1-4/debugtypes.stp
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b9b4dd in dwarf_tag (die=0x0) at dwarf_tag.c:107
107 if (die->abbrev == NULL)
(gdb) bt
#0 0x00007ffff7b9b4dd in dwarf_tag (die=0x0) at dwarf_tag.c:107
#1 0x0000000000546b2a in ?? ()
#2 0x00000000005491e6 in ?? ()
#3 0x00000000004d27eb in ?? ()
#4 0x00000000004b246d in ?? ()
#5 0x00007ffff7bb6ab5 in dwfl_getmodules (dwfl=0x1e03c90, callback=0x4b2310, arg=0x7fffffffcb90, offset=0) at dwfl_getmodules.c:103
#6 0x00000000004c5ffc in ?? ()
#7 0x000000000044c2ff in ?? ()
#8 0x0000000000452fa3 in ?? ()
#9 0x000000000044a8e7 in ?? ()
#10 0x000000000044bebf in ?? ()
#11 0x000000000045e3c3 in ?? ()
#12 0x000000000046af1c in ?? ()
#13 0x0000000000414892 in ?? ()
#14 0x000000000040ff1a in ?? ()
#15 0x00007ffff54c2a55 in __libc_start_main (main=0x40f650, argc=3, ubp_av=0x7fffffffe9a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe998)
at libc-start.c:260
#16 0x0000000000412895 in ?? ()
(gdb) x/4i $rip
=> 0x7ffff7b9b4dd <dwarf_tag+16>: mov 0x10(%rax),%rax
0x7ffff7b9b4e1 <dwarf_tag+20>: test %rax,%rax
0x7ffff7b9b4e4 <dwarf_tag+23>: jne 0x7ffff7b9b559 <dwarf_tag+140>
0x7ffff7b9b4e6 <dwarf_tag+25>: mov -0x18(%rbp),%rax
(gdb) info register
rax 0x0 0
rbx 0x1e07dc0 31489472
rcx 0x1e05d88 31481224
rdx 0x7fffffffc700 140737488340736
rsi 0x7ffff7fdf050 140737354002512
rdi 0x0 0
rbp 0x7fffffffc680 0x7fffffffc680
rsp 0x7fffffffc660 0x7fffffffc660
r8 0x20 32
r9 0x101010101010101 72340172838076673
r10 0x7fffffffc430 140737488340016
r11 0x7ffff553267e 140737309255294
r12 0x1e03630 31471152
r13 0x7fffffffc6e8 140737488340712
r14 0x7fffffffc6e0 140737488340704
r15 0x0 0
rip 0x7ffff7b9b4dd 0x7ffff7b9b4dd <dwarf_tag+16>
eflags 0x10206 [ PF IF RF ]
cs 0xe033 57395
ss 0xe02b 57387
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 18:00 ` Timo Juhani Lindfors
@ 2013-05-17 18:18 ` Mark Wielaard
2013-05-17 18:24 ` Timo Juhani Lindfors
0 siblings, 1 reply; 12+ messages in thread
From: Mark Wielaard @ 2013-05-17 18:18 UTC (permalink / raw)
To: Timo Juhani Lindfors; +Cc: systemtap
On Fri, 2013-05-17 at 21:00 +0300, Timo Juhani Lindfors wrote:
> Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:
> > Thanks! After "echo 0 > /proc/sys/debug/kprobes-optimization" the kernel
> > does not crash anymore and the testsuite completes. I see however a few
> > stap segfaults and OOM killer hits.
>
> First segfault:
>
> lindi3:~/tmp/systemtap-2.2.1/testsuite$ gdb --args stap --rlimit-stack=1 --rlimit-stack=999999999999 -p4 ./systemtap.base/rlimit.stp
> GNU gdb (GDB) 7.4.1-debian
> Copyright (C) 2012 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/bin/stap...(no debugging symbols found)...done.
> (gdb) r
> Starting program: /usr/bin/stap --rlimit-stack=1 --rlimit-stack=999999999999 -p4 ./systemtap.base/rlimit.stp
> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
> warning: Could not load shared library symbols for linux-vdso.so.1.
> Do you need "set solib-search-path" or "set sysroot"?
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Unable to set resource limits for rlimit_stack : Operation not permitted
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff7bb17f9 in dwarf_getsrclines () from /usr/lib/x86_64-linux-gnu/libdw.so.1
> (gdb) bt
> #0 0x00007ffff7bb17f9 in dwarf_getsrclines () from /usr/lib/x86_64-linux-gnu/libdw.so.1
> #1 0x00007ffff7bb5ce1 in dwarf_decl_file () from /usr/lib/x86_64-linux-gnu/libdw.so.1
I am not sure what the rlimit references do here. But if it is limiting
the stack a lot, then that might be it. dwarf_getsrclines () uses some
alloca () calls for temporary memory.
> The version of libdw1 is 0.153-2. I rebuilt it with -O0 -g and now I see bit more:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff7ba24ce in add_new_line (new_line=0x7ffffffdd020, end_sequence=false) at dwarf_getsrclines.c:361
> 361 {
> (gdb) bt
> #0 0x00007ffff7ba24ce in add_new_line (new_line=0x7ffffffdd020, end_sequence=false) at dwarf_getsrclines.c:361
> #1 0x00007ffff7ba1144 in dwarf_getsrclines (cudie=0x7fffffffb7b0, lines=0x7fffffffb778, nlines=0x7fffffffb780) at dwarf_getsrclines.c:421
> #2 0x00007ffff7ba7b14 in dwarf_decl_file (die=0x2e5cd78) at dwarf_decl_file.c:87
[...]
> (gdb) l
> 356 end_seq))) \
> 357 goto invalid_data; \
> 358 } while (0)
> 359
> 360 inline bool add_new_line (struct linelist *new_line, bool end_sequence)
> 361 {
> 362 /* Set the line information. For some fields we use bitfields,
> 363 so we would lose information if the encoded values are too large.
> 364 Check just for paranoia, and call the data "invalid" if it
> 365 violates our assumptions on reasonable limits for the values. */
> (gdb) p *new_line
> $2 = {line = {files = 0x0, addr = 0, file = 0, line = 0, column = 0, is_stmt = 0, basic_block = 0, end_sequence = 0, prologue_end = 0, epilogue_begin = 0, op_index = 0, isa = 0, discriminator = 0},
> next = 0x7ffffffdd060}
> (gdb) p *new_line->next
> $3 = {line = {files = 0x0, addr = 18446744071585905164, file = 4, line = 2702, column = 0, is_stmt = 1, basic_block = 0, end_sequence = 0, prologue_end = 0, epilogue_begin = 0, op_index = 0, isa = 0,
> discriminator = 0}, next = 0x7ffffffdd0a0}
> (gdb) up
> #1 0x00007ffff7ba1144 in dwarf_getsrclines (cudie=0x7fffffffb7b0, lines=0x7fffffffb778, nlines=0x7fffffffb780) at dwarf_getsrclines.c:421
> 421 NEW_LINE (0);
> (gdb) l
> 416 /* Perform the increments. */
> 417 line += line_increment;
> 418 advance_pc ((opcode - opcode_base) / line_range);
> 419
> 420 /* Add a new line with the current state machine values. */
> 421 NEW_LINE (0);
> 422
> 423 /* Reset the flags. */
> 424 basic_block = false;
> 425 prologue_end = false;
The NEW_LINE define uses alloca.
Try the same stap command without the -rlimit-stack=... arguments to see
if that is it.
Cheers,
Mark
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 18:02 ` Timo Juhani Lindfors
@ 2013-05-17 18:20 ` Mark Wielaard
2013-05-17 18:25 ` Timo Juhani Lindfors
0 siblings, 1 reply; 12+ messages in thread
From: Mark Wielaard @ 2013-05-17 18:20 UTC (permalink / raw)
To: Timo Juhani Lindfors; +Cc: systemtap
On Fri, 2013-05-17 at 21:02 +0300, Timo Juhani Lindfors wrote:
> Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:
> > Thanks! After "echo 0 > /proc/sys/debug/kprobes-optimization" the kernel
> > does not crash anymore and the testsuite completes. I see however a few
> > stap segfaults and OOM killer hits.
>
> Second segfault:
>
> lindi3:~/tmp/systemtap-2.2.1/testsuite$ gdb --args stap -p2 ./systemtap.pass1-4/debugtypes.stp
> GNU gdb (GDB) 7.4.1-debian
> Copyright (C) 2012 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/bin/stap...(no debugging symbols found)...done.
> (gdb) r
> Starting program: /usr/bin/stap -p2 ./systemtap.pass1-4/debugtypes.stp
> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
> warning: Could not load shared library symbols for linux-vdso.so.1.
> Do you need "set solib-search-path" or "set sysroot"?
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff7b9b4dd in dwarf_tag (die=0x0) at dwarf_tag.c:107
> 107 if (die->abbrev == NULL)
Since this is elfutils < 0.154 then you might be hitting the bug that
this test was (partially) written for:
commit 581c3f60e2b1fc7ddaf4260bb5a9cb59f8e3f0d0
Author: Tom Tromey <tromey@redhat.com>
Date: Wed Mar 21 08:54:32 2012 -0600
Fix bug using dwarf_next_unit to iterate over .debug_types
Could you upgrade to 0.154 (or the latest 0.155) version of elfutils and
retest?
Thanks,
Mark
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 18:18 ` Mark Wielaard
@ 2013-05-17 18:24 ` Timo Juhani Lindfors
0 siblings, 0 replies; 12+ messages in thread
From: Timo Juhani Lindfors @ 2013-05-17 18:24 UTC (permalink / raw)
To: systemtap
Mark Wielaard <mjw@redhat.com> writes:
> Try the same stap command without the -rlimit-stack=... arguments to see
> if that is it.
If I don't pass --rlimit-stack arguments it does not crash.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 18:20 ` Mark Wielaard
@ 2013-05-17 18:25 ` Timo Juhani Lindfors
0 siblings, 0 replies; 12+ messages in thread
From: Timo Juhani Lindfors @ 2013-05-17 18:25 UTC (permalink / raw)
To: systemtap
Mark Wielaard <mjw@redhat.com> writes:
> Could you upgrade to 0.154 (or the latest 0.155) version of elfutils and
> retest?
No time right now but I pinged the Debian elfutils maintainer to upload
a newer version.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-17 12:21 ` Frank Ch. Eigler
2013-05-17 14:20 ` Timo Juhani Lindfors
@ 2013-05-21 2:45 ` Masami Hiramatsu
2013-05-22 14:22 ` Frank Ch. Eigler
1 sibling, 1 reply; 12+ messages in thread
From: Masami Hiramatsu @ 2013-05-21 2:45 UTC (permalink / raw)
To: Frank Ch. Eigler
Cc: Timo Juhani Lindfors, systemtap, srostedt, yrl.pp-manager.tt
Hi,
I've found the root cause of this bug at last.
In the normal path, an optprobe on an init function is
unregistered when a module goes live.
unregister_kprobe(kp)
-> __unregister_kprobe_top
->__disable_kprobe
->disarm_kprobe(ap == op)
->__disarm_kprobe
->unoptimize_kprobe : the op is queued
on unoptimizing_list
and do nothing in __unregister_kprobe_bottom
After a while (usually wait 5 jiffies), kprobe_optimizer runs
to unoptimize and free optprobe.
kprobe_optimizer
->do_unoptimize_kprobes
->arch_unoptimize_kprobes : moved to free_list
->do_free_cleaned_kprobes
->hlist_del: the op is removed
->free_aggr_kprobe
->arch_remove_optimized_kprobe
->arch_remove_kprobe
->kfree: the op is freed
Here, if kprobes_module_callback is called and the delayed
unoptimizing probe is picked BEFORE kprobe_optimizer runs,
kprobes_module_callback
->kill_kprobe
->kill_optimized_kprobe : dequeued from unoptimizing_list <=!!!
->arch_remove_optimized_kprobe
->arch_remove_kprobe
(but op is not freed, and on the kprobe hash table)
This shouldn't happen if the probe unregistration is done AFTER
kprobes_module_callback is called (because at that time the op
is gone), and kprobe-tracer does it.
So, ftrace never gets this bug, but systemtap does.
Of course, this is an actual bug, I'll fix it. And also, the stap
side can avoid it by unregistering probes after kprobes_module_callback
is called.
Thank you,
(2013/05/17 21:21), Frank Ch. Eigler wrote:
>
> timo.lindfors@iki.fi wrote:
>
>> [35567.567939] stap_ea692c11a3d17766a0d577ba42aeeaaa_14143: systemtap: 2.2.1/0.153, base: ffffffffa05b4000, memory: 24data/28text/12ctx/2058net/33alloc kb, probes: 13
>> [35567.567946] Warning: found a stray unused aggrprobe@ffffffffa01b6000
>> [35567.567963] ------------[ cut here ]------------
>> [35567.567967] kernel BUG at /build/buildd-linux_3.8.12-1-amd64-RaG_7r/linux-3.8.12/kernel/kprobes.c:707!
>> [35567.567972] invalid opcode: 0000 [#1] SMP
>> [35567.567976] Modules linked in: stap_ea692c11a3d17766a0d577ba42aeeaaa_14143 systemtap_test_module1(O) systemtap_test_module2(O) zlib_deflate mtd binfmt_misc fuse nfsv4 nfsd auth_rpcgss nfs_acl nfs lockd dns_resolver fscache sunrpc loop evdev snd_pcm_oss snd_mixer_oss snd_pcm acpi_cpufreq snd_page_alloc snd_timer mperf snd processor soundcore thermal_sys pcspkr ext3 mbcache jbd virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront [last unloaded: stap_4ed9fdb2cd6708e2efa0c51aeb4b64f_13962]
>> [35567.568137] Call Trace:
>> [35567.568142] [<ffffffffa01b6000>] ? 0xffffffffa01b5fff
>> [35567.568148] [<ffffffffa05b93a3>] ? _stp_ctl_write_cmd+0x3d8/0x7f9 [stap_ea692c11a3d17766a0d577ba42aeeaaa_14143]
>> [...]
>> [35567.568185] Code: 75 12 48 8b 75 28 48 c7 c7 76 17 4f 81 31 c0 e8 ab 5f ff ff 83 65 78 fd 48 81 7d 40 39 69 38 81 75 09 83 bd a0 00 00 00 00 75 02 <0f> 0b 48 89 ef e8 48 f6 ff ff f6 45 78 01 74 27 48 89 ef e8 e6
>> [35567.568233] RIP [<ffffffff81387fb6>] register_kprobe+0x1c8/0x418
>> [35567.568238] RSP <ffff88003d791e48>
>> [35567.568261] ---[ end trace 66e9937400424719 ]---
>
>
> This seems like a recurrence of an old kernel bug related to
> optimized-kprobes, see <http://sourceware.org/bugzilla/show_bug.cgi?id=13193>.
> It could also be something related to ftrace/kprobes perhaps.
>
>
> - FChE
>
--
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-21 2:45 ` Masami Hiramatsu
@ 2013-05-22 14:22 ` Frank Ch. Eigler
2013-05-23 2:01 ` Masami Hiramatsu
0 siblings, 1 reply; 12+ messages in thread
From: Frank Ch. Eigler @ 2013-05-22 14:22 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Timo Juhani Lindfors, systemtap, srostedt, yrl.pp-manager.tt
Hi, Masami -
> I've found the root cause of this bug at last.
Thanks for looking into it!
> [...]
> This shouldn't happen if the probe unregistration is done AFTER
> kprobes_module_callback is called (because at that time the op
> is gone), and kprobe-tracer does it.
> So, ftrace never gets this bug, but systemtap does.
I don't quite understand. In systemtap, as per runtime/transport/transport.c,
we use a notifier priority of 1 for the _stp_module_notifier, so that we get
notified after the kprobes_module_callback. I believe this behavior matches
ftrace.
- FChE
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707
2013-05-22 14:22 ` Frank Ch. Eigler
@ 2013-05-23 2:01 ` Masami Hiramatsu
0 siblings, 0 replies; 12+ messages in thread
From: Masami Hiramatsu @ 2013-05-23 2:01 UTC (permalink / raw)
To: Frank Ch. Eigler
Cc: Timo Juhani Lindfors, systemtap, srostedt, yrl.pp-manager.tt
(2013/05/22 23:21), Frank Ch. Eigler wrote:
> Hi, Masami -
>
>> I've found the root cause of this bug at last.
>
> Thanks for looking into it!
>
>> [...]
>> This shouldn't happen if the probe unregistration is done AFTER
>> kprobes_module_callback is called (because at that time the op
>> is gone), and kprobe-tracer does it.
>> So, ftrace never gets this bug, but systemtap does.
>
> I don't quite understand. In systemtap, as per runtime/transport/transport.c,
> we use a notifier priority of 1 for the _stp_module_notifier, so that we get
> notified after the kprobes_module_callback. I believe this behavior matches
> ftrace.
OK, even though, a probe have been removed right before the target module
is unloaded, it also hits the same problem. So, could you try to apply
the patch I posted on LKML and this ML yesterday?
https://lkml.org/lkml/2013/5/22/172
I hope it fixed the bug ...
Thank you,
--
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-05-23 2:01 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-05-17 11:11 systemtap 2.2.1 installcheck => kernel BUG at .. kprobes.c:707 Timo Juhani Lindfors
2013-05-17 12:21 ` Frank Ch. Eigler
2013-05-17 14:20 ` Timo Juhani Lindfors
2013-05-17 18:00 ` Timo Juhani Lindfors
2013-05-17 18:18 ` Mark Wielaard
2013-05-17 18:24 ` Timo Juhani Lindfors
2013-05-17 18:02 ` Timo Juhani Lindfors
2013-05-17 18:20 ` Mark Wielaard
2013-05-17 18:25 ` Timo Juhani Lindfors
2013-05-21 2:45 ` Masami Hiramatsu
2013-05-22 14:22 ` Frank Ch. Eigler
2013-05-23 2:01 ` Masami Hiramatsu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).