[Bug runtime/14804] New: Kernel panic when running hello world stap script

[Bug runtime/14804] New: Kernel panic when running hello world stap script

[jiri.horky at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

             Bug #: 14804
           Summary: Kernel panic when running hello world stap script
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: runtime
        AssignedTo: systemtap@sourceware.org
        ReportedBy: jiri.horky@gmail.com
    Classification: Unclassified


Created attachment 6715
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6715
Various information including backtrace,log,proc-modules etc.

Hi, I get kernel panic - "unable to handle kernel NULL pointer dereference at
0000000000000018" when running simple systemtap script. More information
follows.

System: SLES 11 SP1, kernel 2.6.32.59-0.7-default x86_64, 2x Xeon X5650, 48GB
RAM.
Systemtap: current git clone from 04.11.2012  (2.1/0.148, commit
release-2.0-75-g02bff02 + changes)
Directory where systemtap is installed: /usr/local/ (/usr/local/bin/stap)
The values for all environment variables: included as attachment.
The version of all related packages installed:

elfutils-0.137-8.22.58.x86_64
gcc-4.3-62.198.x86_64
gcc-c++-4.3-62.198.x86_64
gcc43-32bit-4.3.4_20091019-0.7.35.x86_64
gcc43-4.3.4_20091019-0.7.35.x86_64
gcc43-c++-4.3.4_20091019-0.7.35.x86_64
kernel-default-2.6.32.59-0.7.1.x86_64
kernel-default-base-2.6.32.59-0.7.1.x86_64
kernel-default-debuginfo-2.6.32.59-0.7.1.x86_64
kernel-default-devel-2.6.32.59-0.7.1.x86_64
kernel-default-devel-2.6.32.59-0.7.1.x86_64
kernel-source-2.6.32.59-0.7.1.x86_64
libgcc43-32bit-4.3.4_20091019-0.7.35.x86_64
libgcc43-4.3.4_20091019-0.7.35.x86_64
linux-kernel-headers-2.6.32-1.4.13.noarch
sgi-nfs-kernel-server-1.2.1-2.18.1sgi250r1.sles11.x86_64

PLEASE NOTE: systamtap was built ./configure
--with-elfutils=/root/elfutils-0.148 

Stap invocation: stap -k -vvv -e 'probe begin { log ("hello world") }'
Bug: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
Reproducible: Always
GCC: gcc (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]

modinfo /tmp/stapcmHxTQ/stap_5550.ko 
filename:       /tmp/stapcmHxTQ/stap_5550.ko
license:        GPL
description:    systemtap-generated probe
license:        GPL
srcversion:     43DA452BDA49BFCA2E81AAA
depends:        
vermagic:       2.6.32.59-0.7-default SMP mod_unload modversions 
parm:           _stp_bufsize:buffer size (int)


I include more information from kernel crashdump as attachment.

Please let me know if you need more information (I can upload crash dump image
if needed).

Jiri Horky

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[jiri.horky at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

Jiri Horky <jiri.horky at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jiri.horky at gmail dot com

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> 2012-11-23 15:16:15 UTC ---
Could you rebuild your stap module with -B CONFIG_DEBUG_INFO=y, and disassemble
it at the point of the crash?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[jiri.horky at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

--- Comment #2 from Jiri Horky <jiri.horky at gmail dot com> 2012-11-27 13:47:08 UTC ---
Created attachment 6757
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6757
objdump --disassemble of the failing module

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[jiri.horky at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

--- Comment #3 from Jiri Horky <jiri.horky at gmail dot com> 2012-11-27 13:48:36 UTC ---
Hi, 

here are the requested information. I also include the full output of "objdump
-d -S --disassemble stap_5816.ko".

OOPS:
[1816621.562379] Oops: 0002 [#1] SMP 
[1816621.630090] last sysfs file:
/sys/devices/system/cpu/cpu23/cache/index2/shared_cpu_map
[1816621.703173] CPU 11 
[1816621.769412] Modules linked in: stap_4476(N+) nls_utf8 nfs(X) fscache
sgi_cxfs(PX) sgi_xvm_cell(PX) sgi_cell(PX) rpcsec_gss_krb5(X) nfsd(X) lockd(X)
nfs_acl(X) auth_rpcgss(X) sunrpc(X) exportfs binfm
t_misc ipmi_devintf ipmi_si ipmi_msghandler ipv6 8021q garp stp llc bonding
cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq microcode
fuse xfs_quota(X) xfs_dmapi(X) dmapi(X) loop sg
i_xvm(PX) sgi_pm(PX) sgi_os_lib(PX) cme_migrate(X) hwperf(X) jmicron qla2xxx
ixgbe(X) tpm_tis sg scsi_transport_fc ohci1394 pata_jmicron i2c_i801 iTCO_wdt
tpm ioatdma iTCO_vendor_support tpm_bios igb mpt
ctl i2c_core scsi_tgt ieee1394 rtc_cmos pcspkr rtc_core dca joydev rtc_lib
button usbhid hid uhci_hcd ehci_hcd sd_mod crc_t10dif usbcore edd ext3 jbd
mbcache xfs(X) fan thermal processor thermal_sys hwmo
n mptsas mptscsih mptbase scsi_transport_sas ata_piix ahci piix ide_pci_generic
ide_core ata_generic libata megaraid_sas scsi_mod
[1816622.360710] Supported: No, Unsupported modules are loaded
[1816622.441237] Pid: 4846, comm: staprun Tainted: P   M      NX
2.6.32.59-0.7-default #1 C3108-TY11
[1816622.525488] RIP: 0010:[<ffffffff8139e6f8>]  [<ffffffff8139e6f8>]
_spin_lock_irqsave+0x8/0x20
[1816622.609573] RSP: 0018:ffff88059102de00  EFLAGS: 00010092
[1816622.690195] RAX: 0000000000000292 RBX: 0000000000000000 RCX:
00000000000000f7
[1816622.772499] RDX: 0000000000010000 RSI: ffff8806554a0000 RDI:
0000000000000018
[1816622.853869] RBP: 0000000000000002 R08: ffffffffa0774b3f R09:
0000000000000015
[1816622.934430] R10: 00000000a11a5ce9 R11: 00000000a238a149 R12:
0000000000000018
[1816623.013907] R13: ffff8806554a0000 R14: 0000000000000003 R15:
000000000061a010
[1816623.092353] FS:  00007ffa90290700(0000) GS:ffff8806554a0000(0000)
knlGS:0000000000000000
[1816623.171703] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1816623.248684] CR2: 0000000000000018 CR3: 0000000c1999a000 CR4:
00000000000006e0
[1816623.327338] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1816623.405077] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[1816623.481814] Process staprun (pid: 4846, threadinfo ffff88059102c000, task
ffff880623ea4300)
[1816623.560895] Stack:
[1816623.632984]  ffffffffa076fb11 ffffffffa0775b40 0000000000000002
0000000000000043
[1816623.641858] <0> ffffffffa0770f64 ffffffffa0775b40 ffffffffa0775b40
ffffffffa0772f90
[1816623.721214] <0> 0000000000000000 000000000061a060 ffffffffa0771edc
ffffc90000000010
[1816623.870613] Call Trace:
[1816623.943660]  [<ffffffffa076fb11>] _stp_mempool_alloc+0x21/0x80 [stap_4476]
[1816624.022599]  [<ffffffffa0770f64>] _stp_ctl_send+0x44/0x290 [stap_4476]
[1816624.101038]  [<ffffffffa0771edc>] _stp_error+0x4c/0x60 [stap_4476]
[1816624.178816]  [<ffffffffa0772fe5>] init_module+0x55/0xb30 [stap_4476]
[1816624.256063]  [<ffffffff810001e5>] do_one_initcall+0x35/0x190
[1816624.331760]  [<ffffffff8107edf4>] sys_init_module+0xe4/0x270
[1816624.405930]  [<ffffffff81002f7b>] system_call_fastpath+0x16/0x1b
[1816624.478754]  [<00007ffa8f49567a>] 0x7ffa8f49567a
[1816624.549723] Code: 01 00 00 00 75 09 f0 81 07 00 00 00 01 31 d2 89 d0 c3 90
f0 83 2f 01 79 05 e8 75 c4 e4 ff c3 0f 1f 40 00 9c 58 fa ba 00 00 01 00 <f0> 0f
c1 17 0f b7 ca c1 ea 10 39 d1 74 07 f3 90 0
f b7 0f eb f5 
[1816624.705374] RIP  [<ffffffff8139e6f8>] _spin_lock_irqsave+0x8/0x20
[1816624.778499]  RSP <ffff88059102de00>
[1816624.848501] CR2: 0000000000000018

Disassembled piece of source from objdump:

/* allocate a buffer from a memory pool */
static void *_stp_mempool_alloc(_stp_mempool_t *pool)
{
     af0:       48 83 ec 18             sub    $0x18,%rsp
     af4:       4c 89 64 24 10          mov    %r12,0x10(%rsp)
        unsigned long flags;
        struct _stp_mem_buffer *ptr = NULL;
        spin_lock_irqsave(&pool->lock, flags);
     af9:       4c 8d 67 18             lea    0x18(%rdi),%r12
        return NULL;
}

/* allocate a buffer from a memory pool */
static void *_stp_mempool_alloc(_stp_mempool_t *pool)
{
     afd:       48 89 1c 24             mov    %rbx,(%rsp)
     b01:       48 89 fb                mov    %rdi,%rbx
     b04:       48 89 6c 24 08          mov    %rbp,0x8(%rsp)
        unsigned long flags;
        struct _stp_mem_buffer *ptr = NULL;
        spin_lock_irqsave(&pool->lock, flags);
     b09:       4c 89 e7                mov    %r12,%rdi
     b0c:       e8 00 00 00 00          callq  b11 <_stp_mempool_alloc+0x21>
 * list_empty - tests whether a list is empty
 * @head: the list to test.
 */

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> 2012-11-29 20:17:09 UTC ---
OK, the problem appears to be that early during initialization, an _stp_error
is generated, but the mechanisms to transmit that message back to userspace
aren't as yet set up (_stp_mempool_init not yet called).  This is a bug.

Would you mind trying to rerun the stap module with -DSTAP_DEBUG_PRINTK?  That
should echo the error to printk first, just for curiosity.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> 2012-11-30 05:09:25 UTC ---
Can you check whether commit 5fadcec helps?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[jiri.horky at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

--- Comment #6 from Jiri Horky <jiri.horky at gmail dot com> 2012-11-30 08:11:41 UTC ---
Hi, 

the commit 5fadcec seems to fix this problem, "hello world" is printed fine. I
will try to run some more non-trivial stap modules test it a little bit more.

Thanks for your help.

Jiri Horky

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/14804] Kernel panic when running hello world stap script

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14804

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |FIXED

--- Comment #7 from Frank Ch. Eigler <fche at redhat dot com> 2012-11-30 14:49:17 UTC ---
appears slightly less ungood

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.