From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 26915 invoked by alias); 23 Dec 2011 05:00:44 -0000 Received: (qmail 26893 invoked by uid 22791); 23 Dec 2011 05:00:40 -0000 X-SWARE-Spam-Status: No, hits=-1.6 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD,TW_JB,TW_PC,TW_UH X-Spam-Check-By: sourceware.org Received: from smtp1.onthe.net.au (HELO smtp1.onthe.net.au) (203.22.196.249) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Fri, 23 Dec 2011 05:00:24 +0000 Received: from localhost (localhost [127.0.0.1]) by smtp1.onthe.net.au (Postfix) with ESMTP id 0426361145 for ; Fri, 23 Dec 2011 16:00:22 +1100 (EST) Received: from smtp1.onthe.net.au ([127.0.0.1]) by localhost (smtp1.onthe.net.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6+McrnvV24Y8 for ; Fri, 23 Dec 2011 16:00:21 +1100 (EST) Received: from o1.office.otn.net.au (unknown [180.148.179.26]) by smtp1.onthe.net.au (Postfix) with ESMTP id D3CB46114B for ; Fri, 23 Dec 2011 16:00:20 +1100 (EST) Received: from achates.office.onthe.net.au (achates.office.onthe.net.au [10.1.32.53]) by o1.office.otn.net.au (Postfix) with ESMTP id B56CA7A6C7 for ; Fri, 23 Dec 2011 16:00:20 +1100 (EST) Received: by achates.office.onthe.net.au (Postfix, from userid 999) id BE5F32A7; Fri, 23 Dec 2011 16:00:20 +1100 (EST) Date: Fri, 23 Dec 2011 08:21:00 -0000 From: Chris Dunlop To: systemtap@sourceware.org Subject: Error removing module: Device or resource busy Message-ID: <20111223050020.GA11829@onthe.net.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q4/txt/msg00405.txt.bz2 Hi, Linux-3.0.13, systemtap 6.1 and now HEAD(b6a3da4) I've just started using systemtap to help diagnose a problem with zfsonlinux[1]. Whenever I run stap[2], the output ends with: Error removing module 'stap_fcac0085842418e34d8094455dc203e8_1_21605': Device or resource busy. (obviously the module name changes) and the module is still loaded: # lsmod | grep stap stap_fcac0085842418e34d8094455dc203e8_1_21605 2896285 1027571582 [permanent] Using "rmmod -f" on that module results in a kernel NULL pointer dereference[3]. Any suggestions about how to avoid the unloadable module? Cheers, Chris [1] https://github.com/zfsonlinux/zfs/issues/503 ---------------------------------------------------------------------- [2] stap usage ---------------------------------------------------------------------- # stap \ -c "getfattr -d file" \ para-callgraph.stp \ 'module("zfs").function("*@module/zfs/*.c")' \ 'module("zfs").function("zpl_xattr_list")' # cat para-callgraph.stp #! /usr/bin/env stap # # http://sourceware.org/systemtap/examples/general/para-callgraph.stp # function trace(entry_p, extra) { %( $# > 1 %? if (tid() in trace) %) printf("%s%s%s %s\n", thread_indent (entry_p), (entry_p>0?"->":"<-"), probefunc (), extra) } %( $# > 1 %? global trace probe $2.call { trace[tid()] = 1 } probe $2.return { delete trace[tid()] } %) probe $1.call { trace(1, $$parms) } probe $1.return { trace(-1, $$return) } ---------------------------------------------------------------------- [3] kernel message ---------------------------------------------------------------------- kernel: [ 292.856177] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 kernel: [ 292.856392] IP: [] free_percpu+0xa4/0x112 kernel: [ 292.856531] PGD 611554067 PUD 5e1622067 PMD 0 kernel: [ 292.856754] Oops: 0000 [#1] SMP kernel: [ 292.856927] CPU 5 kernel: [ 292.856980] Modules linked in: ext4 mbcache jbd2 crc16 loop fuse nfsd exportfs zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate sha256_generic aesni_intel cryptd aes_x86_64 aes_generic cbc dm_crypt dm_mod nfs lockd auth_rpcgss nfs_acl sunrpc bridge stp llc sg sd_mod usbhid hid uhci_hcd mpt2sas psmouse scsi_transport_sas raid_class scsi_mod i2c_i801 i2c_core ioatdma ehci_hcd igb dca button processor thermal_sys [last unloaded: stap_152d5b1673c7d1a7749e69173a687f9a_12_6436] kernel: [ 292.859810] kernel: [ 292.859886] Pid: 6444, comm: rmmod Tainted: P R 3.0.13-otn-00016-g1306024 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH kernel: [ 292.860152] RIP: 0010:[] [] free_percpu+0xa4/0x112 kernel: [ 292.860319] RSP: 0018:ffff8805d4371e58 EFLAGS: 00010046 kernel: [ 292.860406] RAX: 0000000000000000 RBX: 0000000000000287 RCX: ffff880be0261dde kernel: [ 292.860498] RDX: 00003ffffffff000 RSI: ffff880000000000 RDI: 0000000000000000 kernel: [ 292.860591] RBP: ffff8805d4371e78 R08: ffffffff8117c1ab R09: ffffffff8100da91 kernel: [ 292.860684] R10: ffffffff8107487f R11: ffff88062763dfa0 R12: ffff880c3ffdde40 kernel: [ 292.860777] R13: ffff8805e0221dde R14: ffffffffa0728a28 R15: 0000000000000008 kernel: [ 292.860870] FS: 00007f9150eb8700(0000) GS:ffff880c3fc40000(0000) knlGS:0000000000000000 kernel: [ 292.860989] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [ 292.861078] CR2: 0000000000000020 CR3: 000000060ffa1000 CR4: 00000000000006e0 kernel: [ 292.861171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 292.861263] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 292.861356] Process rmmod (pid: 6444, threadinfo ffff8805d4370000, task ffff88062155c350) kernel: [ 292.861476] Stack: kernel: [ 292.861551] ffffffff8117c1ab ffffffffa0728a18 ffffffffa0728830 ffffffffa0728a18 kernel: [ 292.861853] ffff8805d4371ea8 ffffffff810758f1 ffffffffa0728830 ffffffffa0728880 kernel: [ 292.862154] 0000000000000002 ffff8805d533e670 ffff8805d4371ed8 ffffffff81075a6f kernel: [ 292.862456] Call Trace: kernel: [ 292.862539] [] ? kobject_del+0x38/0x38 kernel: [ 292.862631] [] module_unload_free+0xca/0xd3 kernel: [ 292.862721] [] free_module+0x175/0x1c6 kernel: [ 292.862810] [] sys_delete_module+0x1f3/0x226 kernel: [ 292.862903] [] system_call_fastpath+0x16/0x1b kernel: [ 292.862992] Code: c2 85 d2 4c 0f 45 25 04 52 54 00 eb 22 48 8b 05 9b f7 3e 00 4c 89 ef 65 8b 14 25 f8 db 00 00 48 63 d2 48 03 3c d0 e8 eb e4 00 00 <4c> 8b 60 20 44 89 ee 41 2b 74 24 18 4c 89 e7 e8 dc fd ff ff 41 kernel: [ 292.866166] RIP [] free_percpu+0xa4/0x112 kernel: [ 292.866302] RSP kernel: [ 292.866383] CR2: 0000000000000020 kernel: [ 292.866464] ---[ end trace 2b4d7238c919f6da ]--- ----------------------------------------------------------------------