From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 20077 invoked by alias); 23 Dec 2011 20:12:21 -0000 Received: (qmail 20051 invoked by uid 22791); 23 Dec 2011 20:12:10 -0000 X-SWARE-Spam-Status: No, hits=-7.6 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,SPF_HELO_PASS,TW_ZF X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Fri, 23 Dec 2011 20:11:57 +0000 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pBNKBsXl006376 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 23 Dec 2011 15:11:54 -0500 Received: from [10.3.113.27] (ovpn-113-27.phx2.redhat.com [10.3.113.27]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id pBNKBsc5008901; Fri, 23 Dec 2011 15:11:54 -0500 Message-ID: <4EF4E089.6060008@redhat.com> Date: Fri, 23 Dec 2011 21:10:00 -0000 From: Josh Stone User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:9.0) Gecko/20111222 Thunderbird/9.0 MIME-Version: 1.0 To: Chris Dunlop CC: systemtap@sourceware.org Subject: Re: Error removing module: Device or resource busy References: <20111223050020.GA11829@onthe.net.au> In-Reply-To: <20111223050020.GA11829@onthe.net.au> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q4/txt/msg00418.txt.bz2 On 12/22/2011 09:00 PM, Chris Dunlop wrote: > Hi, > > Linux-3.0.13, systemtap 6.1 and now HEAD(b6a3da4) Your dump looks like x86_64 - is this on any particular distro? Are you building this kernel yourself? > I've just started using systemtap to help diagnose a problem with zfsonlinux[1]. Cool! > Whenever I run stap[2], the output ends with: > > Error removing module 'stap_fcac0085842418e34d8094455dc203e8_1_21605': Device or resource busy. > > (obviously the module name changes) and the module is still loaded: > > # lsmod | grep stap > stap_fcac0085842418e34d8094455dc203e8_1_21605 2896285 1027571582 [permanent] This is odd. Reading kernel/module.c, the "[permanent]" should only come about if the module has no ->exit callback. And 1027571582 is the field for the module refcount, which doesn't look plausible. You must have CONFIG_MODULE_UNLOAD=y, or else the kernel just prints a dummy " - -" in place of the refcount. Though I'm not certain how lsmod translates that, so you might check /proc/modules directly. But it seems like your stap modules are being built without this machinery, so corruption ensues. Are you sure that /lib/modules/`uname -r`/build matches the running kernel? > Using "rmmod -f" on that module results in a kernel NULL pointer dereference[3]. This is bound to crash, even if all else is solved. If the module exit routines don't work, then probes may be left installed, and will certainly die as soon as it tries to call the handler from unloaded memory. So don't waste any time trying to get "rmmod -f" working. Fedora, for instance, does not enable CONFIG_MODULE_FORCE_UNLOAD at all, because there are very few cases where this would do anything sane. > Any suggestions about how to avoid the unloadable module? If you can boot without any of the zfs stuff, then I'd experiment with that first, just to make sure stap is lined up with all the correct kernel build infrastructure. Try something simple, like one of the syscall examples. Josh