From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 32653 invoked by alias); 26 Dec 2011 18:33:49 -0000 Received: (qmail 32645 invoked by uid 22791); 26 Dec 2011 18:33:48 -0000 X-SWARE-Spam-Status: No, hits=-7.0 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,SPF_HELO_PASS,TW_ZF X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 26 Dec 2011 18:33:31 +0000 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id pBQIXRxf024995 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 26 Dec 2011 13:33:27 -0500 Received: from [10.3.113.6] ([10.3.113.6]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id pBQIXQQY028257; Mon, 26 Dec 2011 13:33:27 -0500 Message-ID: <4EF8BDF6.2070105@redhat.com> Date: Tue, 27 Dec 2011 22:06:00 -0000 From: Josh Stone User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:9.0) Gecko/20111222 Thunderbird/9.0 MIME-Version: 1.0 To: Chris Dunlop CC: systemtap@sourceware.org Subject: Re: Error removing module: Device or resource busy References: <20111223050020.GA11829@onthe.net.au> <4EF4E089.6060008@redhat.com> <20111225235927.GA2907@onthe.net.au> In-Reply-To: <20111225235927.GA2907@onthe.net.au> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q4/txt/msg00428.txt.bz2 On 12/25/2011 03:59 PM, Chris Dunlop wrote: >> But it seems like your stap modules are being built without >> this machinery, so corruption ensues. > > By "this machinery", do you mean the ->exit callback, and if so, is this the > ->exit callback? > > # grep module_exit /root/.systemtap/cache/6b/stap_6bbd9bcdc91b5b9122793a314d034588_795.c > static void systemtap_module_exit (void) { I meant the code involved in refcounting and module exits, basically everything guarded by #ifdef CONFIG_MODULE_UNLOAD in struct module. Our exit path is more involved than just systemtap_module_exit, but it starts with cleanup_exit as is standard for all kernel modules. Then the kernel's scripts/mod/modpost.c creates the struct module with .exit = cleanup_module. This too is guarded by #ifdef CONFIG_MODULE_UNLOAD. If you run a stap example with "-p4 -k", it will leave the temporary directory around so you can examine all the build result. The struct module is written into .mod.c. It > Without any of the zfs modules loaded... And a clean boot as such? Just want to make sure in case we're dealing with corruption by zfs, that it doesn't get that chance. > # stap -v -e 'probe begin {printf("foo\n"); exit()}' > Pass 1: parsed user script and 78 library script(s) using 77348virt/21892res/2620shr kb, in 110usr/20sys/170real ms. > Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77876virt/22684res/2852shr kb, in 0usr/0sys/3real ms. > Pass 3: translated to C into "/tmp/stapSBQNZp/stap_6bbd9bcdc91b5b9122793a314d034588_795_src.c" using 77876virt/22768res/2924shr kb, in 0usr/0sys/1real ms. > Pass 4: compiled C into "stap_6bbd9bcdc91b5b9122793a314d034588_795.ko" in 1120usr/160sys/1889real ms. > Pass 5: starting run. > foo > Error removing module 'stap_6bbd9bcdc91b5b9122793a314d03458_5786': Device or resource busy. > WARNING: /usr/bin/staprun exited with status: 1 > Pass 5: run completed in 20usr/0sys/423real ms. > Pass 5: run failed. Try again with another '--vp 00001' option. If only stap was working, so we could write some scripts to track down why stap isn't working... :/ Something else that occurs to me - in 1.6 we added code to auto-rename modules as they are loaded to avoid conflicts. That requires poking in struct module, so that could be causing issues for you. This is in the -R option to staprun, which stap always sets unless you gave your own module name. So two ways you can test around this: - Run stap -p4 to get the module, then staprun it without -R. - Run stap -m NAME to assign a specific module name. Josh