From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 4503 invoked by alias); 28 Dec 2011 08:07:54 -0000 Received: (qmail 4492 invoked by uid 22791); 28 Dec 2011 08:07:50 -0000 X-SWARE-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,TW_ZF X-Spam-Check-By: sourceware.org Received: from smtp1.onthe.net.au (HELO smtp1.onthe.net.au) (203.22.196.249) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Wed, 28 Dec 2011 08:07:35 +0000 Received: from localhost (localhost [127.0.0.1]) by smtp1.onthe.net.au (Postfix) with ESMTP id 4C3E7611C0; Wed, 28 Dec 2011 19:07:32 +1100 (EST) Received: from smtp1.onthe.net.au ([127.0.0.1]) by localhost (smtp1.onthe.net.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BFNzck4GjLMx; Wed, 28 Dec 2011 19:07:29 +1100 (EST) Received: from o1.office.otn.net.au (unknown [180.148.179.26]) by smtp1.onthe.net.au (Postfix) with ESMTP id AC98561081; Wed, 28 Dec 2011 19:07:28 +1100 (EST) Received: from achates.office.onthe.net.au (achates.office.onthe.net.au [10.1.32.53]) by o1.office.otn.net.au (Postfix) with ESMTP id 64F477A6C7; Wed, 28 Dec 2011 19:07:28 +1100 (EST) Received: by achates.office.onthe.net.au (Postfix, from userid 999) id 5041610F; Wed, 28 Dec 2011 19:07:28 +1100 (EST) Date: Wed, 28 Dec 2011 17:08:00 -0000 From: Chris Dunlop To: Josh Stone Cc: systemtap@sourceware.org Subject: Re: Error removing module: Device or resource busy Message-ID: <20111228080728.GA12001@onthe.net.au> References: <20111223050020.GA11829@onthe.net.au> <4EF4E089.6060008@redhat.com> <20111225235927.GA2907@onthe.net.au> <4EF8BDF6.2070105@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EF8BDF6.2070105@redhat.com> User-Agent: Mutt/1.5.20 (2009-06-14) Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q4/txt/msg00430.txt.bz2 On Mon, Dec 26, 2011 at 10:33:26AM -0800, Josh Stone wrote: > On 12/25/2011 03:59 PM, Chris Dunlop wrote: >>> But it seems like your stap modules are being built without >>> this machinery, so corruption ensues. >> >> By "this machinery", do you mean the ->exit callback, and if so, is this the >> ->exit callback? >> >> # grep module_exit /root/.systemtap/cache/6b/stap_6bbd9bcdc91b5b9122793a314d034588_795.c >> static void systemtap_module_exit (void) { > > I meant the code involved in refcounting and module exits, basically > everything guarded by #ifdef CONFIG_MODULE_UNLOAD in struct module. There's just the ".exit = cleanup_module" guarded in stap_test1.mod.c: struct module __this_module __attribute__((section(".gnu.linkonce.this_module"))) = { .name = KBUILD_MODNAME, .init = init_module, #ifdef CONFIG_MODULE_UNLOAD .exit = cleanup_module, #endif .arch = MODULE_ARCH_INIT, }; > Our exit path is more involved than just systemtap_module_exit, but it > starts with cleanup_exit as is standard for all kernel modules. Then > the kernel's scripts/mod/modpost.c creates the struct module with .exit > = cleanup_module. This too is guarded by #ifdef CONFIG_MODULE_UNLOAD. > > If you run a stap example with "-p4 -k", it will leave the temporary > directory around so you can examine all the build result. The struct > module is written into .mod.c. It I tested the #ifdef CONFIG_MODULE_UNLOAD by putting garbage within the #ifdef and manually compiling it (where the compile was per the output of "stap -vvvvvvv"). As expected the compilation failed at the garbage, i.e. CONFIG_MODULE_UNLOAD is definately defined. >> Without any of the zfs modules loaded... > > And a clean boot as such? Just want to make sure in case we're dealing > with corruption by zfs, that it doesn't get that chance. Yes, fresh boot without any zfs (ditto for all subsequent tests). Note: the othe thing about this box that might be slightly unusual is that it's using nfs-root. Unfortunately it would be difficult to try it without that. >> # stap -v -e 'probe begin {printf("foo\n"); exit()}' >> Pass 1: parsed user script and 78 library script(s) using 77348virt/21892res/2620shr kb, in 110usr/20sys/170real ms. >> Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77876virt/22684res/2852shr kb, in 0usr/0sys/3real ms. >> Pass 3: translated to C into "/tmp/stapSBQNZp/stap_6bbd9bcdc91b5b9122793a314d034588_795_src.c" using 77876virt/22768res/2924shr kb, in 0usr/0sys/1real ms. >> Pass 4: compiled C into "stap_6bbd9bcdc91b5b9122793a314d034588_795.ko" in 1120usr/160sys/1889real ms. >> Pass 5: starting run. >> foo >> Error removing module 'stap_6bbd9bcdc91b5b9122793a314d03458_5786': Device or resource busy. >> WARNING: /usr/bin/staprun exited with status: 1 >> Pass 5: run completed in 20usr/0sys/423real ms. >> Pass 5: run failed. Try again with another '--vp 00001' option. > > If only stap was working, so we could write some scripts to track down > why stap isn't working... :/ :/ ... on the other hand, it seems to be working up to the point of unloading. So perhaps? > Something else that occurs to me - in 1.6 we added code to auto-rename > modules as they are loaded to avoid conflicts. That requires poking in > struct module, so that could be causing issues for you. This is in the > -R option to staprun, which stap always sets unless you gave your own > module name. So two ways you can test around this: No joy either way I'm afraid. > - Run stap -p4 to get the module, then staprun it without -R. fresh boot, no zfs... # lsmod | grep zfs # stap -v -p4 -k -m stap_test1 -e 'probe begin {printf("foo\n"); exit()}' Pass 1: parsed user script and 78 library script(s) using 77216virt/21892res/2632shr kb, in 110usr/20sys/248real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77744virt/22684res/2864shr kb, in 0usr/0sys/3real ms. Pass 3: translated to C into "/tmp/stapFBxZ92/stap_test1_src.c" using 77744virt/22684res/2864shr kb, in 0usr/0sys/4real ms. stap_test1.ko Pass 4: compiled C into "stap_test1.ko" in 1160usr/160sys/2679real ms. Keeping temporary directory "/tmp/stapFBxZ92" # staprun stap_test1.ko foo Error removing module 'stap_test1': Device or resource busy. # lsmod | grep stap stap_test1 26018 4294967295 [permanent] kern.log: kernel: [ 256.496145] Kprobes globally unoptimized kernel: [ 256.515740] stap_test1: systemtap: 1.7/0.152, base: ffffffffa0367000, memory: 12data/13text/9ctx/2058net/133alloc kb, probes: 1 > - Run stap -m NAME to assign a specific module name. fresh boot, no zfs... # lsmod | grep zfs # stap -v -k -m stap_test1 -e 'probe begin {printf("foo\n"); exit()}' Pass 1: parsed user script and 78 library script(s) using 77356virt/21900res/2632shr kb, in 110usr/20sys/247real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77884virt/22692res/2864shr kb, in 0usr/0sys/3real ms. Pass 3: translated to C into "/tmp/stapdmru1Y/stap_test1_src.c" using 77884virt/22692res/2864shr kb, in 0usr/0sys/4real ms. Pass 4: compiled C into "stap_test1.ko" in 1150usr/180sys/2798real ms. Pass 5: starting run. foo Error removing module 'stap_test1': Device or resource busy. WARNING: /usr/bin/staprun exited with status: 1 Pass 5: run completed in 0usr/20sys/442real ms. Pass 5: run failed. Try again with another '--vp 00001' option. Keeping temporary directory "/tmp/stapdmru1Y" # lsmod | grep stap stap_test1 26018 4294967295 [permanent] kern.log: b5 kernel: [ 164.802684] Kprobes globally unoptimized b5 kernel: [ 164.820723] stap_test1: systemtap: 1.7/0.152, base: ffffffffa0300000, memory: 12data/13text/9ctx/2058net/133alloc kb, probes: 1