From: Chris Dunlop <chris@onthe.net.au>
To: Josh Stone <jistone@redhat.com>
Cc: systemtap@sourceware.org
Subject: Re: Error removing module: Device or resource busy
Date: Wed, 28 Dec 2011 17:08:00 -0000 [thread overview]
Message-ID: <20111228080728.GA12001@onthe.net.au> (raw)
In-Reply-To: <4EF8BDF6.2070105@redhat.com>
On Mon, Dec 26, 2011 at 10:33:26AM -0800, Josh Stone wrote:
> On 12/25/2011 03:59 PM, Chris Dunlop wrote:
>>> But it seems like your stap modules are being built without
>>> this machinery, so corruption ensues.
>>
>> By "this machinery", do you mean the ->exit callback, and if so, is this the
>> ->exit callback?
>>
>> # grep module_exit /root/.systemtap/cache/6b/stap_6bbd9bcdc91b5b9122793a314d034588_795.c
>> static void systemtap_module_exit (void) {
>
> I meant the code involved in refcounting and module exits, basically
> everything guarded by #ifdef CONFIG_MODULE_UNLOAD in struct module.
There's just the ".exit = cleanup_module" guarded in stap_test1.mod.c:
struct module __this_module
__attribute__((section(".gnu.linkonce.this_module"))) = {
.name = KBUILD_MODNAME,
.init = init_module,
#ifdef CONFIG_MODULE_UNLOAD
.exit = cleanup_module,
#endif
.arch = MODULE_ARCH_INIT,
};
> Our exit path is more involved than just systemtap_module_exit, but it
> starts with cleanup_exit as is standard for all kernel modules. Then
> the kernel's scripts/mod/modpost.c creates the struct module with .exit
> = cleanup_module. This too is guarded by #ifdef CONFIG_MODULE_UNLOAD.
>
> If you run a stap example with "-p4 -k", it will leave the temporary
> directory around so you can examine all the build result. The struct
> module is written into .mod.c. It
I tested the #ifdef CONFIG_MODULE_UNLOAD by putting garbage within the #ifdef
and manually compiling it (where the compile was per the output of "stap
-vvvvvvv"). As expected the compilation failed at the garbage, i.e.
CONFIG_MODULE_UNLOAD is definately defined.
>> Without any of the zfs modules loaded...
>
> And a clean boot as such? Just want to make sure in case we're dealing
> with corruption by zfs, that it doesn't get that chance.
Yes, fresh boot without any zfs (ditto for all subsequent tests).
Note: the othe thing about this box that might be slightly unusual is that it's
using nfs-root. Unfortunately it would be difficult to try it without that.
>> # stap -v -e 'probe begin {printf("foo\n"); exit()}'
>> Pass 1: parsed user script and 78 library script(s) using 77348virt/21892res/2620shr kb, in 110usr/20sys/170real ms.
>> Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77876virt/22684res/2852shr kb, in 0usr/0sys/3real ms.
>> Pass 3: translated to C into "/tmp/stapSBQNZp/stap_6bbd9bcdc91b5b9122793a314d034588_795_src.c" using 77876virt/22768res/2924shr kb, in 0usr/0sys/1real ms.
>> Pass 4: compiled C into "stap_6bbd9bcdc91b5b9122793a314d034588_795.ko" in 1120usr/160sys/1889real ms.
>> Pass 5: starting run.
>> foo
>> Error removing module 'stap_6bbd9bcdc91b5b9122793a314d03458_5786': Device or resource busy.
>> WARNING: /usr/bin/staprun exited with status: 1
>> Pass 5: run completed in 20usr/0sys/423real ms.
>> Pass 5: run failed. Try again with another '--vp 00001' option.
>
> If only stap was working, so we could write some scripts to track down
> why stap isn't working... :/
:/
... on the other hand, it seems to be working up to the point of unloading. So perhaps?
> Something else that occurs to me - in 1.6 we added code to auto-rename
> modules as they are loaded to avoid conflicts. That requires poking in
> struct module, so that could be causing issues for you. This is in the
> -R option to staprun, which stap always sets unless you gave your own
> module name. So two ways you can test around this:
No joy either way I'm afraid.
> - Run stap -p4 to get the module, then staprun it without -R.
fresh boot, no zfs...
# lsmod | grep zfs
# stap -v -p4 -k -m stap_test1 -e 'probe begin {printf("foo\n"); exit()}'
Pass 1: parsed user script and 78 library script(s) using 77216virt/21892res/2632shr kb, in 110usr/20sys/248real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77744virt/22684res/2864shr kb, in 0usr/0sys/3real ms.
Pass 3: translated to C into "/tmp/stapFBxZ92/stap_test1_src.c" using 77744virt/22684res/2864shr kb, in 0usr/0sys/4real ms.
stap_test1.ko
Pass 4: compiled C into "stap_test1.ko" in 1160usr/160sys/2679real ms.
Keeping temporary directory "/tmp/stapFBxZ92"
# staprun stap_test1.ko
foo
Error removing module 'stap_test1': Device or resource busy.
# lsmod | grep stap
stap_test1 26018 4294967295 [permanent]
kern.log:
kernel: [ 256.496145] Kprobes globally unoptimized
kernel: [ 256.515740] stap_test1: systemtap: 1.7/0.152, base: ffffffffa0367000, memory: 12data/13text/9ctx/2058net/133alloc kb, probes: 1
> - Run stap -m NAME to assign a specific module name.
fresh boot, no zfs...
# lsmod | grep zfs
# stap -v -k -m stap_test1 -e 'probe begin {printf("foo\n"); exit()}'
Pass 1: parsed user script and 78 library script(s) using 77356virt/21900res/2632shr kb, in 110usr/20sys/247real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 77884virt/22692res/2864shr kb, in 0usr/0sys/3real ms.
Pass 3: translated to C into "/tmp/stapdmru1Y/stap_test1_src.c" using 77884virt/22692res/2864shr kb, in 0usr/0sys/4real ms.
Pass 4: compiled C into "stap_test1.ko" in 1150usr/180sys/2798real ms.
Pass 5: starting run.
foo
Error removing module 'stap_test1': Device or resource busy.
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run completed in 0usr/20sys/442real ms.
Pass 5: run failed. Try again with another '--vp 00001' option.
Keeping temporary directory "/tmp/stapdmru1Y"
# lsmod | grep stap
stap_test1 26018 4294967295 [permanent]
kern.log:
b5 kernel: [ 164.802684] Kprobes globally unoptimized
b5 kernel: [ 164.820723] stap_test1: systemtap: 1.7/0.152, base: ffffffffa0300000, memory: 12data/13text/9ctx/2058net/133alloc kb, probes: 1
next prev parent reply other threads:[~2011-12-28 8:07 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-23 8:21 Chris Dunlop
2011-12-23 21:10 ` Josh Stone
2011-12-26 18:33 ` Chris Dunlop
2011-12-27 22:06 ` Josh Stone
2011-12-28 17:08 ` Chris Dunlop [this message]
2012-01-21 1:18 ` Josh Stone
2012-01-21 1:36 ` Chris Dunlop
2012-01-21 2:30 ` Chris Dunlop
2012-01-21 7:27 ` Josh Stone
2012-01-23 7:25 ` Chris Dunlop
2012-01-25 12:31 ` Frank Ch. Eigler
2012-01-26 0:30 ` Chris Dunlop
2012-01-26 1:34 ` Josh Stone
2012-01-26 2:28 ` Chris Dunlop
2012-01-26 18:47 ` Josh Stone
2012-01-26 23:32 ` Chris Dunlop
2012-01-21 1:40 ` Chris Dunlop
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111228080728.GA12001@onthe.net.au \
--to=chris@onthe.net.au \
--cc=jistone@redhat.com \
--cc=systemtap@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).