[Bug runtime/15663] New: Failing to remove the stap kernel modules on kernel 3.9.4

[Bug runtime/15663] New: Failing to remove the stap kernel modules on kernel 3.9.4

[agentzh at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15663

            Bug ID: 15663
           Summary: Failing to remove the stap kernel modules on kernel
                    3.9.4
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com

On Linux kernel 3.9.4 x86_64, we're always seeing that stap fails to remove the
kernel module upon exit:

  # stap -e 'probe begin { print("Hello\n") exit() }'
  Hello
  Error removing module 'stap_72392efe884b8779ed83364cfbcd97e0_965': Device or
resource busy.
  WARNING: /usr/local/systemtap/bin/staprun exited with status: 1
  Pass 5: run failed.  [man error::pass5]

And we can see the kernel modules are still there:

  # lsmod | grep stap
  stap_db98a89434ce6c6d11c1e43dd3b758df_11003   242671  0 [permanent]
  stap_41ed56b577691483d1b190b51ee07fe1_12277   241903  0 [permanent]

The systemtap version that we're using:

  # stap --version
  Systemtap translator/driver (version 2.2/0.155, non-git sources)
  Copyright (C) 2005-2013 Red Hat, Inc. and others
  This is free software; see the source for copying conditions.
  enabled features: BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS

And CONFIG_MODULE_UNLOAD is enabled in our kernel:

  # grep CONFIG_MODULE_UNLOAD /boot/config-`uname -r`
  CONFIG_MODULE_UNLOAD=y

Any help will be highly appreciated.

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15663] Failing to remove the stap kernel modules on kernel 3.9.4

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15663

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #1 from Josh Stone <jistone at redhat dot com> ---
(In reply to agentzh from comment #0)
>   # lsmod | grep stap
>   stap_db98a89434ce6c6d11c1e43dd3b758df_11003   242671  0 [permanent]
>   stap_41ed56b577691483d1b190b51ee07fe1_12277   241903  0 [permanent]

The [permanent] is somewhat reminiscent of this thread:
http://sourceware.org/ml/systemtap/2012-q1/msg00041.html

... although that case was also indicated by an implausible reference count,
where you have a seemingly correct 0.  But to summarize, the issue there was
that the gcc version used for the kernel was different than the gcc available
to stap for compiling kernel modules.  That led kbuild to have a difference in
-DCC_HAVE_ASM_GOTO, which then sets HAVE_JUMP_LABEL, which causes the layout of
struct module to change.

So please check the compiler versions first.  If these do match, then we'll
continue looking elsewhere.  Check dmesg around the time stap was active, and
also consider attaching the output of stap-report here.


PS: As a diagnostic for that particular issue, there might actually be a way to
compare sizeof(struct module) from vmlinux to our own modules.  We could
basically compare a @cast "module" between "kernel" (for vmlinux) and
"kernel<linux/module.h>" (thus generated by us) -- although with stap internals
it doesn't have to literally be a @cast, just something that finds those types
and compares the DWARF sizes.  This would of course only work when kernel
debuginfo is available, but it may prevent some headaches.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15663] Failing to remove the stap kernel modules on kernel 3.9.4

[agentzh at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15663

agentzh <agentzh at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #2 from agentzh <agentzh at gmail dot com> ---
Yes, it's indeed caused by a gcc version mismatch. The default gcc when running
stap is gcc 4.4.7 while the kernel was compiled by gcc 4.7.3 and 4.8.0. By
explicitly specifying the "-B gcc-4.7" option by running stap, the problem
disappears immediately.

And yeah, I also hope stap can have a way to check such issues automatically to
prevent such weird issues :) I'll think about Josh Stone's proposal above :)

-- 
You are receiving this mail because:
You are the assignee for the bug.