[Bug runtime/15408] New: procfs probes broken on rawhide

[Bug runtime/15408] New: procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

             Bug #: 15408
           Summary: procfs probes broken on rawhide
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
        AssignedTo: systemtap@sourceware.org
        ReportedBy: dsmith@redhat.com
    Classification: Unclassified


On rawhide (3.8.8-202.fc18.x86_64) procfs probes silently fail. The
/proc/systemtap directory never gets created. No errors are displayed.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

--- Comment #1 from David Smith <dsmith at redhat dot com> 2013-04-26 17:28:55 UTC ---
I found the following in /var/log/messages:

Systemtap Error at _stp_mkdir_proc_module:139 Unable to lock transport
directory.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

--- Comment #2 from David Smith <dsmith at redhat dot com> 2013-04-30 16:24:56 UTC ---
The problem appears to be related to kernel version. Somewhere between 3.6 and
3.8 locking the transport directory repeatedly fails.

Working kernels:

2.6.32-279.14.1.el6.x86_64
3.6.11-1.fc17.x86_64

Failing kernels:

3.8.4-102.fc17.x86_64
3.8.9-200.fc18.x86_64
3.9.0-0.rc8.git0.1.fc20.x86_64

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

--- Comment #3 from David Smith <dsmith at redhat dot com> 2013-05-02 16:04:31 UTC ---
Since kernel 3.7.9-205.fc18.x86_64 also fails here, it looks like this started
failing somewhere between 3.6 and 3.7.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

--- Comment #4 from David Smith <dsmith at redhat dot com> 2013-05-07 15:37:53 UTC ---
I believe this started happening when debugfs was converted to use user
namespaces via the following kernel commit:

====
commit 7dc05881b64792e0ea41293e9595cc962a716225
Author: Eric W. Biederman <ebiederm@xmission.com>
Date:   Tue Apr 3 14:01:31 2012 -0700

    userns: Convert debugfs to use kuid/kgid where appropriate.

    Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
    Acked-by: Serge Hallyn <serge.hallyn@canonical.com>
    Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
====

I've figured out that when this fails, it fails because we can't create the
transport locking directory. That fails because of an EACCESS error. When
creating the transport locking directory works, our euid is 0. When it fails,
it is the user's uid (who doesn't have access to create a debugfs directory).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> 2013-05-07 15:40:34 UTC ---
Then this sounds a little like bug #14223, wherein kernel permission checks are
naive to the extent that they perform this work not at the user-kernel
interface, but several layers down within the kernel (which is sort of
setuid-root already).

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

--- Comment #6 from David Smith <dsmith at redhat dot com> 2013-05-07 16:32:50 UTC ---
Thinking about this some more, here's what's going on.

staprun (which is setuid 0) loads the module. At this point the following call
chain occurs:

  init_module() ->
   _stp_transport_init() ->
     _stp_transport_data_fs_start() ->
       _stp_lock_transport_dir()

So, while staprun is running our euid is 0. We can lock the transport
directory, which really just creates a debugfs directory.

Then staprun execs stapio.

stapio (which isn't a setuid program) sends STP_START, which eventually calls
the module's 'systemtap_module_init()' function. During systemtap_module_init()
when there are procfs probes, we call _stp_lock_transport_dir(). At this point,
since our euid isn't 0, we can't create the debugfs directory.

Note that procfs probes don't really need to lock the transport directory. This
was done so that simultaneously running systemtap modules don't create multiple
/proc/systemtap directories (which the proc file system doesn't disallow).

To fix this problem, we can try to change our euid back to 0 or we could come
up with a different method of handing simultaneously running systemtap modules
trying to create the /proc/systemtap directory. The latter sounds safer.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #7 from David Smith <dsmith at redhat dot com> 2013-05-07 21:23:43 UTC ---
Fixed in commit be66b6e. Now creates top-level procfs directory during
module_init() (when staprun is running), instead of during
systemtap_module_init() (when stapio is running). When staprun is running, we
have root access (euid == 0), since it is a setuid program.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #8 from Josh Stone <jistone at redhat dot com> 2013-05-09 23:10:41 UTC ---
(In reply to comment #7)
> Fixed in commit be66b6e. Now creates top-level procfs directory during
> module_init() (when staprun is running), instead of during
> systemtap_module_init() (when stapio is running). When staprun is running, we
> have root access (euid == 0), since it is a setuid program.

This commit breaks RHEL4 support, so even the simplest script fails as:
> Error inserting module '[stap...].ko': Operation not permitted

The kernel log says:
> Systemtap Error at _stp_register_ctl_channel:597 Error creating systemtap control channel.

Then I looked and see /proc/systemtap/ is left there with a stap_...
subdirectory.  When I tried to read that, the kernel crashes from
proc_lookup().

IIRC, in RHEL4 we use procfs for transport (vs. debugfs everywhere else), so
perhaps that's conflicting with your change?  I'm quite sure though, reverting
just your commit clears it up.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/15408] procfs probes broken on rawhide

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15408

--- Comment #9 from David Smith <dsmith at redhat dot com> 2013-05-10 14:10:41 UTC ---
RHEL4 breakage fixed in commit a2c7d92. Thanks for finding this.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.