[Bug runtime/12164] New: stap script causing ptrace to error with "Device or resource busy"

[Bug runtime/12164] New: stap script causing ptrace to error with "Device or resource busy"

[erick.tryzelaar at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

           Summary: stap script causing ptrace to error with "Device or
                    resource busy"
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
        AssignedTo: systemtap@sources.redhat.com
        ReportedBy: erick.tryzelaar@gmail.com


Hello,

I just discovered that this simple systemtap 1.3 script:

probe kprocess.create {
        printf("hello\n")
}

is causing strace to error out with this message:

strace: ptrace(PTRACE_TRACEME, ...): Device or resource busy

Only after stap and stapio processes have died will strace successfully run. I
have seen this on multiple fedora 11 machines, some running with kernel
2.6.29.4-167.fc11, others with 2.6.31.12-174.2.22.fc11.

I confirmed that this stap script and strace work fine together on systemtap
0.9.9, but does not work for 1.3.2.

One other piece of information is that on the install of systemtap rpm, it does
mention this:

ERROR: Module uprobes does not exist in /proc/modules

Could that be related? Both 1.3.2 and 0.9.9 complain about this issue, so it
may be unrelated.

Thanks for any help!

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12164] stap script causing ptrace to error with "Device or resource busy"

[erick.tryzelaar at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

--- Comment #1 from Erick Tryzelaar <erick.tryzelaar at gmail dot com> 2010-10-27 17:46:20 UTC ---
fche asked for a couple extra datapoints. We saw this happen with "strace df",
on 8 core and 12 core x86_64 machines.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12164] stap script causing ptrace to error with "Device or resource busy"

[erick.tryzelaar at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

--- Comment #2 from Erick Tryzelaar <erick.tryzelaar at gmail dot com> 2010-10-27 17:50:33 UTC ---
Created attachment 5092
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5092
stap-report output

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12164] stap script causing ptrace to error with "Device or resource busy"

[erick.tryzelaar at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

--- Comment #3 from Erick Tryzelaar <erick.tryzelaar at gmail dot com> 2010-10-27 20:11:10 UTC ---
I bisected the bug down to b642c901a918a92fdd6b1167ed02dfe4c50cdd3e:

    Add kprobe user probe semaphore handling.

    Adds stap_kprobe_process_found and stap_kprobe_mmap_found handlers
    and then the task_finder uses those to set the semaphore values.
    The probe adds user_path and user_lib which are used by the
    task_finder to decide what object to sniff.

    * sdt.h (STAP_SEMAPHORE): Also define for kprobes.
    * tapsets.cxx (dwarf_derived_probe): Add user_path and user_lib.
    (dwarf_derived_probe::join_group): Enable task finder for kprobes.
    (dwarf_derived_probe::dwarf_derived_probe): Set user_path and
    user_lib.
    (dwarf_derived_probe_group::emit_module_decls): Emit
    stap_kprobe_mmap_found, sdt_sem_offset, sdt_sem_address, pathname,
    and finder declarations.  Emit stap_kprobe_process_found and
    stap_kprobe_mmap_found.
    (dwarf_derived_probe_group::emit_module_init): Setup task finder.
    (sdt_query::convert_location): Remove TOK_LIBRARY token.
    (dwarf_builder::build): Set user_path and user_lib.
    (kprobe_derived_probe): Add path and library.
    (kprobe_builder::build): Get process and library params

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12164] stap script causing ptrace to error with "Device or resource busy"

[erick.tryzelaar at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

--- Comment #4 from Erick Tryzelaar <erick.tryzelaar at gmail dot com> 2010-10-27 22:53:24 UTC ---
jistone on #systemtap had me run this systemtap script:

probe process.begin {
    printf("hello\n")
}

And we found that I was getting the same "Device or resource busy" when running
the stap script at the same time as "strace df". jistone thinks that this is a
kernel/utrace bug since this was reproducible down to systemtap 0.9.9 on Fedora
11 kernel 2.6.31.12-172.2.22.fc11, but not reproducible on Fedora 13 kernel 13
2.6.33.5-112.fc13.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12164] stap script causing ptrace to error with "Device or resource busy"

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #5 from Josh Stone <jistone at redhat dot com> 2010-10-28 21:05:39 UTC ---
(In reply to comment #3)
> I bisected the bug down to b642c901a918a92fdd6b1167ed02dfe4c50cdd3e:
> 
>     Add kprobe user probe semaphore handling.

So, the reason this commit came up is that it started enabling task_finder when
kprobes are used.  We should probably try to narrow this to just for
SDT-related kprobes, but that's not really the issue here.

(In reply to comment #4)
> jistone on #systemtap had me run this systemtap script: [... process.begin ...]

This was to explicitly use utrace, thus even older stap is affected.

> this was reproducible down to systemtap 0.9.9 on Fedora 11 kernel
> 2.6.31.12-172.2.22.fc11, but not reproducible on Fedora 13 kernel 13
> 2.6.33.5-112.fc13.

I found out from Erick that his F11 kernel is actually a backport from 
2.6.31.12-174.2.22.fc12.  I don't have an F11 VM anymore, but I downloaded this
kernel from koji onto my F12 VM, and sure enough, strace gets EBUSY whenever a
stap-utrace probe is active.  Both i686 and x86_64 have this problem on that
kernel, but work fine on with the latest F12 kernel, 2.6.32.21-168.fc12.

It's not just strace either.  While stap is using utrace, gdb also gets EBUSY
trying to attach to a process, and sees no activity at all when trying to start
its own inferior.  It seems that utrace on that kernel is not meeting its
multiple-client promise...

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug runtime/12164] stap script causing ptrace to error with "Device or resource busy"

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12164

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED
         AssignedTo|systemtap at sources dot    |jistone at redhat dot com
                   |redhat.com                  |

--- Comment #6 from Josh Stone <jistone at redhat dot com> 2010-11-11 21:34:07 UTC ---
(In reply to comment #5)
> (In reply to comment #3)
> > I bisected the bug down to b642c901a918a92fdd6b1167ed02dfe4c50cdd3e:
> > 
> >     Add kprobe user probe semaphore handling.
> 
> So, the reason this commit came up is that it started enabling task_finder when
> kprobes are used.  We should probably try to narrow this to just for
> SDT-related kprobes, but that's not really the issue here.

Added commit 62f52bb "PR12164: Emit kprobes-sdt goo only when needed"

As for the underlying utrace-ptrace problem, we're unlikely to see any action
on it, since this has only been seen on an outdated F12 kernel.  (And F12 is
nearly EOL anyway.)  I'll dig a little in the history to see if there's
something you could backport, but otherwise I'll consider this PR fixed by the
commit above.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.