staprun -c deadlocks often

staprun -c deadlocks often

[Richard W.M. Jones]

systemtap-1.4-9.fc15.x86_64
kernel 2.6.38.7-30.fc15.x86_64

I'm not sure if I'm "doing it wrong", but it appears that the staprun -c
option deadlocks for me very frequently.  Not always, but at least
more often than one time in two.

Here is what I'm doing (as root):

  stap -p4 \
  -e 'probe module("ext2").statement ("*@*.c:*") { printf ("%s\n", pp()); }'

[stap command prints the module name, used below]

  staprun -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko

As I say, more often than half the time, instead of printing out a
directory listing, the staprun command will just hang.  Note that the
filesystem I'm listing is not ext2, so I don't think the probes
themselves have anything to do with it.

When it hangs, a process listing shows:

  13073 pts/4    Ss     0:00  \_ bash
  17930 pts/4    S+     0:00  |   \_ sudo staprun -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko
  17931 pts/4    Sl+    0:00  |       \_ /usr/libexec/systemtap/stapio -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko
  17932 pts/4    S+     0:00  |           \_ /usr/libexec/systemtap/stapio -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko

Note that there is no staprun process, nor any /bin/ls, but two stapio
processes have been left around.

Any ideas?  I had a look at the code in runtime/staprun/mainloop.c but
it just looks fiendishly complex ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

Re: staprun -c deadlocks often

[Frank Ch. Eigler]

Hi, Richard -


rjones wrote:

> systemtap-1.4-9.fc15.x86_64
> kernel 2.6.38.7-30.fc15.x86_64
>
> I'm not sure if I'm "doing it wrong"

Nope, not your fault.

> [...] Here is what I'm doing (as root):
>   stap -p4 \
>   -e 'probe module("ext2").statement ("*@*.c:*") { printf ("%s\n", pp()); }'
> [stap command prints the module name, used below]
>   staprun -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko

Just running stap .... -c /bin/ls should give the same results.  Or
are you cross-compiling?  (If so, there are other methods available
that wouldn't require hand-copying and hand-running staprun.)

> As I say, more often than half the time, instead of printing out a
> directory listing, the staprun command will just hang.  [...]

This sounds like http://sourceware.org/PR10189, where too many warning
messages interfere with non-warning staprun<->module packets.

- FChE

Re: staprun -c deadlocks often

[Richard W.M. Jones]

On Fri, Jun 17, 2011 at 02:43:13PM -0400, Frank Ch. Eigler wrote:
> rjones wrote:
> > [stap command prints the module name, used below]
> >   staprun -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko
> 
> Just running stap .... -c /bin/ls should give the same results.  Or
> are you cross-compiling?  (If so, there are other methods available
> that wouldn't require hand-copying and hand-running staprun.)

The reason I'm using staprun is to avoid recompiling the kernel module
each time.  I'm not sure how necessary this is, since I also noticed
that Systemtap seems to cache the module.  Will it always avoid
recompiling the exact same source?

Anyhow, in order to exclude staprun from the equation, I used just
stap:

sudo \
  stap \
    -o /tmp/stap \
    -c /bin/ls \
    -e 'probe module("ext2").statement ("*@*.c:*") { printf ("%s\n", pp()); }'

I get basically the same problem, although it appears to be slightly
less frequently (1 in 4?  1 in 5?) for some reason.

Process listing when it hangs:

29643 pts/5    Ss     0:00  \_ bash
29934 pts/5    S+     0:00  |   \_ sudo stap -o /tmp/stap -c /bin/ls -e probe mo
29935 pts/5    S+     0:00  |       \_ stap -o /tmp/stap -c /bin/ls -e probe mod
29936 pts/5    Sl+    0:00  |           \_ /usr/libexec/systemtap/stapio -o /tmp
29937 pts/5    S+     0:00  |               \_ /usr/libexec/systemtap/stapio -o


> > As I say, more often than half the time, instead of printing out a
> > directory listing, the staprun command will just hang.  [...]
> 
> This sounds like http://sourceware.org/PR10189, where too many warning
> messages interfere with non-warning staprun<->module packets.

It could be.  I'm still getting warning messages (see previous email).

Is there a way to get it to discard the warning messages?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: staprun -c deadlocks often

[Richard W.M. Jones]

On Fri, Jun 17, 2011 at 02:43:13PM -0400, Frank Ch. Eigler wrote:
> This sounds like http://sourceware.org/PR10189, where too many warning
> messages interfere with non-warning staprun<->module packets.

In fact, reading a bit further into that bug report, it does seem
to be the same thing, since in 'dmesg' I'm seeing:

[64843.757205] ctl_send (type=0 len=8) failed: -12

It's not clear to me if STP_DEFAULT_BUFFERS is something that can be
changed w/o recompiling.  Or is there another workaround?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: staprun -c deadlocks often

[Josh Stone]

On 06/17/2011 08:43 AM, Richard W.M. Jones wrote:
> When it hangs, a process listing shows:
> 
>   13073 pts/4    Ss     0:00  \_ bash
>   17930 pts/4    S+     0:00  |   \_ sudo staprun -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko
>   17931 pts/4    Sl+    0:00  |       \_ /usr/libexec/systemtap/stapio -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko
>   17932 pts/4    S+     0:00  |           \_ /usr/libexec/systemtap/stapio -o /tmp/stap -c /bin/ls /home/rjones/.systemtap/cache/44/stap_44fe38ee6f62656645948ce5d275d678_260427.ko
> 
> Note that there is no staprun process, nor any /bin/ls, but two stapio
> processes have been left around.

It's not surprising that staprun is gone -- once it has loaded the
module, it execs stapio to drop privileges while the script runs.  It
doesn't make any difference here since you're running with sudo, but if
you add yourself to stapusr & stapdev groups and let the setuid staprun
do its thing, then stapio and the -c child will run as your own uid.

Having two stapios supports Frank that this is likely PR10189.  We first
fork stapio so we can report the child's pid to the module, then wait
for the module's STP_START message before the actual child exec.  If
there's no message, we make no progress...

Josh