Getting userspace stack traces with userspace probes ?

Getting userspace stack traces with userspace probes ?

[Daniel P. Berrange]

I'm doing some troubleshooting of the libvirt daemon
using userspace probes. My stap script looks like

  probe process("/usr/lib64/libvirt.so").function("virDomainObjUnref") {
      printf("Unref %p %d\n", $dom, $dom->refs);
  }

  probe process("/usr/lib64/libvirt.so").function("virDomainObjRef") {
      printf("Ref %p %d\n", $dom, $dom->refs);
  }

  probe process("/usr/lib64/libvirt.so").function("virDomainObjNew").return {
      printf("New %p %d\n", $return, $return->refs);
  }

This is printing what I expect. I now want to get associated
userspace stack traces, so I tried adding

      print_ustack(ubacktrace())

And / or

      print_ubacktrace()

To all the probe points, all it ever prints for a stack trace is

 0x3992a52e10 : virDomainObjUnref+0x0/0x100 [libvirt.so.0.8.4]
 0x45900f [libvirtd+0x5900f/0xe8000]

Which is far too small compared to what GDB says the stack
trace is, eg I'm expecting at least 6-8 levels deep

  (gdb) bt
  #0  virDomainObjUnref (dom=0x1b49390) at conf/domain_conf.c:828
  #1  0x0000000000470f33 in qemuMonitorFree (mon=0x1cb6640) at qemu/qemu_monitor.c:202
  #2  qemuMonitorUnref (mon=0x1cb6640) at qemu/qemu_monitor.c:221
  #3  0x0000000000416ebf in virEventCleanupHandles () at event.c:548
  #4  0x0000000000417c8e in virEventRunOnce () at event.c:612
  #5  0x0000000000418593 in qemudOneLoop () at libvirtd.c:2226
  #6  0x00000000004188b9 in qemudRunLoop (opaque=0x1b42850) at libvirtd.c:2335
  #7  0x0000003980e07761 in start_thread (arg=0x7f6e63423700) at pthread_create.c:301
  #8  0x00000039806e098d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Am I using the wrong functions for getting stack traces,
or is it just not possible to get reliable stacks from
userspace probes ?

FYI, I'm using systemtap-1.3-3.fc13.x86_64 and
kernel 2.6.34.7-66.fc13.x86_64

Regards,
Daniel

Re: Getting userspace stack traces with userspace probes ?

[Mark Wielaard]

Hi Daniel,

On Wed, 2011-01-26 at 16:17 +0000, Daniel P. Berrange wrote:
> This is printing what I expect. I now want to get associated
> userspace stack traces, so I tried adding
> 
>       print_ustack(ubacktrace())
> 
> And / or
> 
>       print_ubacktrace()
> 
> To all the probe points, all it ever prints for a stack trace is
> 
>  0x3992a52e10 : virDomainObjUnref+0x0/0x100 [libvirt.so.0.8.4]
>  0x45900f [libvirtd+0x5900f/0xe8000]

   ^- this is a very poor hint from systemtap that it found an
      address in a module (libvirtd) but could find the symbol
      (and by extension the unwind table).

> Which is far too small compared to what GDB says the stack
> trace is, eg I'm expecting at least 6-8 levels deep
> 
>   (gdb) bt
>   #0  virDomainObjUnref (dom=0x1b49390) at conf/domain_conf.c:828
>   #1  0x0000000000470f33 in qemuMonitorFree (mon=0x1cb6640) at qemu/qemu_monitor.c:202
>   #2  qemuMonitorUnref (mon=0x1cb6640) at qemu/qemu_monitor.c:221
>   #3  0x0000000000416ebf in virEventCleanupHandles () at event.c:548
>   #4  0x0000000000417c8e in virEventRunOnce () at event.c:612
>   #5  0x0000000000418593 in qemudOneLoop () at libvirtd.c:2226
>   #6  0x00000000004188b9 in qemudRunLoop (opaque=0x1b42850) at libvirtd.c:2335
>   #7  0x0000003980e07761 in start_thread (arg=0x7f6e63423700) at pthread_create.c:301
>   #8  0x00000039806e098d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> 
> Am I using the wrong functions for getting stack traces,
> or is it just not possible to get reliable stacks from
> userspace probes ?

You are doing the right things, but stap isn't smart enough to figure it
all out on its own. The simplest way to tell stap you want all symbols
and unwind tables is to say something like:

$ stap -d /usr/sbin/libvirtd --ldd [...]

> FYI, I'm using systemtap-1.3-3.fc13.x86_64 and
> kernel 2.6.34.7-66.fc13.x86_64

systemtap 1.4 (just released) should at least give you a hint what to
add if it cannot unwind further.

Cheers,

Mark