user mode backtrace

user mode backtrace

[David Boreham]

I'd like to get a stack trace for the process that made the
system call I'm probing (I'm looking at filesystem access
typically, so reads/writes/syncs etc). The systemtap backtrace
function appears to only get the kernel mode stack which
is not much use to me. I was wondering if anyone had
discovered a good solution to this problem already ?
I was thinking perhaps I could invoke pstack (gdb)
on the current pid/tid. But I'm worried that doing so
might deadlock since the process is inside a system
call.

I'm looking at a very large application that beats up on
the filesystem, in case you're wondering why I want to do
this. It's so large that nobody is quite sure what code
access which files, when and why.

Thanks.

Re: user mode backtrace

[Vara Prasad]

David Boreham wrote:

> I'd like to get a stack trace for the process that made the
> system call I'm probing (I'm looking at filesystem access
> typically, so reads/writes/syncs etc). The systemtap backtrace
> function appears to only get the kernel mode stack which
> is not much use to me. I was wondering if anyone had
> discovered a good solution to this problem already ?

If i read your above message correctly you would like to put probes in 
the kernel but you want to get a full stack which includes kernel mode 
stack and as well as user mode stack, correct.
 
Martine, as a first step, can we dump the stack pointer addresses for 
the user space stack as part of our back trace functionality when 
someone says give me full stack. I understand stack unwinding is not 
always reliable in x86 without frame pointers but we have the same 
problem with the kernel as well and we can deal with this also in the 
same fashion. I think it is safe to assume that the most interesting 
part of the user space stack will be in memory not paged out when we are 
in the kernel handler.  Are there some other complications that i am 
missing?

David, if you want to put probes in the user space itself and do back 
trace we are working on user space probing patch that will let you  see 
the stack just in the application space only. We may be able to give 
some thing for you to play within couple of weeks if you are interested 
in user space probing.

> I was thinking perhaps I could invoke pstack (gdb)
> on the current pid/tid. But I'm worried that doing so
> might deadlock since the process is inside a system
> call.
>
> I'm looking at a very large application that beats up on
> the filesystem, in case you're wondering why I want to do
> this. It's so large that nobody is quite sure what code
> access which files, when and why.


I can completely understand why it is difficult to figure out which code 
path you are taking that is one of the reasons why we are working on 
user space probes.

My next question to you is we are looking to put good load on the system 
while running probes, is this filesystem application is something that 
you can share and it is easy for us to run outside of your environment. 
If this is an application that you can not share outside can you help us 
run this application while putting lots of probes touching various parts 
of the system. We can give you scripts, if you like, that will place 
probes in many parts of the system or specifically in the areas this 
application touches the most. Please let us know.

>
> Thanks.
>
>

Re: user mode backtrace

[David Boreham]

Vara Prasad wrote:

> If i read your above message correctly you would like to put probes in 
> the kernel but you want to get a full stack which includes kernel mode 
> stack and as well as user mode stack, correct.

ack

> David, if you want to put probes in the user space itself and do back 
> trace we are working on user space probing patch that will let you  
> see the stack just in the application space only. We may be able to 
> give some thing for you to play within couple of weeks if you are 
> interested in user space probing.

I'm not sure if this would work. It depends on whether I would need to 
identify the
processes subject to probing in advance. The application spawns 
processes left and right,
many of which are sort-lived. If I could place a probe in the glibc 
layer above
the system calls that might work, but said probes would need to magically
apply to any process that called that code. Is that the feature you have 
in mind ?

But even so, I can imagine cases where it would still be useful to probe 
in the
kernel but show user stack : e.g. I probe physical read from disk 
because I am
not interested in reads served from filesystem cache. But I'd like to 
find out
where the read calls that hit non-cached data are coming from in the 
application.
Hope that makes sense.

> My next question to you is we are looking to put good load on the 
> system while running probes, is this filesystem application is 
> something that you can share and it is easy for us to run outside of 
> your environment. If this is an application that you can not share 
> outside can you help us run this application while putting lots of 
> probes touching various parts of the system. We can give you scripts, 
> if you like, that will place probes in many parts of the system or 
> specifically in the areas this application touches the most. Please 
> let us know.

The application is not supersecret. In fact in binary form it can be
downloaded from a public web site. I have been loading it with
an open source benchmark tool. So it would be possible for
someone else to reproduce my setup. To see source you'd need
to wait for the application to be open source'ed, which it will be soon
but not right now.

For now however I want to be a little cautious about the details
of what I'm working on. Let me go find out how much I am able
to share...

Re: user mode backtrace

[William Cohen]

David Boreham wrote:
> Vara Prasad wrote:
> 
>> If i read your above message correctly you would like to put probes in 
>> the kernel but you want to get a full stack which includes kernel mode 
>> stack and as well as user mode stack, correct.
> 
> 
> ack
> 
>> David, if you want to put probes in the user space itself and do back 
>> trace we are working on user space probing patch that will let you  
>> see the stack just in the application space only. We may be able to 
>> give some thing for you to play within couple of weeks if you are 
>> interested in user space probing.
> 
> 
> I'm not sure if this would work. It depends on whether I would need to 
> identify the
> processes subject to probing in advance. The application spawns 
> processes left and right,
> many of which are sort-lived. If I could place a probe in the glibc 
> layer above
> the system calls that might work, but said probes would need to magically
> apply to any process that called that code. Is that the feature you have 
> in mind ?
> 
> But even so, I can imagine cases where it would still be useful to probe 
> in the
> kernel but show user stack : e.g. I probe physical read from disk 
> because I am
> not interested in reads served from filesystem cache. But I'd like to 
> find out
> where the read calls that hit non-cached data are coming from in the 
> application.
> Hope that makes sense.

The disk I/O schedules the reads and writes. It is unlikely that the matching 
user process is running when the physical read occurs. Would it be useful to 
look at the elapsed time between entry and exit time of the read system call and 
trigger recording of information if it is over some threshold?

Would it make sense for the probing code to mark that userstack is needed and 
then record when the processor is about to return to userspace? The marking 
takes a fixed amount of time to do, so should be safe for probes.

-Will

Re: user mode backtrace

[David Boreham]

> The disk I/O schedules the reads and writes. It is unlikely that the 
> matching user process is running when the physical read occurs. 

Can I clarify the problem here ? : the user process isn't running 
because it's been put to sleep waiting on I/O that is done
in the context of another thread (or whatever we call that animal in the 
Linux kernel) , or is this when read-ahead is being
done on the file, at the behest of the kernel, with no corresponding 
user process read() call having been issued yet ?
(apologies my knowledge of present-day linux kernel details is rather 
patchy).

> Would it be useful to look at the elapsed time between entry and exit 
> time of the read system call and trigger recording of information if 
> it is over some threshold?

Possibly. I wouldn't want to get too hung up on this specific example -- 
I thought it up off the top of my head.
The point I was trying to make is that probing in userland in glibc may 
not do everything that's required because sometimes
one is interested in events that are only visible inside the kernel. 
Physical I/O was just the first one that I could
think of. I suppose a kernel mutex being busy might be another case.

> Would it make sense for the probing code to mark that userstack is 
> needed and then record when the processor is about to return to 
> userspace? The marking takes a fixed amount of time to do, so should 
> be safe for probes.

Yes, that sounds like a good way to do it.

Re: user mode backtrace

[Frank Ch. Eigler]

David Boreham <david_list@boreham.org> writes:

> I'd like to get a stack trace for the process that made the system
> call I'm probing [...]

Yes, that's an understandable need.

> But I'm worried that [calling gdb] might deadlock since the process
> is inside a system call.

Indeed, that would be several orders of magnitude too slow and invasive.

We don't yet have a good answer to user-space stuff in general.  We
may adopt a mixture of two approaches: one where the compiled
systemtap probes get some extract of backtrace-enabling unwind
information for selected user-space programs and libraries (so the
backtrace processing can be done instantly right there); and another
one where the probe may snapshot only an approximation (say a vector
of probable PCs from the thread's stack frame), and rely on a
user-space helper to correct/complete it for output later, off-line.

- FChE

Re: user mode backtrace

[David Boreham]

>
>
>We don't yet have a good answer to user-space stuff in general.  We
>may adopt a mixture of two approaches: one where the compiled
>systemtap probes get some extract of backtrace-enabling unwind
>information for selected user-space programs and libraries (so the
>backtrace processing can be done instantly right there); and another
>one where the probe may snapshot only an approximation (say a vector
>of probable PCs from the thread's stack frame), and rely on a
>user-space helper to correct/complete it for output later, off-line.
>  
>
Given the choice I'd prefer the latter because it'd be less intrusive.

Re: user mode backtrace

[Frank Ch. Eigler]

Hi -

On Thu, Oct 19, 2006 at 06:15:33PM -0600, David Boreham wrote:

> >[...] one where the compiled systemtap probes get some extract of
> >backtrace-enabling unwind information [...]; and another one where
> >the probe may snapshot only an approximation [...]  and rely on a
> >user-space helper to correct/complete it [...]

> Given the choice I'd prefer the latter because it'd be less intrusive.

Actually, intrusiveness is probably not the real trade-off point.
Both approaches would require scanning the user-space stack.  The
former (the probe knowing unwind info) might even be lighter, in that
it would know which portions of the stack need to be sampled.  The
latter may need to be conservative and transcribe a larger number of
words.  With respect to speed, unwinding vs. conservative sampling
should not be too different.

The trade-offs are probably closer to issues such as expressiveness,
module size, complexity of kernel/user interaction.

- FChE

RE: user mode backtrace

[Stone, Joshua I]

On Thursday, October 19, 2006 1:47 PM, David Boreham wrote:
> I'd like to get a stack trace for the process that made the
> system call I'm probing (I'm looking at filesystem access
> typically, so reads/writes/syncs etc). The systemtap backtrace
> function appears to only get the kernel mode stack which
> is not much use to me. I was wondering if anyone had
> discovered a good solution to this problem already ?
> I was thinking perhaps I could invoke pstack (gdb)
> on the current pid/tid. But I'm worried that doing so
> might deadlock since the process is inside a system
> call.
> 
> I'm looking at a very large application that beats up on
> the filesystem, in case you're wondering why I want to do
> this. It's so large that nobody is quite sure what code
> access which files, when and why.
> 
> Thanks.

Deadlock issues aside, there's not really a way for you to invoke a
process (like pstack) from within a SystemTap script.  You could run a
separate user program or script to do this for you though, and then you
just need to coordinate with SystemTap.  Such a method might look like
this:

-----------------------------------------------
/* Main test driver */
-----------------------------------------------
pid = fork_ptraceme_exec("myapp"); // start the app paused
stappid = fork_exec("stap myscript.stp -x " + pid); // start systemtap
ptrace(DETACH, pid, ...); // let the app run
while(pid == waitpid(pid, stat, 0)) {
  if (WIFSTOPPED(stat)) { // app is stopped
    system("pstack " + pid); // dump the stack
    kill(pid, SIGCONT); // continue the app
  }
  else if (WIFEXITED(stat)) {
    break;
  }
}
kill(stappid, SIGINT); // tell systemtap to stop
waitpid(stappid, ...);
-----------------------------------------------

-----------------------------------------------
/* SystemTap script: myscript.stp */
-----------------------------------------------
probe syscall.read {
  if (target() != tid()) next;
  /* log some stuff: filename, etc. */
}
probe syscall.read.return {
  if (target() != tid()) next;
  send_stop() // do this on return to avoid EINTR
}
function send_stop %{
  send_sig(SIGSTOP, current, 1);
%}
-----------------------------------------------

This is all pretty rough, and I haven't actually tried it, so who knows
if it will actually work.

Of course at the end of the day, this is just a convoluted strace with a
stack printout.  You could probably do the same thing by hacking gdb's
backtrace function into strace.  But this SystemTap method would also
let you do probe other things besides just system calls...

If anyone gets this working I would LOVE to hear about it... :)


Josh

Re: user mode backtrace

[David Boreham]

Stone, Joshua I wrote:

>On Thursday, October 19, 2006 1:47 PM, David Boreham wrote:
>  
>
>>I'd like to get a stack trace for the process that made the
>>system call I'm probing (I'm looking at filesystem access
>>typically, so reads/writes/syncs etc). The systemtap backtrace
>>function appears to only get the kernel mode stack which
>>is not much use to me. I was wondering if anyone had
>>discovered a good solution to this problem already ?
>>I was thinking perhaps I could invoke pstack (gdb)
>>on the current pid/tid. But I'm worried that doing so
>>might deadlock since the process is inside a system
>>call.
>>
>>I'm looking at a very large application that beats up on
>>the filesystem, in case you're wondering why I want to do
>>this. It's so large that nobody is quite sure what code
>>access which files, when and why.
>>
>>Thanks.
>>    
>>
>
>Deadlock issues aside, there's not really a way for you to invoke a
>process (like pstack) from within a SystemTap script.  You could run a
>separate user program or script to do this for you though, and then you
>just need to coordinate with SystemTap.  Such a method might look like
>this:
>  
>
Yeah, that was the kind of hack I was thinking of.

>Of course at the end of the day, this is just a convoluted strace with a
>stack printout.  
>
Not really. The application that I'm battling with is resistent to my 
'traditional'
analysis techniques, which would include strace, gdb/pstack, oprofile etc.
The big show stopper is that it isn't one big process that you can look at.

It fires up numerous short lived processes. There isn't even one master
process that exec's the subprocesses: some of them kick in on timers,
watching the modification dates on files, and via local pipes.
Hence the system-wide approach, hooking kernel activity and then looking
upward to figure out what code in the application, in which process, is 
burning
I/O resources.

Re: user mode backtrace

[David Boreham]

>pid = fork_ptraceme_exec("myapp"); // start the app paused
>stappid = fork_exec("stap myscript.stp -x " + pid); // start systemtap
>ptrace(DETACH, pid, ...); // let the app run
>  
>
Actually I don't think this will help me because it looks like
it assumes a specific target process. That's the specific problem that
I have : I don't know which processes are going to be interesting
in advance.

Re: user mode backtrace

[Frank Ch. Eigler]

"Stone, Joshua I" <joshua.i.stone@intel.com> writes:

> [...]  Deadlock issues aside, there's not really a way for you to
> invoke a process (like pstack) from within a SystemTap script.  [...]

It turns out that quite some time ago, Martin implemented a clone of
the dtrace system() function for systemtap, which enqueues a string
for execution by the userspace staprun daemon.  (There is no
synchronization or data exchange though.)

- FChE

RE: user mode backtrace

[Stone, Joshua I]

On Thursday, October 19, 2006 4:25 PM, David Boreham wrote:
>> pid = fork_ptraceme_exec("myapp"); // start the app paused
>> stappid = fork_exec("stap myscript.stp -x " + pid); // start
>> systemtap ptrace(DETACH, pid, ...); // let the app run
>> 
>> 
> Actually I don't think this will help me because it looks like
> it assumes a specific target process. That's the specific problem that
> I have : I don't know which processes are going to be interesting
> in advance.

I just filter on a single tid because it's convenient.  The thing you
have to avoid is probing any of the processes you kick off, like the
pstack.  Otherwise you get yourself in a recursive loop, and
congratulations, you've just fork-bombed the system.  So it's hard to be
smart about which processes NOT to probe.  You could try filtering by
execname, if that's known.

If you can manage that your application is spawned from a central
process, you could try to follow forks from that process:

-----------------------------------------------
global filter
probe begin {
  filter[target()] = 1
}
probe process.create {
  if (filter[tid()])
    filter[new_pid] = 1
}
probe process.exit {
  delete filter[tid()]
}
-----------------------------------------------

Then instead of "if (target() != tid()) next;" you have "if
(!filter[tid()]) next;".


Josh

RE: user mode backtrace

[Stone, Joshua I]

On Thursday, October 19, 2006 5:10 PM, Frank Ch. Eigler wrote:
> "Stone, Joshua I" <joshua.i.stone@intel.com> writes:
> 
>> [...]  Deadlock issues aside, there's not really a way for you to
>> invoke a process (like pstack) from within a SystemTap script.  [...]
> 
> It turns out that quite some time ago, Martin implemented a clone of
> the dtrace system() function for systemtap, which enqueues a string
> for execution by the userspace staprun daemon.  (There is no
> synchronization or data exchange though.)

I forgot we had that.  Take two... :)

-----------------------------------------------
global args
probe syscall.open {
    t = tid()
    if (target() != t) next;
    args[t] = argstr
}
probe syscall.open.return {
    t = tid()
    if (target() != t) next;
    send_stop() // do this on return to avoid EINTR
    printf("\n%d %s open(%s) = %s\n", t, execname(), args[t], retstr)
    system(sprintf("pstack %d && kill -CONT %d", t, t))
    delete args[t]
}
function send_stop() %{
    send_sig(SIGSTOP, current, 1);
%}
-----------------------------------------------

... and this one actually works!  Mostly... when I SIGSTOP an app
started from an interactive shell, the shell seems to take back control,
and I can't get SIGCONT to work nicely.  But, I tried targeting a gvim
process, which is detached, and it worked just fine!

Of course, it's VERY slow -- probably orders of magnitude slower than
other options like recording the stack frame for post-processing.

It's still a fun exercise though.  :)


Josh

RE: user mode backtrace

[Stone, Joshua I]

On Thursday, October 19, 2006 7:13 PM, Stone, Joshua I wrote:
> -----------------------------------------------
> global args
> probe syscall.open {
>     t = tid()
>     if (target() != t) next;
>     args[t] = argstr
> }
> probe syscall.open.return {
>     t = tid()
>     if (target() != t) next;
>     send_stop() // do this on return to avoid EINTR
>     printf("\n%d %s open(%s) = %s\n", t, execname(), args[t], retstr)
>     system(sprintf("pstack %d && kill -CONT %d", t, t))
>     delete args[t]
> }
> function send_stop() %{
>     send_sig(SIGSTOP, current, 1);
> %}
> -----------------------------------------------
> 
> ... and this one actually works!  Mostly... when I SIGSTOP an app
> started from an interactive shell, the shell seems to take back
> control, and I can't get SIGCONT to work nicely.  But, I tried
> targeting a gvim process, which is detached, and it worked just fine!
> 
> Of course, it's VERY slow -- probably orders of magnitude slower than
> other options like recording the stack frame for post-processing.
> 
> It's still a fun exercise though.  :)

Bonus points -

While this method is really too slow for tracing usage, it might be good
for debugging purposes.  I'm thinking of the usage model where you
detect that something interesting happens in your app, and you want to
pause it so you can attach a debugger and inspect it interactively.

For example, consider the simple case when you're debugging an app that
forks, and you want to have gdb attached to *both* ends of the fork.
The gdb docs guide you to "Put a call to sleep in the code which the
child process executes after the fork."  With SystemTap probes on
process.create and/or process.start, you could watch for your app's
fork, SIGSTOP the new process before it goes anywhere, and then attach a
gdb to the new process.  You still need multiple gdb sessions, but this
way you don't need to modify your app with a new sleep call.

If you want to be extra clever, you could even use system() to
automatically launch an xterm with a new gdb session attached, something
like:

  probe process.start {
    if (my_filter()) {
      send_stop()
      system(sprintf("xterm -e gdb %d &", tid()))
    }
  }


Josh