Re: [Ksummit-2008-discuss] DTrace

[Jim Keniston <jkenisto@us.ibm.com>]

On Mon, 2008-06-30 at 13:12 -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> On Mon, Jun 30, 2008 at 01:29:13PM +0200, Christoph Hellwig wrote:
> 
> > [...]  This might be getting a little offtopic for the kernel summit
> > discuss list, but let's start anyway, we can move this to a better
> > suited list, although I can't think of one except for linux-kernel.
> 
> systemtap@sources.redhat.com
> utrace-devel@redhat.com
> 
> 
> > I'm not sure if that's the current design, but I can't find any
> > evidence in the code that it allows running handlers in process
> > context, all that's available is a kernel callback.  [...]

To clarify, it's a kernel callback that runs in the context of the
probed thread -- like other utrace-based callbacks.  And like other
utrace-based callbacks, a uprobes handler can block for stuff like
copy_to/from_user()... although I believe systemtap will support only
non-blocking handlers for now.

> 
> For systemtap's purposes, that is sufficient.  Our probes are meant to
> run non-intrusively (they do not mess with user thread scheduling,
> their VM state, strictly limited time & space consumption), so
> actually injecting equivalent snippets of code into userspace for
> execution there does not seem to buy anything.  Plus, like dtrace, we
> want scripts to be able to intermix probes (=> share data) amongst
> kernel and multiple user-space threads, and this seems most naturally
> done by running the probes themselves in kernel space.

Yes.

> 
> 
> > [...] What we really need is a userspace interface so that it
> > actually can be used by thing like frysk or an implementation of the
> > userspace dtrace hooks.

Userspace dtrace hooks could be probed using systemtap-generated
uprobes, whether or not the hooks all funnel into the same user-space
handler function.

> 
> That will get built as other tools require it.  Systemtap per se will
> likely not.

Two years back, we explored the possibility of systemtap translating a
script into an ad hoc tracer app that used ptrace.  The idea was that
that would suffice in cases where the user doesn't care to see what's
going on in the kernel.  My experience was that ptrace wasn't up to the
task.  Perhaps when we nail down the right utrace-based,
ptrace-replacement system call interface (utracer II, or whatever -- see
the  current discussion on the utrace-devel list), we should revisit
that option.  It would make systemtap accessible to the ordinary
application programmer, without him/her needing root or stapdev to bless
his/her script.

Stuff that's in uprobes (e.g., kprobes-style single-stepping out of
line, to allow real-time tracing of multithreaded apps) can be made
available to the new syscall API and/or utrace.

> 
> 
> > [...] For complex traces doing this in userspace is for sure a better idea.
> 
> Can you elaborate upon this more complex scenario?
> 
> 
> - FChE

Jim Keniston