Java VM support in SystemTap

Java VM support in SystemTap

[Danilov, Oleg V]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

Hi all,

Inspired by the SUN DTrace ability to provide insight information about running java applications we think it makes sense to add this ability to the SystemTap as well. Basically we need a framework which allows investigation of the behavior of underlying operating system combined with the insight information about running Java applications. This can assist in identifying the underlying cause of a performance or functional problem. SystemTap simplify gathering of information about Linux kernel and it is possible to add support of the user applications. An attached document describes our vision on how Java VM support could be integrated into the SystemTap.

Your comments are welcome.

Thanks,
  Oleg


[-- Attachment #2: SystemTap and Java.pdf --]
[-- Type: application/octet-stream, Size: 114590 bytes --]

Re: Java VM support in SystemTap

[Frank Ch. Eigler]

"Danilov, Oleg V" <oleg.v.danilov@intel.com> writes:

> Inspired by the SUN DTrace ability to provide insight information
> about running java applications we think it makes sense to add this
> ability to the SystemTap as well. [...]

It's an interesting bunch of ideas.  It's impressive that you worked
stuff out all the way through possible message formats and probe point
syntax, which is useful analysis regardless of ultimate
implementation.

The key trick is the notion of using stpd as an intermediary between
user-space event sources (in this case, JVM instrumentation hooks
talking to a specialized systemtap-bundled supervisor program) and
normal kernel-space probe handlers.

This is interesting because it suggests a mechanism for general
user-space probing, where instead of using JVM[TPD]I APIs, a specially
built user-space supervisor program could use ptrace or utrace or
whatever to manage a non-JVM program.  It could relay this occurrence
to kernel-side probe handlers via stpd.

A consequence of this type of approach is to suffer multiple context
switches per probe hit.  Further, if these supervisor programs are
general (not parametrized for the particular probe script), then the
quantity of contextual data would be strictly limited.  That's
because, for several reasons, all plausible $target data would need to
be collected at probe-hit time, and packaged up in the first packet
toward stpd to relay to kernel-space.  (Explicit request/reply type
interaction from the kernel probe handlers is not going to work.)

- FChE

RE: Java VM support in SystemTap

[Scherbakov, Sergey V]

Frank,

I'm coauthor of that proposal. First of all, thanks for the quick
evaluation. Please see my question/comments inlined.


-----Original Message-----
From: fche@redhat.com [mailto:fche@redhat.com] 


> The key trick is the notion of using stpd as an intermediary between
> user-space event sources (in this case, JVM instrumentation hooks
> talking to a specialized systemtap-bundled supervisor program) and
> normal kernel-space probe handlers.

Is it means that we really can use stpd as gate between VM and
kernel-space probe handlers? I'm asking because our ideas based not on
deep knowledge of SystemTap but mostly on common sense. 

> This is interesting because it suggests a mechanism for general
> user-space probing, where instead of using JVM[TPD]I APIs, a specially
> built user-space supervisor program could use ptrace or utrace or
> whatever to manage a non-JVM program.  It could relay this occurrence
> to kernel-side probe handlers via stpd.

If you think, it is interesting not only for using with Java VM may be
it is good idea to extend/generalize that communication interface?

> A consequence of this type of approach is to suffer multiple context
> switches per probe hit.  Further, if these supervisor programs are
> general (not parametrized for the particular probe script), then the
> quantity of contextual data would be strictly limited.  That's
> because, for several reasons, all plausible $target data would need to
> be collected at probe-hit time, and packaged up in the first packet
> toward stpd to relay to kernel-space.  (Explicit request/reply type
> interaction from the kernel probe handlers is not going to work.)

Yes, agree. It is an issue. It is why in case of VM we propose to use
special command 'RequestOptionalData' to describe expected data and
avoid unnecessary context switches.

Thanks
  Sergey

Re: Java VM support in SystemTap

[Frank Ch. Eigler]

sergey.v.scherbakov wrote:

> [...]
> > The key trick is the notion of using stpd as an intermediary between
> > user-space event sources (in this case, JVM instrumentation hooks
> > talking to a specialized systemtap-bundled supervisor program) and
> > normal kernel-space probe handlers.
> 
> Is it means that we really can use stpd as gate between VM and
> kernel-space probe handlers? [...]

It sounds like a fit, at least in theory.  There are probably some
other possibilities also (see below).

> > [...]  Further, if these supervisor programs are general (not
> > parametrized for the particular probe script), then the quantity
> > of contextual data would be strictly limited.  [...]
>
> Yes, agree. It is an issue. It is why in case of VM we propose to use
> special command 'RequestOptionalData' to describe expected data and
> avoid unnecessary context switches.

My point is that if the "optional data" is a function of the probe
script (say, the list of actual $obj->obj2->field expressions used),
and if this list is not available ahead of time to the user-side
intermediary programs, then the kernel-side probes won't be able to
get at them at all.  The kernel-side probe handlers cannot practically
query them during hit time.


This concern leads me to speculate about another possible
implementation technique, one that does not use stpd nor a generic
unparametrized JVM interface process.  That is to have the translator
map JVM probes to a custom user-space program that includes
- the JVMDI etc. logic needed to hook up identified JVMs
- the probe handlers proper
- an alternative run-time library that can operate in user-space
If in fact such a JVMDI client program can be dynamically loaded into
the JVM rather than run as an external process, so much the better.

In order to enable a single systemtap script to probe both kernel and
JVM space in this model, they would have to be able to *share memory*
such as
- systemtap script-level global variables
- and their locks!
- supervisory state such as error_count
Of course this would have to be done in a secure and safe manner, which
would require the run-time code to be more paranoid.

Perhaps the execution model is obvious from the above and is similar
to plan B: <http://sourceware.org/ml/systemtap/2006-q2/msg00570.html>.
In particular, it may offer the following advantages over the
generic-intermediary + stpd + kernel-space probe handlers proposal:
- faster performance because of absence of context switching
- full context information available
- simpler in one way (inter-process communication), though
  more complex in another (possible need to share live kernel
  systemtap data)

- FChE

RE: Java VM support in SystemTap

[Scherbakov, Sergey V]

Frank,

We took one-day timeout to read about 'plan B' and discuss your really
useful ideas/comments between us. 

Actually, in beginning of our researches we were thinking about similar
model. We are planned to set a breakpoint to some stab inside JVMTI/PI
agent and then use kernel probe scenario. Unfortunately in this case we
have a lot of almost undecidable issues. 

I guess the main issue that in case of kernel/user probe we know
address, we can unwind stack etc., on other hand (JVM) we have only
event of some class (MethodEnter for instance). We even cannot set a
breakpoint in JIT-ed function (we know its address), because it is
possible to have more that one JIT-ed variant, but VM can choose to use
interpreter. Sure we can generate C/C++ stub corresponding every method
of interest, but it is really difficult (almost impossible in case of
wildcards usage).  

Frank Ch. Eigler wrote:

> My point is that if the "optional data" is a function of the probe
> script (say, the list of actual $obj->obj2->field expressions used),
> and if this list is not available ahead of time to the user-side
> intermediary programs, then the kernel-side probes won't be able to
> get at them at all.  The kernel-side probe handlers cannot practically
> query them during hit time.

The following is a quotation from our internal discussion:

--  The optional data needed [$obj->obj2->field] is defined at script
start --  time.  This could in theory be passed thru stpd to the VM
agent at
--   configuration time.  [i.e., the "RequestOptionalData" could all be
set --  up ahead of time...]
--  e.g.
--      p=RegisterProbe("java/lang/...",enter);
--      RequestOptionalData(p, PARAMETER1,...);
--   
--  Then when the event occurs, the VM agent bundles the event with the 
--  registeredOptionalData and sends it as a packet to stpd who forwards
to --  kernel.  The kernel component won't need to request additional
stuff at --  run time.

Thanks,
   Sergey