next steps

next steps

[Frank Ch. Eigler]

Hi -

I see several broad directions for our project for the next few
months.  In no particular order:


- Improving kprobes:
  - djprobes
  - rcu kprobes
  - user-space probes
  - porting to any other interesting platforms

- Other event sources:
  - remaining dwarf probe point types (.callees etc.)
  - static instrumentation in kernel and user space
  - performance counters

- Tapset scripts:
  - VM
  - I/O
  - networking
  - glibc threading / locking
  - language interpreter events

- Translator:
  - statistics support
  - optimizations

- System testing:
  - over a range of probe sizes, complexity
  - over a range of machine sizes, complexity
  - over a range of kernel/compiler versions
  - static verification?

- Propaganda:
  - tutorial level documentation
  - worked out problem-solving examples
  - scripts that substitute for / combine existing procps type tools
  - script language customizations for emacs/vi


It is not necessary for each of us to continue within the same area
we've been working so far.  I'd like to hear whether people or groups
are interested in taking responsibility for particular line items, or
if they wish to suggest amendments.


- FChE

Re: next steps

[Richard J Moore]

Frank, under improving kprobes I would add:

1) dprobes or a scalable mechanism for performance tracing, but not one
which throws away what we have unless it truly supersedes it
2) watchpoint probes if these aren't already there. This is the capability
to place a probe on a data location. Internally it uses the debug
registers. If this isn't there we do have the patch since this was part of
the original dprobes.
3) add back the ability to place probes in a code location before the
module is loaded. This capability comes automatically with the user-space
probes since probes are tracked by (inode, offset) and fixed up via
readpage() then the code is paged in.  We used to have this capability for
kernel modules. It relied on patches in insmod. Clearly this is useful
especially when wanting to capture initialization module problems.
4) Finally I'd request that we re-instate the SysRq function to disable all
probes instantly. This again was part of the original dprobes and proved
its worth when intensive tracing locked out the command line.

Apart from 1) all of these are simple modification for which the code has
all ready been written.

- -
Richard J Moore
IBM Advanced Linux Response Team - Linux Technology Centre
MOBEX: 264807; Mobile (+44) (0)7739-875237
Office: (+44) (0)1962-817072


                                                                           
             "Frank Ch.                                                    
             Eigler"                                                       
             <fche@redhat.co                                            To 
             m>                        systemtap@sources.redhat.com        
             Sent by:                                                   cc 
             systemtap-owner                                               
             @sources.redhat                                           bcc 
             .com                                                          
                                                                   Subject 
                                       next steps                          
             20/09/2005                                                    
             16:50                                                         
                                                                           




Hi -

I see several broad directions for our project for the next few
months.  In no particular order:


- Improving kprobes:
  - djprobes
  - rcu kprobes
  - user-space probes
  - porting to any other interesting platforms

- Other event sources:
  - remaining dwarf probe point types (.callees etc.)
  - static instrumentation in kernel and user space
  - performance counters

- Tapset scripts:
  - VM
  - I/O
  - networking
  - glibc threading / locking
  - language interpreter events

- Translator:
  - statistics support
  - optimizations

- System testing:
  - over a range of probe sizes, complexity
  - over a range of machine sizes, complexity
  - over a range of kernel/compiler versions
  - static verification?

- Propaganda:
  - tutorial level documentation
  - worked out problem-solving examples
  - scripts that substitute for / combine existing procps type tools
  - script language customizations for emacs/vi


It is not necessary for each of us to continue within the same area
we've been working so far.  I'd like to hear whether people or groups
are interested in taking responsibility for particular line items, or
if they wish to suggest amendments.


- FChE

Re: next steps

[Frank Ch. Eigler]

Hi -

> Frank, under improving kprobes I would add:

Thanks for the suggestions.

> 1) dprobes or a scalable mechanism for performance tracing [...]

Could you elaborate on what exactly "scalable ... tracing" refers to,
and how you believe we're short of this?

> 2) watchpoint probes if these aren't already there. [...]

This is already tracked by bug #1324.  A prerequisite is a debug
register management API in the kernel.

> 3) add back the ability to place probes in a code location before
> the module is loaded. [...]  We used to have this capability for
> kernel modules. It relied on patches in insmod. Clearly this is
> useful especially when wanting to capture initialization module
> problems.

Sounds useful, especially if it can be accomplished without much or
any patching of generic kernel or kernel-utils code.  Might the
register_module_notifier() hook be sufficient for this?  It's worth
creating a bugzilla item for this.  Bug #1145 is a prerequisite.

> 4) Finally I'd request that we re-instate the SysRq function to
> disable all probes instantly. [...]

Please create a bugzilla item.  __sysrq_put_key_op is probably
suitable, as long as a handler doesn't try to do much anything fancy.
Is it safe to instantly remove all kprobes at an arbitrary moment?

> Apart from 1) all of these are simple modification for which the
> code has all ready been written.

These are good, though was intending to list the *larger* missing
chunks: leaps of usefulness that would excite ordinary developers, not
just kernel hackers.


- FChE

Re: next steps

[Martin Hunt]

On Tue, 2005-09-20 at 11:50 -0400, Frank Ch. Eigler wrote:

> - Translator:
>   - statistics support
>   - optimizations

The issues I see so far that I would like fixed include:

If we can't make the language typeless, can we at least add some type of
polymorphism?  The common case I would like to solve is when writing a
probe to handle probepoints and I would like to print out an arg which
may be an int or a string. Maybe just adding typeof() would be enough.

Support for variable args, mostly just for print() so it can take
printf-like formatting. This is needed for printa() too.

Currently the translator writes a large amount of C code for every
probe.  This is unscalable and the code is too slow for tracing or
profiling. Probably we need a different type of specialized probe,
consisting of a djprobe that prints out some standard info.  That same
probe would be inserted everywhere, instead of compiling tens of
thousands of identical probes.

Martin

Re: next steps

[Richard J Moore]

"Frank Ch. Eigler" <fche@redhat.com> wrote on 21/09/2005 21:56:10:

> Hi -
>
> > Frank, under improving kprobes I would add:
>
> Thanks for the suggestions.
>
> > 1) dprobes or a scalable mechanism for performance tracing [...]
>
> Could you elaborate on what exactly "scalable ... tracing" refers to,
> and how you believe we're short of this?
>

Frank, this is a reference to earlier discussion relating to the
inefficiency of the interrupt mechanism.
On the one hand the int3 mechanism allows use to probes in almost all
contexts under almost any condition. But it is relatively slow.
For debugging purposes it's usually good enough.

However, for performance measurements, once need a mechanism that perturbs
the system timing as little as possible - even if some events are lost.
I believe djprobes attempts to fulfil this need, but if my understanding is
correct there are some unfortunate restrictions to its deployment. So, I'm
lodging a request that this issue be further discussed at the very least to
determine whether there is a real need to address it, which I believe
there. For example using the current mechanism to trace interrupts from a
Gb ethernet NIC is not going to work particularly well.

> > 2) watchpoint probes if these aren't already there. [...]
>
> This is already tracked by bug #1324.  A prerequisite is a debug
> register management API in the kernel.
>
> > 3) add back the ability to place probes in a code location before
> > the module is loaded. [...]  We used to have this capability for
> > kernel modules. It relied on patches in insmod. Clearly this is
> > useful especially when wanting to capture initialization module
> > problems.
>
> Sounds useful, especially if it can be accomplished without much or
> any patching of generic kernel or kernel-utils code.  Might the
> register_module_notifier() hook be sufficient for this?  It's worth
> creating a bugzilla item for this.  Bug #1145 is a prerequisite.
>
> > 4) Finally I'd request that we re-instate the SysRq function to
> > disable all probes instantly. [...]
>
> Please create a bugzilla item.  __sysrq_put_key_op is probably
> suitable, as long as a handler doesn't try to do much anything fancy.
> Is it safe to instantly remove all kprobes at an arbitrary moment?
>
> > Apart from 1) all of these are simple modification for which the
> > code has all ready been written.
>
> These are good, though was intending to list the *larger* missing
> chunks: leaps of usefulness that would excite ordinary developers, not
> just kernel hackers.
>
>
> - FChE

Re: next steps

[Frank Ch. Eigler]

Hi -

hunt wrote:

> The issues I see so far that I would like fixed include:
> If we can't make the language typeless, can we at least add some type of
> polymorphism?  [...]
> Support for variable args, mostly just for print() so it can take
> printf-like formatting. This is needed for printa() too.

These are worth considering, though do you think they would constitute
a big step forward in usability?

> Currently the translator writes a large amount of C code for every
> probe.  [...]

This is already being worked on under bug #1330.

- FChE

Re: next steps

[Jose R. Santos]

Martin Hunt <hunt@redhat.com> [050921]:
> Currently the translator writes a large amount of C code for every
> probe.  This is unscalable and the code is too slow for tracing or
> profiling. Probably we need a different type of specialized probe,
> consisting of a djprobe that prints out some standard info.  That same
> probe would be inserted everywhere, instead of compiling tens of
> thousands of identical probes.

Is the creation of a trace probe something thats being worked on?  My
teams is currently in the process of creating a trace tool using
SystemTap and one of the requirements that the tool need is that for
every probe inserted, the cpu, pid, tid and a time-stamp (jiffies?) be
returned as well as some additional information that would be probe
specific.  While it is possible to grab all these using SystemTap, I
wonder if it would make more sense to have a special trace probe that
does all this and then have SystemTap exploit this.

Any thoughts?

-JRS

Re: next steps

[Frank Ch. Eigler]

Hi -

On Mon, Sep 26, 2005 at 10:11:01AM -0500, Jose R. Santos wrote:

> [...]  one of the requirements that the tool need is that for every
> probe inserted, the cpu, pid, tid and a time-stamp (jiffies?) be
> returned as well as some additional information that would be probe
> specific.

function ln /* labeled number */ (name, value) {
  print (name . "=" . string(value) . " ")
}
function trace_common_stuff () {
  ln ("cpu", cpu()) # cpu() to go into context tapset
  ln ("pid", pid()) 
  ln ("jiffies", jiffies()))  # jiffies() to go into in timestamp tapset
}
probe THIS, THAT { 
  trace_common_stuff ()
  print ("probe specific\n")
}
probe OTHER {
  trace_common_stuff ()
  print ("other probe specific\n")
}

> While it is possible to grab all these using SystemTap, I wonder if
> it would make more sense to have a special trace probe that does all
> this and then have SystemTap exploit this. [...]

Perhaps the confusion is one of terminology.  A "trace probe" is not a
thing with an obvious meaning.  Using plain systemtap script pieces
like the one above can likely get you what you need.


- FChE

Re: next steps

[Jose R. Santos]

Frank Ch. Eigler <fche@redhat.com> [050926]:
> Hi -
> 
> On Mon, Sep 26, 2005 at 10:11:01AM -0500, Jose R. Santos wrote:
> 
> > [...]  one of the requirements that the tool need is that for every
> > probe inserted, the cpu, pid, tid and a time-stamp (jiffies?) be
> > returned as well as some additional information that would be probe
> > specific.
> 
> function ln /* labeled number */ (name, value) {
>   print (name . "=" . string(value) . " ")
> }
> function trace_common_stuff () {
>   ln ("cpu", cpu()) # cpu() to go into context tapset
>   ln ("pid", pid()) 
>   ln ("jiffies", jiffies()))  # jiffies() to go into in timestamp tapset
> }
> probe THIS, THAT { 
>   trace_common_stuff ()
>   print ("probe specific\n")
> }
> probe OTHER {
>   trace_common_stuff ()
>   print ("other probe specific\n")
> }
> 

Nice. Thanks for the example.

> > While it is possible to grab all these using SystemTap, I wonder if
> > it would make more sense to have a special trace probe that does all
> > this and then have SystemTap exploit this. [...]
> 
> Perhaps the confusion is one of terminology.  A "trace probe" is not a
> thing with an obvious meaning.  Using plain systemtap script pieces
> like the one above can likely get you what you need.
> 
> 
> - FChE

My question really was one of simplicity and performance.  I've looked
at some of the SystemTap generated code and the size of the code
generated is something that could be of concern if I need to have a
system with 20+ different trace events being captured.  What I'm
interested in is if the overhead of using SystemTap to gather this
information will be equal whether you use a script or a special
kprobe.  Overhead is important for the people folks we are currently
designing this tool for.  The previous poster sort of implied that
there were some performance issues with using SystemTap for tracing
things like this.

We will do some performance overhead testing but I would also like to
know if anybody has done this type of measurements already.

Thanks

-JRS

Re: next steps

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 1081 bytes --]

Hi -

On Mon, Sep 26, 2005 at 11:26:12AM -0500, Jose R. Santos wrote:

> [...]  My question really was one of simplicity and performance.
> I've looked at some of the SystemTap generated code and the size [is
> of concern] [...] What I'm interested in is if the overhead of using
> SystemTap to gather this information will be equal whether you use a
> script or a special kprobe.  [...]

I am certain that we suffer extra overhead over a hand-written C
kprobe.  After all, we include a bunch of somewhat naive checking and
debugging-enabling code.  However, looking at the relevant assembly
code, it's not *that* bad, and could be tightened in the future.

> The previous poster sort of implied that there were some performance
> issues with using SystemTap for tracing things like this.  We will
> do some performance overhead testing but I would also like to know
> if anybody has done this type of measurements already.

The system is changing rapidly enough, and we're all speculating
enough, that only individual measurements can meaningfully answer such
questions.


- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

thoughts

[Mathieu Desnoyers]

Jose R. Santos <jrsantos@austin.ibm.com> :

> Is the creation of a trace probe something thats being worked on?  My
> teams is currently in the process of creating a trace tool using
> SystemTap and one of the requirements that the tool need is that for
> every probe inserted, the cpu, pid, tid and a time-stamp (jiffies?) be
> returned as well as some additional information that would be probe
> specific.  While it is possible to grab all these using SystemTap, I
> wonder if it would make more sense to have a special trace probe that
> does all this and then have SystemTap exploit this.
> 
> Any thoughts?

We are working on a kernel tracing tool that seems to share a lot with the
trace tool you are talking about.

We have a kernel tracer (LTTng) and a trace viewer (LTTV) which are based on
static kernel instrumentation.

One of the main advantages of the LTTng tracing code is that it is fast. It
seems to be a downside of SystemTAP. We use a code generator that takes an event
description to generate the tracing code that can be inserted through module
loading.

I think that you might be interested to have a look at it. Here is the website :
http://ltt.polymtl.ca


Mathieu


Mahtieu Desnoyers
Computer Engineering Master Student
École Polytechnique de Montréal


OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 

Re: thoughts [re LTT]

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

Hi -

On Mon, Sep 26, 2005 at 01:44:42PM -0400, Mathieu Desnoyers wrote:

> [...]  We have a kernel tracer (LTTng) and a trace viewer (LTTV)
> which are based on static kernel instrumentation.  One of the main
> advantages of the LTTng tracing code is that it is fast. It seems to
> be a downside of SystemTAP.

Static instrumentation of course allows one to avoid the breakpoint
fault of kprobes.  At some point, we too will support static
instrumentation, with our own variant and/or attaching to
LTT/LKST/etc. hooks.

> We use a code generator that takes an event description to generate
> the tracing code that can be inserted through module loading.

This is quite clever, but addresses a different clientele.  LTT-type
techniques may be sufficient iff:

 - one may patch one's kernel with the tracing infrastructure
 - the set of trace event points are all hard-coded
 - the data made available at those points are all hard-coded
 - the computation performed upon those data/events is limited to
   little besides filtering / printing

I suspect we can get the best of both worlds over time.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: thoughts [re LTT]

[Mathieu Desnoyers]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

I agree that both projects are made for different purposes.

* Frank Ch. Eigler (fche@redhat.com) wrote:
>  - the computation performed upon those data/events is limited to
>    little besides filtering / printing
> 

The LTTV project proves this argument false. We do have graphical visualisation
of the scheduler's work, listing of events, filtering, all in a modular
pluggable architecture, so people can add their own "views" of the data
(eventually add event rate view, per cpu kernel state view, memory allocator
view, block subsystem view, and so on)


> I suspect we can get the best of both worlds over time.
>

I hope so. It would be interesting to see where the projects could use each
other parts so the work is not duplicated.


Mathieu

OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: thoughts [re LTT]

[Karim Yaghmour]

Frank Ch. Eigler wrote:
> This is quite clever, but addresses a different clientele.  LTT-type
> techniques may be sufficient iff:
> 
>  - one may patch one's kernel with the tracing infrastructure
>  - the set of trace event points are all hard-coded
>  - the data made available at those points are all hard-coded
>  - the computation performed upon those data/events is limited to
>    little besides filtering / printing

I think this misses the point. Mathieu's lttv is mechanism-agnostic.
In fact, it can read any format as long as you provide it with the
XML description of it. That's why creating a new tracing tool
just doesn't make any sense.

In as far as static instrumentation is concerned, it's as old as
programming, every body does it. It's nice to be able to add stuff
dynamically, but a quick browse around the kernel sources should
rapidely reveal how widespread static instrumentation already is.

In general, some kernel developers had shown resistence to static
trace points, but it's beyond me that some people in the tools
development crowd are still stuck on this issue while many kernel
developers (including Andrew) themselves now acknowledge that trace
points such as those provided by LTT are not a problem.

IOW, there's just no point having a debate about static vs.
dynamic instrumentation, the issue is a pointless.

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 1-866-677-4546

Re: next steps

[Jose R. Santos]

Frank Ch. Eigler <fche@redhat.com> [050926]:
> Hi -
> 
> On Mon, Sep 26, 2005 at 10:11:01AM -0500, Jose R. Santos wrote:
> 
> > [...]  one of the requirements that the tool need is that for every
> > probe inserted, the cpu, pid, tid and a time-stamp (jiffies?) be
> > returned as well as some additional information that would be probe
> > specific.
> 
> function ln /* labeled number */ (name, value) {
>   print (name . "=" . string(value) . " ")
> }
> function trace_common_stuff () {
>   ln ("cpu", cpu()) # cpu() to go into context tapset
This would need to look something like this:

function cpu:long () %{
        if (unlikely(in_interrupt()))
                THIS->__retvalue = 0;
        else
                THIS->__retvalue = task_cpu(current);
%}

right?  Seems to be working on my 4-way system.

>   ln ("pid", pid()) 
>   ln ("jiffies", jiffies()))  # jiffies() to go into in timestamp tapset

Just notice the availability of gettimeofday_(us/ms/s) on the
timestamp tapset so I don't think jiffies are needed.  Either
way, it should be quick and painless to implement.

Thanks again.

-JRS

Re: next steps

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

Hi -

On Mon, Sep 26, 2005 at 04:34:11PM -0500, Jose R. Santos wrote:
> [...]
> >   ln ("cpu", cpu()) # cpu() to go into context tapset
> This would need to look something like this:
> 
> function cpu:long () %{
>         [...]
>         THIS->__retvalue = task_cpu(current);
> %}
> right?  Seems to be working on my 4-way system.

Yes, or perhaps just smp_processor_id() if that were more general and
still safe.

> [...]
> >   ln ("jiffies", jiffies()))  # jiffies() to go into in timestamp tapset
> 
> Just notice the availability of gettimeofday_(us/ms/s) on the
> timestamp tapset so I don't think jiffies are needed.  Either
> way, it should be quick and painless to implement.

Exactly.  Feel free to submit a patch for one or both of these
additions to the appropriate .stp files, and brief documentation in
src/stapfuncs.5.  I can commit them for you, or if you forsee becoming
a frequent contributor, you may request cvs write access.

That reminds me.  Time to add a GPL header to the tapset files.


- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: next steps

[Jose R. Santos]

Frank Ch. Eigler <fche@redhat.com> [050926]:
> Hi -
> 
> On Mon, Sep 26, 2005 at 04:34:11PM -0500, Jose R. Santos wrote:
> > [...]
> > >   ln ("cpu", cpu()) # cpu() to go into context tapset
> > This would need to look something like this:
> > 
> > function cpu:long () %{
> >         [...]
> >         THIS->__retvalue = task_cpu(current);
> > %}
> > right?  Seems to be working on my 4-way system.
> 
> Yes, or perhaps just smp_processor_id() if that were more general and
> still safe.
> 
> > [...]
> > >   ln ("jiffies", jiffies()))  # jiffies() to go into in timestamp tapset
> > 
> > Just notice the availability of gettimeofday_(us/ms/s) on the
> > timestamp tapset so I don't think jiffies are needed.  Either
> > way, it should be quick and painless to implement.
> 
> Exactly.  Feel free to submit a patch for one or both of these
> additions to the appropriate .stp files, and brief documentation in
> src/stapfuncs.5.  I can commit them for you, or if you forsee becoming
> a frequent contributor, you may request cvs write access.
> 
> That reminds me.  Time to add a GPL header to the tapset files.

Heres a patch for the cpu() function.  I will request for the cvs
access as well.

Index: stapfuncs.5.in
===================================================================
RCS file: /cvs/systemtap/src/stapfuncs.5.in,v
retrieving revision 1.6
diff -u -r1.6 stapfuncs.5.in
--- stapfuncs.5.in      7 Sep 2005 02:16:59 -0000       1.6
+++ stapfuncs.5.in      27 Sep 2005 15:24:47 -0000
@@ -158,6 +158,10 @@
 target:long ()
 Returns the pid of the target process.
 
+.TP 
+cpu:long ()
+Returns the current cpu of the target process.
+
 .SH FILES
 .nh
 .IR /usr/share/systemtap/tapset
Index: tapset/context.stp
===================================================================
RCS file: /cvs/systemtap/src/tapset/context.stp,v
retrieving revision 1.6
diff -u -r1.6 context.stp
--- tapset/context.stp  6 Sep 2005 17:40:13 -0000       1.6
+++ tapset/context.stp  27 Sep 2005 15:24:47 -0000
@@ -97,3 +97,9 @@
         THIS->__retvalue = _stp_target;
 %}
 
+function cpu:long () %{
+       if (unlikely(in_interrupt()))
+               THIS->__retvalue = 0;
+       else
+               THIS->__retvalue = task_cpu(current);           
+%}

Re: next steps

[Masami Hiramatsu]

Hi, Richard

Richard J Moore wrote:
> However, for performance measurements, once need a mechanism that perturbs
> the system timing as little as possible - even if some events are lost.

I agree.

> I believe djprobes attempts to fulfil this need, but if my understanding is
> correct there are some unfortunate restrictions to its deployment.

As you understood, djprobe have some restriction. But I think
the restriction is not so huge. Djprobe can be used to probe
most of kallsyms address. I believe that is good enough to use
in the real work. And also we can use kprobes instead of djprobe,
when we need to put probes that can not be covered by djprobe.
So, djprobe can coexist with kprobes and it can be deployed, I think.

> So, I'm
> lodging a request that this issue be further discussed at the very least to
> determine whether there is a real need to address it, which I believe
> there. For example using the current mechanism to trace interrupts from a
> Gb ethernet NIC is not going to work particularly well.

All discussions are welcome.

Best Regards,

-- 
Masami HIRAMATSU
2nd Research Dept.
Hitachi, Ltd., Systems Development Laboratory
E-mail: hiramatu@sdl.hitachi.co.jp

Re: thoughts

[Jose R. Santos]

Mathieu Desnoyers <compudj@krystal.dyndns.org> [050926]:
> We are working on a kernel tracing tool that seems to share a lot with the
> trace tool you are talking about.
> 
> We have a kernel tracer (LTTng) and a trace viewer (LTTV) which are based on
> static kernel instrumentation.
> 
> One of the main advantages of the LTTng tracing code is that it is fast. It
> seems to be a downside of SystemTAP. We use a code generator that takes an event
> description to generate the tracing code that can be inserted through module
> loading.
> 
> I think that you might be interested to have a look at it. Here is the website :
> http://ltt.polymtl.ca

Hi Mathieu,

I've been looking at the LTTV component and it is the same basic
concept I wanted to create as part of our project.  It would be nice
not to re-write all this from scratch so I'm looking at how to
integrate LTTV into the SystemTap trace functionality that we are
working on.

Thanks

-JRS