Linux Trace terminology - feedback requested

Linux Trace terminology - feedback requested

[Tim Bird]

Hi all,  please forgive the massive cross-post.

I'm working on coordinating some collaboration between some of the major
tracing systems in Linux, and as a first step I'm working on some terminology
definitions.  This is just to see what people call different things, and is a
precursor to putting together a taxonomy of features and attributes for the
different tracers.

Here is my first attempt:

 * event - an instruction location or system state at a specific point in time
 * event data - information related to an event
    * ie. In some trace systesm, event data size can vary depending on the event.
 * capture - the act of recording event information
 * trace point - a location in the traced software, where an event is "emitted"
 * trace buffer - location where trace data is stored at time of capture
 * trace log - location where trace data is stored long-term
 * configuration interface - the API or mechanism used to configure the tracing engine
 * control interface - the API or mechanism used to control the tracing engine
 * transfer interface - the API or mechanism used to move the trace data from
   kernel to user space
 * trace time - the time when the trace is active
    * ie The trace buffer may be accessed at trace time, that is, while the trace is active.
 * post-processing - manipulation of the trace data after the trace is collected
 * configuration - the set of constraints which determine what events are collected
   and how they are processed in a trace
 * static tracepoint - a trace point statically compiled into the software being traced
 * dynamic tracepoint - a trace point dynamically added to the software being traced
 * aggregation - updating statistics or other analytical information, based on trace events
     * ie. SystemTap can do aggregation at trace time, while KFT and LTTng do
       aggregation during post-processing (mostly).
 * filters - criteria used to limit the events that are processed or captured
 * triggers - criteria used to start and stop tracing automatically

Does your tracing system use different terms for any of the above?
If so, what are they?

Are there any other important terms I am missing?

Your feedback is really appreciated.

Thanks,
 -- Tim

P.S. Nojiri-san - sorry, but I didn't know what list to send to to get
feedback from LKST people.  The english sourceforge list for LKST seems defunct.
Can you forward this as appropriate?  Thanks.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: Linux Trace terminology - feedback requested

[Li Guanglei]

Tim Bird ??:

> I'm working on coordinating ...

In LKET, we have some slight differences, listed below:
> 
> Here is my first attempt:
> 
>  * event - an instruction location or system state at a specific point in time
>  * trace point - a location in the traced software, where an event is "emitted"
I called it "probe point". But "trace point" emphasizes more about 
"tracing".
>  * trace log - location where trace data is stored long-term
> Does your tracing system use different terms for any of the above?
> If so, what are they?
 >

  * event hook - refers to the (trace point definition)/(corresponding 
trace handler) in the *tracing* software for a specific event.

  * trace handler(probe handler) - The codes to be executed when a 
trace point is hit.

Re: Linux Trace terminology - feedback requested

[Richard J Moore]

systemtap-owner@sourceware.org wrote on 19/05/2006 03:55:39:

> Tim Bird ??:
>
> > I'm working on coordinating ...
>
> In LKET, we have some slight differences, listed below:
> >
> > Here is my first attempt:
> >
> >  * event - an instruction location or system state at a specific
> point in time
> >  * trace point - a location in the traced software, where an event
> is "emitted"

And as matter of semantics: point: trace point is better specified as
trace-point or tracepoint;
similarly for probe point;  since these terms do not refer to "points" in
general.


> I called it "probe point". But "trace point" emphasizes more about
> "tracing".
> >  * trace log - location where trace data is stored long-term
> > Does your tracing system use different terms for any of the above?
> > If so, what are they?
>  >
>
>   * event hook - refers to the (trace point definition)/(corresponding
> trace handler) in the *tracing* software for a specific event.
>
>   * trace handler(probe handler) - The codes to be executed when a
> trace point is hit.
>
>

Re: Linux Trace terminology - feedback requested

[Richard J Moore]

systemtap-owner@sourceware.org wrote on 18/05/2006 23:22:48:

> Hi all,  please forgive the massive cross-post.
>
> I'm working on coordinating some collaboration between some of the major
> tracing systems in Linux, and as a first step I'm working on some
terminology
> definitions.  This is just to see what people call different things, and
is a
> precursor to putting together a taxonomy of features and attributes for
the
> different tracers.
>
> Here is my first attempt:
>
>  * event - an instruction location or system state at a specific point in
time

tracepoint is the location that as been chosen for tracing. It may be
active or inactive.

event is the "firing" of the tracepoint, which generally generates event
data.

event data that is recorded is trace data.


>  * event data - information related to an event
>     * ie. In some trace systesm, event data size can vary depending
> on the event.
>  * capture - the act of recording event information
>  * trace point - a location in the traced software, where an event
> is "emitted"
>  * trace buffer - location where trace data is stored at time of capture
>  * trace log - location where trace data is stored long-term

For binary tracing "log" is generally replaced by "file"

>  * configuration interface - the API or mechanism used to configure
> the tracing engine
>  * control interface - the API or mechanism used to control the tracing
engine
>  * transfer interface - the API or mechanism used to move the trace data
from
>    kernel to user space
>  * trace time - the time when the trace is active
>     * ie The trace buffer may be accessed at trace time, that is,
> while the trace is active.
>  * post-processing - manipulation of the trace data after the trace
> is collected

after then entire tracing episode has ended? I.e post mortem so to speak?


>  * configuration - the set of constraints which determine what
> events are collected
>    and how they are processed in a trace
>  * static tracepoint - a trace point statically compiled into the
> software being traced
>  * dynamic tracepoint - a trace point dynamically added to the
> software being traced

Need to define tracepoint for that definition to work :-)

>  * aggregation - updating statistics or other analytical
> information, based on trace events
>      * ie. SystemTap can do aggregation at trace time, while KFT and
LTTng do
>        aggregation during post-processing (mostly).
>  * filters - criteria used to limit the events that are processed or
captured

There are also data filters that operate following each trace event before
the event data is recorded in the trace buffer.

>  * triggers - criteria used to start and stop tracing automatically
>
> Does your tracing system use different terms for any of the above?
> If so, what are they?
>
> Are there any other important terms I am missing?
>
> Your feedback is really appreciated.
>
> Thanks,
>  -- Tim
>


regards,

Richard J M

Re: [ltt-dev] Linux Trace terminology - feedback requested

[Michel Dagenais]

> as a first step I'm working on some terminology
> definitions.

Very good idea!

>  * event - an instruction location or system state at a specific point in time
>  * event data - information related to an event
>     * ie. In some trace systesm, event data size can vary depending on the event.
>  * capture - the act of recording event information
event capture

>  * trace point - a location in the traced software, where an event is "emitted"
>  * trace buffer - location where trace data is stored at time of capture
>  * trace log - location where trace data is stored long-term
>  * configuration interface - the API or mechanism used to configure the tracing engine
You may want to define the "tracing engine" and then use the name
"tracing engine configuration interface".

>  * control interface - the API or mechanism used to control the tracing engine
tracing engine control interface

>  * transfer interface - the API or mechanism used to move the trace data from
>    kernel to user space
trace transfer interface

>  * trace time - the time when the trace is active
>     * ie The trace buffer may be accessed at trace time, that is, while the trace is active.
>  * post-processing - manipulation of the trace data after the trace is collected
trace post-processing

>  * configuration - the set of constraints which determine what events are collected
>    and how they are processed in a trace
tracing engine configuration

>  * static tracepoint - a trace point statically compiled into the software being traced
>  * dynamic tracepoint - a trace point dynamically added to the software being traced

You may want to distinguish tracepoints which are labeled directly in
the source code ("in source tracepoint", e.g. printk like statement
permanently in the source file) from those which are described elsewhere
("externally described tracepoint", e.g. gdb breakpoint described in a
gdb script or dynamic kernel probe described in a script). In source
tracepoints could be "statically compiled out" (through proper C
preprocessor magic) or compiled in as "static tracepoints". They may
even be statically compiled out but later inserted as dynamic
tracepoints. Similarly, "externally described tracepoints" could be
compiled in as static tracepoints through the use of a special source
code preprocessor, or could be inserted as dynamic tracepoints.

Of course the typical case is "in source tracepoints" used as "static
tracepoints" and "externally described tracepoints" used as "dynamic
tracepoints". Nonetheless, it is important to at least stress the fact
that static tracepoints can be compiled out for zero overhead.

>  * aggregation - updating statistics or other analytical information, based on trace events
>      * ie. SystemTap can do aggregation at trace time, while KFT and LTTng do
>        aggregation during post-processing (mostly).
Aggregation during postprocessing is a given. Perhaps you want to more
specifically refer to dynamic aggregation (simple computation performed
at event capture time?

>  * filters - criteria used to limit the events that are processed or captured
>  * triggers - criteria used to start and stop tracing automatically

Trace point activation is an important topic for which more options
should appear in LTT in the near future. The activation of a group of
trace points (e.g. networking events) could be performed statically (at
compile time) or dynamically. It could be implemented at the trace point
location (for instance as self modifying code containing a conditional
instruction testing an immediate boolean or NOPs used as placeholders
for the tracing instructions), in which case the activation takes effect
globally. It could also be implemented as filtering on a trace by trace
basis (user A wants all his processes traced for scheduling and I/O
events, user B wants one process traced for system calls and networking
events).

Perhaps we should talk about "static trace point activation" (compiled
in or not) and "dynamic trace point activation" (activated or not at the
trace point, for all traces). Then, "filtering" could be used to decide
if a specific event gets stored in a specific trace when more than one
trace is recorded at a time.

Re: [ltt-dev] Linux Trace terminology - feedback requested

[Frank Ch. Eigler]

Hi -


> [...] In source tracepoints could be "statically compiled out"
> (through proper C preprocessor magic) or compiled in as "static
> tracepoints". They may even be statically compiled out but later
> inserted as dynamic tracepoints.

Have you heard of any system that does that effectively?  In systemtap
we've found that it is not unusual for the kernel compiler to mangle
code and data flows so much that there is little hope for an
arbitrarily placed *comment* ("compiled out") to be preserved in a
sensible way.  There is no PC associated with a piece of code that the
compiler can't see - thus no dynamic breakpoint insertable "there".


> [...]  Of course the typical case is "in source tracepoints" used as
> "static tracepoints" and "externally described tracepoints" used as
> "dynamic tracepoints". Nonetheless, it is important to at least
> stress the fact that static tracepoints can be compiled out for zero
> overhead.

This distinction seems to offer little value.  A really compiled-out
static probe is just as useful as a dynamic-probe script that is not
running.

If, on the other hand, "compiled-out" means something milder, like
leaving a number of NOPs in the object code, then we should consider a
different term, since even NOPs don't come free.


- FChE

Re: [ltt-dev] Linux Trace terminology - feedback requested

[Michel Dagenais]

> Have you heard of any system that does that effectively?  In systemtap
> we've found that it is not unusual for the kernel compiler to mangle
> code and data flows so much that there is little hope for an
> arbitrarily placed *comment* ("compiled out") to be preserved in a
> sensible way.

This is unfortunately very true. Perhaps you can leave a label or a
special directive that effectively reserves that location in the code
and somewhat limits what the optimizer will do there.

> This distinction seems to offer little value.  A really compiled-out
> static probe is just as useful as a dynamic-probe script that is not
> running.
...
> If, on the other hand, "compiled-out" means something milder, like
> leaving a number of NOPs in the object code, then we should consider a
> different term, since even NOPs don't come free.

Then indeed we may need a name for these things; do we have something in
the source code, is there something in the binary (NOPS place holder,
conditional instruction...).