Re: [Ksummit-2008-discuss] Topics for KS 2008 : tracing integration with the kernel

Re: [Ksummit-2008-discuss] Topics for KS 2008 : tracing integration with the kernel

[Masami Hiramatsu]

Hi Mathieu,

Mathieu Desnoyers wrote:
> Hi,
> 
> I've looked at the Dtrace discussion and at the following proposal of a
> SystemTAP discussion at KS2008. I think the broader topic of tracing
> integration with the kernel would encompass the more general subject, in
> which SystemTAP would be one element.
> 
> Talking about various subtopic which have only been partially addressed
> so far such as :
> 
> Instrumentation
>   - We currently have kprobes, linux kernel markers and soon
>     tracepoints. The goal of tracepoints is to try to identify clearly,
>     with a well-defined interface, what tracepoints can be expected from
>     a kernel. Those are defined in include/trace/*.h.

We also has ftrace's function entry probe now. It can be a good
probe facility. And James' simple trace is also possible one.

> Probes
>   - Writing tracing probes meant to be connected to the instrumentation
>     presents a lot of difficulties, ranging from reentrancy wrt
>     interrupts, non-maskable interrupts, other CPUs to dealing with
>     locks already taken in the instrumentation site's execution context.
>   - Various types of probes can re-use the same instrumentation
>     - Specialized probes (e.g. ftrace, blktrace)
>     - Script-like probes, customized by the users (SystemTAP). Mainly
>       meant to detect some pre-defined conditions and to perform some
>       counting.
>     - General purpose tracer. Basically exports the full data flow to
>       userspace through a highly optimized buffering mechanism (LTTng).

Various probes can re-use each basic implementations
 - Time logging code
 - Buffer formatting code
 - Memory object pool
 - Basic calculation code (like statistics in stap)

> 
> Relaying of data to userspace
>   - Pretty much covered by relay and debugfs.

Ftrace has a special logging buffer. We might be able to integrate
those infrastructures to a general configurable communication
interface.

> Questions which are still open issues :
> - Dealing with userspace tracing.
>   Giving the ability to perform early boot-time tracing of processes
>   such as init) adds requirements for new kernel ABIs. Also, do we want
>   to pay the penality cost of adding a system call per probe or should
>   we designed a shared-memory buffer scheme with userspace for tracing ?
>   If we think of pthread mutex instrumentation, the latter is much
>   preferred.

And also we should consider about kernel and user log data merging.
in that case, former is simple and it might have fair performance.

> - How do we assign names to instrumentation sites in a way that would
>   permit tracing the kernel, libraries and userspace programs.
>   Namespacing becomes an issue here.
 - Where and what information are good for us. What kind of information
   can be exposed to userspace.

Thank you,

> 
> I'd definitely be interested to discuss those issues at KS2008,
> especially given the growing interest the community has for tracing
> tools.
> 
> Mathieu
> 

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com