Re: Potential Systemtap topics for the Kernel Summit

[Masami Hiramatsu <mhiramat@redhat.com>]

Theodore Tso wrote:
> On Wed, Jul 02, 2008 at 03:25:19PM -0400, Frank Ch. Eigler wrote:
>> So, one way a kernel developer could help write a tapset piece for us
>> is to encapsulate this into a tapset script fragment:
>>
>> probe vfs.read = kernel.function ("vfs_read")
>>   {
>>     dev_nr = $...expression
>>     inode_nr = $...expression
>>   }
>>
>> ****  or  ****
>>
>> Kernel maintainers could add a marker or two right into their C code:
>>
>> {
>>     /* ... */
>>     trace_mark (vfs_read, "dev %u inode %u whatever %s",
>>                           expression1, expression2, whatever);
>>     /* ... */
>> }
> 
> So it sounds like potential Systemtap topics for the kernel summit
> might include:
> 
> A)  Enhancements to kbuild to better support kernel developers who want
> to use systemtap.
> 
> B) Discussion of list of tapsets and markers that would be useful for
> System administrators wanting to use systemtap.  This is one place
> where if someone could volunteer to examine some of the Dtrace
> examples and blog entries where Dtrace users have raved about how
> Dtrace saved their bacon by instantly identifying some performance
> problem, and then assemble a "tapset or marker WANTED" bounty list,
> that would be very useful.  One potential problem is that I suspect
> kernel developers may not know or have the intuition of what sort of
> markers or tapsets would be most useful.  Having a targetted wish list
> would be very useful.  (We might then have some discussions about
> whether a particular tapset or marker is too hard to maintain, or
> represents too much of a performance hit, but at least would be
> dealing with concrete requests.)

I think, not only tapset, but also (a part of) runtime and building script
should be merged into kernel tree. Some of them depends on kernel features
and currently we uses autoconf-like method for checking whether we can use
those features. Obviously, this method has a limitation. :-(

> C) Whether tapsets/markers should be maintained inside the kernel, and
> if so, how.

We have been discussing how we can push markers into upstream on LKML.
http://lkml.org/lkml/2008/6/2/463
http://lkml.org/lkml/2008/6/20/250

as far as I know, current discussion point of markers are;
- Interface Ugliness
  Markers needs printf-style format string, and some people
  complains that is ugly.
- KABI exposure
  some Markers expose internal kernel structures to user space,
  it is not useful because those internal structures rapidly change.
  And also, might we treat it as stable ABIs?
- Maintenance issue
  We might have to consider maintenance cost of markers which are
  scattered throughout the kernel. If those markers perpetuate
  the logic and code, it can prevent linux's evolution.
  And subsystem maintainers might want not to get involved
  in maintenance of markers embedded in their code.

For resolving first and second issues technically, Mathieu
proposed tracepoint layer.
http://lkml.org/lkml/2008/6/25/426

I think we still have maintenance issue. So we need to discuss
clear policies for marker maintenance which don't prevent
linux evolution.

For example,
- Subsystem maintainers or developers can freely modify
  or remove trace points by their patch, if needed.
- Marker/Tracepoint maintainers should follow upstream changes.
- No one can NAK a patch from the reason that it
  modifies or removes trace points.
etc.

> 
> D) What is the right way to do user probes.
> 
> Of course, if some of these topics are handled via e-mail before the
> kernel summit, even better.  But somehow, I'm guessing there will
> still be more to talk about.  :-)
> 
> The bottom line is more communication between the kernel and systemtap
> developers is a good thing (and getting more kernel developers to use
> systemtap would be a good start).

Yeah, we need to have a party to talk each other. :-P


Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com