New kernel marker wiki page

New kernel marker wiki page

[David Smith]

I've gotten requests for more kernel marker information, so I decided to
summarize what I know on a new wiki page:

<http://sourceware.org/systemtap/wiki/UsingMarkers>

(It was a useful exercise, since I discovered bug 5608 while writing the
page.)

If you see any additional areas (or have additional questions) that need
covering send me an email or add the info yourself.

I hope this helps.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: New kernel marker wiki page

[K.Prasad]

Hi David and all,
		Thanks for posting the wiki.

I thought I would bring the issue of handling structure pointers in SystemTap
script for the members to deliberate upon.

Quoting the wiki here to bring the context:

"How are marker arguments handled that are structure pointers?

Not well. Because systemtap has no DWARF information for marker arguments, it
really doesn't know what type they are. In a DWARF-based probe, you could write
something like this:

probe kernel.function("kfunc") { printf("inode number: %d\n", i->i_ino) }

However, if you try something similar with a marker-based probe, you'll get an
error because systemtap doesn't know that:
* $arg1 is a pointer
* the type of what $arg1 points to"

Solutions to this problem, as detailed in the wiki, and their shortcomings as I
see are:

* Solution proposed - "Use Guru mode (-g option) to embed C code that type-casts
the datatype to the requisite format" - It would be less desirable to clutter
the scripts with embedded C code, and would potentially take away the
'abstraction' and security that these SystemTap scripts are expected to provide.

* Solution proposed - "go ahead and put the interesting structure fields in the
marker itself" - From the previous discussions related to marker patches
(submitted for RCU debugging infrastructure), it was apparent that the community
desired the least amount of instrumentation code in the kernel. This would be
more pronounced when markers are proposed in critical code regions which would
have in the performance of the system. In such cases we may have to export the
parent data structure (which will enable us to extract all the data-members of
the structure of interest, in the marker handler/SystemTap script). Say, if we
have a structure struct X { a, b, c,.....z} and if we are interested in only a
few members (although not all), it would still be worthwhile to embed the
instance of 'struct X' in the trace_mark() - to optimise the marker for a
not-taken case.

* The other method would be to make a similar declaration of the data structure
and use them. But again, this comes with costs such as - bloating of SystemTap
scripts, added complexity and maintenance overhead (keep the data type of the
parameters in the script in sync with the kernel).

Given these usability issues for SystemTap over markers in the absence of DWARF
information, would it still be a good idea to exclude DWARF information or are
there other plans to mitigate them?

Thanks,
K.Prasad

On Mon, 14 Jan 2008 21:38:15 +0530, David Smith <dsmith@redhat.com> wrote:

> I've gotten requests for more kernel marker information, so I decided to
> summarize what I know on a new wiki page:
>
> <http://sourceware.org/systemtap/wiki/UsingMarkers>
>
> (It was a useful exercise, since I discovered bug 5608 while writing the
> page.)
>
> If you see any additional areas (or have additional questions) that need
> covering send me an email or add the info yourself.
>
> I hope this helps.
>

Re: New kernel marker wiki page

[David Smith]

K.Prasad wrote:
> Hi David and all,
> 		Thanks for posting the wiki.

Thanks for reviewing it.

> I thought I would bring the issue of handling structure pointers in SystemTap
> script for the members to deliberate upon.
> 
> Quoting the wiki here to bring the context:
> 
> "How are marker arguments handled that are structure pointers?
> 
> Not well. Because systemtap has no DWARF information for marker arguments, it
> really doesn't know what type they are.

...

> Solutions to this problem, as detailed in the wiki, and their shortcomings as I
> see are:
> 
> * Solution proposed - "Use Guru mode (-g option) to embed C code that type-casts
> the datatype to the requisite format" - It would be less desirable to clutter
> the scripts with embedded C code, and would potentially take away the
> 'abstraction' and security that these SystemTap scripts are expected to provide.
> 
> * Solution proposed - "go ahead and put the interesting structure fields in the
> marker itself" - From the previous discussions related to marker patches
> (submitted for RCU debugging infrastructure), it was apparent that the community
> desired the least amount of instrumentation code in the kernel. This would be
> more pronounced when markers are proposed in critical code regions which would
> have in the performance of the system. In such cases we may have to export the
> parent data structure (which will enable us to extract all the data-members of
> the structure of interest, in the marker handler/SystemTap script). Say, if we
> have a structure struct X { a, b, c,.....z} and if we are interested in only a
> few members (although not all), it would still be worthwhile to embed the
> instance of 'struct X' in the trace_mark() - to optimise the marker for a
> not-taken case.

Note that in the case of markers, we have to avoid looking at things in
a too systemtap-specific way and remember there are other marker users.
 The biggest one, of course, is ltt.  The only solution that ltt has
(that I know of) is the second one listed above - embedding all the
interesting structure fields in the marker itself.  Because of this, I
expect most markers will take that approach - especially since ltt is
also writing the marker patches that are trying to make it to the
upstream kernel.

As to your suggestion of somehow using DWARF information to help us out
with markers, I thought we were trying to figure out ways of moving away
from such a great dependence on DWARF information, not trying to add
more dependence.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: New kernel marker wiki page

[Frank Ch. Eigler]

"K.Prasad" <prasad@linux.vnet.ibm.com> writes:

> [...]
> > "How are marker arguments handled that are structure pointers?
> >
> > Not well. Because systemtap has no DWARF information for marker arguments, it
> > really doesn't know what type they are. [...]

> Solutions to this problem, as detailed in the wiki, and their shortcomings as I
> see are [...]

... are fairly straightforward to understand.


> * Solution proposed - "go ahead and put the interesting structure fields in the
> marker itself" - From the previous discussions related to marker patches
> (submitted for RCU debugging infrastructure), it was apparent that the community
> desired the least amount of instrumentation code in the kernel. This would be
> more pronounced when markers are proposed in critical code regions which would
> have in the performance of the system. [...]

It turns out that adding extra marker parameters is essentially a
performance no-op, consider that even the evaluation of those
parameters is placed inside an unlikely() / -freorder-blocks type
block.  One vs. several makes little difference.

That said, one has to use judgement and context.  A few key parameters
are not too bad; a dozen is probably excessive.


> [...] Given these usability issues for SystemTap over markers in the
> absence of DWARF information, would it still be a good idea to
> exclude DWARF information or are there other plans to mitigate them?

Remember, it's not that we *exclude* dwarf data.  For markers, don't
have any natural references as to where to start looking.  There is no
requirement that a marker name be unique across the kernel.  There is
no single PC address associated with a given marker call site, which
is what we'd need to look in dwarf for variables/locations.  There is
not even a single compilation unit.  Even if we could map a marker
name to a set of PCs, at the time that a marker callback function is
received, we don't know which one (call site) it was.  We also don't
get a copy of pt_regs to start extracting registers from.

Now, if we restrict the complexity a great deal, and are willing to
settle for heuristics, we may be able to do something.  But it's
certainly not a trivial matter of "not excluding" dwarf information.


- FChE