tapset feedback

tapset feedback

[Martin Hunt]

I have a bunch of issues with the current tapsets and the way tapsets
are implemented.  Others have written more tapsets than me and I'd
really like to know what your experiences are and what you think of the
following.

1. My biggest issue with the way tapsets are implemented is that there
are no namespaces or ways to limit the scope of a function. Plus the
concepts of code libraries and tapsets are overlapped.

Why is this a problem? Consider the functions contained in
aux_syscalls.stp. Some of these might be tempting for systemtap script
writers to use. However, some of these functions use embedded-C and are
safe to call only from certain contexts. Allowing any script writer to
use them bypasses our safety checks.

Secondly, we have a problem when two tapset writers mistakenly use the
same name for a function. This is a namespace issue that could be solved
by use of some simple naming convention. So it is not serious like the
first problem. But still an annoyance that needs addressing now, before
we have many people writing tapsets. And the solution for the first
problem might be able to solve both.

One thing I would recommend is a conceptual split between "tapsets",
which export probe points and a system library, which would export
general-purpose safe functions.

2. I am not running 2.6.14, 2.6.9-20.ELsmp, or 2.6.9-24.ELsmp on any of
my machines, so I cannot use the syscalls tapset. I mention this not to
pick on that tapset, but to point out that we have a serious maintenance
problem created by the way we have implemented tapsets. 

3. We need to rethink how we set probes on sets of functions.  By this I
mean when we do things like:

probe kernel.syscall.* or
probe kernel.function("sys_*)

We need to consider the following cases:
A. What if one of the functions matched does not exist in the current
kernel?  Right now the compilation fails.
B. What if one of the functions cannot be probed because it is
blacklisted?
C. What if the function is inlined? See
http://sourceware.org/bugzilla/show_bug.cgi?id=1570

So what does this have to do with tapsets? Plenty. Tapsets are basically
a bunch of probe points with code to safely extract data.  Often we are
interested in all the probe points in a tapset, such as with
"kernel.syscall.*". But doing that means that we must have an exact list
of the current syscalls in our running kernel, which brings us back to
problem #2 above.

I can think of several possible solutions. Here's one:

Implement a new probe point "kernel.func" which would be like
"kernel.function" except it would just ignore any function that did not
exist in the currently running system. Also it would match inlines.  Add
to the context tapset a function inlined() which would return 1 if the
current probe point was inlined.

So then we could safely do 
probe kernel.func ("sys_*") 

and for tapsets
probe kernel.syscall.get_mempolicy = kernel.func("sys_get_mempolicy")
{...}
probe kernel.syscall.* {...}

This greatly simplifies the maintenance of tapsets because functions get
added and removed from the kernel all the time both because the kernel
changed and because it can be compiled with different options enabled.

Martin

Re: tapset feedback

[Roland McGrath]

> One thing I would recommend is a conceptual split between "tapsets",
> which export probe points and a system library, which would export
> general-purpose safe functions.

Why is this advantageous?  The problems you've cited argue for some kind of
name spaces or module system for systemtap functions and probes.  But I
don't off hand see how they lead you to conclude that distinguishing these
two kinds of libraries (function libraries, and probe libraries, which some
people like to call tapsets).  

> 2. I am not running 2.6.14, 2.6.9-20.ELsmp, or 2.6.9-24.ELsmp on any of
> my machines, so I cannot use the syscalls tapset. I mention this not to
> pick on that tapset, but to point out that we have a serious maintenance
> problem created by the way we have implemented tapsets. 

Hopefully most of these kinds of troubles point to areas where we need to
consider alternative implementation strategies that would be less kludgey
and version-dependent in this way.  

> A. What if one of the functions matched does not exist in the current
> kernel?  Right now the compilation fails.

What does that mean?  A wildcard match will produce some set of actual
probe points, and all of those will exist in the kernel that matches the
debuginfo examined.  Do you mean the situation when one of the patterns
matches no functions at all?

The differences in semantics you suggest for "func" are sensible things to
want.  I certainly don't think that one thing called "function" and one
thing called "func" with slightly different semantics in these couple of
ways is the right way to express it in any language I'd want to write in.
Maybe someone can think of a better approach to satisfy these needs.


Thanks,
Roland

Re: tapset feedback

[Martin Hunt]

On Thu, 2006-01-05 at 01:05 -0800, Roland McGrath wrote:
> > One thing I would recommend is a conceptual split between "tapsets",
> > which export probe points and a system library, which would export
> > general-purpose safe functions.
> 
> Why is this advantageous?  The problems you've cited argue for some kind of
> name spaces or module system for systemtap functions and probes.  But I
> don't off hand see how they lead you to conclude that distinguishing these
> two kinds of libraries (function libraries, and probe libraries, which some
> people like to call tapsets).  

All I'm proposing is that we have a well-defined and documented set of
library functions.  And regardless of how it is implemented, I don't
think we should call it a tapset because it doesn't act like a tapset.

> > A. What if one of the functions matched does not exist in the current
> > kernel?  Right now the compilation fails.
> 
> What does that mean?  A wildcard match will produce some set of actual
> probe points, and all of those will exist in the kernel that matches the
> debuginfo examined.  

I'm thinking about tapsets here. For example, "kernel.syscall.*" may
contain functions that the current kernel doesn't implement.

Martin

Re: tapset feedback

[Roland McGrath]

> All I'm proposing is that we have a well-defined and documented set of
> library functions.  

For general-purpose library functions, I agree.

> And regardless of how it is implemented, I don't think we should call it
> a tapset because it doesn't act like a tapset.

I have never found the term "tapset" especially useful at all, but some
people seem to like it.  To me, it means nothing other than a library
intended to be useful for a particular topic.  It makes sense to me that a
topical library might well provide both probes and functions that user
scripts can use to interrogate the system about that topic.  I certainly
agree that what a library exports to users (probes and/or functions) should
be well-defined and documented, and clearly distinguished from any named
entities used for internal implementation purposes within the library.

> I'm thinking about tapsets here. For example, "kernel.syscall.*" may
> contain functions that the current kernel doesn't implement.

Here I think we should consider first what are the desireable high-level
semantics for referring to exported library-defined probe types such as
these, and then separately the implementation issues in the library code
and in the language to make that happen.

The status quo is not what we want, where kernel.syscall.foobar can
translate into something that won't resolve and kernel.syscall.* just
matches that along with everything else and fails to resolve it later.

I see two reasonable choices for the semantics a user of the language would
want.  Firstly, that probes exported by a library should work, and a
library should not export any probes that cannot work.  So syscalls.stp
would not export kernel.syscall.foobar at all if sys_foobar doesn't exist.
Hence "kernel.syscall.*" won't match foobar and won't ever consider it so
as to have a problem.  If a user script might want to use
kernel.syscall.foobar directly, then it will have to use % conditionals to
avoid those uses on kernels where the probe won't be defined because it
can't work; i.e., have to know the kernel versions to test for, or perhaps
allow libraries/tapsets to export symbolic conditionals so a user can test
with %(have_foobar) in conditionals.  

The other choice is to define a special notion of "never" probes.  Here the
complete set of probe names is a fixed part of the particular tapset's API
to user scripts, across all kernel versions.  So there is always a
kernel.syscall.foobar known to the translator when referenced from a user
script, distinguished from kernel.syscall.somerandomname that doesn't exist
at all.  But depending on the kernel version (or other vagaries of the
individual tapset implementation), kernel.syscall.foobar might actually
boil down to a function entry probe, or it might be defined as a "never"
probe.  A "never" probe never fires, and so no code is emitted associated
with it.  But never probes can have particular effects at translation time,
such as giving you a warning message or an error.  With some combination of
language constructs in scripts and/or in tapsets, and/or translator flags,
you can say things like: warn/err when a never probe is used by name, when
a never probe is one of multiple matches for a probe wildcard, when never
probes are the only matches for a probe wildcard, etc.  For example, it
might make sense to let the syscalls tapset define that
kernel.syscall.wildcard matches should silently ignore never probes, since
if one wants all the syscalls one doesn't care what syscalls known to man
don't exist on this system.  But another tapset might have purely topical
names such that some wildcards should cause complaints about unavailable
probe types.  Likewise, some scripts will be written not to care about the
omitted probes (and should be able to easily say to suppress warning about
such), while some will use more precise interactions between probes and
want errors if any probe is not going to be installed; still others might
want to programatically know in one probe when another probe is never going
to fire (a la the %(have_foobar) conditionals).


I tend toward the latter choice, though it clearly has many details to be
decided.


In implementing either approach, what we can already use in tapsets is the
version conditionals.  i.e., in the first approach use conditionals to omit
probe definitions entirely; and in the second case, to define probe aliases
as "never" instead of as function probes and such.  If doing "never" probes, 
it seems useful to give tapset-writers some more implicit mechanisms, e.g.:

probe kernel.syscall.foobar = kernel.function("sys_foobar") or never { ... }

(This syntax not to be taken seriously.)  i.e., specify to the translator
that a probe alias uses some low-level probe flavor if that matches
anything, or falls back to being a "never" probe if not.



Thanks,
Roland

Re: tapset feedback

[Martin Hunt]

On Thu, 2006-01-05 at 03:17 -0800, Roland McGrath wrote:

>  If a user script might want to use
> kernel.syscall.foobar directly, then it will have to use % conditionals to
> avoid those uses on kernels where the probe won't be defined because it
> can't work; i.e., have to know the kernel versions to test for,

This is exactly the ugly situation we now have. For every function that
gets added/removed we have to know the exact kernel versions it happened
in. And this means systemtap becomes less useful for nonstandard kernels
because we won't know what options they were built with.

>  or perhaps
> allow libraries/tapsets to export symbolic conditionals so a user can test
> with %(have_foobar) in conditionals.

That would have to be a runtime conditional.

> The other choice is to define a special notion of "never" probes.  

So the idea here is that we define for each probe what to do if the
probe does not match any current function. Isn't this kind of similar to
my (perhaps poorly named) kernel.func probe point? Using kernel.func vs
kernel.function means "do nothing if this probe point fails to match".

What I REALLY would like to see is a way to say "here is a list of probe
points. Set probes on as many as possible."

Martin

Re: tapset feedback

[Frank Ch. Eigler]

hunt wrote:

> 1. My biggest issue with the way tapsets are implemented is that there
> are no namespaces or ways to limit the scope of a function. Plus the
> concepts of code libraries and tapsets are overlapped. [...]

The HACKING file includes simple suggestions for prefixing internal
identifiers to avoid namespace collisions.

> 2. I am not running 2.6.14, 2.6.9-20.ELsmp, or 2.6.9-24.ELsmp on any
> of my machines, so I cannot use the syscalls tapset.  [...]

The syscalls tapset has undergone several drastic changes.  This
latest repackaging was checked in without consultation, and indeed I
don't think it's quite right.  So please don't use that as a sole
example to indict the whole scheme.


> > [...] a user script might want to use kernel.syscall.foobar
> > directly, then it will have to use % conditionals to avoid those
> > uses on kernels where the probe won't be defined because it can't
> > work; i.e., have to know the kernel versions to test for,
> 
> This is exactly the ugly situation we now have. 

I don't understand why you think this is "ugly".  If an user script
specifically asks for a particular probe (.foobar), and it does not
exist, why should the user be surprised by an error or having to work
around it?


> For every function that gets added/removed we have to know the exact
> kernel versions it happened in.

That's a different "we" - a systemtap library script developer.  Yes,
if such a developer decides to target a very fluid API, then she will
pay the price of having to conditionalize it sufficiently.


> [...]  What I REALLY would like to see is a way to say "here is a
> list of probe points. Set probes on as many as possible."

Why exactly would you want to say something like that, and not by
using wildcards?


- FChE

Re: tapset feedback

[Roland McGrath]

> > allow libraries/tapsets to export symbolic conditionals so a user can test
> > with %(have_foobar) in conditionals.
> 
> That would have to be a runtime conditional.

How do you figure?  The value I'm talking about is known at translation time.

Re: tapset feedback

[Roland McGrath]

> So the idea here is that we define for each probe what to do if the
> probe does not match any current function. Isn't this kind of similar to
> my (perhaps poorly named) kernel.func probe point? Using kernel.func vs
> kernel.function means "do nothing if this probe point fails to match".

My suggestion percolates the "failed to match" notion up from the tapset
implementation to the user script so that the user can decide about "do
nothing" or "give me an error" or "warn me", in a general way for any such
case from any tapset.