user instruction tracing patch?

user instruction tracing patch?

[Dave Nomura]

Have you had a chance to review the user instruction tracing patch I 
re-posted on 11/1?

-- 
Dave Nomura
LTC Linux Power Toolchain

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Dave Nomura <dcnltc@us.ibm.com> writes:

> Have you had a chance to review the user instruction tracing patch I
> re-posted on 11/1?

Thanks for demonstrating the implementation as something layered
nicely on top of utrace.  (Sorry for not collecting my notes earlier.)

In order to integrate this facility nicely into systemtap, it should
be integrated as a first class probe point syntax.  You suggested:

  probe usr_itrace.single_step
  probe usr_itrace.block_step

but that's rather unspecific (particularly: which processes to
probe?).  So, here's a possibly better way.  The script-side interface
would look like:

  probe process(PID).itrace { }
  probe process(PID).btrace { }   # block trace
  probe process(NAME).itrace { }
  probe process(NAME).btrace { }

The name variant could be implemented by a startup-time mapping of
execname->pid.  There's probably a utrace hook that will let us be
notified of execs, so that we can keep the mapping up-to-date during
run time.

Since utrace will provide the pt_regs structure, the probe handler
bodies will be able to call e.g. backtrace(), probefunc(), and really
should have some structured access to the registers (a new tapset
function like register:long ("name") ?)

It's also a bit risky to expose the on/off functionality as explicit
functions (though an end-user script or tapset could be more assured
of proper cleanup using "probe end,error {...}".)  But of course it's
desirable to turn things on/off dynamically, so how?  A special case
of bug #4935 might be the ticket, once that is complete:

  probe process(PID).itrace if (condition) { }
  probe process(PID).function("NAME") { condition = 1 }
  probe process(PID).function("NAME").return { condition = 0 }

(The .function("NAME") stuff is of course waiting for symbolic
user-space probing support, in progress separately.)

It seems like with these three separate bits of incoming functions
(itrace, on-the-fly probe on/off, user-space probes), we should get a
comfortable user experience.

- FChE

Re: user instruction tracing patch?

[Roland McGrath]

> Since utrace will provide the pt_regs structure, the probe handler
> bodies will be able to call e.g. backtrace(), probefunc(), and really
> should have some structured access to the registers (a new tapset
> function like register:long ("name") ?)

In general to access all registers you can't always use pt_regs directly;
the details vary by machine.  You can get many of them on the common
machines (x86, ppc).  A simple embedded-C tapset approach for each machine
is certainly an easy place to start.  For full generality, you'll need to
use the formal utrace_regset interfaces instead.  The elfutils libraries
have full information on mapping register names and DWARF numbers to the
formats these interfaces expose, though they could use more sugar.  There
is a variety of ways you could use that info in the translator to expose
register access to the script language.

>   probe process(PID).itrace if (condition) { }

So this is special syntax constrained to just testing a script variable?
Or it's any complex expression reducible to only reading global script
variables?  Or what exactly?  

I like this syntax when it's just a general thing for skipping out probe
hits.  (In fact, you can call it awk and say it's really just a
one-statement probe body since {...} is just a statement.)  

But when its real use is to cause acts that change the value of evaluating
the condition to hook into immediately changing the state of low-level
tracing features that produce the probe hits, then I want to know exactly
what the language semantics guarantees about the timely control of that state.

>   probe process(PID).function("NAME") { condition = 1 }
>   probe process(PID).function("NAME").return { condition = 0 }

For likely answers to my first question, doesn't seem much different from
an explicit enable/disable command, just more obscure when reading the script.
If the variable is not otherwise used, wouldn't that compile to the same as:

    probe process(PID).function("NAME") { enable process(PID).itrace; }
    probe process(PID).function("NAME").return { disable process(PID).itrace; }

or whatever syntax explicit enable/disable has?


Thanks,
Roland

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi, Roland -

On Wed, Nov 14, 2007 at 01:59:40PM -0800, Roland McGrath wrote:
> > Since utrace will provide the pt_regs structure, the probe handler
> [...]
> In general to access all registers you can't always use pt_regs directly;
> the details vary by machine.  You can get many of them on the common
> machines (x86, ppc).  A simple embedded-C tapset approach for each machine
> is certainly an easy place to start. [...]

OK.

> [...]
> >   probe process(PID).itrace if (condition) { }
> 
> So this is special syntax constrained to just testing a script variable?
> Or it's any complex expression reducible to only reading global script
> variables?  Or what exactly?  

It will be defined carefully, something like this: the condition needs
to be an expression that uses only script-level global variables.

> [...]  But when its real use is to cause acts that change the value
> of evaluating the condition to hook into immediately changing the
> state of low-level tracing features that produce the probe hits,
> then I want to know exactly what the language semantics guarantees
> about the timely control of that state.

We discussed this briefly on the mailing list [1].  The provided
timeliness will have to depend on the kind of probe.  There will not
be spurious events (probe hits when the condition is false), but there
may be absent events (the condition is true but probe rearming could
not be accomplished quickly enough).  I expect that we'll use some
lightweight facility such as workqueues to minimize the time window.

[1] http://sourceware.org/ml/systemtap/2007-q3/msg00023.html


> [...] If the variable is not otherwise used, wouldn't that compile to the same as:
> 
>     probe process(PID).function("NAME") { enable process(PID).itrace; }
>     probe process(PID).function("NAME").return { disable process(PID).itrace; }

Yes, but the cases such as controlling many probes at once, or doing
so indirectly (e.g., setting individual values of an alternation),
makes the proposed syntax significantly more expressive, and IMO not
really more opaque.

> or whatever syntax explicit enable/disable has?

There isn't an explicit syntax for that at all.  It's partly in desire
to avoid having to invent it that we're implementing the conditional
widget instead.

- FChE

Re: user instruction tracing patch?

[Roland McGrath]

> Yes, but the cases such as controlling many probes at once, or doing
> so indirectly (e.g., setting individual values of an alternation),
> makes the proposed syntax significantly more expressive, and IMO not
> really more opaque.

I see the point on expressivity for doing many probes.

> > or whatever syntax explicit enable/disable has?
> 
> There isn't an explicit syntax for that at all.  It's partly in desire
> to avoid having to invent it that we're implementing the conditional
> widget instead.

I understand that.  I'm talking about the expressivity and opacity of the
two options compared.  

I can't really agree that it's not substantially more opaque.  Perhaps if
there is a hard requirement on a special sort of variable name for it to be
an "active" one, but that again seems only very cosmetically different from
an explicit enabling syntax.  

What I find opaque in a way I don't like is the idea that I might need to
go through every probe and function body looking for things that might
modify "foo" to know when that means "trigger enabling/disabling itrace",
and then keep track of all such "foo"s by remembering the names used in all
the probe conditions I've read for all itrace (or similarly "active")
probes I've read.  If I were looking for "enable foo;" and "disable foo;"
or some other dedicated syntax, then I would feel much more comfortable
that I could understand what a large body of script code might be doing.
If it's "active:foo = ...", that's fine too, if I can be sure that looking
at all /\<active:.*/ occurrences is exhaustive.  

Basically, my opinion of the fiction that instruction tracing with a
condition is just like a rarely-hit probe with body { if (!condition) { ... }}
is that it's a double-edged sword that's at least as sharp on the back edge
as on the front.  I like the straightforward expressivity.  But I also like
my concise syntax to have unmistakably unambiguous semantics about what's
setting a script-level flag and what's enabling a special kind of continual
sampling with extremely high overhead. 


Thanks,
Roland

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi, Roland -

On Wed, Nov 14, 2007 at 03:38:25PM -0800, Roland McGrath wrote:

> [...]  What I find opaque in a way I don't like is the idea that I
> might need to go through every probe and function body looking for
> things that might modify "foo" to know when that means "trigger
> enabling/disabling itrace", [...]

But there wouldn't be that many probes in an ordinary hand-written
script.  Most fit on a page or two, so finding the control variables
would be doable "by inspection".

- FChE

Re: user instruction tracing patch?

[Jim Keniston]

On Wed, 2007-11-14 at 22:58 -0500, Frank Ch. Eigler wrote:
> Hi, Roland -
> 
> On Wed, Nov 14, 2007 at 03:38:25PM -0800, Roland McGrath wrote:
> 
> > [...]  What I find opaque in a way I don't like is the idea that I
> > might need to go through every probe and function body looking for
> > things that might modify "foo" to know when that means "trigger
> > enabling/disabling itrace", [...]
> 
> But there wouldn't be that many probes in an ordinary hand-written
> script.  Most fit on a page or two, so finding the control variables
> would be doable "by inspection".
> 
> - FChE

I think it would help if we were clear on the sorts of probing we expect
SystemTap to support as it exploits utrace, uprobes, and instruction
tracing (AKA itrace).

For example, will stap provide access to all the task-lifetime events
that utrace reports (on a per-task basis, BTW) -- fork/clone, exec,
exit, signal, syscall, etc?  I hope so.

I often see examples like
  probe process(PID).function("NAME") { ... }

But wouldn't real-life scripts be just as likely to do something like
  probe program("PATHNAME").pid(TBD).function("NAME") { ... }
where PATHNAME specifies the a.out file (w/ dwarf info) and the PID is
somehow specified on the fly -- e.g., after fork and exec probes have
been triggered courtesy of utrace?

Similarly, you'd often want to specify a process ID or (more likely) a
thread/task ID when enabling a utrace- or itrace-based probe.

I can even envision things like
  probe program("PATHNAME").descendent_of($1 /*pid*/).function("NAME")

The above suggests (to me, anyway) that we should either:
1) support probe enablement via a statement in a probe handler, as
Roland and Dave N. suggested (so that pid or tid can be computed and
specified on the fly); or
2) expand the probe statement syntax to accommodate same.

Jim

Re: user instruction tracing patch?

[Frank Ch. Eigler]

jkenisto wrote:

> [...]  For example, will stap provide access to all the
> task-lifetime events that utrace reports (on a per-task basis, BTW)
> -- fork/clone, exec, exit, signal, syscall, etc?  I hope so.

These sound sensible.  It could be interesting to use utrace's system
call interception facilities instead of the system-wide ones we were
discussing yesterday, when the focus can be appropriately restricted.

>[...]
> But wouldn't real-life scripts be just as likely to do something like
>   probe program("PATHNAME").pid(TBD).function("NAME") { ... }
> [...]
>   probe program("PATHNAME").descendent_of($1 /*pid*/).function("NAME")

One can get carried away with encoding too much in the probe point
syntax.  Generally, they are meant to be statically resolvable sorts
of things identifying the parts of the system possibly being affected
by probing.  I'd like to add dynamic things only tastefully - if they
are common, compact, obvious; if translator/runtime automation
significantly assists performance or safety; if they cannot be
reasonably coded up in plain script code.

> The above suggests (to me, anyway) that we should either:
> 1) support probe enablement via a statement in a probe handler [...]
> 2) expand the probe statement syntax to accommodate same.

You may be mixing things up.  The two alternate syntaxes for
on-the-fly probe enable/disable operations are roughly equally capable
in this regard.  (If it takes script code to determine enablement,
then that script code could just as easily use either activating
variables or function calls.)


- FChE

Re: user instruction tracing patch?

[Jim Keniston]

On Fri, 2007-11-16 at 12:16 -0500, Frank Ch. Eigler wrote:
> jkenisto wrote:
> 
...
> 
> >[...]
> > But wouldn't real-life scripts be just as likely to do something like
> >   probe program("PATHNAME").pid(TBD).function("NAME") { ... }
> > [...]
> >   probe program("PATHNAME").descendent_of($1 /*pid*/).function("NAME")
> 
> One can get carried away with encoding too much in the probe point
> syntax.

Yeah, I wasn't proposing that as actual syntax.  The point is that
  probe program("PATHNAME").function("NAME") { ... }
provides everything stap needs to establish the probepoint and handler,
except the pid; and in many cases (it seems to me) the pid(s) won't be
known until after staprun starts (-c and -x notwithstanding).  So how do
we plug in the pid when we enable the probe?

> Generally, they are meant to be statically resolvable sorts
> of things identifying the parts of the system possibly being affected
> by probing.  I'd like to add dynamic things only tastefully - if they
> are common, compact, obvious; if translator/runtime automation
> significantly assists performance or safety; if they cannot be
> reasonably coded up in plain script code.

I'm not sure of the intent of that last phrase.  I assume we don't want
stap scripts to have to resort to embedded C with explicit calls to
[un]register_u[ret]probe() and such.

> > The above suggests (to me, anyway) that we should either:
> > 1) support probe enablement via a statement in a probe handler [...]
> > 2) expand the probe statement syntax to accommodate same.
> 
> You may be mixing things up.  The two alternate syntaxes for
> on-the-fly probe enable/disable operations are roughly equally capable
> in this regard.  (If it takes script code to determine enablement,
> then that script code could just as easily use either activating
> variables or function calls.)

Sure.  But again, I think that we will see cases where the pid/tid can't
be encoded in the static probe definition, and must be specified on the
fly.  I'd like to see that taken into consideration when we're
discussing syntax alternatives.

> 
> 
> - FChE

Jim

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

On Fri, Nov 16, 2007 at 11:36:11AM -0800, Jim Keniston wrote:

> [...]  probe program("PATHNAME").function("NAME") { ... } provides
> everything stap needs to establish the probepoint and handler,
> except the pid [...] So how do we plug in the pid when we enable the
> probe?

See at the bottom.


> > Generally, they are meant to be statically resolvable sorts
> > of things identifying the parts of the system possibly being affected
> > by probing.  I'd like to add dynamic things only tastefully - if they
> > are common, compact, obvious; if translator/runtime automation
> > significantly assists performance or safety; if they cannot be
> > reasonably coded up in plain script code.
> 
> I'm not sure of the intent of that last phrase.  

Maybe the example at the bottom makes this more clear.

> I assume we don't want stap scripts to have to resort to embedded C
> with explicit calls to [un]register_u[ret]probe() and such.

Of course we don't.


> > > The above suggests (to me, anyway) that we should either:
> > > 1) support probe enablement via a statement in a probe handler [...]
> > > 2) expand the probe statement syntax to accommodate same.
> > 
> > You may be mixing things up.  The two alternate syntaxes for
> > on-the-fly probe enable/disable operations are roughly equally capable
> > in this regard.  [...]

> Sure.  But again, I think that we will see cases where the pid/tid can't
> be encoded in the static probe definition, and must be specified on the
> fly.  I'd like to see that taken into consideration when we're
> discussing syntax alternatives.

But that issue is separate from the syntax alternatives for on-the-fly
enablement being discussed.  Here they are, doing something like
"activate the malloc probe for the first 300 ms within an interesting
thread".


(a - being implemented):

   probe program("/bin/vi").function("malloc") if (mp) { .. }
   probe syscall.fork.return { if (interesting_p(pid))  mp = 1 }
   probe timer.ms(300) { mp = 0 }

(b - roland's proposal):

   probe program("/bin/vi").function("malloc") { ... }
   probe syscall.fork.return { if (interesting_p(pid))
                         enable program("/bin/vi").function("malloc"); }
   probe begin,timer.ms(300) { disable program("/bin/vi").function("malloc"); }

These two alternative cases have in common that the logic that
determines whether the "malloc" probe should be enabled is itself
first-class *script code*.  The interesting_p() function can perform
the "shares-ancestry-with -c/-x program" test if desired.  This is
what I was talking about with the "if they cannot be reasonably coded
up in plain script code" line above.


This can be contrasted with how one might implement

   probe program("/bin/vi").function("malloc") { .... }

in terms of uprobes/utrace at all, which operate at the PID level.
Without the runtime providing execname->tid mappings, and some
hidden-from-script machinery to dynamically add/remove uprobes, a user
might have to do something awful like this:

   probe process("*").function("malloc") if (vi_p) { .... }
   probe scheduler.cpu_on { vi_p = (execname() == "/bin/vi") }

And even that relies on fictional wildcarded PIDs, so never mind.
Choosing the "enable" syntax above *makes no difference at all* in
this.  Rather, this is the sort of probe point syntax that I think we
should have language-level support for: where expressing it with
existing mechanisms is just too approximate or painful.


- FChE

Re: user instruction tracing patch?

[Jim Keniston]

On Fri, 2007-11-16 at 14:20 -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> On Fri, Nov 16, 2007 at 11:36:11AM -0800, Jim Keniston wrote:
> 
> > [...]  probe program("PATHNAME").function("NAME") { ... } provides
> > everything stap needs to establish the probepoint and handler,
> > except the pid [...] So how do we plug in the pid when we enable the
> > probe?
> 
> See at the bottom.
...
> 
> > Sure.  But again, I think that we will see cases where the pid/tid can't
> > be encoded in the static probe definition, and must be specified on the
> > fly.  I'd like to see that taken into consideration when we're
> > discussing syntax alternatives.
> 
> But that issue is separate from the syntax alternatives for on-the-fly
> enablement being discussed.  Here they are, doing something like
> "activate the malloc probe for the first 300 ms within an interesting
> thread".
> 
> 
> (a - being implemented):
> 
>    probe program("/bin/vi").function("malloc") if (mp) { .. }
>    probe syscall.fork.return { if (interesting_p(pid))  mp = 1 }
>    probe timer.ms(300) { mp = 0 }
> 
> (b - roland's proposal):
> 
>    probe program("/bin/vi").function("malloc") { ... }
>    probe syscall.fork.return { if (interesting_p(pid))
>                          enable program("/bin/vi").function("malloc"); }
>    probe begin,timer.ms(300) { disable program("/bin/vi").function("malloc"); }

Well, Frank's syntax is prettier in this example.  I had assumed that
Roland's approach included some way to name a probe, so that the enable
statement/function could use that instead of the explicit probe
specification.  As a script evolves during a debugging session, it's
certainly possible for it to accumulate more than one handler for the
same probepoint, so the explicit probe specification would be ambiguous.
Frank's syntax avoids that ambiguity.

But what both examples assume is that a task can dynamically
enable/disable probes only for itself (in the case of utrace/itrace) or
its own process (uprobes).  The underlying kernel facilities don't
impose this constraint.  I expect that self-probing (so to speak) covers
the vast majority of cases, and I'm happy to see you start with that,
but it'll be interesting to see whether a need for "cross-probing"
develops.

...
> 
> 
> - FChE

Jim

Re: user instruction tracing patch?

[Roland McGrath]

> Well, Frank's syntax is prettier in this example.  I had assumed that
> Roland's approach included some way to name a probe, so that the enable
> statement/function could use that instead of the explicit probe
> specification.

I wasn't specific about the real syntax, but I did expect something like
that.  My point really was not about the specificity of the identifier;
the points you bring up about that are good ones that I haven't addressed.
I was simply reacting to the idea of "changing any script variable can run
magic hooks" as opposed to more explicit syntax for magic hooks.


Thanks,
Roland

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

On Fri, Nov 16, 2007 at 01:20:51PM -0800, Jim Keniston wrote:
> [...]
> But what both examples assume is that a task can dynamically
> enable/disable probes only for itself (in the case of utrace/itrace)
> or its own process (uprobes).

Not at all - the script fragments were merely examples of a general
syntax.  There is no reason that the probes being enabled/disabled
might not belong to an altogether "foreign" facility such as a timer,
or a kprobe, or another process's uprobe.  (This one reason why the
implementation of disableable probes will be a bit tricky.)


- FChE

Re: user instruction tracing patch?

[Jim Keniston]

On Fri, 2007-11-16 at 15:37 -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> On Fri, Nov 16, 2007 at 01:20:51PM -0800, Jim Keniston wrote:
> > [...]
> > But what both examples assume is that a task can dynamically
> > enable/disable probes only for itself (in the case of utrace/itrace)
> > or its own process (uprobes).
> 
> Not at all - the script fragments were merely examples of a general
> syntax.  There is no reason that the probes being enabled/disabled
> might not belong to an altogether "foreign" facility such as a timer,
> or a kprobe, or another process's uprobe.  (This one reason why the
> implementation of disableable probes will be a bit tricky.)

Sorry, I still don't get it then.  Given
	probe program("/bin/vi").function("malloc") if (mp) { .. }
if some handler executes
	mp = 1;
how does stap know which instance(s) of vi to probe?

> 
> 
> - FChE

Jim

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

On Fri, Nov 16, 2007 at 01:07:47PM -0800, Jim Keniston wrote:
> [...]
> Sorry, I still don't get it then.  Given
> 	probe program("/bin/vi").function("malloc") if (mp) { .. }
> if some handler executes
> 	mp = 1;
> how does stap know which instance(s) of vi to probe?

None of this exists yet, but it should work similarly to kprobes
where lack of probe point qualification means "all of them".  So

   probe program("/bin/vi").function("malloc")  { }

would be a request to probe all running (and forseeably, any future)
copies of vi.  To filter further, I imagine additional qualifiers such
as ".pid(NUM)", ".uid(NUM)".  These are all to identify the potential
probing target processes.


The conditional enable/disable stuff is intended to be orthogonal.
They do not create or destroy probing targets/opportunities, merely
turn them on or off.

- FChE

Re: user instruction tracing patch?

[Jim Keniston]

On Fri, 2007-11-16 at 16:24 -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> On Fri, Nov 16, 2007 at 01:07:47PM -0800, Jim Keniston wrote:
> > [...]
> > Sorry, I still don't get it then.  Given
> > 	probe program("/bin/vi").function("malloc") if (mp) { .. }
> > if some handler executes
> > 	mp = 1;
> > how does stap know which instance(s) of vi to probe?
> 
> None of this exists yet, but it should work similarly to kprobes
> where lack of probe point qualification means "all of them".  So
> 
>    probe program("/bin/vi").function("malloc")  { }
> 
> would be a request to probe all running (and forseeably, any future)
> copies of vi.  To filter further, I imagine additional qualifiers such
> as ".pid(NUM)", ".uid(NUM)".  These are all to identify the potential
> probing target processes.

In the above examples, if NUM could be a variable name, then I think
we're in business... especially if it could be a set of pids or uids
(implemented as, or perhaps explicitly, an associative array?).  (I can
always dream...)

> The conditional enable/disable stuff is intended to be orthogonal.
> They do not create or destroy probing targets/opportunities, merely
> turn them on or off.
> 
> - FChE

Jim

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

> > [...]
> >    probe program("/bin/vi").function("malloc")  { }
> > 
> > would be a request to probe all running (and forseeably, any future)
> > copies of vi.  To filter further, I imagine additional qualifiers such
> > as ".pid(NUM)", ".uid(NUM)".  These are all to identify the potential
> > probing target processes.
> 
> In the above examples, if NUM could be a variable name, then I think
> we're in business... especially if it could be a set of pids or uids
> [...]

Well, it depends how variable that data is.  Another way to think
about it is to think of the probe point type of specification as
"additive", and as conditionals as "subtractive".  If the data is
fixed at/near startup time, then we can "add" probes only to the
targeted processes and not bother any of the others.  If the data is
highly variable, it may be more appropriate to designate probes in
them all, and "subtract" out the uninteresting ones on the fly by
means of dynamic disabling conditions.

- FChE

Re: user instruction tracing patch?

[Dave Nomura]

Did you have any comments on the actual content of my patch?

Did you want to resolve the translator syntax changes to support 
instruction tracing first?

Most of the discussion regarding the translator syntax changes was 
beyond my understanding of SystemTap.  It sounded like some of the 
issues were broader than just instruction tracing.  Did you come to any 
consensus on what translator changes should be made?

Frank Ch. Eigler wrote:
> Hi -
>
>   
>>> [...]
>>>    probe program("/bin/vi").function("malloc")  { }
>>>
>>> would be a request to probe all running (and forseeably, any future)
>>> copies of vi.  To filter further, I imagine additional qualifiers such
>>> as ".pid(NUM)", ".uid(NUM)".  These are all to identify the potential
>>> probing target processes.
>>>       
>> In the above examples, if NUM could be a variable name, then I think
>> we're in business... especially if it could be a set of pids or uids
>> [...]
>>     
>
> Well, it depends how variable that data is.  Another way to think
> about it is to think of the probe point type of specification as
> "additive", and as conditionals as "subtractive".  If the data is
> fixed at/near startup time, then we can "add" probes only to the
> targeted processes and not bother any of the others.  If the data is
> highly variable, it may be more appropriate to designate probes in
> them all, and "subtract" out the uninteresting ones on the fly by
> means of dynamic disabling conditions.
>
> - FChE
>   


-- 
Dave Nomura
LTC Linux Power Toolchain

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

> Did you have any comments on the actual content of my patch?

Yes, it was a good proof-of-concept of using utrace as the
instruction-stepping infrastructure.

> Did you want to resolve the translator syntax changes to support
> instruction tracing first?

I believe we do need proper translator-side support for this feature,
beyond what the tapset-only prototype provides.  We need end-user
scripts using itrace should not to have to use embedded-c code, and
we need the implementation to be robust (i.e., not require users
to match each "turn on itrace for pid PID" with "... off ...".

The first pair of probe point extensions listed in
http://tinyurl.com/2e77sb:

  probe process(PID).itrace { }
  probe process(PID).btrace { }   # block trace

should be reasonably easy to implement.  The embedded-C code currently
in the tapset would become some mixture of code inlined by the
translator, or else copied into the runtime.  Adding dynamic
designation of target processes would be a small followup step.


> Most of the discussion regarding the translator syntax changes was
> beyond my understanding of SystemTap.  [...]

We did go a bit far afield from the initial question, sorry about that.


- FChE

Re: user instruction tracing patch?

[Dave Nomura]

So what do you think the next steps are to move forward on this issue?  

Did you have some expectations on who would do this translator work?

 From your discussion it sounded like some of these issues were more 
general than just instruction tracing.
Are there plans to do any of that work?

Frank Ch. Eigler wrote:
> Hi -
>
>   
>> Did you have any comments on the actual content of my patch?
>>     
>
> Yes, it was a good proof-of-concept of using utrace as the
> instruction-stepping infrastructure.
>
>   
>> Did you want to resolve the translator syntax changes to support
>> instruction tracing first?
>>     
>
> I believe we do need proper translator-side support for this feature,
> beyond what the tapset-only prototype provides.  We need end-user
> scripts using itrace should not to have to use embedded-c code, and
> we need the implementation to be robust (i.e., not require users
> to match each "turn on itrace for pid PID" with "... off ...".
>
> The first pair of probe point extensions listed in
> http://tinyurl.com/2e77sb:
>
>   probe process(PID).itrace { }
>   probe process(PID).btrace { }   # block trace
>
> should be reasonably easy to implement.  The embedded-C code currently
> in the tapset would become some mixture of code inlined by the
> translator, or else copied into the runtime.  Adding dynamic
> designation of target processes would be a small followup step.
>
>
>   
>> Most of the discussion regarding the translator syntax changes was
>> beyond my understanding of SystemTap.  [...]
>>     
>
> We did go a bit far afield from the initial question, sorry about that.
>
>
> - FChE
>   


-- 
Dave Nomura
LTC Linux Power Toolchain

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

> So what do you think the next steps are to move forward on this issue?  
> Did you have some expectations on who would do this translator work?

We need that famous "someone" to do the work.  Would you like to try
it?  It should not be much harder than (say) the timer.profile
implementation.

> From your discussion it sounded like some of these issues were more
> general than just instruction tracing.  Are there plans to do any of
> that work?

Yes, some of them are underway.

- FChE

Re: user instruction tracing patch?

[Dave Nomura]

The Perfomance Inspector group, who currently host the ITRACE tool on, 
are interested in exploring the use of Systemtap to build their ITRACE  
tool on RedHat.  In an effort to jump start them I am planning on 
submitting my user instruction tracing patch to them and helping them 
build an ITRACE RPM that does userspace level instruction tracing as 
well as some other functionality that they are interested in.  This does 
not diminish our commitment to getting this functionality into 
Systemtap, but is more of a temporary measure.  In the long term I 
envision the basic userspace instruction tracing functionality 
integrated into Systemtap, and PI possibly building on this framework to 
support some of the other features that they are interested in. 

I should be able to start doing the Systemtap translator work as soon as 
I finish up helping with the PI effort.   Since I am a SystemTap 
neophyte I am hoping to get some guidance from you, the Systemtap 
community, and hopefully with local help from Jim Keniston.

Frank Ch. Eigler wrote:
> Hi -
>
>   
>> So what do you think the next steps are to move forward on this issue?  
>> Did you have some expectations on who would do this translator work?
>>     
>
> We need that famous "someone" to do the work.  Would you like to try
> it?  It should not be much harder than (say) the timer.profile
> implementation.
>
>   
>> From your discussion it sounded like some of these issues were more
>> general than just instruction tracing.  Are there plans to do any of
>> that work?
>>     
>
> Yes, some of them are underway.
>
> - FChE
>   


-- 
Dave Nomura
LTC Linux Power Toolchain

Re: user instruction tracing patch?

[Dave Nomura]

I am finally at a point where I can start looking at the translator 
changes to support instruction tracing.  There was a discussion of the 
pros/cons of your proposed support, only about half of which I could 
follow between Jim Keniston, Roland McGrath, and yourself.  Did that 
discussion result in any modifications to your proposal below?

I have a few other questions (inline below).
Frank Ch. Eigler wrote:
> So, here's a possibly better way.  The script-side interface
> would look like:
>
>   probe process(PID).itrace { }
>   probe process(PID).btrace { }   # block trace
>   probe process(NAME).itrace { }
>   probe process(NAME).btrace { }
>
> The name variant could be implemented by a startup-time mapping of
> execname->pid.  There's probably a utrace hook that will let us be
> notified of execs, so that we can keep the mapping up-to-date during
> run time.
>   
So the NAME variant would be like "probe process("/usr/bin/vi").btrace 
and this would get mapped to all instances of "/usr/bin/vi" processes at 
startup time, and we would use a utrace hook to add in probes for newly 
created instances of vi?

These probes simply establish that instruction tracing, either itrace, 
or btrace, is requested for specific processes, and associating a 
generated handler corresponding to the { ... } part.

> Since utrace will provide the pt_regs structure, the probe handler
> bodies will be able to call e.g. backtrace(), probefunc(), and really
> should have some structured access to the registers (a new tapset
> function like register:long ("name") ?)
>
> It's also a bit risky to expose the on/off functionality as explicit
> functions (though an end-user script or tapset could be more assured
> of proper cleanup using "probe end,error {...}".)  But of course it's
> desirable to turn things on/off dynamically, so how?  A special case
> of bug #4935 might be the ticket, once that is complete:
>
>   probe process(PID).itrace if (condition) { }
>   probe process(PID).function("NAME") { condition = 1 }
>   probe process(PID).function("NAME").return { condition = 0 }
>
>   
So, rather than have the user explicitly call routines to turn on/off 
tracing for a PID, you set this condition variable. After talking with 
Jim Keniston, my understanding is that, in this example, the fact that 
there is an "if (condition)" on the it .itrace probe the translator will 
be looking for any modifications to "condition" as a signal to insert a 
call to turn on/off instruction tracing.
The user must be aware that "condition = 1" has the side effect of 
turning on instruction tracing. He would have to avoid things like:
       condition = 0;  // default
       if (criteria1) then
          condtion = 1;
      endif
Would modification of "condition" have this side-effect anywhere in 
their tapset, or only in the context of a "probe process..." probe?

My understanding is that this if(condition) idea is not specific to 
instruction tracing and there is some ongoing effort to implement it.  
How is that going? 

Is it premature for me to start on instruction tracing support until the 
"if(expr)" support is complete?



-- 
Dave Nomura
LTC Linux Power Toolchain

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi, Dave -

> I am finally at a point where I can start looking at the translator 
> changes to support instruction tracing.  [...]

> >  probe process(PID).itrace { }
> >  probe process(PID).btrace { }   # block trace

This is the only variant you need to worry about, as

> >  probe process(NAME).itrace { }
> >  probe process(NAME).btrace { }

this stuff will come later, based on process-lifetime-tracking code
being currently built by dsmith.


> [...]
> These probes simply establish that instruction tracing, either itrace, 
> or btrace, is requested for specific processes, and associating a 
> generated handler corresponding to the { ... } part.
> 
> >Since utrace will provide the pt_regs structure, the probe handler
> >bodies will be able to call e.g. backtrace(), probefunc(), and really
> >should have some structured access to the registers (a new tapset
> >function like register:long ("name") ?)

The tapset extensions for context access are very similar to the needs
of symbol-only probing being worked on jkenisto.


> >  probe process(PID).itrace if (condition) { }
> >  probe process(PID).function("NAME") { condition = 1 }
> >  probe process(PID).function("NAME").return { condition = 0 }

This part will already work.


 FChE

Re: user instruction tracing patch?

[Frank Ch. Eigler]

Hi -

> Can you point me at the patch(s) that implement conditional probes? 

Search for usage of the condition field in probe_point's.

> Presumably, I'm going to somehow have to identify the inputs [...]
> Can you give me an overview of how the mechanism works in the
> translator, and an idea of how these actions are attached to
> "condition".

You won't need to worry about this part.  It will be a generic feature
that will be activated for all probe families as a matter of course,
once I finish the first one (for ordinary kprobes).  The syntax is
already active, and checking of the condition expressions is already
present (early during pass 2).

- FChE