user-space probes -- plan B from outer space

user-space probes -- plan B from outer space

[Frank Ch. Eigler]

Hi -

Here is an outline of how systemtap might support user-space probes,
even in the absence of kernel-based user kprobes.  This is a "plan B"
only, a desperate stopgap until lkml sees the light.  Maybe "plan Z"
is more appropriate, considering the limitations I'm about to outline.

The idea is to support limited systemtap scripts that refer only to
user-space probe targets such as existing processes.  These scripts
would be translated to a user-space probe program instead of a kernel
probe module.

Probes would be specified with a probe point syntax such as:

   user.process(233).statement(0xfeedface)
   user("fche").process("/bin/vi").function("*init*")

Instead of kprobes of a probe module, this probe program would use
ptrace to insert breakpoints into any target processes, perhaps using
code from RDA or GDB.  Given the process-id or process name, systemtap
should be able to locate the necessary debugging information at
translation time.  When probes are hit, the probe process would run
the compiled probe handlers in much the same way as now.  Access to
$target vars should be possible.  The runtime code would have to have
a new variant to use some user-level facility (plain pipes?)  to
communicate with the front-end.


Q: Wouldn't this be slow?
A: Oh yes, quite.  Several ptrace context-switch round-trips per
   probe hit.  Lots more if we want to pull out target-side
   state like $variables or stack backtraces.

Q: What about concurrency?
A: You mean like probes concurrently hit in several target processes,
   like SMP kprobes?  If there was any indication that this was
   worthwhile, then we could make the systemtap-generated probe
   process be multi-threaded (one probe thread per target thread).

Q: Any other limitations?
A: Because of ptrace, any process can be supervised by only one
   process at a time.  So if you run systemtap on a user process,
   you won't be able to run gdb or another systemtap session on it.

Q: What about probing the kernel and user space together?
A: Maybe this scheme would work if kernel-space systemtap probes
   run concurrently, and arrange to share systemtap globals with
   userspace somehow (mmap?).  Shared variables like this would
   likely cause many more locking timeouts (=> skipped probes)
   than now.  There are also additional security concerns.

Q: What about probing shared libraries?
A: Because of the way ptrace works, we'd have to turn these into
   process-level probes, including probes that just sit around
   monitoring the threads and all their children to dlopen/mmap
   the named libraries.

Q: Is it worth it to try?  Is there a better way?
A: You tell me.


- FChE

Re: user-space probes -- plan B from outer space

[Jim Keniston]

On Tue, 2006-06-06 at 12:07, Frank Ch. Eigler wrote:
> Hi -
> 
> Here is an outline of how systemtap might support user-space probes,
> even in the absence of kernel-based user kprobes.  This is a "plan B"
> only, a desperate stopgap until lkml sees the light.  Maybe "plan Z"
> is more appropriate, considering the limitations I'm about to outline.

I'm just now prototyping something very much like what you've
described.  See below for more info.

> 
> The idea is to support limited systemtap scripts that refer only to
> user-space probe targets such as existing processes.  These scripts
> would be translated to a user-space probe program instead of a kernel
> probe module.

I was thinking a user-mode (instrumentation) program + a kernel module
that defines handlers that could be invoked from the instrumentation
program.  The latter (which requires kernel enhancements) is necessary
only for convenient & efficient coordination of user-space and
kernel-space instrumentation.  (But that's what we're after, right?)

> 
> Probes would be specified with a probe point syntax such as:
> 
>    user.process(233).statement(0xfeedface)
>    user("fche").process("/bin/vi").function("*init*")
> 
> Instead of kprobes of a probe module, this probe program would use
> ptrace to insert breakpoints into any target processes,

Got that running, although the API needs to be generalized.

> perhaps using
> code from RDA or GDB.  Given the process-id or process name, systemtap
> should be able to locate the necessary debugging information at
> translation time.  When probes are hit, the probe process would run
> the compiled probe handlers in much the same way as now.  Access to
> $target vars should be possible.  The runtime code would have to have
> a new variant to use some user-level facility (plain pipes?)  to
> communicate with the front-end.

Comm with front end not undertaken yet.

> 
> 
> Q: Wouldn't this be slow?
> A: Oh yes, quite.  Several ptrace context-switch round-trips per
>    probe hit.  Lots more if we want to pull out target-side
>    state like $variables or stack backtraces.

Yes, pretty slow.  In my prototype, my user-mode handler just increments
a counter.  On my Pentium M, overhead per probepoint hit is ~14.2 usec,
compared with 1.03 usec for the uprobes version last posted to LKML.

For comparison, using "gdb -batch" to do the same thing cost 111 usec
per hit, and tracing one syscall with strace cost ~10 usec per hit.  (Of
course, strace can be more efficient than ad hoc probing because ptrace
has special support for syscall tracing; and a C-code handler can do all
sorts of things that a gdb command-script can't.)

> 
> Q: What about concurrency?
> A: You mean like probes concurrently hit in several target processes,
>    like SMP kprobes?  If there was any indication that this was
>    worthwhile, then we could make the systemtap-generated probe
>    process be multi-threaded (one probe thread per target thread).

Yes.  I haven't taken that on.

> 
> Q: Any other limitations?
> A: Because of ptrace, any process can be supervised by only one
>    process at a time.  So if you run systemtap on a user process,
>    you won't be able to run gdb or another systemtap session on it.

Yes.

> 
> Q: What about probing the kernel and user space together?
> A: Maybe this scheme would work if kernel-space systemtap probes
>    run concurrently, and arrange to share systemtap globals with
>    userspace somehow (mmap?).  Shared variables like this would
>    likely cause many more locking timeouts (=> skipped probes)
>    than now.  There are also additional security concerns.

My proposed approach to user/kernel data sharing is a new system call or
ptrace request that just passes a pid, a handler ID, and a pointer to an
area in user space that the handler (installed via a kernel module) can
read and/or write.  Again, there are security concerns.  But a Bad Guy
would have to have the help of somebody who has permission to install
the module.

If all kernel/user comm were initiated by the instrumentation program,
the kernel handler could sleep as needed.

> 
> Q: What about probing shared libraries?
> A: Because of the way ptrace works, we'd have to turn these into
>    process-level probes, including probes that just sit around
>    monitoring the threads and all their children to dlopen/mmap
>    the named libraries.

Yes.

> 
> Q: Is it worth it to try?  Is there a better way?
> A: You tell me.

There are certainly ways that perform better and lack some of these
limitations.  Selling them on LKML is another matter.

An "incremental approach" might enhance ptrace to reduce probepoint
overhead -- e.g., let the kernel handle single-stepping and continuing
all in one ptrace call.

> 
> 
> - FChE

Jim

Re: user-space probes -- plan B from outer space

[Vara Prasad]

Frank Ch. Eigler wrote:

>Hi -
>
>Here is an outline of how systemtap might support user-space probes,
>even in the absence of kernel-based user kprobes.  This is a "plan B"
>only, a desperate stopgap until lkml sees the light.  Maybe "plan Z"
>is more appropriate, considering the limitations I'm about to outline.
>
>The idea is to support limited systemtap scripts that refer only to
>user-space probe targets such as existing processes.  These scripts
>would be translated to a user-space probe program instead of a kernel
>probe module.
>  
>
Well another  approach is similar to Ptrace actions you have predefined 
handlers that some one can activate through a new systemcall. This gives 
the ability execute simple handlers without a need for loading kernel 
modules. For those who wants to do complicated aggregations, correlation 
etc., you can load your own handlers through a module and activate them 
through the same systemcall.  This gives the flexibility for power users 
and simplicity for regular users and we can get away from module 
loading/root permission issues for simple tracing.

>Probes would be specified with a probe point syntax such as:
>
>   user.process(233).statement(0xfeedface)
>   user("fche").process("/bin/vi").function("*init*")
>
>  
>
In my opinion user space probes is most useful in the case of server 
class kind of complicated programs which are usually long living, with 
that background i feel there is not much value with system wide tracing 
instead we should focus on process specific tracing. We could also allow 
users to start a program using systemtap so that we have a chance to 
place probes before running the process, this could be useful to track 
problems during the startup, and of course in this case no pid will be 
specified. 
I am not sure i see the value of process("process name") syntax if our 
focus is process specific tracing.

>Instead of kprobes of a probe module, this probe program would use
>ptrace to insert breakpoints into any target processes, perhaps using
>code from RDA or GDB.  Given the process-id or process name, systemtap
>should be able to locate the necessary debugging information at
>translation time.  When probes are hit, the probe process would run
>the compiled probe handlers in much the same way as now.  Access to
>$target vars should be possible.  The runtime code would have to have
>a new variant to use some user-level facility (plain pipes?)  to
>communicate with the front-end.
>  
>
I am not sure i see lot of value of this solution compared to a gdb 
batch job, but for bit better performance than the heavy weight gdb.
I keep receiving complains from people that strace performance overhead 
makes it prohibitive to solve performance related problems and this is 
going to be even more slow than that so we have to keep that in mind as 
well.

>
>Q: Wouldn't this be slow?
>A: Oh yes, quite.  Several ptrace context-switch round-trips per
>   probe hit.  Lots more if we want to pull out target-side
>   state like $variables or stack backtraces.
>  
>
Jim last week did a simple performance comparison of current uprobes 
solution vs strace solutions to track a process his results were strace 
was 16x slower than current in kernel handlers.

>Q: What about concurrency?
>A: You mean like probes concurrently hit in several target processes,
>   like SMP kprobes?  If there was any indication that this was
>   worthwhile, then we could make the systemtap-generated probe
>   process be multi-threaded (one probe thread per target thread).
>  
>

I would say it is probably not worth the complication instead one 
systemtap process can trace only one user process, so if one wants to 
track several process from several windows run several sessions of  
systemtap, i think this may be a good start than making complicated 
multithreaded stap.

>Q: Any other limitations?
>A: Because of ptrace, any process can be supervised by only one
>   process at a time.  So if you run systemtap on a user process,
>   you won't be able to run gdb or another systemtap session on it.
>
>Q: What about probing the kernel and user space together?
>A: Maybe this scheme would work if kernel-space systemtap probes
>   run concurrently, and arrange to share systemtap globals with
>   userspace somehow (mmap?).  Shared variables like this would
>   likely cause many more locking timeouts (=> skipped probes)
>   than now.  There are also additional security concerns.
>  
>
We could introduce a new systemcall that can be called from the handler 
to execute a kernel handler to get correlated info but as you said other 
security concerns might not make it possible to get this accepted.

>Q: What about probing shared libraries?
>A: Because of the way ptrace works, we'd have to turn these into
>   process-level probes, including probes that just sit around
>   monitoring the threads and all their children to dlopen/mmap
>   the named libraries.
>
>  
>
Well if we just admit that we allow process based probing we dont need 
to do any thing more for shared libraries specifically.

>Q: Is it worth it to try?  Is there a better way?
>A: You tell me.
>
>  
>
Not really unless we can't come up with a simple enough of solution that 
lets handlers run in the kernel which gives the performance that we need 
even then we have to justify what is the value of this duplication of 
gdb functionality.

>- FChE
>  
>

Re: user-space probes -- plan B from outer space

[Frank Ch. Eigler]

jkenisto wrote:

> [...]  I was thinking a user-mode (instrumentation) program + a
> kernel module that defines handlers that could be invoked from the
> instrumentation program.  The latter (which requires kernel
> enhancements) is necessary only for convenient & efficient
> coordination of user-space and kernel-space instrumentation.  [...]

This scheme is more advanced than the hypothetical "plan B", which
would require no kernel changes.

> [...]  Yes, pretty slow.  In my prototype, my user-mode handler just
> increments a counter.  On my Pentium M, overhead per probepoint hit
> is ~14.2 usec, compared with 1.03 usec for the uprobes version last
> posted to LKML.  [...]

It would be good to see your code.  I'm surprised the slowdown is
*only* a factor of 14.

> My proposed approach to user/kernel data sharing is a new system
> call or ptrace request that just passes a pid, a handler ID, and a
> pointer to an area in user space that the handler (installed via a
> kernel module) can read and/or write. [...]

What would this user space area be used for?  If all the handlers run
in kernel space, why not keep all the data there too?

- FChE

Re: user-space probes -- plan B from outer space

[Jim Keniston]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Wed, 2006-06-07 at 12:58, Frank Ch. Eigler wrote:
> jkenisto wrote:
> 
...
> 
> > [...]  Yes, pretty slow.  In my prototype, my user-mode handler just
> > increments a counter.  On my Pentium M, overhead per probepoint hit
> > is ~14.2 usec, compared with 1.03 usec for the uprobes version last
> > posted to LKML.  [...]
> 
> It would be good to see your code.  I'm surprised the slowdown is
> *only* a factor of 14.

Code (and README, tests, bug report) attached.

The slowdown is still about 14x (over LKML uprobes), even after
generalizing for multiple probes and multiple processes.  It slows down
significantly (and often breaks -- see README) when probing multiple
heavily-probe-burdened processes simultaneously.

I'm out of the office 'til Monday, BTW.

Jim

[-- Attachment #2: libptp.tar.gz --]
[-- Type: application/x-gzip, Size: 9118 bytes --]

Re: user-space probes -- plan B from outer space

[Frank Ch. Eigler]

Hi -

> Code (and README, tests, bug report) attached.
> The slowdown is still about 14x (over LKML uprobes) [...]

Thanks, well done.  It looks a lot like plan B: no kernel-side changes.

> [...]  It slows down significantly (and often breaks -- see README)
> when probing multiple heavily-probe-burdened processes
> simultaneously. [...]

Unless one can get around it with clever signal manipulation, this may
be a good reason to recast libptp as a multiprocessing system.  (One
would have to experiment to see whether it could be "simply"
multithreaded or a full-blown multi-process widget is required.)  In
any case, it would avoid multiplexing multiple sources of ptrace
SIGCHLD on any particular probing process.

Another major stumbling block will be when attempting to instrument
multithreaded target programs, where single-stepping and opcode
replacement need to be synchronized.  This is a source of complexity
in real debuggers.

- FChE

Re: user-space probes -- plan B from outer space

[Frank Ch. Eigler]

prasadav wrote:

> [...]  Well another approach is similar to Ptrace actions you have
> predefined handlers that some one can activate through a new
> systemcall. [...] 

A fixed pool of predefined handlers seem like an antithesis of
systemtap.  Did you have some user interface in mind for these?

> >   user.process(233).statement(0xfeedface)
> >   user("fche").process("/bin/vi").function("*init*")
>
> In my opinion user space probes is most useful in the case of server
> class kind of complicated programs which are usually long living,
> [...]  there is not much value with system wide tracing instead we
> should focus on process specific tracing. [...]

There is no contradiction here.  System-wide probing can be
accomplished by a collection of process-specific probes.

> [...] I am not sure i see the value of process("process name")
> syntax if our focus is process specific tracing.

It would be one way of identifying present or future processes to
probe.  For processes that do not yet exist, what other scheme do you
have in mind?

> I am not sure i see lot of value of this solution compared to a gdb
> batch job, but for bit better performance than the heavy weight gdb.
> [...]

How would this gdb batch job alternative work?  Are you intending to
compare the expressity of systemtap script with gdb macros?

> >Q: Is it worth it to try?  Is there a better way?
> >A: You tell me.
>
> Not really unless we can't come up with a simple enough of solution
> that lets handlers run in the kernel which gives the performance
> that we need [...]

Sure - that is why I called it "plan B": something to consider if
"plan A" does not come to fruition.

- FChE

Re: user-space probes -- plan B from outer space

[Vara Prasad]

Frank Ch. Eigler wrote:

>prasadav wrote:
>
>  
>
>>[...]  Well another approach is similar to Ptrace actions you have
>>predefined handlers that some one can activate through a new
>>systemcall. [...] 
>>    
>>
>
>A fixed pool of predefined handlers seem like an antithesis of
>systemtap.  Did you have some user interface in mind for these?
>  
>
Like i mentioned in response to Prasanna's user space probes approach  
mail  there will be two types of handlers for user space probes. One 
type is predefined for specific tasks like dump the stack, get registers 
etc., these handlers are built into the kernel. The second type of 
handlers are the ones that can be added on demand using a module load 
and register them similar to kprobes. SystemTap will use the second 
approach.

Association of a break point to handler is done as a separate step and 
either type of the handlers can be associated with the breakpoint. The 
idea of predefined handlers is if you don't want to write any 
complicated handlers and probe only your own process you could use the 
preexisting handlers and don't need to write kernel module and possibly 
don't even need root permission to trace.  This addresses one of the 
comments we got for earlier userspace implementation posting in LKML.

>  
>
>>>  user.process(233).statement(0xfeedface)
>>>  user("fche").process("/bin/vi").function("*init*")
>>>      
>>>
>>In my opinion user space probes is most useful in the case of server
>>class kind of complicated programs which are usually long living,
>>[...]  there is not much value with system wide tracing instead we
>>should focus on process specific tracing. [...]
>>    
>>
>
>There is no contradiction here.  System-wide probing can be
>accomplished by a collection of process-specific probes.
>
>  
>
>>[...] I am not sure i see the value of process("process name")
>>syntax if our focus is process specific tracing.
>>    
>>
>
>It would be one way of identifying present or future processes to
>probe.  For processes that do not yet exist, what other scheme do you
>have in mind?
>  
>
The scheme i am thinking is you could start a new process using 
systemtap, systemtap will then fork a copy of itself and exec the new 
executable on the child. This gives systemtap ability to put probes on 
the child process from the start similar to what gdb does if you start a 
program under debugger.

With the ability to probe already running process and as well as able to 
start new  process under systemtap we can probe most applications.

The one case that we can not cover with the above two is if an 
application has a complicated startup. For example if script A starts 
Script B and script B starts an executable C and executable C is a short 
lived one, since C is a short lived one it will not give us a chance to 
put probes before it exits. I think these kinds of cases are not too 
many to worry for now at least. Do you think of any other useful cases 
that we need to probe not covered by the above two approaches?

>  
>
>>I am not sure i see lot of value of this solution compared to a gdb
>>batch job, but for bit better performance than the heavy weight gdb.
>>[...]
>>    
>>
>
>How would this gdb batch job alternative work?  Are you intending to
>compare the expressity of systemtap script with gdb macros?
>
>  
>
I am not saying gdb macros are as powerful as systemtap scripts. All i 
meant is one can use gdb batch scripts to print the variables you need 
and use all kinds of post processing using your favorite scripting 
language. As the handlers are run in the userspace i am not seeing much 
advantage of  using systemtap language to do filtering in the userspace 
vs post processing using your favorite scripting language.

[...]

>
>- FChE
>  
>

Re: user-space probes -- plan B from outer space

[Frank Ch. Eigler]

Hi -

varap wrote:
> [...]
> >A fixed pool of predefined handlers seem like an antithesis of
> >systemtap.  Did you have some user interface in mind for these?
>
> Like i mentioned in response to Prasanna's user space probes approach  
> mail there will be two types of handlers for user space probes [...]

I understand that this is what you're proposing.

> SystemTap will use the second approach.

OK, so no user interface required for the first.  But of course if
you're building the first approach also, you must have some tool in
mind to at least demonstrate it.


> [...] you could use the preexisting handlers and don't need to write
> kernel module and possibly don't even need root permission to trace.
> [...]

With some cleverness, we can keep separated the issues of these
predefined handlers (and the hypothetical non-systemtap tool that
might use them) and unprivileged probing.


> >>[...] I am not sure i see the value of process("process name")
> >>syntax if our focus is process specific tracing.
>
> >It would be one way of identifying present or future processes to
> >probe.  For processes that do not yet exist, what other scheme do you
> >have in mind?
>
> The scheme i am thinking is you could start a new process [...]

But you were concerned about the process("name") *syntax*.  Sure,
implementing any of these various probes may involve forked observer
processes.  But if you don't have that *syntax*, how else do you want
a script to target a particular process (other than by pid)?


> >>I am not sure i see lot of value of this solution compared to a gdb
> >>batch job, but for bit better performance than the heavy weight gdb.
> >>[...]
> >
> >How would this gdb batch job alternative work?  Are you intending to
> >compare the expressity of systemtap script with gdb macros?
> 
> I am not saying gdb macros are as powerful as systemtap scripts. All i 
> meant is one can use gdb batch scripts to print the variables you need 
> and use all kinds of post processing using your favorite scripting 
> language. As the handlers are run in the userspace i am not seeing much 
> advantage of using systemtap language to do filtering in the userspace 
> vs post processing using your favorite scripting language.

It has the same kind of advantage in user space as it does in kernel
space: the option of safely and compactly
filtering/analyzing/acting-on data in situ rather than archiving
masses of trace data and passively analyzing it after the fact.

- FChE

Re: user-space probes -- plan B from outer space

[Vara Prasad]

Frank Ch. Eigler wrote:

>Hi -
>
>varap wrote:
>  
>
>>[...]
>>    
>>
>>>A fixed pool of predefined handlers seem like an antithesis of
>>>systemtap.  Did you have some user interface in mind for these?
>>>      
>>>
>>Like i mentioned in response to Prasanna's user space probes approach  
>>mail there will be two types of handlers for user space probes [...]
>>    
>>
>
>I understand that this is what you're proposing.
>
>  
>
>>SystemTap will use the second approach.
>>    
>>
>
>OK, so no user interface required for the first.  But of course if
>you're building the first approach also, you must have some tool in
>mind to at least demonstrate it.
>  
>
I don't have any specific tool in mind but giving this interface 
satisfies the need for those who wants to write their own programs to 
trace user space without needing to use systemtap. I am not sure in 
reality how many people will do that but comments seem to ask for such 
an interface.

>
>  
>
>>[...] you could use the preexisting handlers and don't need to write
>>kernel module and possibly don't even need root permission to trace.
>>[...]
>>    
>>
>
>With some cleverness, we can keep separated the issues of these
>predefined handlers (and the hypothetical non-systemtap tool that
>might use them) and unprivileged probing.
>
>  
>
I am fine to keep the issues separate but if this lets us solve the root 
problem in limited cases we can use it until complete solution is found.

On the topic of the root problem, if i understand correctly the main 
reason we need root to use systemtap now is we need to load a module. 
One way to solve that is use pre-compiled modules but we have to deal 
with how we can parameterize them for the specific needs of a script. 
Another way to solve  this is someone comes up with a security model 
where we can designate certain types of modules are "safe" so non root 
users can load or  some security folks comes up with a fine grained 
roles in the system and we have a role that lets users to load systemtap 
kind of modules without needing root privileges.  None of these are easy 
implementations but who am i to say there are no better and easy to 
implement ideas.

>  
>
>>>>[...] I am not sure i see the value of process("process name")
>>>>syntax if our focus is process specific tracing.
>>>>        
>>>>
>>>It would be one way of identifying present or future processes to
>>>probe.  For processes that do not yet exist, what other scheme do you
>>>have in mind?
>>>      
>>>
>>The scheme i am thinking is you could start a new process [...]
>>    
>>
>
>But you were concerned about the process("name") *syntax*.  Sure,
>implementing any of these various probes may involve forked observer
>processes.  But if you don't have that *syntax*, how else do you want
>a script to target a particular process (other than by pid)?
>
>
>  
>
In the scheme i mentioned the syntax i was thinking is a command line 
option some thing like below
stap -e <myexecutable name>

In the above method when we start the process we get the pid and we can 
use the pid syntax so no need to have process("name") syntax to place 
probes. In other words if we are only go to allow process based tracing 
which meant to me as unique process id then what is the purpose of 
process ("name") which is not unique, may be i am missing something.

>>>>I am not sure i see lot of value of this solution compared to a gdb
>>>>batch job, but for bit better performance than the heavy weight gdb.
>>>>[...]
>>>>        
>>>>
>>>How would this gdb batch job alternative work?  Are you intending to
>>>compare the expressity of systemtap script with gdb macros?
>>>      
>>>
>>I am not saying gdb macros are as powerful as systemtap scripts. All i 
>>meant is one can use gdb batch scripts to print the variables you need 
>>and use all kinds of post processing using your favorite scripting 
>>language. As the handlers are run in the userspace i am not seeing much 
>>advantage of using systemtap language to do filtering in the userspace 
>>vs post processing using your favorite scripting language.
>>    
>>
>
>It has the same kind of advantage in user space as it does in kernel
>space: the option of safely and compactly
>filtering/analyzing/acting-on data in situ rather than archiving
>masses of trace data and passively analyzing it after the fact.
>
>  
>
No disagreements in use of systemtap language in kernel and doing it in 
situ. In the userspace however one can pipe the output to the post 
processing process without needing to write to disk.

>- FChE
>  
>

Re: user-space probes -- plan B from outer space

[Frank Ch. Eigler]

Hi -

varap wrote:

> [...]  I don't have any specific tool in mind but giving this
> [predefined ptrace handler] interface satisfies the need for those
> who wants to write their own programs to trace user space without
> needing to use systemtap. I am not sure in reality how many people
> will do that but comments seem to ask for such an interface.

It seems to me that without a specific prototype, you will not know
whether the choice of predefined handlers are actually useful.


> [...]
> On the topic of the root problem, if i understand correctly the main 
> reason we need root to use systemtap now is we need to load a module. 
> One way to solve that is use pre-compiled modules but we have to deal 
> with how we can parameterize them for the specific needs of a script. 

We already have some support for run-time parametrization (setting of
systemtap globals).

> Another way [...]

Yes, possibly, but there are yet other different approaches.  Once we
have workable user-space probing, it will be time to deal with probing
by unprivileged users.


> [...]
> >But you were concerned about the process("name") *syntax*.  [...]
>
> In the scheme i mentioned the syntax i was thinking is a command line 
> option some thing like below
> stap -e <myexecutable name>

I assume you mean "stap -c CMD" or "stap -x PID".  That's fine for
probing a single process, but it may be desirable and not too difficult
to let a script refer to a whole family of processes at once:
- everyone who runs vi
- every future descendant of a given process that has a given name pattern 
- several running processes with different pids

So, like kernel.function("*"), one might think of the user-space probe
point specifiers as possible filters that tell systemtap to focus on
some *subset* of the complete list of present/future processes/threads.


> >>>>I am not sure i see lot of value of this solution compared to a gdb
> >>>>batch job, but for bit better performance than the heavy weight gdb.
> >>>>[...]
>
> No disagreements in use of systemtap language in kernel and doing it
> in situ. In the userspace however one can pipe the output to the
> post processing process without needing to write to disk.

I see no difference between kernel & user space probing in this
regard.  Neither requires copying to disk in principle: in both cases
it should be possible to both filter data in situ and later via a
buffered trace data stream.


- FChE