From: Vara Prasad <prasadav@us.ibm.com>
To: "Frank Ch. Eigler" <fche@redhat.com>
Cc: systemtap@sources.redhat.com
Subject: Re: user-space probes -- plan B from outer space
Date: Wed, 07 Jun 2006 01:38:00 -0000 [thread overview]
Message-ID: <44862DF2.3000402@us.ibm.com> (raw)
In-Reply-To: <y0mhd2yt85g.fsf@ton.toronto.redhat.com>
Frank Ch. Eigler wrote:
>Hi -
>
>Here is an outline of how systemtap might support user-space probes,
>even in the absence of kernel-based user kprobes. This is a "plan B"
>only, a desperate stopgap until lkml sees the light. Maybe "plan Z"
>is more appropriate, considering the limitations I'm about to outline.
>
>The idea is to support limited systemtap scripts that refer only to
>user-space probe targets such as existing processes. These scripts
>would be translated to a user-space probe program instead of a kernel
>probe module.
>
>
Well another approach is similar to Ptrace actions you have predefined
handlers that some one can activate through a new systemcall. This gives
the ability execute simple handlers without a need for loading kernel
modules. For those who wants to do complicated aggregations, correlation
etc., you can load your own handlers through a module and activate them
through the same systemcall. This gives the flexibility for power users
and simplicity for regular users and we can get away from module
loading/root permission issues for simple tracing.
>Probes would be specified with a probe point syntax such as:
>
> user.process(233).statement(0xfeedface)
> user("fche").process("/bin/vi").function("*init*")
>
>
>
In my opinion user space probes is most useful in the case of server
class kind of complicated programs which are usually long living, with
that background i feel there is not much value with system wide tracing
instead we should focus on process specific tracing. We could also allow
users to start a program using systemtap so that we have a chance to
place probes before running the process, this could be useful to track
problems during the startup, and of course in this case no pid will be
specified.
I am not sure i see the value of process("process name") syntax if our
focus is process specific tracing.
>Instead of kprobes of a probe module, this probe program would use
>ptrace to insert breakpoints into any target processes, perhaps using
>code from RDA or GDB. Given the process-id or process name, systemtap
>should be able to locate the necessary debugging information at
>translation time. When probes are hit, the probe process would run
>the compiled probe handlers in much the same way as now. Access to
>$target vars should be possible. The runtime code would have to have
>a new variant to use some user-level facility (plain pipes?) to
>communicate with the front-end.
>
>
I am not sure i see lot of value of this solution compared to a gdb
batch job, but for bit better performance than the heavy weight gdb.
I keep receiving complains from people that strace performance overhead
makes it prohibitive to solve performance related problems and this is
going to be even more slow than that so we have to keep that in mind as
well.
>
>Q: Wouldn't this be slow?
>A: Oh yes, quite. Several ptrace context-switch round-trips per
> probe hit. Lots more if we want to pull out target-side
> state like $variables or stack backtraces.
>
>
Jim last week did a simple performance comparison of current uprobes
solution vs strace solutions to track a process his results were strace
was 16x slower than current in kernel handlers.
>Q: What about concurrency?
>A: You mean like probes concurrently hit in several target processes,
> like SMP kprobes? If there was any indication that this was
> worthwhile, then we could make the systemtap-generated probe
> process be multi-threaded (one probe thread per target thread).
>
>
I would say it is probably not worth the complication instead one
systemtap process can trace only one user process, so if one wants to
track several process from several windows run several sessions of
systemtap, i think this may be a good start than making complicated
multithreaded stap.
>Q: Any other limitations?
>A: Because of ptrace, any process can be supervised by only one
> process at a time. So if you run systemtap on a user process,
> you won't be able to run gdb or another systemtap session on it.
>
>Q: What about probing the kernel and user space together?
>A: Maybe this scheme would work if kernel-space systemtap probes
> run concurrently, and arrange to share systemtap globals with
> userspace somehow (mmap?). Shared variables like this would
> likely cause many more locking timeouts (=> skipped probes)
> than now. There are also additional security concerns.
>
>
We could introduce a new systemcall that can be called from the handler
to execute a kernel handler to get correlated info but as you said other
security concerns might not make it possible to get this accepted.
>Q: What about probing shared libraries?
>A: Because of the way ptrace works, we'd have to turn these into
> process-level probes, including probes that just sit around
> monitoring the threads and all their children to dlopen/mmap
> the named libraries.
>
>
>
Well if we just admit that we allow process based probing we dont need
to do any thing more for shared libraries specifically.
>Q: Is it worth it to try? Is there a better way?
>A: You tell me.
>
>
>
Not really unless we can't come up with a simple enough of solution that
lets handlers run in the kernel which gives the performance that we need
even then we have to justify what is the value of this duplication of
gdb functionality.
>- FChE
>
>
next prev parent reply other threads:[~2006-06-07 1:38 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2006-06-06 19:08 Frank Ch. Eigler
2006-06-06 21:50 ` Jim Keniston
2006-06-07 19:59 ` Frank Ch. Eigler
2006-06-09 11:57 ` Jim Keniston
2006-06-09 18:42 ` Frank Ch. Eigler
2006-06-07 1:38 ` Vara Prasad [this message]
2006-06-14 19:11 ` Frank Ch. Eigler
2006-06-14 22:42 ` Vara Prasad
2006-06-14 23:46 ` Frank Ch. Eigler
2006-06-15 1:02 ` Vara Prasad
2006-06-19 17:13 ` Frank Ch. Eigler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=44862DF2.3000402@us.ibm.com \
--to=prasadav@us.ibm.com \
--cc=fche@redhat.com \
--cc=systemtap@sources.redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).