documentation for user-space usage?

documentation for user-space usage?

[Grant Edwards]

I've figured out how to do kernel-tracing on an embedded system by
cross-compiling .stp files and then running them on a target that has
only staprun installed.

But, I've been trying on-and-off for a few months to figure out what
system-tap user-space tracing is and how it works, and I've made very
little progress.

What I need is basically a low overhead alternative to syslog for
embedded system use.

I want to put logging statements in user application source code that
log various events along with maybe one or two parameter values along
with an event.  I need to log up to a few thousand events/second for
short periods of time into a circular buffer and then transfer the raw
log data to a different machine for display/analysis.  I need to start
stop tracing and enable/disable various subsets of those trace points
at runtime (hopefully without having to restart the processes doing
the logging).

About the only documentation I can find on systemtap user-space stuff
is http://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps,
and that's not been very helpful -- it seems mostly to be about
autoconf and RPM. I was hoping to see some example of instrumented C
source, but if it's there, I'm too dumb to find it.

I was under the impression that uprobes support was required, but that
page says only mentions utrace as a requirement.

Is ubprobes support required, or just utrace?

All the examples I can find of user-space probes seem to require that
stap be installed on the machine where the probed user-processes are
being run.  If that's true, then using systemtap isn't feasible for
me.

Am I on the right track looking at systemtap for user application
event logging, or not?

-- 
Grant Edwards               grant.b.edwards        Yow! Today, THREE WINOS
                                  at               from DETROIT sold me a
                              gmail.com            framed photo of TAB HUNTER
                                                   before his MAKEOVER!

Re: documentation for user-space usage?

[Frank Ch. Eigler]

Hi, Grant -


grant.b.edwards wrote:

> I've figured out how to do kernel-tracing on an embedded system by
> cross-compiling .stp files and then running them on a target that has
> only staprun installed.

Great.  (Were you doing this by manual copying, or via the stap-server
code?)

> [...]
> I want to put logging statements in user application source code that
> log various events along with maybe one or two parameter values along
> with an event.  I need to log up to a few thousand events/second for
> short periods of time into a circular buffer and then transfer the raw
> log data to a different machine for display/analysis.  I need to start
> stop tracing and enable/disable various subsets of those trace points
> at runtime (hopefully without having to restart the processes doing
> the logging).

All those look within the realm of systemtap (and probably some other
tools).

> About the only documentation I can find on systemtap user-space stuff
> is http://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps,

The beginner's guide is going to get more content about this soon.
In the mean time, stapprobes(3stap) includes documentation.  See
the
     process("/path/to/binary").{statement,function,mark}(...)
probes.

> I was under the impression that uprobes support was required, but
> that page says only mentions utrace as a requirement.

uprobes (for several platforms) is currently provided as a part of the
systemtap distribution.  It prereqs utrace in the kernel.  Depending
on your architecture and kernel distribution of interest, this could
be all done, or else a moderate amount of porting work.

> All the examples I can find of user-space probes seem to require that
> stap be installed on the machine where the probed user-processes are
> being run.  [...]

User-space probing and kernel-space probing are basically identical
from the point of view of the stap user.  You can cross-compile probes
the same way.

> Am I on the right track looking at systemtap for user application
> event logging, or not?

The theoretical fit is pretty good.  If in practice you are blocked by
some missing unported layer, you could decide between helping port,
prototyping on x86 while someone else ports, and/or investigating
other logging-related tools/libraries.

- FChE

Re: documentation for user-space usage?

[Grant Edwards]

On 2010-10-31, Frank Ch. Eigler <fche@redhat.com> wrote:

> grant.b.edwards wrote:
>
>> I've figured out how to do kernel-tracing on an embedded system by
>> cross-compiling .stp files and then running them on a target that has
>> only staprun installed.
>
> Great.  (Were you doing this by manual copying, or via the stap-server
> code?)

I copy the compiled script to the target via scp/tftp/etc. and then
run it with the staprun command.

>> [...]
>> I want to put logging statements in user application source code that
>> log various events along with maybe one or two parameter values along
>> with an event.  I need to log up to a few thousand events/second for
>> short periods of time into a circular buffer and then transfer the raw
>> log data to a different machine for display/analysis.  I need to start
>> stop tracing and enable/disable various subsets of those trace points
>> at runtime (hopefully without having to restart the processes doing
>> the logging).
>
> All those look within the realm of systemtap (and probably some other
> tools).

One of the other options I'm looking at is LTTng, but the user-space
stuff has apparently never been tested on ARM, and if it does work the
user space event tracing is going to involve a context switch to fetch
a timestamp and even then you won't be able to correlate kernel-space
and user-space events.  LTTng also appears to be pretty heavyweight
and complicated.  For each thread/process that you want to instrument,
you have to have a companion thread that hands the thread's log data
off to a kernel module which then hands it off to daemon, which then
writes it to disk.

>> About the only documentation I can find on systemtap user-space stuff
>> is http://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps,
>
> The beginner's guide is going to get more content about this soon.
> In the mean time, stapprobes(3stap) includes documentation.  See
> the
>      process("/path/to/binary").{statement,function,mark}(...)
> probes.

Assuming utrace/uprobes gets ported to ARM, does such a trace events
involve a context switch to kernel-mode and then back to user-mode?

>> I was under the impression that uprobes support was required, but
>> that page says only mentions utrace as a requirement.
>
> uprobes (for several platforms) is currently provided as a part of
> the systemtap distribution.  It prereqs utrace in the kernel.
> Depending on your architecture and kernel distribution of interest,
> this could be all done, or else a moderate amount of porting work.

I'm using ARM9.  Utrace support patches have apparently been rejected
by the kernel maintainers, so I'd hae to maintain my own fork of the
kernel as well as port uprobes.  I think.

>> All the examples I can find of user-space probes seem to require that
>> stap be installed on the machine where the probed user-processes are
>> being run.  [...]
>
> User-space probing and kernel-space probing are basically identical
> from the point of view of the stap user.

Is the timing impace of a user-space probe no different than that of a
kernel-space probe?

> You can cross-compile probes the same way.
>
>> Am I on the right track looking at systemtap for user application
>> event logging, or not?
>
> The theoretical fit is pretty good.  If in practice you are blocked
> by some missing unported layer, you could decide between helping
> port, prototyping on x86 while someone else ports, and/or
> investigating other logging-related tools/libraries.

I wouldn't mind working on porting uprobes if I was confident that it
would be accepted upstream.  Since utrace support was apparently not
accepted, I'm not too optimistic.

-- 
Grant Edwards               grant.b.edwards        Yow! Does someone from
                                  at               PEORIA have a SHORTER
                              gmail.com            ATTENTION span than me?

Re: documentation for user-space usage?

[Frank Ch. Eigler]

grant.b.edwards wrote:

> [...]
> Assuming utrace/uprobes gets ported to ARM, does such a trace events
> involve a context switch to kernel-mode and then back to user-mode?

Yes.

> [...]  I'm using ARM9.  Utrace support patches have apparently been
> rejected by the kernel maintainers, so I'd hae to maintain my own
> fork of the kernel as well as port uprobes.  I think.

How close is your current kernel to fedora's?

>> [...] User-space probing and kernel-space probing are basically identical
>> from the point of view of the stap user.
>
> Is the timing impact of a user-space probe no different than that of a
> kernel-space probe?

It's very similar.

>> [...]
>> The theoretical fit is pretty good.  If in practice you are blocked
>> by some missing unported layer, you could decide between helping
>> port, prototyping on x86 while someone else ports, and/or
>> investigating other logging-related tools/libraries.
>
> I wouldn't mind working on porting uprobes if I was confident that it
> would be accepted upstream.  Since utrace support was apparently not
> accepted, I'm not too optimistic.

Another option is to go ahead and try to port uprobes, leave
ARM/utrace to us / fedora people.  When/if the newer lkml-track
uprobes gets merged, the hypothetical ARM port could go into the main
kernel that way, bypassing the utrace kerfuffle.  IOW, doing an ARM
port of the current systemtap-resident uprobes would not be a wasted
effort, if LKML gets its act together and merges the other one.


- FChE

Re: documentation for user-space usage?

[Josh Stone]

On 10/31/2010 09:26 AM, Frank Ch. Eigler wrote:
> User-space probing and kernel-space probing are basically identical
> from the point of view of the stap user.  You can cross-compile probes
> the same way.

I'm not so sure: When we examine kernel probes, we're able to consider
different paths for local vs. cross-compiled kernels.  But when trying
to probe "/usr/bin/foo", we're only considering what's local.

Perhaps we need a "--rootfs" option? This could give a local path to
find binaries that will be located on the root of the target.  i.e. for
translation we'd look at "$rootfs/usr/bin/foo", but at runtime
task-finder would match simply "/usr/bin/foo".

Josh

Re: documentation for user-space usage?

[William Cohen]

On 10/31/2010 12:26 PM, Frank Ch. Eigler wrote:
> 
> Hi, Grant -
> 
> 
> grant.b.edwards wrote:
...
>> About the only documentation I can find on systemtap user-space stuff
>> is http://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps,
> 
> The beginner's guide is going to get more content about this soon.
> In the mean time, stapprobes(3stap) includes documentation.  See
> the
>      process("/path/to/binary").{statement,function,mark}(...)
> probes.

I just checked in some material for the Systemtap beginner's guide into the systemtap git tree. This is to address PR11823 - document user-space probes and other recent features in beginner's guide:

http://sources.redhat.com/bugzilla/show_bug.cgi?id=11823

There are some todo left on this material:

-compiling the uprobes.ko module when needed
-pretty printing $ $$ suffixes
-user-space stack backtraces

Any comments or feedback on the material would be appreciated.

-Will

Re: documentation for user-space usage?

[Grant Edwards]

On 2010-11-01, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> grant.b.edwards wrote:
>
>> [...]
>> Assuming utrace/uprobes gets ported to ARM, does such a trace events
>> involve a context switch to kernel-mode and then back to user-mode?
>
> Yes.

That might be a concern if it affects timings too much.  If we roll
our own, the plan would be to keep it purely user-space in order to
avoid the impact of context switching when logging is enabled.

>> [...]  I'm using ARM9.  Utrace support patches have apparently been
>> rejected by the kernel maintainers, so I'd hae to maintain my own
>> fork of the kernel as well as port uprobes.  I think.
>
> How close is your current kernel to fedora's?

I've no idea.  I'm currently using vanilla 2.6.30 sources with some
AT91-specific patches from Atmel.

>>> [...] User-space probing and kernel-space probing are basically
>>> identical from the point of view of the stap user.
>>
>> Is the timing impact of a user-space probe no different than that of
>> a kernel-space probe?
>
> It's very similar.
>
>>> [...]
>>> The theoretical fit is pretty good.  If in practice you are blocked
>>> by some missing unported layer, you could decide between helping
>>> port, prototyping on x86 while someone else ports, and/or
>>> investigating other logging-related tools/libraries.
>>
>> I wouldn't mind working on porting uprobes if I was confident that it
>> would be accepted upstream.  Since utrace support was apparently not
>> accepted, I'm not too optimistic.
>
> Another option is to go ahead and try to port uprobes, leave
> ARM/utrace to us / fedora people.  When/if the newer lkml-track
> uprobes gets merged, the hypothetical ARM port could go into the main
> kernel that way, bypassing the utrace kerfuffle.  IOW, doing an ARM
> port of the current systemtap-resident uprobes would not be a wasted
> effort, if LKML gets its act together and merges the other one.

OK, thanks.  Can anybody provide a guess as to how much porting needs
to done (assuming a competent kernel-mode programmer who knows nothing
about the tracing stuff)?

-- 
Grant Edwards               grant.b.edwards        Yow! Spreading peanut
                                  at               butter reminds me of
                              gmail.com            opera!!  I wonder why?

Re: documentation for user-space usage?

[Jim Keniston]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

On Mon, 2010-11-01 at 20:53 +0000, Grant Edwards wrote:
> On 2010-11-01, Frank Ch. Eigler <fche@redhat.com> wrote:
...
> >
> > Another option is to go ahead and try to port uprobes, leave
> > ARM/utrace to us / fedora people.  When/if the newer lkml-track
> > uprobes gets merged, the hypothetical ARM port could go into the main
> > kernel that way, bypassing the utrace kerfuffle.  IOW, doing an ARM
> > port of the current systemtap-resident uprobes would not be a wasted
> > effort, if LKML gets its act together and merges the other one.
> 
> OK, thanks.  Can anybody provide a guess as to how much porting needs
> to done (assuming a competent kernel-mode programmer who knows nothing
> about the tracing stuff)?
> 

Attached is the uprobes porting guide, updated to reflect SystemTap's
version of uprobes.  Most of the functions and macros you'd need to
provide are very simple, assuming you know about how ARM's "breakpoint"
instruction works.  You can get most of this info from the kprobes code.

It looks like the ARM version of kprobes emulates instructions rather
than single-stepping them.  So the bulk of your work will be:
a) deciding which instructions you'll allow users to probe; and
b) making the emulation code (which was presumably written for kernel
instructions) work for user-space instructions.

Aside from (a) and (b) above, the uprobes port should be pretty much
paint-by-numbers coding, and then (once the utrace port gets done)
testing and debugging.

Jim Keniston

[-- Attachment #2: port_stap_uprobes.txt --]
[-- Type: text/plain, Size: 11104 bytes --]

0. TERMINOLOGY

myarch: The architecture you're porting to.

regs->ip: On myarch, the member of the pt_regs structure that contains the
instruction pointer.

1. BASIC PORT
=============
The basic port gives you uprobes, using single-stepping inline (SSIL).
To add support for return probes (uretprobes), see "PORTING URETPROBES".
To add single-stepping out of line (SSOL), see "PORTING SSOL".  SSOL
ensures that no probepoints are missed even in multithreaded apps.
It also typically yields better performance.

On some architectures, emulating some or all instructions may be
preferable to using SSIL or SSOL.  For example, emulation avoids
the cost of the second (single-step) trap for each probepoint hit.
See "EMULATING INSTRUCTIONS".

runtime/uprobes2/uprobes_myarch.h
---------------------------------
Create this file, defining the following:

typedef ___ uprobe_opcode_t;
An integer type that is the appropriate size for holding myarch's
breakpoint instruction -- e.g., u8 for i386's 1-byte int3 instruction.

#define BREAKPOINT_INSTRUCTION 0x___
myarch's breakpoint instruction -- e.g., 0xcc for i386's int3
instruction.

#define BP_INSN_SIZE ___
sizeof(uprobe_opcode_t) -- e.g., 1 for i386's int3

#define MAX_UINSN_BYTES ___
The number of bytes in the longest possible (probeable) instruction
in myarch's instruction set.  Round up, if necessary, to an integral
multiple of BP_INSN_SIZE.  E.g., 16 for i386 and x86_64, where the
longest possible instruction is actually 15 bytes.

Note: You do NOT need to define the SLOT_IP macro, which is defined
in some other architectures' headers.

#define BREAKPOINT_SIGNAL ___
The signal generated when myarch's breakpoint instruction is
executed.  Typically SIGTRAP.

#define SSTEP_SIGNAL ___
The signal generated when an instruction is single-stepped.  Typically
SIGTRAP.

#define ARCH_BP_INST_PTR(inst_ptr) ___
A breakpoint has just been hit.  inst_ptr is the (unsigned long)
current value of the instruction pointer.  Return the (unsigned
long) address of the probepoint that was hit.  On i386 and x86_64,
the instruction pointer is AFTER the breakpoint instruction at this
point, so ARCH_BP_INST_PTR returns inst_ptr-BP_INSN_SIZE.  On other
architectures, the instruction pointer is AT the breakpoint, so
ARCH_BP_INST_PTR return inst_ptr.

[typically static inline]
unsigned long arch_get_probept(struct pt_regs *regs);
A breakpoint has just been hit.  regs contains the saved registers.
Return the address of the probepoint.  This function typically returns
the equivalent of ARCH_BP_INST_PTR(regs->ip).

[typically static inline]
int uprobe_emulate_insn(struct pt_regs *regs, struct uprobe_probept *ppt);
If you don't emulate any instructions, this function just returns 0.
Otherwise, see "EMULATING INSTRUCTIONS."

[typically static inline]
void arch_reset_ip_for_sstep(struct pt_regs *regs);
A breakpoint has just been hit.  regs contains the saved registers.
Make sure regs->ip points to the probepoint.  This function typically
does the equivalent of
	regs->ip = ARCH_BP_INST_PTR(regs->ip);
which is a nop on architectures where the breakpoint instruction leaves
the instruction pointer at the breakpoint.

struct uprobe_probept_arch_info { ... };
myarch-specific data for a probepoint.  Can be used, e.g., to remember
info about how the probed instruction should be single-stepped (see
"PORTING SSOL").  Typically empty.

struct uprobe_task_arch_info { ... };
myarch-specific data for a probed task.  Can be used, e.g., to pass
data from uprobe_pre_ssout() to uprobe_post_ssout() during probepoint
processing (see "PORTING SSOL").  Typically empty.

[extern or static]
int arch_validate_probed_insn(struct uprobe_probept *ppt,
					struct task_struct *tsk);
ppt->insn[] contains a copy of the instruction to be probed.
ppt->vaddr contains its address.  arch_validate_probed_insn()
returns 0 if you support probing that instruction at that address,
or a negative errno (e.g., -EINVAL or -EPERM) otherwise.  The caller
will reject requests to probe BREAKPOINT_INSTRUCTION, or to probe
addresses outside executable VM areas (see uprobe_validate_vaddr());
for these cases, arch_validate_probed_insn() won't be called.

arch_validate_probed_insn() is your opportunity to collect and
remember (e.g., in ppt->arch_info) information about the probed
instruction (see "PORTING SSOL" and "EMULATING INSTRUCTIONS").
Note that in at least one case (x86_64 rip-relative instructions),
the instruction actually single-stepped is a modified version of the
probed instruction.  For such cases, you actually modify ppt->insn[]
in arch_validate_probed_insn().  (If you need to remember the original
instruction, save a copy in ppt->arch_info.)

runtime/uprobes2/uprobes_myarch.c
---------------------------------
Create this file, implementing any functions that you declared extern
in include/asm-myarch/uprobes.h.  Be sure to do
#define UPROBES_IMPLEMENTATION 1
before
#include <linux/uprobes.h>
On some architectures, everything mentioned so far can be implemented
as static inlines in include/asm-myarch/uprobes.h, so this .c file
may be essentially empty (which is OK).  But the build scheme requires
this file, and you'll definitely need it when you port SSOL.

runtime/uprobes2/uprobes_arch.c
runtime/uprobes2/uprobes_arch.h
-------------------------------
Modifiy these files to #include your uprobes_myarch.h and uprobes_myarch.c
files.

2. PORTING URETPROBES
=====================
This step adds myarch support for register_uretprobe() and
unregister_uretprobe().  This should be pretty easy.

runtime/uprobes2/uprobes_myarch.h
---------------------------------
Add the following:

#define CONFIG_URETPROBES 1

[extern or static]
unsigned long arch_hijack_uret_addr(unsigned long trampoline_addr,
		struct pt_regs *regs, struct uprobe_task *utask);
We have just hit the probepoint at the entry to a uretprobed
function.  Remember the real* return address, replace it with
trampoline_addr, and return the real* return address.

If for some reason you can't replace the return address, return 0.
If you somehow leave the return address in a corrupted state,
also set utask->doomed = 1.

*This is called for every uretprobe registered on the probed
function.  If there's more than one, only the first call to
arch_hijack_uret_addr() will return the real return address; the rest
will return trampoline_addr.  That's the desired behavior.

[typically static inline]
void arch_restore_uret_addr(unsigned long ret_addr, struct pt_regs *regs);
Called after the uretprobed function executes its return and all
associated uretprobe handlers have been run.  Set regs->ip = ret_addr.

[typically static inline]
unsigned long arch_get_cur_sp(struct pt_regs *regs);
Returns the current value of the user-mode stack pointer, as obtained from regs.

[typically static inline]
unsigned long arch_predict_sp_at_ret(struct pt_regs *regs,
						struct task_struct *tsk);
Called right after arch_hijack_uret_addr() (see above) -- i.e., right
after the probed function has been called.  Returns the expected value
of the user-mode stack pointer after this function's return instruction
has been executed.  If your architecture passes the return address
in a register, then this function typically returns the current value
of the stack pointer.  On the other hand, if your architecture passes
the return address on the stack, then this function typically returns
the current value of the stack pointer plus the size of the return
address (to reflect the fact that the return address is on the stack
when the function is called, but is popped off as it returns).

runtime/uprobes2/uprobes_myarch.c
------------------------------------
Add any of the above-described functions that aren't implemented in
runtime/uprobes2/uprobes_myarch.h.

3. PORTING SSOL
This step adds support for single-stepping out of line (SSOL).
This step requires pretty extensive knowledge of myarch's
instruction set.  Fortunately, if there's a kprobes port for
myarch, most of the thinking has already been done for you.

As previously mentioned, SSOL ensures that no probepoints are missed
even in multithreaded apps, and typically yields better performance.
The basic idea is that we need to leave the breakpoint instruction
in place at all times (to avoid probepoint misses), and so must
single-step a copy of the probed instruction.  Uprobes puts the
instruction-copy in one of a set of "instruction slots" allocated
from a special VM area.

runtime/uprobes2/uprobes_myarch.h
---------------------------------
Add the following line:
#define CONFIG_UPROBES_SSOL 1

runtime/uprobes2/uprobes_myarch.c
------------------------------------
Add the following functions:

void uprobe_pre_ssout(struct uprobe_task *utask,
		struct uprobe_probept *ppt, struct pt_regs *regs);
Called when the indicated task is about to single-step the instruction
at the indicated probepoint.  Call uprobe_get_insn_slot() to ensure
that there's an instruction slot in the SSOL vma reserved for this
probepoint, and that the slot contains the instruction-copy to be
single-stepped.  If uprobe_get_insn_slot() returns NULL, it means
uprobes couldn't populate the instruction slot; just set utask->doomed
= 1.  Otherwise set regs->ip and utask->singlestep_addr to the address
of the instruction slot.  Perform any myarch-specific pre-single-step
work (typically none; x86_64 is an exception).

void uprobe_post_ssout(struct uprobe_task *utask,
		struct uprobe_probept *ppt, struct pt_regs *regs);
Called after the instruction copy has been single-stepped.
Call up_read(&ppt->slot->rwsem) to release the instruction slot.
Perform any myarch-specific fixups required due to the fact that
we single-stepped the instruction copy at utask->singlestep_addr
rather than the original instruction at ppt->vaddr.

For most instructions, this just means adjusting regs->ip so that it
points back to the next instruction in the probed instruction stream.
Typical exceptions are return instructions and absolute or indirect
call/jump instructions, for which no regs->ip adjustment is necessary.
Also for call instructions, you typically need to adjust the return
address.

If myarch has a kprobes port, you can model uprobe_post_ssout()
after resume_execution() in arch/myarch/kernel/kprobes.c.

4. EMULATING INSTRUCTIONS
If you emulate one or more instructions, then your uprobe_emulate_insn()
function must do more than just return 0.

int uprobe_emulate_insn(struct pt_regs *regs, struct uprobe_probept *ppt);
Called after a probepoint has been hit and the associated handlers have
been run.  If you're not emulating the instruction specified by ppt,
simply return 0.  Otherwise, this function must:

- Perform the action of the probed instruction.  Keep in mind that the
instruction ordinarily runs in user space, so you must appropriately
handle stray memory references, illegal and privileged instructions,
malevolent code, etc.

- Point regs->ip at the next instruction to be executed.

- Return 1.

Instructions that can be neither emulated nor single-stepped should
be rejected by arch_validate_probed_insn() (see above).

Re: documentation for user-space usage?

[Roland McGrath]

> Perhaps we need a "--rootfs" option? This could give a local path to
> find binaries that will be located on the root of the target.  i.e. for
> translation we'd look at "$rootfs/usr/bin/foo", but at runtime
> task-finder would match simply "/usr/bin/foo".

This is the analogue of the standard "sysroot" feature in other tools.