precompiled probing scenarios

["Frank Ch. Eigler" <fche@elastic.org>]

Hi -

Here are some ideas about pre-compiled probes.  It all tries to bring
together some concepts we've kicked around about caching,
pre-compilation, remote deployment, run-time probe parametrization,
and the stap/staprun split.

One part of the plan would be a cache of compiled probe modules.
Unless disabled, the translator would compute a hash of the probe
script and environmental parameters (e.g.  sess.kernel_release,
architecture, invoking userid).  This hash value would be used to
identify the module in a persistent cache
($HOME/.systemtap/cache/HASH.ko just like ccache), so that the same
script for the same machine would map to the same module name.  (A
person deliberately running the same script twice concurrently would
have to "salt" the cache/hash for the duplicates.)  A "stap -p4" run
would print the cached module's file name.

To run a compiled probe, one would normally use "staprun HASH.ko".
Since we already support parametrization (extra module parameters can
initialize string/number global script variables), staprun should be
extended to pass those on to insmod.  (By the way, as we discussed,
staprun will be extended to send a variant of /proc/modules to the
module, so it can relocate its module probes.)

Cross-compilation could come in by letting users specify a target name
for probing.  This name would be mapped to a kernel version,
architecture, cpu type, and maybe a build-root, all via a
configuration file $HOME/.systemtap/known_hosts.  (A host may have
multiple target names, for example for different kernel versions.)
The translator might update its own host's details to that file
automatically, so a network-wide file could be grown by running the
translator once on each client machine and concatenating the files.

Once such a cross-compiled module is built, it could be saved in the
local cache (-p4), and for -p5 even shipped to a named remote machine
via scp and executed there via ssh/staprun.  (Probing several remote
machines concurrently could be easily accomodated.)

We've mentioned somehow securely identifying of compiled modules to
represent a special permission to execute.  This would be a way of
having a security expert dude formally designate a module for use on a
locked-down deployment machine.  Given that the modsign code in
FC/RHEL is not widespread or general enough, a proper kernel-enforced
crypto signature may be out of reach.  Maybe we can list (say) md5sums
of approved module .ko's in a /etc/systemtap/authorized_probes file,
and have a new staprun.auth variant that checks it before submitting a
module to insmod(8) (or actually better, to sys_init_module(2)
directly).

So on to some examples:

% stap -p4 -e "probe foo { ... }"
/home/fche/.systemtap/cache/0xdeadbeef.ko
% stap -p4 -e "probe foo { ... }"
/home/fche/.systemtap/cache/0xdeadbeef.ko  # instant: already cached
% staprun /home/fche/.systemtap/cache/0xdeadbeef.ko global1=value global2=value
(sudo password if needed)
(probe output)
^C

# grep TARGET $HOME/.systemtap/known_hosts
TARGET execute=ssh:user@host.name:sudo kernel=2.6.18-78234.327 arch=i686 cpu=p4
% stap -T TARGET -p4 -e "probe foo { ... }"
/home/fche/.systemtap/cache/0xfeedface.ko  # precompile
% stap -T TARGET -e "probe foo { ... }"
(scp 0xfeedface.ko to host.name:/tmp)
(ssh host.name staprun /tmp/0xfeedface.ko)
(sudo password if needed)
(probe output)

# grep TARGET2 $HOME/.systemtap/known_hosts
TARGET2 execute=ssh:user@host.name:auth kernel=2.6.18-78234.327 arch=i686 cpu=p4
% md5sum /home/fche/.systemtap/cache/0xfeedface.ko
982734982739487239487234
% ssh root@host.name echo md5:982734982739487239487234 >> /etc/systemtap/authorized_probes  # bless this module
% stap -T TARGET2 -e "probe foo { ... }" -x CMD
(scp 0xfeedface.ko to user@host.name:/tmp)
(ssh user@host.name staprun.auth /tmp/0xfeedface.ko -x CMD)
(no sudo password needed!)
(CMD forked under real-uid privileges; module loaded under setuid)
(probe output)


Is this plausible enough to build?
 

- FChE