public inbox for systemtap@sourceware.org
 help / color / mirror / Atom feed
* precompiled probing scenarios
@ 2006-10-06 19:08 Frank Ch. Eigler
  2006-10-06 20:33 ` David Smith
  0 siblings, 1 reply; 19+ messages in thread
From: Frank Ch. Eigler @ 2006-10-06 19:08 UTC (permalink / raw)
  To: systemtap

Hi -

Here are some ideas about pre-compiled probes.  It all tries to bring
together some concepts we've kicked around about caching,
pre-compilation, remote deployment, run-time probe parametrization,
and the stap/staprun split.

One part of the plan would be a cache of compiled probe modules.
Unless disabled, the translator would compute a hash of the probe
script and environmental parameters (e.g.  sess.kernel_release,
architecture, invoking userid).  This hash value would be used to
identify the module in a persistent cache
($HOME/.systemtap/cache/HASH.ko just like ccache), so that the same
script for the same machine would map to the same module name.  (A
person deliberately running the same script twice concurrently would
have to "salt" the cache/hash for the duplicates.)  A "stap -p4" run
would print the cached module's file name.

To run a compiled probe, one would normally use "staprun HASH.ko".
Since we already support parametrization (extra module parameters can
initialize string/number global script variables), staprun should be
extended to pass those on to insmod.  (By the way, as we discussed,
staprun will be extended to send a variant of /proc/modules to the
module, so it can relocate its module probes.)

Cross-compilation could come in by letting users specify a target name
for probing.  This name would be mapped to a kernel version,
architecture, cpu type, and maybe a build-root, all via a
configuration file $HOME/.systemtap/known_hosts.  (A host may have
multiple target names, for example for different kernel versions.)
The translator might update its own host's details to that file
automatically, so a network-wide file could be grown by running the
translator once on each client machine and concatenating the files.

Once such a cross-compiled module is built, it could be saved in the
local cache (-p4), and for -p5 even shipped to a named remote machine
via scp and executed there via ssh/staprun.  (Probing several remote
machines concurrently could be easily accomodated.)

We've mentioned somehow securely identifying of compiled modules to
represent a special permission to execute.  This would be a way of
having a security expert dude formally designate a module for use on a
locked-down deployment machine.  Given that the modsign code in
FC/RHEL is not widespread or general enough, a proper kernel-enforced
crypto signature may be out of reach.  Maybe we can list (say) md5sums
of approved module .ko's in a /etc/systemtap/authorized_probes file,
and have a new staprun.auth variant that checks it before submitting a
module to insmod(8) (or actually better, to sys_init_module(2)
directly).

So on to some examples:

% stap -p4 -e "probe foo { ... }"
/home/fche/.systemtap/cache/0xdeadbeef.ko
% stap -p4 -e "probe foo { ... }"
/home/fche/.systemtap/cache/0xdeadbeef.ko  # instant: already cached
% staprun /home/fche/.systemtap/cache/0xdeadbeef.ko global1=value global2=value
(sudo password if needed)
(probe output)
^C

# grep TARGET $HOME/.systemtap/known_hosts
TARGET execute=ssh:user@host.name:sudo kernel=2.6.18-78234.327 arch=i686 cpu=p4
% stap -T TARGET -p4 -e "probe foo { ... }"
/home/fche/.systemtap/cache/0xfeedface.ko  # precompile
% stap -T TARGET -e "probe foo { ... }"
(scp 0xfeedface.ko to host.name:/tmp)
(ssh host.name staprun /tmp/0xfeedface.ko)
(sudo password if needed)
(probe output)

# grep TARGET2 $HOME/.systemtap/known_hosts
TARGET2 execute=ssh:user@host.name:auth kernel=2.6.18-78234.327 arch=i686 cpu=p4
% md5sum /home/fche/.systemtap/cache/0xfeedface.ko
982734982739487239487234
% ssh root@host.name echo md5:982734982739487239487234 >> /etc/systemtap/authorized_probes  # bless this module
% stap -T TARGET2 -e "probe foo { ... }" -x CMD
(scp 0xfeedface.ko to user@host.name:/tmp)
(ssh user@host.name staprun.auth /tmp/0xfeedface.ko -x CMD)
(no sudo password needed!)
(CMD forked under real-uid privileges; module loaded under setuid)
(probe output)


Is this plausible enough to build?
 

- FChE

^ permalink raw reply	[flat|nested] 19+ messages in thread
* RE: precompiled probing scenarios
@ 2006-10-19 20:33 Stone, Joshua I
  2006-10-19 20:41 ` David Smith
  0 siblings, 1 reply; 19+ messages in thread
From: Stone, Joshua I @ 2006-10-19 20:33 UTC (permalink / raw)
  To: David Smith, Frank Ch. Eigler; +Cc: systemtap

On Thursday, October 19, 2006 12:50 PM, David Smith wrote:
> Frank Ch. Eigler wrote:
>> kernel.function("*") should match exactly what was there before.
>> Probes on module("*").FOO would be redefined to mean something like
>> "all modules that we know at translation time that *might* exist,
>> that also happen to be *loaded* at run time.  This aspect of wildcard
>> expansion would thus take place at run time rather than translate
>> time.  It just so happens that the same module might probe a greater
>> or lesser number of modules on an actual system.  With that proviso,
>> a script-source-level hash still seems to work.
>[...]
> The hash is computed using the following data:
>[...]
> - pass 2 script output
>[...]
> Stuff left to do:
> 
> - Testing, testing, testing
> - Correct module handling (which Frank outlined above)

Actually, correct module handling should already be covered if you're
hashing the pass-2 output.  Pass-2 is elaboration, which uses debuginfo
to locate the actual probe points.  Suppose I run a script like this:

	probe module("*").function("*interrupt*") { log(probefunc()) }

Pass-2 output will include all of the probe points matching my wildcard:

  ...
  module("ahci").function("ahci_interrupt@drivers/scsi/ahci.c:889"),
 
module("libata").function("ata_interrupt@drivers/scsi/libata-core.c:4198
"),
  ...

If I then load a new module that also has a "*interrupt*" function, the
pass-2 output will include the new probe point, and will thus get a
different hash.

In my estimation, there are two lengthy tasks in script compilation:
pass-2 elaboration (digging through lots of debuginfo) and pass-4
C-compilation.  This caching mechanism removes the pain of pass-4, which
is probably the worse of the two.  But if you find a caching scheme to
also avoid pass-2, then the module handling will need to be considered.


Josh

^ permalink raw reply	[flat|nested] 19+ messages in thread
* RE: precompiled probing scenarios
@ 2006-10-20 18:44 Stone, Joshua I
  2006-10-20 19:26 ` David Smith
  0 siblings, 1 reply; 19+ messages in thread
From: Stone, Joshua I @ 2006-10-20 18:44 UTC (permalink / raw)
  To: David Smith, Frank Ch. Eigler; +Cc: systemtap

On Friday, October 20, 2006 6:50 AM, David Smith wrote:
> Frank Ch. Eigler wrote:
>>> Note that currently several tests in the testsuite fail after a
>>> first run to seed the cache because they don't expect to see the
>>> skip from pass 2 to pass 5.
>> 
>> How do you mean they fail?  -p3 or -p4 should still work.
> 
> Here's what goes on.  The '-p3' and '-p4' options still work.  But,
> several run ('-p5') tests use testsuite/lib/stap_run.exp or
> testsuite/lib/stap_run2.exp.  Those two tcl files expect to see "Pass
> [12345]" in the output.  They get confused when only seeing "Pass
> [125]" and then think the test has timed out.

Would it make sense to print "dummy" pass 3/4 messages when a cached
version is used?  Something like:

Pass 1: parsed user script and 53 library script(s) in
310usr/0sys/326real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 global(s) in
10usr/0sys/5real ms.
Pass 3: (cached) in 0usr/0sys/0real ms.
Pass 4: (cached) in 0usr/0sys/0real ms.
Pass 5: starting run.

The timing info doesn't need to be hardcoded zero, I just expect it
would be very small.

Side question - do you still use caching when someone calls '-p3' or
'-p4'?  And with verbosity increased, what would this output, given that
you're not actually doing the work?  (e.g., you wouldn't have a compiler
output on a cached pass-4.)


Josh

^ permalink raw reply	[flat|nested] 19+ messages in thread
* RE: precompiled probing scenarios
@ 2006-10-20 20:51 Stone, Joshua I
  0 siblings, 0 replies; 19+ messages in thread
From: Stone, Joshua I @ 2006-10-20 20:51 UTC (permalink / raw)
  To: Frank Ch. Eigler, David Smith; +Cc: systemtap

On Friday, October 20, 2006 1:13 PM, Frank Ch. Eigler wrote:
> Hi -
> 
> dsmith wrote:
>> [...]
>> I don't hash the getuid(), since:
>> - by default your cache is stored in your home directory
>> - who you are doesn't change the pass 2-4 output
> 
> That's all true.  One possible reason for including getuid() anyway is
> so that two different sudo-empowered people can run the same script at
> the same time without one having to disable his cache.

Also, uid isn't always in sync with $HOME.  Besides people doing weird
things tweaking it, a simple example is that sudo may or may not change
$HOME to ~root.

Josh

^ permalink raw reply	[flat|nested] 19+ messages in thread
* RE: precompiled probing scenarios
@ 2006-10-24  0:29 Stone, Joshua I
  2006-10-24 15:16 ` David Smith
  0 siblings, 1 reply; 19+ messages in thread
From: Stone, Joshua I @ 2006-10-24  0:29 UTC (permalink / raw)
  To: David Smith; +Cc: systemtap

I saw that you checked in the caching code, so I finally got around to
trying it.  :)

For the most part, it seems to work really nicely.  The caching is
essentially transparent, which makes for a positive experience when your
scripts startup faster.

There's a few trials I did though where caching opportunities were
missed.  I'll admit freely that these are perhaps too nitpicky, so we
can treat it as a low-priority enhancement.

1. probe begin { exit() }
2. probe begin { exit(); }
3. probe begin { exit() a=1 }

2 and 3 actually hash the same, since elision turns 'a=1' into an empty
statement (';').  We should to be able to tell that these are all the
same, but since the pass-2 output leaves in all semi-colons, the hash is
different.  It ought to be pretty easy to normalize empty statements
away, so minor differences like this don't matter.

A harder scenario to address is this:

4. probe begin, end { exit() }
5. probe end, begin { exit() }

Again, with some fancy normalization, we should be able to identify
these as equal.  And actually, the seemingly-unrelated work in
probe-grouping would probably help here, if the pass-2 output were
ordered in a deterministic manner.  Stap already does some of this,
e.g., by ordering functions before probes.

It's likely rare that the differences between scripts will be so small,
so these optimizations may not matter.  But if anyone's bored, or has an
intern with nothing to do, this may be a simple enhancement.


Josh

^ permalink raw reply	[flat|nested] 19+ messages in thread
* RE: precompiled probing scenarios
@ 2006-10-25 18:54 Stone, Joshua I
  2006-10-26  1:07 ` Frank Ch. Eigler
  0 siblings, 1 reply; 19+ messages in thread
From: Stone, Joshua I @ 2006-10-25 18:54 UTC (permalink / raw)
  To: David Smith; +Cc: systemtap

On Tuesday, October 24, 2006 8:17 AM, David Smith wrote:
> Stone, Joshua I wrote:
>> 1. probe begin { exit() }
>> 2. probe begin { exit(); }
>> 
>> 4. probe begin, end { exit() }
>> 5. probe end, begin { exit() }
> 
> Hmm.  Just for fun, I decided to see if the pass 3 output of [1. 2.]
> or [4. 5.] would compare equally.  They don't.

That's partly my point.  Those pairings are functionally equivalent,
right?  So why should the code we generate show any differences?

The difference between 1 & 2 is basically just the line "/* null */;" in
the probe's generated C -- a useless statement.  Between 4 & 5 the only
difference is whether the begin or end is generated first.  Because
these differences are insignificant, we should be able to treat them the
same for caching purposes.

Another normalization example is with braces:

6. probe begin { if(foo) exit() }
7. probe begin { if(foo) { exit() } }
8. probe begin { if(foo) { { exit() } } }

My hope is that someday the translator will also treat less obvious
cases like these as identical:

9.  probe begin { log("foo") }
10. probe begin { if(1) log("foo") }
11. probe begin { i=1; if(i) log("foo") }
12. probe begin { while(!i++) log("foo") }

This has more to do with optimization of the generated code, but it
could help caching if the optimization is done before pass-2 output.


Josh

^ permalink raw reply	[flat|nested] 19+ messages in thread
end of thread, other threads:[~2006-10-26  1:07 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-10-06 19:08 precompiled probing scenarios Frank Ch. Eigler
2006-10-06 20:33 ` David Smith
2006-10-06 20:40   ` Frank Ch. Eigler
2006-10-19 19:49     ` David Smith
2006-10-19 21:53       ` Frank Ch. Eigler
2006-10-20 13:50         ` David Smith
2006-10-19 20:33 Stone, Joshua I
2006-10-19 20:41 ` David Smith
2006-10-20 18:44 Stone, Joshua I
2006-10-20 19:26 ` David Smith
2006-10-20 19:32   ` Frank Ch. Eigler
2006-10-20 19:50     ` David Smith
2006-10-20 20:13       ` Frank Ch. Eigler
2006-10-23 20:36         ` David Smith
2006-10-20 20:51 Stone, Joshua I
2006-10-24  0:29 Stone, Joshua I
2006-10-24 15:16 ` David Smith
2006-10-25 18:54 Stone, Joshua I
2006-10-26  1:07 ` Frank Ch. Eigler

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).