[Bug runtime/2035] investigate boot-time probing

[Bug runtime/2035] investigate boot-time probing

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2035

Jonathan Lebon <jlebon at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jlebon at redhat dot com

--- Comment #6 from Jonathan Lebon <jlebon at redhat dot com> ---
I've started looking into this. Thankfully dracut helps out a lot here. It
provides us a straightforward method of hooking scripts into the various stages
of the bootup (see dracut.bootup(7)).

After some work and research, I successfully inserted a module in the initramfs
that gets loaded as early as possible.

I'll just go over the steps of what I did:

First, we need to create a dracut module. This is no more than a directory and
some shell scripts that will get pulled in whenever dracut is called to create
a new image.

Create the directory e.g. '01stap' under '/lib/dracut/modules.d'. There, we
have two simple scripts. The first script is module-setup.sh, which contains
functions dracut will call during image creation.

$ cat module-setup.sh
#!/bin/bash

check() {
    return 0
}

depends() {
    echo ""
}

install() {
    inst_hook cmdline 01 "$moddir/start-staprun.sh"
}
$

The 'inst_hook' line is telling dracut that we want the 'start-staprun.sh'
script to be executed at the cmdline hook, which is the earliest one. In
'start-staprun.sh' we have:

$ cat start-staprun.sh
. /lib/dracut-lib.sh
/home/vm/codebase/systemtap/install/bin/staprun -L module_watcher
$

The kernel module 'module_watcher' is a simple stap test script I wrote to
showcase what kind of things we could do if running at boot. It watches for
module load/unload events. Here is the script:

$ cat module_watcher.stp
global args

probe begin {
   println("0.000000 - Started module_watcher")
   start_stopwatch("timer")
}

probe syscall.init_module {
   args = uargs
}

probe kernel.function("do_init_module") {
   timer = read_stopwatch_us("timer")
   printf("%d.%.6d - ", timer/1000000, timer%1000000)
   printf("Loading module %s", kernel_string($mod->name))
   if (args != "")
      printf(" with args %s ", args)
   println("")
}

probe syscall.delete_module {
   timer = read_stopwatch_us("timer")
   printf("%d.%.6d - ", timer/1000000, timer%1000000)
   printf("Unloading module %s with flags %x\n", user_string($name_user),
flags);
}

probe end {
   println("Exiting module_watcher")
}
$

I compiled the script and placed it in /lib/modules/`uname -r`/systemtap. I
also ran depmod after.

All that's left to do is to create a new image in which the compiled SystemTap
kernel module we want, staprun and stapio are included:

# dracut --force --install '/home/vm/codebase/systemtap/install/bin/staprun
/home/vm/codebase/systemtap/install/libexec/systemtap/stapio' --add-drivers
module_watcher
# 

I think we could forgo '--add-drivers module_watcher' if we instead specify it
in the install() function of our module-setup.sh script. Haven't played with
that yet.

Reboot, open a terminal, and reconnect to the module (note we need to do this
as root because the module belongs to root, otherwise staprun won't be happy):

# /home/vm/codebase/systemtap/install/bin/staprun -A module_watcher
0.000000 - Started module_watcher
0.102458 - Loading module pata_acpi
0.103298 - Loading module ata_generic
0.108215 - Loading module i2c_core
0.108700 - Loading module virtio_net
0.112446 - Loading module drm
0.139114 - Loading module virtio_blk
0.139537 - Loading module ttm
0.142624 - Loading module vmwgfx
1.205084 - Loading module uinput
1.271417 - Loading module mperf
1.273269 - Loading module acpi_cpufreq
1.291843 - Loading module i2c_piix4
1.298882 - Loading module microcode
1.304980 - Loading module virtio_balloon
1.307476 - Loading module ghash_clmulni_intel
1.315539 - Loading module serio_raw
1.326341 - Loading module crc32c_intel
1.335179 - Loading module crc32_pclmul
1.967812 - Loading module iptable_raw
1.979030 - Loading module iptable_security
1.990345 - Loading module iptable_mangle
1.994897 - Loading module nf_conntrack
1.996680 - Loading module nf_nat
1.997620 - Loading module nf_nat_ipv4
1.999513 - Loading module nf_defrag_ipv4
2.000688 - Loading module nf_conntrack_ipv4
2.001808 - Loading module iptable_nat
2.006181 - Loading module ip6_tables
2.013440 - Loading module ip6table_filter
2.053304 - Loading module ip6table_raw
2.065209 - Loading module ip6table_security
2.080719 - Loading module ip6table_mangle
2.119615 - Loading module nf_nat_ipv6
2.124040 - Loading module nf_defrag_ipv6
2.125342 - Loading module nf_conntrack_ipv6
2.126901 - Loading module ip6table_nat
2.193750 - Loading module ebtables
2.199242 - Loading module ebtable_filter
2.207249 - Loading module llc
2.208277 - Loading module stp
2.212893 - Loading module bridge
2.215973 - Loading module ebtable_broute
2.225101 - Loading module ebtable_nat
2.481361 - Loading module rfkill
2.487218 - Loading module bluetooth
2.491427 - Loading module bnep
2.500985 - Loading module xt_conntrack
2.606265 - Loading module ip6t_REJECT
2.786338 - Loading module ipt_MASQUERADE
2.831228 - Loading module nf_conntrack_broadcast
2.832305 - Loading module nf_conntrack_netbios_ns
^CExiting module_watcher
#

And voila!

Now, the goal is to streamline this process and skip the parts that can be
skipped.

One issue which comes to mind right away is the permission issue, since the
module gets inserted as real root. But then again, if they are privileged
enough to be allowed to have a module inserted at bootup time, maybe we don't
have to change anything.

For the interface, I was thinking of introducing a 'list' paradigm. E.g. we
could have the following switches for stap:
--boot-add --> add script to list of boot scripts
--boot-list --> list all scripts currently enabled on boot
--boot-remove --> remove script from list of boot scripts
--boot-once --> add script to list just for the next boot

Of course in the background, we would be rerunning dracut whenever necessary.
This can be a time-consuming process (in my VM, it can take up to 30s).

Not sure yet how much that makes sense. I guess it depends on how people will
use the feature.

Also, do we want to support platforms pre-dracut (e.g. RHEL5)? I haven't played
with mkinitrd, although I do see that it has a --preload=MODULE option which
might do the trick.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/2035] investigate boot-time probing

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2035

--- Comment #7 from Jonathan Lebon <jlebon at redhat dot com> ---
Created attachment 7312
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7312&action=edit
systemtap-initscript documentation patch for boot-time probing

This is how I'm thinking of changing the interface of the initscript.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/2035] investigate boot-time probing

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2035

--- Comment #8 from Jonathan Lebon <jlebon at redhat dot com> ---
I've just created a branch (jlebon/boot-time) containing the work so far. The
easiest way to test it is to 'make rpm' and install the rpms. However if you'd
like to do it manually, you will need to make and
- Copy initscript/99stap/{module-setup.sh,start-staprun.sh} into
/lib/dracut/modules.d/99stap (remove {prefix})
- Copy initscript/systemtap to /etc/init.d/ (also remove {prefix})

Once installed, you can test it by doing the following (also see section 5.9 of
initscript/README.systemtap):

# cat > /etc/systemtap/script.d/hello.stp
probe begin {
   println("Hello World!")
}
# service systemtap onboot hello
# reboot
---
# staprun -A hello

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/2035] investigate boot-time probing

[jlebon at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2035

Jonathan Lebon <jlebon at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from Jonathan Lebon <jlebon at redhat dot com> ---
Now in master (commit 527e696 and previous ones).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/2035] investigate boot-time probing

[fche at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|hunt at redhat dot com      |systemtap at sources dot
                   |                            |redhat dot com
             Status|NEW                         |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=2035

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.