Re: [Ksummit-2008-discuss] DTrace

[Theodore Tso <tytso@mit.edu>]

On Tue, Jul 01, 2008 at 01:06:32PM -0700, Roland McGrath wrote:
> Like I said, the essential command is eu-strip -f.  It is simple to use.
> 
> For one's own local hacking purposes, there is no real reason to bother
> with strip-to-file complexities.  You can just copy the unstripped files
> before stripping them.  The effect is the same (or easier for you, with
> most of the tools), and the extra 6M (stripped) where you have disk space
> for the 60M (debuginfo) is never an issue (i.e. if it's 66M unstripped).

Well, actually, it *does* matter, at least to me.  Sometimes when I am
trying to track down a problem, either using git-bisect or evaluating
multiple patches, I might have five, six, seven, eight or more kernels
installed.  And on a number of my systems, the amount of space on the
parititons where /boot and /lib live can't take the space demands of
compiling the kernel and modules with -g.

And remember, for the average kernel developer, the question is
whether using SystemTap is easier than inserting a bunch of printk's
and recompiling.  And one of the major flaws of the Linux's RAS tools
is that the LKML development community doesn't use them; and to the
extent that tapsets would be written more quickly if they are easy for
kernel developers who aren't depending on distro packaging and distro
building of systemtap.  (Especially if systemtap is so fast moving
that people shouldn't depend on stable releases but rather the git
repository or weekly automated snapshots.)  So actually, being able to
install stripped modules and vmlinux into /boot and /linux, and then
being able to put the unstripped binaries somewhere else, without
having to use the !@#@! complicated RPM macros by Fedora/RHEL is
actually **very** important to me.

I don't know how many people considered that a showstopper; but James
mentioned on another thread that figuring out the magic, undocumanted
--enable-staticdw flag hit him as well.  Yes, I know that's been fixed
as of last Friday in the git repository, but again, it's these little
things that cause people to throw up their hands in frustration and
say, "Eh!  I'll just use printk's and recompilations instead; it's
easier."

In the past two years, I've on average tried Systemtap every 9 months
or so, and each time, I'd hit a different annoying roadblock, and then
I was so busy I would move on to a more productive way of solving my
problems.  And I've asked various different Systemtap developers and
architects (mostly inside IBM), and I'd get the same answers that
Ulrich spouted just recently on this list.  "Tapsets?  Yeah, we're
depending on kernel subsystem experts to write them; we don't know how
to get inside the internals of the various subsystems."  "Building it?
Stable releases?  That's a distro problem; just use what your distro
uses."  "Ooooh, sorry, that's an ancient version of Systemtap, blame
your distro provider for doing a sucky job."  And my reaction each
time was, "OK, back to printk debugging; and if you want me to write
tapsets for you, you're in fantasy land."

So I think this issue is very much a potential topic for the kernel
summit, namely --- why is it that so few kernel developers are using
RAS tools like Systemtap, and what can be done to improve this
situation?  Or if the Systemtap team doesn't need any help, and can
write all of these tapsets without kernel developer's participation,
or maybe assume that System administrators can write Systemtap scripts
that do things like:

probe kernel.function ("vfs_write"),
      kernel.function ("vfs_read")
{
  dev_nr = $file->f_dentry->d_inode->i_sb->s_dev
  inode_nr = $file->f_dentry->d_inode->i_ino

  if (dev_nr == ($1 << 20 | $2) # major/minor device
      && inode_nr == $3)
    printf ("%s(%d) %s 0x%x/%u\n",
      execname(), pid(), probefunc(), dev_nr, inode_nr)
}

and still be a credible competition to the audience served by DTrace,
hey, knock yourself out.  But I think there may be a connection
between problems which Systemtap developers seem to continually assert
a Sombody Else's Problem field around, and the lack of uptake by the
LKML community.  Maybe.  Just a guess on my part.

						- Ted