Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso <tytso@mit.edu>]

On Thu, Jan 08, 2009 at 01:22:37AM -0800, Roland McGrath wrote:
> 
> Since 2.6.2[78], the whole of the new code is utrace proper, and all under
> the CONFIG_UTRACE option, which "depends on EXPERIMENTAL".  However broken
> utrace might be, you can configure it out and have my patches make zero
> code changes in your kernel.
> 
> I had hoped that this zero impact for the uninterested would make the
> threshold of acceptance for merging utrace reasonably attainable without
> much controversy or rigamarole.  However, in various recent conversations,
> several kernel folks said we should not submit utrace on its own at all.
> They told us a new in-kernel API can't go in without something that uses
> it, and "systemtap doesn't count".  They want us to come up with some
> completed kernel feature, small or large, that makes use of utrace to do
> something or other, that they like and want to merge.  Only then, they
> said, should we submit utrace patches, along with more patches adding this
> thing that uses the utrace API.
> 
> That seems counterproductive to me, frankly.  Since you seem to be
> encouraging me to just get utrace merged ASAP, I hope you agree with my
> view here.  Everyone who ever says they'd like to help writing nice, useful
> things based on the utrace API complains about utrace not being merged and
> that this discourages work based on it.  Now other people say that it
> mustn't be merged until people have done that work.  Catch 22.
> (Everybody's way out is for me to just write everything, apparently.  
> Silly other people want me to also work on other projects entirely at the
> same time, so this idea has not been getting things done quicker.)

Sigh.  So part of the problem with the reputation Systemtap has as a
project out of control which doesn't listen to the kernel development
community is that no one is willing to bend the rules.  Code needs
in-tree users; and out of tree moudles have never counted.  This is
true regardless of whether the out of tree module is something evil,
like Nvidia's proprietary drivers with memory pointer bugs that induce
random memory corruption in Linux kernels, or out of tree filesystems
like OpenAFS (even if they are open source), or SystemTap.

There can and have been exceptions, or talented kernel developers can
find easy ways to make the code useful for purposes.  The one piece of
advice I can give you is to ***ask*** someone like Ingo Molndar or
Steven Rostedt for their help in shepherding the utrace patch into the
kernel.  You needs someone who can vouch for the patch, and can make
the necessary changes and then work the politics to get the patch in
(probably as a replacement back-end for ptrace, and demonstrate that
it is cleaner/faster/better for long-term maintenance than the
existing ptrace code).

Look, let me be blunt.  Frank doesn't seem to believe me when I say
that Red Hat kernel engineers I've talked to are massively dismissive
about SystemTap; and apparently when a member of the LF Technical
Advisory Board talked to Brian Stevens about the perceived dysfunction
of the SystemTap team, nothing happened.  This fact, combined with the
observation that apparently Red Hat kernel developers are unwilling to
speak aloud problems with SystemTap internally within Red Hat, *and*
the fact that SystemTap hasn't been cancelled despite its many years
of toilage with very little results to show for it, leads me to
believe that SystemTap has massive political clout and masive
political support within Red Hat in support of the project.  

If that is true, I would suggest you *use* some of that massive amount
of political clout to get a senior Red Hat Kernel developer assigned
to the project to render assistance.  You need the help; seriously.
Otherwise, I very much doubt you'll be able to get kernel developers
to bend the rules that are in force for all other code submissions.

Things might have been different if SystemTap team had been more
willing to listen to kernel developers, and less resistant to
preferences of moving systemtap support into the kernel, maybe there
would be some positive karma that you could have drawn upon, and
gotten more help from various kernel developers.  But at this point,
anything SystemTap related starts with a huge negative bias against
it, especially when the general assumption from the kernel developers
is that if they ever hope to have something which is useful to them
and to others, they will need to do it themselves.

The work replacing markers with tracepoints, and the enhancements in
the ftrace framework, are all part of a general effort to provide a
general-purpose tracing facility so that SystemTap can be pushed
aside.  I'm not sure if anyone has a good story for userspace tracing
that doens't involve SystemTap, so I suspect that's going to be
SystemTap's last saving throw.

But what I would do, since you asked for my advice, is:

*) Reach out through Red Hat Management.  Tell them that several years
of investment in SystemTap is at risk.  Get a senior Red Hat
Kernel developer assigned to help out.

*) **Listen** to that developer's recommendations.  It will probably
involve getting the SystemTap kernel support code into the kernel, and
probably many other changes.  You may think it's bullshit, but don't
blow him off the way you've done with me or James' suggestions.  You
*need* his expertise to make the chnages acceptable to the standard
mainstream kernel acceptance criteria, and you *need* his good well so
he's willing to vouch for the code.  Hey, maybe my word is worthless,
and I don't know what the hell I'm talking about, but if someone like
Ingo Molnar tells you "You Need To Do This", maybe you'll pay
attention to him.

*) And do this quickly.  The longer you wait, the longer more people
will have time to work on alternatives to SystemTap, and the harder it
will be for people to believe that SystemTap is worth saving.

     	    	      	      	   	     	- Ted