Re: Discussion at Linux Foundation Japan Symposium

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi -

On Fri, Dec 19, 2008 at 03:50:58PM +0000, Richard J Moore wrote:

> [...]  I still think there's a case to make ST only weakly dependent
> on debuginfo.  The original dprobes would use the public names in
> the ELF headers to resolve symbols and only resort to debuginfo if
> needed.

It turns it's needed rather a lot. :-( Without it, we have no typing
information with which pointer/struct values can be intelligently
processed.  We also don't have location information to *find* data
such as parameters and locals.  It is also our current solution for
prologue analysis, so we know where to insert probes on plain
(non-regparms) functions compiled by current gcc (where location list
data can be simply wrong).

Some of this has been predicated on the belief that debuginfo quality
will improve enough that we do not need to resort to hard-coded
per-architecture per-compiler hacks like some in gdb.


> [...] Would could build some scripts with the binaries where debug
> info is available, interpret that info as relative offsets to public
> symbols then dispense with the info on the shipped production
> script.  See where I'm going here?

It sounds like an interesting sort of shortcut - one that we have not
seriously considered.  It may be far too optimistic, considering the
degree of configuration/optimization change possible between two
similar kernel builds, and the prevalent hostility toward reusable
kernel modules.

> And even for one-off scripts, this is still useful. I have on many
> occasions started with a dump, found the information I need
> extracting with a dynamic tracepoint and developed the script in
> terms relative to public symbols. No debug info needed..

Could you elaborate with an example?

- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Sat, Dec 20, 2008 at 07:38:31PM -0500, Frank Ch. Eigler wrote:
> 
> Some of this has been predicated on the belief that debuginfo quality
> will improve enough that we do not need to resort to hard-coded
> per-architecture per-compiler hacks like some in gdb.

Yes, but how long will people need to wait before debuginfo quality
will improve?  And when such a hypothetical improved compiler be
available and trusted to compile the kernel correctly?  (And how much
slower will it be compared to current versions of gcc's?   :-)

More importantly, what will systemtap development do in the meantime?
There are a couple of choices:

1) Do nothing.  Kernel developers will continue to largely give up on
Systemtap, mostly out of frustration.  Frustration that their needs
aren't being met; frustration that suggestions that they make are
blown off as not being important, or concerns that are in the
minority, or whatever.  The process of getting userspace probes in
will go slowly (if at all).  I haven't seen a new version of the
patches submitted in months --- and the 2.6.29 merge window will be
opening soon.  Whoever is managing getting the necessary userspace
probe patches into mainline seems to be totally ignorant about
mainline processes --- why hasn't a "this is what we intend to merge"
been sent to LKML already?   

2) Explain to developers how to use Systemtap in a painless way as
possible.  This means no modules, since debuginfo is a disaster with
modules (so helping people create configurations that don't use
modules would be helpful); and/or with a focus on markers as a
debugging mechanism, even if that means that you have to recompile the
kernel first before they can be used.  If making markers more helpful
is the goal, then work to make the Modules.marker file consistently
installed in the same place, or to make Systemtap look to find it in
multiple places, is going to be important.

3) Actually try to find a way for Systemtap to be more useful
*without* waiting for Godot^H^H^H^H^H^H some hypothetical future,
probably at least 18-24 months away, when new gcc and toolchain
becomes available that will magically and mystically Solve All
Problems.  Consider how ftrace works; it very cleverly uses/abuses the
profiling hook inserted gcc -pg.  Note that it doesn't wait for magic
new capabilities in the toolchain.  

In fact, given the utterly worthless (and vastly huge) debuginfo, the
only advantage SystemTap has over ftrace is (a) Systemtap has access
to the function parameters, and (b) Systemtap allows you to filter
logging information based on conditions, which is more efficient than
filtering in userspace.  Being able to probe individual lines usually
doesn't work, so most of the time what I find is the only thing I can
count on is add a tap at the beginning and end of the function ---
which I can do using ftrace.

> > [...] Would could build some scripts with the binaries where debug
> > info is available, interpret that info as relative offsets to public
> > symbols then dispense with the info on the shipped production
> > script.  See where I'm going here?
> 
> It sounds like an interesting sort of shortcut - one that we have not
> seriously considered.  It may be far too optimistic, considering the
> degree of configuration/optimization change possible between two
> similar kernel builds, and the prevalent hostility toward reusable
> kernel modules.

Another solution which I've suggested in the past, but which has been
utterly ignored, (since, after all, what does a Kernel Developer
know?) is to simply generate the debuginfo information on demand so
the function parameters on entry can be decoded.  We have the header
files after all, so it shouldn't be hard to generate a pseudo function:

extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
       void *ptr)
{
	return 0;
}

... and then use that to generate the debuginfo information for that
information so Systemtap knows how to decode the function parameters
on entry.  This would be enough such that Systemtap can probe any
public function in System.map --- and obtain the function parameters
for any function which is present in the kernel headers --- all
without requiring users to download or create hundreds of megabytes of
largely useless (on a value-per-megabyte-basis) !@#!@# debuginfo
information.

						- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi, Ted -


Thanks for commenting.


On Mon, Dec 22, 2008 at 01:19:21PM -0500, Theodore Tso wrote:
> > Some of this has been predicated on the belief that debuginfo quality
> > will improve enough that we do not need to resort to hard-coded
> > per-architecture per-compiler hacks like some in gdb.
> 
> Yes, but how long will people need to wait before debuginfo quality
> will improve?  And when such a hypothetical improved compiler be
> available and trusted to compile the kernel correctly?  [...]

You misinterpret the status quo.  In reality, debuginfo quality with
current gcc has been satisfactory for many purposes.  That is what
enables user-space debuggers to work as well as they do, and indeed
enables systemtap to work as well as it does.  The major current
debuginfo quality shortcoming involves only those intra-function
statement probes.


> More importantly, what will systemtap development do in the meantime?
> There are a couple of choices:

> 1) Do nothing.  Kernel developers will continue to largely give up on
> Systemtap, mostly out of frustration.  [...]

If you're still talking about debuginfo per se, I am led to believe
that only a small number of kernel developers are negatively affected:
those that build large kernel configurations but on such miniscule
hardware that the sheer disk I/O of the -g data is deemed intolerable.

If you're talking about other aspects, please elaborate.


> The process of getting userspace probes in will go slowly (if at
> all).  I haven't seen a new version of the patches submitted in
> months --- and the 2.6.29 merge window will be opening soon.  [...]

Roland?  Please advise.


> 2) Explain to developers how to use Systemtap in a painless way as
> possible.  

The http://sources.redhat.com/systemtap/wiki/SystemTapWithSelfBuiltKernel
page is becoming simpler and simpler.  Please try it.

> This means no modules, since debuginfo is a disaster with modules
> (so helping people create configurations that don't use modules
> would be helpful);

Please elaborate what you mean.  Why systemtap is forced to use
(create) modules at all has been explained often.


> and/or with a focus on markers as a debugging mechanism, even if
> that means that you have to recompile the kernel first before they
> can be used.  If making markers more helpful is the goal, then work
> to make the Modules.marker file consistently installed in the same
> place, or to make Systemtap look to find it in multiple places, is
> going to be important.

Sure, upstream changes in the location of Modules.marker are something
we can easily adapt to.  Are you aware of a current problem with that?
On the other hand, we plan to work on also connecting straight to
tracepoints, and to various slices of ftrace, for which a file such
the troubled-history Modules.markers will not be necessary.


> 3) Actually try to find a way for Systemtap to be more useful
> *without* waiting for [new gcc]

As I said above, the main thing we're expecting from a future gcc is
improvement on statement (arbitraty source line) probing.  Otherwise
quality is fine and we're not having to wait.  (As everyone knows,
quantity (compression) is not great but at least this is only a
usability rather than functionality matter.)

> Consider how ftrace works; it very cleverly uses/abuses the
> profiling hook inserted gcc -pg.

Yes, it's clever (though somewhat fragile).

> Note that it doesn't wait for magic new capabilities in the
> toolchain.

Nor do we.  In fact, we've been as unimposing as possible upon both
kernel and user-space build systems -- perhaps to a fault.

> [...] the only advantage SystemTap has over ftrace is (a) Systemtap
> has access to the function parameters, and (b) Systemtap allows you
> to filter logging information based on conditions, which is more
> efficient than filtering in userspace.

Beyond those, there is really plenty of difference in capability.
Line-by-line function tracing?  User-space?  Concurrent users?  The
list goes on...


> > > [...] Would could build some scripts with the binaries where debug
> > > info is available, interpret that info as relative offsets to public
> > > symbols then dispense with the info on the shipped production
> > > script.  See where I'm going here?
> > 
> > It sounds like an interesting sort of shortcut - one that we have not
> > seriously considered.  It may be far too optimistic, considering the
> > degree of configuration/optimization change possible between two
> > similar kernel builds, and the prevalent hostility toward reusable
> > kernel modules.
> 
> Another solution which I've suggested in the past, 

(I must have missed it.  This one's new to me.)

> [...] is to simply generate the debuginfo information on demand so
> the function parameters on entry can be decoded.  We have the header
> files after all, so it shouldn't be hard to generate a pseudo function:
> 
> extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
>        void *ptr)
> {
>         return 0;
> }

It has some potential.  On first blush this seems to require parsing
all the C header files (accounting for arch/kconfig cpp macros) to
look for a matching function declaration.  Then the parameter/return
types may require listing additional C headers.  As you know, kernel
headers are complicated and rich with #ifdefs and macros that may
expand to inlines inlines, so a naive text processing inspired by the
following may not be worthwhile.  How do you imagine it working?

fgrep -v '[' /proc/kallsyms | while read addr type fn 
do
echo $fn; find /lib/modules/`uname -r`/build/include -type f | xargs egrep " ${fn}[ ]*\("
done


- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Mon, Dec 22, 2008 at 03:37:48PM -0500, Frank Ch. Eigler wrote:
> > Yes, but how long will people need to wait before debuginfo quality
> > will improve?  And when such a hypothetical improved compiler be
> > available and trusted to compile the kernel correctly?  [...]
> 
> You misinterpret the status quo.  In reality, debuginfo quality with
> current gcc has been satisfactory for many purposes.  That is what
> enables user-space debuggers to work as well as they do, and indeed
> enables systemtap to work as well as it does.  The major current
> debuginfo quality shortcoming involves only those intra-function
> statement probes.

I can only speak from my personal experience, which is given the
kernel default compilation options, and the inline functions and
static functions in the parts of the kernel that I normally play with
(i.e., ext3/ext4 filesystem code), in nearly every single time where
I've had to try to place a Systemtap probe, if it hasn't been at a
non-static function entry/exit point, the vast majority of the time,
it Simply Doesn't Work.

With user-space debuggers, it's not so great, either, but normally
when I do debugging e2fsprogs, I recompile w/o optimizations to avoid
these frustrations.  Furthermore, e2fsprogs doesn't have the same
large number of static functions which gcc is far too happy to
in-line, and which currently makes the debuginfo largely useless.

Worse yet, the error handling with Systemtap is so uniformly awful (to
the point that even Roland has complained --- "stap -vvvvvvv" anyone?)
that when it doesn't work, it's highly frustrating to figure out what
is going on.  Basically, I've written off anything other than
non-static function entry/exits as being too frustrating to even try using.

It might be that the right answer is to tell kernel developers to
compile their development kernels without optimization.  OK, it that
goes against the Systemtap religion that you must never need to
recompile or use special built options.  But that mantra is causing
massive frustration, since after multiple times of trying to set probe
points inside ext4 code, and watching it Utterly Fail, it causes me to
curse debuginfo, and Systemtap's reliance on it, as Utter Crap.

You can go ahead, and dismiss my reports --- after all, I'm one of the
few kernel developers who's even bothering to try System Tap at this
point, and adding markers into the kernel --- and the first North
American kernel developer and CTO of the Linux Foundation can't
possibly know anything about, you know, kernel development
workflows....

> If you're still talking about debuginfo per se, I am led to believe
> that only a small number of kernel developers are negatively affected:
> those that build large kernel configurations but on such miniscule
> hardware that the sheer disk I/O of the -g data is deemed intolerable.

Heh.  Only a very few number of kernel developers are even *trying* to
use Systemtap at this point.  Trying to keep the few remaining kernel
developers happy might be considered a good thing.  You saw the number
of hands that went up at the Kernel Summit.  After the Plumber's
Conference and the Kernel Summit, Aryan van der Ven told me and a
group of of kernel developers that as a senior manager, he planned to
pull Intel's last resources working on SystemTap off the project,
based on SystemTap developers not being willing to listen to input.
And other kernel engineers employed at Red Hat told me that they
thought the Red Hat SystempTap developers were totally out of control
and they had but very little respect for the project.  You are in a
*hole*, brother.  And the first rule is, when you are in a hole, stop
digging....

> > The process of getting userspace probes in will go slowly (if at
> > all).  I haven't seen a new version of the patches submitted in
> > months --- and the 2.6.29 merge window will be opening soon.  [...]
> 
> Roland?  Please advise.

Note that the best way to get kernel patches in is to handle the
comments, and repost new versions quickly and often with the comments
addressed.  Look at how Eric Paris from Red Hat posted fanotify
patches; multiple versions, addressing comments, every few days.  And
even though many people are dubious about his stated goal of helping
the virus scanner partner's of Red Hat, he's made enough core
infrastructure improvements to other notification systems, such as
inotify and dnotify, that those patches will likely make it into the
mainline.

Quite frankly, the pace at which new versions of the patches are
published in response to comments reminds me doomed patches such as
reiserfs4....

> > This means no modules, since debuginfo is a disaster with modules
> > (so helping people create configurations that don't use modules
> > would be helpful);
> 
> Please elaborate what you mean.  Why systemtap is forced to use
> (create) modules at all has been explained often.

What I mean is that if you build a kernel with a large number of
modules, the bloat of systemtap becomes disastrous.  With a
distro-style kernel, the compilation time literally goes up by a
factor of 4-5 or so, and the disk space utilization goes up by a
factor of 8 or so.  It's not so bad if you only the device drivers you
into a single kernel, and not create any unneeded modules.  This is
often not the common way at least some kernel developers build
systems, since (a) modules make it easier to load and unload
subsystems without rebooting, and this can shorten the
compile-edit-debug cycle times, and (b) some kernel developers want to
be sure they know whether any infrastructural changes they might cause
some random device driver or other random subsystem to break.
Unfortunately, building a distro-style kernel is a horrible disaster
with debuginfo enabled, and so warning people off about this
particular issue would be helpful.

> Sure, upstream changes in the location of Modules.marker are something
> we can easily adapt to.  Are you aware of a current problem with that?
> On the other hand, we plan to work on also connecting straight to
> tracepoints, and to various slices of ftrace, for which a file such
> the troubled-history Modules.markers will not be necessary.

Only that "make install" or "make modules_install" doesn't currently
install Modules.marker anywhere useful.  And there is no standard
location for where Modules.marker will be installed given current
distribution packaging tools.  Attention to consumability of SystemTap
given a variety of different kernel installation schemes is a Really
Good Idea.  It's not just Fedora RPM-style macros and "make install".
There are many other ways that people create and install kernels; and
this kind of attention to detail will stop people from getting
Frustrated with SystemTap.

> > 3) Actually try to find a way for Systemtap to be more useful
> > *without* waiting for [new gcc]
> 
> As I said above, the main thing we're expecting from a future gcc is
> improvement on statement (arbitraty source line) probing.  Otherwise
> quality is fine and we're not having to wait.  (As everyone knows,
> quantity (compression) is not great but at least this is only a
> usability rather than functionality matter.)

It's a rather important usability issue.  Part of the reason why
SystemTap is so frustrating is precisely because there hasn't been a
big enough focus on usability, IMHO.  Maybe systemtap developers get
used to stap -vvvvvv to figure out why something doesn't work, but it
still strikes me as a reminder of how awful and kldugy the technology
is.

> Nor do we.  In fact, we've been as unimposing as possible upon both
> kernel and user-space build systems -- perhaps to a fault.
> 
> > [...] the only advantage SystemTap has over ftrace is (a) Systemtap
> > has access to the function parameters, and (b) Systemtap allows you
> > to filter logging information based on conditions, which is more
> > efficient than filtering in userspace.
> 
> Beyond those, there is really plenty of difference in capability.
> Line-by-line function tracing?  User-space?  Concurrent users?  The
> list goes on...

Line-by-line function tracing doesn't work for me.  I've given up on
it as useless.   Yawn.

User-space tracing --- is a patch that I don't forsee going mainline
anytime soon, given a lack of understanding about how to respond to
comments and how to get patches upstream.  (Again, there are people in
Red Hat who know how to do this; why isn't Roland reaching out to
people like Eric Paris, David Jones, Alan Cox, etc.?)

Concurrent users -- doesn't matter to kernel developers.  And exactly
who do you expect is going to be adding markers to the kernel sources,
if you continually blow off the concerns of kernel developers such as
James and myself?

> > [...] is to simply generate the debuginfo information on demand so
> > the function parameters on entry can be decoded.  We have the header
> > files after all, so it shouldn't be hard to generate a pseudo function:
> > 
> > extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
> >        void *ptr)
> > {
> >         return 0;
> > }
> 
> It has some potential.  On first blush this seems to require parsing
> all the C header files (accounting for arch/kconfig cpp macros) to
> look for a matching function declaration.  Then the parameter/return
> types may require listing additional C headers.  As you know, kernel
> headers are complicated and rich with #ifdefs and macros that may
> expand to inlines inlines, so a naive text processing inspired by the
> following may not be worthwhile.  How do you imagine it working?

Well, the original way I was thinking about doing this involved
support from the kernel sources --- so changes to kbuild to generate
debugging information that is far more compact, only optimized to
provide kernel function parameter information --- since as I have
mentioend, line-by-line probing in practice doesn't work for me, so
why pay for a factor 8 explosion of debuginfo overhead when most of it
seems largely useless given standard kernel compile options.

However, it might be doable without having made any kernel changes.
Finding which header file a function parameter can be found is pretty
simple.  There are some places where the function names is created
using magic cpp concatenation games, but these tend to be inline
functions which systemtap can't probe anyway.  The hard part is
knowing which other header files might be needed.  This can be easily
extracted from source files --- so the question is whether Systemtap
ships a database of which header files are needed before it's safe to
include a particular .h file containing the function prototype; or
whether we need a kbuild patch to extract this information; or do both.

	   	  	       	  	       		    - Ted

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Theodore Tso wrote:
>>> This means no modules, since debuginfo is a disaster with modules
>>> (so helping people create configurations that don't use modules
>>> would be helpful);
>> Please elaborate what you mean.  Why systemtap is forced to use
>> (create) modules at all has been explained often.
> 
> What I mean is that if you build a kernel with a large number of
> modules, the bloat of systemtap becomes disastrous.  With a
> distro-style kernel, the compilation time literally goes up by a
> factor of 4-5 or so, and the disk space utilization goes up by a
> factor of 8 or so.  It's not so bad if you only the device drivers you
> into a single kernel, and not create any unneeded modules.  This is
> often not the common way at least some kernel developers build
> systems, since (a) modules make it easier to load and unload
> subsystems without rebooting, and this can shorten the
> compile-edit-debug cycle times, and (b) some kernel developers want to
> be sure they know whether any infrastructural changes they might cause
> some random device driver or other random subsystem to break.
> Unfortunately, building a distro-style kernel is a horrible disaster
> with debuginfo enabled, and so warning people off about this
> particular issue would be helpful.

Hmm, why would you think all or nothing?

I agreed with your opinion that the kernel compilation time longer
if we configure more drivers compiling as modules.

However, even if you build all required modules into the kernel,
you can still enable module support. So, I think it's not an issue.


>> Sure, upstream changes in the location of Modules.marker are something
>> we can easily adapt to.  Are you aware of a current problem with that?
>> On the other hand, we plan to work on also connecting straight to
>> tracepoints, and to various slices of ftrace, for which a file such
>> the troubled-history Modules.markers will not be necessary.
> 
> Only that "make install" or "make modules_install" doesn't currently
> install Modules.marker anywhere useful.  And there is no standard
> location for where Modules.marker will be installed given current
> distribution packaging tools.  Attention to consumability of SystemTap
> given a variety of different kernel installation schemes is a Really
> Good Idea.  It's not just Fedora RPM-style macros and "make install".
> There are many other ways that people create and install kernels; and
> this kind of attention to detail will stop people from getting
> Frustrated with SystemTap.

Hm, perhaps, we should support an option or environment variable to
specify (several) path of Modules.marker. After fixing that path, we can
deprecate the option.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Mon, Dec 22, 2008 at 06:34:11PM -0500, Masami Hiramatsu wrote:
> 
> Hmm, why would you think all or nothing?
> 
> I agreed with your opinion that the kernel compilation time longer
> if we configure more drivers compiling as modules.
> 
> However, even if you build all required modules into the kernel,
> you can still enable module support. So, I think it's not an issue.

Yes, that's what I was talking about; my apologies for not being
clear.  So what I do when I want to use debuginfo is I build a single
reduced-functionality kernel, with only device drivers I need built
into the kernel, but I allow modules because SystemTap requires them.
My suggestion was *documenting* this in the Wiki to make it clear that
(a) debuginfo sucks and terribly bloats with lots of modules, and (b)
developers who want to use SystemTap with debuginfo would be well
advised to avoid using large numbers of modules.  This is the kind of
usability issue that SystemTap is badly lacking.

Also, if you want to advertise using line-by-line probes as a
SystemTap feature, you might want to suggest a kernel patch that
disables gcc optimizations such that probes have a half-way decent
chance of actually working.  Yes it violates the SystemTap religion
about never needing to recompile the kernel; but it recognizes the
reality that the gcc toolchain has incredibly sucky debuginfo that
doesn't take into account CSE, function reordering, and inlining of
static functions.

> Hm, perhaps, we should support an option or environment variable to
> specify (several) path of Modules.marker. After fixing that path, we can
> deprecate the option.

There are only a few places where it can go; one is
/boot/Module.marker-<kver>; another is
/lib/modules/<kver>/Module.markers.  The last
place is /lib/modules/<kver>/build/Module.marker .

							-Ted

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

> There are only a few places where it can go; one is
> /boot/Module.marker-<kver>; another is
> /lib/modules/<kver>/Module.markers.  The last
> place is /lib/modules/<kver>/build/Module.marker .

The latter is where Fedora kernel rpms put it (this places it in the
kernel-devel subpackage, the one required for compiling loadable modules).
The logic was that Module.symvers is there also, and--at least at the
time--markers were another thing used only in building a kernel module, so
seemed similar.  If it's desireable to standardize on one of the other
locations as preferable, we can certainly change the Fedora packaging
however is most useful.


Thanks,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Theodore Tso wrote:
> On Mon, Dec 22, 2008 at 06:34:11PM -0500, Masami Hiramatsu wrote:
>> Hmm, why would you think all or nothing?
>>
>> I agreed with your opinion that the kernel compilation time longer
>> if we configure more drivers compiling as modules.
>>
>> However, even if you build all required modules into the kernel,
>> you can still enable module support. So, I think it's not an issue.
> 
> Yes, that's what I was talking about; my apologies for not being
> clear.  So what I do when I want to use debuginfo is I build a single
> reduced-functionality kernel, with only device drivers I need built
> into the kernel, but I allow modules because SystemTap requires them.
> My suggestion was *documenting* this in the Wiki to make it clear that
> (a) debuginfo sucks and terribly bloats with lots of modules, and (b)
> developers who want to use SystemTap with debuginfo would be well
> advised to avoid using large numbers of modules.  This is the kind of
> usability issue that SystemTap is badly lacking.

OK, so kernel developers would better configure reducing or embedding
drivers as much as possible, since the amount of debuginfo strongly
depends on the number of modules. That's a good tip.

> Also, if you want to advertise using line-by-line probes as a
> SystemTap feature, you might want to suggest a kernel patch that
> disables gcc optimizations such that probes have a half-way decent
> chance of actually working.  Yes it violates the SystemTap religion
> about never needing to recompile the kernel; but it recognizes the
> reality that the gcc toolchain has incredibly sucky debuginfo that
> doesn't take into account CSE, function reordering, and inlining of
> static functions.

Thank you for a good advice. I think, at least, we might have to
clarify what optimizations will effect to line-by-line probing.

I also heard that -fno-inline-functions-called-once which is
enabled by CONFIG_DEBUG_SECTION_MISMATCH enables us to access
the parameters of some static functions.

>> Hm, perhaps, we should support an option or environment variable to
>> specify (several) path of Modules.marker. After fixing that path, we can
>> deprecate the option.
> 
> There are only a few places where it can go; one is
> /boot/Module.marker-<kver>; another is
> /lib/modules/<kver>/Module.markers.  The last
> place is /lib/modules/<kver>/build/Module.marker .

Additionally, if users want to use markers in their out-of-tree
driver, they will have another Module.marker file in the local
(module build) directory. :-)

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi -

On Mon, Dec 22, 2008 at 05:37:41PM -0500, Theodore Tso wrote:

> [...]
> > > Yes, but how long will people need to wait before debuginfo quality
> > > will improve?  And when such a hypothetical improved compiler be
> > > available and trusted to compile the kernel correctly?  [...]
> > 
> > You misinterpret the status quo.  In reality, debuginfo quality with
> > current gcc has been satisfactory for many purposes.  That is what
> > enables user-space debuggers to work as well as they do, and indeed
> > enables systemtap to work as well as it does.  The major current
> > debuginfo quality shortcoming involves only those intra-function
> > statement probes.
> 
> I can only speak from my personal experience, which is given the
> kernel default compilation options, and the inline functions and
> static functions in the parts of the kernel that I normally play with

Some inline/static functions become de facto intra-function statement
probes that we have problems with.  Note that can't probe those
functions based on symbol data (kallsyms) because they don't show up
there at all!  Same with ftrace, I think.  Keep that in mind if you
wish to fairly compare alternative technologies.


> [...]
> Worse yet, the error handling with Systemtap is so uniformly awful (to
> the point that even Roland has complained --- "stap -vvvvvvv" anyone?)

This has been since improved.


> It might be that the right answer is to tell kernel developers to
> compile their development kernels without optimization.  OK, it that
> goes against the Systemtap religion that you must never need to
> recompile or use special built options.  [...]

It's not a "religion".  It's sometimes an absolute user requirement.
It is also a way of letting the rest of the open-source "ecosystem"
evolve without us imposing ourselves.


> You can go ahead, and dismiss my reports [...]

I have endavoured to address each of your points.  If I overlooked
one, I apologize, and if you point it out I'll be glad to retry.


> > If you're still talking about debuginfo per se, I am led to believe
> > that only a small number of kernel developers are negatively affected:
> > those that build large kernel configurations but on such miniscule
> > hardware that the sheer disk I/O of the -g data is deemed intolerable.
> 
> Heh.  Only a very few number of kernel developers are even *trying* to
> use Systemtap at this point.  [...]

One short-term benefit you are likely to experience is the restoration
of the symbol-table-only probing, which should give you and others
some amount of probing capability without CONFIG_DEBUG_INFO.  Other
intermediate approaches (debuginfo subsetting, compression) appear to
require development well outside systemtap.

> After the Plumber's Conference and the Kernel Summit, [Arjan] van der
> Ven told me [...]  And other kernel engineers employed at Red Hat
> told me [...]

It is hard to respond to this, if these other gentlemen don't take up
our manifold invitations to comment/advise us directly as opposed to
each other.  (The last thing I recall Arjan ever suggesting was
packaging some systemtap tapset scripts along with the kernel, in the
hope that they would be maintained by subsystem maintainers there.
That idea appeared to be mooted by the apparent kernel summit
consensus for adding static compiled-in tracing hooks into the
kernel.)


> > > This means no modules, since debuginfo is a disaster with modules
> > > (so helping people create configurations that don't use modules
> > > would be helpful);
> > 
> > Please elaborate what you mean.  Why systemtap is forced to use
> > (create) modules at all has been explained often.

> What I mean is that if you build a kernel with a large number of
> modules, the bloat of systemtap becomes disastrous.  [...]

It's more fair to label this "bloat of debuginfo", since systemtap is
not responsible for how much code you choose to build.

> Unfortunately, building a distro-style kernel is a horrible disaster
> with debuginfo enabled, and so warning people off about this
> particular issue would be helpful.

Is http://tinyurl.com/96dza8 what you had in mind?

> > Sure, upstream changes in the location of Modules.marker are something
> > we can easily adapt to.  Are you aware of a current problem with that?
> > [...]
> 
> Only that "make install" or "make modules_install" doesn't currently
> install Modules.marker anywhere useful.  [...]

OK, I believe you mean that as the sole (?) user of this file,
systemtap folks should fix this kernel oversight by submitting a patch
against kbuild makefiles.


> > [...]  (As everyone knows, quantity (compression) is not great but
> > at least this is only a usability rather than functionality
> > matter.)
>
> It's a rather important usability issue.  [...]

Agreed.

> Line-by-line function tracing doesn't work for me.  [...]

I hope people will like it once it works robustly.

> User-space tracing --- is a patch that I don't forsee going mainline
> anytime soon [...]

I hope there will be progress on this soon.

> Concurrent users -- doesn't matter to kernel developers.  

I don't think that's completely true.  Right now, ftrace is a
single-configuration-at-a-time tool, and I recall reading complaints
about that.


> And exactly who do you expect is going to be adding markers to the
> kernel sources, [...]

It turns out that, among others, several Red Hat kernel engineers are
actively pursuing this, including sending tracepoint-laden patches
upstreamward via lkml/lttng.


> > > [...] is to simply generate the debuginfo information on demand so
> > > the function parameters on entry can be decoded.  We have the header
> > > files after all, so it shouldn't be hard to generate a pseudo function:
> > > 
> > > extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
> > >        void *ptr)
> > > {
> > >         return 0;
> > > }

> > It has some potential.  On first blush this seems to require
> > parsing all the C header files (accounting for arch/kconfig cpp
> > macros) to look for a matching function declaration.  [...]  How
> > do you imagine it working?

> Well, the original way I was thinking about doing this involved
> support from the kernel sources --- so changes to kbuild to generate
> debugging information that is far more compact, only optimized to
> provide kernel function parameter information

That's does not sound like a simple kbuild preference.  That may need
a whole new tool well beyond strip/objcopy.

> [...] However, it might be doable without having made any kernel
> changes.  Finding which header file a function parameter can be
> found is pretty simple.  [...]  The hard part is knowing which other
> header files might be needed.  This can be easily extracted from
> source files [...]

So more C source parsing.

> so the question is whether Systemtap ships a database of which
> header files are needed [...]

Maybe, but this sounds about as kernel-version-fragile as some of our
tapset scripts.

> or whether we need a kbuild patch to extract this information; or do
> both.

OK, so we agree we're talking about a nontrivial amount of code here.


- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

> > The process of getting userspace probes in will go slowly (if at
> > all).  I haven't seen a new version of the patches submitted in
> > months --- and the 2.6.29 merge window will be opening soon.  [...]
> 
> Roland?  Please advise.

I've never had much facility for soap opera, so I'll ignore Ted's
hyperbole about my personal ability to relate to the kernel community.
I think my record of kernel contributions speaks for itself (for however
good or ill you construe that record).  But it's worth noting that of
the 231 upstream commits since 2.6.24 that GIT attributes to me, only a
handful are *not* part of the groundwork for utrace (or whatever it may
become or be supplanted by).

I believe the actual reasons for slow progress are far more mundane than
any dramatic intercommunity conflicts or failures to address the kernel
development culture.  It's the variety of effects of having ~0.5 person
ever work on the core enabling piece (utrace) that has to go in to
enable other work.  (And 0.5 of a bit of a scatterbrain, at that.)  As
I've seen it, there has been a great deal of hand-wringing and fret, and
some amount of blame-seeking, but not a lot of straight-up collaborative
hacking.  I consider this an organizational failure of the project's
community and the organizations supporting it, of which I'm a member and
a party to that failure.

I'll freely admit to being disappointed about that, but I don't see any
point in dwelling on any past lacks--I'm sure I'm lousy at discussions
hashing them over, and I'm sure I have no patience for them either.  We
simply have to work harder and more cohesively together to make progress
now, and I think we all know what it takes to do that.  
Less yakin', more hackin'.

We recently had a meeting (among several main systemtap contributors) on
this.  I did not then characterize the situation in such stark terms,
but we discussed how to collaborate better on utrace and related code
that we need to get in shape to go into the kernel.  (Some notes from
that meeting were posted on the utrace-devel@redhat.com mailing list,
and I'm not sure if those were posted here too.)  I can only hope that
when we resume work in the new year, we'll have turned over a new leaf
and see contributors driving hard and participating with gusto in every
aspect of the kernel work we need.

This will certainly be a test, as it comes in a context of less than 0.5
of my time allocated to kernel work in the coming period that will cover
at least the next one or two kernel merge cycles.  Progress on utrace
has been regrettably slow this fall, and it will get even slower now
without major collaboration.  (It so happens this refocusing of my time
is intended to jump-start the DWARF size reduction work.  I won't deny
this may well be "12-18 months out" for stable release deployments, but
it can't even be that if it doesn't start sometime, and this is an
approach to the debuginfo issues that most people other than Ted seem to
agree could solve many important problems for systemtap.)


Thanks and Happy Holidays,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Roland,

Roland McGrath wrote:
> This will certainly be a test, as it comes in a context of less than 0.5
> of my time allocated to kernel work in the coming period that will cover
> at least the next one or two kernel merge cycles.  Progress on utrace
> has been regrettably slow this fall, and it will get even slower now
> without major collaboration.  (It so happens this refocusing of my time
> is intended to jump-start the DWARF size reduction work.  I won't deny
> this may well be "12-18 months out" for stable release deployments, but
> it can't even be that if it doesn't start sometime, and this is an
> approach to the debuginfo issues that most people other than Ted seem to
> agree could solve many important problems for systemtap.)

I'm interested in your work. Could you tell me your plan? Would you
develop a tool to reduce debuginfo like objcopy?

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

> I'm interested in your work. Could you tell me your plan? Would you
> develop a tool to reduce debuginfo like objcopy?

It's roughly along those lines.  The wiki at elfutils.fedorahosted.org and
the elfutils-devel@lists.fedorahosted.org mailing list are where that
planning is taking place.


Thanks,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Roland McGrath wrote:
>> I'm interested in your work. Could you tell me your plan? Would you
>> develop a tool to reduce debuginfo like objcopy?
> 
> It's roughly along those lines.  The wiki at elfutils.fedorahosted.org and
> the elfutils-devel@lists.fedorahosted.org mailing list are where that
> planning is taking place.

As far as I read the wiki, it seems that you'll make a tool to compress
debuginfo. Out of curiosity, how small would you expect that the tool
can reduce debuginfo? (Can we select(edit) how much info the dwarf
includes by that tool, or just use compressed format?)

By the way, if the development will take more than two years, it might be
belated, at last. Unfortunately, we are in a severe situation. I don't mean
that you give up the effort. Instead of that, I recommend you to run the
project with target users (for example, kernel developers and systemtap
developers), do rapid prototyping and embrace their feedbacks. It could
take more time, but you can continue to attract users and give us hope!

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

> As far as I read the wiki, it seems that you'll make a tool to compress
> debuginfo. Out of curiosity, how small would you expect that the tool
> can reduce debuginfo? 

I wouldn't like to speculate.  I plan to just get on with it and progress
quickly to tools sufficient to make concrete measurements.

> (Can we select(edit) how much info the dwarf
> includes by that tool, or just use compressed format?)

I have not made particular plans for anything that would lose information.
But the intent is to take an approach that will make it fairly easy to
write and experiment with new DWARF-transforming tools in the future.

> By the way, if the development will take more than two years, it might be
> belated, at last.

I don't think I understood this sentence.  I certainly would be much
happier if we'd embarked on this work two years ago, but we didn't,
and we have yet to really begin.  The idea that it could take
anything like two years of actual development on this to produce
results seems just ludicrous to me, if that's what you meant.
Unless the effort goes horribly awry, I'll know a great deal more
concrete about it in a month, and the hope going in is to have quite
a lot of it done in two months.

> Unfortunately, we are in a severe situation. I don't mean
> that you give up the effort. Instead of that, I recommend you to run the
> project with target users (for example, kernel developers and systemtap
> developers), do rapid prototyping and embrace their feedbacks. It could
> take more time, but you can continue to attract users and give us hope!

I don't know what it means to "run the project with target users".
The actual tool development is something for DWARF and toolchain
experts to do, and I don't expect kernel developers to devote their
time to that just because they would benefit from the deployment of
such tools (as will many other kinds of developers--this is not
solely for the benefit of systemtap).  Of course the project is open
to everyone who wants to contribute, and of course development of
new tools will be driven by the concerns of their potential users.


Thanks,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Roland,

Roland McGrath wrote:
>> By the way, if the development will take more than two years, it might be
>> belated, at last.
> 
> I don't think I understood this sentence.  I certainly would be much
> happier if we'd embarked on this work two years ago, but we didn't,
> and we have yet to really begin.  The idea that it could take
> anything like two years of actual development on this to produce
> results seems just ludicrous to me, if that's what you meant.
> Unless the effort goes horribly awry, I'll know a great deal more
> concrete about it in a month, and the hope going in is to have quite
> a lot of it done in two months.

Would you mean you can release a usable prototype in two months
if no trouble? Great! If so, I think it's enough short for us and kernel
developers to use the tool.

>> Unfortunately, we are in a severe situation. I don't mean
>> that you give up the effort. Instead of that, I recommend you to run the
>> project with target users (for example, kernel developers and systemtap
>> developers), do rapid prototyping and embrace their feedbacks. It could
>> take more time, but you can continue to attract users and give us hope!
> 
> I don't know what it means to "run the project with target users".

Sorry about confusing you.
I just mean that we'd better notify users who are suffered from
huge debuginfo that there is a plan to make a tool which can compress
the debuginfo (and they can use it soon).
Unless that, we will waste time on finding another way to solve this
issue :(.

> The actual tool development is something for DWARF and toolchain
> experts to do, and I don't expect kernel developers to devote their
> time to that just because they would benefit from the deployment of
> such tools (as will many other kinds of developers--this is not
> solely for the benefit of systemtap).

Indeed, and at least there are some people who will receive benefit
from that tool.

>  Of course the project is open
> to everyone who wants to contribute, and of course development of
> new tools will be driven by the concerns of their potential users.

Sure, and that's a good news for "Frustrated" systemtap users.

Thank you for your work!

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

> Would you mean you can release a usable prototype in two months
> if no trouble? Great! 

We'll see how it goes.

> Sorry about confusing you.
> I just mean that we'd better notify users who are suffered from
> huge debuginfo that there is a plan to make a tool which can compress
> the debuginfo (and they can use it soon).

Ok.  I think we have been saying for a long time that this is a likely
solution and one we intended to have at some point.  Until there is
something concrete to show, I wouldn't like to start offering new
promises about it.

> Unless that, we will waste time on finding another way to solve this
> issue :(.

I don't know that one solution is going to address all of everyone's
concerns, and certainly the initial versions will have some trade-offs
that won't suit some.  I wouldn't necessarily say that the presence of
this effort should preclude exploring other approaches.


Thanks,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Wed, Dec 24, 2008 at 01:08:48AM -0500, Masami Hiramatsu wrote:
>> I don't think I understood this sentence.  I certainly would be much
>> happier if we'd embarked on this work two years ago, but we didn't,
>> and we have yet to really begin.  The idea that it could take
>> anything like two years of actual development on this to produce
>> results seems just ludicrous to me, if that's what you meant.
>> Unless the effort goes horribly awry, I'll know a great deal more
>> concrete about it in a month, and the hope going in is to have quite
>> a lot of it done in two months.
>
> Would you mean you can release a usable prototype in two months
> if no trouble? Great! If so, I think it's enough short for us and kernel
> developers to use the tool.

The key question is whether said prototype is a stand-alone tool, or
something which is a patch to the current development tree of gcc.  If
the development work is merely a way to compress the debuginfo, either
by outright using gzip, or eliminating common type information ala
Open Solaris' CTF, it could very well be a standalone tool, at which
point two months is really no big deal.  (A tool that stripped out
line-by-line debuginfo but left function parameter information might
also be useful, given my experience about exactly how useful per-line
probing has actually worked for me in practice.)

If however the patches need to go into the gcc development, we end up
getting blocked for the next gcc release, that could be a very long
time.  Even after the compiler is released, the Linux kernel is very
sensitive to compiler vagrancies.  Regardless of whose side you chose
on the "the C spec allows us to do this, and any code that relies on
previous behaviour is buggy" vs. "no sane intepretation of the
standard would allow that; it's the compiler's fault", the fact of the
matter is that a bleeding edge compiler can't be trusted to compile a
Linux kernel w/o bugs for quite some time --- whether you call it
making changes to the kernel to work around gcc bugs, or fixing bugs
in the kernel, the net result is the same.

Note that some of the changes to allow line-by-line probes to be more
reliable in the face of CSE, static function inlining, code
reordering, etc., changes almost certainly have to be made to the
compiler.

Hence, it might make sense to figure out what could be done that does
_not_ require gcc modifications, but rather could be done as a
standalone program, as well as what a long-term development plan that
requires making Compiler toolchain modifications.

To be clear, it is the gcc changes that I suspect will require 12-18
months (and so therefore makes me debious it could be done before RHEL
6), because of the gcc release and correct kernel compilation issues.
If a prototype of a standalone program which can compress the debug
info in two months, that would be highly helpful, I agree.

     	   					- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

Our current approach is a post-processing tool that does not include
changes to the compiler or linker.  Having the original output directly
from the toolchain be more compact is also desireable, but it is not our
near-term focus.  This work I'm talking about will address size concerns.
The quality/usefulness of the info is another matter, which certainly does
entail compiler work.  There are some long-term efforts going in that area,
but that is a separate issue from what I'm doing now.


Thanks,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Tue, Dec 23, 2008 at 01:13:06PM -0800, Roland McGrath wrote:

> I believe the actual reasons for slow progress are far more mundane than
> any dramatic intercommunity conflicts or failures to address the kernel
> development culture.  It's the variety of effects of having ~0.5 person
> ever work on the core enabling piece (utrace) that has to go in to
> enable other work.  (And 0.5 of a bit of a scatterbrain, at that.)  As
> I've seen it, there has been a great deal of hand-wringing and fret, and
> some amount of blame-seeking, but not a lot of straight-up collaborative
> hacking.  I consider this an organizational failure of the project's
> community and the organizations supporting it, of which I'm a member and
> a party to that failure.

How much more utrace work is needed?  Fedora is shipping it, right?
What still needs to be done before it can be merged?  The
characterization that I heard at the plumber's conference was that the
work was largely done, and it was just a matter of it getting merged.

/me refers to the utrace-devel mailing list...

Ah, I see, there are still some ptrace regression bugs.  All code has
bugs; heck, the ext4 code has had plenty of bugs, and all of my ext4
development happens on my own time, late at night or when I am on an
airplane.... 

May I gently suggest that if SystemTap project had been more
aggressively making SystemTap usable for kernel developers, and it had
an active user community that included may more LKML denizens, that
perhaps these bugs might have been fixed more quickly?  I don't know
how to make a better case that an attitude of "kernel developers don't
matter; our primary customers are enterprise end users" is in the end
quite self-defeating.

The other suggestion I would make is to work through your organization
and get someone like Ingo Molnar to spend a week or so going through
the utrace code.  He might be able to spot race conditions and other
problems quite quickly, and otherwise make suggestions for to get the
patch ready in time for the next merge window.  After all, if this is
going to make the next RHEL cutoff, you probably need to get the code
in via this merge window or the next one to make Fedora 11.  And the
fact that the code hasn't started down the LKML merge window is
something that if *I* were in Red Hat technical management, would make
me be highly concerned indeed, and I would be trying to engage some
expert resources to assure success, and not a last minute scramble.

> (It so happens this refocusing of my time is intended to jump-start
> the DWARF size reduction work.  I won't deny this may well be "12-18
> months out" for stable release deployments, but it can't even be
> that if it doesn't start sometime, and this is an approach to the
> debuginfo issues that most people other than Ted seem to agree could
> solve many important problems for systemtap.)

I agree it would solve many problems for SystemTap, I just don't see
it reaching fruition until RHEL 7, and there is such a thing as
missing the window, where projects get defunded by companies, and
kernel developers start working on alternate schemes (like ftrace and
LTTng) because they've given up on SystemTap.  I would probably be
making similar recommendations at my company, except that I see the
potential in SystemTap, and realize there are may things that these
alternate solutions would not be able to do --- for example, Userspace
Tracing, filtering log decisions in kernel space, etc.  It's just been
frustrating seeing a project with so much potential not living up to
that potential.

						- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Ted,

Theodore Tso wrote:
> May I gently suggest that if SystemTap project had been more
> aggressively making SystemTap usable for kernel developers, and it had
> an active user community that included may more LKML denizens, that
> perhaps these bugs might have been fixed more quickly?  I don't know
> how to make a better case that an attitude of "kernel developers don't
> matter; our primary customers are enterprise end users" is in the end
> quite self-defeating.

OK, so I would like to hear your suggestions seriously.
(I believe that we know on what we are working).
I think it's time to embody action items. Just criticizing can't go
things ahead. So, what we can do for you?

- Communication issue
 - Notifying releasing on LKML frequently.
 - Preparing more documents for kernel developers.
 - Feeding back kernel developers' opinions.

- Solving debuginfo issue
 - in the short term, notifying how to minimize debuginfo size.
 - in the mid term, making dummy (function-entry) debuginfo.
 - in the long term, Roland's dwarf compressor can solve this issue.

- Upstream first issue
 - Utrace/Uprobe
  - Develop code on LKML and embrace feedbacks.
  - Involving main kernel maintainer to speed up the code merging.
 - Systemtap itself
  - Merging a part of runtime and basic tapset to kernel tree, which
   will provide us a permanent API for systemtap generated code.
  (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
   generating ftrace-plugin tracers. This will be more acceptable for
   upstream because we don't need to add so many code).

Any other issues and ideas?

>> (It so happens this refocusing of my time is intended to jump-start
>> the DWARF size reduction work.  I won't deny this may well be "12-18
>> months out" for stable release deployments, but it can't even be
>> that if it doesn't start sometime, and this is an approach to the
>> debuginfo issues that most people other than Ted seem to agree could
>> solve many important problems for systemtap.)
> 
> I agree it would solve many problems for SystemTap, I just don't see
> it reaching fruition until RHEL 7, and there is such a thing as
> missing the window, where projects get defunded by companies, and
> kernel developers start working on alternate schemes (like ftrace and
> LTTng) because they've given up on SystemTap.  I would probably be
> making similar recommendations at my company, except that I see the
> potential in SystemTap, and realize there are may things that these
> alternate solutions would not be able to do --- for example, Userspace
> Tracing, filtering log decisions in kernel space, etc.  It's just been
> frustrating seeing a project with so much potential not living up to
> that potential.

It's a nightmare scenario :(. I hope we can avoid it.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Tue, Dec 23, 2008 at 07:54:46PM -0500, Masami Hiramatsu wrote:
> 
> OK, so I would like to hear your suggestions seriously.
> (I believe that we know on what we are working).
> I think it's time to embody action items. Just criticizing can't go
> things ahead. So, what we can do for you?
> 
> - Communication issue
>  - Notifying releasing on LKML frequently.
>  - Preparing more documents for kernel developers.
>  - Feeding back kernel developers' opinions.
> 
> - Solving debuginfo issue
>  - in the short term, notifying how to minimize debuginfo size.
>  - in the mid term, making dummy (function-entry) debuginfo.
>  - in the long term, Roland's dwarf compressor can solve this issue.
> 
> - Upstream first issue
>  - Utrace/Uprobe
>   - Develop code on LKML and embrace feedbacks.
>   - Involving main kernel maintainer to speed up the code merging.
>  - Systemtap itself
>   - Merging a part of runtime and basic tapset to kernel tree, which
>    will provide us a permanent API for systemtap generated code.
>   (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
>    generating ftrace-plugin tracers. This will be more acceptable for
>    upstream because we don't need to add so many code).
> 
> Any other issues and ideas?

That's a pretty good list.  Other things to think about

*) One of the really important reasons why SystemTap needs to move
   runtime into the kernel is because especially for the community
   distributions, Fedora for example releases regular bleeding-edge
   kernels so that end users can help provide badly-needed testing of
   development kernel.  So if Systemtap needs to be updated from git
   to deal with changes in the development kernel, Fedora users who
   are using the bleeding edge kernels won't be able to use Systemtap
   unless it is released very often as Fedora, Open Suse, and
   otherwise packaged for other community distributions.  If the
   SystemTap runtime is bundled with the kernel, then it's much more
   likely that end users who want to use the daily kernel RPM's will
   also be able to use SystemTap.

*) Systemtap needs to be adpted to use tracepoints, quickly; the
   kernel markers for the scheduler have disappeared and replaced with
   tracepoints.  I've getting pressure to replace ext4's markers with
   tracepoints, and eventually I suspect markers infrastructure will
   probably disappear entirely since tracepoints are perceived as
   better and as their replacement.  Personally, I haven't had a
   chance to analyze them deeply enough to know whether this is true,
   but it's clearly the long-term direction.  (And yes, this is part
   of efforts to provide a tracing solution that assumes SystemTap is
   not part of the picture.)

The idea of using the ftrace infrastructure is a good one; since
SystemTap resisted making an effort get its runtime into the kernel,
now that the ftrace infrastructure is in the kernel, the principle of
not merging code that duplicates functionality means that this makes
Systemtap now needs to adapt what infrastructure did get merged for
its purposes, since it will be much more difficult get parallel
functionality merged into the kernel.

Regards,

						- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi -

On Fri, Jan 09, 2009 at 09:48:10PM -0500, Theodore Tso wrote:
> [...]
> *) One of the really important reasons why SystemTap needs to move
>    runtime into the kernel is because especially for the community
>    distributions, [e.g. rawhide].
>    If the SystemTap runtime is bundled with the kernel, then it's
>    much more likely that end users who want to use the daily kernel
>    RPM's will also be able to use SystemTap.

That is one possible solution for this specific problem -- rawhide
systemtap users who're unable/unwilling to build systemtap out of git
occasoinally.  (Remember that the recent 2.6.28 breakage took a few
hours to fix.)

Another solution would be for rawhide-style distributions to
aggressively package systemtap snapshots into their development
streams -- as some are doing already.  (We could have semiautomated
snapshots going straight into fedora rawhide too, and could increase
the rate of minor releases.)


> *) Systemtap needs to be adpted to use tracepoints, quickly [...]

We will work on this very soon.

>    and eventually I suspect markers infrastructure will probably
>    disappear entirely since tracepoints are perceived as better and
>    as their replacement.

(That would probably hurt lttng more than it would systemtap.)

>    Personally, I haven't had a chance to analyze them deeply enough
>    to know whether this is true, but it's clearly the long-term
>    direction. [...]

I hope that before this long-term direction is brought into effect,
someone does sit down and analyze the issue deeply enough.


> SystemTap resisted making an effort get its runtime into the kernel,
> now that the ftrace infrastructure is in the kernel, the principle of
> not merging code that duplicates functionality means that this makes
> Systemtap now needs to adapt what infrastructure did get merged for
> its purposes, since it will be much more difficult get parallel
> functionality merged into the kernel.

You overestimate the amount of duplicated functionality.  For modern
kernels, systemtap uses the standard in-kernel relay API, with a tiny
shim.  (If the relay API were ever removed, systemtap would just
switch to another existing API.  Plain buffering is not rocket
science.)


- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi

Frank Ch. Eigler wrote:
> That is one possible solution for this specific problem -- rawhide
> systemtap users who're unable/unwilling to build systemtap out of git
> occasoinally.  (Remember that the recent 2.6.28 breakage took a few
> hours to fix.)
> 
> Another solution would be for rawhide-style distributions to
> aggressively package systemtap snapshots into their development
> streams -- as some are doing already.  (We could have semiautomated
> snapshots going straight into fedora rawhide too, and could increase
> the rate of minor releases.)

Even if we do that, we have to clarify which package can be applied
to which kernel version. I'm still not sure when some bugs reported
on this ml are fixed - e.g.
http://sources.redhat.com/ml/systemtap/2009-q1/msg00029.html

BTW, more those kind of bugfix are committed, more autoconfs are
introduced. We already has 15(!) autoconfs, and these autoconfs
increase compilation time (this will be avoided by caching the result
per kernel...).

I think current method is good for short-term solution, but that
obviously has a limit.


>> *) Systemtap needs to be adpted to use tracepoints, quickly [...]
> 
> We will work on this very soon.

That's a good news!

> 
>>    and eventually I suspect markers infrastructure will probably
>>    disappear entirely since tracepoints are perceived as better and
>>    as their replacement.
> 
> (That would probably hurt lttng more than it would systemtap.)

(Now, lttng is working with kernel community, I think this doesn't hurt
lttng so much.)

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi -

On Mon, Jan 12, 2009 at 01:17:49PM -0500, Masami Hiramatsu wrote:
> [...]
> Frank Ch. Eigler wrote:
> > That is one possible solution for this specific problem -- rawhide
> > systemtap users who're unable/unwilling to build systemtap out of git
> > occasoinally.  (Remember that the recent 2.6.28 breakage took a few
> > hours to fix.)
> > 
> > Another solution would be for rawhide-style distributions to
> > aggressively package systemtap snapshots into their development
> > streams [...]
> 
> Even if we do that, we have to clarify which package can be applied
> to which kernel version. 

In what way?  There would be one package in rawhide, which should work
on every kernel version that we've ever worked with.  It would be
replaced frequently - perhaps every few days.


> I'm still not sure when some bugs reported on this ml are fixed -
> e.g.  http://sources.redhat.com/ml/systemtap/2009-q1/msg00029.html

There was no bugzilla report associated with it, but it was fixed the
same day.


> BTW, more those kind of bugfix are committed, more autoconfs are
> introduced. We already has 15(!) autoconfs,

Yes, but as you know, their cost has a benefit: they enable our
operation with a whole spectrum of kernel versions.


> and these autoconfs increase compilation time (this will be avoided
> by caching the result per kernel...).[...]

FWIW, on my workstation, they seem to add about a second.


> >>    and eventually I suspect markers infrastructure will probably
> >>    disappear entirely since tracepoints are perceived as better and
> >>    as their replacement.
> > 
> > (That would probably hurt lttng more than it would systemtap.)
> 
> (Now, lttng is working with kernel community, I think this doesn't hurt
> lttng so much.)

I'm simply saying that currently lttng thoroughly depends on markers,
which systemtap does not.


- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Frank,

Frank Ch. Eigler wrote:
> Hi -
> 
> On Mon, Jan 12, 2009 at 01:17:49PM -0500, Masami Hiramatsu wrote:
>> [...]
>> Frank Ch. Eigler wrote:
>>> That is one possible solution for this specific problem -- rawhide
>>> systemtap users who're unable/unwilling to build systemtap out of git
>>> occasoinally.  (Remember that the recent 2.6.28 breakage took a few
>>> hours to fix.)
>>>
>>> Another solution would be for rawhide-style distributions to
>>> aggressively package systemtap snapshots into their development
>>> streams [...]
>> Even if we do that, we have to clarify which package can be applied
>> to which kernel version. 
> 
> In what way?  There would be one package in rawhide, which should work
> on every kernel version that we've ever worked with.  It would be
> replaced frequently - perhaps every few days.

For example, commenting in each snapshot release note.
e.g. - tested on 2.6.28.

>> I'm still not sure when some bugs reported on this ml are fixed -
>> e.g.  http://sources.redhat.com/ml/systemtap/2009-q1/msg00029.html
> 
> There was no bugzilla report associated with it, but it was fixed the
> same day.

Thank you. I think we should notice that fix to reporters.

>> BTW, more those kind of bugfix are committed, more autoconfs are
>> introduced. We already has 15(!) autoconfs,
> 
> Yes, but as you know, their cost has a benefit: they enable our
> operation with a whole spectrum of kernel versions.

I know that it is so useful for older kernels. However, once the
runtime is merged to upstream, we don't need to prepare those autoconfs
and to recompile systemtap itself.

>> and these autoconfs increase compilation time (this will be avoided
>> by caching the result per kernel...).[...]
> 
> FWIW, on my workstation, they seem to add about a second.

Sure, autoconf compilation overhead may not be so big problem...

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Sun, Jan 11, 2009 at 11:29:12AM -0500, Frank Ch. Eigler wrote:
> >    and eventually I suspect markers infrastructure will probably
> >    disappear entirely since tracepoints are perceived as better and
> >    as their replacement.
> 
> (That would probably hurt lttng more than it would systemtap.)

Why do you say that, out of curiosity?  LTTng already has full
tracepoint support, and it is the LTTng maintainer, working in
cooperation with the ftrace developer and others, who has been pushing
tracepoints into mainline.  He's been quite helpful in terms of
working with me so that the ext4 markers can be translated into
tracepoints with no loss of functionality over ext4 markers+systemtap.

> >    Personally, I haven't had a chance to analyze them deeply enough
> >    to know whether this is true, but it's clearly the long-term
> >    direction. [...]
> 
> I hope that before this long-term direction is brought into effect,
> someone does sit down and analyze the issue deeply enough.

What do you believe is wrong with tracepoints?  They are not currently
feature-for-feature identical with markers+Systemtap, today, yes
(which is why I haven't applied LTTng's patch to migrate ext4 markers
to tracepoint, but they are carrying that in the LTTng's kernel git
tree) but aside from needing to write custome probe modules for the
0.01% edge case when I need to filter on something other than a single
device or inode number to reduce the amount of data going to userspace
before I do userspace-side filtering, it looks like it's pretty close.

> You overestimate the amount of duplicated functionality.  For modern
> kernels, systemtap uses the standard in-kernel relay API, with a tiny
> shim.  (If the relay API were ever removed, systemtap would just
> switch to another existing API.  Plain buffering is not rocket
> science.)

Well, if it's not a lot of code, great!  Then it shouldn't be that
hard to prepare the code for upstream submission, assuming you can get
someone like Ingo or Steven to shepherd the code submission.  What is
there does seem to be problematic enough that people need to track
bleeding-edge Systemtap if they want to use bleeding-edge kernels,
which in the long term is a lot of work for all concerned....

      	     	       	    	   	- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi -

> > >    Personally, I haven't had a chance to analyze them deeply enough
> > >    to know whether this is true, but it's clearly the long-term
> > >    direction. [...]
> > 
> > I hope that before this long-term direction is brought into effect,
> > someone does sit down and analyze the issue deeply enough.
> 
> What do you believe is wrong with tracepoints?  [...]

I would not say "wrong", just that they represent a different set of
trade-offs.  Markers are easy to attach to by code that is not
hand-coded for each event type.  Tracepoints (and ftrace engines) are
on the other extreme: they need hand- or hard-coded code for each
individual event.  This implies that generic markers can be used from
systemtap without debuginfo, whereas generic tracepoints need
debuginfo or else future tech in the form of C code parsing.

There are also some other minor issues, such as dormant markers being
lighter weight than dormant tracepoints.


> [...] but aside from needing to write custome probe modules for the
> 0.01% edge case when I need to filter on something other than a
> single device or inode number to reduce the amount of data going to
> userspace before I do userspace-side filtering, it looks like it's
> pretty close.

If that's all you need to do, and you're up for writing all the
kernel- and user-side machinery by hand, that's great.  Many users are
not of that calibre.


> > You overestimate the amount of duplicated functionality.  For modern
> > kernels, systemtap uses the standard in-kernel relay API, with a tiny
> > shim.  (If the relay API were ever removed, systemtap would just
> > switch to another existing API.  Plain buffering is not rocket
> > science.)
> 
> Well, if it's not a lot of code, great!  Then it shouldn't be that
> hard to prepare the code for upstream submission [...]

Perhaps.  I was talking about not a lot of *duplicated functionality*
code.  There is a bunch of code that is highly systemtap-specialized
and approximately useless to "in-tree" users.  I'll review and resend
an older analysis so you can see the specifics of this issue.


> What is there does seem to be problematic enough that people need to
> track bleeding-edge Systemtap if they want to use bleeding-edge
> kernels, which in the long term is a lot of work for all
> concerned....

True, it is *some* work, and it is *ongoing* work.

There is another side to it though.  The systemtap runtime is not
done: it is changing regularly.  If instead of being shipped with
stap, there were a variety of versions of it frozen into given
kernels, we would have to fork the userspace side of systemtap all the
time.  While some may have the luxury of caring solely about the
bleeding edge, we have a broader clientele.

- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Jason Baron]

On Fri, Jan 09, 2009 at 09:48:10PM -0500, Theodore Tso wrote:
> On Tue, Dec 23, 2008 at 07:54:46PM -0500, Masami Hiramatsu wrote:
> > 
> > OK, so I would like to hear your suggestions seriously.
> > (I believe that we know on what we are working).
> > I think it's time to embody action items. Just criticizing can't go
> > things ahead. So, what we can do for you?
> > 
> > - Communication issue
> >  - Notifying releasing on LKML frequently.
> >  - Preparing more documents for kernel developers.
> >  - Feeding back kernel developers' opinions.
> > 
> > - Solving debuginfo issue
> >  - in the short term, notifying how to minimize debuginfo size.
> >  - in the mid term, making dummy (function-entry) debuginfo.
> >  - in the long term, Roland's dwarf compressor can solve this issue.
> > 
> > - Upstream first issue
> >  - Utrace/Uprobe
> >   - Develop code on LKML and embrace feedbacks.
> >   - Involving main kernel maintainer to speed up the code merging.
> >  - Systemtap itself
> >   - Merging a part of runtime and basic tapset to kernel tree, which
> >    will provide us a permanent API for systemtap generated code.
> >   (BTW, I'd like to suggest modifying ftrace plug-able and systemtap
> >    generating ftrace-plugin tracers. This will be more acceptable for
> >    upstream because we don't need to add so many code).
> > 
> > Any other issues and ideas?
> 
> That's a pretty good list.  Other things to think about
> 
> *) One of the really important reasons why SystemTap needs to move
>    runtime into the kernel is because especially for the community
>    distributions, Fedora for example releases regular bleeding-edge
>    kernels so that end users can help provide badly-needed testing of
>    development kernel.  So if Systemtap needs to be updated from git
>    to deal with changes in the development kernel, Fedora users who
>    are using the bleeding edge kernels won't be able to use Systemtap
>    unless it is released very often as Fedora, Open Suse, and
>    otherwise packaged for other community distributions.  If the
>    SystemTap runtime is bundled with the kernel, then it's much more
>    likely that end users who want to use the daily kernel RPM's will
>    also be able to use SystemTap.
> 
> *) Systemtap needs to be adpted to use tracepoints, quickly; the
>    kernel markers for the scheduler have disappeared and replaced with
>    tracepoints.  I've getting pressure to replace ext4's markers with
>    tracepoints, and eventually I suspect markers infrastructure will
>    probably disappear entirely since tracepoints are perceived as
>    better and as their replacement.  Personally, I haven't had a
>    chance to analyze them deeply enough to know whether this is true,
>    but it's clearly the long-term direction.  (And yes, this is part
>    of efforts to provide a tracing solution that assumes SystemTap is
>    not part of the picture.)
> 
> The idea of using the ftrace infrastructure is a good one; since
> SystemTap resisted making an effort get its runtime into the kernel,
> now that the ftrace infrastructure is in the kernel, the principle of
> not merging code that duplicates functionality means that this makes
> Systemtap now needs to adapt what infrastructure did get merged for
> its purposes, since it will be much more difficult get parallel
> functionality merged into the kernel.
> 
> Regards,
> 
> 						- Ted

hi,

We have been actively looking at and adding tracepoints to the lttng
kernel tree via the ltt-dev list to support Systemtap. These tracepoints are
being added at "key" kernel points in the fs, vm, scheduler, and other
subsystems. Unfortunately, we just realized that these tracepoints are not
going to be proposed for a merge until lttng is proposed for merge. Systemtap
can not be held up by this.

Therefore, I was thinking of proposing 100+ tracepoints that are
currently in the lttng tree (and not upstream, but many have already
been reviewed upstream), on lkml. If we also propose Systemtap specific set of 
markers to interface, with these tracepoints, then Systemtap will work out of
the box with no debuginfo, no gcc changes, and be effective immediately to 
filter ext4 debug information.

Longer term, we can look at merging markers into tracepoints, having
Systemtap directly interface with tracepoints, and merging
utrace/probes. This proposal makes Systemtap immediately more useful on 
upstream kernels, while longer term issues are addressed. thoughts?

thanks,

-Jason

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Mon, Jan 12, 2009 at 02:04:01PM -0500, Jason Baron wrote:
> 
> We have been actively looking at and adding tracepoints to the lttng
> kernel tree via the ltt-dev list to support Systemtap. These tracepoints are
> being added at "key" kernel points in the fs, vm, scheduler, and other
> subsystems. Unfortunately, we just realized that these tracepoints are not
> going to be proposed for a merge until lttng is proposed for merge. Systemtap
> can not be held up by this.

Huh?  Last I checked Systemtap didn't support tracepoints at all.  Did
I miss something?

And what what do you mean by "adding tracepoints to support
Systemtap"?  Do you mean that these would help you write better
tapsets, for when Systemtap could support tracepoints?

Also, the trace points won't necessarily be helpd up for merge until
lttng is proposed for merge.  What is necessary is a way to access
those tracepoints without needing some big, harry, complex, userspace
package (whether it is called Lttng or Systemtap), since said packages
often are written with massive distro-dependencies, or are written in
C++ so kernel developers have a hard time customizing/fixing them to
meet their needs, and so on.

So what Linus Torvalds and other senior kernel developers proposed at
the Kernel Summit was a simple debugfs/proc interface which would
allow individual activation of a tracepoint/marker, and which would
dump out the data collected by that marker as a simple text file
accessed via a pseudo-filesystem.  This would be the "in-mainline
user" of the markers/tracepoints, and would guarantee that tracepoints
could be made *useful* by kernel developers using grep, awk, and perl
of that text file.  Simple filtering for bandwidth reasons might be
done via debugfs knobs, only for the 99.9% common cases.

> Therefore, I was thinking of proposing 100+ tracepoints that are
> currently in the lttng tree (and not upstream, but many have already
> been reviewed upstream), on lkml.

Linus has basically said at the Kernel Summit that he was not going to
accept new markers until there was a way to make sure that they could
actually be made *useful* for real, live kernel maintainers, via this
simple text interface.  There were some questions about whether text
or a compressed binary would be used to ship the log to userspace, but
in the latter case, a simple .c file shipped with the kernel sources
in the examples directory must be all that would be necessary to
generate the text stream that would then be processed via grep/awk ---
not a massive out-of-tree C++ program.

I was able to sneak in some markers for ext4, but that's primarily
because it was maintainer's discretion and ext4 isn't in widespread
use and is in late-development stage, so Linus doesn't pay as close
attention.  :-) However, if you are going to try to get 100+
tracepoints into the core kernel, that *will* draw notice, and the
first question people will ask is "what's the in-tree consume of
tracepoints".

I pinged Steven Rostedt at Red Hat, and he indicated that this was
still on his todo list.  So my recommendation to you would be to reach
out to Steven Rostedt, and see if you can help with trying to get the
"simple text output" enhancements to ftrace completed so it can get
merged into mainline.  There has already been general approval of that
game plan for at the Kernel Summit, so this is basically a question of
"Show Me The Code".  Once this is done, getting the tracepoints you
want into the kernel should be relatively straightforward.

> If we also propose Systemtap specific set of 
> markers to interface, with these tracepoints, then Systemtap will work out of
> the box with no debuginfo, no gcc changes, and be effective immediately to 
> filter ext4 debug information.

That assumes that SystemTap can access tracepoints, but I assume
that's a Small Matter of Programming.  :-)

> Longer term, we can look at merging markers into tracepoints, having
> Systemtap directly interface with tracepoints, and merging
> utrace/probes. This proposal makes Systemtap immediately more useful on 
> upstream kernels, while longer term issues are addressed. thoughts?

Markers are probably just going to disappear.  Most of the markers
that were in the core kernel have already disappeared; all that's left
was a handful of Markers arch-specific code, the KVM subsystem, and
the ext4 subsystem.

I don't think you'll be able to get the tracepoints into the main
kernel until we get a way to access tracepoints directly via debugfs
and getting exported output files via either a simple .c file
accessing a binary log output, or simply using "cat
/debugfs/..../foo.txt", but I suspect that can happen very quickly.

I'll admit that once this is done, I may end up using the plain text
output mechanism far more often that SystemTap, since it will be more
conveient that creating stap scripts like this:

probe kernel.mark("ext4_sync_file")
{
	t = gettimeofday_ms();
	printf("%d.%d:ext4_sync_file: dev %s datasync %d ino %d parent %d\n",
		      t / 1000, t % 1000, $arg1, $arg2, $arg3, $arg4)
}

... but maybe that's OK, the SystemTap developers will probably get
many fewer annoying complaints from kernel developers complaining
about how SystemTap doesn't work with 2.6.29-rc1, or how "xmlto pdf"
or elfutils doesn't work exactly the same (or not at all) on
non-RedHat distributions, etc.

						- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

HI -

On Mon, Jan 12, 2009 at 02:51:59PM -0500, Theodore Tso wrote:
> [...]
> > We have been actively looking at and adding tracepoints to the lttng
> > kernel tree via the ltt-dev list to support Systemtap. [...]

> Huh?  Last I checked Systemtap didn't support tracepoints at all.  Did
> I miss something?

Not *directly*.  Just like lttng, systemtap can currently use
tracepoints via a hand-written intermediary module that maps them to
markers.


> Also, the trace points won't necessarily be helpd up for merge until
> lttng is proposed for merge.  [...]
> 
> So what Linus Torvalds and other senior kernel developers proposed at
> the Kernel Summit was a simple debugfs/proc interface which would
> allow individual activation of a tracepoint/marker, and which would
> dump out the data collected by that marker as a simple text file
> accessed via a pseudo-filesystem.  

Consider acking http://lkml.org/lkml/2008/12/30/297, a widget that
aims to connect generic markers to ftrace.


- FChE

Re: Discussion at Linux Foundation Japan Symposium

[Roland McGrath]

Picking up the discussion after the holiday break.  Happy New Year.

> How much more utrace work is needed?  Fedora is shipping it, right?
> What still needs to be done before it can be merged?  The
> characterization that I heard at the plumber's conference was that the
> work was largely done, and it was just a matter of it getting merged.

It is largely done in the sense that it is a new experimental in-kernel API
that we can use now and that will surely get thrashed and refined as we go
on.  I would certainly like to see it merged sooner rather than later, and
then improved in the tree.

The code at http://people.redhat.com/roland/utrace/ and also in
git://git.kernel.org/pub/scm/linux/kernel/git/frob/linux-2.6-utrace.git
is based on the current Linus tree and can merge painlessly.

Since 2.6.2[78], the whole of the new code is utrace proper, and all under
the CONFIG_UTRACE option, which "depends on EXPERIMENTAL".  However broken
utrace might be, you can configure it out and have my patches make zero
code changes in your kernel.

I had hoped that this zero impact for the uninterested would make the
threshold of acceptance for merging utrace reasonably attainable without
much controversy or rigamarole.  However, in various recent conversations,
several kernel folks said we should not submit utrace on its own at all.
They told us a new in-kernel API can't go in without something that uses
it, and "systemtap doesn't count".  They want us to come up with some
completed kernel feature, small or large, that makes use of utrace to do
something or other, that they like and want to merge.  Only then, they
said, should we submit utrace patches, along with more patches adding this
thing that uses the utrace API.

That seems counterproductive to me, frankly.  Since you seem to be
encouraging me to just get utrace merged ASAP, I hope you agree with my
view here.  Everyone who ever says they'd like to help writing nice, useful
things based on the utrace API complains about utrace not being merged and
that this discourages work based on it.  Now other people say that it
mustn't be merged until people have done that work.  Catch 22.
(Everybody's way out is for me to just write everything, apparently.  
Silly other people want me to also work on other projects entirely at the
same time, so this idea has not been getting things done quicker.)

I agree with your attitude about bugs.  If it's merged, we will fix and
change it in the tree as we hash out problems.  People at large find it
easier to consider, test, or help work on the code that way.  It would
certainly be easier for me and for the systemtap community.  People who are
concerned it might be unstable can see "EXPERIMENTAL" when configuring
their kernels and just say "n" (CONFIG_UTRACE is off in defconfig).

Since the kernel summit I got in little time to actually do very much, so
there's really a moderately small amount of change since then.  I had fixed
several bugs and introduced more regressions along the way.  Just today I
finally got a little time to get back to it and fixed most of the
regressions, so it's not in a pathological state.  But it presently has at
least one known intermittent regression in ptrace and a WARN_ON that I'm
confused about, and I'm not certain that it can't still be crashed by some
of the test cases that LKML reviewers showed before (though our suite has
variants of those tests and I haven't seen any crashes lately).  (I'll be
spending some time on the known bugs, but a fairly constrained portion of
my time.  It still remains to be seen what can be done about getting other
hackers' eyeballs active on fixing code in this layer.)

So, please advise me.


Thanks,
Roland

Re: Discussion at Linux Foundation Japan Symposium

[Theodore Tso]

On Thu, Jan 08, 2009 at 01:22:37AM -0800, Roland McGrath wrote:
> 
> Since 2.6.2[78], the whole of the new code is utrace proper, and all under
> the CONFIG_UTRACE option, which "depends on EXPERIMENTAL".  However broken
> utrace might be, you can configure it out and have my patches make zero
> code changes in your kernel.
> 
> I had hoped that this zero impact for the uninterested would make the
> threshold of acceptance for merging utrace reasonably attainable without
> much controversy or rigamarole.  However, in various recent conversations,
> several kernel folks said we should not submit utrace on its own at all.
> They told us a new in-kernel API can't go in without something that uses
> it, and "systemtap doesn't count".  They want us to come up with some
> completed kernel feature, small or large, that makes use of utrace to do
> something or other, that they like and want to merge.  Only then, they
> said, should we submit utrace patches, along with more patches adding this
> thing that uses the utrace API.
> 
> That seems counterproductive to me, frankly.  Since you seem to be
> encouraging me to just get utrace merged ASAP, I hope you agree with my
> view here.  Everyone who ever says they'd like to help writing nice, useful
> things based on the utrace API complains about utrace not being merged and
> that this discourages work based on it.  Now other people say that it
> mustn't be merged until people have done that work.  Catch 22.
> (Everybody's way out is for me to just write everything, apparently.  
> Silly other people want me to also work on other projects entirely at the
> same time, so this idea has not been getting things done quicker.)

Sigh.  So part of the problem with the reputation Systemtap has as a
project out of control which doesn't listen to the kernel development
community is that no one is willing to bend the rules.  Code needs
in-tree users; and out of tree moudles have never counted.  This is
true regardless of whether the out of tree module is something evil,
like Nvidia's proprietary drivers with memory pointer bugs that induce
random memory corruption in Linux kernels, or out of tree filesystems
like OpenAFS (even if they are open source), or SystemTap.

There can and have been exceptions, or talented kernel developers can
find easy ways to make the code useful for purposes.  The one piece of
advice I can give you is to ***ask*** someone like Ingo Molndar or
Steven Rostedt for their help in shepherding the utrace patch into the
kernel.  You needs someone who can vouch for the patch, and can make
the necessary changes and then work the politics to get the patch in
(probably as a replacement back-end for ptrace, and demonstrate that
it is cleaner/faster/better for long-term maintenance than the
existing ptrace code).

Look, let me be blunt.  Frank doesn't seem to believe me when I say
that Red Hat kernel engineers I've talked to are massively dismissive
about SystemTap; and apparently when a member of the LF Technical
Advisory Board talked to Brian Stevens about the perceived dysfunction
of the SystemTap team, nothing happened.  This fact, combined with the
observation that apparently Red Hat kernel developers are unwilling to
speak aloud problems with SystemTap internally within Red Hat, *and*
the fact that SystemTap hasn't been cancelled despite its many years
of toilage with very little results to show for it, leads me to
believe that SystemTap has massive political clout and masive
political support within Red Hat in support of the project.  

If that is true, I would suggest you *use* some of that massive amount
of political clout to get a senior Red Hat Kernel developer assigned
to the project to render assistance.  You need the help; seriously.
Otherwise, I very much doubt you'll be able to get kernel developers
to bend the rules that are in force for all other code submissions.

Things might have been different if SystemTap team had been more
willing to listen to kernel developers, and less resistant to
preferences of moving systemtap support into the kernel, maybe there
would be some positive karma that you could have drawn upon, and
gotten more help from various kernel developers.  But at this point,
anything SystemTap related starts with a huge negative bias against
it, especially when the general assumption from the kernel developers
is that if they ever hope to have something which is useful to them
and to others, they will need to do it themselves.

The work replacing markers with tracepoints, and the enhancements in
the ftrace framework, are all part of a general effort to provide a
general-purpose tracing facility so that SystemTap can be pushed
aside.  I'm not sure if anyone has a good story for userspace tracing
that doens't involve SystemTap, so I suspect that's going to be
SystemTap's last saving throw.

But what I would do, since you asked for my advice, is:

*) Reach out through Red Hat Management.  Tell them that several years
of investment in SystemTap is at risk.  Get a senior Red Hat
Kernel developer assigned to help out.

*) **Listen** to that developer's recommendations.  It will probably
involve getting the SystemTap kernel support code into the kernel, and
probably many other changes.  You may think it's bullshit, but don't
blow him off the way you've done with me or James' suggestions.  You
*need* his expertise to make the chnages acceptable to the standard
mainstream kernel acceptance criteria, and you *need* his good well so
he's willing to vouch for the code.  Hey, maybe my word is worthless,
and I don't know what the hell I'm talking about, but if someone like
Ingo Molnar tells you "You Need To Do This", maybe you'll pay
attention to him.

*) And do this quickly.  The longer you wait, the longer more people
will have time to work on alternatives to SystemTap, and the harder it
will be for people to believe that SystemTap is worth saving.

     	    	      	      	   	     	- Ted

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Ted,

Theodore Tso wrote:
> 3) Actually try to find a way for Systemtap to be more useful
> *without* waiting for Godot^H^H^H^H^H^H some hypothetical future,
> probably at least 18-24 months away, when new gcc and toolchain
> becomes available that will magically and mystically Solve All
> Problems.  Consider how ftrace works; it very cleverly uses/abuses the
> profiling hook inserted gcc -pg.  Note that it doesn't wait for magic
> new capabilities in the toolchain.  

Actually, I'd like to find a way (or a bridge of the gap amoung us)
to solve this problem positively...

Hm, what issues are making systemtap useless?

- Using debuginfo
 Pros.
  - with debuginfo, systemtap can access local variables and
    query members of data structures by name.
  - debuginfo also enables line-by-line tracing(need to be improved),
    userspace probing, and so on.
  - kernel developers can build its debuginfo by themselves.
 Cons.
  - debuginfo is too huge to distribute all developers and users.
  - debuginfo is too huge and complex so that it increases compilation
    time and disk space consumption.
  - most of developers may not need line-by-line tracing(and it
    might not work in some cases). they just want to trace function
    boundary (maybe, with parameters).

- Module-base
 Pros.
  - Systemtap allows you to filter logging information based on conditions.
  - no-module configuration is not popular...
 Cons.
  - some people build kernel with no-module configuration.

- Non upstream first
 Pros.
  - systemtap is just a tool.
  - systemtap can catch up with upstream in several days after releasing.
 Cons.
  - systemtap works as a tool, but it strongly depends on the kernel.
    if upstream kernel is updated, systemtap will be broken.
  - because of waiting for systemtap update, kernel developers can't
    use systemtap on the development kernel.
  - maintenance cost will increase, especially on runtime and tapset.
  - utrace and uprobe are a part of kernel, but no patch was posted
    these months.

Are these items right? (I think some items should be updated.)


> Another solution which I've suggested in the past, but which has been
> utterly ignored, (since, after all, what does a Kernel Developer
> know?) is to simply generate the debuginfo information on demand so
> the function parameters on entry can be decoded.  We have the header
> files after all, so it shouldn't be hard to generate a pseudo function:
> 
> extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
>        void *ptr)
> {
> 	return 0;
> }
> 
> ... and then use that to generate the debuginfo information for that
> information so Systemtap knows how to decode the function parameters
> on entry.  This would be enough such that Systemtap can probe any
> public function in System.map --- and obtain the function parameters
> for any function which is present in the kernel headers --- all
> without requiring users to download or create hundreds of megabytes of
> largely useless (on a value-per-megabyte-basis) !@#!@# debuginfo
> information.

Good idea! It seems much easier than introducing new CTF-like compressed
debuginfo... why did anyone ignore...?

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Discussion at Linux Foundation Japan Symposium

[Satoshi OSHIMA]

Hi all,

Long time no see and sorry for my late report.

I attended 9th Linux Foundation Japan Symposium and 
discussed on issues of systemtap project with Ted Ts'o, 
James Bottomley and Jonathan Corbet.

In my understanding, they demand the following things:

(1) Follow upstream first

Utrace and uprobe features are currently available only 
on Fedora and Red Hat Enterprise Linux, since those 
patches are not merged into upstream kernel yet.

my suggestion:

To reduce complaints of upstream kernel developers, 
systemtap project may need to postpone adding new 
uprobe features until getting utrace (and uprobe) 
patch set accepted in mainline.


(2) Maintain tapset

Systemtap users (including kernel developers) get 
frustrated because tapsets often do not work on 
the latest kernel. Moreover, sometimes users 
have to fix the tapset incompatibility of kernels.

my suggestion:

If systemtap procjet can fix this kind of incompatibilities
within a few hours or days as Myths about systemtap 
on the wiki claims, releasing new systemtap minor release
tarball for each upstream kernel release would help users.


(3) Make no debuginfo version

Systemtap always requires kernel debuginfo to use. 
Unfortunately, it is hard for users of some distributions 
to have debuginfo.

my suggestion:

If systemtap has a build option to make no debuginfo version,
this complain will be reduced. I know we had had it before.
We should provide it again.


(4) Have conversations frequently with Kernel Community

I understand that Frank has tried to communicate with upstream
kernel community. However, it seems that developers of upstream 
kernel feel it is not enough.

my suggestion:

I know that systemtap is a bit different from other part of
the kernel. Usual kernel subsystem maintainers are chosen 
on activities in lkml. On the other hand, systemtap maintainer's
activities are invisible for almost all of the kernel developers.

This may be one of the reasons of their frastration.
To solve this problem, we should periodically make announcements
of systemtap update and require questions or comments.

To do this, systemtap project might need more ambassadors
for kernel community.

In addition, since criticisms of systemtap occur in events sponsored
by Linux Foundation, systemtap project ambassadaor(s) should talk 
with Linux Foundation people(Andrew Morton, Ted, James).


I hope my suggestion help you to understand the current 
developers feelings.

Regards,
Satoshi Oshima

Re: Discussion at Linux Foundation Japan Symposium

[KOSAKI Motohiro]

Hi!

> Hi all,
> 
> Long time no see and sorry for my late report.
> 
> I attended 9th Linux Foundation Japan Symposium and 
> discussed on issues of systemtap project with Ted Ts'o, 
> James Bottomley and Jonathan Corbet.
> 
> In my understanding, they demand the following things:

Great!


> (1) Follow upstream first
> 
> Utrace and uprobe features are currently available only 
> on Fedora and Red Hat Enterprise Linux, since those 
> patches are not merged into upstream kernel yet.
> 
> my suggestion:
> 
> To reduce complaints of upstream kernel developers, 
> systemtap project may need to postpone adding new 
> uprobe features until getting utrace (and uprobe) 
> patch set accepted in mainline.

Sure!

> (2) Maintain tapset
> 
> Systemtap users (including kernel developers) get 
> frustrated because tapsets often do not work on 
> the latest kernel. Moreover, sometimes users 
> have to fix the tapset incompatibility of kernels.
> 
> my suggestion:
> 
> If systemtap procjet can fix this kind of incompatibilities
> within a few hours or days as Myths about systemtap 
> on the wiki claims, releasing new systemtap minor release
> tarball for each upstream kernel release would help users.

In recently lttng discusstion, Andrew Morton sugessted to don't only
merge lttng kernel part, but also merge userland tool.

I think SystemTap also can try above approach. it is happy for
mainline kernel developer.

Honestly, current SystemTap can't use so easily for mainline kernel
developer.
at least, I tried it in this year five times. but all challenge is fail ;-)

I wonder why systemtap doesn't merge any part into mainline.
At least, systemtap runtime depend on kernel interface largely.
merging is better, imho.


Otherwise, lkml developer (include other tracing developer, e.g. mathieu, steven)
don't understand systemtap design and requirement correctly.


Thanks!

Re: Discussion at Linux Foundation Japan Symposium

[Jun Koi]

On Thu, Dec 18, 2008 at 5:09 PM, Satoshi OSHIMA
<satoshi.oshima.fk@hitachi.com> wrote:
> Hi all,
>
> Long time no see and sorry for my late report.
>
> I attended 9th Linux Foundation Japan Symposium and
> discussed on issues of systemtap project with Ted Ts'o,
> James Bottomley and Jonathan Corbet.
>
> In my understanding, they demand the following things:
>
> (1) Follow upstream first
>
> Utrace and uprobe features are currently available only
> on Fedora and Red Hat Enterprise Linux, since those
> patches are not merged into upstream kernel yet.
>
> my suggestion:
>
> To reduce complaints of upstream kernel developers,
> systemtap project may need to postpone adding new
> uprobe features until getting utrace (and uprobe)
> patch set accepted in mainline.
>
>
> (2) Maintain tapset
>
> Systemtap users (including kernel developers) get
> frustrated because tapsets often do not work on
> the latest kernel. Moreover, sometimes users
> have to fix the tapset incompatibility of kernels.
>
> my suggestion:
>
> If systemtap procjet can fix this kind of incompatibilities
> within a few hours or days as Myths about systemtap
> on the wiki claims, releasing new systemtap minor release
> tarball for each upstream kernel release would help users.
>
>
> (3) Make no debuginfo version
>
> Systemtap always requires kernel debuginfo to use.
> Unfortunately, it is hard for users of some distributions
> to have debuginfo.
>

How is it possible to do that without kernel debug info? Currently
systemtap extracts lots of information on kernel layout from debug
info, so I dont understand why we can survive without that.

Thanks,
Jun

Re: Discussion at Linux Foundation Japan Symposium

[jidong xiao]

On Thu, Dec 18, 2008 at 4:57 PM, Jun Koi <junkoi2004@gmail.com> wrote:
> On Thu, Dec 18, 2008 at 5:09 PM, Satoshi OSHIMA
> <satoshi.oshima.fk@hitachi.com> wrote:
>> Hi all,
>>
>> Long time no see and sorry for my late report.
>>
>> I attended 9th Linux Foundation Japan Symposium and
>> discussed on issues of systemtap project with Ted Ts'o,
>> James Bottomley and Jonathan Corbet.
>>
>> In my understanding, they demand the following things:
>>
>> (1) Follow upstream first
>>
>> Utrace and uprobe features are currently available only
>> on Fedora and Red Hat Enterprise Linux, since those
>> patches are not merged into upstream kernel yet.
>>
>> my suggestion:
>>
>> To reduce complaints of upstream kernel developers,
>> systemtap project may need to postpone adding new
>> uprobe features until getting utrace (and uprobe)
>> patch set accepted in mainline.
>>
>>
>> (2) Maintain tapset
>>
>> Systemtap users (including kernel developers) get
>> frustrated because tapsets often do not work on
>> the latest kernel. Moreover, sometimes users
>> have to fix the tapset incompatibility of kernels.
>>
>> my suggestion:
>>
>> If systemtap procjet can fix this kind of incompatibilities
>> within a few hours or days as Myths about systemtap
>> on the wiki claims, releasing new systemtap minor release
>> tarball for each upstream kernel release would help users.
>>
>>
>> (3) Make no debuginfo version
>>
>> Systemtap always requires kernel debuginfo to use.
>> Unfortunately, it is hard for users of some distributions
>> to have debuginfo.
>>
>
> How is it possible to do that without kernel debug info? Currently
> systemtap extracts lots of information on kernel layout from debug
> info, so I dont understand why we can survive without that.
>
> Thanks,
> Jun
>
But not every distribution contains kernel debuginfo packages,
therefore this makes many people not easy to use systemtap.

Regards
Jason

Re: Discussion at Linux Foundation Japan Symposium

[Jun Koi]

On Thu, Dec 18, 2008 at 6:06 PM, jidong xiao <jidong.xiao@gmail.com> wrote:
> On Thu, Dec 18, 2008 at 4:57 PM, Jun Koi <junkoi2004@gmail.com> wrote:
>> On Thu, Dec 18, 2008 at 5:09 PM, Satoshi OSHIMA
>> <satoshi.oshima.fk@hitachi.com> wrote:
>>> Hi all,
>>>
>>> Long time no see and sorry for my late report.
>>>
>>> I attended 9th Linux Foundation Japan Symposium and
>>> discussed on issues of systemtap project with Ted Ts'o,
>>> James Bottomley and Jonathan Corbet.
>>>
>>> In my understanding, they demand the following things:
>>>
>>> (1) Follow upstream first
>>>
>>> Utrace and uprobe features are currently available only
>>> on Fedora and Red Hat Enterprise Linux, since those
>>> patches are not merged into upstream kernel yet.
>>>
>>> my suggestion:
>>>
>>> To reduce complaints of upstream kernel developers,
>>> systemtap project may need to postpone adding new
>>> uprobe features until getting utrace (and uprobe)
>>> patch set accepted in mainline.
>>>
>>>
>>> (2) Maintain tapset
>>>
>>> Systemtap users (including kernel developers) get
>>> frustrated because tapsets often do not work on
>>> the latest kernel. Moreover, sometimes users
>>> have to fix the tapset incompatibility of kernels.
>>>
>>> my suggestion:
>>>
>>> If systemtap procjet can fix this kind of incompatibilities
>>> within a few hours or days as Myths about systemtap
>>> on the wiki claims, releasing new systemtap minor release
>>> tarball for each upstream kernel release would help users.
>>>
>>>
>>> (3) Make no debuginfo version
>>>
>>> Systemtap always requires kernel debuginfo to use.
>>> Unfortunately, it is hard for users of some distributions
>>> to have debuginfo.
>>>
>>
>> How is it possible to do that without kernel debug info? Currently
>> systemtap extracts lots of information on kernel layout from debug
>> info, so I dont understand why we can survive without that.
>>
>> Thanks,
>> Jun
>>
> But not every distribution contains kernel debuginfo packages,
> therefore this makes many people not easy to use systemtap.
>

But my question is: how can we survive without debug info? In that
case, we cannot tap into kernel anymore? (I guess tap on userspace
still works)


Thanks,
Jun

Re: Discussion at Linux Foundation Japan Symposium

[KOSAKI Motohiro]

> > (3) Make no debuginfo version
> >
> > Systemtap always requires kernel debuginfo to use.
> > Unfortunately, it is hard for users of some distributions
> > to have debuginfo.
> >
> 
> How is it possible to do that without kernel debug info? Currently
> systemtap extracts lots of information on kernel layout from debug
> info, so I dont understand why we can survive without that.

At least, dtrace don't need debuginfo.
On Solaris 10, kernel and all executable binary have own tracing purpose
information in special elf section.
it can't be stripped and gurantee exist although product application binary.

http://blogs.sun.com/ali/entry/what_is_sunw_ldynsym


Of cource, this approach indicate systemtap lost some feature.
(e.g. above section don't have line number information and local
 variable name information)

but it is still useful for average tracing user.

Re: Discussion at Linux Foundation Japan Symposium

[Jun Koi]

On Thu, Dec 18, 2008 at 6:27 PM, KOSAKI Motohiro
<kosaki.motohiro@jp.fujitsu.com> wrote:
>> > (3) Make no debuginfo version
>> >
>> > Systemtap always requires kernel debuginfo to use.
>> > Unfortunately, it is hard for users of some distributions
>> > to have debuginfo.
>> >
>>
>> How is it possible to do that without kernel debug info? Currently
>> systemtap extracts lots of information on kernel layout from debug
>> info, so I dont understand why we can survive without that.
>
> At least, dtrace don't need debuginfo.
> On Solaris 10, kernel and all executable binary have own tracing purpose
> information in special elf section.
> it can't be stripped and gurantee exist although product application binary.
>
> http://blogs.sun.com/ali/entry/what_is_sunw_ldynsym
>
>
> Of cource, this approach indicate systemtap lost some feature.
> (e.g. above section don't have line number information and local
>  variable name information)
>
> but it is still useful for average tracing user.

So the way is to include, by default, debug information we need into
kernel binary? Yes, that solves the problem, provided that they
(kernel developers) accept that.

Thanks,
Jun

Re: Discussion at Linux Foundation Japan Symposium

[Jun Koi]

On Thu, Dec 18, 2008 at 6:34 PM, Jun Koi <junkoi2004@gmail.com> wrote:
> On Thu, Dec 18, 2008 at 6:27 PM, KOSAKI Motohiro
> <kosaki.motohiro@jp.fujitsu.com> wrote:
>>> > (3) Make no debuginfo version
>>> >
>>> > Systemtap always requires kernel debuginfo to use.
>>> > Unfortunately, it is hard for users of some distributions
>>> > to have debuginfo.
>>> >
>>>
>>> How is it possible to do that without kernel debug info? Currently
>>> systemtap extracts lots of information on kernel layout from debug
>>> info, so I dont understand why we can survive without that.
>>
>> At least, dtrace don't need debuginfo.
>> On Solaris 10, kernel and all executable binary have own tracing purpose
>> information in special elf section.
>> it can't be stripped and gurantee exist although product application binary.
>>
>> http://blogs.sun.com/ali/entry/what_is_sunw_ldynsym
>>
>>
>> Of cource, this approach indicate systemtap lost some feature.
>> (e.g. above section don't have line number information and local
>>  variable name information)
>>
>> but it is still useful for average tracing user.
>
> So the way is to include, by default, debug information we need into
> kernel binary? Yes, that solves the problem, provided that they
> (kernel developers) accept that.
>

The downside is that the kernel binary becomes much much bigger.

Thanks,
Jun
>

Re: Discussion at Linux Foundation Japan Symposium

[KOSAKI Motohiro]

> >> At least, dtrace don't need debuginfo.
> >> On Solaris 10, kernel and all executable binary have own tracing purpose
> >> information in special elf section.
> >> it can't be stripped and gurantee exist although product application binary.
> >>
> >> http://blogs.sun.com/ali/entry/what_is_sunw_ldynsym
> >>
> >>
> >> Of cource, this approach indicate systemtap lost some feature.
> >> (e.g. above section don't have line number information and local
> >>  variable name information)
> >>
> >> but it is still useful for average tracing user.
> >
> > So the way is to include, by default, debug information we need into
> > kernel binary? Yes, that solves the problem, provided that they
> > (kernel developers) accept that.
> >
> 
> The downside is that the kernel binary becomes much much bigger.

No.

I agree with full DWARF information is _very_ big.
but if we want only dtrace level tracing, tracing section is less than few mega byte.


Then, I suggest to implement restrected (dtrace level) tracing feature.
only the user of using debuginfo can do full tracing.

Actually, average tracing user don't need line number proving.

Re: Discussion at Linux Foundation Japan Symposium

[K.Prasad]

On Thu, Dec 18, 2008 at 06:27:32PM +0900, KOSAKI Motohiro wrote:
> > > (3) Make no debuginfo version
> > >
> > > Systemtap always requires kernel debuginfo to use.
> > > Unfortunately, it is hard for users of some distributions
> > > to have debuginfo.
> > >
> > 
> > How is it possible to do that without kernel debug info? Currently
> > systemtap extracts lots of information on kernel layout from debug
> > info, so I dont understand why we can survive without that.
> 
> At least, dtrace don't need debuginfo.
> On Solaris 10, kernel and all executable binary have own tracing purpose
> information in special elf section.
> it can't be stripped and gurantee exist although product application binary.
> 
> http://blogs.sun.com/ali/entry/what_is_sunw_ldynsym
> 
> 
> Of cource, this approach indicate systemtap lost some feature.
> (e.g. above section don't have line number information and local
>  variable name information)
> 
> but it is still useful for average tracing user.
>
I'm not sure if the issue is discussed without being aware of the
dwarfless probing that's available in SystemTap since a while ago. More
details at: http://sourceware.org/systemtap/wiki/MakeDoWithoutDebugInfo

Thanks,
K.Prasad
 

Re: Discussion at Linux Foundation Japan Symposium

[KOSAKI Motohiro]

> > Of cource, this approach indicate systemtap lost some feature.
> > (e.g. above section don't have line number information and local
> >  variable name information)
> > 
> > but it is still useful for average tracing user.
> >
> I'm not sure if the issue is discussed without being aware of the
> dwarfless probing that's available in SystemTap since a while ago. More
> details at: http://sourceware.org/systemtap/wiki/MakeDoWithoutDebugInfo

Cool.

but, I feel "must asmlinkage" is too strong restriction.
Now, ftrace can trace any function by -pg compile option and
self modification code.

Can we do the same thing?
I think ftrace approach have very few performance degression and
we can apply production kernel.

Re: Discussion at Linux Foundation Japan Symposium

[K.Prasad]

On Thu, Dec 18, 2008 at 06:57:57PM +0900, KOSAKI Motohiro wrote:
> > > Of cource, this approach indicate systemtap lost some feature.
> > > (e.g. above section don't have line number information and local
> > >  variable name information)
> > > 
> > > but it is still useful for average tracing user.
> > >
> > I'm not sure if the issue is discussed without being aware of the
> > dwarfless probing that's available in SystemTap since a while ago. More
> > details at: http://sourceware.org/systemtap/wiki/MakeDoWithoutDebugInfo
> 
> Cool.
> 
> but, I feel "must asmlinkage" is too strong restriction.
> Now, ftrace can trace any function by -pg compile option and
> self modification code.
> 
> Can we do the same thing?
> I think ftrace approach have very few performance degression and
> we can apply production kernel.
>

investigate ftrace interface:
http://sources.redhat.com/bugzilla/show_bug.cgi?id=6594

Thanks,
K.Prasad

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

KOSAKI Motohiro wrote:
>>> Of cource, this approach indicate systemtap lost some feature.
>>> (e.g. above section don't have line number information and local
>>>  variable name information)
>>>
>>> but it is still useful for average tracing user.
>>>
>> I'm not sure if the issue is discussed without being aware of the
>> dwarfless probing that's available in SystemTap since a while ago. More
>> details at: http://sourceware.org/systemtap/wiki/MakeDoWithoutDebugInfo
> 
> Cool.
> 
> but, I feel "must asmlinkage" is too strong restriction.

Agreed, logically, no-asmlinkage function can be also easily traced
at least x86/x86-64. (I'm not sure whether we can identify which
functions are asmlinkage or not.)

Thanks,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Satoshi,

Thank you for reporting!

Satoshi OSHIMA wrote:
> Hi all,
> 
> Long time no see and sorry for my late report.
> 
> I attended 9th Linux Foundation Japan Symposium and 
> discussed on issues of systemtap project with Ted Ts'o, 
> James Bottomley and Jonathan Corbet.

FYI, we also discuss this topic on below bugzilla.
http://sourceware.org/bugzilla/show_bug.cgi?id=7042

There are links to videos of the symposium on that bugzilla.

> In my understanding, they demand the following things:
> 
> (1) Follow upstream first
> 
> Utrace and uprobe features are currently available only 
> on Fedora and Red Hat Enterprise Linux, since those 
> patches are not merged into upstream kernel yet.
> 
> my suggestion:
> 
> To reduce complaints of upstream kernel developers, 
> systemtap project may need to postpone adding new 
> uprobe features until getting utrace (and uprobe) 
> patch set accepted in mainline.
> 
> 
> (2) Maintain tapset
> 
> Systemtap users (including kernel developers) get 
> frustrated because tapsets often do not work on 
> the latest kernel. Moreover, sometimes users 
> have to fix the tapset incompatibility of kernels.
> 
> my suggestion:
> 
> If systemtap procjet can fix this kind of incompatibilities
> within a few hours or days as Myths about systemtap 
> on the wiki claims, releasing new systemtap minor release
> tarball for each upstream kernel release would help users.

Agreed, We have a weekly snapshot for it. however, that means
it could delay 1 week. And also, we should provide which
release can work on the latest kernel.

> (3) Make no debuginfo version
> 
> Systemtap always requires kernel debuginfo to use. 
> Unfortunately, it is hard for users of some distributions 
> to have debuginfo.
> 
> my suggestion:
> 
> If systemtap has a build option to make no debuginfo version,
> this complain will be reduced. I know we had had it before.
> We should provide it again.

As Prasad said, systemtap had supported no-dwarf mode, however,
currently it is disabled (due to uprobe issue?). I strongly
recommend to re-enable it only for kernel-space probes as soon as possible.

> (4) Have conversations frequently with Kernel Community
> 
> I understand that Frank has tried to communicate with upstream
> kernel community. However, it seems that developers of upstream 
> kernel feel it is not enough.
> 
> my suggestion:
> 
> I know that systemtap is a bit different from other part of
> the kernel. Usual kernel subsystem maintainers are chosen 
> on activities in lkml. On the other hand, systemtap maintainer's
> activities are invisible for almost all of the kernel developers.
> 
> This may be one of the reasons of their frastration.
> To solve this problem, we should periodically make announcements
> of systemtap update and require questions or comments.

From Linux foundation video, I think we also would better work
with some kernel (subsystem) maintainers, and ask them
what they need to use systemtap easy.

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Satoshi OSHIMA <satoshi.oshima.fk@hitachi.com> writes:

> [...]  Utrace and uprobe features are currently available only on
> Fedora and Red Hat Enterprise Linux, since those patches are not
> merged into upstream kernel yet.

That's correct, but we all hope this is temporary.  Both utrace and
uprobes developers are keen to push the work into the kernel ASAP.

> To reduce complaints of upstream kernel developers, systemtap
> project may need to postpone adding new uprobe features until
> getting utrace (and uprobe) patch set accepted in mainline.

I am not sure I understand - surely kernel developers are not offended
by our delightful and necessary progress in user-space probing.  Even
if they were, the cure for that is not for us to stop development in
systemtap proper, but for them and others to offer support for
utrace/uprobes to get that code upstream.


> (2) Maintain tapset

> Systemtap users (including kernel developers) get frustrated because
> tapsets often do not work on the latest kernel.

It would be very helpful if people who run into actual problems got in
touch with us.  It is disheartening to hear of someone trying to run
systemtap against some kernel versions, failing for some reason, and
giving up instead of asking for help or reporting the bug.


> my suggestion:
>
> If systemtap project can fix this kind of incompatibilities
> within a few hours or days as Myths about systemtap 
> on the wiki claims, releasing new systemtap minor release
> tarball for each upstream kernel release would help users.

That particular myth entry was more about runtime incompatibilities
that prevent building at all.  Those things are indeed fixed quickly.
Do you think that snapshots even more frequent than the weekly
snapshots are likely to be used?  Or perhaps we could tag the git tree
with kernel release numbers that it has been tested against.

Script tapsets are somewhat fuzzier, in that we may have one or two
test suite failures but nothing catastrophic happen if they go
out-of-date.  At the kernel summit the preferred solution appeared to
be to have kernel people insert static more instrumentation that we
could also use.


> (3) Make no debuginfo version

> Systemtap always requires kernel debuginfo to use.  Unfortunately,
> it is hard for users of some distributions to have debuginfo.

Those distributions ought to be lobbied to improve this situation,
since there are many non-systemtap uses of debugging data - such as
debuggers, kernel- and user-space crash dump analysis tools.

> my suggestion:
>
> If systemtap has a build option to make no debuginfo version,
> this complain will be reduced. I know we had had it before.
> We should provide it again.

The limitations of operating without debugging information are severe,
but indeed we should revive this functionality.

If tools appear to package CTF-like debuginfo subsets, and if linux
developers/distributors are willing to use them, systemtap should of
course support it too.


- FChE

Re: Discussion at Linux Foundation Japan Symposium

[KOSAKI Motohiro]

> > (3) Make no debuginfo version
> 
> > Systemtap always requires kernel debuginfo to use.  Unfortunately,
> > it is hard for users of some distributions to have debuginfo.
> 
> Those distributions ought to be lobbied to improve this situation,
> since there are many non-systemtap uses of debugging data - such as
> debuggers, kernel- and user-space crash dump analysis tools.

I don't agree.

In enterprise usage, crash dump analysis is provided by vendor.
but tapset run on customer.

They use the same OS/kernel. but different person, different machine.

Re: Discussion at Linux Foundation Japan Symposium

[Frank Ch. Eigler]

Hi -

On Fri, Dec 19, 2008 at 08:58:58AM +0900, KOSAKI Motohiro wrote:
> > > Systemtap always requires kernel debuginfo to use.  Unfortunately,
> > > it is hard for users of some distributions to have debuginfo.
> > 
> > Those distributions ought to be lobbied to improve this situation,
> > since there are many non-systemtap uses of debugging data - such as
> > debuggers, kernel- and user-space crash dump analysis tools.
> 
> [...]
> In enterprise usage, crash dump analysis is provided by vendor.
> but tapset run on customer. [...]

That is just one scenario.  Another compelling use for debuginfo is a
tool like oprofile.

- FChE

Re: Discussion at Linux Foundation Japan Symposium

[KOSAKI Motohiro]

> > > > Systemtap always requires kernel debuginfo to use.  Unfortunately,
> > > > it is hard for users of some distributions to have debuginfo.
> > > 
> > > Those distributions ought to be lobbied to improve this situation,
> > > since there are many non-systemtap uses of debugging data - such as
> > > debuggers, kernel- and user-space crash dump analysis tools.
> > 
> > [...]
> > In enterprise usage, crash dump analysis is provided by vendor.
> > but tapset run on customer. [...]
> 
> That is just one scenario.  Another compelling use for debuginfo is a
> tool like oprofile.

Yes and No.

I agreed with Oprofile also need debuginfo.
However, enterprise customer doesn't use Oprofile too.

From enterprise customer view, that is consultant or vendor business.

Re: Discussion at Linux Foundation Japan Symposium

[William Cohen]

Frank Ch. Eigler wrote:
> Hi -
> 
> On Fri, Dec 19, 2008 at 08:58:58AM +0900, KOSAKI Motohiro wrote:
>>>> Systemtap always requires kernel debuginfo to use.  Unfortunately,
>>>> it is hard for users of some distributions to have debuginfo.
>>> Those distributions ought to be lobbied to improve this situation,
>>> since there are many non-systemtap uses of debugging data - such as
>>> debuggers, kernel- and user-space crash dump analysis tools.
>> [...]
>> In enterprise usage, crash dump analysis is provided by vendor.
>> but tapset run on customer. [...]
> 
> That is just one scenario.  Another compelling use for debuginfo is a
> tool like oprofile.
> 
> - FChE

For the case where oprofile data is being mapped back to lines in the source 
code debuginfo is needed. However, for the case just want to know which 
functions the code is spending time in oprofile doesn't use debugging 
information produced by -g; it just looks for the function locations in the 
code.  Unfortunately, the strip operations used for binaries in distribution 
strip out all information from the binary leaving just the minimal executable 
bits, making the debuginfo the only place to get that information.

It would be nice if there was something between the extremes of a huge amount of 
information in the debuginfo or nothing at all.


-Will

Re: Discussion at Linux Foundation Japan Symposium

[Richard J Moore]

systemtap-owner@sourceware.org wrote on 19/12/2008 19:53:35:

> [image removed]
>
> Re: Discussion at Linux Foundation Japan Symposium
>
> William Cohen
>
> to:
>
> Frank Ch. Eigler
>
> 19/12/2008 19:54
>
> Sent by:
>
> systemtap-owner@sourceware.org
>
> Cc:
>
> KOSAKI Motohiro, Satoshi OSHIMA, systemtap, mhiramat,
> hisashi.hashimoto.wh, Yumiko SUGITA
>
> Frank Ch. Eigler wrote:
> > Hi -
> >
> > On Fri, Dec 19, 2008 at 08:58:58AM +0900, KOSAKI Motohiro wrote:
> >>>> Systemtap always requires kernel debuginfo to use.  Unfortunately,
> >>>> it is hard for users of some distributions to have debuginfo.
> >>> Those distributions ought to be lobbied to improve this situation,
> >>> since there are many non-systemtap uses of debugging data - such as
> >>> debuggers, kernel- and user-space crash dump analysis tools.
> >> [...]
> >> In enterprise usage, crash dump analysis is provided by vendor.
> >> but tapset run on customer. [...]
> >
> > That is just one scenario.  Another compelling use for debuginfo is a
> > tool like oprofile.
> >
> > - FChE
>
> For the case where oprofile data is being mapped back to lines in the
source
> code debuginfo is needed. However, for the case just want to know which
> functions the code is spending time in oprofile doesn't use debugging
> information produced by -g; it just looks for the function locations in
the
> code.  Unfortunately, the strip operations used for binaries in
distribution
> strip out all information from the binary leaving just the minimal
executable
> bits, making the debuginfo the only place to get that information.
>
> It would be nice if there was something between the extremes of a
> huge amount of
> information in the debuginfo or nothing at all.
>
>
> -Will

I attempted to reply to this earlier but it never made the mailing list,
apologies for repeating myself to those who did read my earlier response.:

My suggestion and hope is that we can weakening the dependency between
SystemTap
and debuginfo. On taking a look back to the antecedents to kprobes i.e
dprobes
and before that dtrace for OS/2 we didn't rely explicitly on debuginfo.

Both dprobes and dtrace would use the debuginfo if available but would also
work
with public symbols in the module header and with expressions based on
these
symbols.

Is this useable? Yes it is:

for one-off debugging scenarios I often used a crashdump to begin the
anbalysis
and would follow up with a probe based on what I say in the dump. It's
relatively easy, when dealing with the binary data to develop probe
expressions
based on offsets to public symbols. This would be useful but would require
System
Tap to change.

For performance investigation and regular tracing, the probes can generally
be
developed in advance of their need. What we did with OS/2 was to build
probe
binaries based on debuginfo as part of the module build. Once the binary
was build
there was no need for debuginfo to be present to execute the probe. So we
stripped
the debuginfo from the module and shipped the stripped module along with
its
probe binary.

We could do something similar with SystemTap especially for the pre-built
probe
libraries. Debug info is only really needed when developing probes without
access
to a dump. I think a lot of mileage could be gained from System Tap by
having
pre-built probe modules that matched a given distro service pack or kernel
build.
We wouldn't need to change System Tap to support this. We could then
concentrate
on providing a host of useful pre-built probe modules.

Richard M.

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

William Cohen wrote:
> For the case where oprofile data is being mapped back to lines in the 
> source code debuginfo is needed. However, for the case just want to know 
> which functions the code is spending time in oprofile doesn't use 
> debugging information produced by -g; it just looks for the function 
> locations in the code.  Unfortunately, the strip operations used for 
> binaries in distribution strip out all information from the binary 
> leaving just the minimal executable bits, making the debuginfo the only 
> place to get that information.
> 
> It would be nice if there was something between the extremes of a huge 
> amount of information in the debuginfo or nothing at all.

Agreed, I think CTF which Frank commented is one possible candidate, right?
I think we should involve gcc developers if we propose this kind of diet
debuginfo. They might know a better way to do that.

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: Discussion at Linux Foundation Japan Symposium

[Masami Hiramatsu]

Hi Frank,

Frank Ch. Eigler wrote:
> If tools appear to package CTF-like debuginfo subsets, and if linux
> developers/distributors are willing to use them, systemtap should of
> course support it too.

I think no other people but systemtap people are willing to use them.
So, should WE propose that kind of minimum debuginfo to upstream?

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com