From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 11833 invoked by alias); 22 Dec 2008 22:43:38 -0000 Received: (qmail 11823 invoked by uid 22791); 22 Dec 2008 22:43:37 -0000 X-SWARE-Spam-Status: No, hits=0.0 required=5.0 tests=BAYES_50,SPF_PASS X-Spam-Status: No, hits=0.0 required=5.0 tests=BAYES_50,SPF_PASS X-Spam-Check-By: sourceware.org Received: from BISCAYNE-ONE-STATION.MIT.EDU (HELO biscayne-one-station.mit.edu) (18.7.7.80) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 22 Dec 2008 22:42:55 +0000 Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by biscayne-one-station.mit.edu (8.13.6/8.9.2) with ESMTP id mBMMfpWS000898; Mon, 22 Dec 2008 17:41:54 -0500 (EST) Received: from closure.thunk.org (c-98-216-98-217.hsd1.ma.comcast.net [98.216.98.217]) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id mBMMbfqu014640 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 22 Dec 2008 17:37:42 -0500 (EST) Received: from tytso by closure.thunk.org with local (Exim 4.69) (envelope-from ) id 1LEtPB-0001le-GP; Mon, 22 Dec 2008 17:37:41 -0500 Date: Tue, 23 Dec 2008 00:33:00 -0000 From: Theodore Tso To: "Frank Ch. Eigler" Cc: systemtap@sources.redhat.com Subject: Re: Discussion at Linux Foundation Japan Symposium Message-ID: <20081222223741.GM23723@mit.edu> References: <20081221003831.GG24081@redhat.com> <20081222181921.GH23723@mit.edu> <20081222203747.GA4195@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081222203747.GA4195@redhat.com> User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-Scanned-By: MIMEDefang 2.42 X-Spam-Score: 0.00 X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2008-q4/txt/msg00629.txt.bz2 On Mon, Dec 22, 2008 at 03:37:48PM -0500, Frank Ch. Eigler wrote: > > Yes, but how long will people need to wait before debuginfo quality > > will improve? And when such a hypothetical improved compiler be > > available and trusted to compile the kernel correctly? [...] > > You misinterpret the status quo. In reality, debuginfo quality with > current gcc has been satisfactory for many purposes. That is what > enables user-space debuggers to work as well as they do, and indeed > enables systemtap to work as well as it does. The major current > debuginfo quality shortcoming involves only those intra-function > statement probes. I can only speak from my personal experience, which is given the kernel default compilation options, and the inline functions and static functions in the parts of the kernel that I normally play with (i.e., ext3/ext4 filesystem code), in nearly every single time where I've had to try to place a Systemtap probe, if it hasn't been at a non-static function entry/exit point, the vast majority of the time, it Simply Doesn't Work. With user-space debuggers, it's not so great, either, but normally when I do debugging e2fsprogs, I recompile w/o optimizations to avoid these frustrations. Furthermore, e2fsprogs doesn't have the same large number of static functions which gcc is far too happy to in-line, and which currently makes the debuginfo largely useless. Worse yet, the error handling with Systemtap is so uniformly awful (to the point that even Roland has complained --- "stap -vvvvvvv" anyone?) that when it doesn't work, it's highly frustrating to figure out what is going on. Basically, I've written off anything other than non-static function entry/exits as being too frustrating to even try using. It might be that the right answer is to tell kernel developers to compile their development kernels without optimization. OK, it that goes against the Systemtap religion that you must never need to recompile or use special built options. But that mantra is causing massive frustration, since after multiple times of trying to set probe points inside ext4 code, and watching it Utterly Fail, it causes me to curse debuginfo, and Systemtap's reliance on it, as Utter Crap. You can go ahead, and dismiss my reports --- after all, I'm one of the few kernel developers who's even bothering to try System Tap at this point, and adding markers into the kernel --- and the first North American kernel developer and CTO of the Linux Foundation can't possibly know anything about, you know, kernel development workflows.... > If you're still talking about debuginfo per se, I am led to believe > that only a small number of kernel developers are negatively affected: > those that build large kernel configurations but on such miniscule > hardware that the sheer disk I/O of the -g data is deemed intolerable. Heh. Only a very few number of kernel developers are even *trying* to use Systemtap at this point. Trying to keep the few remaining kernel developers happy might be considered a good thing. You saw the number of hands that went up at the Kernel Summit. After the Plumber's Conference and the Kernel Summit, Aryan van der Ven told me and a group of of kernel developers that as a senior manager, he planned to pull Intel's last resources working on SystemTap off the project, based on SystemTap developers not being willing to listen to input. And other kernel engineers employed at Red Hat told me that they thought the Red Hat SystempTap developers were totally out of control and they had but very little respect for the project. You are in a *hole*, brother. And the first rule is, when you are in a hole, stop digging.... > > The process of getting userspace probes in will go slowly (if at > > all). I haven't seen a new version of the patches submitted in > > months --- and the 2.6.29 merge window will be opening soon. [...] > > Roland? Please advise. Note that the best way to get kernel patches in is to handle the comments, and repost new versions quickly and often with the comments addressed. Look at how Eric Paris from Red Hat posted fanotify patches; multiple versions, addressing comments, every few days. And even though many people are dubious about his stated goal of helping the virus scanner partner's of Red Hat, he's made enough core infrastructure improvements to other notification systems, such as inotify and dnotify, that those patches will likely make it into the mainline. Quite frankly, the pace at which new versions of the patches are published in response to comments reminds me doomed patches such as reiserfs4.... > > This means no modules, since debuginfo is a disaster with modules > > (so helping people create configurations that don't use modules > > would be helpful); > > Please elaborate what you mean. Why systemtap is forced to use > (create) modules at all has been explained often. What I mean is that if you build a kernel with a large number of modules, the bloat of systemtap becomes disastrous. With a distro-style kernel, the compilation time literally goes up by a factor of 4-5 or so, and the disk space utilization goes up by a factor of 8 or so. It's not so bad if you only the device drivers you into a single kernel, and not create any unneeded modules. This is often not the common way at least some kernel developers build systems, since (a) modules make it easier to load and unload subsystems without rebooting, and this can shorten the compile-edit-debug cycle times, and (b) some kernel developers want to be sure they know whether any infrastructural changes they might cause some random device driver or other random subsystem to break. Unfortunately, building a distro-style kernel is a horrible disaster with debuginfo enabled, and so warning people off about this particular issue would be helpful. > Sure, upstream changes in the location of Modules.marker are something > we can easily adapt to. Are you aware of a current problem with that? > On the other hand, we plan to work on also connecting straight to > tracepoints, and to various slices of ftrace, for which a file such > the troubled-history Modules.markers will not be necessary. Only that "make install" or "make modules_install" doesn't currently install Modules.marker anywhere useful. And there is no standard location for where Modules.marker will be installed given current distribution packaging tools. Attention to consumability of SystemTap given a variety of different kernel installation schemes is a Really Good Idea. It's not just Fedora RPM-style macros and "make install". There are many other ways that people create and install kernels; and this kind of attention to detail will stop people from getting Frustrated with SystemTap. > > 3) Actually try to find a way for Systemtap to be more useful > > *without* waiting for [new gcc] > > As I said above, the main thing we're expecting from a future gcc is > improvement on statement (arbitraty source line) probing. Otherwise > quality is fine and we're not having to wait. (As everyone knows, > quantity (compression) is not great but at least this is only a > usability rather than functionality matter.) It's a rather important usability issue. Part of the reason why SystemTap is so frustrating is precisely because there hasn't been a big enough focus on usability, IMHO. Maybe systemtap developers get used to stap -vvvvvv to figure out why something doesn't work, but it still strikes me as a reminder of how awful and kldugy the technology is. > Nor do we. In fact, we've been as unimposing as possible upon both > kernel and user-space build systems -- perhaps to a fault. > > > [...] the only advantage SystemTap has over ftrace is (a) Systemtap > > has access to the function parameters, and (b) Systemtap allows you > > to filter logging information based on conditions, which is more > > efficient than filtering in userspace. > > Beyond those, there is really plenty of difference in capability. > Line-by-line function tracing? User-space? Concurrent users? The > list goes on... Line-by-line function tracing doesn't work for me. I've given up on it as useless. Yawn. User-space tracing --- is a patch that I don't forsee going mainline anytime soon, given a lack of understanding about how to respond to comments and how to get patches upstream. (Again, there are people in Red Hat who know how to do this; why isn't Roland reaching out to people like Eric Paris, David Jones, Alan Cox, etc.?) Concurrent users -- doesn't matter to kernel developers. And exactly who do you expect is going to be adding markers to the kernel sources, if you continually blow off the concerns of kernel developers such as James and myself? > > [...] is to simply generate the debuginfo information on demand so > > the function parameters on entry can be decoded. We have the header > > files after all, so it shouldn't be hard to generate a pseudo function: > > > > extern int foo_kernel_function(struct file *f, off_t offset, size_t len, > > void *ptr) > > { > > return 0; > > } > > It has some potential. On first blush this seems to require parsing > all the C header files (accounting for arch/kconfig cpp macros) to > look for a matching function declaration. Then the parameter/return > types may require listing additional C headers. As you know, kernel > headers are complicated and rich with #ifdefs and macros that may > expand to inlines inlines, so a naive text processing inspired by the > following may not be worthwhile. How do you imagine it working? Well, the original way I was thinking about doing this involved support from the kernel sources --- so changes to kbuild to generate debugging information that is far more compact, only optimized to provide kernel function parameter information --- since as I have mentioend, line-by-line probing in practice doesn't work for me, so why pay for a factor 8 explosion of debuginfo overhead when most of it seems largely useless given standard kernel compile options. However, it might be doable without having made any kernel changes. Finding which header file a function parameter can be found is pretty simple. There are some places where the function names is created using magic cpp concatenation games, but these tend to be inline functions which systemtap can't probe anyway. The hard part is knowing which other header files might be needed. This can be easily extracted from source files --- so the question is whether Systemtap ships a database of which header files are needed before it's safe to include a particular .h file containing the function prototype; or whether we need a kbuild patch to extract this information; or do both. - Ted