From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 24413 invoked by alias); 22 Dec 2008 18:22:53 -0000 Received: (qmail 24406 invoked by uid 22791); 22 Dec 2008 18:22:52 -0000 X-SWARE-Spam-Status: No, hits=0.0 required=5.0 tests=BAYES_50,SPF_PASS X-Spam-Status: No, hits=0.0 required=5.0 tests=BAYES_50,SPF_PASS X-Spam-Check-By: sourceware.org Received: from BISCAYNE-ONE-STATION.MIT.EDU (HELO biscayne-one-station.mit.edu) (18.7.7.80) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 22 Dec 2008 18:22:10 +0000 Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by biscayne-one-station.mit.edu (8.13.6/8.9.2) with ESMTP id mBMILpMT025026; Mon, 22 Dec 2008 13:21:54 -0500 (EST) Received: from closure.thunk.org (c-98-216-98-217.hsd1.ma.comcast.net [98.216.98.217]) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id mBMIJLsk028655 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 22 Dec 2008 13:19:22 -0500 (EST) Received: from tytso by closure.thunk.org with local (Exim 4.69) (envelope-from ) id 1LEpNB-0008M9-St; Mon, 22 Dec 2008 13:19:21 -0500 Date: Mon, 22 Dec 2008 20:38:00 -0000 From: Theodore Tso To: "Frank Ch. Eigler" Cc: systemtap@sources.redhat.com Subject: Re: Discussion at Linux Foundation Japan Symposium Message-ID: <20081222181921.GH23723@mit.edu> References: <20081221003831.GG24081@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081221003831.GG24081@redhat.com> User-Agent: Mutt/1.5.17+20080114 (2008-01-14) X-Scanned-By: MIMEDefang 2.42 X-Spam-Score: 0.00 X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2008-q4/txt/msg00624.txt.bz2 On Sat, Dec 20, 2008 at 07:38:31PM -0500, Frank Ch. Eigler wrote: > > Some of this has been predicated on the belief that debuginfo quality > will improve enough that we do not need to resort to hard-coded > per-architecture per-compiler hacks like some in gdb. Yes, but how long will people need to wait before debuginfo quality will improve? And when such a hypothetical improved compiler be available and trusted to compile the kernel correctly? (And how much slower will it be compared to current versions of gcc's? :-) More importantly, what will systemtap development do in the meantime? There are a couple of choices: 1) Do nothing. Kernel developers will continue to largely give up on Systemtap, mostly out of frustration. Frustration that their needs aren't being met; frustration that suggestions that they make are blown off as not being important, or concerns that are in the minority, or whatever. The process of getting userspace probes in will go slowly (if at all). I haven't seen a new version of the patches submitted in months --- and the 2.6.29 merge window will be opening soon. Whoever is managing getting the necessary userspace probe patches into mainline seems to be totally ignorant about mainline processes --- why hasn't a "this is what we intend to merge" been sent to LKML already? 2) Explain to developers how to use Systemtap in a painless way as possible. This means no modules, since debuginfo is a disaster with modules (so helping people create configurations that don't use modules would be helpful); and/or with a focus on markers as a debugging mechanism, even if that means that you have to recompile the kernel first before they can be used. If making markers more helpful is the goal, then work to make the Modules.marker file consistently installed in the same place, or to make Systemtap look to find it in multiple places, is going to be important. 3) Actually try to find a way for Systemtap to be more useful *without* waiting for Godot^H^H^H^H^H^H some hypothetical future, probably at least 18-24 months away, when new gcc and toolchain becomes available that will magically and mystically Solve All Problems. Consider how ftrace works; it very cleverly uses/abuses the profiling hook inserted gcc -pg. Note that it doesn't wait for magic new capabilities in the toolchain. In fact, given the utterly worthless (and vastly huge) debuginfo, the only advantage SystemTap has over ftrace is (a) Systemtap has access to the function parameters, and (b) Systemtap allows you to filter logging information based on conditions, which is more efficient than filtering in userspace. Being able to probe individual lines usually doesn't work, so most of the time what I find is the only thing I can count on is add a tap at the beginning and end of the function --- which I can do using ftrace. > > [...] Would could build some scripts with the binaries where debug > > info is available, interpret that info as relative offsets to public > > symbols then dispense with the info on the shipped production > > script. See where I'm going here? > > It sounds like an interesting sort of shortcut - one that we have not > seriously considered. It may be far too optimistic, considering the > degree of configuration/optimization change possible between two > similar kernel builds, and the prevalent hostility toward reusable > kernel modules. Another solution which I've suggested in the past, but which has been utterly ignored, (since, after all, what does a Kernel Developer know?) is to simply generate the debuginfo information on demand so the function parameters on entry can be decoded. We have the header files after all, so it shouldn't be hard to generate a pseudo function: extern int foo_kernel_function(struct file *f, off_t offset, size_t len, void *ptr) { return 0; } ... and then use that to generate the debuginfo information for that information so Systemtap knows how to decode the function parameters on entry. This would be enough such that Systemtap can probe any public function in System.map --- and obtain the function parameters for any function which is present in the kernel headers --- all without requiring users to download or create hundreds of megabytes of largely useless (on a value-per-megabyte-basis) !@#!@# debuginfo information. - Ted