From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 23098 invoked by alias); 23 Dec 2008 00:34:50 -0000 Received: (qmail 23091 invoked by uid 22791); 23 Dec 2008 00:34:48 -0000 X-SWARE-Spam-Status: No, hits=-2.1 required=5.0 tests=AWL,BAYES_00,KAM_MX,SPF_HELO_PASS,SPF_PASS X-Spam-Status: No, hits=-2.1 required=5.0 tests=AWL,BAYES_00,KAM_MX,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Tue, 23 Dec 2008 00:34:13 +0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id mBN0YBsg022540 for ; Mon, 22 Dec 2008 19:34:11 -0500 Received: from ns3.rdu.redhat.com (ns3.rdu.redhat.com [10.11.255.199]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id mBN0YBrS018102 for ; Mon, 22 Dec 2008 19:34:11 -0500 Received: from ton.toronto.redhat.com (ton.yyz.redhat.com [10.15.16.15]) by ns3.rdu.redhat.com (8.13.8/8.13.8) with ESMTP id mBN0YBx1023516; Mon, 22 Dec 2008 19:34:11 -0500 Received: from ton.toronto.redhat.com (localhost.localdomain [127.0.0.1]) by ton.toronto.redhat.com (8.13.1/8.13.1) with ESMTP id mBN0YAdo022222; Mon, 22 Dec 2008 19:34:10 -0500 Received: (from fche@localhost) by ton.toronto.redhat.com (8.13.1/8.13.1/Submit) id mBN0YAi2022221; Mon, 22 Dec 2008 19:34:10 -0500 Date: Tue, 23 Dec 2008 14:28:00 -0000 From: "Frank Ch. Eigler" To: Theodore Tso Cc: systemtap@sources.redhat.com Subject: Re: Discussion at Linux Foundation Japan Symposium Message-ID: <20081223003410.GC4195@redhat.com> References: <20081221003831.GG24081@redhat.com> <20081222181921.GH23723@mit.edu> <20081222203747.GA4195@redhat.com> <20081222223741.GM23723@mit.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20081222223741.GM23723@mit.edu> User-Agent: Mutt/1.4.1i X-Scanned-By: MIMEDefang 2.58 on 172.16.52.254 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2008-q4/txt/msg00632.txt.bz2 Hi - On Mon, Dec 22, 2008 at 05:37:41PM -0500, Theodore Tso wrote: > [...] > > > Yes, but how long will people need to wait before debuginfo quality > > > will improve? And when such a hypothetical improved compiler be > > > available and trusted to compile the kernel correctly? [...] > > > > You misinterpret the status quo. In reality, debuginfo quality with > > current gcc has been satisfactory for many purposes. That is what > > enables user-space debuggers to work as well as they do, and indeed > > enables systemtap to work as well as it does. The major current > > debuginfo quality shortcoming involves only those intra-function > > statement probes. > > I can only speak from my personal experience, which is given the > kernel default compilation options, and the inline functions and > static functions in the parts of the kernel that I normally play with Some inline/static functions become de facto intra-function statement probes that we have problems with. Note that can't probe those functions based on symbol data (kallsyms) because they don't show up there at all! Same with ftrace, I think. Keep that in mind if you wish to fairly compare alternative technologies. > [...] > Worse yet, the error handling with Systemtap is so uniformly awful (to > the point that even Roland has complained --- "stap -vvvvvvv" anyone?) This has been since improved. > It might be that the right answer is to tell kernel developers to > compile their development kernels without optimization. OK, it that > goes against the Systemtap religion that you must never need to > recompile or use special built options. [...] It's not a "religion". It's sometimes an absolute user requirement. It is also a way of letting the rest of the open-source "ecosystem" evolve without us imposing ourselves. > You can go ahead, and dismiss my reports [...] I have endavoured to address each of your points. If I overlooked one, I apologize, and if you point it out I'll be glad to retry. > > If you're still talking about debuginfo per se, I am led to believe > > that only a small number of kernel developers are negatively affected: > > those that build large kernel configurations but on such miniscule > > hardware that the sheer disk I/O of the -g data is deemed intolerable. > > Heh. Only a very few number of kernel developers are even *trying* to > use Systemtap at this point. [...] One short-term benefit you are likely to experience is the restoration of the symbol-table-only probing, which should give you and others some amount of probing capability without CONFIG_DEBUG_INFO. Other intermediate approaches (debuginfo subsetting, compression) appear to require development well outside systemtap. > After the Plumber's Conference and the Kernel Summit, [Arjan] van der > Ven told me [...] And other kernel engineers employed at Red Hat > told me [...] It is hard to respond to this, if these other gentlemen don't take up our manifold invitations to comment/advise us directly as opposed to each other. (The last thing I recall Arjan ever suggesting was packaging some systemtap tapset scripts along with the kernel, in the hope that they would be maintained by subsystem maintainers there. That idea appeared to be mooted by the apparent kernel summit consensus for adding static compiled-in tracing hooks into the kernel.) > > > This means no modules, since debuginfo is a disaster with modules > > > (so helping people create configurations that don't use modules > > > would be helpful); > > > > Please elaborate what you mean. Why systemtap is forced to use > > (create) modules at all has been explained often. > What I mean is that if you build a kernel with a large number of > modules, the bloat of systemtap becomes disastrous. [...] It's more fair to label this "bloat of debuginfo", since systemtap is not responsible for how much code you choose to build. > Unfortunately, building a distro-style kernel is a horrible disaster > with debuginfo enabled, and so warning people off about this > particular issue would be helpful. Is http://tinyurl.com/96dza8 what you had in mind? > > Sure, upstream changes in the location of Modules.marker are something > > we can easily adapt to. Are you aware of a current problem with that? > > [...] > > Only that "make install" or "make modules_install" doesn't currently > install Modules.marker anywhere useful. [...] OK, I believe you mean that as the sole (?) user of this file, systemtap folks should fix this kernel oversight by submitting a patch against kbuild makefiles. > > [...] (As everyone knows, quantity (compression) is not great but > > at least this is only a usability rather than functionality > > matter.) > > It's a rather important usability issue. [...] Agreed. > Line-by-line function tracing doesn't work for me. [...] I hope people will like it once it works robustly. > User-space tracing --- is a patch that I don't forsee going mainline > anytime soon [...] I hope there will be progress on this soon. > Concurrent users -- doesn't matter to kernel developers. I don't think that's completely true. Right now, ftrace is a single-configuration-at-a-time tool, and I recall reading complaints about that. > And exactly who do you expect is going to be adding markers to the > kernel sources, [...] It turns out that, among others, several Red Hat kernel engineers are actively pursuing this, including sending tracepoint-laden patches upstreamward via lkml/lttng. > > > [...] is to simply generate the debuginfo information on demand so > > > the function parameters on entry can be decoded. We have the header > > > files after all, so it shouldn't be hard to generate a pseudo function: > > > > > > extern int foo_kernel_function(struct file *f, off_t offset, size_t len, > > > void *ptr) > > > { > > > return 0; > > > } > > It has some potential. On first blush this seems to require > > parsing all the C header files (accounting for arch/kconfig cpp > > macros) to look for a matching function declaration. [...] How > > do you imagine it working? > Well, the original way I was thinking about doing this involved > support from the kernel sources --- so changes to kbuild to generate > debugging information that is far more compact, only optimized to > provide kernel function parameter information That's does not sound like a simple kbuild preference. That may need a whole new tool well beyond strip/objcopy. > [...] However, it might be doable without having made any kernel > changes. Finding which header file a function parameter can be > found is pretty simple. [...] The hard part is knowing which other > header files might be needed. This can be easily extracted from > source files [...] So more C source parsing. > so the question is whether Systemtap ships a database of which > header files are needed [...] Maybe, but this sounds about as kernel-version-fragile as some of our tapset scripts. > or whether we need a kbuild patch to extract this information; or do > both. OK, so we agree we're talking about a nontrivial amount of code here. - FChE