From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 7305 invoked by alias); 5 Oct 2007 23:05:19 -0000 Received: (qmail 7291 invoked by uid 22791); 5 Oct 2007 23:05:17 -0000 X-Spam-Status: No, hits=-1.2 required=5.0 tests=AWL,BAYES_00,DK_POLICY_SIGNSOME,FORGED_RCVD_HELO,SPF_HELO_PASS,SPF_PASS,TW_DW X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.31) with ESMTP; Fri, 05 Oct 2007 23:05:14 +0000 Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by mx1.redhat.com (8.13.8/8.13.1) with ESMTP id l95N5CCH002454 for ; Fri, 5 Oct 2007 19:05:12 -0400 Received: from gateway.sf.frob.com (vpn-15-31.rdu.redhat.com [10.11.15.31]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l95N5Bcx025496 for ; Fri, 5 Oct 2007 19:05:12 -0400 Received: from magilla.localdomain (magilla.sf.frob.com [198.49.250.228]) by gateway.sf.frob.com (Postfix) with ESMTP id C1CC9357B; Fri, 5 Oct 2007 16:05:10 -0700 (PDT) Received: by magilla.localdomain (Postfix, from userid 5281) id 27F6E4D04B7; Fri, 5 Oct 2007 16:05:10 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit From: Roland McGrath To: Srikar Dronamraju Cc: SystemTap Subject: Re: debuginfo search path in i386 and powerpc are different In-Reply-To: Srikar Dronamraju's message of Friday, 5 October 2007 12:12:21 +0530 <20071005064221.GC12795@linux.vnet.ibm.com> References: <20071004103908.GB12795@linux.vnet.ibm.com> <20071004210930.DC5914D04B7@magilla.localdomain> <20071005064221.GC12795@linux.vnet.ibm.com> X-Windows: a terminal disease. Message-Id: <20071005230510.27F6E4D04B7@magilla.localdomain> Date: Fri, 05 Oct 2007 23:05:00 -0000 X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2007-q4/txt/msg00085.txt.bz2 > On my setup I copied the unstripped vmlinux to > /lib/modules/2.6.18-rhel5.1snap2/vmlinux and the stripped vmlinux to > /boot/vmlinux-2.6.18-rhel5.1snap2. This is the key issue. This is not a combination the code expects. What systemtap (libdwfl) is doing is in two phases. First, it looks for a "main ELF file" for the kernel. This can be either stripped or unstripped. If that file has DWARF info, nothing else is consulted. If the file is stripped, it looks for a corresponding separate debuginfo file. (That is, it's expecting to find a .debug file made by eu-strip -f, not the original unstripped file. However, it should work fine to be seeing an unstripped file in place of a .debug file, as long as it's exactly the file the stripped main file came from. It uses the DWARF data from the debuginfo file, but combined with addresses and other information from the main file, so they have to match.) > open("/boot/vmlinux-2.6.18-rhel5.1snap2", O_RDONLY|O_LARGEFILE) = 3 This is it finding the main ELF file in the first place it looks. Since this file is stripped, it then looks for a debuginfo file in the standard way. (This is the exactly same method that is applied to non-kernel executable and DSO fils by elfutils or gdb.) That follows a default path in the manner described by the libdwfl.h comment near the declaration of dwfl_standard_find_debuginfo. > open("/boot/vmlinux-2.6.18-rhel5.1snap2.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/boot/.debug/vmlinux-2.6.18-rhel5.1snap2.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/usr/lib/debug/boot/vmlinux-2.6.18-rhel5.1snap2.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) These are what that default path yields. The last one, in /usr/lib/debug, is the canonical way to package separate debuginfo for any installed binary. i.e. /usr/lib/debug/FULL/PATH/NAME.debug If you are installing a kernel by hand, you have a few options. If /boot has space for the unstripped image and it works to just boot that file directly, the simplest thing is to just put it there and nowhere else. > On this setup also I copied the unstripped vmlinux to > /lib/modules/2.6.18-rhel5.1snap2/vmlinux and the stripped vmlinux to > /boot/vmlinuz-2.6.18-rhel5.1snap2. vmlinuz is the name usually used for a bootable compressed kernel image. On x86, this is not an ELF file at all, though it contains a compressed copy of the stripped vmlinux file. On some other machines a similar bootable image sometimes is an ELF file; though it contains the stripped vmlinux data, this is not the stripped vmlinux file. Such image files have no role in debugging. Looking at them when they are ELF files would confuse the tool, which would think it was looking at the stripped vmlinux file. For these reasons, the library never looks for any "vmlinuz" file. To confuse matters further, the Fedora/RHEL kernel packaging on ppc does use a plain stripped vmlinux file to boot from, but installs this as /boot/vmlinuz-RELEASE. I think this is just an artifact of the installer and tools that edit boot loader configuration files and so forth, which assume "vmlinuz" names uniformly across machines. But the wisdom of that and issues with changing it are beyond the scope of this discussion. Because of the prevalence of the x86 style where there is no ELF file installed in /boot or anywhere else just to use the kernel, the kernel-debuginfo rpms for all machines do not produce the vanilla split into separate installed ELF binary and .debug files for vmlinux. Instead, they copy the unstripped vmlinux to /usr/lib/debug/lib/modules/RELEASE/vmlinux. The package installs no file outside /usr/lib/debug for debugging the kernel image, since there is no file involved in debugging that is needed anywhere but in the kernel-debuginfo rpm. > open("/boot/vmlinux-2.6.18-rhel5.1snap2", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/boot/vmlinux-2.6.18-rhel5.1snap2", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/boot/.debug/vmlinux-2.6.18-rhel5.1snap2", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/usr/lib/debug/boot/vmlinux-2.6.18-rhel5.1snap2", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) As you saw on ppc, this is the first place it looks for the main vmlinux ELF file. It's looking in /boot/vmlinux-RELEASE, and then for the file vmlinux-RELEASE in each of the directories the debuginfo path would look for a debuginfo file associated with a binary in /boot. > open("/boot/vmlinux-2.6.18-rhel5.1snap2.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/boot/.debug/vmlinux-2.6.18-rhel5.1snap2.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/usr/lib/debug/boot/vmlinux-2.6.18-rhel5.1snap2.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) The next thing it tries is all the places that it would look for a debuginfo file if it had a stripped file called /boot/vmlinux-RELEASE.debug. > open("/lib/modules/2.6.18-rhel5.1snap2/vmlinux", O_RDONLY|O_LARGEFILE) = 3 Next it tries /lib/modules/RELEASE/vmlinux. If this had failed, it would have tried all the same derivatives of that directory + file name that it first tried with /boot/vmlinux-RELEASE. Between these defaults and --debuginfo-path, a person installing their own file by hand has a variety of ways to choose the most convenient place for them. > open("/lib/modules/2.6.18-rhel5.1snap2/vmlinux.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/lib/modules/2.6.18-rhel5.1snap2/.debug/vmlinux.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) > open("/usr/lib/debug/lib/modules/2.6.18-rhel5.1snap2/vmlinux.debug", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) This is the normal debuginfo path search done when /lib/modules/2.6.18-rhel5.1snap2/vmlinux is the main ELF file. I think it's an unrelated bug, that I was already aware of and is harmless, that causes libdwfl to look for a debuginfo file even though the main ELF file is stripped. So, everything is working as expected (except the final bit, the harmless bug). If you are doing a hand install and just installing the unstripped file in /boot/vmlinux-RELEASE works for booting, then do that. If you cannot boot directly from an unstripped vmlinux file for whatever reason, then either give the stripped + .debug (or unstripped copy) a pair of directory and file names that the standard debuginfo association will find (including using a --debuginfo-path option to affect that, if you prefer), or don't install the stripped file by the name /boot/vmlinux-RELEASE at all and place the unstripped vmlinux in one of the places the library searches for the kernel's main ELF file as described above (including using a --debuginfo-path option to affect that, if you prefer). Thanks, Roland