From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 78277 invoked by alias); 26 Jan 2017 21:27:10 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 77865 invoked by uid 89); 26 Jan 2017 21:27:10 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.4 required=5.0 tests=BAYES_00,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RCVD_IN_SORBS_SPAM,SPF_PASS autolearn=no version=3.3.2 spammy=H*i:sk:7671b79, H*MI:sk:7671b79, H*f:sk:7671b79, f X-HELO: mail-lf0-f67.google.com Received: from mail-lf0-f67.google.com (HELO mail-lf0-f67.google.com) (209.85.215.67) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 26 Jan 2017 21:27:00 +0000 Received: by mail-lf0-f67.google.com with SMTP id x1so24748542lff.0 for ; Thu, 26 Jan 2017 13:26:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=YMdcVCVFJrCBn1fosal3WZK7ILqR76DBuMLv2YiJFH4=; b=UZPLMTstBokjwpZb/S4x3W9rwVPAM0GzoChUk5VmmEXIHSN3Sk1M3TKtrVpgIZXZrL unez+h8Ybx4unIiid53y8iktqtGG8OrwMkuwifCUmZyqwMovpfagZd2dzaTUCvzQR6kA uaziCnucVHUz+4C2BoxAfU7b2whOZ/FbnAXjqMPy/cxyXqMX8DPr0cd/IJB45lirMe/5 eKOmADkQR/msdZVKi7WirK9+9aJMDDdVN8XzL5dKJe+ANlF1WI015ZJWjAWXESbrlfRO B2VNL0UQKp0WeemEhLvfleLTUsdXiPpsmk0qLt/JHuMRRyfsvWS87pUXFGqjNzhfjHmy ZJZg== X-Gm-Message-State: AIkVDXK7uoIj3oLwZ8MpZCKqsuT/arrGuFT5ITfCw/N0em/dPzUiV8tGJXLqOcV6wCm9Cg== X-Received: by 10.46.77.149 with SMTP id c21mr1892100ljd.69.1485466017609; Thu, 26 Jan 2017 13:26:57 -0800 (PST) Received: from [192.168.2.21] (staticline4507.toya.net.pl. [85.89.169.166]) by smtp.googlemail.com with ESMTPSA id c1sm776675lfg.9.2017.01.26.13.26.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Jan 2017 13:26:56 -0800 (PST) Subject: Re: Problem with vfs probe on Proxmox kernel To: David Smith , systemtap@sourceware.org References: <13b0aebd-4bbe-f0ca-bf30-1605da0097f5@gmail.com> <1fba1b99-f649-5e0a-80d1-370ca321a80b@redhat.com> <7671b79f-80d5-b6d7-1912-6378653380d9@redhat.com> From: Adam Guderski Message-ID: <9452cf75-cce6-66e6-512f-13b2987e1b82@gmail.com> Date: Thu, 26 Jan 2017 21:27:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: <7671b79f-80d5-b6d7-1912-6378653380d9@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SW-Source: 2017-q1/txt/msg00070.txt.bz2 On 26/01/17 20:05, David Smith wrote: > On 01/26/2017 12:42 PM, Adam Guderski wrote: >> On 26/01/17 18:37, David Smith wrote: >>> On 01/25/2017 04:44 PM, Adam Guderski wrote: >>>> Hello, >>>> >>>> I've got problem running sample traceio.stp script (obtained from >>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) >>>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which >>>> uses a custom Ubuntu-based kernel. The Proxmox repository does not >>>> provide a kernel with debug symbols, so I had to compile it from >>>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos >>>> (jessie) provide systemtap in version 2.9, which from what I read does >>>> not support 4.4 kernels, so I compiled systemtap from sources (from >>>> master branch, version 3.1/0.159). >>>> >>>> With such setup, I'm able to run sample "hello world" stap script and >>>> couple of others, but all scripts with vfs probes fail. I attach full >>>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think >>>> the main error can be explained below. What can I do about it? I >>>> searched for couple of hours but did not find anything that would help. >>> >>> As the following states: >>> >>>> # stap -v traceio2.stp 0xfb00 >>>> Pass 1: parsed user script and 119 library scripts using >>>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. >>>> semantic error: while resolving probe point: identifier 'kernel' at >>>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 >>>> source: probe vfs.write = kernel.function("vfs_write") >>>> ^ >>>> >>>> semantic error: missing x86_64 kernel/module debuginfo [man >>>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' >>> >>> Note that a systemtap "hello world" program doesn't need kernel >>> debuginfo, but a script using vfs probes does need kernel debuginfo. >>> Does any script that needs debuginfo work, or is it just vfs probes? To >>> test, try 'stap -l 'kernel.function("*_write")'. You should see items >>> like 'sys_write' and 'vfs_write' in the output. >> >> Sadly, the command "stap -l 'kernel.function("*_write")'" returns >> nothing at all. > > That's actually good. This definitely means that stap can't find any > debuginfo (vs. the problem being stap found the debuginfo but didn't > find vfs_write() in the debuginfo). > >>> systemtap is expecting to find the kernel's debuginfo at >>> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need >>> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment >>> variable to tell systemtap where you've put the debuginfo. >> >> Could you elaborate how debuginfo files look like? This is what I've got >> in the directory: > > For unstripped userspace exes, the debuginfo is present in the exe. If > you probe an unstripped userspace exe, systemtap shouldn't have any > trouble finding the debuginfo. To make the exes smaller, the debuginfo > can be extracted and put in another file, typically in > /usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for > /bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug. > > For kernels, things are a bit more tricky. You typically boot off a > compressed stripped kernel image, most likely called > /boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The > debuginfo is typically stored in an unstripped uncompressed kernel file > down in /usr/lib/debug. Here's on a Fedora 25 system I see: > > ==== > # ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64 > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux > -rwxr-xr-x. 1 root root 6891144 Jan 20 07:43 > /boot/vmlinuz-4.9.5-200.fc25.x86_64 > -rwxr-xr-x. 1 root root 541141280 Jan 20 07:48 > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux > # file /boot/vmlinuz-4.9.5-200.fc25.x86_64 > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux > /boot/vmlinuz-4.9.5-200.fc25.x86_64: Linux kernel > x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64 > (mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6, > Normal VGA > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB > executable, x86-64, version 1 (SYSV), statically linked, > BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped > ==== > > As you can see the vmlinux file is much bigger than the vmlinuz file, > since it is unstripped and uncompressed. > > Now those paths are all for distro files, and you've got a self-built > kernel. But, the principle is still the same. > >> # ls -la /lib/modules/4.4.35-2-pve/build/ >> total 1564 >> drwxr-xr-x 26 root root 4096 Jan 22 02:09 . >> drwxr-xr-x 3 root root 4096 Jan 18 00:52 .. >> drwxr-xr-x 33 root root 4096 Jan 22 02:09 arch >> drwxr-xr-x 3 root root 4096 Jan 22 02:09 block >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 certs >> -rw-r--r-- 1 root root 189740 Jan 22 01:29 .config >> drwxr-xr-x 4 root root 4096 Jan 22 02:09 crypto >> drwxr-xr-x 127 root root 4096 Jan 22 02:09 drivers >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 firmware >> drwxr-xr-x 74 root root 4096 Jan 22 02:09 fs >> drwxr-xr-x 30 root root 4096 Jan 22 02:09 include >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 init >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 ipc >> -rw-r--r-- 1 root root 2622 Dec 22 08:43 Kbuild >> -rw-r--r-- 1 root root 252 Dec 22 08:43 Kconfig >> drwxr-xr-x 15 root root 4096 Jan 22 02:09 kernel >> drwxr-xr-x 12 root root 4096 Jan 22 02:09 lib >> -rw-r--r-- 1 root root 56164 Jan 22 00:46 Makefile >> drwxr-xr-x 3 root root 4096 Jan 22 02:09 mm >> -rw-r--r-- 1 root root 1218952 Jan 22 01:29 Module.symvers >> drwxr-xr-x 60 root root 4096 Jan 22 02:09 net >> drwxr-xr-x 16 root root 4096 Jan 22 02:09 samples >> drwxr-xr-x 13 root root 20480 Jan 22 02:09 scripts >> drwxr-xr-x 9 root root 4096 Jan 22 02:09 security >> drwxr-xr-x 23 root root 4096 Jan 22 02:09 sound >> drwxr-xr-x 10 root root 4096 Jan 22 02:09 spl >> drwxr-xr-x 21 root root 4096 Jan 22 02:09 tools >> drwxr-xr-x 8 root root 4096 Jan 22 02:09 ubuntu >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 usr >> drwxr-xr-x 4 root root 4096 Jan 22 02:09 virt >> drwxr-xr-x 13 root root 4096 Jan 22 02:09 zfs >> >> Am I missing something? Are debuginfo files output somewhere else during >> compilation from source? > > Based on the paths stored when compiling your kernel, systemtap is > looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. > Obviously it isn't there. Can you search around and see if you can find > it and link it to its proper place? I can't find it (find / -type f -name '*vmlinux*'). When I compiled from source, I noticed this lines in Makefile for Proxmox VE kernel: # strip debug info find tmp/lib/modules -name \*.ko -print | while read f ; do strip --strip-debug "$$f"; done I thought that when I comment this out my kernel will have debug info compiled into it. Interestingly, the config file for my kernel says that CONFIG_DEBUG_INFO is enabled: # grep DEBUG_INFO /boot/config-4.4.35-2-pve CONFIG_DEBUG_INFO=y # CONFIG_DEBUG_INFO_REDUCED is not set # CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF4=y