* Problem with vfs probe on Proxmox kernel @ 2017-01-25 22:44 Adam Guderski 2017-01-26 4:51 ` Arkady 2017-01-26 17:38 ` David Smith 0 siblings, 2 replies; 10+ messages in thread From: Adam Guderski @ 2017-01-25 22:44 UTC (permalink / raw) To: systemtap Hello, I've got problem running sample traceio.stp script (obtained from https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which uses a custom Ubuntu-based kernel. The Proxmox repository does not provide a kernel with debug symbols, so I had to compile it from provided sources - version 4.4.35-2-pve, to be specific. Debian repos (jessie) provide systemtap in version 2.9, which from what I read does not support 4.4 kernels, so I compiled systemtap from sources (from master branch, version 3.1/0.159). With such setup, I'm able to run sample "hello world" stap script and couple of others, but all scripts with vfs probes fail. I attach full verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think the main error can be explained below. What can I do about it? I searched for couple of hours but did not find anything that would help. Best regards, Adam # stap -v traceio2.stp 0xfb00 Pass 1: parsed user script and 119 library scripts using 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. semantic error: while resolving probe point: identifier 'kernel' at /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 source: probe vfs.write = kernel.function("vfs_write") ^ semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' semantic error: resolution failed in alias expansion builder semantic error: while resolving probe point: identifier 'vfs' at traceio2.stp:14:7 source: probe vfs.write, vfs.read ^ semantic error: no match semantic error: while resolving probe point: identifier 'kernel' at /usr/local/share/systemtap/tapset/linux/vfs.stp:915:18 source: probe vfs.read = kernel.function("vfs_read") ^ semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' semantic error: resolution failed in alias expansion builder semantic error: while resolving probe point: identifier 'vfs' at traceio2.stp:14:18 source: probe vfs.write, vfs.read ^ semantic error: no match Pass 2: analyzed script: 1 probe, 33 functions, 1 embed, 1 global using 72384virt/46080res/6016shr/40680data kb, in 40usr/100sys/146real ms. Pass 2: analysis failed. [man error::pass2] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-25 22:44 Problem with vfs probe on Proxmox kernel Adam Guderski @ 2017-01-26 4:51 ` Arkady 2017-01-26 17:38 ` David Smith 1 sibling, 0 replies; 10+ messages in thread From: Arkady @ 2017-01-26 4:51 UTC (permalink / raw) To: Adam Guderski; +Cc: systemtap I am running Systemtap translator/driver (version 2.9/0.165, Debian version 2.9-2ubuntu2 (xenial)) on Linux ubuntu 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux and vfsread/vfswrite work fine. On Thu, Jan 26, 2017 at 12:44 AM, Adam Guderski <a.guderski@gmail.com> wrote: > Hello, > > I've got problem running sample traceio.stp script (obtained from > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) > on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which > uses a custom Ubuntu-based kernel. The Proxmox repository does not > provide a kernel with debug symbols, so I had to compile it from > provided sources - version 4.4.35-2-pve, to be specific. Debian repos > (jessie) provide systemtap in version 2.9, which from what I read does > not support 4.4 kernels, so I compiled systemtap from sources (from > master branch, version 3.1/0.159). > > With such setup, I'm able to run sample "hello world" stap script and > couple of others, but all scripts with vfs probes fail. I attach full > verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think > the main error can be explained below. What can I do about it? I > searched for couple of hours but did not find anything that would help. > > Best regards, > Adam > > > # stap -v traceio2.stp 0xfb00 > Pass 1: parsed user script and 119 library scripts using > 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. > semantic error: while resolving probe point: identifier 'kernel' at > /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 > source: probe vfs.write = kernel.function("vfs_write") > ^ > > semantic error: missing x86_64 kernel/module debuginfo [man > warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' > > semantic error: resolution failed in alias expansion builder > > semantic error: while resolving probe point: identifier 'vfs' at > traceio2.stp:14:7 > source: probe vfs.write, vfs.read > ^ > > semantic error: no match > > semantic error: while resolving probe point: identifier 'kernel' at > /usr/local/share/systemtap/tapset/linux/vfs.stp:915:18 > source: probe vfs.read = kernel.function("vfs_read") > ^ > > semantic error: missing x86_64 kernel/module debuginfo [man > warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' > > semantic error: resolution failed in alias expansion builder > > semantic error: while resolving probe point: identifier 'vfs' at > traceio2.stp:14:18 > source: probe vfs.write, vfs.read > ^ > > semantic error: no match > > Pass 2: analyzed script: 1 probe, 33 functions, 1 embed, 1 global using > 72384virt/46080res/6016shr/40680data kb, in 40usr/100sys/146real ms. > Pass 2: analysis failed. [man error::pass2] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-25 22:44 Problem with vfs probe on Proxmox kernel Adam Guderski 2017-01-26 4:51 ` Arkady @ 2017-01-26 17:38 ` David Smith 2017-01-26 18:43 ` Adam Guderski 1 sibling, 1 reply; 10+ messages in thread From: David Smith @ 2017-01-26 17:38 UTC (permalink / raw) To: Adam Guderski, systemtap On 01/25/2017 04:44 PM, Adam Guderski wrote: > Hello, > > I've got problem running sample traceio.stp script (obtained from > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) > on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which > uses a custom Ubuntu-based kernel. The Proxmox repository does not > provide a kernel with debug symbols, so I had to compile it from > provided sources - version 4.4.35-2-pve, to be specific. Debian repos > (jessie) provide systemtap in version 2.9, which from what I read does > not support 4.4 kernels, so I compiled systemtap from sources (from > master branch, version 3.1/0.159). > > With such setup, I'm able to run sample "hello world" stap script and > couple of others, but all scripts with vfs probes fail. I attach full > verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think > the main error can be explained below. What can I do about it? I > searched for couple of hours but did not find anything that would help. As the following states: > # stap -v traceio2.stp 0xfb00 > Pass 1: parsed user script and 119 library scripts using > 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. > semantic error: while resolving probe point: identifier 'kernel' at > /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 > source: probe vfs.write = kernel.function("vfs_write") > ^ > > semantic error: missing x86_64 kernel/module debuginfo [man > warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' Note that a systemtap "hello world" program doesn't need kernel debuginfo, but a script using vfs probes does need kernel debuginfo. Does any script that needs debuginfo work, or is it just vfs probes? To test, try 'stap -l 'kernel.function("*_write")'. You should see items like 'sys_write' and 'vfs_write' in the output. systemtap is expecting to find the kernel's debuginfo at /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment variable to tell systemtap where you've put the debuginfo. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 17:38 ` David Smith @ 2017-01-26 18:43 ` Adam Guderski 2017-01-26 19:05 ` David Smith 0 siblings, 1 reply; 10+ messages in thread From: Adam Guderski @ 2017-01-26 18:43 UTC (permalink / raw) To: David Smith, systemtap On 26/01/17 18:37, David Smith wrote: > On 01/25/2017 04:44 PM, Adam Guderski wrote: >> Hello, >> >> I've got problem running sample traceio.stp script (obtained from >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) >> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which >> uses a custom Ubuntu-based kernel. The Proxmox repository does not >> provide a kernel with debug symbols, so I had to compile it from >> provided sources - version 4.4.35-2-pve, to be specific. Debian repos >> (jessie) provide systemtap in version 2.9, which from what I read does >> not support 4.4 kernels, so I compiled systemtap from sources (from >> master branch, version 3.1/0.159). >> >> With such setup, I'm able to run sample "hello world" stap script and >> couple of others, but all scripts with vfs probes fail. I attach full >> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think >> the main error can be explained below. What can I do about it? I >> searched for couple of hours but did not find anything that would help. > > As the following states: > >> # stap -v traceio2.stp 0xfb00 >> Pass 1: parsed user script and 119 library scripts using >> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. >> semantic error: while resolving probe point: identifier 'kernel' at >> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 >> source: probe vfs.write = kernel.function("vfs_write") >> ^ >> >> semantic error: missing x86_64 kernel/module debuginfo [man >> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' > > Note that a systemtap "hello world" program doesn't need kernel > debuginfo, but a script using vfs probes does need kernel debuginfo. > Does any script that needs debuginfo work, or is it just vfs probes? To > test, try 'stap -l 'kernel.function("*_write")'. You should see items > like 'sys_write' and 'vfs_write' in the output. Sadly, the command "stap -l 'kernel.function("*_write")'" returns nothing at all. > systemtap is expecting to find the kernel's debuginfo at > /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need > to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment > variable to tell systemtap where you've put the debuginfo. Could you elaborate how debuginfo files look like? This is what I've got in the directory: # ls -la /lib/modules/4.4.35-2-pve/build/ total 1564 drwxr-xr-x 26 root root 4096 Jan 22 02:09 . drwxr-xr-x 3 root root 4096 Jan 18 00:52 .. drwxr-xr-x 33 root root 4096 Jan 22 02:09 arch drwxr-xr-x 3 root root 4096 Jan 22 02:09 block drwxr-xr-x 2 root root 4096 Jan 22 02:09 certs -rw-r--r-- 1 root root 189740 Jan 22 01:29 .config drwxr-xr-x 4 root root 4096 Jan 22 02:09 crypto drwxr-xr-x 127 root root 4096 Jan 22 02:09 drivers drwxr-xr-x 2 root root 4096 Jan 22 02:09 firmware drwxr-xr-x 74 root root 4096 Jan 22 02:09 fs drwxr-xr-x 30 root root 4096 Jan 22 02:09 include drwxr-xr-x 2 root root 4096 Jan 22 02:09 init drwxr-xr-x 2 root root 4096 Jan 22 02:09 ipc -rw-r--r-- 1 root root 2622 Dec 22 08:43 Kbuild -rw-r--r-- 1 root root 252 Dec 22 08:43 Kconfig drwxr-xr-x 15 root root 4096 Jan 22 02:09 kernel drwxr-xr-x 12 root root 4096 Jan 22 02:09 lib -rw-r--r-- 1 root root 56164 Jan 22 00:46 Makefile drwxr-xr-x 3 root root 4096 Jan 22 02:09 mm -rw-r--r-- 1 root root 1218952 Jan 22 01:29 Module.symvers drwxr-xr-x 60 root root 4096 Jan 22 02:09 net drwxr-xr-x 16 root root 4096 Jan 22 02:09 samples drwxr-xr-x 13 root root 20480 Jan 22 02:09 scripts drwxr-xr-x 9 root root 4096 Jan 22 02:09 security drwxr-xr-x 23 root root 4096 Jan 22 02:09 sound drwxr-xr-x 10 root root 4096 Jan 22 02:09 spl drwxr-xr-x 21 root root 4096 Jan 22 02:09 tools drwxr-xr-x 8 root root 4096 Jan 22 02:09 ubuntu drwxr-xr-x 2 root root 4096 Jan 22 02:09 usr drwxr-xr-x 4 root root 4096 Jan 22 02:09 virt drwxr-xr-x 13 root root 4096 Jan 22 02:09 zfs Am I missing something? Are debuginfo files output somewhere else during compilation from source? ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 18:43 ` Adam Guderski @ 2017-01-26 19:05 ` David Smith 2017-01-26 21:27 ` Adam Guderski 0 siblings, 1 reply; 10+ messages in thread From: David Smith @ 2017-01-26 19:05 UTC (permalink / raw) To: Adam Guderski, systemtap On 01/26/2017 12:42 PM, Adam Guderski wrote: > On 26/01/17 18:37, David Smith wrote: >> On 01/25/2017 04:44 PM, Adam Guderski wrote: >>> Hello, >>> >>> I've got problem running sample traceio.stp script (obtained from >>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) >>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which >>> uses a custom Ubuntu-based kernel. The Proxmox repository does not >>> provide a kernel with debug symbols, so I had to compile it from >>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos >>> (jessie) provide systemtap in version 2.9, which from what I read does >>> not support 4.4 kernels, so I compiled systemtap from sources (from >>> master branch, version 3.1/0.159). >>> >>> With such setup, I'm able to run sample "hello world" stap script and >>> couple of others, but all scripts with vfs probes fail. I attach full >>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think >>> the main error can be explained below. What can I do about it? I >>> searched for couple of hours but did not find anything that would help. >> >> As the following states: >> >>> # stap -v traceio2.stp 0xfb00 >>> Pass 1: parsed user script and 119 library scripts using >>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. >>> semantic error: while resolving probe point: identifier 'kernel' at >>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 >>> source: probe vfs.write = kernel.function("vfs_write") >>> ^ >>> >>> semantic error: missing x86_64 kernel/module debuginfo [man >>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' >> >> Note that a systemtap "hello world" program doesn't need kernel >> debuginfo, but a script using vfs probes does need kernel debuginfo. >> Does any script that needs debuginfo work, or is it just vfs probes? To >> test, try 'stap -l 'kernel.function("*_write")'. You should see items >> like 'sys_write' and 'vfs_write' in the output. > > Sadly, the command "stap -l 'kernel.function("*_write")'" returns > nothing at all. That's actually good. This definitely means that stap can't find any debuginfo (vs. the problem being stap found the debuginfo but didn't find vfs_write() in the debuginfo). >> systemtap is expecting to find the kernel's debuginfo at >> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need >> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment >> variable to tell systemtap where you've put the debuginfo. > > Could you elaborate how debuginfo files look like? This is what I've got > in the directory: For unstripped userspace exes, the debuginfo is present in the exe. If you probe an unstripped userspace exe, systemtap shouldn't have any trouble finding the debuginfo. To make the exes smaller, the debuginfo can be extracted and put in another file, typically in /usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for /bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug. For kernels, things are a bit more tricky. You typically boot off a compressed stripped kernel image, most likely called /boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The debuginfo is typically stored in an unstripped uncompressed kernel file down in /usr/lib/debug. Here's on a Fedora 25 system I see: ==== # ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64 /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux -rwxr-xr-x. 1 root root 6891144 Jan 20 07:43 /boot/vmlinuz-4.9.5-200.fc25.x86_64 -rwxr-xr-x. 1 root root 541141280 Jan 20 07:48 /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux # file /boot/vmlinuz-4.9.5-200.fc25.x86_64 /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux /boot/vmlinuz-4.9.5-200.fc25.x86_64: Linux kernel x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64 (mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6, Normal VGA /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped ==== As you can see the vmlinux file is much bigger than the vmlinuz file, since it is unstripped and uncompressed. Now those paths are all for distro files, and you've got a self-built kernel. But, the principle is still the same. > # ls -la /lib/modules/4.4.35-2-pve/build/ > total 1564 > drwxr-xr-x 26 root root 4096 Jan 22 02:09 . > drwxr-xr-x 3 root root 4096 Jan 18 00:52 .. > drwxr-xr-x 33 root root 4096 Jan 22 02:09 arch > drwxr-xr-x 3 root root 4096 Jan 22 02:09 block > drwxr-xr-x 2 root root 4096 Jan 22 02:09 certs > -rw-r--r-- 1 root root 189740 Jan 22 01:29 .config > drwxr-xr-x 4 root root 4096 Jan 22 02:09 crypto > drwxr-xr-x 127 root root 4096 Jan 22 02:09 drivers > drwxr-xr-x 2 root root 4096 Jan 22 02:09 firmware > drwxr-xr-x 74 root root 4096 Jan 22 02:09 fs > drwxr-xr-x 30 root root 4096 Jan 22 02:09 include > drwxr-xr-x 2 root root 4096 Jan 22 02:09 init > drwxr-xr-x 2 root root 4096 Jan 22 02:09 ipc > -rw-r--r-- 1 root root 2622 Dec 22 08:43 Kbuild > -rw-r--r-- 1 root root 252 Dec 22 08:43 Kconfig > drwxr-xr-x 15 root root 4096 Jan 22 02:09 kernel > drwxr-xr-x 12 root root 4096 Jan 22 02:09 lib > -rw-r--r-- 1 root root 56164 Jan 22 00:46 Makefile > drwxr-xr-x 3 root root 4096 Jan 22 02:09 mm > -rw-r--r-- 1 root root 1218952 Jan 22 01:29 Module.symvers > drwxr-xr-x 60 root root 4096 Jan 22 02:09 net > drwxr-xr-x 16 root root 4096 Jan 22 02:09 samples > drwxr-xr-x 13 root root 20480 Jan 22 02:09 scripts > drwxr-xr-x 9 root root 4096 Jan 22 02:09 security > drwxr-xr-x 23 root root 4096 Jan 22 02:09 sound > drwxr-xr-x 10 root root 4096 Jan 22 02:09 spl > drwxr-xr-x 21 root root 4096 Jan 22 02:09 tools > drwxr-xr-x 8 root root 4096 Jan 22 02:09 ubuntu > drwxr-xr-x 2 root root 4096 Jan 22 02:09 usr > drwxr-xr-x 4 root root 4096 Jan 22 02:09 virt > drwxr-xr-x 13 root root 4096 Jan 22 02:09 zfs > > Am I missing something? Are debuginfo files output somewhere else during > compilation from source? Based on the paths stored when compiling your kernel, systemtap is looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. Obviously it isn't there. Can you search around and see if you can find it and link it to its proper place? -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 19:05 ` David Smith @ 2017-01-26 21:27 ` Adam Guderski 2017-01-26 21:34 ` David Smith 0 siblings, 1 reply; 10+ messages in thread From: Adam Guderski @ 2017-01-26 21:27 UTC (permalink / raw) To: David Smith, systemtap On 26/01/17 20:05, David Smith wrote: > On 01/26/2017 12:42 PM, Adam Guderski wrote: >> On 26/01/17 18:37, David Smith wrote: >>> On 01/25/2017 04:44 PM, Adam Guderski wrote: >>>> Hello, >>>> >>>> I've got problem running sample traceio.stp script (obtained from >>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2) >>>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which >>>> uses a custom Ubuntu-based kernel. The Proxmox repository does not >>>> provide a kernel with debug symbols, so I had to compile it from >>>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos >>>> (jessie) provide systemtap in version 2.9, which from what I read does >>>> not support 4.4 kernels, so I compiled systemtap from sources (from >>>> master branch, version 3.1/0.159). >>>> >>>> With such setup, I'm able to run sample "hello world" stap script and >>>> couple of others, but all scripts with vfs probes fail. I attach full >>>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think >>>> the main error can be explained below. What can I do about it? I >>>> searched for couple of hours but did not find anything that would help. >>> >>> As the following states: >>> >>>> # stap -v traceio2.stp 0xfb00 >>>> Pass 1: parsed user script and 119 library scripts using >>>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms. >>>> semantic error: while resolving probe point: identifier 'kernel' at >>>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19 >>>> source: probe vfs.write = kernel.function("vfs_write") >>>> ^ >>>> >>>> semantic error: missing x86_64 kernel/module debuginfo [man >>>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build' >>> >>> Note that a systemtap "hello world" program doesn't need kernel >>> debuginfo, but a script using vfs probes does need kernel debuginfo. >>> Does any script that needs debuginfo work, or is it just vfs probes? To >>> test, try 'stap -l 'kernel.function("*_write")'. You should see items >>> like 'sys_write' and 'vfs_write' in the output. >> >> Sadly, the command "stap -l 'kernel.function("*_write")'" returns >> nothing at all. > > That's actually good. This definitely means that stap can't find any > debuginfo (vs. the problem being stap found the debuginfo but didn't > find vfs_write() in the debuginfo). > >>> systemtap is expecting to find the kernel's debuginfo at >>> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need >>> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment >>> variable to tell systemtap where you've put the debuginfo. >> >> Could you elaborate how debuginfo files look like? This is what I've got >> in the directory: > > For unstripped userspace exes, the debuginfo is present in the exe. If > you probe an unstripped userspace exe, systemtap shouldn't have any > trouble finding the debuginfo. To make the exes smaller, the debuginfo > can be extracted and put in another file, typically in > /usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for > /bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug. > > For kernels, things are a bit more tricky. You typically boot off a > compressed stripped kernel image, most likely called > /boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The > debuginfo is typically stored in an unstripped uncompressed kernel file > down in /usr/lib/debug. Here's on a Fedora 25 system I see: > > ==== > # ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64 > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux > -rwxr-xr-x. 1 root root 6891144 Jan 20 07:43 > /boot/vmlinuz-4.9.5-200.fc25.x86_64 > -rwxr-xr-x. 1 root root 541141280 Jan 20 07:48 > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux > # file /boot/vmlinuz-4.9.5-200.fc25.x86_64 > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux > /boot/vmlinuz-4.9.5-200.fc25.x86_64: Linux kernel > x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64 > (mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6, > Normal VGA > /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB > executable, x86-64, version 1 (SYSV), statically linked, > BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped > ==== > > As you can see the vmlinux file is much bigger than the vmlinuz file, > since it is unstripped and uncompressed. > > Now those paths are all for distro files, and you've got a self-built > kernel. But, the principle is still the same. > >> # ls -la /lib/modules/4.4.35-2-pve/build/ >> total 1564 >> drwxr-xr-x 26 root root 4096 Jan 22 02:09 . >> drwxr-xr-x 3 root root 4096 Jan 18 00:52 .. >> drwxr-xr-x 33 root root 4096 Jan 22 02:09 arch >> drwxr-xr-x 3 root root 4096 Jan 22 02:09 block >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 certs >> -rw-r--r-- 1 root root 189740 Jan 22 01:29 .config >> drwxr-xr-x 4 root root 4096 Jan 22 02:09 crypto >> drwxr-xr-x 127 root root 4096 Jan 22 02:09 drivers >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 firmware >> drwxr-xr-x 74 root root 4096 Jan 22 02:09 fs >> drwxr-xr-x 30 root root 4096 Jan 22 02:09 include >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 init >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 ipc >> -rw-r--r-- 1 root root 2622 Dec 22 08:43 Kbuild >> -rw-r--r-- 1 root root 252 Dec 22 08:43 Kconfig >> drwxr-xr-x 15 root root 4096 Jan 22 02:09 kernel >> drwxr-xr-x 12 root root 4096 Jan 22 02:09 lib >> -rw-r--r-- 1 root root 56164 Jan 22 00:46 Makefile >> drwxr-xr-x 3 root root 4096 Jan 22 02:09 mm >> -rw-r--r-- 1 root root 1218952 Jan 22 01:29 Module.symvers >> drwxr-xr-x 60 root root 4096 Jan 22 02:09 net >> drwxr-xr-x 16 root root 4096 Jan 22 02:09 samples >> drwxr-xr-x 13 root root 20480 Jan 22 02:09 scripts >> drwxr-xr-x 9 root root 4096 Jan 22 02:09 security >> drwxr-xr-x 23 root root 4096 Jan 22 02:09 sound >> drwxr-xr-x 10 root root 4096 Jan 22 02:09 spl >> drwxr-xr-x 21 root root 4096 Jan 22 02:09 tools >> drwxr-xr-x 8 root root 4096 Jan 22 02:09 ubuntu >> drwxr-xr-x 2 root root 4096 Jan 22 02:09 usr >> drwxr-xr-x 4 root root 4096 Jan 22 02:09 virt >> drwxr-xr-x 13 root root 4096 Jan 22 02:09 zfs >> >> Am I missing something? Are debuginfo files output somewhere else during >> compilation from source? > > Based on the paths stored when compiling your kernel, systemtap is > looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. > Obviously it isn't there. Can you search around and see if you can find > it and link it to its proper place? I can't find it (find / -type f -name '*vmlinux*'). When I compiled from source, I noticed this lines in Makefile for Proxmox VE kernel: # strip debug info find tmp/lib/modules -name \*.ko -print | while read f ; do strip --strip-debug "$$f"; done I thought that when I comment this out my kernel will have debug info compiled into it. Interestingly, the config file for my kernel says that CONFIG_DEBUG_INFO is enabled: # grep DEBUG_INFO /boot/config-4.4.35-2-pve CONFIG_DEBUG_INFO=y # CONFIG_DEBUG_INFO_REDUCED is not set # CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF4=y ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 21:27 ` Adam Guderski @ 2017-01-26 21:34 ` David Smith 2017-01-26 21:52 ` Adam Guderski 0 siblings, 1 reply; 10+ messages in thread From: David Smith @ 2017-01-26 21:34 UTC (permalink / raw) To: Adam Guderski, systemtap On 01/26/2017 03:26 PM, Adam Guderski wrote: ... stuff deleted ... >> Based on the paths stored when compiling your kernel, systemtap is >> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. >> Obviously it isn't there. Can you search around and see if you can find >> it and link it to its proper place? > > I can't find it (find / -type f -name '*vmlinux*'). When I compiled from > source, I noticed this lines in Makefile for Proxmox VE kernel: > > # strip debug info > find tmp/lib/modules -name \*.ko -print | while read f ; do strip > --strip-debug "$$f"; done > > I thought that when I comment this out my kernel will have debug info > compiled into it. That line you commented out is stripping debuginfo from kernel modules (*.ko), not the kernel itself. > Interestingly, the config file for my kernel says that > CONFIG_DEBUG_INFO is enabled: > > # grep DEBUG_INFO /boot/config-4.4.35-2-pve > CONFIG_DEBUG_INFO=y > # CONFIG_DEBUG_INFO_REDUCED is not set > # CONFIG_DEBUG_INFO_SPLIT is not set > CONFIG_DEBUG_INFO_DWARF4=y I'd guess you missed another line in the Proxmox kernel Makefile where it strips debuginfo from the kernel itself. What does "file" say on the kernel you built? -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 21:34 ` David Smith @ 2017-01-26 21:52 ` Adam Guderski 2017-01-26 22:41 ` David Smith 0 siblings, 1 reply; 10+ messages in thread From: Adam Guderski @ 2017-01-26 21:52 UTC (permalink / raw) To: David Smith, systemtap On 26/01/17 22:34, David Smith wrote: > On 01/26/2017 03:26 PM, Adam Guderski wrote: > > ... stuff deleted ... > >>> Based on the paths stored when compiling your kernel, systemtap is >>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. >>> Obviously it isn't there. Can you search around and see if you can find >>> it and link it to its proper place? >> >> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from >> source, I noticed this lines in Makefile for Proxmox VE kernel: >> >> # strip debug info >> find tmp/lib/modules -name \*.ko -print | while read f ; do strip >> --strip-debug "$$f"; done >> >> I thought that when I comment this out my kernel will have debug info >> compiled into it. > > That line you commented out is stripping debuginfo from kernel modules > (*.ko), not the kernel itself. > >> Interestingly, the config file for my kernel says that >> CONFIG_DEBUG_INFO is enabled: >> >> # grep DEBUG_INFO /boot/config-4.4.35-2-pve >> CONFIG_DEBUG_INFO=y >> # CONFIG_DEBUG_INFO_REDUCED is not set >> # CONFIG_DEBUG_INFO_SPLIT is not set >> CONFIG_DEBUG_INFO_DWARF4=y > > I'd guess you missed another line in the Proxmox kernel Makefile where > it strips debuginfo from the kernel itself. > > What does "file" say on the kernel you built? > It says: # file /boot/vmlinuz-4.4.35-2-pve /boot/vmlinuz-4.4.35-2-pve: Linux kernel x86 boot executable bzImage, version 4.4.35-2-pve (root@pve-test) #1 SMP Sun Jan 22 00:59:19 CET 201, RO-rootFS, swap_dev 0x6, Normal VGA While looking for the vmlinux file, I found a script that (I think) uncompresses a compressed kernel, so I did some testing: # /usr/src/linux-headers-4.4.35-2-pve/scripts/extract-vmlinux /boot/vmlinuz-4.4.35-2-pve > vmlinux # file vmlinux vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=c23f97de3c8765073fbecf0b529383f1305c8e81, stripped So yes, this appears that the kernel I compiled is still stripped, so I missed something, I'll give it another look but the documentation is sparse and I don't see another strip command in it. If you wonder how this Makefile looks like, it's here: https://git.proxmox.com/?p=pve-kernel.git;a=blob_plain;f=Makefile;hb=HEAD Best regards and many thanks for your help. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 21:52 ` Adam Guderski @ 2017-01-26 22:41 ` David Smith 2017-01-27 2:04 ` Adam Guderski 0 siblings, 1 reply; 10+ messages in thread From: David Smith @ 2017-01-26 22:41 UTC (permalink / raw) To: Adam Guderski, systemtap On 01/26/2017 03:51 PM, Adam Guderski wrote: > On 26/01/17 22:34, David Smith wrote: >> On 01/26/2017 03:26 PM, Adam Guderski wrote: >> >> ... stuff deleted ... >> >>>> Based on the paths stored when compiling your kernel, systemtap is >>>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. >>>> Obviously it isn't there. Can you search around and see if you can find >>>> it and link it to its proper place? >>> >>> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from >>> source, I noticed this lines in Makefile for Proxmox VE kernel: >>> >>> # strip debug info >>> find tmp/lib/modules -name \*.ko -print | while read f ; do strip >>> --strip-debug "$$f"; done >>> >>> I thought that when I comment this out my kernel will have debug info >>> compiled into it. >> >> That line you commented out is stripping debuginfo from kernel modules >> (*.ko), not the kernel itself. >> >>> Interestingly, the config file for my kernel says that >>> CONFIG_DEBUG_INFO is enabled: >>> >>> # grep DEBUG_INFO /boot/config-4.4.35-2-pve >>> CONFIG_DEBUG_INFO=y >>> # CONFIG_DEBUG_INFO_REDUCED is not set >>> # CONFIG_DEBUG_INFO_SPLIT is not set >>> CONFIG_DEBUG_INFO_DWARF4=y I doubt this will make a difference, but CONFIG_DEBUG_INFO_DWARF4 isn't usually set on the kernels we use. See <https://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel>. >> I'd guess you missed another line in the Proxmox kernel Makefile where >> it strips debuginfo from the kernel itself. >> >> What does "file" say on the kernel you built? > > It says: > > # file /boot/vmlinuz-4.4.35-2-pve > /boot/vmlinuz-4.4.35-2-pve: Linux kernel x86 boot executable bzImage, > version 4.4.35-2-pve (root@pve-test) #1 SMP Sun Jan 22 00:59:19 CET 201, > RO-rootFS, swap_dev 0x6, Normal VGA > > While looking for the vmlinux file, I found a script that (I think) > uncompresses a compressed kernel, so I did some testing: > > # /usr/src/linux-headers-4.4.35-2-pve/scripts/extract-vmlinux > /boot/vmlinuz-4.4.35-2-pve > vmlinux > # file vmlinux > vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically > linked, BuildID[sha1]=c23f97de3c8765073fbecf0b529383f1305c8e81, stripped > > So yes, this appears that the kernel I compiled is still stripped, so I > missed something, I'll give it another look but the documentation is > sparse and I don't see another strip command in it. If you wonder how > this Makefile looks like, it's here: > https://git.proxmox.com/?p=pve-kernel.git;a=blob_plain;f=Makefile;hb=HEAD > > Best regards and many thanks for your help. I looked through that Makefile and I didn't see where the kernel gets stripped either. You are going to have to dig deeper to find where vmlinux gets deleted. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Problem with vfs probe on Proxmox kernel 2017-01-26 22:41 ` David Smith @ 2017-01-27 2:04 ` Adam Guderski 0 siblings, 0 replies; 10+ messages in thread From: Adam Guderski @ 2017-01-27 2:04 UTC (permalink / raw) To: David Smith, systemtap On 26/01/17 23:41, David Smith wrote: > On 01/26/2017 03:51 PM, Adam Guderski wrote: >> On 26/01/17 22:34, David Smith wrote: >>> On 01/26/2017 03:26 PM, Adam Guderski wrote: >>> >>> ... stuff deleted ... >>> >>>>> Based on the paths stored when compiling your kernel, systemtap is >>>>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'. >>>>> Obviously it isn't there. Can you search around and see if you can find >>>>> it and link it to its proper place? >>>> >>>> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from >>>> source, I noticed this lines in Makefile for Proxmox VE kernel: >>>> >>>> # strip debug info >>>> find tmp/lib/modules -name \*.ko -print | while read f ; do strip >>>> --strip-debug "$$f"; done >>>> >>>> I thought that when I comment this out my kernel will have debug info >>>> compiled into it. >>> >>> That line you commented out is stripping debuginfo from kernel modules >>> (*.ko), not the kernel itself. >>> >>>> Interestingly, the config file for my kernel says that >>>> CONFIG_DEBUG_INFO is enabled: >>>> >>>> # grep DEBUG_INFO /boot/config-4.4.35-2-pve >>>> CONFIG_DEBUG_INFO=y >>>> # CONFIG_DEBUG_INFO_REDUCED is not set >>>> # CONFIG_DEBUG_INFO_SPLIT is not set >>>> CONFIG_DEBUG_INFO_DWARF4=y > > I doubt this will make a difference, but CONFIG_DEBUG_INFO_DWARF4 isn't > usually set on the kernels we use. See > <https://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel>. > >>> I'd guess you missed another line in the Proxmox kernel Makefile where >>> it strips debuginfo from the kernel itself. >>> >>> What does "file" say on the kernel you built? >> >> It says: >> >> # file /boot/vmlinuz-4.4.35-2-pve >> /boot/vmlinuz-4.4.35-2-pve: Linux kernel x86 boot executable bzImage, >> version 4.4.35-2-pve (root@pve-test) #1 SMP Sun Jan 22 00:59:19 CET 201, >> RO-rootFS, swap_dev 0x6, Normal VGA >> >> While looking for the vmlinux file, I found a script that (I think) >> uncompresses a compressed kernel, so I did some testing: >> >> # /usr/src/linux-headers-4.4.35-2-pve/scripts/extract-vmlinux >> /boot/vmlinuz-4.4.35-2-pve > vmlinux >> # file vmlinux >> vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically >> linked, BuildID[sha1]=c23f97de3c8765073fbecf0b529383f1305c8e81, stripped >> >> So yes, this appears that the kernel I compiled is still stripped, so I >> missed something, I'll give it another look but the documentation is >> sparse and I don't see another strip command in it. If you wonder how >> this Makefile looks like, it's here: >> https://git.proxmox.com/?p=pve-kernel.git;a=blob_plain;f=Makefile;hb=HEAD >> >> Best regards and many thanks for your help. > > I looked through that Makefile and I didn't see where the kernel gets > stripped either. You are going to have to dig deeper to find where > vmlinux gets deleted. > I have found the vmlinux file - it was in directory ubuntu-xenial/vmlinux (relative to root directory of Proxmox kernel sources), and the reason I haven't found it earlier is because I ran "make clean" some time earlier. Apparently, when I linked this file to /lib/modules/4.4.35-2-pve/build/vmlinux, stap no longer complains when I run scripts requiring vfs kernel probles. David, I can't express how much you helped me, I thank you one more time and wish you very best! ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-01-27 2:04 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-01-25 22:44 Problem with vfs probe on Proxmox kernel Adam Guderski 2017-01-26 4:51 ` Arkady 2017-01-26 17:38 ` David Smith 2017-01-26 18:43 ` Adam Guderski 2017-01-26 19:05 ` David Smith 2017-01-26 21:27 ` Adam Guderski 2017-01-26 21:34 ` David Smith 2017-01-26 21:52 ` Adam Guderski 2017-01-26 22:41 ` David Smith 2017-01-27 2:04 ` Adam Guderski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).