Problem with vfs probe on Proxmox kernel

Problem with vfs probe on Proxmox kernel

[Adam Guderski]

Hello,

I've got problem running sample traceio.stp script (obtained from
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
uses a custom Ubuntu-based kernel. The Proxmox repository does not
provide a kernel with debug symbols, so I had to compile it from
provided sources - version 4.4.35-2-pve, to be specific. Debian repos
(jessie) provide systemtap in version 2.9, which from what I read does
not support 4.4 kernels, so I compiled systemtap from sources (from
master branch, version 3.1/0.159).

With such setup, I'm able to run sample "hello world" stap script and
couple of others, but all scripts with vfs probes fail. I attach full
verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
the main error can be explained below. What can I do about it? I
searched for couple of hours but did not find anything that would help.

Best regards,
Adam


# stap -v traceio2.stp 0xfb00
Pass 1: parsed user script and 119 library scripts using
71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
semantic error: while resolving probe point: identifier 'kernel' at
/usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
        source: probe vfs.write = kernel.function("vfs_write")
                                  ^

semantic error: missing x86_64 kernel/module debuginfo [man
warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'

semantic error: resolution failed in alias expansion builder

semantic error: while resolving probe point: identifier 'vfs' at
traceio2.stp:14:7
        source: probe vfs.write, vfs.read
                      ^

semantic error: no match

semantic error: while resolving probe point: identifier 'kernel' at
/usr/local/share/systemtap/tapset/linux/vfs.stp:915:18
        source: probe vfs.read = kernel.function("vfs_read")
                                 ^

semantic error: missing x86_64 kernel/module debuginfo [man
warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'

semantic error: resolution failed in alias expansion builder

semantic error: while resolving probe point: identifier 'vfs' at
traceio2.stp:14:18
        source: probe vfs.write, vfs.read
                                 ^

semantic error: no match

Pass 2: analyzed script: 1 probe, 33 functions, 1 embed, 1 global using
72384virt/46080res/6016shr/40680data kb, in 40usr/100sys/146real ms.
Pass 2: analysis failed.  [man error::pass2]

Re: Problem with vfs probe on Proxmox kernel

[Arkady]

I am running
Systemtap translator/driver (version 2.9/0.165, Debian version
2.9-2ubuntu2 (xenial))
on
Linux ubuntu 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux

and vfsread/vfswrite work fine.

On Thu, Jan 26, 2017 at 12:44 AM, Adam Guderski <a.guderski@gmail.com> wrote:
> Hello,
>
> I've got problem running sample traceio.stp script (obtained from
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
> uses a custom Ubuntu-based kernel. The Proxmox repository does not
> provide a kernel with debug symbols, so I had to compile it from
> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
> (jessie) provide systemtap in version 2.9, which from what I read does
> not support 4.4 kernels, so I compiled systemtap from sources (from
> master branch, version 3.1/0.159).
>
> With such setup, I'm able to run sample "hello world" stap script and
> couple of others, but all scripts with vfs probes fail. I attach full
> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
> the main error can be explained below. What can I do about it? I
> searched for couple of hours but did not find anything that would help.
>
> Best regards,
> Adam
>
>
> # stap -v traceio2.stp 0xfb00
> Pass 1: parsed user script and 119 library scripts using
> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
> semantic error: while resolving probe point: identifier 'kernel' at
> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>         source: probe vfs.write = kernel.function("vfs_write")
>                                   ^
>
> semantic error: missing x86_64 kernel/module debuginfo [man
> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
>
> semantic error: resolution failed in alias expansion builder
>
> semantic error: while resolving probe point: identifier 'vfs' at
> traceio2.stp:14:7
>         source: probe vfs.write, vfs.read
>                       ^
>
> semantic error: no match
>
> semantic error: while resolving probe point: identifier 'kernel' at
> /usr/local/share/systemtap/tapset/linux/vfs.stp:915:18
>         source: probe vfs.read = kernel.function("vfs_read")
>                                  ^
>
> semantic error: missing x86_64 kernel/module debuginfo [man
> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
>
> semantic error: resolution failed in alias expansion builder
>
> semantic error: while resolving probe point: identifier 'vfs' at
> traceio2.stp:14:18
>         source: probe vfs.write, vfs.read
>                                  ^
>
> semantic error: no match
>
> Pass 2: analyzed script: 1 probe, 33 functions, 1 embed, 1 global using
> 72384virt/46080res/6016shr/40680data kb, in 40usr/100sys/146real ms.
> Pass 2: analysis failed.  [man error::pass2]

Re: Problem with vfs probe on Proxmox kernel

[David Smith]

On 01/25/2017 04:44 PM, Adam Guderski wrote:
> Hello,
> 
> I've got problem running sample traceio.stp script (obtained from
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
> uses a custom Ubuntu-based kernel. The Proxmox repository does not
> provide a kernel with debug symbols, so I had to compile it from
> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
> (jessie) provide systemtap in version 2.9, which from what I read does
> not support 4.4 kernels, so I compiled systemtap from sources (from
> master branch, version 3.1/0.159).
> 
> With such setup, I'm able to run sample "hello world" stap script and
> couple of others, but all scripts with vfs probes fail. I attach full
> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
> the main error can be explained below. What can I do about it? I
> searched for couple of hours but did not find anything that would help.

As the following states:

> # stap -v traceio2.stp 0xfb00
> Pass 1: parsed user script and 119 library scripts using
> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
> semantic error: while resolving probe point: identifier 'kernel' at
> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>         source: probe vfs.write = kernel.function("vfs_write")
>                                   ^
> 
> semantic error: missing x86_64 kernel/module debuginfo [man
> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'

Note that a systemtap "hello world" program doesn't need kernel
debuginfo, but a script using vfs probes does need kernel debuginfo.
Does any script that needs debuginfo work, or is it just vfs probes? To
test, try 'stap -l 'kernel.function("*_write")'. You should see items
like 'sys_write' and 'vfs_write' in the output.

systemtap is expecting to find the kernel's debuginfo at
/lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need
to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment
variable to tell systemtap where you've put the debuginfo.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Problem with vfs probe on Proxmox kernel

[Adam Guderski]

On 26/01/17 18:37, David Smith wrote:
> On 01/25/2017 04:44 PM, Adam Guderski wrote:
>> Hello,
>>
>> I've got problem running sample traceio.stp script (obtained from
>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
>> uses a custom Ubuntu-based kernel. The Proxmox repository does not
>> provide a kernel with debug symbols, so I had to compile it from
>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
>> (jessie) provide systemtap in version 2.9, which from what I read does
>> not support 4.4 kernels, so I compiled systemtap from sources (from
>> master branch, version 3.1/0.159).
>>
>> With such setup, I'm able to run sample "hello world" stap script and
>> couple of others, but all scripts with vfs probes fail. I attach full
>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
>> the main error can be explained below. What can I do about it? I
>> searched for couple of hours but did not find anything that would help.
> 
> As the following states:
> 
>> # stap -v traceio2.stp 0xfb00
>> Pass 1: parsed user script and 119 library scripts using
>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
>> semantic error: while resolving probe point: identifier 'kernel' at
>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>>         source: probe vfs.write = kernel.function("vfs_write")
>>                                   ^
>>
>> semantic error: missing x86_64 kernel/module debuginfo [man
>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
> 
> Note that a systemtap "hello world" program doesn't need kernel
> debuginfo, but a script using vfs probes does need kernel debuginfo.
> Does any script that needs debuginfo work, or is it just vfs probes? To
> test, try 'stap -l 'kernel.function("*_write")'. You should see items
> like 'sys_write' and 'vfs_write' in the output. 

Sadly, the command "stap -l 'kernel.function("*_write")'" returns
nothing at all.

> systemtap is expecting to find the kernel's debuginfo at
> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need
> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment
> variable to tell systemtap where you've put the debuginfo.

Could you elaborate how debuginfo files look like? This is what I've got
in the directory:

# ls -la /lib/modules/4.4.35-2-pve/build/
total 1564
drwxr-xr-x  26 root root    4096 Jan 22 02:09 .
drwxr-xr-x   3 root root    4096 Jan 18 00:52 ..
drwxr-xr-x  33 root root    4096 Jan 22 02:09 arch
drwxr-xr-x   3 root root    4096 Jan 22 02:09 block
drwxr-xr-x   2 root root    4096 Jan 22 02:09 certs
-rw-r--r--   1 root root  189740 Jan 22 01:29 .config
drwxr-xr-x   4 root root    4096 Jan 22 02:09 crypto
drwxr-xr-x 127 root root    4096 Jan 22 02:09 drivers
drwxr-xr-x   2 root root    4096 Jan 22 02:09 firmware
drwxr-xr-x  74 root root    4096 Jan 22 02:09 fs
drwxr-xr-x  30 root root    4096 Jan 22 02:09 include
drwxr-xr-x   2 root root    4096 Jan 22 02:09 init
drwxr-xr-x   2 root root    4096 Jan 22 02:09 ipc
-rw-r--r--   1 root root    2622 Dec 22 08:43 Kbuild
-rw-r--r--   1 root root     252 Dec 22 08:43 Kconfig
drwxr-xr-x  15 root root    4096 Jan 22 02:09 kernel
drwxr-xr-x  12 root root    4096 Jan 22 02:09 lib
-rw-r--r--   1 root root   56164 Jan 22 00:46 Makefile
drwxr-xr-x   3 root root    4096 Jan 22 02:09 mm
-rw-r--r--   1 root root 1218952 Jan 22 01:29 Module.symvers
drwxr-xr-x  60 root root    4096 Jan 22 02:09 net
drwxr-xr-x  16 root root    4096 Jan 22 02:09 samples
drwxr-xr-x  13 root root   20480 Jan 22 02:09 scripts
drwxr-xr-x   9 root root    4096 Jan 22 02:09 security
drwxr-xr-x  23 root root    4096 Jan 22 02:09 sound
drwxr-xr-x  10 root root    4096 Jan 22 02:09 spl
drwxr-xr-x  21 root root    4096 Jan 22 02:09 tools
drwxr-xr-x   8 root root    4096 Jan 22 02:09 ubuntu
drwxr-xr-x   2 root root    4096 Jan 22 02:09 usr
drwxr-xr-x   4 root root    4096 Jan 22 02:09 virt
drwxr-xr-x  13 root root    4096 Jan 22 02:09 zfs

Am I missing something? Are debuginfo files output somewhere else during
compilation from source?

Re: Problem with vfs probe on Proxmox kernel

[David Smith]

On 01/26/2017 12:42 PM, Adam Guderski wrote:
> On 26/01/17 18:37, David Smith wrote:
>> On 01/25/2017 04:44 PM, Adam Guderski wrote:
>>> Hello,
>>>
>>> I've got problem running sample traceio.stp script (obtained from
>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
>>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
>>> uses a custom Ubuntu-based kernel. The Proxmox repository does not
>>> provide a kernel with debug symbols, so I had to compile it from
>>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
>>> (jessie) provide systemtap in version 2.9, which from what I read does
>>> not support 4.4 kernels, so I compiled systemtap from sources (from
>>> master branch, version 3.1/0.159).
>>>
>>> With such setup, I'm able to run sample "hello world" stap script and
>>> couple of others, but all scripts with vfs probes fail. I attach full
>>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
>>> the main error can be explained below. What can I do about it? I
>>> searched for couple of hours but did not find anything that would help.
>>
>> As the following states:
>>
>>> # stap -v traceio2.stp 0xfb00
>>> Pass 1: parsed user script and 119 library scripts using
>>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
>>> semantic error: while resolving probe point: identifier 'kernel' at
>>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>>>         source: probe vfs.write = kernel.function("vfs_write")
>>>                                   ^
>>>
>>> semantic error: missing x86_64 kernel/module debuginfo [man
>>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
>>
>> Note that a systemtap "hello world" program doesn't need kernel
>> debuginfo, but a script using vfs probes does need kernel debuginfo.
>> Does any script that needs debuginfo work, or is it just vfs probes? To
>> test, try 'stap -l 'kernel.function("*_write")'. You should see items
>> like 'sys_write' and 'vfs_write' in the output. 
> 
> Sadly, the command "stap -l 'kernel.function("*_write")'" returns
> nothing at all.

That's actually good. This definitely means that stap can't find any
debuginfo (vs. the problem being stap found the debuginfo but didn't
find vfs_write() in the debuginfo).

>> systemtap is expecting to find the kernel's debuginfo at
>> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need
>> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment
>> variable to tell systemtap where you've put the debuginfo.
> 
> Could you elaborate how debuginfo files look like? This is what I've got
> in the directory:

For unstripped userspace exes, the debuginfo is present in the exe. If
you probe an unstripped userspace exe, systemtap shouldn't have any
trouble finding the debuginfo. To make the exes smaller, the debuginfo
can be extracted and put in another file, typically in
/usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for
/bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug.

For kernels, things are a bit more tricky. You typically boot off a
compressed stripped kernel image, most likely called
/boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The
debuginfo is typically stored in an unstripped uncompressed kernel file
down in /usr/lib/debug. Here's on a Fedora 25 system I see:

====
# ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
-rwxr-xr-x. 1 root root   6891144 Jan 20 07:43
/boot/vmlinuz-4.9.5-200.fc25.x86_64
-rwxr-xr-x. 1 root root 541141280 Jan 20 07:48
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
# file /boot/vmlinuz-4.9.5-200.fc25.x86_64
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
/boot/vmlinuz-4.9.5-200.fc25.x86_64:                      Linux kernel
x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64
(mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6,
Normal VGA
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB
executable, x86-64, version 1 (SYSV), statically linked,
BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped
====

As you can see the vmlinux file is much bigger than the vmlinuz file,
since it is unstripped and uncompressed.

Now those paths are all for distro files, and you've got a self-built
kernel. But, the principle is still the same.

> # ls -la /lib/modules/4.4.35-2-pve/build/
> total 1564
> drwxr-xr-x  26 root root    4096 Jan 22 02:09 .
> drwxr-xr-x   3 root root    4096 Jan 18 00:52 ..
> drwxr-xr-x  33 root root    4096 Jan 22 02:09 arch
> drwxr-xr-x   3 root root    4096 Jan 22 02:09 block
> drwxr-xr-x   2 root root    4096 Jan 22 02:09 certs
> -rw-r--r--   1 root root  189740 Jan 22 01:29 .config
> drwxr-xr-x   4 root root    4096 Jan 22 02:09 crypto
> drwxr-xr-x 127 root root    4096 Jan 22 02:09 drivers
> drwxr-xr-x   2 root root    4096 Jan 22 02:09 firmware
> drwxr-xr-x  74 root root    4096 Jan 22 02:09 fs
> drwxr-xr-x  30 root root    4096 Jan 22 02:09 include
> drwxr-xr-x   2 root root    4096 Jan 22 02:09 init
> drwxr-xr-x   2 root root    4096 Jan 22 02:09 ipc
> -rw-r--r--   1 root root    2622 Dec 22 08:43 Kbuild
> -rw-r--r--   1 root root     252 Dec 22 08:43 Kconfig
> drwxr-xr-x  15 root root    4096 Jan 22 02:09 kernel
> drwxr-xr-x  12 root root    4096 Jan 22 02:09 lib
> -rw-r--r--   1 root root   56164 Jan 22 00:46 Makefile
> drwxr-xr-x   3 root root    4096 Jan 22 02:09 mm
> -rw-r--r--   1 root root 1218952 Jan 22 01:29 Module.symvers
> drwxr-xr-x  60 root root    4096 Jan 22 02:09 net
> drwxr-xr-x  16 root root    4096 Jan 22 02:09 samples
> drwxr-xr-x  13 root root   20480 Jan 22 02:09 scripts
> drwxr-xr-x   9 root root    4096 Jan 22 02:09 security
> drwxr-xr-x  23 root root    4096 Jan 22 02:09 sound
> drwxr-xr-x  10 root root    4096 Jan 22 02:09 spl
> drwxr-xr-x  21 root root    4096 Jan 22 02:09 tools
> drwxr-xr-x   8 root root    4096 Jan 22 02:09 ubuntu
> drwxr-xr-x   2 root root    4096 Jan 22 02:09 usr
> drwxr-xr-x   4 root root    4096 Jan 22 02:09 virt
> drwxr-xr-x  13 root root    4096 Jan 22 02:09 zfs
> 
> Am I missing something? Are debuginfo files output somewhere else during
> compilation from source?

Based on the paths stored when compiling your kernel, systemtap is
looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
Obviously it isn't there. Can you search around and see if you can find
it and link it to its proper place?

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Problem with vfs probe on Proxmox kernel

[Adam Guderski]

On 26/01/17 20:05, David Smith wrote:
> On 01/26/2017 12:42 PM, Adam Guderski wrote:
>> On 26/01/17 18:37, David Smith wrote:
>>> On 01/25/2017 04:44 PM, Adam Guderski wrote:
>>>> Hello,
>>>>
>>>> I've got problem running sample traceio.stp script (obtained from
>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
>>>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
>>>> uses a custom Ubuntu-based kernel. The Proxmox repository does not
>>>> provide a kernel with debug symbols, so I had to compile it from
>>>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
>>>> (jessie) provide systemtap in version 2.9, which from what I read does
>>>> not support 4.4 kernels, so I compiled systemtap from sources (from
>>>> master branch, version 3.1/0.159).
>>>>
>>>> With such setup, I'm able to run sample "hello world" stap script and
>>>> couple of others, but all scripts with vfs probes fail. I attach full
>>>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
>>>> the main error can be explained below. What can I do about it? I
>>>> searched for couple of hours but did not find anything that would help.
>>>
>>> As the following states:
>>>
>>>> # stap -v traceio2.stp 0xfb00
>>>> Pass 1: parsed user script and 119 library scripts using
>>>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
>>>> semantic error: while resolving probe point: identifier 'kernel' at
>>>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>>>>         source: probe vfs.write = kernel.function("vfs_write")
>>>>                                   ^
>>>>
>>>> semantic error: missing x86_64 kernel/module debuginfo [man
>>>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
>>>
>>> Note that a systemtap "hello world" program doesn't need kernel
>>> debuginfo, but a script using vfs probes does need kernel debuginfo.
>>> Does any script that needs debuginfo work, or is it just vfs probes? To
>>> test, try 'stap -l 'kernel.function("*_write")'. You should see items
>>> like 'sys_write' and 'vfs_write' in the output. 
>>
>> Sadly, the command "stap -l 'kernel.function("*_write")'" returns
>> nothing at all.
> 
> That's actually good. This definitely means that stap can't find any
> debuginfo (vs. the problem being stap found the debuginfo but didn't
> find vfs_write() in the debuginfo).
> 
>>> systemtap is expecting to find the kernel's debuginfo at
>>> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need
>>> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment
>>> variable to tell systemtap where you've put the debuginfo.
>>
>> Could you elaborate how debuginfo files look like? This is what I've got
>> in the directory:
> 
> For unstripped userspace exes, the debuginfo is present in the exe. If
> you probe an unstripped userspace exe, systemtap shouldn't have any
> trouble finding the debuginfo. To make the exes smaller, the debuginfo
> can be extracted and put in another file, typically in
> /usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for
> /bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug.
> 
> For kernels, things are a bit more tricky. You typically boot off a
> compressed stripped kernel image, most likely called
> /boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The
> debuginfo is typically stored in an unstripped uncompressed kernel file
> down in /usr/lib/debug. Here's on a Fedora 25 system I see:
> 
> ====
> # ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
> -rwxr-xr-x. 1 root root   6891144 Jan 20 07:43
> /boot/vmlinuz-4.9.5-200.fc25.x86_64
> -rwxr-xr-x. 1 root root 541141280 Jan 20 07:48
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
> # file /boot/vmlinuz-4.9.5-200.fc25.x86_64
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
> /boot/vmlinuz-4.9.5-200.fc25.x86_64:                      Linux kernel
> x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64
> (mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6,
> Normal VGA
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB
> executable, x86-64, version 1 (SYSV), statically linked,
> BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped
> ====
> 
> As you can see the vmlinux file is much bigger than the vmlinuz file,
> since it is unstripped and uncompressed.
> 
> Now those paths are all for distro files, and you've got a self-built
> kernel. But, the principle is still the same.
> 
>> # ls -la /lib/modules/4.4.35-2-pve/build/
>> total 1564
>> drwxr-xr-x  26 root root    4096 Jan 22 02:09 .
>> drwxr-xr-x   3 root root    4096 Jan 18 00:52 ..
>> drwxr-xr-x  33 root root    4096 Jan 22 02:09 arch
>> drwxr-xr-x   3 root root    4096 Jan 22 02:09 block
>> drwxr-xr-x   2 root root    4096 Jan 22 02:09 certs
>> -rw-r--r--   1 root root  189740 Jan 22 01:29 .config
>> drwxr-xr-x   4 root root    4096 Jan 22 02:09 crypto
>> drwxr-xr-x 127 root root    4096 Jan 22 02:09 drivers
>> drwxr-xr-x   2 root root    4096 Jan 22 02:09 firmware
>> drwxr-xr-x  74 root root    4096 Jan 22 02:09 fs
>> drwxr-xr-x  30 root root    4096 Jan 22 02:09 include
>> drwxr-xr-x   2 root root    4096 Jan 22 02:09 init
>> drwxr-xr-x   2 root root    4096 Jan 22 02:09 ipc
>> -rw-r--r--   1 root root    2622 Dec 22 08:43 Kbuild
>> -rw-r--r--   1 root root     252 Dec 22 08:43 Kconfig
>> drwxr-xr-x  15 root root    4096 Jan 22 02:09 kernel
>> drwxr-xr-x  12 root root    4096 Jan 22 02:09 lib
>> -rw-r--r--   1 root root   56164 Jan 22 00:46 Makefile
>> drwxr-xr-x   3 root root    4096 Jan 22 02:09 mm
>> -rw-r--r--   1 root root 1218952 Jan 22 01:29 Module.symvers
>> drwxr-xr-x  60 root root    4096 Jan 22 02:09 net
>> drwxr-xr-x  16 root root    4096 Jan 22 02:09 samples
>> drwxr-xr-x  13 root root   20480 Jan 22 02:09 scripts
>> drwxr-xr-x   9 root root    4096 Jan 22 02:09 security
>> drwxr-xr-x  23 root root    4096 Jan 22 02:09 sound
>> drwxr-xr-x  10 root root    4096 Jan 22 02:09 spl
>> drwxr-xr-x  21 root root    4096 Jan 22 02:09 tools
>> drwxr-xr-x   8 root root    4096 Jan 22 02:09 ubuntu
>> drwxr-xr-x   2 root root    4096 Jan 22 02:09 usr
>> drwxr-xr-x   4 root root    4096 Jan 22 02:09 virt
>> drwxr-xr-x  13 root root    4096 Jan 22 02:09 zfs
>>
>> Am I missing something? Are debuginfo files output somewhere else during
>> compilation from source?
> 
> Based on the paths stored when compiling your kernel, systemtap is
> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
> Obviously it isn't there. Can you search around and see if you can find
> it and link it to its proper place?

I can't find it (find / -type f -name '*vmlinux*'). When I compiled from
source, I noticed this lines in Makefile for Proxmox VE kernel:

# strip debug info
find tmp/lib/modules -name \*.ko -print | while read f ; do strip
--strip-debug "$$f"; done

I thought that when I comment this out my kernel will have debug info
compiled into it. Interestingly, the config file for my kernel says that
CONFIG_DEBUG_INFO is enabled:

# grep  DEBUG_INFO /boot/config-4.4.35-2-pve
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y

Re: Problem with vfs probe on Proxmox kernel

[David Smith]

On 01/26/2017 03:26 PM, Adam Guderski wrote:

... stuff deleted ...

>> Based on the paths stored when compiling your kernel, systemtap is
>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
>> Obviously it isn't there. Can you search around and see if you can find
>> it and link it to its proper place?
> 
> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from
> source, I noticed this lines in Makefile for Proxmox VE kernel:
> 
> # strip debug info
> find tmp/lib/modules -name \*.ko -print | while read f ; do strip
> --strip-debug "$$f"; done
> 
> I thought that when I comment this out my kernel will have debug info
> compiled into it.

That line you commented out is stripping debuginfo from kernel modules
(*.ko), not the kernel itself.

> Interestingly, the config file for my kernel says that
> CONFIG_DEBUG_INFO is enabled:
> 
> # grep  DEBUG_INFO /boot/config-4.4.35-2-pve
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF4=y

I'd guess you missed another line in the Proxmox kernel Makefile where
it strips debuginfo from the kernel itself.

What does "file" say on the kernel you built?

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Problem with vfs probe on Proxmox kernel

[Adam Guderski]

On 26/01/17 22:34, David Smith wrote:
> On 01/26/2017 03:26 PM, Adam Guderski wrote:
> 
> ... stuff deleted ...
> 
>>> Based on the paths stored when compiling your kernel, systemtap is
>>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
>>> Obviously it isn't there. Can you search around and see if you can find
>>> it and link it to its proper place?
>>
>> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from
>> source, I noticed this lines in Makefile for Proxmox VE kernel:
>>
>> # strip debug info
>> find tmp/lib/modules -name \*.ko -print | while read f ; do strip
>> --strip-debug "$$f"; done
>>
>> I thought that when I comment this out my kernel will have debug info
>> compiled into it.
> 
> That line you commented out is stripping debuginfo from kernel modules
> (*.ko), not the kernel itself.
> 
>> Interestingly, the config file for my kernel says that
>> CONFIG_DEBUG_INFO is enabled:
>>
>> # grep  DEBUG_INFO /boot/config-4.4.35-2-pve
>> CONFIG_DEBUG_INFO=y
>> # CONFIG_DEBUG_INFO_REDUCED is not set
>> # CONFIG_DEBUG_INFO_SPLIT is not set
>> CONFIG_DEBUG_INFO_DWARF4=y
> 
> I'd guess you missed another line in the Proxmox kernel Makefile where
> it strips debuginfo from the kernel itself.
> 
> What does "file" say on the kernel you built?
> 


It says:

# file /boot/vmlinuz-4.4.35-2-pve
/boot/vmlinuz-4.4.35-2-pve: Linux kernel x86 boot executable bzImage,
version 4.4.35-2-pve (root@pve-test) #1 SMP Sun Jan 22 00:59:19 CET 201,
RO-rootFS, swap_dev 0x6, Normal VGA

While looking for the vmlinux file, I found a script that (I think)
uncompresses a compressed kernel, so I did some testing:

# /usr/src/linux-headers-4.4.35-2-pve/scripts/extract-vmlinux
/boot/vmlinuz-4.4.35-2-pve > vmlinux
# file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically
linked, BuildID[sha1]=c23f97de3c8765073fbecf0b529383f1305c8e81, stripped

So yes, this appears that the kernel I compiled is still stripped, so I
missed something, I'll give it another look but the documentation is
sparse and I don't see another strip command in it. If you wonder how
this Makefile looks like, it's here:
https://git.proxmox.com/?p=pve-kernel.git;a=blob_plain;f=Makefile;hb=HEAD

Best regards and many thanks for your help.

Re: Problem with vfs probe on Proxmox kernel

[David Smith]

On 01/26/2017 03:51 PM, Adam Guderski wrote:
> On 26/01/17 22:34, David Smith wrote:
>> On 01/26/2017 03:26 PM, Adam Guderski wrote:
>>
>> ... stuff deleted ...
>>
>>>> Based on the paths stored when compiling your kernel, systemtap is
>>>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
>>>> Obviously it isn't there. Can you search around and see if you can find
>>>> it and link it to its proper place?
>>>
>>> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from
>>> source, I noticed this lines in Makefile for Proxmox VE kernel:
>>>
>>> # strip debug info
>>> find tmp/lib/modules -name \*.ko -print | while read f ; do strip
>>> --strip-debug "$$f"; done
>>>
>>> I thought that when I comment this out my kernel will have debug info
>>> compiled into it.
>>
>> That line you commented out is stripping debuginfo from kernel modules
>> (*.ko), not the kernel itself.
>>
>>> Interestingly, the config file for my kernel says that
>>> CONFIG_DEBUG_INFO is enabled:
>>>
>>> # grep  DEBUG_INFO /boot/config-4.4.35-2-pve
>>> CONFIG_DEBUG_INFO=y
>>> # CONFIG_DEBUG_INFO_REDUCED is not set
>>> # CONFIG_DEBUG_INFO_SPLIT is not set
>>> CONFIG_DEBUG_INFO_DWARF4=y

I doubt this will make a difference, but CONFIG_DEBUG_INFO_DWARF4 isn't
usually set on the kernels we use. See
<https://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel>.

>> I'd guess you missed another line in the Proxmox kernel Makefile where
>> it strips debuginfo from the kernel itself.
>>
>> What does "file" say on the kernel you built?
> 
> It says:
> 
> # file /boot/vmlinuz-4.4.35-2-pve
> /boot/vmlinuz-4.4.35-2-pve: Linux kernel x86 boot executable bzImage,
> version 4.4.35-2-pve (root@pve-test) #1 SMP Sun Jan 22 00:59:19 CET 201,
> RO-rootFS, swap_dev 0x6, Normal VGA
> 
> While looking for the vmlinux file, I found a script that (I think)
> uncompresses a compressed kernel, so I did some testing:
> 
> # /usr/src/linux-headers-4.4.35-2-pve/scripts/extract-vmlinux
> /boot/vmlinuz-4.4.35-2-pve > vmlinux
> # file vmlinux
> vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically
> linked, BuildID[sha1]=c23f97de3c8765073fbecf0b529383f1305c8e81, stripped
> 
> So yes, this appears that the kernel I compiled is still stripped, so I
> missed something, I'll give it another look but the documentation is
> sparse and I don't see another strip command in it. If you wonder how
> this Makefile looks like, it's here:
> https://git.proxmox.com/?p=pve-kernel.git;a=blob_plain;f=Makefile;hb=HEAD
> 
> Best regards and many thanks for your help.

I looked through that Makefile and I didn't see where the kernel gets
stripped either. You are going to have to dig deeper to find where
vmlinux gets deleted.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Problem with vfs probe on Proxmox kernel

[Adam Guderski]

On 26/01/17 23:41, David Smith wrote:
> On 01/26/2017 03:51 PM, Adam Guderski wrote:
>> On 26/01/17 22:34, David Smith wrote:
>>> On 01/26/2017 03:26 PM, Adam Guderski wrote:
>>>
>>> ... stuff deleted ...
>>>
>>>>> Based on the paths stored when compiling your kernel, systemtap is
>>>>> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
>>>>> Obviously it isn't there. Can you search around and see if you can find
>>>>> it and link it to its proper place?
>>>>
>>>> I can't find it (find / -type f -name '*vmlinux*'). When I compiled from
>>>> source, I noticed this lines in Makefile for Proxmox VE kernel:
>>>>
>>>> # strip debug info
>>>> find tmp/lib/modules -name \*.ko -print | while read f ; do strip
>>>> --strip-debug "$$f"; done
>>>>
>>>> I thought that when I comment this out my kernel will have debug info
>>>> compiled into it.
>>>
>>> That line you commented out is stripping debuginfo from kernel modules
>>> (*.ko), not the kernel itself.
>>>
>>>> Interestingly, the config file for my kernel says that
>>>> CONFIG_DEBUG_INFO is enabled:
>>>>
>>>> # grep  DEBUG_INFO /boot/config-4.4.35-2-pve
>>>> CONFIG_DEBUG_INFO=y
>>>> # CONFIG_DEBUG_INFO_REDUCED is not set
>>>> # CONFIG_DEBUG_INFO_SPLIT is not set
>>>> CONFIG_DEBUG_INFO_DWARF4=y
> 
> I doubt this will make a difference, but CONFIG_DEBUG_INFO_DWARF4 isn't
> usually set on the kernels we use. See
> <https://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel>.
> 
>>> I'd guess you missed another line in the Proxmox kernel Makefile where
>>> it strips debuginfo from the kernel itself.
>>>
>>> What does "file" say on the kernel you built?
>>
>> It says:
>>
>> # file /boot/vmlinuz-4.4.35-2-pve
>> /boot/vmlinuz-4.4.35-2-pve: Linux kernel x86 boot executable bzImage,
>> version 4.4.35-2-pve (root@pve-test) #1 SMP Sun Jan 22 00:59:19 CET 201,
>> RO-rootFS, swap_dev 0x6, Normal VGA
>>
>> While looking for the vmlinux file, I found a script that (I think)
>> uncompresses a compressed kernel, so I did some testing:
>>
>> # /usr/src/linux-headers-4.4.35-2-pve/scripts/extract-vmlinux
>> /boot/vmlinuz-4.4.35-2-pve > vmlinux
>> # file vmlinux
>> vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically
>> linked, BuildID[sha1]=c23f97de3c8765073fbecf0b529383f1305c8e81, stripped
>>
>> So yes, this appears that the kernel I compiled is still stripped, so I
>> missed something, I'll give it another look but the documentation is
>> sparse and I don't see another strip command in it. If you wonder how
>> this Makefile looks like, it's here:
>> https://git.proxmox.com/?p=pve-kernel.git;a=blob_plain;f=Makefile;hb=HEAD
>>
>> Best regards and many thanks for your help.
> 
> I looked through that Makefile and I didn't see where the kernel gets
> stripped either. You are going to have to dig deeper to find where
> vmlinux gets deleted.
> 

I have found the vmlinux file - it was in directory
ubuntu-xenial/vmlinux (relative to root directory of Proxmox kernel
sources), and the reason I haven't found it earlier is because I ran
"make clean" some time earlier. Apparently, when I linked this file to
/lib/modules/4.4.35-2-pve/build/vmlinux, stap no longer complains when I
run scripts requiring vfs kernel probles.

David, I can't express how much you helped me, I thank you one more time
and wish you very best!