[Bug kprobes/22772] New: missing kernel/module debuginfo for custom lt kernel

[Bug kprobes/22772] New: missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

            Bug ID: 22772
           Summary: missing kernel/module debuginfo for custom lt kernel
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: kprobes
          Assignee: systemtap at sourceware dot org
          Reporter: ocket8888 at gmail dot com
  Target Milestone: ---

Created attachment 10770
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10770&action=edit
kernel configuration

I've been trying (and failing) to get systemtap to work on my CentOS machine.
```
    $ uname -a
    Linux ipcdn-cache-51.cdnlab.comcast.net 4.9.70-1.el7.centos.x86_64 #1 SMP
Thu Feb 1 15:34:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
    $ cat /etc/centos-release```
    CentOS Linux release 7.4.1708 (Core)
    $ stap --version
    Systemtap translator/driver (version 3.1/0.168, rpm 3.1-5.el7_4)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.10-rc8
enabled features: AVAHI BOOST_STRING_REF DYNINST JAVA PYTHON2 LIBRPM LIBSQLITE3
LIBVIRT LIBXML2 NLS NSS READLINE
```

I'm attaching the configuration used to build the kernel, note that it follows
the configuration specified in the FAQ (Q2). When I run the test command `stap
-ve 'probe vfs.read{exit()}` it outputs

```
    Pass 1: parsed user script and 529 library scripts using
378212virt/192180res/6484shr/188296data kb, in 2030usr/40sys/2078real ms.
    semantic error: while resolving probe point: identifier 'kernel' at
/usr/share/systemtap/tapset/linux/vfs.stp:915:18
        source: probe vfs.read = kernel.function("vfs_read")
                                 ^

    semantic error: missing x86_64 kernel/module debuginfo [man
warning::debuginfo] under '/lib/modules/4.9.70-1.el7.centos.x86_64/build'

    semantic error: resolution failed in alias expansion builder

    semantic error: while resolving probe point: identifier 'vfs' at
<input>:1:7
            source: probe vfs.read{exit()}
                          ^

    semantic error: no match

    Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using
393364virt/210976res/10668shr/202812data kb, in 150usr/120sys/271real ms.
    Pass 2: analysis failed.  [man error::pass2]
```

However, I can show that I have installed all of the necessary packages:

```
    $ yum list installed | grep kernel
    kernel-lt.x86_64           4.9.70-1.el7.centos installed
    kernel-lt-debuginfo.x86_64 4.9.70-1.el7.centos installed
    kernel-lt-devel.x86_64     4.9.70-1.el7.centos installed
    kernel-lt-headers.x86_64   4.9.70-1.el7.centos installed
    kernel-lt-tools.x86_64     4.9.70-1.el7.centos installed
    kernel-lt-tools-libs.x86_64
    kernel-lt-tools-libs-devel.x86_64
```

note that the version numbers and release numbers match `uname -r`. As per the
FAQ (Q3), I've also verified that the kernel exists under /boot/:

```
    $ ls /boot/ | grep `uname -r`
    config-4.9.70-1.el7.centos.x86_64
    initramfs-4.9.70-1.el7.centos.x86_64.img
    symvers-4.9.70-1.el7.centos.x86_64.gz
    System.map-4.9.70-1.el7.centos.x86_64
    vmlinuz-4.9.70-1.el7.centos.x86_64
```

I'm not sure what else would be helpful, but I'm certainly willing to provide
any more information on request.

I suspect the problem has something to do with the fact that I'm using an lt
kernel, as `stap-prep` seems to be looking for the incorrect packages.

```
    ...
    No package kernel-4.9.70-1.el7.centos.x86_64 available.
    No package kernel-devel-4.9.70-1.el7.centos.x86_64 available.
    No package kernel-debuginfo-4.9.70-1.el7.centos.x86_64 available.
    ...
```

not to mention that the proper - kernel-lt-* - packages are installed as noted
above.

To put it simply: what do?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #1 from ocket8888 <ocket8888 at gmail dot com> ---
I totally forgot all the boilerplate stuff:

stap version:          3.1/0.168, rpm 3.1-5.el7_4
installed stap from:   yum repos - packages systemtap-client,
systemtap-runtime,      systemtap-sdt-devel
other staps installed: None
stap binary location:  /usr/bin/stap
tapset location:       /usr/share/systemtap/tapset
execution environment: using sudo - see attachment 'env'
compiler version:      4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
problematic script:    'probe vfs.read{exit()}'
custom tapsets:        None
uname:                 Linux <hostname redacted> 4.9.70-1.el7.centos.x86_64 #1
SMP Thu Feb 1 15:34:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
kernel configuration:  see attachment 'kernel configuration'
kernel compiled with:  (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
system architecture:   x86_64
related packages info: see attachment 'packages'
gdb backtrace:         'No stack.'

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #2 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10771
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10771&action=edit
list of (possibly) related packages installed on the system

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #3 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10772
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10772&action=edit
execution environment

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #4 from David Smith <dsmith at redhat dot com> ---
(In reply to ocket8888 from comment #0)
> Created attachment 10770 [details]
> kernel configuration
> 
> I've been trying (and failing) to get systemtap to work on my CentOS machine.
> ```
>     $ uname -a
>     Linux ipcdn-cache-51.cdnlab.comcast.net 4.9.70-1.el7.centos.x86_64 #1
> SMP Thu Feb 1 15:34:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>     $ cat /etc/centos-release```
>     CentOS Linux release 7.4.1708 (Core)
>     $ stap --version
>     Systemtap translator/driver (version 3.1/0.168, rpm 3.1-5.el7_4)
> Copyright (C) 2005-2017 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> tested kernel versions: 2.6.18 ... 4.10-rc8
> enabled features: AVAHI BOOST_STRING_REF DYNINST JAVA PYTHON2 LIBRPM
> LIBSQLITE3 LIBVIRT LIBXML2 NLS NSS READLINE
> ```
> 
> I'm attaching the configuration used to build the kernel, note that it
> follows the configuration specified in the FAQ (Q2). When I run the test
> command `stap -ve 'probe vfs.read{exit()}` it outputs
> 
> ```
>     Pass 1: parsed user script and 529 library scripts using
> 378212virt/192180res/6484shr/188296data kb, in 2030usr/40sys/2078real ms.
>     semantic error: while resolving probe point: identifier 'kernel' at
> /usr/share/systemtap/tapset/linux/vfs.stp:915:18
>         source: probe vfs.read = kernel.function("vfs_read")
>                                  ^
> 
>     semantic error: missing x86_64 kernel/module debuginfo [man
> warning::debuginfo] under '/lib/modules/4.9.70-1.el7.centos.x86_64/build'

Interesting. I'm betting the '-lt' tag is going to show up in the filenames
installed by one (or more) of the kernel-lt packages. Pay close attention to
the following packages: kernel-lt, kernel-lt-debuginfo, and kernel-lt-devel.
Does '-lt' show up in any of the filenames from those packages?

If so, an semi-easy workaround here would be to create a symbolic link from the
path systemtap expects to the path containing '-lt'.

If I'm right, I'm not sure what the best solution to this problem would be in
systemtap to automatically recognize the '-lt' kernels.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> ---
Try 
% stap -p4 -vvv -e 'probe kernel.function("sys_open") { }'
And also % strace stap ...
to see where it's looking for kernel build / debug data.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #6 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10773
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10773&action=edit
strace output after attempting to probe kernel

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #7 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10774
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10774&action=edit
output of `udo stap -p4 -vvv -e 'probe kernel.function("sys_open"){}'`

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #8 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to Frank Ch. Eigler from comment #5)
> Try 
> % stap -p4 -vvv -e 'probe kernel.function("sys_open") { }'
> And also % strace stap ...
> to see where it's looking for kernel build / debug data.

See my two new attachments

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #9 from Frank Ch. Eigler <fche at redhat dot com> ---
(In reply to ocket8888 from comment #6)
> Created attachment 10773 [details]
> strace output after attempting to probe kernel

I meant

% strace stap -p4 -e 'probe kernel.function("sys_open") {}'

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #10 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to David Smith from comment #4)
> (In reply to ocket8888 from comment #0)
> > Created attachment 10770 [details]
> > kernel configuration
> > 
> > I've been trying (and failing) to get systemtap to work on my CentOS machine.
> > ```
> >     $ uname -a
> >     Linux ipcdn-cache-51.cdnlab.comcast.net 4.9.70-1.el7.centos.x86_64 #1
> > SMP Thu Feb 1 15:34:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >     $ cat /etc/centos-release```
> >     CentOS Linux release 7.4.1708 (Core)
> >     $ stap --version
> >     Systemtap translator/driver (version 3.1/0.168, rpm 3.1-5.el7_4)
> > Copyright (C) 2005-2017 Red Hat, Inc. and others
> > This is free software; see the source for copying conditions.
> > tested kernel versions: 2.6.18 ... 4.10-rc8
> > enabled features: AVAHI BOOST_STRING_REF DYNINST JAVA PYTHON2 LIBRPM
> > LIBSQLITE3 LIBVIRT LIBXML2 NLS NSS READLINE
> > ```
> > 
> > I'm attaching the configuration used to build the kernel, note that it
> > follows the configuration specified in the FAQ (Q2). When I run the test
> > command `stap -ve 'probe vfs.read{exit()}` it outputs
> > 
> > ```
> >     Pass 1: parsed user script and 529 library scripts using
> > 378212virt/192180res/6484shr/188296data kb, in 2030usr/40sys/2078real ms.
> >     semantic error: while resolving probe point: identifier 'kernel' at
> > /usr/share/systemtap/tapset/linux/vfs.stp:915:18
> >         source: probe vfs.read = kernel.function("vfs_read")
> >                                  ^
> > 
> >     semantic error: missing x86_64 kernel/module debuginfo [man
> > warning::debuginfo] under '/lib/modules/4.9.70-1.el7.centos.x86_64/build'
> 
> Interesting. I'm betting the '-lt' tag is going to show up in the filenames
> installed by one (or more) of the kernel-lt packages. Pay close attention to
> the following packages: kernel-lt, kernel-lt-debuginfo, and kernel-lt-devel.
> Does '-lt' show up in any of the filenames from those packages?
> 
> If so, an semi-easy workaround here would be to create a symbolic link from
> the path systemtap expects to the path containing '-lt'.
> 
> If I'm right, I'm not sure what the best solution to this problem would be
> in systemtap to automatically recognize the '-lt' kernels.

Here's what I got:

```
    $ repoquery -l kernel-lt-`uname -r` | grep -e "-lt"
    /etc/ld.so.conf.d/kernel-lt-4.9.70-1.el7.centos.x86_64.conf
```

the devel package installs nothing with '-lt' in the name, while the debuginfo
package dumped too much stuff to paste, but what might be the issue is
everything it puts under /usr/src is under a 'kernel-lt-4.9.70' directory.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

ocket8888 <ocket8888 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #10773|0                           |1
        is obsolete|                            |
  Attachment #10774|0                           |1
        is obsolete|                            |

--- Comment #11 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10775
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10775&action=edit
output of `sudo strace stap -p4 -e 'probe kernel.function("sys_open"){}'`

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #12 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to Frank Ch. Eigler from comment #9)
> (In reply to ocket8888 from comment #6)
> > Created attachment 10773 [details]
> > strace output after attempting to probe kernel
> 
> I meant
> 
> % strace stap -p4 -e 'probe kernel.function("sys_open") {}'

I uploaded the new output in 'strace.err' (output was only 62431 lines to
stderr, 0 on stdout)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #13 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to ocket8888 from comment #10)
> (In reply to David Smith from comment #4)
> > (In reply to ocket8888 from comment #0)
> > > Created attachment 10770 [details]
> > > kernel configuration
> > > 
> > > I've been trying (and failing) to get systemtap to work on my CentOS machine.
> > > ```
> > >     $ uname -a
> > >     Linux ipcdn-cache-51.cdnlab.comcast.net 4.9.70-1.el7.centos.x86_64 #1
> > > SMP Thu Feb 1 15:34:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> > >     $ cat /etc/centos-release```
> > >     CentOS Linux release 7.4.1708 (Core)
> > >     $ stap --version
> > >     Systemtap translator/driver (version 3.1/0.168, rpm 3.1-5.el7_4)
> > > Copyright (C) 2005-2017 Red Hat, Inc. and others
> > > This is free software; see the source for copying conditions.
> > > tested kernel versions: 2.6.18 ... 4.10-rc8
> > > enabled features: AVAHI BOOST_STRING_REF DYNINST JAVA PYTHON2 LIBRPM
> > > LIBSQLITE3 LIBVIRT LIBXML2 NLS NSS READLINE
> > > ```
> > > 
> > > I'm attaching the configuration used to build the kernel, note that it
> > > follows the configuration specified in the FAQ (Q2). When I run the test
> > > command `stap -ve 'probe vfs.read{exit()}` it outputs
> > > 
> > > ```
> > >     Pass 1: parsed user script and 529 library scripts using
> > > 378212virt/192180res/6484shr/188296data kb, in 2030usr/40sys/2078real ms.
> > >     semantic error: while resolving probe point: identifier 'kernel' at
> > > /usr/share/systemtap/tapset/linux/vfs.stp:915:18
> > >         source: probe vfs.read = kernel.function("vfs_read")
> > >                                  ^
> > > 
> > >     semantic error: missing x86_64 kernel/module debuginfo [man
> > > warning::debuginfo] under '/lib/modules/4.9.70-1.el7.centos.x86_64/build'
> > 
> > Interesting. I'm betting the '-lt' tag is going to show up in the filenames
> > installed by one (or more) of the kernel-lt packages. Pay close attention to
> > the following packages: kernel-lt, kernel-lt-debuginfo, and kernel-lt-devel.
> > Does '-lt' show up in any of the filenames from those packages?
> > 
> > If so, an semi-easy workaround here would be to create a symbolic link from
> > the path systemtap expects to the path containing '-lt'.
> > 
> > If I'm right, I'm not sure what the best solution to this problem would be
> > in systemtap to automatically recognize the '-lt' kernels.
> 
> Here's what I got:
> 
> ```
>     $ repoquery -l kernel-lt-`uname -r` | grep -e "-lt"
>     /etc/ld.so.conf.d/kernel-lt-4.9.70-1.el7.centos.x86_64.conf
> ```
> 
> the devel package installs nothing with '-lt' in the name, while the
> debuginfo package dumped too much stuff to paste, but what might be the
> issue is everything it puts under /usr/src is under a 'kernel-lt-4.9.70'
> directory.

Also, I made the links
```
    /usr/lib/debug.dwz/kernel-4.9.70-1.el7.centos.x86_64 ->
/usr/lib/debug.dwz/kernel-lt-4.9.70-1.el7.centos.x86_64
    /usr/src/debug/kernel-4.9.70 -> /usr/src/debug/kernel-lt-4.9.70
    /etc/ld.so.conf.d/kernel-4.9.70-1.el7.centos.x86_64.conf ->
/etc/ld.so.conf.d/kernel-/t-4.9.70-1.el7.centos.x86_64.conf
```
and that hasn't fixed it, but I haven't tried rebooting or reinstalling stap
yet, if you think that'd help. I can't see how it would tbh, which is why I
haven't done it yet, but clearly I have no idea what I'm doing so.... ¯\_(ツ)_/¯

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #14 from David Smith <dsmith at redhat dot com> ---
(In reply to ocket8888 from comment #12)
> (In reply to Frank Ch. Eigler from comment #9)
> > (In reply to ocket8888 from comment #6)
> > > Created attachment 10773 [details]
> > > strace output after attempting to probe kernel
> > 
> > I meant
> > 
> > % strace stap -p4 -e 'probe kernel.function("sys_open") {}'
> 
> I uploaded the new output in 'strace.err' (output was only 62431 lines to
> stderr, 0 on stdout)

So, let's see where systemtap looked for the kernel's debug file (which should
be at /usr/lib/debug/lib/modules/`uname -r`/vmlinux on a RHEL-type system):

====
grep ^open strace_err.txt  | fgrep ENOENT | fgrep
'"/usr/lib/debug/lib/modules/' | fgrep vmlinux
open("/usr/lib/debug/lib/modules/4.9.70-1.el7.centos.x86_64/vmlinux.debug",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/lib/modules/4.9.70-1.el7.centos.x86_64/vmlinux", O_RDONLY)
= -1 ENOENT (No such file or directory)
====

So, where is the vmlinux file provided by your kernel-lt-debuginfo package?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #15 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to David Smith from comment #14)
> (In reply to ocket8888 from comment #12)
> > (In reply to Frank Ch. Eigler from comment #9)
> > > (In reply to ocket8888 from comment #6)
> > > > Created attachment 10773 [details]
> > > > strace output after attempting to probe kernel
> > > 
> > > I meant
> > > 
> > > % strace stap -p4 -e 'probe kernel.function("sys_open") {}'
> > 
> > I uploaded the new output in 'strace.err' (output was only 62431 lines to
> > stderr, 0 on stdout)
> 
> So, let's see where systemtap looked for the kernel's debug file (which
> should be at /usr/lib/debug/lib/modules/`uname -r`/vmlinux on a RHEL-type
> system):
> 
> ====
> grep ^open strace_err.txt  | fgrep ENOENT | fgrep
> '"/usr/lib/debug/lib/modules/' | fgrep vmlinux
> open("/usr/lib/debug/lib/modules/4.9.70-1.el7.centos.x86_64/vmlinux.debug",
> O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/usr/lib/debug/lib/modules/4.9.70-1.el7.centos.x86_64/vmlinux",
> O_RDONLY) = -1 ENOENT (No such file or directory)
> ====
> 
> So, where is the vmlinux file provided by your kernel-lt-debuginfo package?

```
    $ ls -A /usr/lib/debug/lib/modules/`uname -r` 
    kernel
```

('kernel' is a directory) Also:

```
    $ repoquery -l kernel-lt-debuginfo.x86_64 | grep vmlin
   
/usr/src/debug/kernel-lt-4.9.70/linux-4.9.70-1.el7.centos.x86_64/tools/perf/tests/vmlinux-kallsyms.c
```

So it doesn't appear to be providing one. Just to be safe, I ran:

```
    $ find / -name '*vmlinux*'
    /usr/src/kernels/4.9.70-1.el7.centos.x86_64/scripts/gdb/vmlinux-gdb.pyc
    /usr/src/kernels/4.9.70-1.el7.centos.x86_64/scripts/gdb/vmlinux-gdb.py
    /usr/src/kernels/4.9.70-1.el7.centos.x86_64/scripts/gdb/vmlinux-gdb.pyo
    /usr/src/kernels/4.9.70-1.el7.centos.x86_64/scripts/link-vmlinux.sh
    /usr/src/kernels/4.9.70-1.el7.centos.x86_64/scripts/extract-vmlinux
   
/usr/src/kernels/4.9.70-1.el7.centos.x86_64/include/asm-generic/vmlinux.lds.h
```

and I'm not totally sure, but I don't think any of that helps... Could there be
something wrong with my spec file? Cuz it is generating the debuginfo package,
and I do have it installed, so the only thing I can think of is something weird
happening to its contents (disclaimer: I didn't write the spec file, a coworker
did).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #16 from Frank Ch. Eigler <fche at redhat dot com> ---
My guess is that the debuginfo subrpm you have is not the Real Thing.  Maybe
the kernel was configured without CONFIG_DEBUGINFO; maybe something else is
messed up w.r.t. the way the dwarf files are separated.  Hard to know without
the .spec file.

It does not seem like systemtap's doing anything wrong.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #17 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to Frank Ch. Eigler from comment #16)
> My guess is that the debuginfo subrpm you have is not the Real Thing.  Maybe
> the kernel was configured without CONFIG_DEBUGINFO; maybe something else is
> messed up w.r.t. the way the dwarf files are separated.  Hard to know
> without the .spec file.
> 
> It does not seem like systemtap's doing anything wrong.

It's definitely using CONFIG_DEBUGINFO (you can see for yourself in the config
I uploaded), which makes me think it's probably the spec file. I'm gonna upload
it in a second, once I get confirmation that there's no company secrets
embedded in the kernel specfile for some reason.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #18 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10776
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10776&action=edit
specfile used to build all kernel packages

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #19 from David Smith <dsmith at redhat dot com> ---
(In reply to ocket8888 from comment #18)
> Created attachment 10776 [details]
> specfile used to build all kernel packages

A Fedora/RHEL kernel specfile is typically the most complicated spec file in
the distro (or at least in the top 10). Since the kernel isn't a typical ELF
executable, lots of semi-tricky things are done.

What would be more useful than the full specfile itself would be a diff between
your specfile and the specfile you started with. If you just added a custom
kernel patch for example or use a custom config file, that shouldn't effect the
debuginfo generation.

Another thing to try. If you have access to your kernel's rpm build tree, you
could look for a vmlinux or vmlinux.debug file and see if one exists that
didn't get packaged.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #21 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10790
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10790&action=edit
diff with 4.10-ml kernel

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #20 from ocket8888 <ocket8888 at gmail dot com> ---
Created attachment 10789
  --> https://sourceware.org/bugzilla/attachment.cgi?id=10789&action=edit
diff with 4.4-lt kernel

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[ocket8888 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #22 from ocket8888 <ocket8888 at gmail dot com> ---
(In reply to David Smith from comment #19)
> (In reply to ocket8888 from comment #18)
> > Created attachment 10776 [details]
> > specfile used to build all kernel packages
> 
> A Fedora/RHEL kernel specfile is typically the most complicated spec file in
> the distro (or at least in the top 10). Since the kernel isn't a typical ELF
> executable, lots of semi-tricky things are done.
> 
> What would be more useful than the full specfile itself would be a diff
> between your specfile and the specfile you started with. If you just added a
> custom kernel patch for example or use a custom config file, that shouldn't
> effect the debuginfo generation.
> 
> Another thing to try. If you have access to your kernel's rpm build tree,
> you could look for a vmlinux or vmlinux.debug file and see if one exists
> that didn't get packaged.

Sorry I took so long to respond; I work for Comcast and the Super Bowl is a
very busy time here. I asked the guy who built the kernel what specs he
modified, and he directed me to the "kernel-lt-4.4" and "kernel-ml-4.10"
specfiles. He says the 4.9-lt (the one I'm having issues with) is a combination
of the two, so I uploaded a diff with each of them. I'm not totally sure how
helpful that is, so if you want a diff with a specific, standard kernel just
let me know.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

--- Comment #23 from David Smith <dsmith at redhat dot com> ---
(In reply to ocket8888 from comment #22)
> (In reply to David Smith from comment #19)
> > (In reply to ocket8888 from comment #18)
> > > Created attachment 10776 [details]
> > > specfile used to build all kernel packages
> > 
> > A Fedora/RHEL kernel specfile is typically the most complicated spec file in
> > the distro (or at least in the top 10). Since the kernel isn't a typical ELF
> > executable, lots of semi-tricky things are done.
> > 
> > What would be more useful than the full specfile itself would be a diff
> > between your specfile and the specfile you started with. If you just added a
> > custom kernel patch for example or use a custom config file, that shouldn't
> > effect the debuginfo generation.
> > 
> > Another thing to try. If you have access to your kernel's rpm build tree,
> > you could look for a vmlinux or vmlinux.debug file and see if one exists
> > that didn't get packaged.
> 
> Sorry I took so long to respond; I work for Comcast and the Super Bowl is a
> very busy time here. I asked the guy who built the kernel what specs he
> modified, and he directed me to the "kernel-lt-4.4" and "kernel-ml-4.10"
> specfiles. He says the 4.9-lt (the one I'm having issues with) is a
> combination of the two, so I uploaded a diff with each of them. I'm not
> totally sure how helpful that is, so if you want a diff with a specific,
> standard kernel just let me know.

I took a look at where you got your 'kernel-lt' from:
<http://elrepo.org/tiki/kernel-lt>

I didn't realize kernel-lt wasn't from CentOS. Looking at
<http://elrepo.org/linux/kernel/el7/x86_64/RPMS/>, I see that they don't
produce debuginfo RPMs. They have an open bug to do so, but I don't see any
real activity on that bug: <http://elrepo.org/bugs/view.php?id=684>

So, since the spec file you started with didn't produce good debuginfo RPMs, it
isn't surprising that your modified kernel-lt spec file doesn't produce good
debuginfo RPMs.

So, to fix this, you are going to have to dig into the CentOS kernel specfile:

<https://git.centos.org/blob/rpms!kernel.git/6824b576c2a6102b57d66a9b9db39e6a11b9921d/SPECS!kernel.spec>

Look for the '%if %{with_debuginfo}' sections and copy them into your spec
file. When you are done, you might even submit those changes back to
elrepo.org.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[lennordocdoc0921 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

Lennor <lennordocdoc0921 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lennordocdoc0921 at gmail dot com

--- Comment #24 from Lennor <lennordocdoc0921 at gmail dot com> ---
Great question and answer website! Thank you so much, keep it up the good deed!
 Please visit our sites also:
https://www.bradentonrescreening.com/screen-repair--re-screening--st-petersburg-fl.html/
:
https://www.bradentonrescreening.com/screen-repair--re-screening--sarasota-fl.html/
: https://www.vahomeloanmi.com/

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug kprobes/22772] missing kernel/module debuginfo for custom lt kernel

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=22772

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |NOTABUG
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #25 from David Smith <dsmith at redhat dot com> ---
I'm closing this one since the reporter hasn't given us an update in over two
years and this isn't a systemtap problem but a debuginfo problem with -lt
kernels.

-- 
You are receiving this mail because:
You are the assignee for the bug.