[Bug translator/29986] New: kernel.function can not work on fedora34 with aarch64 architecture

[Bug translator/29986] New: kernel.function can not work on fedora34 with aarch64 architecture

[lijunlong at openresty dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

            Bug ID: 29986
           Summary: kernel.function can not work on fedora34 with aarch64
                    architecture
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: translator
          Assignee: systemtap at sourceware dot org
          Reporter: lijunlong at openresty dot com
  Target Milestone: ---

OS: 5.11.12-300.fc34.aarch64
stap verion: Systemtap translator/driver (version 4.9/0.186,
release-4.8-15-gdf41f00d).   (built from the source code)


Use stap -l to list the kernel functions, it will throw errors like below.


stap -vv -l 'kernel.function("vfs_*")'  
Systemtap translator/driver (version 4.9/0.186, release-4.8-15-gdf41f00d)
Copyright (C) 2005-2022 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.32 ... 6.1.0-rc3
enabled features: LIBXML2 NLS
Created temporary directory "/tmp/staptw6f30"
Session arch: arm64 release: 5.11.12-300.fc34.aarch64
Build tree: "/lib/modules/5.11.12-300.fc34.aarch64/build"
Searched for library macro files:
"/usr/local/openresty-stap/share/systemtap/tapset/linux", found: 7, processed:
7
Searched for library macro files:
"/usr/local/openresty-stap/share/systemtap/tapset", found: 11, processed: 11
Searched: "/usr/local/openresty-stap/share/systemtap/tapset/linux/arm64",
found: 2, processed: 2
Searched: "/usr/local/openresty-stap/share/systemtap/tapset/linux", found: 413,
processed: 413
Searched: "/usr/local/openresty-stap/share/systemtap/tapset/arm64", found: 1,
processed: 1
Searched: "/usr/local/openresty-stap/share/systemtap/tapset", found: 40,
processed: 40
Pass 1: parsed user script and 474 library scripts using
119052virt/104412res/9336shr/94904data kb, in 100usr/30sys/149real ms.
derive-probes (location #0): kernel.function("vfs_*") of keyword at <input>:1:1
semantic error: resolution failed in DWARF builder
   thrown from: elaborate.cxx:1094
semantic error: resolution failed in DWARF builder
   thrown from: elaborate.cxx:1094
semantic error: while resolving probe point: identifier 'kernel' at <input>:1:7
   thrown from: elaborate.cxx:1082
        source: probe kernel.function("vfs_*") {}
                      ^

semantic error: no match
   thrown from: elaborate.cxx:1045
Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using
121052virt/107052res/9948shr/96904data kb, in 150usr/10sys/149real ms.
Running rm -rf /tmp/staptw6f30
Spawn waitpid result (0x0): 0
Removed temporary directory "/tmp/staptw6f30"

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wcohen at redhat dot com

--- Comment #1 from William Cohen <wcohen at redhat dot com> ---
The kernel.function("*") probes are going to need the kernel-debuginfo* rpms. 
What is the output of:

rpm -qa |grep kernel-

Are there kernel-debuginfo and kernel-debuginfo-common that match the
kernel-5.11.12-300.fc34.aarch64 that is currently running?

I set up a fc34 machine locally and found that the kernels that were installed
were kernel-5.17.12-100.fc34.aarch64, a later version that used in this bug. 
Most of the rpm repos remove the older versions of the debuginfo for pacage and
you may need to download the matching kernel-debuginfo-* rpms directly from
koji rather than using the stap-prep command as stap-prep won't find the
appropriate kernel-debuginfo* rpms.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
(why wouldn't stap-prep work on arm64?)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

--- Comment #3 from William Cohen <wcohen at redhat dot com> ---
The problem isn't stap-prep or that things are running on aarch64, but rather
than various RPM repos purge the older versions debuginfo.  Able to install
kernel-5.11.12-300.fc34.aarch64.rpm, but no matching debuginfo found:

[root@ampere-mtsnow-altramax-03-vm-04 ~]# dnf install
kernel-5.11.12-300.fc34.aarch64
Last metadata expiration check: 1:48:23 ago on Wed 11 Jan 2023 10:13:12 AM EST.
Dependencies resolved.
================================================================================
 Package            Arch        Version                 Repository         Size
================================================================================
Installing:
 kernel             aarch64     5.11.12-300.fc34        beaker-Fedora     151 k
Installing dependencies:
 kernel-core        aarch64     5.11.12-300.fc34        beaker-Fedora      38 M
 kernel-modules     aarch64     5.11.12-300.fc34        beaker-Fedora      26 M

Transaction Summary
================================================================================
Install  3 Packages

Total download size: 64 M
Installed size: 126 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): kernel-5.11.12-300.fc34.aarch64.rpm      3.7 MB/s | 151 kB     00:00    
(2/3): kernel-modules-5.11.12-300.fc34.aarch64.  43 MB/s |  26 MB     00:00    
(3/3): kernel-core-5.11.12-300.fc34.aarch64.rpm  47 MB/s |  38 MB     00:00    
--------------------------------------------------------------------------------
Total                                            80 MB/s |  64 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : kernel-core-5.11.12-300.fc34.aarch64                   1/3 
  Running scriptlet: kernel-core-5.11.12-300.fc34.aarch64                   1/3 
  Installing       : kernel-modules-5.11.12-300.fc34.aarch64                2/3 
  Running scriptlet: kernel-modules-5.11.12-300.fc34.aarch64                2/3 
  Installing       : kernel-5.11.12-300.fc34.aarch64                        3/3 
  Running scriptlet: kernel-core-5.11.12-300.fc34.aarch64                   3/3 
  Running scriptlet: kernel-5.11.12-300.fc34.aarch64                        3/3 
  Verifying        : kernel-5.11.12-300.fc34.aarch64                        1/3 
  Verifying        : kernel-core-5.11.12-300.fc34.aarch64                   2/3 
  Verifying        : kernel-modules-5.11.12-300.fc34.aarch64                3/3 

Installed:
  kernel-5.11.12-300.fc34.aarch64          kernel-core-5.11.12-300.fc34.aarch64 
  kernel-modules-5.11.12-300.fc34.aarch64 

Complete!
[root@ampere-mtsnow-altramax-03-vm-04 ~]# dnf debuginfo-install
kernel-5.11.12-300.fc34.aarch64
enabling fedora-cisco-openh264-debuginfo repository
enabling fedora-modular-debuginfo repository
enabling updates-debuginfo repository
enabling updates-modular-debuginfo repository
Last metadata expiration check: 1:49:10 ago on Wed 11 Jan 2023 10:13:12 AM EST.
Could not find debuginfo package for the following installed packages:
kernel-5.11.12-300.fc34.aarch64
Could not find debugsource package for the following installed packages:
kernel-5.11.12-300.fc34.aarch64
Dependencies resolved.
Nothing to do.
Complete!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[lijunlong at openresty dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

--- Comment #4 from lijunlong <lijunlong at openresty dot com> ---
I am sorry I did not install the kernel-debuginfo package.
I will update to the latest kernel and install the kernel-debuginfo.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[lijunlong at openresty dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

--- Comment #5 from lijunlong <lijunlong at openresty dot com> ---
Can stap give an accurate error message on this?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

--- Comment #6 from William Cohen <wcohen at redhat dot com> ---
Were you successful in getting a matching debuginfo for the kernel installed
and getting systemtap working?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/29986] kernel.function can not work on fedora34 with aarch64 architecture

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29986

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #7 from William Cohen <wcohen at redhat dot com> ---
This appears to be a missing or missmatched kernel-debugfino rpm.  On aarch64
Fedora 38 this worked for me with both the offical systemtap-4.8-3.fc38.aarch64
and a locally built systemtap from git commit
fc6519089d3f9366470ce442b648d69ed9b56f53.

-- 
You are receiving this mail because:
You are the assignee for the bug.