[Bug tapsets/30407] New: Tapset function print_ubacktrace_fileline() no longer works on kernel 6.2

[Bug tapsets/30407] New: Tapset function print_ubacktrace_fileline() no longer works on kernel 6.2

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

            Bug ID: 30407
           Summary: Tapset function print_ubacktrace_fileline() no longer
                    works on kernel 6.2
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: tapsets
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com
  Target Milestone: ---

I've noted that the standard tapset function `print_ubacktrace_fileline()` no
longer works on the latest 6.2.12 kernel of fedora 36 x86_64.

To reproduce the problem, let's use the following minimal C program as the
target:

```
int main(void) {
    return 0;
}
```

We name it `a.c`.

Then compile it with debug symbols:

```
gcc -g a.c
```

It generates the binary program ./a.out.

Now run the following stap oneliner:

```
$ /opt/stap/bin/stap -c ./a.out --ldd -e 'probe process.function("main") {
print_ubacktrace_fileline() }'
 0x40110a : main+0x4/0xe [/tmp/a.out]
 0x7f896e867510 : __libc_start_call_main+0x80/0xb0 [/usr/lib64/libc.so.6]
 0x7f896e8675c9 : __libc_start_main@GLIBC_2.2.5+0x89/0x150
[/usr/lib64/libc.so.6]
 0x401045 : _start+0x25/0x30 [/tmp/a.out]
```

We can see there are no source file names or line numbers.

For comparison, let's run the same example on an older kernel (5.0.16):

```
$ /opt/stap/bin/stap -c ./a.out --ldd -e 'probe process.function("main") {
print_ubacktrace_fileline() }'
 0x40049a : main+0x4/0x1a at /tmp/a.c:2 [/tmp/a.out]
 0x7fbc6289011b : __libc_start_main+0xeb/0x1c0 [/usr/lib64/libc-2.27.so]
 0x4003da : _start+0x2a/0x30 [/tmp/a.out]
```

It works on this older kernel as expected.

I'm using the latest master branch of the upstream systemtap repo:

```
commit 418f0a45ca4473491385b5c7eef777607bbdb3b7 (HEAD -> master, origin/master,
origin/HEAD)
Author: Frank Ch. Eigler <fche@redhat.com>
Date:   Fri Apr 28 12:07:15 2023 -0400

    NEWS++
```

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works on kernel 6.2

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

--- Comment #1 from agentzh <agentzh at gmail dot com> ---
The 6.1.18 kernel of Fedora 36 x86_64 also has this problem.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works on kernel 6.2/6.1

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

agentzh <agentzh at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Tapset function             |Tapset function
                   |print_ubacktrace_fileline() |print_ubacktrace_fileline()
                   |no longer works on kernel   |no longer works on kernel
                   |6.2                         |6.2/6.1

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works on kernel 6.2/6.1

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

Martin Cermak <mcermak at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mcermak at redhat dot com

--- Comment #2 from Martin Cermak <mcermak at redhat dot com> ---
This can be replicated with rhel8 (works) versusus rhel9 (doesn't work).  But I
don't think it's related to kernel:  After copying my test binary from rhel8
over to rhel9 (and making sure it's md5-identical, things started to work even
on rhel9!

The following comes from my rhel8 binary where things work fine:

> 8 x86_64 # eu-readelf --debug-dump=info a.out 
> 
> DWARF section [26] '.debug_info' at offset 0x2e70:
>  [Offset]
>  Compilation unit at offset 0:
>  Version: 4, Abbreviation section offset: 0, Address size: 8, Offset size: 4
>  [     b]  compile_unit         abbrev: 1
>            producer             (strp) "GNU C17 8.5.0 20210514 (Red Hat 8.5.0-19) -mtune=generic -march=x86-64 -g"
>            language             (data1) C99 (12)
>            name                 (strp) "test.c"
>            comp_dir             (strp) "/root/build"
>            low_pc               (addr) 0x0000000000400536 <main>
>            high_pc              (data8) 11 (0x0000000000400541)
>            stmt_list            (sec_offset) 0
>  [    2d]    subprogram           abbrev: 2
>              external             (flag_present) yes
>              name                 (strp) "main"
>              decl_file            (data1) test.c (1)
>              decl_line            (data1) 1
>              decl_column          (data1) 5
>              prototyped           (flag_present) yes
>              type                 (ref4) [    4b]
>              low_pc               (addr) 0x0000000000400536 <main>
>              high_pc              (data8) 11 (0x0000000000400541)
>              frame_base           (exprloc) 
>               [ 0] call_frame_cfa
>              GNU_all_call_sites   (flag_present) yes
>  [    4b]    base_type            abbrev: 3
>              byte_size            (data1) 4
>              encoding             (data1) signed (5)
>              name                 (string) "int"
> 8 x86_64 #

Now here's what my rhel9 binary shows:

> 9 x86_64 # eu-readelf --debug-dump=info a.out 
> 
> DWARF section [27] '.debug_info' at offset 0x47a0:
>  [Offset]
>  Compilation unit at offset 0:
>  Version: 5, Abbreviation section offset: 0, Address size: 8, Offset size: 4
>  Unit type: compile (1)
>  [     c]  compile_unit         abbrev: 1
>            producer             (strp) "GNU C17 11.3.1 20221121 (Red Hat 11.3.1-4) -mtune=generic -march=x86-64-v2 -g"
>            language             (data1) C11 (29)
>            name                 (line_strp) "test.c"
>            comp_dir             (line_strp) "/root/build"
>            low_pc               (addr) 0x0000000000401106 <main>
>            high_pc              (data8) 11 (0x0000000000401111)
>            stmt_list            (sec_offset) 0
>  [    2e]    subprogram           abbrev: 2
>              external             (flag_present) yes
>              name                 (strp) "main"
>              decl_file            (data1) test.c (1)
>              decl_line            (data1) 1
>              decl_column          (data1) 5
>              prototyped           (flag_present) yes
>              type                 (ref4) [    4c]
>              low_pc               (addr) 0x0000000000401106 <main>
>              high_pc              (data8) 11 (0x0000000000401111)
>              frame_base           (exprloc) 
>               [ 0] call_frame_cfa
>              call_all_calls       (flag_present) yes
>  [    4c]    base_type            abbrev: 3
>              byte_size            (data1) 4
>              encoding             (data1) signed (5)
>              name                 (string) "int"
> 9 x86_64 # 

I've verified that -march, doesn't cause the difference.  But enforcing DWARF4
on rhel9 with `gcc -gdwarf-4` makes things work again on rhel9!  So I think
it's the DWARF version that makes the difference :)


The print_ubacktrace_fileline() call propagates down to
_stp_linenumber_lookup(address, task, &filename, ...) which already returns the
NULL filename.  Stepping in, this calls _stp_linenumber_lookup(), which returns
struct _stp_module with a NULL debug_line member.  Looking farther...

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works with DWARF5

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

Martin Cermak <mcermak at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Tapset function             |Tapset function
                   |print_ubacktrace_fileline() |print_ubacktrace_fileline()
                   |no longer works on kernel   |no longer works with DWARF5
                   |6.2/6.1                     |

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works with DWARF5

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wcohen at redhat dot com

--- Comment #3 from William Cohen <wcohen at redhat dot com> ---
Had some additional discussion with Martin on email on how to fix this. For
this PR (and 29984) there are two parts that need to be addressed:

1) recognize the dwarf 5 format line info in translate.cxx.  It looks like
https://sourceware.org/bugzilla/show_bug.cgi?id=29984 mentions an area that
needs to be changed to so systemtap copies the line information.

2) have the runtime _stp_linenumber_lookup in runtime/sym.c be able to handle
the dwarf 5 line number headers.

Need to refer to the dwarf debuginfo manuals for dwarf4 and dwarf5 from
https://dwarfstd.org/, section 6.2.4 "The Line Number Program Header" for
details on the differences.  There are differences in the dwarf5 line number
header: address_size, segment_selector_size, directories_count, directories,
etc. in dwarf5 .

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works with DWARF5

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

--- Comment #4 from Martin Cermak <mcermak at redhat dot com> ---
Created attachment 14964
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14964&action=edit
A WIP patch showing line numbers, but no file names yet

First time I see DWARF this closely.  The attached WIP patch allows for showing
the line numbers already.  The DWARF structure change roughly is as follows:

DWARF-4                                      DWARF-5
==================================================================
unit_length                                  unit_length
version (uhalf)                              version (uhalf)
---                                          address_size (ubyte)
---                                          segment_selector_size (ubyte)
header_length                                header_length
minimum_instruction_length (ubyte)           minimum_instruction_length (ubyte)
maximum_operations_per_instruction (ubyte)   maximum_operations_per_instruction
(ubyte)
default_is_stmt (ubyte)                      default_is_stmt (ubyte)
line_base (sbyte)                            line_base (sbyte)
line_range (ubyte)                           line_range (ubyte)
opcode_base (ubyte)                          opcode_base (ubyte)
standard_opcode_lengths (array of ubyte)     standard_opcode_lengths (array of
ubyte)
include_directories (sequence of path names) ---
file_names (sequence of file entries)        ---
---                                          directory_entry_format_count
(ubyte)
---                                          directory_entry_format (sequence
of ULEB128 pairs)
---                                          directories_count (ULEB128)
---                                          directories (sequence of directory
names)
---                                          file_name_entry_format_count
(ubyte)
---                                          file_name_entry_format (sequence
of ULEB128 pairs)
---                                          file_names_count (ULEB128)
---                                          file_names (sequence of file name
entries)



So, the attached WIP patch merely jumps over address_size and
segment_selector_size for now. Looking into how to display the file name now...

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works with DWARF5

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works with DWARF5

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

--- Comment #5 from Martin Cermak <mcermak at redhat dot com> ---
Created attachment 15161
  --> https://sourceware.org/bugzilla/attachment.cgi?id=15161&action=edit
WIP patch

The attached patch doesn't properly parse the .debug_line header (yet) to find
correct pointers to the .debug_line_str section.  Next version will try to do
that.

Demo:
-----------------------------------
# cat a.c
int main(void) {
    return 0;
}
# 
# gcc -g -gdwarf-5 a.c -o prog5
# gcc -g -gdwarf-4 a.c -o prog4
#
# eu-readelf --debug-dump=info prog4 | fgrep -i produc
           producer             (strp) "GNU C17 13.1.1 20230614 (Red Hat
13.1.1-4) -mtune=generic -march=x86-64 -g -gdwarf-4"
# eu-readelf --debug-dump=info prog5 | fgrep -i produc
           producer             (strp) "GNU C17 13.1.1 20230614 (Red Hat
13.1.1-4) -mtune=generic -march=x86-64 -g -gdwarf-5"
# 
# stap --ldd -e 'probe process.function("main") { print_ubacktrace_fileline()
}' -c ./prog4 
 0x40110a : main+0x4/0xe at /root/test/a.c:2 [/root/test/prog4]
 0x7f9d3c42814a : __libc_start_call_main+0x7a/0xb0 [/usr/lib64/libc.so.6]
 0x7f9d3c42820b : __libc_start_main@GLIBC_2.2.5+0x8b/0x160
[/usr/lib64/libc.so.6]
 0x401045 : _start+0x25/0x30 [/root/test/prog4]
# 
# stap --ldd -e 'probe process.function("main") { print_ubacktrace_fileline()
}' -c ./prog5
 0x40110a : main+0x4/0xe at /root/test/a.c:2 [/root/test/prog5]
 0x7fba8ae2814a : __libc_start_call_main+0x7a/0xb0 [/usr/lib64/libc.so.6]
 0x7fba8ae2820b : __libc_start_main@GLIBC_2.2.5+0x8b/0x160
[/usr/lib64/libc.so.6]
 0x401045 : _start+0x25/0x30 [/root/test/prog5]
#

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/30407] Tapset function print_ubacktrace_fileline() no longer works with DWARF5

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30407

Martin Cermak <mcermak at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2023-10-10
           Assignee|systemtap at sourceware dot org    |mcermak at redhat dot com
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |ASSIGNED

-- 
You are receiving this mail because:
You are the assignee for the bug.