[Bug tapsets/23879] New: print_ubacktrace can not print function name

[Bug tapsets/23879] New: print_ubacktrace can not print function name

[idealities at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

            Bug ID: 23879
           Summary: print_ubacktrace can not print function name
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: tapsets
          Assignee: systemtap at sourceware dot org
          Reporter: idealities at gmail dot com
  Target Milestone: ---

On archlinux, with kernel 4.18.16-arch1-1-ARCH, systemtap (version 4.0/0.174,
non-git sources), elfutils 0.174.

If use stap on target process originated from system packages,
print_ubacktrace() works fine:

$ stap -v  -e 'probe process.function("*") { print_ubacktrace(); exit() }' -x
4924
Pass 1: parsed user script and 470 library scripts using
75324virt/51500res/6800shr/44864data kb, in 100usr/10sys/110real ms.
Pass 2: analyzed script: 234 probes, 2 functions, 0 embeds, 0 globals using
77040virt/54192res/7668shr/46580data kb, in 10usr/0sys/11real ms.
Pass 3: translated to C into
"/tmp/stapOV3tv1/stap_1bc1f6af99b98d562e15bc3b6e4ee1a2_52098_src.c" using
77184virt/54628res/7924shr/46724data kb, in 10usr/100sys/111real ms.
Pass 4: compiled C into "stap_1bc1f6af99b98d562e15bc3b6e4ee1a2_52098.ko" in
3070usr/460sys/3418real ms.
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry
with -DDEBUG_UNWIND)'
 0x55c3c7f8f9c0 : ProcessList_scan+0x0/0x1a0 [/usr/bin/htop]
 0x55c3c7f90497 : ScreenManager_run+0x107/0x11e0 [/usr/bin/htop]
 0x55c3c7f86c51 : main+0x451/0x660 [/usr/bin/htop]
 0x7f067850d223
Pass 5: run completed in 10usr/20sys/1033real ms.


However if I compile a program manually, stap can only print an address,
without function name:

$ stap -v -DSTP_NO_BUILDID_CHECK -e 'probe process.function("*") {
print_ubacktrace(); exit() }' -x 5465
Pass 1: parsed user script and 470 library scripts using
75332virt/51852res/7144shr/44872data kb, in 90usr/10sys/106real ms.
Pass 2: analyzed script: 341 probes, 2 functions, 0 embeds, 0 globals using
77444virt/55028res/8080shr/46984data kb, in 10usr/0sys/14real ms.
Pass 3: using cached
/root/.systemtap/cache/3d/stap_3d92b79fdcedbfc63746fbaa7ff985f2_90414.c
Pass 4: using cached
/root/.systemtap/cache/3d/stap_3d92b79fdcedbfc63746fbaa7ff985f2_90414.ko
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry
with -DDEBUG_UNWIND)'
 0x5608a9394100
Pass 5: run completed in 10usr/20sys/1038real ms.

Is it the compatible error between new gcc versions and systemtap (or its
dependencies like elfutils) ? Thanks.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/23879] print_ubacktrace can not print function name

[idealities at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

--- Comment #1 from Shang Yuanchun <idealities at gmail dot com> ---
Sorry forgot to provide gcc version, it's gcc (GCC) 8.2.1 20180831

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/23879] print_ubacktrace can not print function name

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
It depends how the two versions of the target program were compiled.  Perhaps
the second one was stripped (including symbol info), so stap (and tools like
gdb) would have no chance to resolve addresses to names.  Perhaps the unwind
data was not generated the same way (possibly CFLAGS without
-fasynchronous-unwind-tables?).

A comparison with "readelf -S" between the two binaries could help clear this
up.

Also, that advice re. rerunning with -DDEBUG_UNWIND is a good one, the data
would help.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/23879] print_ubacktrace can not print function name

[idealities at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

--- Comment #3 from Shang Yuanchun <idealities at gmail dot com> ---
(In reply to Frank Ch. Eigler from comment #2)
> It depends how the two versions of the target program were compiled. 
> Perhaps the second one was stripped (including symbol info), so stap (and
> tools like gdb) would have no chance to resolve addresses to names.  Perhaps
> the unwind data was not generated the same way (possibly CFLAGS without
> -fasynchronous-unwind-tables?).
> 
> A comparison with "readelf -S" between the two binaries could help clear
> this up.
> 
> Also, that advice re. rerunning with -DDEBUG_UNWIND is a good one, the data
> would help.

Thanks.

Actually the first one was strippped (but with right print_ubacktrace()), the
second not:

$ file /bin/htop
/bin/htop: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically
linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0,
BuildID[sha1]=f124a5c165c15004fab233235b393e078a52f4f5, stripped

$ file ./htop
./htop: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically
linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0,
BuildID[sha1]=b6c11a7eda8d30b90a46f940fc9178e3665b399a, not stripped

I used CFLAGS and LDFLAGS as below, to make it same as package built by
archlinux (-fasynchronous-unwind-tables is added by your advice, but the
problem still occurs):

CFLAGS="-fasynchronous-unwind-tables -march=x86-64 -mtune=generic -O2 -pipe
-fstack-protector-strong -fno-plt "
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"


With -DDEBUG_UNWIND:

$ stap -v -DSTP_NO_BUILDID_CHECK -DDEBUG_UNWIND -e 'probe process.function("*")
{ print_ubacktrace(); exit() }' -x 1174
Pass 1: parsed user script and 470 library scripts using
75332virt/51916res/7208shr/44872data kb, in 90usr/20sys/112real ms.
Pass 2: analyzed script: 341 probes, 2 functions, 0 embeds, 0 globals using
77444virt/55156res/8208shr/46984data kb, in 10usr/10sys/15real ms.
Pass 3: translated to C into
"/tmp/stap9fZbm5/stap_c1cd55b8b696f02a065ad588361b3e36_90426_src.c" using
77884virt/55708res/8400shr/47424data kb, in 20usr/100sys/119real ms.
Pass 4: compiled C into "stap_c1cd55b8b696f02a065ad588361b3e36_90426.ko" in
3170usr/420sys/3493real ms.
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry
with -DDEBUG_UNWIND)'
_stp_stack_unwind_one_user:496: STARTING user unwind
 0x56235b12d100
_stp_stack_unwind_one_user:512: CONTINUING user unwind to depth 1
unwind:1483: pc=56235b12d100, 56235b12d100
unwind:1522: No module found for pc=56235b12d100
_stp_stack_unwind_one_user:537: ret=-22 PC=56235b12d100 SP=7ffedbacaaa8
Pass 5: run completed in 10usr/30sys/1238real ms.


readelf output:

$ LANG=en_US.UTF-8 readelf -S ./htop
There are 27 section headers, starting at offset 0x33b18:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .interp           PROGBITS         00000000000002a8  000002a8
       000000000000001c  0000000000000000   A       0     0     1
  [ 2] .note.ABI-tag     NOTE             00000000000002c4  000002c4
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .note.gnu.build-i NOTE             00000000000002e4  000002e4
       0000000000000024  0000000000000000   A       0     0     4
  [ 4] .gnu.hash         GNU_HASH         0000000000000308  00000308
       0000000000000fa0  0000000000000000   A       5     0     8
  [ 5] .dynsym           DYNSYM           00000000000012a8  000012a8
       00000000000029d0  0000000000000018   A       6     1     8
  [ 6] .dynstr           STRTAB           0000000000003c78  00003c78
       00000000000019b8  0000000000000000   A       0     0     1
  [ 7] .gnu.version      VERSYM           0000000000005630  00005630
       000000000000037c  0000000000000002   A       5     0     2
  [ 8] .gnu.version_r    VERNEED          00000000000059b0  000059b0
       00000000000000a0  0000000000000000   A       6     2     8
  [ 9] .rela.dyn         RELA             0000000000005a50  00005a50
       00000000000055b0  0000000000000018   A       5     0     8
  [10] .init             PROGBITS         000000000000b000  0000b000
       000000000000001b  0000000000000000  AX       0     0     4
  [11] .text             PROGBITS         000000000000b020  0000b020
       0000000000014df5  0000000000000000  AX       0     0     16
  [12] .fini             PROGBITS         000000000001fe18  0001fe18
       000000000000000d  0000000000000000  AX       0     0     4
  [13] .rodata           PROGBITS         0000000000020000  00020000
       00000000000034a0  0000000000000000   A       0     0     16
  [14] .eh_frame_hdr     PROGBITS         00000000000234a0  000234a0
       0000000000000a84  0000000000000000   A       0     0     4
  [15] .eh_frame         PROGBITS         0000000000023f28  00023f28
       0000000000003df0  0000000000000000   A       0     0     8
  [16] .init_array       INIT_ARRAY       0000000000029c50  00028c50
       0000000000000008  0000000000000008  WA       0     0     8
  [17] .fini_array       FINI_ARRAY       0000000000029c58  00028c58
       0000000000000008  0000000000000008  WA       0     0     8
  [18] .data.rel.ro      PROGBITS         0000000000029c60  00028c60
       0000000000000d90  0000000000000000  WA       0     0     32
  [19] .dynamic          DYNAMIC          000000000002a9f0  000299f0
       00000000000001d0  0000000000000010  WA       6     0     8
  [20] .got              PROGBITS         000000000002abc0  00029bc0
       0000000000000438  0000000000000008  WA       0     0     8
  [21] .data             PROGBITS         000000000002b000  0002a000
       0000000000002a80  0000000000000000  WA       0     0     32
  [22] .bss              NOBITS           000000000002da80  0002ca80
       0000000000000eb0  0000000000000000  WA       0     0     32
  [23] .comment          PROGBITS         0000000000000000  0002ca80
       000000000000001a  0000000000000001  MS       0     0     1
  [24] .symtab           SYMTAB           0000000000000000  0002caa0
       0000000000004218  0000000000000018          25   260     8
  [25] .strtab           STRTAB           0000000000000000  00030cb8
       0000000000002d5f  0000000000000000           0     0     1
  [26] .shstrtab         STRTAB           0000000000000000  00033a17
       00000000000000fd  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/23879] print_ubacktrace can not print function name

[idealities at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

--- Comment #4 from Shang Yuanchun <idealities at gmail dot com> ---
I find out which really matters is the version of compiler (maybe different
version of gcc turn on different default options or enable new default
functions) and the executable file it generate.

Because if I compile a program on another host with gcc 6.3.0, and copy it to
current host and running stap again, print_ubacktrace() works as expected.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/23879] print_ubacktrace can not print function name

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |critical

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> ---
There is a recent systemic problem in the way userspace probes are identified,
and especially ubacktrace()d-from.  It appears to be something that relates to
the different way modern executables are linked / loaded.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/23879] print_ubacktrace can not print function name

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23879

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> ---
commit 4ae4592f1106e941023a5768d34c2381cc869631 fixes

-- 
You are receiving this mail because:
You are the assignee for the bug.