A scenario where Perf cannot unwind the user stack by dwarf with libdw.so

A scenario where Perf cannot unwind the user stack by dwarf with libdw.so

[liuxu (AJ)]

[-- Attachment #1: Type: text/plain, Size: 6333 bytes --]

Hi all,

I am failing to unwind the user stack with libdw on Ubuntu 18.04 arm64  environment, the toolchain used is clang 12.0.1.  form the unwinding process, I found that the pc value passing to handle_cfi is 0x760, which is 0x10000 offset from the corresponding cfi address 0x10760.

Futher analysis show that the bias calculated in __libdwfl_elf_address_range for ET_DYN type pick the first PT_LOAD segment,  not the segment corresponds to code segment. So the bias result is wrong in current situation.

My temporary repair plan is shown in the bottom of the email.I'm not sure if this is the correct way to solve this problem, any advice?

Many thanks again!
--
Liu Xu

PS:
Diff.patch:
diff --git a/libdwfl/dwfl_report_elf.c b/libdwfl/dwfl_report_elf.c
index 3fc9384..e75fef1 100644
--- a/libdwfl/dwfl_report_elf.c
+++ b/libdwfl/dwfl_report_elf.c
@@ -185,9 +185,9 @@ __libdwfl_elf_address_range (Elf *elf, GElf_Addr base, bool add_p_vaddr,
      GElf_Phdr phdr_mem, *ph = gelf_getphdr (elf, i, &phdr_mem);
      if (unlikely (ph == NULL))
        goto elf_error;
-     if (ph->p_type == PT_LOAD)
+     if (ph->p_type == PT_LOAD && (ph->p_flags & PF_X))
        {
-         vaddr = ph->p_vaddr & -ph->p_align;
+         vaddr = (ph->p_vaddr - ph->p_offset) & -ph->p_align;
          address_sync = ph->p_vaddr + ph->p_memsz;
          break;
        }


Source code:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/time.h>
#define PER_RUNNING_TIME 500

void test_005()
{
    int i = 0;
    i++;
    while (1) {};
}

void test_004()
{
    for (int i = 10;i > 0; i--) {
        test_005();
    }
}

void test_003()
{
        test_004();
}

void test_002()
{
        test_003();
}

void test_001()
{
    test_002();
}

int main()
{
    test_001();
    return 0;
}

Compile command: clang main_dwarf.c -fomit-frame-pointer -o test_dwarf_nofp.

Sampling command: perf record --call-graph dwarf

Report command: perf report -v

The Program Headers of elf file :
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x0000000000000230 0x0000000000000230  R      0x8
  INTERP         0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x000000000000001a 0x000000000000001a  R      0x1
      [Requesting program interpreter: /lib/ld-musl-aarch64.so.1]
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x000000000000065c 0x000000000000065c  R      0x10000
  LOAD           0x000000000000065c 0x000000000001065c 0x000000000001065c
                 0x0000000000000234 0x0000000000000234  R E    0x10000
  LOAD           0x0000000000000890 0x0000000000020890 0x0000000000020890
                 0x00000000000001b0 0x00000000000001b0  RW     0x10000
  LOAD           0x0000000000000a40 0x0000000000030a40 0x0000000000030a40
                 0x0000000000000040 0x0000000000000089  RW     0x10000
  DYNAMIC        0x00000000000008a0 0x00000000000208a0 0x00000000000208a0
                 0x0000000000000170 0x0000000000000170  RW     0x8
  GNU_RELRO      0x0000000000000890 0x0000000000020890 0x0000000000020890
                 0x00000000000001b0 0x0000000000000770  R      0x1
  GNU_EH_FRAME   0x00000000000004f8 0x00000000000004f8 0x00000000000004f8
                 0x000000000000004c 0x000000000000004c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x0
Section to Segment mapping:
  Segment Sections...
   00
   01     .interp
   02     .interp .dynsym .gnu.hash .dynstr .rela.dyn .rela.plt .eh_frame_hdr .eh_frame
   03     .text .init .fini .plt
   04     .init_array .fini_array .dynamic .got
   05     .data .got.plt .bss
   06     .dynamic
   07     .init_array .fini_array .dynamic .got
   08     .eh_frame_hdr
   09

.eh_frame of elf file:
00000000 0000000000000014 00000000 CIE "zR" cf=1 df=-4 ra=30
   LOC           CFA
0000000000000000 sp+0

00000018 000000000000001c 0000001c FDE cie=00000000 pc=0000000000010678..00000000000106c4
   LOC           CFA      x29   ra
0000000000010678 sp+0     u     u
0000000000010680 x29+16   c-16  c-8

00000038 000000000000001c 0000003c FDE cie=00000000 pc=00000000000106c4..0000000000010724
   LOC           CFA      x29   ra
00000000000106c4 sp+0     u     u
00000000000106cc x29+16   c-16  c-8

00000058 0000000000000014 0000005c FDE cie=00000000 pc=0000000000010724..000000000001073c
   LOC           CFA
0000000000010724 sp+0
0000000000010728 sp+16

00000070 0000000000000014 00000074 FDE cie=00000000 pc=000000000001073c..0000000000010778
   LOC           CFA      ra
000000000001073c sp+0     u
0000000000010744 sp+32    c-16

00000088 0000000000000014 0000008c FDE cie=00000000 pc=0000000000010778..0000000000010788
   LOC           CFA      ra
0000000000010778 sp+0     u
000000000001077c sp+16    c-16

000000a0 0000000000000014 000000a4 FDE cie=00000000 pc=0000000000010788..0000000000010798
   LOC           CFA      ra
0000000000010788 sp+0     u
000000000001078c sp+16    c-16

000000b8 0000000000000014 000000bc FDE cie=00000000 pc=0000000000010798..00000000000107a8
   LOC           CFA      ra
0000000000010798 sp+0     u
000000000001079c sp+16    c-16

000000d0 0000000000000014 000000d4 FDE cie=00000000 pc=00000000000107a8..00000000000107d0
   LOC           CFA      ra
00000000000107a8 sp+0     u
00000000000107b0 sp+32    c-16

000000e8 ZERO terminator


Contents of the .debug_frame section:


00000000 0000000000000014 ffffffff CIE "" cf=1 df=-4 ra=30
   LOC           CFA
0000000000000000 sp+0

00000018 0000000000000014 00000000 FDE cie=00000000 pc=0000000000010650..0000000000010678

The actual parameters of handle_cfi when I report:
Breakpoint 1, handle_cfi (state=state@entry=0xaaaaaadc45b0, pc=1888, cfi=0xaaaaaadc4900, bias=757661696) at frame_unwind.c:540
warning: Source file is more recent than executable.
540     {
(gdb) p/x pc
$1 = 0x760
(gdb) p/x bias
$2 = 0x2d290000

Re: A scenario where Perf cannot unwind the user stack by dwarf with libdw.so

[Mark Wielaard]

Hi,

On Mon, Feb 13, 2023 at 01:44:39AM +0000, liuxu (AJ) via Elfutils-devel wrote:
> I am failing to unwind the user stack with libdw on Ubuntu 18.04
> arm64 environment, the toolchain used is clang 12.0.1.  form the
> unwinding process, I found that the pc value passing to handle_cfi
> is 0x760, which is 0x10000 offset from the corresponding cfi address
> 0x10760.
>
> Futher analysis show that the bias calculated in
> __libdwfl_elf_address_range for ET_DYN type pick the first PT_LOAD
> segment, not the segment corresponds to code segment. So the bias
> result is wrong in current situation.

I have some problem replicating this because I don't have the same
toolchain and don't know enough about perf to know how to replicate.

So if I understand correctly you are using clang and lld to link
against a musl based libc/ld.so on aarch64? Does this generate PIE
executables?

Have you tried a different setup? I expect the issue is the PT_LOAD
segments generated by lld are different from ld (bfd). But it would be
good to know if this is the difference that makes things fail.

Are you able to show the issue using eu-stack instead of perf?

If not, could you show how to see the issue with perf?

> The actual parameters of handle_cfi when I report:
> Breakpoint 1, handle_cfi (state=state@entry=0xaaaaaadc45b0, pc=1888, cfi=0xaaaaaadc4900, bias=757661696) at frame_unwind.c:540
> warning: Source file is more recent than executable.

Could you show the backtrace at this point?  (Note that gdb believes
you have changed the sources, but haven't rebuild libdw.so)

Thanks,

Mark

Re: A scenario where Perf cannot unwind the user stack by dwarf with libdw.so

[Mark Wielaard]

Hi,

On Thu, 2023-02-16 at 23:48 +0100, Mark Wielaard wrote:
> On Mon, Feb 13, 2023 at 01:44:39AM +0000, liuxu (AJ) via Elfutils-devel wrote:
> > I am failing to unwind the user stack with libdw on Ubuntu 18.04
> > arm64 environment, the toolchain used is clang 12.0.1.  form the
> > unwinding process, I found that the pc value passing to handle_cfi
> > is 0x760, which is 0x10000 offset from the corresponding cfi address
> > 0x10760.
> > 
> > Futher analysis show that the bias calculated in
> > __libdwfl_elf_address_range for ET_DYN type pick the first PT_LOAD
> > segment, not the segment corresponds to code segment. So the bias
> > result is wrong in current situation.
> 
> I have some problem replicating this because I don't have the same
> toolchain and don't know enough about perf to know how to replicate.

We have discussed this a bit more offlist. But I still haven't been
able to replicate this issue. It seems to depend on a couple of things
I don't fully understand, clang, lld, musl, PIE executables, whether
the fallback frame based unwinder can/is used, etc.

I am hoping someone can sent me a binary to investigate.

Thanks,

Mark