Re: How to debug broken unwinding?

[Milian Wolff <mail@milianw.de>]

[-- Attachment #1: Type: text/plain, Size: 2840 bytes --]

On Donnerstag, 1. Juni 2017 22:57:12 CEST Milian Wolff wrote:
> Hey all,
> 
> on my ArchLinux box I regularly see cases where libdw fails to unwind
> properly. I can reproduce this both with upstream perf as well as with the
> perfparser utility.
> 
> How should I debug this, or how can I report a good bug report for this? I
> guess I could upload a perf archive and document the steps required to build
> perf with libdw as the unwinder, as it allows to easily compare libunwind
> and libdw for unwinding. When I then diff the output of `perf script` for
> the two unwinders for one and the same perf.data file, I see issues like
> this:
> 
> $ diff -u script.libunwind script.elfutils
> --- script.libunwind    2017-06-01 22:30:52.418029474 +0200
> +++ script.elfutils2    2017-06-01 22:35:10.987823055 +0200
> @@ -510,10 +510,6 @@
>                     e8ed _dl_fixup (/usr/lib/ld-2.25.so)
>                    15f06 _dl_runtime_resolve_sse_vex (/usr/lib/ld-2.25.so)
>                    ed94c KDynamicJobTracker::KDynamicJobTracker
> (/home/milian/ projects/compiled/kf5/lib64/libKF5KIOWidgets.so.5.35.0)
> -                  608f3 _GLOBAL__sub_I_kdynamicjobtracker.cpp
> (/home/milian/ projects/compiled/kf5/lib64/libKF5KIOWidgets.so.5.35.0)
> -                   f199 call_init.part.0 (/usr/lib/ld-2.25.so)
> -                   f2a5 _dl_init (/usr/lib/ld-2.25.so)
> -                    db9 _dl_start_user (/usr/lib/ld-2.25.so)
> 
> NOTE: it seems as if unwinding through _GLOBAL__sub* always fails?

This part I now investigated with extensive debug output and figured out the 
issue: For the last frame that works in both cases, i.e. ed94c, libdwfl says 
it knows that this address belongs to /usr/lib/ld-2.25.so. In reality, it 
belongs to libKF5KIOWidgets.so.5.35.0.

Previously, perf just checked whether any module is known to libdwfl for a 
given address and then trusted it to do the right thing. Now I created a patch 
that double-checks whether the mapping known to libdwfl matches what perf 
knows. If not, we report the correct map (as known to perf) and this fixes the 
issue.

In general, I believe that libdwfl's API is lacking here. Both perf and 
perfparser know the exact mapping of a file, i.e. the file, it's start and end  
address, as well as the pgoff. But the integration with dwfl simply calls 
dwfl_report_elf, which only takes a start address. For things like ld-2.25.so 
this is often not enough. Is there any chance to expand the API to let us set 
the explicit mapping addresses?

I see there's dwfl_report_module, which at least takes start and end address, 
but so far I always failed to use it for unwinding - it seems as if that 
function is not setting up the internal ELF file and thus all of the functions 
relying on that will break.

Thanks

-- 
Milian Wolff
mail@milianw.de
http://milianw.de

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]