How to debug broken unwinding?

[Milian Wolff <mail@milianw.de>]

[-- Attachment #1: Type: text/plain, Size: 5766 bytes --]

Hey all,

on my ArchLinux box I regularly see cases where libdw fails to unwind 
properly. I can reproduce this both with upstream perf as well as with the 
perfparser utility.

How should I debug this, or how can I report a good bug report for this? I 
guess I could upload a perf archive and document the steps required to build 
perf with libdw as the unwinder, as it allows to easily compare libunwind and 
libdw for unwinding. When I then diff the output of `perf script` for the two 
unwinders for one and the same perf.data file, I see issues like this:

$ diff -u script.libunwind script.elfutils
--- script.libunwind    2017-06-01 22:30:52.418029474 +0200
+++ script.elfutils2    2017-06-01 22:35:10.987823055 +0200
@@ -510,10 +510,6 @@
                    e8ed _dl_fixup (/usr/lib/ld-2.25.so)
                   15f06 _dl_runtime_resolve_sse_vex (/usr/lib/ld-2.25.so)
                   ed94c KDynamicJobTracker::KDynamicJobTracker (/home/milian/
projects/compiled/kf5/lib64/libKF5KIOWidgets.so.5.35.0)
-                  608f3 _GLOBAL__sub_I_kdynamicjobtracker.cpp (/home/milian/
projects/compiled/kf5/lib64/libKF5KIOWidgets.so.5.35.0)
-                   f199 call_init.part.0 (/usr/lib/ld-2.25.so)
-                   f2a5 _dl_init (/usr/lib/ld-2.25.so)
-                    db9 _dl_start_user (/usr/lib/ld-2.25.so)

NOTE: it seems as if unwinding through _GLOBAL__sub* always fails?

 heaptrack_gui  2228 135073.400474:     613969 cycles: 
                  108c8e [unknown] (/usr/lib/libQt5Core.so.5.8.0)
@@ -533,8 +529,6 @@
                  2170af [unknown] (/usr/lib/libQt5Core.so.5.8.0)
                  297c53 QCoreApplicationPrivate::init (/usr/lib/
libQt5Core.so.5.8.0)
                   f7cde QGuiApplicationPrivate::init (/usr/lib/libQt5Gui.so.
5.8.0)
-                 1589e8 QApplicationPrivate::init (/usr/lib/libQt5Widgets.so.
5.8.0)
-                  78622 main (/home/milian/projects/compiled/other/bin/
heaptrack_gui)
                   20439 __libc_start_main (/usr/lib/libc-2.25.so)
                   78299 _start (/home/milian/projects/compiled/other/bin/
heaptrack_gui)
 
NOTE: this is super odd, it simply misses two frames in the middle?!

@@ -557,14 +551,7 @@
                  2780c7 QFactoryLoader::update (/usr/lib/libQt5Core.so.5.8.0)
                  279525 QFactoryLoader::QFactoryLoader (/usr/lib/
libQt5Core.so.5.8.0)
                   e5bd0 QPlatformIntegrationFactory::create (/usr/lib/
libQt5Gui.so.5.8.0)
-                  f5a1c QGuiApplicationPrivate::createPlatformIntegration (/
usr/lib/libQt5Gui.so.5.8.0)
-                  f650c QGuiApplicationPrivate::createEventDispatcher (/usr/
lib/libQt5Gui.so.5.8.0)
-                 298524 QCoreApplicationPrivate::init (/usr/lib/
libQt5Core.so.5.8.0)
-                  f7cde QGuiApplicationPrivate::init (/usr/lib/libQt5Gui.so.
5.8.0)
-                 1589e8 QApplicationPrivate::init (/usr/lib/libQt5Widgets.so.
5.8.0)
-                  78622 main (/home/milian/projects/compiled/other/bin/
heaptrack_gui)
-                  20439 __libc_start_main (/usr/lib/libc-2.25.so)
-                  78299 _start (/home/milian/projects/compiled/other/bin/
heaptrack_gui)
+                 723dbf [unknown] ([unknown])
 
NOTE: this is sadly a very common issue with libdw where it fails to unwind 
after some frames. The computed address for the next frame is totally off 
(723dbf vs f5a1c) and then everything stops...

 heaptrack_gui  2228 135073.402820:     569191 cycles: 
                    90a2 do_lookup_x (/usr/lib/ld-2.25.so)
@@ -732,23 +670,12 @@
                    b768 _dl_relocate_object (/usr/lib/ld-2.25.so)
                   135e2 dl_open_worker (/usr/lib/ld-2.25.so)
                  126c93 _dl_catch_error (/usr/lib/libc-2.25.so)
-                  12f28 _dl_open (/usr/lib/ld-2.25.so)
                     f25 [unknown] (/usr/lib/libdl-2.25.so)
                  126c93 _dl_catch_error (/usr/lib/libc-2.25.so)
                    15f4 [unknown] (/usr/lib/libdl-2.25.so)
                     fb0 dlopen (/usr/lib/libdl-2.25.so)
                  28dd00 [unknown] (/usr/lib/libQt5Core.so.5.8.0)
-                 287164 [unknown] (/usr/lib/libQt5Core.so.5.8.0)
-                 2773af QFactoryLoader::instance (/usr/lib/libQt5Core.so.
5.8.0)
-                  e5b3e QPlatformIntegrationFactory::create (/usr/lib/
libQt5Gui.so.5.8.0)
-                  f5a1c QGuiApplicationPrivate::createPlatformIntegration (/
usr/lib/libQt5Gui.so.5.8.0)
-                  f650c QGuiApplicationPrivate::createEventDispatcher (/usr/
lib/libQt5Gui.so.5.8.0)
-                 298524 QCoreApplicationPrivate::init (/usr/lib/
libQt5Core.so.5.8.0)
-                  f7cde QGuiApplicationPrivate::init (/usr/lib/libQt5Gui.so.
5.8.0)
-                 1589e8 QApplicationPrivate::init (/usr/lib/libQt5Widgets.so.
5.8.0)
-                  78622 main (/home/milian/projects/compiled/other/bin/
heaptrack_gui)
-                  20439 __libc_start_main (/usr/lib/libc-2.25.so)
-                  78299 _start (/home/milian/projects/compiled/other/bin/
heaptrack_gui)
+                 72499f [unknown] ([unknown])
 
NOTE: a mixture of missing frames in the middle as well as broken backtrace at 
the end

Since this is (sadly) very easy for me to reproduce, I'd be willing to invest 
some time to get this properly fixed. I fail to come up with a way to put this 
into a MWE that does not depend on tons of DSOs on my machine. I fear it's the 
only way to reproduce it though... But since I'm far from an expert in DWARF I 
have no clue on how to even begin to tackle this issue. Would someone more 
involved with this matter accept a `perf archive` to investigate what perf + 
libdw are doing with my DSOs?

Thanks
-- 
Milian Wolff
mail@milianw.de
http://milianw.de

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 195 bytes --]