[HELP] ubacktrace() problem

[HELP] ubacktrace() problem

[Jinhua Luo]

Hi All,

On Debian8 (jessie), I use the sample-bt-off-cpu script to diagnose
the postgresql performance. I already install the debuginfo packages
of kernel, glibc and the postgresql, and I compile the systemtap 2.9
to use.

See https://github.com/openresty/nginx-systemtap-toolkit#sample-bt-off-cpu
for the script.

It fails to unwind some user backtrace (which is a big fraction of the
total samples), and just show some hex adddress, e.g. 0xb77a5d40.

I check that address belongs to the vdso mapping. This issue is not
specific to postgresql, but also for other programs running on the
same system. I try to reinstall the debian8 in a clean box, the issue
is the same.

I am wondering what's wrong with debian8?

And, I am also get confused about the warnings:

WARNING: Too many CFI instuctions

WARNING: DWARF expression stack underflow in CFI

WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear search
for stap_b9bfb422b7bd73acc448d7f775aaacd_37477


I try to run the same script on Ubuntu 14.04 and CentOS 7. They could
work and show me the correct user backtrace. But, the above warnings
exist too! Moreover, the warnings does not necessarily appear at
different runs of the same script! Why systemtap outputs those
warnings on these popular linux distributions? The dwarf info
associated with the kernel, glibc and other programs is not so
qualified? Then what system we need so that we could run systemtap
without any common problems?

Anybody could help me out? Thanks!


Regards,
Jinhua Luo

Re: [HELP] ubacktrace() problem

[Frank Ch. Eigler]

Hi -

luajit.io wrote:

> [...]
> See https://github.com/openresty/nginx-systemtap-toolkit#sample-bt-off-cpu
> for the script.

It's a perl-generated stap script/invocation:
https://github.com/openresty/nginx-systemtap-toolkit/blob/master/sample-bt-off-cpu


> It fails to unwind some user backtrace (which is a big fraction of the
> total samples), and just show some hex adddress, e.g. 0xb77a5d40.
>
> I check that address belongs to the vdso mapping. [...]

Please check if your kernel dbgsym packages include the vdso debuginfo
files.  In Fedora for example:

 % rpm -ql kernel-debuginfo | grep vdso
/usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso
/usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-int80.so.debug
/usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-syscall.so.debug
/usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-sysenter.so.debug
/usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso64.so.debug

These may be necessary to be able to unwind through the kernel-user
transition (if a process is interrupted mid-syscall).  I seem to
recall that, years ago, the Fedora kernel team had some struggles to
ensure that these userspace libraries' debug data was generated &
included properly.  Maybe the Debian/Ubuntu folks are about to have
the same struggle.


> And, I am also get confused about the warnings:
>
> WARNING: Too many CFI instuctions
> WARNING: DWARF expression stack underflow in CFI
> WARNING: no or bad debug frame hdr
> WARNING: No binary search table for eh frame, doing slow linear search
> for stap_b9bfb422b7bd73acc448d7f775aaacd_37477

These generally reflect imperfections in the gcc-generated unwind
information.  These are not something a user can do anything about; we
should probably stop warning about them (or give more information like
the suspect module name).

- FChE

Re: [HELP] ubacktrace() problem

[Jinhua Luo]

Hi,

2016-01-12 22:32 GMT+08:00 Frank Ch. Eigler <fche@redhat.com>:
>  % rpm -ql kernel-debuginfo | grep vdso
> /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso
> /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-int80.so.debug
> /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-syscall.so.debug
> /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-sysenter.so.debug
> /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso64.so.debug
>
> These may be necessary to be able to unwind through the kernel-user
> transition (if a process is interrupted mid-syscall).  I seem to
> recall that, years ago, the Fedora kernel team had some struggles to
> ensure that these userspace libraries' debug data was generated &
> included properly.  Maybe the Debian/Ubuntu folks are about to have
> the same struggle.
>

Thanks for the information! The debian indeed lacks of vdso files in
the kernel package.


>
>> And, I am also get confused about the warnings:
>>
>> WARNING: Too many CFI instuctions
>> WARNING: DWARF expression stack underflow in CFI
>> WARNING: no or bad debug frame hdr
>> WARNING: No binary search table for eh frame, doing slow linear search
>> for stap_b9bfb422b7bd73acc448d7f775aaacd_37477
>
> These generally reflect imperfections in the gcc-generated unwind
> information.  These are not something a user can do anything about; we
> should probably stop warning about them (or give more information like
> the suspect module name).
>

I am still wondering if the warnings denotes that the collected
backtraces are not so accurate and complete.

Regards,
Jinhua Luo

Re: [HELP] ubacktrace() problem

[Frank Ch. Eigler]

Hi -

> >  % rpm -ql kernel-debuginfo | grep vdso
> > /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso
> > [...]
> > These may be necessary to be able to unwind through the kernel-user
> > transition [...]
> 
> Thanks for the information! The debian indeed lacks of vdso files in
> the kernel package.

Please let us know when/if a debian bug is opened, so we can track it.


> [...]  I am still wondering if the warnings denotes that the
> collected backtraces are not so accurate and complete.

Well, they will be as accurate and complete as systemtap can make it,
with the imperfect data it is given.  Use the warnings to help (us)
diagnose known problem cases, otherwise please don't worry about them.


- FChE

Re: [HELP] ubacktrace() problem

[Jinhua Luo]

Hi,

I reported the issue to debian-kernel mailing list:

https://lists.debian.org/debian-kernel/2016/01/msg00266.html

I recompile the kernel, include the vdso files, now I could see the
backtrace, but it's not complete.
It lacks of main, __libc_start_main, _start.
And some functions should not be the first entry in the backtrace.

The correct graph on Ubuntu 14.04:
http://luajit.io/systemtap/pgsql/postgres_1.svg

The wrong graph on Debian8:
http://luajit.io//systemtap/pgsql/postgres_wrong.svg

Maybe the glibc and its debuginfo on Debian8 is bad?

Regards,
Jinhua Luo


2016-01-13 23:23 GMT+08:00 Frank Ch. Eigler <fche@redhat.com>:
> Hi -
>
>> >  % rpm -ql kernel-debuginfo | grep vdso
>> > /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso
>> > [...]
>> > These may be necessary to be able to unwind through the kernel-user
>> > transition [...]
>>
>> Thanks for the information! The debian indeed lacks of vdso files in
>> the kernel package.
>
> Please let us know when/if a debian bug is opened, so we can track it.
>
>
>> [...]  I am still wondering if the warnings denotes that the
>> collected backtraces are not so accurate and complete.
>
> Well, they will be as accurate and complete as systemtap can make it,
> with the imperfect data it is given.  Use the warnings to help (us)
> diagnose known problem cases, otherwise please don't worry about them.
>
>
> - FChE

Re: [HELP] ubacktrace() problem

[Mark Wielaard]

On Thu, Jan 14, 2016 at 11:40:19AM +0800, Jinhua Luo wrote:
> Maybe the glibc and its debuginfo on Debian8 is bad?

Unfortunately it is indeed. In theory you won't need debuginfo just for
backtraces if debian could enable -fasynchronous-unwind-tables:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746426
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746516

The other issue is that depending on version debian uses
.zdebug or SHF_COMPRESSED debug sections in their debug
packages. This is partially supported by systemtap (through elfutils)
for some debuginfo, but not for .debug_frame if that is
required for unwinding. This has also been an issue for
other tools like valgrind:
https://bugs.kde.org/show_bug.cgi?id=303877
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810295

Cheers,

Mark