debugging pthreads

debugging pthreads

[Aras Vaichas]

Hello,

I am having trouble debugging pthreads on my target. I am not sure
what the cause is, but I suspect it could be either my kernel or my
libraries.

I am using crosstool-ng-1.16.0 to create tools for for an
i486-linux-unknown-gnu target, gcc 4.4.6, glibc 2.9, kernel 3.3.8.

I have created the target gdb and gdbserver using ctng.

I can run multithreaded apps on my target, but I cannot debug them
with gdb or Google Breakpad.

e.g. gdb simply fails

--- snip ---
$ ./gdb --batch -n -ex "thread apply all backtrace" ./a.out --pid 716

dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1:
undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.

0xb75aa78c in nanosleep () from /lib/libc.so.6

<< there should be backtraces for several threads here>>
--- snip ---


I understand the "undefined symbol" shouldn't be a problem because of
this: http://permalink.gmane.org/gmane.linux.debian.devel.glibc/4409

If I run the same app on my host, I get backtraces for all the threads.

e.g. gdb on my host with the same test application:

--- snip ---
sudo gdb --batch -n -ex "thread apply all backtrace" ./a.out --pid 17806
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
[New Thread 0xb53ffb40 (LWP 17811)]
[New Thread 0xb5f42b40 (LWP 17810)]
[New Thread 0xb6743b40 (LWP 17809)]
[New Thread 0xb6f44b40 (LWP 17808)]
[New Thread 0xb7745b40 (LWP 17807)]
0x00bec416 in __kernel_vsyscall ()
Thread 6 (Thread 0xb7745b40 (LWP 17807)):
#0  0x00bec416 in __kernel_vsyscall ()
#1  0x0074ec16 in nanosleep () from /lib/i386-linux-gnu/libc.so.6
#2  0x0074ea0f in sleep () from /lib/i386-linux-gnu/libc.so.6
#3  0x0804a3b2 in thread () at test.cpp:8
#4  0x0804b919 in boost::detail::thread_data<void (*)()>::run
(this=0x8a22008) at /usr/include/boost/thread/detail/thread.hpp:61
#5  0x0096f1cc in ?? () from /usr/lib/libboost_thread.so.1.48.0
#6  0x00134d4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0x00785ace in clone () from /lib/i386-linux-gnu/libc.so.6

... etc...

Thread 1 (Thread 0xb7747700 (LWP 17806)):
#0  0x00bec416 in __kernel_vsyscall ()
#1  0x0074ec16 in nanosleep () from /lib/i386-linux-gnu/libc.so.6
#2  0x0074ea0f in sleep () from /lib/i386-linux-gnu/libc.so.6
#3  0x0804a3d2 in f4 () at test.cpp:14
#4  0x0804a3eb in f3 () at test.cpp:20
#5  0x0804a3f8 in f2 () at test.cpp:25
#6  0x0804a405 in f1 () at test.cpp:30
#7  0x0804a4a2 in main () at test.cpp:41
--- snip ---

Google Breakpad simply fails to product a minidump for any application
that contains threads. It does not fail on my target when the test app
is single threaded. The same Google Breakpad library works on my host
with a multi threaded test application.

Is there something fundamental that needs to be in place for pthread
debugging that I may have omitted from my build or system?

Does libthread_db have any other dependencies?

Aras

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: debugging pthreads

[Bill Pringlemeir]

On  7 Sep 2012, aras.vaichas@gmail.com wrote:

> I am having trouble debugging pthreads on my target. I am not sure
> what the cause is, but I suspect it could be either my kernel or my
> libraries.

It seems that you need symbols to debug threads.  I guess that gdb is
looking for variables to determine what threads/stacks are where.  It
cann't seem to get back traces without symbols in the 'libc', etc.

Have you stripped everything before you put it on your target? 

  http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

I found that on my ARM target, I could strip the libraries and use
'--add-gnu-debug-link' method.  I then can transfer the '.debug' files
when I have an issue and gdb will read them and suddenly, I can use the
pthread facility.

...probably not a CTNG issue.

hth,
Bill Pringlemeir.

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: debugging pthreads

[Aras Vaichas]

On 7 September 2012 13:26, Bill Pringlemeir <bpringle@sympatico.ca> wrote:
> On  7 Sep 2012, aras.vaichas@gmail.com wrote:
>
>> I am having trouble debugging pthreads on my target. I am not sure
>> what the cause is, but I suspect it could be either my kernel or my
>> libraries.
>
> It seems that you need symbols to debug threads.  I guess that gdb is
> looking for variables to determine what threads/stacks are where.  It
> cann't seem to get back traces without symbols in the 'libc', etc.
>
> Have you stripped everything before you put it on your target?

Nope. Libraries are used as-is from Crosstools and Buildroot. The only
build difference between my test app build for host and target is the
C++ compiler. Yet GDB works on my host, but does not work on my target
for multithreaded apps. GDB does work for single threaded apps on my
target.

I found some interesting leads. Denys Dmytriyenko from TI said:

"Normally, GDB is not very picky and you can mix and match different
versions of gdb and gdbserver. But, unfortunately, if you need to
debug multi-threaded apps, there are some dependencies for specific
APIs...
For example, this is one of the messages you may see if you didn't
build GDB properly for the thread support:
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1:
undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads."

ref: http://e2e.ti.com/support/embedded/linux/f/354/t/52720.aspx

This is the same error message that I get.

The GDB FAQ says a cause of problems with GDB and threads can be:

"[a] mismatch between libpthread.so.0 and libthread_db.so.1
GDB itself does not know how to decode "thread control blocks"
maintained by glibc and considered to be glibc private implementation
detail. It uses libthread_db.so.1 (part of glibc) to help it do so.
Therefore, libthread_db.so.1 and libpthread.so.0 must match in version
and compilation flags. In addition, libthread_db.so.1 requires certain
non-global symbols to be present in libpthread.so.0. Solution: use
strip --strip-debug libpthread.so.0 instead of strip libpthread.so.0."

ref: http://sourceware.org/gdb/wiki/FAQ


> ...probably not a CTNG issue.

GDB, libpthread and libthread_db are all configured and built by
crosstools, so it seems the most likely place to look for the answers.

Aras

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: debugging pthreads

[Aras Vaichas]

On 7 September 2012 13:26, Bill Pringlemeir <bpringle@sympatico.ca> wrote:
> On  7 Sep 2012, aras.vaichas@gmail.com wrote:
>
>> I am having trouble debugging pthreads on my target. I am not sure
>> what the cause is, but I suspect it could be either my kernel or my
>> libraries.
>
> It seems that you need symbols to debug threads.  I guess that gdb is
> looking for variables to determine what threads/stacks are where.

Yes, you were half right. I was building a *static* gdb and that meant
that the .dynsym table was stripped from the native gdb. This meant
that thread_db could not find the ps_lgetfpregs symbol in gdb and
therefore thread debugging was not available.

i.e. -static causes -rdynamic to be functionally negated.

AFAIK to make a static gdb work with threads, you need to build a
static thread_db as well.

I rebuilt crosstools with a non-static gdb and I can now debug threads
on my target.


> ...probably not a CTNG issue.

A warning in menuconfig would be nice. :)

Aras

--
For unsubscribe information see http://sourceware.org/lists.html#faq