gdb debugging threads help

gdb debugging threads help

[Teresa Rivas]

Hi,

I need to debug multi-threaded applications that runs on an arm platform
in my x86 platform but I can't make that gdb compile and build the
thread-db sources. This is the way I configured gdb:

./configure --host=i386-linux --program-prefix=arm-linux-
--target=arm-linux

and this is the way I configure gdbserver:

./configure --host=arm-linux --program-prefix=arm-linux-
--target=arm-linux

this are the libraries in the target and which paths I set in gdb using
set solib-absolute-prefix and set solib-search-path commands:

-rwxr-xr-x  1 trivas trivas   91306 2005-11-17 08:12
libpthread-0.9.26.so
lrwxrwxrwx  1 trivas trivas      20 2005-11-17 08:12 libpthread.so ->
libpthread-0.9.26.so
lrwxrwxrwx  1 trivas trivas      20 2005-11-17 08:12 libpthread.so.0 ->
libpthread-0.9.26.so
-rwxr-xr-x  1 trivas trivas   19300 2005-11-17 16:36
libthread_db-0.9.26.so
lrwxrwxrwx  1 trivas trivas      22 2005-11-17 08:12 libthread_db.so ->
libthread_db-0.9.26.so
lrwxrwxrwx  1 trivas trivas      22 2005-11-17 08:12 libthread_db.so.0
-> libthread_db-0.9.26.so
lrwxrwxrwx  1 trivas trivas      22 2005-11-17 17:03 libthread_db.so.1
-> libthread_db-0.9.26.so

the problem is that I can't see the treads created when I am debugging
the application, only the main thread:

on the target:
# gdbserver :5234 simple
Process simple created; pid = 91
Listening on port 5234
Remote debugging from host 192.168.200.29
thread 1 created
thread 2 created
Glory, employees contents was always consistent

Child exited with retcode = 0

Child exited with status 0
GDBserver exiting


on the host:
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "--host=i686-linux --target=arm-linux".
(gdb) set solib-absolute-prefix /target/arm-linux/lib
(gdb) set solib-search-path /target/arm-linux/lib
(gdb) dir /target/arm-linux/lib
Source directories searched: /target/arm-linux/lib:$cdir:$cwd
(gdb) set endian little
The target is assumed to be little endian
(gdb) target remote 192.168.200.199:5234
Remote debugging using 192.168.200.199:5234
0x40000c60 in ?? ()
(gdb) handle SIG32 nostop noprint pass
Signal        Stop      Print   Pass to program Description
SIG32         No        No      Yes             Real-time event 32
(gdb) add-symbol-file simple
add symbol table from file "simple" at
(y or n) y
Reading symbols
from /home/trivas/dm320-appro/fs/examples/threads/simple...done.
(gdb) b 87
Breakpoint 1 at 0x8834: file simple.c, line 87.
(gdb) b 122
Breakpoint 2 at 0x8a68: file simple.c, line 122.
(gdb) c
Continuing.

Breakpoint 1, main (argc=1, argv=0xbefffed4) at simple.c:90
90          for (i=0; i<60000; i++) {
(gdb) info threads
  1 Thread 47  main (argc=1, argv=0xbefffed4) at simple.c:90
(gdb) c
Continuing.

Breakpoint 2, main (argc=1, argv=0xbefffed4) at simple.c:123
123         printf("Glory, employees contents was always consistent\n");
(gdb) info threads
  1 Thread 47  main (argc=1, argv=0xbefffed4) at simple.c:123
(gdb) c
Continuing.

Program exited normally.
(gdb)

and if I try a info shared it says "No shared libraries loaded at this
time"

I see there is a thread-db.o target on the gdb/Makefile file but I can't
configure gdb so it execute this thread-db.o target which I believe add
the gdb debbug thread support.

Have had somebody a similar problem or an idea that can help me?

Re: gdb debugging threads help

[Jim Blandy]

On 11/17/05, Teresa Rivas <teresa.rivas@cadenux.com> wrote:
> I see there is a thread-db.o target on the gdb/Makefile file but I can't
> configure gdb so it execute this thread-db.o target which I believe add
> the gdb debbug thread support.

I haven't used gdbserver much, but I can at least tell you not to
worry about thread-db.o.  That's not appropriate for cross-debugging
configurations; it's only used for native debugging.  That file acts
as the interface between GDB and the libthread_db.so provided by the
operating system; when cross-debugging, it's gdbserver that needs to
link against the libthread_db.so, not GDB.

Re: gdb debugging threads help

[Daniel Jacobowitz]

On Thu, Nov 17, 2005 at 06:36:10PM -0600, Teresa Rivas wrote:
> on the target:
> # gdbserver :5234 simple

Is gdbserver linked to libthread_db?  Check the build log, or check the
gdbserver binary.

> (gdb) set solib-absolute-prefix /target/arm-linux/lib

That's probably not right.  The paths below solib-absolute-prefix
should match the target filesystem.  So if solib-absolute-prefix
is "/target/arm-linux", then the target /lib/ld-linux.so.2 should be in
/target/arm-linux/lib/ld-linux.so.2.

> (gdb) set solib-search-path /target/arm-linux/lib

You should not need to set that.

> (gdb) dir /target/arm-linux/lib

Or this.

> Source directories searched: /target/arm-linux/lib:$cdir:$cwd
> (gdb) set endian little
> The target is assumed to be little endian

Or this.

> (gdb) handle SIG32 nostop noprint pass
> Signal        Stop      Print   Pass to program Description
> SIG32         No        No      Yes             Real-time event 32

Don't do that; it never helps.

> (gdb) add-symbol-file simple
> add symbol table from file "simple" at
> (y or n) y
> Reading symbols
> from /home/trivas/dm320-appro/fs/examples/threads/simple...done.

Don't do this either.  Use "file simple", and do it before issuing the
"target remote" command.

> and if I try a info shared it says "No shared libraries loaded at this
> time"

Thread debugging will not work until this works.  It should show you
the list of libraries and say that they have symbols loaded (at main,
not when you first connect).

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: gdb debugging threads help

[Teresa Rivas]

> > on the target:
> > # gdbserver :5234 simple
> 
> Is gdbserver linked to libthread_db?  Check the build log, or check the
> gdbserver binary.

I check the config.log and I have this errors when checking for the
thread_db library:

configure:1510: checking for libthread_db
configure:1531: gcc -o conftest -g -O2   conftest.c  -lthread_db 1>&5
configure:1596: gcc -o conftest -g -O2   -rdynamic conftest.c
-lthread_db 1>&5
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_pdwrite'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_pglobal_lookup'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_lsetfpregs'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_getpid'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_lsetregs'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_pdread'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_lgetfpregs'
/usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
undefined reference to `ps_lgetregs'
collect2: ld returned 1 exit status
configure: failed program was:
#line 1589 "configure"
#include "confdefs.h"


I see that is wrong that looks for the thread_db library in my host
libs, it should look in the target libs, so I add the param
--libdir=/target/libs to the configure and now the config.log doesn't
say any error about thread_db library and I can compile gdbserver fine.

Which command can I use to see if it is linked against the correct
thread_db library?? I use a nm -D gdbserver but all I see are the
symbols from the dynamic libraries not the libraries.

.....
0000eeec T target_signal_to_name
0000ee90 T target_signal_to_string
         U tcgetattr
         U tcsetattr
         U tcsetpgrp
         U td_symbol_list
         U td_ta_event_addr
         U td_ta_event_getmsg
         U td_ta_new
         U td_ta_set_event
         U td_ta_thr_iter
         U td_thr_event_enable
         U td_thr_get_info
0001f42c D the_low_target
00020784 B the_target
00014664 T thread_db_init
00020690 B thread_from_wait
......

Now I get a different error from gdb:

on the target:

# gdbserver :5234 simple
Process simple created; pid = 41
Listening on port 5234


on the host:

GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "--host=i686-linux --target=arm-linux".
(gdb) set solib-absolute-prefix /lib
(gdb) file simple
Reading symbols
from /home/trivas/dm320-appro/fs/examples/threads/simple...done.
(gdb) target remote 192.168.200.199:5234
Remote debugging using 192.168.200.199:5234
0x40000c60 in ?? ()
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
(gdb) info shared
No shared libraries loaded at this time.
(gdb)


> > (gdb) set solib-absolute-prefix /target/arm-linux/lib
> 
> That's probably not right.  The paths below solib-absolute-prefix
> should match the target filesystem.  So if solib-absolute-prefix
> is "/target/arm-linux", then the target /lib/ld-linux.so.2 should be in
> /target/arm-linux/lib/ld-linux.so.2.
> 
> > (gdb) set solib-search-path /target/arm-linux/lib
> 
> You should not need to set that.
> 
> > (gdb) dir /target/arm-linux/lib
> 
> Or this.
> 
> > Source directories searched: /target/arm-linux/lib:$cdir:$cwd
> > (gdb) set endian little
> > The target is assumed to be little endian
> 
> Or this.
> 
> > (gdb) handle SIG32 nostop noprint pass
> > Signal        Stop      Print   Pass to program Description
> > SIG32         No        No      Yes             Real-time event 32
> 
> Don't do that; it never helps.
> 
> > (gdb) add-symbol-file simple
> > add symbol table from file "simple" at
> > (y or n) y
> > Reading symbols
> > from /home/trivas/dm320-appro/fs/examples/threads/simple...done.
> 
> Don't do this either.  Use "file simple", and do it before issuing the
> "target remote" command.
> 
> > and if I try a info shared it says "No shared libraries loaded at this
> > time"
> 
> Thread debugging will not work until this works.  It should show you
> the list of libraries and say that they have symbols loaded (at main,
> not when you first connect).
> 
-- 
Teresa R. Rivas
teresa.rivas@cadenux.com
Cadenux, LLC

Re: gdb debugging threads help

[Teresa Rivas]

Also I tried a c in the gdb and I get this errors:

(gdb) c
Continuing.
Error while mapping shared library sections:
/lib/libc.so.0: No such file or directory.
Error while mapping shared library sections:
/lib/libpthread.so.0: No such file or directory.
Error while mapping shared library sections:
/lib/ld-uClibc.so.0: No such file or directory.
Error while reading shared library symbols:
/lib/libc.so.0: No such file or directory.
Error while reading shared library symbols:
/lib/ld-uClibc.so.0: No such file or directory.

Program exited normally.
(gdb) quit

so even when this libraries are in the target file system I am debugging
remote from my host so on my host these libraries do not exist
under /lib instead they are under target/fs/lib.


> > > on the target:
> > > # gdbserver :5234 simple
> > 
> > Is gdbserver linked to libthread_db?  Check the build log, or check the
> > gdbserver binary.
> 
> I check the config.log and I have this errors when checking for the
> thread_db library:
> 
> configure:1510: checking for libthread_db
> configure:1531: gcc -o conftest -g -O2   conftest.c  -lthread_db 1>&5
> configure:1596: gcc -o conftest -g -O2   -rdynamic conftest.c
> -lthread_db 1>&5
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_pdwrite'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_pglobal_lookup'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_lsetfpregs'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_getpid'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_lsetregs'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_pdread'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_lgetfpregs'
> /usr/lib/gcc/i486-linux-gnu/3.4.5/../../../../lib/libthread_db.so:
> undefined reference to `ps_lgetregs'
> collect2: ld returned 1 exit status
> configure: failed program was:
> #line 1589 "configure"
> #include "confdefs.h"
> 
> 
> I see that is wrong that looks for the thread_db library in my host
> libs, it should look in the target libs, so I add the param
> --libdir=/target/libs to the configure and now the config.log doesn't
> say any error about thread_db library and I can compile gdbserver fine.
> 
> Which command can I use to see if it is linked against the correct
> thread_db library?? I use a nm -D gdbserver but all I see are the
> symbols from the dynamic libraries not the libraries.
> 
> .....
> 0000eeec T target_signal_to_name
> 0000ee90 T target_signal_to_string
>          U tcgetattr
>          U tcsetattr
>          U tcsetpgrp
>          U td_symbol_list
>          U td_ta_event_addr
>          U td_ta_event_getmsg
>          U td_ta_new
>          U td_ta_set_event
>          U td_ta_thr_iter
>          U td_thr_event_enable
>          U td_thr_get_info
> 0001f42c D the_low_target
> 00020784 B the_target
> 00014664 T thread_db_init
> 00020690 B thread_from_wait
> ......
> 
> Now I get a different error from gdb:
> 
> on the target:
> 
> # gdbserver :5234 simple
> Process simple created; pid = 41
> Listening on port 5234
> 
> 
> on the host:
> 
> GNU gdb 6.3
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "--host=i686-linux --target=arm-linux".
> (gdb) set solib-absolute-prefix /lib
> (gdb) file simple
> Reading symbols
> from /home/trivas/dm320-appro/fs/examples/threads/simple...done.
> (gdb) target remote 192.168.200.199:5234
> Remote debugging using 192.168.200.199:5234
> 0x40000c60 in ?? ()
> warning: Unable to find dynamic linker breakpoint function.
> GDB will be unable to debug shared library initializers
> and track explicitly loaded dynamic code.
> (gdb) info shared
> No shared libraries loaded at this time.
> (gdb)
> 
> 
> > > (gdb) set solib-absolute-prefix /target/arm-linux/lib
> > 
> > That's probably not right.  The paths below solib-absolute-prefix
> > should match the target filesystem.  So if solib-absolute-prefix
> > is "/target/arm-linux", then the target /lib/ld-linux.so.2 should be in
> > /target/arm-linux/lib/ld-linux.so.2.
> > 
> > > (gdb) set solib-search-path /target/arm-linux/lib
> > 
> > You should not need to set that.
> > 
> > > (gdb) dir /target/arm-linux/lib
> > 
> > Or this.
> > 
> > > Source directories searched: /target/arm-linux/lib:$cdir:$cwd
> > > (gdb) set endian little
> > > The target is assumed to be little endian
> > 
> > Or this.
> > 
> > > (gdb) handle SIG32 nostop noprint pass
> > > Signal        Stop      Print   Pass to program Description
> > > SIG32         No        No      Yes             Real-time event 32
> > 
> > Don't do that; it never helps.
> > 
> > > (gdb) add-symbol-file simple
> > > add symbol table from file "simple" at
> > > (y or n) y
> > > Reading symbols
> > > from /home/trivas/dm320-appro/fs/examples/threads/simple...done.
> > 
> > Don't do this either.  Use "file simple", and do it before issuing the
> > "target remote" command.
> > 
> > > and if I try a info shared it says "No shared libraries loaded at this
> > > time"
> > 
> > Thread debugging will not work until this works.  It should show you
> > the list of libraries and say that they have symbols loaded (at main,
> > not when you first connect).
> > 
-- 
Teresa R. Rivas
teresa.rivas@cadenux.com
Cadenux, LLC

Re: gdb debugging threads help

[Daniel Jacobowitz]

On Fri, Nov 18, 2005 at 08:30:37AM -0600, Teresa Rivas wrote:
> Which command can I use to see if it is linked against the correct
> thread_db library?? I use a nm -D gdbserver but all I see are the
> symbols from the dynamic libraries not the libraries.

"ldd" on your target.

> .....
> 0000eeec T target_signal_to_name
> 0000ee90 T target_signal_to_string
>          U tcgetattr
>          U tcsetattr
>          U tcsetpgrp
>          U td_symbol_list
>          U td_ta_event_addr
>          U td_ta_event_getmsg
>          U td_ta_new
>          U td_ta_set_event
>          U td_ta_thr_iter
>          U td_thr_event_enable
>          U td_thr_get_info
> 0001f42c D the_low_target
> 00020784 B the_target
> 00014664 T thread_db_init
> 00020690 B thread_from_wait

Looks good.

> (gdb) set solib-absolute-prefix /lib
> (gdb) file simple

No, /target/arm-linux.

The _full path_ to libraries will be appended to this prefix:

> > > (gdb) set solib-absolute-prefix /target/arm-linux/lib
> > 
> > That's probably not right.  The paths below solib-absolute-prefix
> > should match the target filesystem.  So if solib-absolute-prefix
> > is "/target/arm-linux", then the target /lib/ld-linux.so.2 should be in
> > /target/arm-linux/lib/ld-linux.so.2.


-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: gdb debugging threads help

[Teresa Rivas]

HI,


On Fri, 2005-11-18 at 10:28 -0500, Daniel Jacobowitz wrote:
> On Fri, Nov 18, 2005 at 08:30:37AM -0600, Teresa Rivas wrote:
> > Which command can I use to see if it is linked against the correct
> > thread_db library?? I use a nm -D gdbserver but all I see are the
> > symbols from the dynamic libraries not the libraries.
> 
> "ldd" on your target.

I don't have ldd on my target so Im gonna trust in the nm -D an also in
the information from the /prco/PID/maps of the gdbserver when running
which says this:

on the target:

# gdbserver :5234 simple
Process simple created; pid = 58
Listening on port 5234

from a telnet window to the target:

 # ps
  PID  Uid     VmSize Stat Command
    1 root        308 S   init
    2 root            SWN [ksoftirqd/0]
    3 root            SW< [events/0]
    4 root            SW< [kblockd/0]
    5 root            SW< [pdflush]
    6 root            SW< [pdflush]
    8 root            SW< [aio/0]
    7 root            SW  [kswapd0]
    9 root            SW  [rpciod]
   35 root        284 S   /usr/sbin/inetd /etc/inetd.conf
   45 root        408 S   -sh
   46 root        324 S   gdbserver :5234 simple
   47 root         16 T   simple
   48 root        352 S   /usr/sbin/telnetd
   49 user        344 S   -sh
   50 root        408 S   sh
   51 root        308 R   ps
# cat /proc/46/maps
00008000-00017000 r-xp 00000000 00:08 3096786    /usr/bin/gdbserver
0001e000-00020000 rw-p 0000e000 00:08 3096786    /usr/bin/gdbserver
00020000-00022000 rwxp 00000000 00:00 0
40000000-40005000 r-xp 00000000 00:08
73959      /lib/ld-uClibc-0.9.26.so
40005000-40006000 rw-p 00000000 00:00 0
4000c000-4000d000 rw-p 00004000 00:08
73959      /lib/ld-uClibc-0.9.26.so
4000d000-40010000 r-xp 00000000 00:08
74075      /lib/libthread_db-0.9.26.so
40010000-40017000 ---p 00003000 00:00 0
40017000-40019000 rw-p 00002000 00:08
74075      /lib/libthread_db-0.9.26.so
40019000-4005f000 r-xp 00000000 00:08
73962      /lib/libuClibc-0.9.26.so
4005f000-40066000 ---p 00046000 00:00 0
40066000-40069000 rw-p 00045000 00:08
73962      /lib/libuClibc-0.9.26.so
40069000-40072000 rw-p 00000000 00:00 0
beffe000-bf000000 rwxp fffff000 00:00 0

I can see it is mapping /lib/libthread_db-0.9.26.so so I guess is linked
correctly against it.

now I use this on the host and I am able to see debug the threads :)

(gdb) set solib-absolute-prefix /target/arm-linux
(gdb) file simple
Load new symbol table from
"/home/trivas/dm320-appro/fs/examples/threads/simple"? (y or n) y
Reading symbols
from /home/trivas/dm320-appro/fs/examples/threads/simple...done.
(gdb) target remote 192.168.200.199:5234
Remote debugging using 192.168.200.199:5234
0x40000c60 in ?? ()
(gdb) b 87
Breakpoint 1 at 0x8834: file simple.c, line 87.
(gdb) b 122
Breakpoint 2 at 0x8a68: file simple.c, line 122.
(gdb) c
Continuing.
[New Thread 1024]
[Switching to Thread 1024]

Breakpoint 1, main (argc=1, argv=0xbefffed4) at simple.c:90
90          for (i=0; i<60000; i++) {
(gdb) info threads
  4 Thread 2051  0x4004a678 in sched_yield ()
from /target/arm-linux/lib/libc.so.0
  3 Thread 1026  0x4004a678 in sched_yield ()
from /target/arm-linux/lib/libc.so.0
  2 Thread 2049  0x4004a8ac in poll ()
from /target/arm-linux/lib/libc.so.0
* 1 Thread 1024  main (argc=1, argv=0xbefffed4) at simple.c:90

thanks a lot for your help ;)


-- 
Teresa R. Rivas
teresa.rivas@cadenux.com
Cadenux, LLC