remote debugging and source files

remote debugging and source files

[Hinko Kocevar]

Hi,

I'm remote debugging a programm using gdbserver on target side and gdb 
on host side. My target CPU is ARM and host is i686. Debugging same 
program just on host (no cross compilaton) allows me to see the source 
of each file when doing 'step'. That does not work when same program is 
cross-compiled, copied to target and started there with:
	gdbserver host:2345 slider

On host I run the debugger- 'ddd --debugger arm-linu-gdb', load the 
program and specify the target:
	target remote ipaq:2345

I can see slider.c source displayed, but any other functions, defined 
outside, are not stepped into. Maybe it is worth mentioning that this 
functions are in a shared library.

I tried adding directory of the actual source with 'dir /path/to/source' 
but it didn't help.

Also, I pointed 'set solib-search-path' to libs containing those functions.

What am I missing here?

regards,
h

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files

[Hinko Kocevar]

Hinko Kocevar wrote:
> Hi,
> 
...

A little more info from me - running info shared produces:

(gdb) info shared
 From        To          Syms Read   Shared Object Library
0x40020068  0x40029b4c  No 
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/usr/local/lib/libnano-X.so
0x4004a8b0  0x40131260  No 
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/lib/libc.so.6
0x40155974  0x4015bec4  No 
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/lib/libgcc_s.so.1
0x40001460  0x40011914  No 
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/lib/ld-linux.so.2
(gdb)

IMHO libraries are found and loaded, but sysbols are not read... Why?

regards,
h

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files

[Hinko Kocevar]

Hinko Kocevar wrote:
...
> 
> IMHO libraries are found and loaded, but sysbols are not read... Why?
> 
> regards,
> h
> 

Ok, I got it to load the symbols manualy with
sharedlibrary 
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/usr/local/lib/libnano-X.so

I wonder if this could be automated. As I stated earlier on host, 
debugging program compiled for host, it loads symbols automaticaly. If 
debugging program for ARM target I have to manualy load the symbols ?!

'Autoloading of shared library symbols' is set to true in ddd/gdb so 
this should do it, right?

regards,
h

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files

[Kris Warkentin]

auto-solib-add will only work properly if gdb successfully sets the 
solib event breakpoint in the loader.  Try 'maint info breakpoints' and 
see if it's being set.  You could also try setting stop-on-solib-events 
to see if it's actually stopping.  Sometimes you have problems if there 
is a different version of the loader on host and target.

cheers,

Kris

Hinko Kocevar wrote:

> Hinko Kocevar wrote:
> ...
>
>>
>> IMHO libraries are found and loaded, but sysbols are not read... Why?
>>
>> regards,
>> h
>>
>
> Ok, I got it to load the symbols manualy with
> sharedlibrary 
> /opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/usr/local/lib/libnano-X.so 
>
>
> I wonder if this could be automated. As I stated earlier on host, 
> debugging program compiled for host, it loads symbols automaticaly. If 
> debugging program for ARM target I have to manualy load the symbols ?!
>
> 'Autoloading of shared library symbols' is set to true in ddd/gdb so 
> this should do it, right?
>
> regards,
> h
>

Re: remote debugging and source files

[Hinko Kocevar]

Kris Warkentin wrote:
> auto-solib-add will only work properly if gdb successfully sets the 
> solib event breakpoint in the loader.  Try 'maint info breakpoints' and 
> see if it's being set.  You could also try setting stop-on-solib-events 
> to see if it's actually stopping.  Sometimes you have problems if there 
> is a different version of the loader on host and target.
> 

It seems that solib events have enabled breakpoint,

(gdb) maintenance info breakpoints
Num Type           Disp Enb Address    What
-2  longjmp resume keep n   0x00000000
-3  shlib events   keep y   0x4000b8d8

but execution won't stop if I do
	set stop-on-solib-events 1


regards,
h

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files

[Daniel Jacobowitz]

On Tue, Aug 24, 2004 at 07:42:57PM +0200, Hinko Kocevar wrote:
> Kris Warkentin wrote:
> >auto-solib-add will only work properly if gdb successfully sets the 
> >solib event breakpoint in the loader.  Try 'maint info breakpoints' and 
> >see if it's being set.  You could also try setting stop-on-solib-events 
> >to see if it's actually stopping.  Sometimes you have problems if there 
> >is a different version of the loader on host and target.
> >
> 
> It seems that solib events have enabled breakpoint,
> 
> (gdb) maintenance info breakpoints
> Num Type           Disp Enb Address    What
> -2  longjmp resume keep n   0x00000000
> -3  shlib events   keep y   0x4000b8d8
> 
> but execution won't stop if I do
> 	set stop-on-solib-events 1

Check whether that breakpoint is in the right place.  Probably it
isn't.  Probably Kris is right that there is a mismatch...

-- 
Daniel Jacobowitz

Re: remote debugging and source files

[Kris Warkentin]

Hmm...according to your info shared, you have the loader here:

0x40001460  0x40011914  No 
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/lib/ld-linux.so.2 


so the breakpoint doesn't seem entirely unreasonable.  Is there any 
possibility that you have a different ld-linux.so.2 on the host and 
target?  Then gdb would be calculating the breakpoint incorrectly based 
on the host loader.  If you take a look at /usr/include/link.h, you'll 
find the r_debug structure.  The comments in there will help you 
calculate the breakpoint by hand (look at the r_brk element) and then 
you can compare it to what gdb is coming up with.

cheers,

Kris

Hinko Kocevar wrote:

> Kris Warkentin wrote:
>
>> auto-solib-add will only work properly if gdb successfully sets the 
>> solib event breakpoint in the loader.  Try 'maint info breakpoints' 
>> and see if it's being set.  You could also try setting 
>> stop-on-solib-events to see if it's actually stopping.  Sometimes you 
>> have problems if there is a different version of the loader on host 
>> and target.
>>
>
> It seems that solib events have enabled breakpoint,
>
> (gdb) maintenance info breakpoints
> Num Type           Disp Enb Address    What
> -2  longjmp resume keep n   0x00000000
> -3  shlib events   keep y   0x4000b8d8
>
> but execution won't stop if I do
>     set stop-on-solib-events 1
>
>
> regards,
> h
>

Re: remote debugging and source files

[Hinko Kocevar]

Kris Warkentin wrote:
> Hmm...according to your info shared, you have the loader here:
> 
> 0x40001460  0x40011914  No 
> /opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/lib/ld-linux.so.2 
> 
> 
> so the breakpoint doesn't seem entirely unreasonable.  Is there any 
> possibility that you have a different ld-linux.so.2 on the host and 
> target? 

It is same version on the host and on the target - glibc-2.3.2., but the 
one on target is cross compiled for ARM architecture, though. Is there 
any other kind of difference we are interested here?

If debugging host application debugger stops on dynamic library load as 
expected and 'info shared' says
(gdb) info sharedlibrary
 From        To          Syms Read   Shared Object Library
0x40034680  0x400aadb4  Yes         /usr/X11R6/lib/libX11.so.6
0x40134b40  0x4022f7b4  Yes         /lib/i686/libc.so.6
0x40252eb0  0x40253de4  Yes         /lib/libdl.so.2
0x40000c00  0x400139ef  Yes         /lib/ld-linux.so.2

> Then gdb would be calculating the breakpoint incorrectly based 
> on the host loader.  If you take a look at /usr/include/link.h, you'll 
> find the r_debug structure.  The comments in there will help you 
> calculate the breakpoint by hand (look at the r_brk element) and then 
> you can compare it to what gdb is coming up with.
> 

Sounds easy, but to me, it is not - at all :).
OK, here goes...
I did 'maintenance info breakpoints' while debugging native app:
...
-16 shlib events   keep y   0x4000dd60 <_dl_debug_state_internal>
         breakpoint already hit 3 times
...

So debugger stoped on shared library event 3 times (three dynamic libs 
were loaded) at <_dl_debug_state_internal> symbol found in ld-2.3.2.so. 
Using objdump I got the same offset, 0x0000dd60, for 
_dl_debug_state_internal.

On ARM target using the same technique, shlib events are at 0x4000b8d8 
while arm-linux-objdump says that _dl_debug_state_internal is at offset 
0x0000c258. After setting breakpoint to 0x0000c258 debugger stopped 7 
times (according to comments around r_debug that is OK) before my app 
was stated.

Offset 0x0000b8d8 is location in _dl_signal_error.

So how do I convince gdb to use correct address for shlib event?


regards,
h

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files

[Daniel Jacobowitz]

On Wed, Aug 25, 2004 at 03:58:20PM +0200, Hinko Kocevar wrote:
> Kris Warkentin wrote:
> >Hmm...according to your info shared, you have the loader here:
> >
> >0x40001460  0x40011914  No 
> >/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu/lib/ld-linux.so.2 
> >
> >
> >so the breakpoint doesn't seem entirely unreasonable.  Is there any 
> >possibility that you have a different ld-linux.so.2 on the host and 
> >target? 
> 
> It is same version on the host and on the target - glibc-2.3.2., but the 
> one on target is cross compiled for ARM architecture, though. Is there 
> any other kind of difference we are interested here?

No, the copy of glibc on the target and the copy in
/opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu.  Are they
identical builds?


-- 
Daniel Jacobowitz

Re: remote debugging and source files

[Hinko Kocevar]

Daniel Jacobowitz wrote:
...
> 
> No, the copy of glibc on the target and the copy in
> /opt/arm-linux/gcc-3.3.3-glibc-2.3.2/arm-softfloat-linux-gnu.  Are they
> identical builds?
> 
> 

Ooo, not they are not. Ahh - I get it now!!

I'm using several different ARM boards and some use different build of 
libs while other use the same as found on the host. I was soooo sure 
that this one has the same libs, that I didn't even bother to look.

This would certainly be the case then, I'll update that board and repost...

regards,
h

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files - SOLVED

[Hinko Kocevar]

Hinko Kocevar wrote:

> 
> This would certainly be the case then, I'll update that board and repost...
> 

As you guys pointed out, it helps having same libs and binaries on both 
target and host side of debugging enviroment.

Thanks for sharing your wisdom.

But, now that symbols get loaded automagically on shared library event, 
other set and enabled breakpoints in debugged program don't stop 
execution?! I already post another topic about this.

Now I'm more confident in what is the reason - its probably libraries 
itself, since debugging worked just fine (except for the symbol loading) 
with the old ones.

Is there any special case with building libraries to be used process of 
debugging?

-- 
hinko <dot> kocevar <at> iskramedical <dot> si
Hinko Kocevar, developer
Iskra Medical d.o.o., Stegne 23, 1k LJ, SLO-EU

	"Aì rén"	|	[Analects XII:22]

Re: remote debugging and source files

[Kris Warkentin]

Hinko Kocevar wrote:

> It is same version on the host and on the target - glibc-2.3.2., but 
> the one on target is cross compiled for ARM architecture, though. Is 
> there any other kind of difference we are interested here?

No, what I mean is that gdb loads the arm binaries from the host.  Those 
binaries need to be the same binaries as the target is running.

>
> Sounds easy, but to me, it is not - at all :).
> OK, here goes...
> I did 'maintenance info breakpoints' while debugging native app:
> ...
> -16 shlib events   keep y   0x4000dd60 <_dl_debug_state_internal>
>         breakpoint already hit 3 times
> ...
>
> So debugger stoped on shared library event 3 times (three dynamic libs 
> were loaded) at <_dl_debug_state_internal> symbol found in 
> ld-2.3.2.so. Using objdump I got the same offset, 0x0000dd60, for 
> _dl_debug_state_internal.
>
> On ARM target using the same technique, shlib events are at 0x4000b8d8 
> while arm-linux-objdump says that _dl_debug_state_internal is at 
> offset 0x0000c258. After setting breakpoint to 0x0000c258 debugger 
> stopped 7 times (according to comments around r_debug that is OK) 
> before my app was stated.
>
> Offset 0x0000b8d8 is location in _dl_signal_error.
>
> So how do I convince gdb to use correct address for shlib event?


At this stage I'm beginning to get confused about which binaries are 
being discussed here since you're comparing the native version with the 
remote.  What I need to verify first is whether or not the ARM binaries 
(libs and app) on the host are EXACTLY the same as the binaries on the 
target.  Then we need to verify that gdb is finding the correct ones.  
If all that is true and it's still not working, then we may need to dig 
deeper into why the solib breakpoint is being calculated incorrectly.  
Can you verify that auto-solib-add is on as well?

cheers,

Kris