Issues debugging remote leon3 system

Issues debugging remote leon3 system

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 2097 bytes --]

Greetings

I have a Spartan6 FPGA in a Xilinx SP605 board configured with a stock
Leon3 SPARC processor. Currently, I am running GNU/Linux on the device
(kernel version 3.14.26, uClibc version 0.9.33.2, BusyBox 1.23.0) and I
am attempting to run gdbserver on the device.

Whenever I try to connect remotely through TCP, I get the following
error on the GDB client:

$ sparc-leon3-linux-gdb hw
GNU gdb 7.8.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu
--target=sparc-leon3-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://hardwaresecurity.org>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hw...done.
(gdb) target remote leon3:1234
Remote debugging using leon3:1234
Remote register badly formatted:
T050e:0000000000000000;1e:00000000efc4d6a0;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37d.37d;core:0;
here:
00000000;1e:00000000efc4d6a0;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37d.37d;core:0;
(gdb)


Attached you can find the the output of gdbserver (debugging enabled,
not placed here). I have done a packet capture of the protocol for an
ARM target and compared it to a capture for the Leon3 target and noticed
that there a few XML files being transmitted on the ARM target which are
not present in the SPARC target. I am not sure if this is necessary.

I do not have any other SPARC units on which I can test this issue and
searching online and the mailing lists has yielded no results. Please
advice. Thank you.

Cheers,
Orlando.

[-- Attachment #1.2: gdbserver.out --]
[-- Type: text/plain, Size: 6353 bytes --]

~ # gdbserver --remote-debug --debug --debug-format=all --once :1234 ./hw 
559:336201 new_argv[0] = "./hw"
Process ./hw created; pid = 906
559:349540 >>>> entering linux_wait_1
559:355678 linux_wait_1: [Process 906]
my_waitpid (-1, 0x40000001)
my_waitpid (-1, 0x80000001): status(0), -1
559:365493 LWFE: waitpid(-1, ...) returned -1, No child processes
559:371413 leader_pid=906, leader_lp!=NULL=1, num_lwps=1, zombie=0
559:373281 sigsuspend'ing
sigchld_handler
my_waitpid (-1, 0x40000001)
my_waitpid (-1, 0x1): status(57f), 906
559:376950 LWFE: waitpid(-1, ...) returned 906, ERRNO-OK
559:378180 LLW: waitpid 906 received Trace/breakpoint trap (stopped)
559:381324 stop pc is 00000000
559:382730 linux_low_filter_event: pc is 0x0
559:383949 stop pc is 00000000
559:385298 pc is 0x0
559:386121 stop pc is 0x0
my_waitpid (907, 0x0)
my_waitpid (907, 0x0): status(117f), 907
my_waitpid (907, 0x0)
my_waitpid (907, 0x0): status(1057f), 907
my_waitpid (908, 0x0)
my_waitpid (908, 0x0): status(117f), 908
my_waitpid (908, 0x0)
my_waitpid (908, 0x0): status(9), 908
my_waitpid (907, 0x0)
my_waitpid (907, 0x0): status(0), 907
sigchld_handler
559:417415 Hit a non-gdbserver trap event.
559:418210 >>>> entering stop_all_lwps
559:419389 stop_all_lwps (stop, except=none)
559:420459 wait_for_sigstop: pulling events
my_waitpid (-1, 0x40000001)
my_waitpid (-1, 0x80000001): status(0), -1
559:423532 LWFE: waitpid(-1, ...) returned -1, No child processes
559:428196 leader_pid=906, leader_lp!=NULL=1, num_lwps=1, zombie=0
559:429807 LLW: exit (no unwaited-for LWP)
559:430570 stop_all_lwps done, setting stopping_threads back to !stopping
559:431540 <<<< exiting stop_all_lwps
559:432560 Checking whether LWP 906 needs to move out of the jump pad...no
559:433695 linux_wait_1 ret = LWP 906.906, 1, 5
559:435130 <<<< exiting linux_wait_1
Listening on port 1234
574:261967 handling possible accept event
Remote debugging from host 192.168.0.254
574:264846 linux_async (0), previous=0
574:266390 handling possible serial event
[getpkt: discarding char '+']
getpkt ("qSupported:multiprocess+;qRelocInsn+");  [sending ack] 
[sent ack]
putpkt ("$PacketSize=3fff;QPassSignals+;QProgramSignals+;qXfer:libraries-svr4:read+;augmented-libraries-svr4-read+;qXfer:auxv:read+;qXfer:spu:read+;qXfer:spu:write+;qXfer:siginfo:read+;qXfer:siginfo:write+;qXfer:features:read+;QStartNoAckMode+;qXfer:osdata:read+;multiprocess+;QNonStop+;QDisableRandomization+;qXfer:threads:read+;ConditionalBreakpoints+;BreakpointCommands+;QAgent+#4f"); [looking for ack]
[received '+' (0x2b)]
574:274834 handling possible serial event
getpkt ("QStartNoAckMode");  [sending ack] 
[sent ack]
[noack mode enabled]
putpkt ("$OK#9a"); [noack mode]
574:279631 handling possible serial event
[getpkt: discarding char '+']
getpkt ("QProgramSignals:0;1;3;4;6;7;8;9;a;b;c;d;e;f;10;11;12;13;14;15;16;17;18;19;1a;1b;1c;1d;1e;1f;20;21;22;23;24;25;26;27;28;29;2a;2b;2c;2d;2e;2f;30;31;32;33;34;35;36;37;38;39;3a;3b;3c;3d;3e;3f;40;41;42;43;44;45;46;47;48;49;4a;4b;4c;4d;4e;4f;50;51;52;53;54;55;56;57;58;59;5a;5b;5c;5d;5e;5f;60;61;62;63;64;65;66;67;68;69;6a;6b;6c;6d;6e;6f;70;71;72;73;74;75;76;77;78;79;7a;7b;7c;7d;7e;7f;80;81;82;83;84;85;86;87;88;89;8a;8b;8c;8d;8e;8f;90;91;92;93;94;95;96;");  [no ack sent] 
putpkt ("$OK#9a"); [noack mode]
574:285069 handling possible serial event
getpkt ("Hgp0.0");  [no ack sent] 
putpkt ("$OK#9a"); [noack mode]
574:288535 handling possible serial event
getpkt ("qXfer:features:read:target.xml:0,fff");  [no ack sent] 
putpkt ("$E01#a6"); [noack mode]
574:292454 handling possible serial event
getpkt ("qXfer:auxv:read::0,1000");  [no ack sent] 
putpkt ("$l"); [noack mode]
574:297364 handling possible serial event
getpkt ("QNonStop:0");  [no ack sent] 
574:298783 linux_async (0), previous=0
[all-stop mode enabled]
putpkt ("$OK#9a"); [noack mode]
574:302452 handling possible serial event
getpkt ("qTStatus");  [no ack sent] 
putpkt ("$#00"); [noack mode]
574:305949 handling possible serial event
getpkt ("?");  [no ack sent] 
574:307499 >>>> entering stop_all_lwps
574:308396 stop_all_lwps (stop, except=none)
574:309445 wait_for_sigstop: pulling events
my_waitpid (-1, 0x40000001)
my_waitpid (-1, 0x80000001): status(3f), -1
574:312342 Lrandom: nonblocking pool is initialized
WFE: waitpid(-1, ...) returned -1, No child processes
574:316472 leader_pid=906, leader_lp!=NULL=1, num_lwps=1, zombie=0
574:318093 LLW: exit (no unwaited-for LWP)
574:318834 stop_all_lwps done, setting stopping_threads back to !stopping
574:319798 <<<< exiting stop_all_lwps
574:320672 Checking whether LWP 906 needs to move out of the jump pad...no
574:321849 Writing resume reply for LWP 906.906:1
putpkt ("$T050e:6172795f63747874;1e:0*"08eff5a730;50:0*,;51:0*,;0f:5f73776974636865;thread:p38a.38a;core:0;#68"); [noack mode]
574:328334 handling possible serial event
getpkt ("qXfer:threads:read::0,fff");  [no ack sent] 
putpkt ("$l<threads>
<thread id="p38a.38a" core="0"/>
</threads>
#c2"); [noack mode]
574:334761 handling possible serial event
getpkt ("qAttached:38a");  [no ack sent] 
putpkt ("$0#30"); [noack mode]
574:338374 handling possible serial event
getpkt ("Hc-1");  [no ack sent] 
putpkt ("$E01#a6"); [noack mode]
574:341823 handling possible serial event
getpkt ("qOffsets");  [no ack sent] 
putpkt ("$#00"); [noack mode]
574:345427 handling possible serial event
readchar: Got EOF
[getpkt: discarding char '�']
Killing process(es): 906
574:349066 >>>> entering stop_all_lwps
574:350020 stop_all_lwps (stop, except=none)
574:351380 wait_for_sigstop: pulling events
my_waitpid (-1, 0x40000001)
my_waitpid (-1, 0x80000001): status(0), -1
574:354045 LWFE: waitpid(-1, ...) returned -1, No child processes
574:358146 leader_pid=906, leader_lp!=NULL=1, num_lwps=1, zombie=0
574:360059 LLW: exit (no unwaited-for LWP)
574:360813 stop_all_lwps done, setting stopping_threads back to !stopping
574:361512 <<<< exiting stop_all_lwps
574:362511 lkop: is last of process LWP 906.906
574:363623 kwl: killing lwp 906, for pid: 906
sigchld_handler
574:367175 LKL:  kill (SIGKILL) LWP 906.906, 0, 0 (OK)
574:368481 LKL:  PTRACE_KILL LWP 906.906, 0, 0 (OK)
my_waitpid (906, 0x0)
my_waitpid (906, 0x0): status(9), 906
574:372275 deleting 906
574:373353 >>>> entering unstop_all_lwps
574:374265 unstopping all lwps
574:375244 unstop_all_lwps done
574:375945 <<<< exiting unstop_all_lwps
~ # 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Issues debugging remote leon3 system

[Orlando Arias]

[-- Attachment #1: Type: text/plain, Size: 2988 bytes --]

Greetings,

In hope anybody is seeing this, I have updated to GDB 7.8.2 and the
issue is still persistent. I am configuring gdb in its own build
directory with

mkdir gdb-build && cd gdb-build
../configure --prefix=/usr \
	--target=sparc-leon3-linux \
	--host=x86_64-unknown-linux-gnu \
	--build=x86_64-unknonw-linux-gnu \
	--with-sysroot=/usr/sparc-leon3-linux \
	--without-guile \
	--disable-nls \
	--with-python=/usr/bin/python2 \
	--with-system-readline

and gdbserver as a separate package with
../configure --prefix=/usr \
	--build=x86_64-unknown-linux-gnu \
	--host=sparc-leon3-linux \
	--program-prefix="" \
	--disable-nls

Please let me know if I am missing something. Thank you.

Cheers,
Orlando.



On 01/09/2015 11:23 AM, Orlando Arias wrote:
> Greetings
> 
> I have a Spartan6 FPGA in a Xilinx SP605 board configured with a stock
> Leon3 SPARC processor. Currently, I am running GNU/Linux on the device
> (kernel version 3.14.26, uClibc version 0.9.33.2, BusyBox 1.23.0) and I
> am attempting to run gdbserver on the device.
> 
> Whenever I try to connect remotely through TCP, I get the following
> error on the GDB client:
> 
> $ sparc-leon3-linux-gdb hw
> GNU gdb 7.8.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "--host=x86_64-unknown-linux-gnu
> --target=sparc-leon3-linux".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://hardwaresecurity.org>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from hw...done.
> (gdb) target remote leon3:1234
> Remote debugging using leon3:1234
> Remote register badly formatted:
> T050e:0000000000000000;1e:00000000efc4d6a0;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37d.37d;core:0;
> here:
> 00000000;1e:00000000efc4d6a0;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37d.37d;core:0;
> (gdb)
> 
> 
> Attached you can find the the output of gdbserver (debugging enabled,
> not placed here). I have done a packet capture of the protocol for an
> ARM target and compared it to a capture for the Leon3 target and noticed
> that there a few XML files being transmitted on the ARM target which are
> not present in the SPARC target. I am not sure if this is necessary.
> 
> I do not have any other SPARC units on which I can test this issue and
> searching online and the mailing lists has yielded no results. Please
> advice. Thank you.
> 
> Cheers,
> Orlando.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Issues debugging remote leon3 system

[Orlando Arias]

[-- Attachment #1: Type: text/plain, Size: 3732 bytes --]

Greetings,

Further providing information, I have found the following whilst
debugging the issue:

At some point, gdbserver sends the following string:
$T050e:0*,;1e:0*"00ef90f6a0;50:0*,;51:0*,;0f:0*,;thread:p37b.37b;core:0;#54

and gdb attempts to process:
T050e:0000000000000000;1e:0008a6a000080e14;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37e.37e;core:0;

At the moment, I am not sure as to what I am looking at. I will continue
working on finding the root cause. If anybody has any ideas, please let
me know. Thank you.

Cheers,
Orlando.

On 01/15/2015 03:41 PM, Orlando Arias wrote:
> Greetings,
> 
> In hope anybody is seeing this, I have updated to GDB 7.8.2 and the
> issue is still persistent. I am configuring gdb in its own build
> directory with
> 
> mkdir gdb-build && cd gdb-build
> ../configure --prefix=/usr \
> 	--target=sparc-leon3-linux \
> 	--host=x86_64-unknown-linux-gnu \
> 	--build=x86_64-unknonw-linux-gnu \
> 	--with-sysroot=/usr/sparc-leon3-linux \
> 	--without-guile \
> 	--disable-nls \
> 	--with-python=/usr/bin/python2 \
> 	--with-system-readline
> 
> and gdbserver as a separate package with
> ../configure --prefix=/usr \
> 	--build=x86_64-unknown-linux-gnu \
> 	--host=sparc-leon3-linux \
> 	--program-prefix="" \
> 	--disable-nls
> 
> Please let me know if I am missing something. Thank you.
> 
> Cheers,
> Orlando.
> 
> 
> 
> On 01/09/2015 11:23 AM, Orlando Arias wrote:
>> Greetings
>>
>> I have a Spartan6 FPGA in a Xilinx SP605 board configured with a stock
>> Leon3 SPARC processor. Currently, I am running GNU/Linux on the device
>> (kernel version 3.14.26, uClibc version 0.9.33.2, BusyBox 1.23.0) and I
>> am attempting to run gdbserver on the device.
>>
>> Whenever I try to connect remotely through TCP, I get the following
>> error on the GDB client:
>>
>> $ sparc-leon3-linux-gdb hw
>> GNU gdb 7.8.1
>> Copyright (C) 2014 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "--host=x86_64-unknown-linux-gnu
>> --target=sparc-leon3-linux".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <https://hardwaresecurity.org>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from hw...done.
>> (gdb) target remote leon3:1234
>> Remote debugging using leon3:1234
>> Remote register badly formatted:
>> T050e:0000000000000000;1e:00000000efc4d6a0;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37d.37d;core:0;
>> here:
>> 00000000;1e:00000000efc4d6a0;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37d.37d;core:0;
>> (gdb)
>>
>>
>> Attached you can find the the output of gdbserver (debugging enabled,
>> not placed here). I have done a packet capture of the protocol for an
>> ARM target and compared it to a capture for the Leon3 target and noticed
>> that there a few XML files being transmitted on the ARM target which are
>> not present in the SPARC target. I am not sure if this is necessary.
>>
>> I do not have any other SPARC units on which I can test this issue and
>> searching online and the mailing lists has yielded no results. Please
>> advice. Thank you.
>>
>> Cheers,
>> Orlando.
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Issues debugging remote leon3 system

[Sterling Augustine]

On Fri, Jan 16, 2015 at 7:46 AM, Orlando Arias <oarias@knights.ucf.edu> wrote:
> Greetings,
>
> Further providing information, I have found the following whilst
> debugging the issue:
>
> At some point, gdbserver sends the following string:
> $T050e:0*,;1e:0*"00ef90f6a0;50:0*,;51:0*,;0f:0*,;thread:p37b.37b;core:0;#54
>
> and gdb attempts to process:
> T050e:0000000000000000;1e:0008a6a000080e14;50:0000000000000000;51:0000000000000000;0f:0000000000000000;thread:p37e.37e;core:0;
>
> At the moment, I am not sure as to what I am looking at. I will continue
> working on finding the root cause. If anybody has any ideas, please let
> me know. Thank you.
>
> Cheers,
> Orlando.

Since no one is giving you anything to work with.

I know nothing about this processor or your architecture, or how it
interacts with gdb, but sometimes this type of problem occurs when gdb
and gdbserver disagree on the number of registers or their size of
registers at the target.

There can be many different sources of the problem, from mismatched
gdb versions to poorly written gdb stubs, to miscompiled programs.

Don't know if that will help, but that is where I would look.