Insight + ARM-9 + BDI2000: Hang on exec

Insight + ARM-9 + BDI2000: Hang on exec

[Toralf Lund]

I'm using insight built for ARM along with the BDI2000 (BDIGDB) debug 
"POD" to debug a custom interface card based on the MC9328 (ARM-9 core) 
processor.

Now, I can download the code, access memory, display registers etc. just 
fine. I was even able to start the application, stop at breakpoints, 
steptrace etc. for a while, but now when I hit "run", insight will hang 
completely. The window is  still displayed, but none of the buttons or 
menus, or the sliders, respond to input, and the "run" icon doesn't even 
change to "stop". The only way out is to kill the app. gdb from the same 
build appears to work fine. As is often the case when those things 
happen, I'm unable to tell what has changed since everything worked. 
I've tried removing .gdbtkinit & reconfigured target settings to make 
sure the setup was clean, but this didn't help.

Any ideas what may be wrong?

Insight version is 6.0. Host OS is Red Hat Linux 9

- Toralf

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Keith Seitz]

On Mon, 2004-04-19 at 05:55, Toralf Lund wrote:
> I'm using insight built for ARM along with the BDI2000 (BDIGDB) debug 
> "POD" to debug a custom interface card based on the MC9328 (ARM-9 core) 
> processor.

First things first: does it run under console gdb? ("insight -nw" or
"gdb")?

There are numerous reasons for failures of this kind. Typically, it is
an error in the target code in gdb. Testing command-line gdb is the
first step to ascertaining where the problem lies.

Also, could you send a more complete, step-by-step description of what
you did? From your post, I'm not entirely sure how you got it to hang.

Keith

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Toralf Lund]

Keith Seitz wrote:

>On Mon, 2004-04-19 at 05:55, Toralf Lund wrote:
>  
>
>>I'm using insight built for ARM along with the BDI2000 (BDIGDB) debug 
>>"POD" to debug a custom interface card based on the MC9328 (ARM-9 core) 
>>processor.
>>    
>>
>
>First things first: does it run under console gdb? ("insight -nw" or
>"gdb")?
>  
>
Yes, it runs under gdb (I thought I said that). Well, actually, the code 
crashes after a while, but I can at least set some breakpoints, have it 
stop there, step the code etc.

One problem, though: I sometimes get

<host>:2001: Connection refused.

for no apparent reason, when issuing a "target" command. If I try the 
same operation again, it usually works fine.

Also, even insight was OK for a while, then something (related to the 
application - insight hasn't changed) happened that made it stop working 
- and I can't figure out what it is ;-/

>There are numerous reasons for failures of this kind. Typically, it is
>an error in the target code in gdb. Testing command-line gdb is the
>first step to ascertaining where the problem lies.
>
>Also, could you send a more complete, step-by-step description of what
>you did? From your post, I'm not entirely sure how you got it to hang.
>  
>
I got it to hang simply by starting the code (which is why I didn't 
think it necessary to give any steps), i.e.

   1. arm-coff-insight <application>
   2. Run->Download
   3. Run->Run

After I wrote the original message, I discovered that there are other 
ways, like

   1. arm-coff-insight <application>
   2. Run->Download
   3. File->Exit

(But Exit does work if I do File->Disconnect first.)

Furthermore,

   1. arm-coff-insight <application>
   2. Run->Download
   3. Control->Next Asm Inst (since execution start with start() written
      in assembler.
   4. Other "step" commands...

Actually works, but if I try e.g. View->Registers, I get another hang, 
but of a slightly different type; this time I get a "stop" icon and 
"busy" cursor, but I'm not really allowed to stop the process.

 

>Keith
>
>  
>

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Keith Seitz]

I think you're running into multiple problems...

On Tue, 2004-04-20 at 01:12, Toralf Lund wrote:
> Yes, it runs under gdb (I thought I said that). Well, actually, the code 
> crashes after a while, but I can at least set some breakpoints, have it 
> stop there, step the code etc.

Ok, that's a first good step.

> One problem, though: I sometimes get
> <host>:2001: Connection refused.
> for no apparent reason, when issuing a "target" command. If I try the 
> same operation again, it usually works fine.

My guess would be that since you're reusing the same port on the target,
you're hitting the 2MSL wait state.

> Also, even insight was OK for a while, then something (related to the 
> application - insight hasn't changed) happened that made it stop working 
> - and I can't figure out what it is ;-/

:-(

> I got it to hang simply by starting the code (which is why I didn't 
> think it necessary to give any steps), i.e.
> 
>    1. arm-coff-insight <application>
>    2. Run->Download
>    3. Run->Run

[Note: You can just use the Run button to start your application. The
Target Settings dialog defines the default behavior of the run button
for various targets. For Remote targets, it connects to the target,
downloads and continues execution.]

So, when you do this, the UI just "hangs"? Control buttons disabled
(grayed-out)? If so, open a console window and type "tk
gdb_target_has_execution". What's the return value? Also try "tk set
::gdb_running".

> After I wrote the original message, I discovered that there are other 
> ways, like
> 
>    1. arm-coff-insight <application>
>    2. Run->Download
>    3. File->Exit
>
> (But Exit does work if I do File->Disconnect first.)

This is reminiscent of a long-standing gdb bug. But I could be wrong.
The only way to know for sure is to step through the code and find out
why it is hanging. Having the console window open might help, too.

> Furthermore,
> 
>    1. arm-coff-insight <application>
>    2. Run->Download
>    3. Control->Next Asm Inst (since execution start with start() written
>       in assembler.
>    4. Other "step" commands...
> 
> Actually works, but if I try e.g. View->Registers, I get another hang, 
> but of a slightly different type; this time I get a "stop" icon and 
> "busy" cursor, but I'm not really allowed to stop the process.

I think that there is something wrong with register display in that
version. Gdb has changed a whole lot since I last did any active work on
Insight -- which means that no one else has active worked on it. As I
recall, there have been numerous changes to register and memory
handling, but let's deal with one issue at a time.

Keith

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Toralf Lund]

Keith Seitz wrote:

>I think you're running into multiple problems...
>
>On Tue, 2004-04-20 at 01:12, Toralf Lund wrote:
>  
>
>>Yes, it runs under gdb (I thought I said that). Well, actually, the code 
>>crashes after a while, but I can at least set some breakpoints, have it 
>>stop there, step the code etc.
>>    
>>
>
>Ok, that's a first good step.
>
>  
>
>>One problem, though: I sometimes get
>><host>:2001: Connection refused.
>>for no apparent reason, when issuing a "target" command. If I try the 
>>same operation again, it usually works fine.
>>    
>>
>
>My guess would be that since you're reusing the same port on the target,
>you're hitting the 2MSL wait state.
>  
>
Probably something like that.

According to Abatron (the company that makes the BDI2000) support, you 
can't expect re-connect to work due to timing issues. (They blame gdb 
for this, not their own unit...)

>  
>
>>Also, even insight was OK for a while, then something (related to the 
>>application - insight hasn't changed) happened that made it stop working 
>>- and I can't figure out what it is ;-/
>>    
>>
>
>:-(
>
>  
>
>>I got it to hang simply by starting the code (which is why I didn't 
>>think it necessary to give any steps), i.e.
>>
>>   1. arm-coff-insight <application>
>>   2. Run->Download
>>   3. Run->Run
>>    
>>
>
>[Note: You can just use the Run button to start your application. The
>Target Settings dialog defines the default behavior of the run button
>for various targets. For Remote targets, it connects to the target,
>downloads and continues execution.]
>  
>
Yes. I figured that out eventually.

If I use Remote and select "run" without download first, the hang does 
not occur directly. I'll get it a a later stage, though.

>So, when you do this, the UI just "hangs"? Control buttons disabled
>(grayed-out)?
>
No. Nothing changes; the UI just stops responding to input. I can't 
select icons, open menus, scroll the source view, or even close the window.

> If so, open a console window and type "tk
>gdb_target_has_execution". What's the return value? Also try "tk set
>::gdb_running".
>  
>
No can do. I can't open the console after the hang occurs, and if I have 
one open before it does, that stops responding to input, too

>>After I wrote the original message, I discovered that there are other 
>>ways, like
>>
>>   1. arm-coff-insight <application>
>>   2. Run->Download
>>   3. File->Exit
>>
>>(But Exit does work if I do File->Disconnect first.)
>>    
>>
>
>This is reminiscent of a long-standing gdb bug. But I could be wrong.
>The only way to know for sure is to step through the code and find out
>why it is hanging. Having the console window open might help, too.
>  
>
The console window doesn't tell my anything, either...

>>Furthermore,
>>
>>   1. arm-coff-insight <application>
>>   2. Run->Download
>>   3. Control->Next Asm Inst (since execution start with start() written
>>      in assembler.
>>   4. Other "step" commands...
>>
>>Actually works, but if I try e.g. View->Registers, I get another hang, 
>>but of a slightly different type; this time I get a "stop" icon and 
>>"busy" cursor, but I'm not really allowed to stop the process.
>>    
>>
>
>I think that there is something wrong with register display in that
>version. Gdb has changed a whole lot since I last did any active work on
>Insight -- which means that no one else has active worked on it. As I
>recall, there have been numerous changes to register and memory
>handling, but let's deal with one issue at a time.
>  
>
OK

>Keith
>
>
>  
>

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Toralf Lund]

Toralf Lund wrote:

> Keith Seitz wrote:
>
>> I think you're running into multiple problems...
>>
>> On Tue, 2004-04-20 at 01:12, Toralf Lund wrote:
>

Hello again,

I just found an even simpler way to get reproduce the hang. Simply

   1. Start insight
   2. Run->Connect to target

is enough. I also get the hang if I disconnect first.

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Keith Seitz]

On Wed, 2004-04-21 at 02:04, Toralf Lund wrote:
> >So, when you do this, the UI just "hangs"? Control buttons disabled
> >(grayed-out)?
> >
> No. Nothing changes; the UI just stops responding to input. I can't 
> select icons, open menus, scroll the source view, or even close the window.

Ok, this screams of a malignant back-end or similar problem. If the UI
doesn't even do screen updates, then I can pretty much guarantee that
the back-end code for your target is locking the UI out (blocked in a
system call). This happens rather more frequently than I wished. I get
reports of this every few months. [However, you're using a standard
"remote" target, and this should not happen, so something else must be
amiss.]

The problem is that gdb has several event loops. Insight further burdens
the debugging model by adding yet another one. Let me quickly clarify.

The two main event loops in gdb are the target event loop (where the
target code is waiting for debug messages from the target) and the UI
(or mi or cli) event loop, where the user interface is waiting for input
from the user.

Unfortunately, gdb and insight don't share the same event loops for
these things, and they probably never will until all target event loops
go async. In the meantime, we came up with a hack: have the target code
periodically call out to the UI to allow it to run it's event loop. This
is the evil "ui_loop_hook" that is in gdb.

For example, one of the little nasty things that is done (for which I am
regrettably responsible many years ago) involves breaking up serial and
network read requests into small chunks. Instead of waiting N seconds
for a reply from the target, gdb will wait 1 second, N times. After
every second, it calls the ui_loop_hook to update the UI (in this case,
only insight needs to register a hook callback).

So, now for your problems, the question is: why isn't the ui_loop_hook
being called? Better yet, why isn't the connect command completing and
returning control to gdb-proper?

To answer these questions may not be too difficult with a little
diligence on your part.

So, here are some questions/comments:
1) Let's see if we cannot get gdb/Insight to timeout after the connect
and return control to the UI. Before connecting to the target, can you
open a console and enter the command "set remote timeout 1". Now try
connecting. Does it ever return control to you, i.e., does the UI update
at all?

2) I believe you said you were using a remote tcp target, right? Hmm.
Looking over src/gdb/ser-tcp.c, I see calls to ui_loop_hook. Well, to
check this, we have no choice: you're going to have to load insight into
gdb and set a breakpoint on "x_event" in gdbtk-hooks.c. When you attach
to the target, does it ever stop at the breakpoint and run the hook?

3) The easy way: Run insight on gdb. Connect to your target (where the
hang occurs), and interrupt insight and get a backtrace of where it
stopped. Demonstrate anything? Blocked in a system call?

4) The hard way: If all else fails, you're going to have to set a break
in remote_open in remote.c and follow it through to the protocol level
(for tcp, that's in ser-tcp.c). Does it do the N 1-second timeouts,
calling ui loop hook? Or is it blocked?

5) The really hard way: Use strace to find out what's going on.

6) Can you remind me what version of gdb/insight you're using? (Show
complete output of "insight -v".)

Keith

Re: Insight + ARM-9 + BDI2000: Hang on exec

[Toralf Lund]

Keith Seitz wrote:

>On Wed, 2004-04-21 at 02:04, Toralf Lund wrote:
>  
>
>>>So, when you do this, the UI just "hangs"? Control buttons disabled
>>>(grayed-out)?
>>>
>>>      
>>>
>>No. Nothing changes; the UI just stops responding to input. I can't 
>>select icons, open menus, scroll the source view, or even close the window.
>>    
>>
I suddenly discovered what's been going on now - just as I was trying to 
debug the debugger ;-) It was something quite different from what we 
expected, and I should have noticed it earlier, but I just didn't 
consider the possibility, perhaps because it was too simple...

Anyhow, the situation was simply that at the point of the "hang", 
insight would display a modal dialog window of some kind. On connect it 
would contain the message "Successfully connected", on "Run" it asked 
whether I wanted to restart the program, while View->Registers lead to 
an "internal error" popup (I'll investigate that later.)

The problem was that I never could see these popups - not because I'm 
blind or something, but because they would always be opened in the 
"minimized" (iconic) state - so instead of the popup, I just got a 
tasklist entry that I never noticed. (Hmmm... Maybe I should run the 
"window list" applet after all, rather than just using the task pulldown 
of the menu panel. I otherwise find it quite useless when I have more 
than about 3 windows open, and I always do...)

I still haven't figured out why this happened, and if it was insight 
itself or the window manager that caused it, but everything is OK now, 
after I played around a bit with minimize and unminize (or whatever you 
call it) of the popups and the main window.

Sorry for the trouble...

- Toralf

>
>Ok, this screams of a malignant back-end or similar problem. If the UI
>doesn't even do screen updates, then I can pretty much guarantee that
>the back-end code for your target is locking the UI out (blocked in a
>system call). This happens rather more frequently than I wished. I get
>reports of this every few months. [However, you're using a standard
>"remote" target, and this should not happen, so something else must be
>amiss.]
>
>The problem is that gdb has several event loops. Insight further burdens
>the debugging model by adding yet another one. Let me quickly clarify.
>
>The two main event loops in gdb are the target event loop (where the
>target code is waiting for debug messages from the target) and the UI
>(or mi or cli) event loop, where the user interface is waiting for input
>from the user.
>
>Unfortunately, gdb and insight don't share the same event loops for
>these things, and they probably never will until all target event loops
>go async. In the meantime, we came up with a hack: have the target code
>periodically call out to the UI to allow it to run it's event loop. This
>is the evil "ui_loop_hook" that is in gdb.
>
>For example, one of the little nasty things that is done (for which I am
>regrettably responsible many years ago) involves breaking up serial and
>network read requests into small chunks. Instead of waiting N seconds
>for a reply from the target, gdb will wait 1 second, N times. After
>every second, it calls the ui_loop_hook to update the UI (in this case,
>only insight needs to register a hook callback).
>
>So, now for your problems, the question is: why isn't the ui_loop_hook
>being called? Better yet, why isn't the connect command completing and
>returning control to gdb-proper?
>
>To answer these questions may not be too difficult with a little
>diligence on your part.
>
>So, here are some questions/comments:
>1) Let's see if we cannot get gdb/Insight to timeout after the connect
>and return control to the UI. Before connecting to the target, can you
>open a console and enter the command "set remote timeout 1". Now try
>connecting. Does it ever return control to you, i.e., does the UI update
>at all?
>
>2) I believe you said you were using a remote tcp target, right? Hmm.
>Looking over src/gdb/ser-tcp.c, I see calls to ui_loop_hook. Well, to
>check this, we have no choice: you're going to have to load insight into
>gdb and set a breakpoint on "x_event" in gdbtk-hooks.c. When you attach
>to the target, does it ever stop at the breakpoint and run the hook?
>
>3) The easy way: Run insight on gdb. Connect to your target (where the
>hang occurs), and interrupt insight and get a backtrace of where it
>stopped. Demonstrate anything? Blocked in a system call?
>
>4) The hard way: If all else fails, you're going to have to set a break
>in remote_open in remote.c and follow it through to the protocol level
>(for tcp, that's in ser-tcp.c). Does it do the N 1-second timeouts,
>calling ui loop hook? Or is it blocked?
>
>5) The really hard way: Use strace to find out what's going on.
>
>6) Can you remind me what version of gdb/insight you're using? (Show
>complete output of "insight -v".)
>
>Keith
>
>  
>


-- 
Toralf Lund <toralf@procaptura.com> +47 66 85 51 22
ProCaptura AS                       +47 66 85 51 00 (switchboard)
http://www.procaptura.com/~toralf   +47 66 85 51 01 (fax)