Is multiprocessor debugging multithreaded debugging?

Is multiprocessor debugging multithreaded debugging?

[Anupama Chandwani]

Hello, i would like gdb to remote debug a homogeneous multiprocessor
chip (with no OS) from my linux box. I want to use only a single
session of gdb. How different is it from debugging multi-threaded
programs? i.e Can each processor be treated as a thread? However there
are different code images on each processor..
regards,
anupama

Re: Is multiprocessor debugging multithreaded debugging?

[Daniel Jacobowitz]

On Sat, Sep 24, 2005 at 12:26:00PM +0530, Anupama Chandwani wrote:
> Hello, i would like gdb to remote debug a homogeneous multiprocessor
> chip (with no OS) from my linux box. I want to use only a single
> session of gdb. How different is it from debugging multi-threaded
> programs? i.e Can each processor be treated as a thread? However there
> are different code images on each processor..

GDB doesn't support debugging multiple code images at one time, only
single-program-single-data.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: Is multiprocessor debugging multithreaded debugging?

[Aaron S. Kurland]

On Sat, 2005-09-24 at 11:14 -0400, Daniel Jacobowitz wrote:
> On Sat, Sep 24, 2005 at 12:26:00PM +0530, Anupama Chandwani wrote:
> > Hello, i would like gdb to remote debug a homogeneous multiprocessor
> > chip (with no OS) from my linux box. I want to use only a single
> > session of gdb. How different is it from debugging multi-threaded
> > programs? i.e Can each processor be treated as a thread? However there
> > are different code images on each processor..
> 
> GDB doesn't support debugging multiple code images at one time, only
> single-program-single-data.

Will it ever? Is this something that is planned? I'd be willing to
dedicate a full time resource to such a project if anyone is interested
in running it.

--
Aaron S. Kurland
Founder and CTO, Boston Circuits, Inc.
http://www.bostoncircuits.com/

Re: Is multiprocessor debugging multithreaded debugging?

[Daniel Jacobowitz]

On Sat, Sep 24, 2005 at 01:24:05PM -0400, Aaron S. Kurland wrote:
> On Sat, 2005-09-24 at 11:14 -0400, Daniel Jacobowitz wrote:
> > On Sat, Sep 24, 2005 at 12:26:00PM +0530, Anupama Chandwani wrote:
> > > Hello, i would like gdb to remote debug a homogeneous multiprocessor
> > > chip (with no OS) from my linux box. I want to use only a single
> > > session of gdb. How different is it from debugging multi-threaded
> > > programs? i.e Can each processor be treated as a thread? However there
> > > are different code images on each processor..
> > 
> > GDB doesn't support debugging multiple code images at one time, only
> > single-program-single-data.
> 
> Will it ever? Is this something that is planned? I'd be willing to
> dedicate a full time resource to such a project if anyone is interested
> in running it.

Standard answer: it will if someone is interested in implementing and
maintaining it.  In this case, that mostly means thinking about the
user interface and MI interface issues involved; how to present this to
the user and to IDEs.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: Is multiprocessor debugging multithreaded debugging?

[Anupama Chandwani]

In continuation with my prev mail..
I want to extend gdb to debug homing ogenous multiprocessor system
(say multiple ARM or x86 processors on single chip) by remote
debugging in a single session of gdb.

What i want to know is are there enough applications being written on
such multi processors? Also are there different executables being
required to be debugged simultaneously? Coz this is what i want to
extend in further.. Each processor running a different executable so
the processors dont share memory & run with different images of code.

An application of such debugger could be while building an OS but that
wouldnt involve different executables.. So are there applications
requiring to run different executables on each processor? Say for
example a prog gives a certain bug on when there is certain other
program running on the other processor or something similar to
this....

As far as i know this done by multiplexing the JTAG interface (for
x86) &different sessions of gdb right now. Any other? And any flaws or
inconvenience with present methods?

Regards,
anupama

Re: Is multiprocessor debugging multithreaded debugging?

[Steven Johnson]

Anupama Chandwani wrote:

>In continuation with my prev mail..
>I want to extend gdb to debug homing ogenous multiprocessor system
>  
>
>(say multiple ARM or x86 processors on single chip) by remote
>debugging in a single session of gdb.
>
>What i want to know is are there enough applications being written on
>such multi processors? Also are there different executables being
>required to be debugged simultaneously? Coz this is what i want to
>extend in further.. Each processor running a different executable so
>the processors dont share memory & run with different images of code.
>  
>
This is commonly called "Asynchronous" Multi Processing.

>An application of such debugger could be while building an OS but that
>wouldnt involve different executables.. So are there applications
>requiring to run different executables on each processor? Say for
>example a prog gives a certain bug on when there is certain other
>program running on the other processor or something similar to
>this....
>  
>
Yes in the embedded world, there are many examples of Asynchronous Multi 
Processor designs.  They are by far the easiest multi processor design 
to implement.  I for example have worked on a board that had 3 MSP430's, 
each had a unique function, and they intercommunicated over a custom 
parallel bus to coordinate their activities.  Worked sweet, had high 
performance, and was really cheap.

>As far as i know this done by multiplexing the JTAG interface (for
>x86) &different sessions of gdb right now. Any other? And any flaws or
>inconvenience with present methods?
>  
>
This is exactly how it is done, multiple sessions of GDB.  This, in my 
opinion is the right way to go.  Not all Asynchronous multi processor 
designs have homogeneous pprocessors (ie, you may have an MPC860 
handling comms, and a MIPS Chip doing some number crunching.  1 is a 
power PC, the other is a MIPS.  Both have different debug interfaces. 

Now if you had a system say, where you had 3 MIPS Chips, hooked up on 
the same EJTAG interface, you would need to handle that with some nifty 
EJTAG code in your (pseudo) stub to ensure each device was uniquely 
addressed and they didnt interfere with one another, so that you could 
start up 3 GDB sessions to debug your 3 processors, but then it becomes 
a problem for the stub. 

What im saying is I dont think a single instance of GDB needs to be 
complicated to try and debug multiple "tasks" simultaneously.  I dont 
have any problems with running GDB as many times as I want.  For example 
with the MSP430 example, I had (at various times) GDB running 5 times on 
the one PC.  One was debugging a local PC app that talked to my MSP430 
board.  3 were talking to the MSP430 board, the last was talking to yet 
another device (that had an MPC862 as its processor), I just ran each in 
a separate "Desktop" under KDE and then switched to the one i had to 
deal with at the time.  No problems, worked easily.

Hope that gives you insight into one application of what you discussed.
Steven

Re: Is multiprocessor debugging multithreaded debugging?

[Ramana Radhakrishnan]

Hi ,

As much as I agree with you there might be other use cases that are not
possible with the setup you describe. One example is to be able to be
able to single step 2 processors and let a 3rd continue merrily or
single step in lockstep a group of processors. 

Providing relational breakpoints between multiple processors might be
another nifty feature. 

Lets say something like b <here> if val_in_pgm_in_proc2 is 0xcafebabe. 

My 2 bits on the topic. 

cheers
Ramana



On Wed, 2005-09-28 at 21:16 +1100, Steven Johnson wrote:
> Anupama Chandwani wrote:
> 
> >In continuation with my prev mail..
> >I want to extend gdb to debug homing ogenous multiprocessor system
> >  
> >
> >(say multiple ARM or x86 processors on single chip) by remote
> >debugging in a single session of gdb.
> >
> >What i want to know is are there enough applications being written on
> >such multi processors? Also are there different executables being
> >required to be debugged simultaneously? Coz this is what i want to
> >extend in further.. Each processor running a different executable so
> >the processors dont share memory & run with different images of code.
> >  
> >
> This is commonly called "Asynchronous" Multi Processing.
> 
> >An application of such debugger could be while building an OS but that
> >wouldnt involve different executables.. So are there applications
> >requiring to run different executables on each processor? Say for
> >example a prog gives a certain bug on when there is certain other
> >program running on the other processor or something similar to
> >this....
> >  
> >
> Yes in the embedded world, there are many examples of Asynchronous Multi 
> Processor designs.  They are by far the easiest multi processor design 
> to implement.  I for example have worked on a board that had 3 MSP430's, 
> each had a unique function, and they intercommunicated over a custom 
> parallel bus to coordinate their activities.  Worked sweet, had high 
> performance, and was really cheap.
> 
> >As far as i know this done by multiplexing the JTAG interface (for
> >x86) &different sessions of gdb right now. Any other? And any flaws or
> >inconvenience with present methods?
> >  
> >
> This is exactly how it is done, multiple sessions of GDB.  This, in my 
> opinion is the right way to go.  Not all Asynchronous multi processor 
> designs have homogeneous pprocessors (ie, you may have an MPC860 
> handling comms, and a MIPS Chip doing some number crunching.  1 is a 
> power PC, the other is a MIPS.  Both have different debug interfaces. 
> 
> Now if you had a system say, where you had 3 MIPS Chips, hooked up on 
> the same EJTAG interface, you would need to handle that with some nifty 
> EJTAG code in your (pseudo) stub to ensure each device was uniquely 
> addressed and they didnt interfere with one another, so that you could 
> start up 3 GDB sessions to debug your 3 processors, but then it becomes 
> a problem for the stub. 
> 
> What im saying is I dont think a single instance of GDB needs to be 
> complicated to try and debug multiple "tasks" simultaneously.  I dont 
> have any problems with running GDB as many times as I want.  For example 
> with the MSP430 example, I had (at various times) GDB running 5 times on 
> the one PC.  One was debugging a local PC app that talked to my MSP430 
> board.  3 were talking to the MSP430 board, the last was talking to yet 
> another device (that had an MPC862 as its processor), I just ran each in 
> a separate "Desktop" under KDE and then switched to the one i had to 
> deal with at the time.  No problems, worked easily.
> 
> Hope that gives you insight into one application of what you discussed.
> Steven
-- 
Ramana Radhakrishnan
GNU Tools
codito ergo sum (www.codito.com)

Re: Is multiprocessor debugging multithreaded debugging?

[Steven Johnson]

Ramana Radhakrishnan wrote:

>Hi ,
>
>As much as I agree with you there might be other use cases that are not
>possible with the setup you describe. One example is to be able to be
>able to single step 2 processors and let a 3rd continue merrily or
>single step in lockstep a group of processors. 
>  
>
I dont think "lock step" is achievable with Asynchronous 
multiprocessing.  Each processor has a different clock domain, they also 
do not have any "synchronizing" features in hardware.  The best you 
could approximate is "command 2 targets to Single step at once" which 
one actually single steps first would of course be unknown. And in fact 
if one processor is faster than another the processor that actually 
single stepped last, could be the first to complete its instruction.

In my case, if i needed to do this, i worked out which instruction 
should be executed first (for my particular test), went to that instance 
of GDB, single stepped.  Went to the next instance of GDB and then 
stepped it.  Given the asynchronous nature of what you describe i can 
imagine all sorts of race conditions which would mean you would get 
inconsistent debug results.  Things like, why did that happen, that bit 
shouldnt have been set yet type stuff.

>Providing relational breakpoints between multiple processors might be
>another nifty feature. 
>
>Lets say something like b <here> if val_in_pgm_in_proc2 is 0xcafebabe. 
>  
>
I also think to do this GDB would need to "simulate" the behavior.  It 
would have to put an unconditional break on <here>.  Every time the 
target breaks on <here> break the processor with the val_in_pgm_in_proc2 
variable, inspect it. restart the proc2 target (which might then 
actually change the value from 0xcafebabe, so you are not in the state 
you think you are) and then make a decision about whether to continue 
from the break at <here> or report it to the user.  Ive yet to see a 
JTAG of BDM type interface that allows "live" inspection of data, while 
the CPU is executing code.

I think all of these things are advanced User Interface concerns, and 
actually have little if anything to do with the fundamental GDB itself.  
Maybe what is really required, if anything, is a GDB front end, that can 
interface to multiple GDB back end instances (through MI), each one a 
potentially different processor architecture, and do these types of 
operations.  Tabbed pages for each CPU, simulated simultaneous step on n 
of m processors, cross processor break trigger conditioning.  Sounds 
like a bitching front end, but i dont think GDB itself should be made to 
do these things natively, as they are not actually native debug issues, 
but simulated "pseudo" operations.

Now if such a front end was GPL, could work on the majority of hosts 
that support GDB, and was assigned to the FSF it might (if the 
maintainers agree) even be able to live in the GDB CVS Tree (like the 
TUI front end), and be a standard example of an MI/GUI front end. (But 
now im dreaming).  Id shudder to think how unwieldy multi CPU debugging 
would be without multiple "screens" of information, you can switch 
between, so i really dont know how you would achieve anything really 
useful without a GUI in any event.

In fact what you describe is really no different to debugging two 
applications on the one PC, which interact, at the same time.  Again, 
something people do not infrequently, but they do it with multiple 
instances of the debugger running.

Maybe if Anupama can highlight what he thinks the deficiencies with the 
current arrangement are, i could see the problems with my view.

Now if you were talking about getting GDB to be able to read/set JTAG 
boundary scan information (on targets where that is applicable), and 
process BSDL files to show the states of all pins in a meaningful way I 
think that would be a useful project in this realm, but i think making 
GDB natively handle asynchronous multiprocessor debugging is a waste of 
time that could be better spent, as there is no problem to solve that i 
can see.

Steven

>My 2 bits on the topic. 
>
>cheers
>Ramana
>
>
>
>On Wed, 2005-09-28 at 21:16 +1100, Steven Johnson wrote:
>  
>
>>Anupama Chandwani wrote:
>>
>>    
>>
>>>In continuation with my prev mail..
>>>I want to extend gdb to debug homing ogenous multiprocessor system
>>> 
>>>
>>>(say multiple ARM or x86 processors on single chip) by remote
>>>debugging in a single session of gdb.
>>>
>>>What i want to know is are there enough applications being written on
>>>such multi processors? Also are there different executables being
>>>required to be debugged simultaneously? Coz this is what i want to
>>>extend in further.. Each processor running a different executable so
>>>the processors dont share memory & run with different images of code.
>>> 
>>>
>>>      
>>>
>>This is commonly called "Asynchronous" Multi Processing.
>>
>>    
>>
>>>An application of such debugger could be while building an OS but that
>>>wouldnt involve different executables.. So are there applications
>>>requiring to run different executables on each processor? Say for
>>>example a prog gives a certain bug on when there is certain other
>>>program running on the other processor or something similar to
>>>this....
>>> 
>>>
>>>      
>>>
>>Yes in the embedded world, there are many examples of Asynchronous Multi 
>>Processor designs.  They are by far the easiest multi processor design 
>>to implement.  I for example have worked on a board that had 3 MSP430's, 
>>each had a unique function, and they intercommunicated over a custom 
>>parallel bus to coordinate their activities.  Worked sweet, had high 
>>performance, and was really cheap.
>>
>>    
>>
>>>As far as i know this done by multiplexing the JTAG interface (for
>>>x86) &different sessions of gdb right now. Any other? And any flaws or
>>>inconvenience with present methods?
>>> 
>>>
>>>      
>>>
>>This is exactly how it is done, multiple sessions of GDB.  This, in my 
>>opinion is the right way to go.  Not all Asynchronous multi processor 
>>designs have homogeneous pprocessors (ie, you may have an MPC860 
>>handling comms, and a MIPS Chip doing some number crunching.  1 is a 
>>power PC, the other is a MIPS.  Both have different debug interfaces. 
>>
>>Now if you had a system say, where you had 3 MIPS Chips, hooked up on 
>>the same EJTAG interface, you would need to handle that with some nifty 
>>EJTAG code in your (pseudo) stub to ensure each device was uniquely 
>>addressed and they didnt interfere with one another, so that you could 
>>start up 3 GDB sessions to debug your 3 processors, but then it becomes 
>>a problem for the stub. 
>>
>>What im saying is I dont think a single instance of GDB needs to be 
>>complicated to try and debug multiple "tasks" simultaneously.  I dont 
>>have any problems with running GDB as many times as I want.  For example 
>>with the MSP430 example, I had (at various times) GDB running 5 times on 
>>the one PC.  One was debugging a local PC app that talked to my MSP430 
>>board.  3 were talking to the MSP430 board, the last was talking to yet 
>>another device (that had an MPC862 as its processor), I just ran each in 
>>a separate "Desktop" under KDE and then switched to the one i had to 
>>deal with at the time.  No problems, worked easily.
>>
>>Hope that gives you insight into one application of what you discussed.
>>Steven
>>    
>>

Re: Is multiprocessor debugging multithreaded debugging?

[Ramana Radhakrishnan]

Hi Steven, 

> >
> I dont think "lock step" is achievable with Asynchronous 
> multiprocessing.  Each processor has a different clock domain, they also 
> do not have any "synchronizing" features in hardware.  The best you 
> could approximate is "command 2 targets to Single step at once" which 
> one actually single steps first would of course be unknown. And in fact 
> if one processor is faster than another the processor that actually 
> single stepped last, could be the first to complete its instruction.

Yes it is this approximation that I meant. If you wanted to define an
order of stepping that could be one extra feature. But you really cannot
have a granularity more than an instruction.


> 
> In my case, if i needed to do this, i worked out which instruction 
> should be executed first (for my particular test), went to that instance 
> of GDB, single stepped.  Went to the next instance of GDB and then 
> stepped it.  Given the asynchronous nature of what you describe i can 
> imagine all sorts of race conditions which would mean you would get 
> inconsistent debug results.  Things like, why did that happen, that bit 
> shouldnt have been set yet type stuff.
> 
> >Providing relational breakpoints between multiple processors might be
> >another nifty feature. 
> >
> >Lets say something like b <here> if val_in_pgm_in_proc2 is 0xcafebabe. 
> >  
> >
> I also think to do this GDB would need to "simulate" the behavior.  It 
> would have to put an unconditional break on <here>.  Every time the 
> target breaks on <here> break the processor with the val_in_pgm_in_proc2 
> variable, inspect it. restart the proc2 target (which might then 
> actually change the value from 0xcafebabe, so you are not in the state 
> you think you are) and then make a decision about whether to continue 
> from the break at <here> or report it to the user.  Ive yet to see a 
> JTAG of BDM type interface that allows "live" inspection of data, while 
> the CPU is executing code.

Agreed ! Neither have I . 

> 
> I think all of these things are advanced User Interface concerns, and 
> actually have little if anything to do with the fundamental GDB itself.  
> Maybe what is really required, if anything, is a GDB front end, that can 
> interface to multiple GDB back end instances (through MI), each one a 
> potentially different processor architecture, and do these types of 
> operations.  Tabbed pages for each CPU, simulated simultaneous step on n 
> of m processors, cross processor break trigger conditioning.  Sounds 
> like a bitching front end, but i dont think GDB itself should be made to 
> do these things natively, as they are not actually native debug issues, 
> but simulated "pseudo" operations.


User Interfaces are a concern and if one could not use these features
from the command line or emacs for that matter (which happens to be my
favourite GDB frontend.)  . 

> 
> Now if such a front end was GPL, could work on the majority of hosts 
> that support GDB, and was assigned to the FSF it might (if the 
> maintainers agree) even be able to live in the GDB CVS Tree (like the 
> TUI front end), and be a standard example of an MI/GUI front end. (But 
> now im dreaming).  Id shudder to think how unwieldy multi CPU debugging 
> would be without multiple "screens" of information, you can switch 
> between, so i really dont know how you would achieve anything really 
> useful without a GUI in any event.

I agree about the UI design part, a per CPU command or a per CPU view /
a process command and viewing registers seamlessly depending on the
context of the debuggee under consideration would be a nifty feature to
have. 

A set context to cpu#x would be another command. The problem with
frontends again is that users have different choices of frontends, so to
get people to migrate from one to another is another set of issues. And
again keeping it in gdb allows users to continue using their frontends
or customizing it for features in gdb  . Also it allows users to
seamlessly continue using their existing features. 

> 
> In fact what you describe is really no different to debugging two 
> applications on the one PC, which interact, at the same time.  Again, 
> something people do not infrequently, but they do it with multiple 
> instances of the debugger running.

Agreed again ! What I have been interested in at some point in the past,
was designing a unified frontend and possibly looking at an interface
within GDB to do such stuff. We did do some prototyping with exactly
this kind of a setup where we could look at multiple binaries within the
same session and switch between processes and so on and so forth.  


> 
> Maybe if Anupama can highlight what he thinks the deficiencies with the 
> current arrangement are, i could see the problems with my view.

Its a she actually ! 

> 
> Now if you were talking about getting GDB to be able to read/set JTAG 
> boundary scan information (on targets where that is applicable), and 
> process BSDL files to show the states of all pins in a meaningful way I 
> think that would be a useful project in this realm, but i think making 
> GDB natively handle asynchronous multiprocessor debugging is a waste of 
> time that could be better spent, as there is no problem to solve that i 
> can see.

If I understand this right you are talking about reading and setting
JTAG boundary scan info , processing BSDL regarding the design for the
particular board under consideration and then debug the model using that
for the program being debugged and relate the two up.Map between test
program source and the state of the pins . 

Care to elaborate this further ? If it is not really relevant to
discussion on gdb@ maybe we can carry this conversation offlist ? 

cheers
Ramana

> 
> Steven
> 
> >My 2 bits on the topic. 
> >
> >cheers
> >Ramana
> >
> >
> >
> >On Wed, 2005-09-28 at 21:16 +1100, Steven Johnson wrote:
> >  
> >
> >>Anupama Chandwani wrote:
> >>
> >>    
> >>
> >>>In continuation with my prev mail..
> >>>I want to extend gdb to debug homing ogenous multiprocessor system
> >>> 
> >>>
> >>>(say multiple ARM or x86 processors on single chip) by remote
> >>>debugging in a single session of gdb.
> >>>
> >>>What i want to know is are there enough applications being written on
> >>>such multi processors? Also are there different executables being
> >>>required to be debugged simultaneously? Coz this is what i want to
> >>>extend in further.. Each processor running a different executable so
> >>>the processors dont share memory & run with different images of code.
> >>> 
> >>>
> >>>      
> >>>
> >>This is commonly called "Asynchronous" Multi Processing.
> >>
> >>    
> >>
> >>>An application of such debugger could be while building an OS but that
> >>>wouldnt involve different executables.. So are there applications
> >>>requiring to run different executables on each processor? Say for
> >>>example a prog gives a certain bug on when there is certain other
> >>>program running on the other processor or something similar to
> >>>this....
> >>> 
> >>>
> >>>      
> >>>
> >>Yes in the embedded world, there are many examples of Asynchronous Multi 
> >>Processor designs.  They are by far the easiest multi processor design 
> >>to implement.  I for example have worked on a board that had 3 MSP430's, 
> >>each had a unique function, and they intercommunicated over a custom 
> >>parallel bus to coordinate their activities.  Worked sweet, had high 
> >>performance, and was really cheap.
> >>
> >>    
> >>
> >>>As far as i know this done by multiplexing the JTAG interface (for
> >>>x86) &different sessions of gdb right now. Any other? And any flaws or
> >>>inconvenience with present methods?
> >>> 
> >>>
> >>>      
> >>>
> >>This is exactly how it is done, multiple sessions of GDB.  This, in my 
> >>opinion is the right way to go.  Not all Asynchronous multi processor 
> >>designs have homogeneous pprocessors (ie, you may have an MPC860 
> >>handling comms, and a MIPS Chip doing some number crunching.  1 is a 
> >>power PC, the other is a MIPS.  Both have different debug interfaces. 
> >>
> >>Now if you had a system say, where you had 3 MIPS Chips, hooked up on 
> >>the same EJTAG interface, you would need to handle that with some nifty 
> >>EJTAG code in your (pseudo) stub to ensure each device was uniquely 
> >>addressed and they didnt interfere with one another, so that you could 
> >>start up 3 GDB sessions to debug your 3 processors, but then it becomes 
> >>a problem for the stub. 
> >>
> >>What im saying is I dont think a single instance of GDB needs to be 
> >>complicated to try and debug multiple "tasks" simultaneously.  I dont 
> >>have any problems with running GDB as many times as I want.  For example 
> >>with the MSP430 example, I had (at various times) GDB running 5 times on 
> >>the one PC.  One was debugging a local PC app that talked to my MSP430 
> >>board.  3 were talking to the MSP430 board, the last was talking to yet 
> >>another device (that had an MPC862 as its processor), I just ran each in 
> >>a separate "Desktop" under KDE and then switched to the one i had to 
> >>deal with at the time.  No problems, worked easily.
> >>
> >>Hope that gives you insight into one application of what you discussed.
> >>Steven
> >>    
> >>