Re: Non-stop multi-threaded debugging

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Hello,
I am collaborating with Ericsson on implementing GDB support in Eclipse 
and I'm very much looking forward to seeing this functionality 
implemented in GDB.   With the Eclipse use of the MI protocol in mind I 
have a couple of comments and questions on the proposal:

> P1) Non-stop multi-threaded native debugging
<snip>
>     - The MI '-exec-interrupt' command shall stop all threads.  This
>       will always generate an 'EXEC-ASYNC-OUTPUT' record, even if all
>       threads were already stopped.  (This helps users handle the case
>       where the thread stops of its own accord just as the user sends
>       it an '-exec-interrupt' command.)
>
>     - The MI '-thread-select' command shall stop the thread selected,
>       if it is running.  The previously selected thread is left in its
>       former state, either stopped or running.  A '-thread-select'
>       command shall always generate an 'EXEC-ASYNC-OUTPUT' record,
>       even if the thread was already stopped.
I think using -thread-select to suspend a thread will have negative 
consequences to the MI-client.  The -thread-select is used to change the 
protocol state before requesting data related to that thread.  Adding 
the side-effect of suspending the thread, will force the client to add 
additional logic to avoid suspending threads accidentally as a result of 
attempting to read data.  I think it would be much safer to use an 
explicit -exec-interrupt request (after a -thread-select) to suspend a 
thread. 
Note that either way GDB will have to handle and return an errors to 
requests for data (such as stack or expressions) while a selected thread 
is running.  If a client sends a -exec-continue, I assume that whatever 
thread was selected last will still be selected.  So a 
-stack-list-frames should result in an error.

> P7) Non-stop multi-threaded multi-process native debugging
<snip>
>     - MI shall provide a command to stop all the threads in a given
>       process, and a command to resume all stopped threads in a given
>       process.

I would like to make a suggestion here.  Instead of inventing a new set 
of parallel commands and events to deal with process-wide events, it 
would be cleaner to simply assign thread IDs to processes and use 
existing commands and events to operate on them.  A -thread-select on an 
ID of a process followed by an -exec-continue would resume an entire 
process, while a -thread-select of a thread's ID followed by a continue 
would resume only that thread.  This could also be applied to all other 
commands that need to operate on a process, such as -thread-list-ids, 
-break-insert, etc.

While events could also be re-used to report without any changes to the 
event format, it would be very helpful for the client if the events 
included both the process's and thread's thread ID, as well as whether 
all threads stopped as the result of the event:
...
-thread-select 2
^done
-exec-interrupt
*stopped, thread-id="2", process-thread-id="1", all-threads="false", ....
^done
...
-thread-select 1
^done
-exec-continue
^running, thread-id="1", process-thread-id="1", all-threads="true"
...

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Nick Roberts]

 >                                           ... A -thread-select on an 
 > ID of a process followed by an -exec-continue would resume an entire 
 > process, while a -thread-select of a thread's ID followed by a continue 
 > would resume only that thread.  This could also be applied to all other 
 > commands that need to operate on a process, such as -thread-list-ids, 
 > -break-insert, etc.

This would change the current behaviour of these commands.  If a new command
is undesirable then perhaps optional parameters could be used:

     -exec-continue [ -p THREAD-ID/PROCESS-ID ]
     -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]

It appears that -break-insert already has such an option for threads.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Nick Roberts wrote:
>  >                                           ... A -thread-select on an 
>  > ID of a process followed by an -exec-continue would resume an entire 
>  > process, while a -thread-select of a thread's ID followed by a continue 
>  > would resume only that thread.  This could also be applied to all other 
>  > commands that need to operate on a process, such as -thread-list-ids, 
>  > -break-insert, etc.
>
> This would change the current behaviour of these commands.  If a new command
> is undesirable then perhaps optional parameters could be used:
>
>      -exec-continue [ -p THREAD-ID/PROCESS-ID ]
>      -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]
>
> It appears that -break-insert already has such an option for threads.
>
>   
 From Eclipse's point of view it actually doesn't make much difference 
whether -thread-select or -p option is used to specify the thread. 

That said, I would argue that adding the non-stop debugging feature 
changes the behavior of the entire system, so it could be expected that 
some commands will behave somewhat differently as they relate to this 
new feature.  Actually, with non-stop debugging feature turned off, and 
without attaching to multiple processes, these commands would still 
behave exactly as they do now. 

In fact, the original post states:
>     - Execution commands like '-exec-continue' and '-exec-step' shall
>       resume only the selected thread, without affecting the other
>       threads in the process.
So it seems that Jim's proposal would already change the existing 
behavior of -exec-continue and -exec-step commands.


Also:
> At the user interface level:
>
>     - MI shall provide a command to stop all the threads in a given
>       process, and a command to resume all stopped threads in a given
>       process.
Which to me implies that -exec-continue and -exec-interrupt would act on 
a single thread, although this is contradicted by the other statement 
about -exec-interrupt.


Lastly, since my post I thought a little about the CLI aspect in my 
suggested use of thread IDs.  It would probably be a little awkward to 
use the "thread <id>" command to switch to a different process, or to 
use it with a process-id to initiate process-wide operations.  So for 
the CLI interface, while I still think it would make sense to use the 
same ID name-space for processes and threads, it would help to add a 
couple of new commands:

"process <id>" - To select a new active process where the active thread 
would be the last active thread in this process.  This would be the 
equivalent of a -thread-select with the last active thread in that process.

"thread all" - To un-select the current thread in order to perform 
operations on all threads in the process.  Equivalent of -thread-select 
with the ID of the process.

Cheers,
Pawel

gdb over RNDIS to PDA

[Steve DeLaney]

 
anyone doing anything with gdb over RNDIS?  
I'm working on a windows mobile project using cegcc and am trying to get
gdbserver to work
the PDA is sync'd to linux FC6 with RNDIS/USB using SynCE

details posted in cegcc-devel@lists.sourceforge.net 
topic "mingw32ce gdb to wm2005 over RNDIS"

thanks!

/steverino2

Re: Non-stop multi-threaded debugging

[Vladimir Prus]

Pawel Piech wrote:

> Nick Roberts wrote:
>>  >                                           ... A -thread-select on an
>>  > ID of a process followed by an -exec-continue would resume an entire
>>  > process, while a -thread-select of a thread's ID followed by a
>>  > continue
>>  > would resume only that thread.  This could also be applied to all
>>  > other commands that need to operate on a process, such as
>>  > -thread-list-ids, -break-insert, etc.
>>
>> This would change the current behaviour of these commands.  If a new
>> command is undesirable then perhaps optional parameters could be used:
>>
>>      -exec-continue [ -p THREAD-ID/PROCESS-ID ]
>>      -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]
>>
>> It appears that -break-insert already has such an option for threads.
>>
>>   
>  From Eclipse's point of view it actually doesn't make much difference
> whether -thread-select or -p option is used to specify the thread.
> 
> That said, I would argue that adding the non-stop debugging feature
> changes the behavior of the entire system, so it could be expected that
> some commands will behave somewhat differently as they relate to this
> new feature.  Actually, with non-stop debugging feature turned off, and
> without attaching to multiple processes, these commands would still
> behave exactly as they do now.

Given the choice between:

1. Changing the behaviour of the existing command, and adding new one
that behaves like existing one, and

2. Adding new command

I think adding new command (or option to existing command), is a smaller change.
So yes, -exec-continue -t <xxx> might be a better choice.

- Volodya

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Vladimir Prus wrote:
> Pawel Piech wrote:
>
>   
>> Nick Roberts wrote:
>>     
>>>  >                                           ... A -thread-select on an
>>>  > ID of a process followed by an -exec-continue would resume an entire
>>>  > process, while a -thread-select of a thread's ID followed by a
>>>  > continue
>>>  > would resume only that thread.  This could also be applied to all
>>>  > other commands that need to operate on a process, such as
>>>  > -thread-list-ids, -break-insert, etc.
>>>
>>> This would change the current behaviour of these commands.  If a new
>>> command is undesirable then perhaps optional parameters could be used:
>>>
>>>      -exec-continue [ -p THREAD-ID/PROCESS-ID ]
>>>      -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]
>>>
>>> It appears that -break-insert already has such an option for threads.
>>>
>>>   
>>>       
>>  From Eclipse's point of view it actually doesn't make much difference
>> whether -thread-select or -p option is used to specify the thread.
>>
>> That said, I would argue that adding the non-stop debugging feature
>> changes the behavior of the entire system, so it could be expected that
>> some commands will behave somewhat differently as they relate to this
>> new feature.  Actually, with non-stop debugging feature turned off, and
>> without attaching to multiple processes, these commands would still
>> behave exactly as they do now.
>>     
>
> Given the choice between:
>
> 1. Changing the behaviour of the existing command, and adding new one
> that behaves like existing one, and
>
> 2. Adding new command
>
> I think adding new command (or option to existing command), is a smaller change.
> So yes, -exec-continue -t <xxx> might be a better choice.
>
> - Volodya
>   
I don't believe that I suggested adding any new command.  Instead, I 
suggested only changing the behavior of some of the existing commands to 
use the currently selected thread.  It was Jim's proposal that included 
adding new commands.  My point was that instead of adding new commands 
it would be cleaner to extend the functionality of -thread-select in 
order to select a process context, i.e. a context that includes all 
threads. This way existing commands, which currently can only operate on 
a global context, could now operate on a process or a thread context.

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Vladimir Prus]

On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:

>  I don't believe that I suggested adding any new command.
>  Instead, I suggested only changing the behavior of 
>  some of the existing commands to use the currently selected thread.
>  It was Jim's proposal that included adding new commands.
>  My point was that instead of adding new commands it would be cleaner
>  to extend the functionality of -thread-select in order to select 
>  a process context, i.e. a context that includes all threads. 
>  This way existing commands, which currently can only operate
>  on a global context, could now operate on a process or a thread context.        

Ok, we have those choices:

1. Make -exec-continue work on all threads, until -thread-select is used.
2. Make -exec-continue work on one thread, and add another command
to operate on all threads.
3. Make -exec-continue still operate on all threads, unless
and explicit option to make it operate on a thread is given.

You've indicated that (1) and (3) are about the same in complexity for you --
am I right? I personally prefer (3), since it does not implicitly changes
the meaning of existing commands.

Surely, non-stop mode does require some changes in frontend, but the
fewer changes are, the better, IMO.

- Volodya

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Vladimir Prus wrote:
> On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:
>
>   
>>  I don't believe that I suggested adding any new command.
>>  Instead, I suggested only changing the behavior of 
>>  some of the existing commands to use the currently selected thread.
>>  It was Jim's proposal that included adding new commands.
>>  My point was that instead of adding new commands it would be cleaner
>>  to extend the functionality of -thread-select in order to select 
>>  a process context, i.e. a context that includes all threads. 
>>  This way existing commands, which currently can only operate
>>  on a global context, could now operate on a process or a thread context.        
>>     
>
> Ok, we have those choices:
>
> 1. Make -exec-continue work on all threads, until -thread-select is used.
> 2. Make -exec-continue work on one thread, and add another command
> to operate on all threads.
> 3. Make -exec-continue still operate on all threads, unless
> and explicit option to make it operate on a thread is given.
>
> You've indicated that (1) and (3) are about the same in complexity for you --
> am I right? 
This is correct.
> I personally prefer (3), since it does not implicitly changes
> the meaning of existing commands.
>
> Surely, non-stop mode does require some changes in frontend, but the
> fewer changes are, the better, IMO.
>
> - Volodya
>   

In that case I'll try to convince you otherwise :-)

-exec-continue is not the only command that would need to be modified.  
-exec-interrupt, would all need to take the -p parameter, and in order 
to implement multi-process debugging, many of the commands that 
currently operate on a global context (too many to try to list) would 
all require an additional parameter to specify which process they are to 
act on.  There seems to be a well established paradigm in the MI (and 
CLI) protocol, where special commands: -thread-select and 
-stack-frame-select change the state of the protocol so that commands 
following these operate on the context selected by these commands.  My 
main point is to extend the functionality of these state-changing 
commands in order to add the ability to select an active context, and to 
select a context which will allow commands to operate on all the threads 
of a process. 

IMO, the question of whether -exec-continue takes a -p argument is a 
rather minor one.  But for sake of consistency with other -exec-* 
commands I think it would be a mistake to add this parameter.  That's 
because the stepping commands already do operate on the currently 
selected thread.  While all the threads are resumed when stepping, 
execution does not stop until the next line of code is reached by the 
thread that was selected.  With non-stop debugging, stepping commands 
will continue to operate on the selected thread with the difference that 
other suspended threads will remain suspended.  It seems logical to me 
that -exec-continue and -exec-interrupt should operate in the same way.  
In the degenerate case where the target is not capable non-stop 
multi-threaded debugging, conceptually these commands can still be 
thought of as operating on the selected thread, with the side effect 
that when one thread is resumed/suspended, all threads are 
resumed/suspended as well.

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Vladimir Prus]

On Thursday 29 November 2007 20:36:18 Pawel Piech wrote:
> Vladimir Prus wrote:
> > On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:
> >
> >   
> >>  I don't believe that I suggested adding any new command.
> >>  Instead, I suggested only changing the behavior of 
> >>  some of the existing commands to use the currently selected thread.
> >>  It was Jim's proposal that included adding new commands.
> >>  My point was that instead of adding new commands it would be cleaner
> >>  to extend the functionality of -thread-select in order to select 
> >>  a process context, i.e. a context that includes all threads. 
> >>  This way existing commands, which currently can only operate
> >>  on a global context, could now operate on a process or a thread context.        
> >>     
> >
> > Ok, we have those choices:
> >
> > 1. Make -exec-continue work on all threads, until -thread-select is used.
> > 2. Make -exec-continue work on one thread, and add another command
> > to operate on all threads.
> > 3. Make -exec-continue still operate on all threads, unless
> > and explicit option to make it operate on a thread is given.
> >
> > You've indicated that (1) and (3) are about the same in complexity for you --
> > am I right? 
> This is correct.
> > I personally prefer (3), since it does not implicitly changes
> > the meaning of existing commands.
> >
> > Surely, non-stop mode does require some changes in frontend, but the
> > fewer changes are, the better, IMO.
> >
> > - Volodya
> >   
> 
> In that case I'll try to convince you otherwise :-)
> 
> -exec-continue is not the only command that would need to be modified.  
> -exec-interrupt, would all need to take the -p parameter, and in order 
> to implement multi-process debugging, many of the commands that 
> currently operate on a global context (too many to try to list) would 
> all require an additional parameter to specify which process they are to 
> act on.  There seems to be a well established paradigm in the MI (and 
> CLI) protocol, where special commands: -thread-select and 
> -stack-frame-select change the state of the protocol so that commands 
> following these operate on the context selected by these commands.  

There are different opinions about those commands ;-) For example, I believe
that stateless protocol would be much easier.

> My  
> main point is to extend the functionality of these state-changing 
> commands in order to add the ability to select an active context, and to 
> select a context which will allow commands to operate on all the threads 
> of a process. 
> 
> IMO, the question of whether -exec-continue takes a -p argument is a 
> rather minor one.  But for sake of consistency with other -exec-* 
> commands I think it would be a mistake to add this parameter.  That's 
> because the stepping commands already do operate on the currently 
> selected thread.  While all the threads are resumed when stepping, 
> execution does not stop until the next line of code is reached by the 
> thread that was selected.  With non-stop debugging, stepping commands 
> will continue to operate on the selected thread with the difference that 
> other suspended threads will remain suspended.  

Interesting. Presently, in CLI gdb set 'scheduler mode' that controls
if step resumes all thread, or not. MI does not have a counterpart of that.
Maybe, we should start by changing -exec-step to accept a thread parameter, too,
which would mean "step only this thread, keep others suspended". Then,
-exec-continue taking thread parameter will be quite consistent with -exec-step.
I would say such change to -exec-step will be valuable in itself.

- Volodya

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Vladimir Prus wrote:
> On Thursday 29 November 2007 20:36:18 Pawel Piech wrote:
>   
>> Vladimir Prus wrote:
>>     
>>> On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:
>>>
>>>   
>>>       
>>>>  I don't believe that I suggested adding any new command.
>>>>  Instead, I suggested only changing the behavior of 
>>>>  some of the existing commands to use the currently selected thread.
>>>>  It was Jim's proposal that included adding new commands.
>>>>  My point was that instead of adding new commands it would be cleaner
>>>>  to extend the functionality of -thread-select in order to select 
>>>>  a process context, i.e. a context that includes all threads. 
>>>>  This way existing commands, which currently can only operate
>>>>  on a global context, could now operate on a process or a thread context.        
>>>>     
>>>>         
>>> Ok, we have those choices:
>>>
>>> 1. Make -exec-continue work on all threads, until -thread-select is used.
>>> 2. Make -exec-continue work on one thread, and add another command
>>> to operate on all threads.
>>> 3. Make -exec-continue still operate on all threads, unless
>>> and explicit option to make it operate on a thread is given.
>>>
>>> You've indicated that (1) and (3) are about the same in complexity for you --
>>> am I right? 
>>>       
>> This is correct.
>>     
>>> I personally prefer (3), since it does not implicitly changes
>>> the meaning of existing commands.
>>>
>>> Surely, non-stop mode does require some changes in frontend, but the
>>> fewer changes are, the better, IMO.
>>>
>>> - Volodya
>>>   
>>>       
>> In that case I'll try to convince you otherwise :-)
>>
>> -exec-continue is not the only command that would need to be modified.  
>> -exec-interrupt, would all need to take the -p parameter, and in order 
>> to implement multi-process debugging, many of the commands that 
>> currently operate on a global context (too many to try to list) would 
>> all require an additional parameter to specify which process they are to 
>> act on.  There seems to be a well established paradigm in the MI (and 
>> CLI) protocol, where special commands: -thread-select and 
>> -stack-frame-select change the state of the protocol so that commands 
>> following these operate on the context selected by these commands.  
>>     
>
> There are different opinions about those commands ;-) For example, I believe
> that stateless protocol would be much easier.
>
>   
I completely agree with you, for a "machine interface" a stateless 
protocol would be much easier to use.  But GDB/MI is what it is and at 
this point trying to make it stateless would probably be counter productive.
>> My  
>> main point is to extend the functionality of these state-changing 
>> commands in order to add the ability to select an active context, and to 
>> select a context which will allow commands to operate on all the threads 
>> of a process. 
>>
>> IMO, the question of whether -exec-continue takes a -p argument is a 
>> rather minor one.  But for sake of consistency with other -exec-* 
>> commands I think it would be a mistake to add this parameter.  That's 
>> because the stepping commands already do operate on the currently 
>> selected thread.  While all the threads are resumed when stepping, 
>> execution does not stop until the next line of code is reached by the 
>> thread that was selected.  With non-stop debugging, stepping commands 
>> will continue to operate on the selected thread with the difference that 
>> other suspended threads will remain suspended.  
>>     
>
> Interesting. Presently, in CLI gdb set 'scheduler mode' that controls
> if step resumes all thread, or not. MI does not have a counterpart of that.
> Maybe, we should start by changing -exec-step to accept a thread parameter, too,
> which would mean "step only this thread, keep others suspended". Then,
> -exec-continue taking thread parameter will be quite consistent with -exec-step.
> I would say such change to -exec-step will be valuable in itself.
>
> - Volodya
>   
I'm rather confused here.  I thought that the point of non-stop 
multi-threaded debugging was to allow for this behavior.  Is "scheduler 
mode" some kind of partial implementation?  Either way though, I have no 
opinion whether it would be useful for the command line user to be able 
to specify the scheduling mode as an option to individual commands, but 
in an IDE such behavior would most likely be controlled with a toggle 
button, so a mode switch command would be sufficient.  If -exec-step 
would take a -p parameter to determine whether to resume all threads, 
then -exec-step without -p would still operate on the selected thread, 
in that it would suspend the process only when the selected thread 
reached the next line.  IHMO, this would only add to the confusion.

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Folks should treat this post:

  http://sourceware.org/ml/gdb/2007-11/msg00198.html

as an attempt to mention all the relevant issues --- not as a fully
worked-out design.  The plan was to hash out the details through
discussion amongst the front end authors on this list.  I don't have
experience writing clients for MI; I've only dealt with implementing
it in GDB, and even then not much.  So you people who work on
real-life front ends are in a much better position to see what would
work best.

So I encourage you, Vlad, Nick, and the rest of the MI fans to hash it
all out and come up with something everyone is satisfied with.  We
just need to make sure that the issues the analysis raises all get
addressed (or dismissed).

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Hi Jim,
I'm glad to hear that you're receptive to feedback in defining the 
protocol for these new features.  I will be glad to write up my proposal 
as a starting point for discussion.  I've never participated on this 
list before or in any other GDB forum, so forgive my ignorance, but is 
this mailing list the appropriate place for this discussion?  Is there a 
bugzilla or other bug database where I could post this document?  And 
what is the best format for such design document.

Thanks
Pawel

Jim Blandy wrote:
> Folks should treat this post:
>
>   http://sourceware.org/ml/gdb/2007-11/msg00198.html
>
> as an attempt to mention all the relevant issues --- not as a fully
> worked-out design.  The plan was to hash out the details through
> discussion amongst the front end authors on this list.  I don't have
> experience writing clients for MI; I've only dealt with implementing
> it in GDB, and even then not much.  So you people who work on
> real-life front ends are in a much better position to see what would
> work best.
>
> So I encourage you, Vlad, Nick, and the rest of the MI fans to hash it
> all out and come up with something everyone is satisfied with.  We
> just need to make sure that the issues the analysis raises all get
> addressed (or dismissed).
>   

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Pawel Piech <pawel.piech at windriver.com> writes:
> I'm glad to hear that you're receptive to feedback in defining the
> protocol for these new features.  I will be glad to write up my
> proposal as a starting point for discussion.

That'd be great!

> I've never participated
> on this list before or in any other GDB forum, so forgive my
> ignorance, but is this mailing list the appropriate place for this
> discussion?  Is there a bugzilla or other bug database where I could
> post this document?  And what is the best format for such design
> document.

This is the right list for general discussion like this.  If one has a
patch for discussion or review, that belongs on gdb-patches.

We haven't made a practice of putting documents in an issue tracker; I
don't think anyone has proposed it.  We just link to the archive, as I
did in the message you replied to.

The GDB documentation is written in texinfo.  When I've written draft
documentation for discussion, I've written it in texinfo, and then
posted the output from 'makeinfo --plaintext' for review and
discussion.  But if it's not intended to go into a manual, then plain
text is what we usually use.

Non-stop multi-threaded debugging

[Nathan Sidwell]

Hi all,

Jim Blandy prepared this, but is on vacation this week.  So, I'm announcing it 
in his absence.  Pretend I wrote 'sudo jimb ...'

A client of CodeSourcery's has contracted with us to implement a
number of new features in GDB, some of which have been on the
frequently requested list for quite some time:

- We're to implement non-stop multi-threaded debugging in GDB.

   At present, if you are debugging a multi-threaded program, when one
   thread stops (for a breakpoint, watchpoint, exception, or the like),
   GDB stops all other threads in the program while you interact with
   the thread of interest.  When you continue or step a thread, you can
   allow the other threads to run, or have them remain stopped, but
   while you inspect any thread's state, all threads stop.

   In non-stop mode, when one thread stops, other threads can continue
   to run freely.  You'll be able to treat each thread independently,
   leaving it stopped or free to run as needed.

   Non-stop mode will be selectable; the old all-stop behavior will
   still be available.

- We're to implement asynchronous interaction with GDB.

   GDB will be responsive to commands while the program is running.
   This is mostly a consequence of supporting non-stop multi-threaded
   debugging: it's the degenerate case where no threads happen to be
   stopped.

- We're to implement a limited form of multi-process debugging.

   Full multi-process debugging would entail changes to
   1) process management code,
   2) target interfaces, and
   3) symbol tables.

   For our client, however, the case where processes have different
   memory maps is not (yet) of interest, so they have sponsored us to
   do 1) and 2), but not 3).  This will yield a GDB that can (for
   example) follow both parent and child after a fork, but not follow
   processes across exec or dlopen/dlclose operations.  If a process
   carries out one of these operations, GDB will ask the user whether
   to follow that process only, or detach from it and stick with the
   others.

   So our goal here is to carry out steps 1) and 2) in such a way that
   anyone can easily pick up 3) and complete the feature.  In other
   words, we want the restrictions simply to be a matter of leaving
   work undone, and not of embedding simplifying assumptions into the
   code that would make full support difficult.

Our client would very much like for this work to be incorporated into
the public GDB sources (although they understand that the decision is
in the public project's hands), so we'll be posting our design
thoughts for general discussion.  In particular, I believe the
multi-process work may overlap with some of the work IBM has done to
support the Cell processor; we'd very much like to work with IBM to
ensure that the final model is appropriate for both our client and for
Cell developers.

Our client is only interested in the MI interface; they intend to use
all these facilities via Eclipse.  So we will not be implementing
command-line support any more than is helpful to us in development.
But again, we want to do this work in a way that leaves CLI support
for these features a simple matter of coding, so that our work is
still forward progress, which anyone can complete.

Our client is interested in non-stop, multi-process debugging via the
remote protocol.  However, we will be implementing these for native
debugging first, in order to break the work into manageable steps.

The below is taken from a more detailed document we put together
proposing the work.  It is in two sections:

- The "Architectural Challenges" section explains limitations of GDB's
   current architecture that make it difficult to implement non-stop
   and multi-process debugging at present.

- The "Projects" section presents a series of well-defined engineering
   projects which remove limitations or add features to meet one or
   more of our client's requirements.

Our intention is to help the list understand why each piece of work is
needed and what it would accomplish.


Architectural Challenges

GDB's present architecture imposes a number of barriers to
implementing non-stop and multi-process debugging:

C1) While the user inspects the state of a stopped thread, GDB stops
     all other threads.  This approach simplifies GDB's user interface,
     as there is no need to report events taking place in other threads
     while the user inspects one thread.  However, these
     simplifications are no longer valid in non-stop debugging

C2) Stopping all threads also simplifies GDB's execution management
     code, as GDB can pause all threads, manage interesting events, and
     then assume the system is quiet.  As above, these simplifications
     are no longer valid in non-stop debugging.

C3) Stopping all threads further allows GDB to remove all breakpoints
     from the program's memory while the program is stopped, and
     re-insert them only when resuming one or more threads, making it
     less likely that an abrupt disconnection will abandon a debuggee
     with breakpoint instructions patched into its code.  However, this
     behavior is clearly unsuitable if the user wants other threads to
     continue to execute while she stops one for inspection.

C4) Finally, stopping all threads simplifies GDB's remote protocol.
     At present, GDB's remote protocol notifies GDB of exactly one
     thread's state in response to each 'continue' or 'step' operation,
     permitting no further packets from the stub until GDB resumes
     some thread.

C5) GDB breakpoints are currently per-thread or global.  To satisfy
     our client's requirements, we must adapt these structures to
     distinguish per-process and global breakpoints, where 'global'
     breakpoints are set in all attached processes.

C6) [Our client elected not to address this issue yet.]

C7) GDB currently operates on a single process at a time: the list of
     known threads is global, and the ID of the process being debugged
     is global.  This conflicts with the needs of multi-process
     debugging.

C8) GDB currently maintains a single global map of the address space.
     It cannot represent multiple processes with code and data
     appearing at different addresses in different processes.  This is
     not a problem for our client, because code and variables appear at
     the same addresses in all processes on their system.  However, it
     is a requirement for multi-process debugging on Linux.

C9) GDB will not currently relocate different segments of an
     executable or shared library by different offsets from the
     addresses they are assigned in the ELF file.  The client's
     operating system may relocate each section of a load module by a
     different amount.


Projects

This section breaks down the work necessary into well-defined
engineering tasks.  For each proposed project, we explain the work
entailed, the benefits provided, and how it depends on other projects,
if at all.


P1) Non-stop multi-threaded native debugging

     This project allows GDB to stop one thread for inspection on a
     native system while allowing others to run.

     To prepare GDB to debug one process while other processes continue
     to run freely (the feature our client is interested in), we will
     first implement the ability to debug one thread while other
     threads in that process continue to run freely.

     As described in C1, C2, and C3, GDB assumes in its user interface
     and code that no execution occurs while the user is inspecting a
     thread's state.  This project removes that simplifying assumption.

     At the user interface level, GDB's Machine Interface ('MI', the
     command set used by Eclipse) shall behave as follows:

     - MI shall provide a command to allow the user to choose between
       the older 'all-stop' and the new 'non-stop' multi-threaded
       debugging behaviors.  In all-stop mode, GDB shall behave as it
       does now.  The following points describe non-stop debugging
       mode.

     - GDB shall always prompt for and respond to MI commands,
       regardless of whether any threads are running or not.

     - When a thread finishes a command like '-exec-next' or
       '-exec-finish', hits a breakpoint, or encounters a fault, GDB
       shall stop that thread, without affecting the other threads in
       the process.

     - Execution commands like '-exec-continue' and '-exec-step' shall
       resume only the selected thread, without affecting the other
       threads in the process.

     - The MI '-exec-interrupt' command shall stop all threads.  This
       will always generate an 'EXEC-ASYNC-OUTPUT' record, even if all
       threads were already stopped.  (This helps users handle the case
       where the thread stops of its own accord just as the user sends
       it an '-exec-interrupt' command.)

     - The MI '-thread-select' command shall stop the thread selected,
       if it is running.  The previously selected thread is left in its
       former state, either stopped or running.  A '-thread-select'
       command shall always generate an 'EXEC-ASYNC-OUTPUT' record,
       even if the thread was already stopped.

     - MI shall provide a command to continue all stopped threads.

     - GDB shall send 'EXEC-ASYNC-OUTPUT' MI records to notify the user
       of events that have occurred in threads, even while GDB is
       waiting for an MI command.  Every thread GDB stops shall be
       mentioned in some 'EXEC-ASYNC-OUTPUT' record; when GDB stops all
       threads, the EXEC-ASYNC-OUTPUT record shall include a
       'thread-id="all"' result.

     - The MI '-thread-info' and '-thread-list-all-threads' commands
       shall be implemented.  Their output shall indicate whether each
       thread listed is currently stopped by GDB, or whether it is
       allowed to run.

     - GDB shall use 'EXEC-ASYNC-OUTPUT' MI records to report thread
       creation and termination.  These records shall include the GDB
       thread number as a result.  After sending a thread termination
       record, GDB shall not include the thread in the output of
       '-thread-list-ids' or '-thread-list-all-threads'.

     (Adapting GDB's command-line interface to non-stop debugging is
     more involved; whereas MI need only be accurate and sufficient,
     the command-line interface must also respect human interface
     issues.  Since GDB's command-line interface is of limited interest
     to our client, we have not included it here.)

     To implement the behavior described above, a number of areas
     within GDB will need modification:

     - GDB's event loop must be responsive to user input and thread
       events from the debuggee simultaneously.

     - GDB's execution control code must avoid stopping all threads
       when one reports an event, and must make the processing of
       thread stops independent of resumption: it must no longer assume
       that events only arrive after resumptions, and resumptions only
       happen after events.

     - GDB must insert breakpoints into code being executed by live
       threads in a manner supported by the target architecture.

     - GDB's breakpoint support code must leave breakpoints inserted at
       all times.  Even while GDB steps a thread past a breakpoint,
       the breakpoint must remain in effect for all other threads.

     These are each reasonably substantial pieces of work, the design
     of which should be discussed on the public GDB list to ensure that
     the work will be acceptable for inclusion in the public sources
     when it is complete.


P4) Stub for client's OS

     This project will mostly be non-GDB work.  However, there are some
     changes to the remote protocol we would like to introduce at this
     point:

     The remote protocol presently leaves the process to be debugged
     implicit; users generally specify it when they start the stub.
     However, to satisfy our client's requirements, we must be able to
     connect to a system, list the processes present, and attach to one
     of them.  This entails making some straightforward extensions to
     the GDB remote protocol, and thus to GDB as well.

     The stub for our client should use the 'library' stop reply
     packets and the 'qXfer:libraries:read' packet to report load
     module events.  However, because the client's OS may bring each
     section of a load module into memory at a different offset from
     the VMA given in the ELF file, we will need to extend the format
     of the library list the latter packet returns, as it currently
     assumes that each library needs only one offset, and extend GDB to
     allow each segment to appear at a different offset (C9).


P6) Multi-threaded limited-multi-process native debugging

     This project provides multi-threaded debugging of multiple
     processes simultaneously.  The debugger stops all threads in all
     attached processes while the user inspects the state of any
     thread.  This work is independent of P1; we combine P1 and P6 in
     the next project, P7.

     At the user interface level:

     - MI shall provide new commands to attach and detach a process;
       unlike GDB's existing 'attach' and 'detach' commands, the new
       'attach' command will not require GDB to detach from any
       currently attached processes.

     - MI shall provide a command to list all currently attached
       processes.

     - MI shall provide a command to list all the threads in a given
       attached process.

     - The output of the MI '-thread-info' and
       '-thread-list-all-threads' commands shall include the process ID
       of each thread listed.  The process ID shall be a separate MI
       'result' from the string provided by the
       'target_extra_thread_info' function, so that Eclipse can access
       it reliably.

     - GDB shall stop all threads in all attached processes while
       interacting with the user.  Attaching to a process shall stop
       all threads in that process.  Detaching from a process shall
       allow its threads to run again.

     - MI shall report faults encountered by threads in any attached
       process.

     - MI shall report the termination of any attached process.  After
       such a report, GDB will no longer be attached to the process.

     - MI's '-thread-select' command shall be able to select any thread
       in any attached process.

     - MI's existing breakpoint commands shall set breakpoints global
       to all attached processes.

     To support those facilities, we need the following changes:

     - GDB shall maintain a table of attached processes.  The remote
       protocol shall provide packets directing the stub to attach to a
       new process, and to detach from a currently attached process.

     - The remote protocol shall carry process IDs as well as thread
       IDs in stop reply packets, thread selection packets, thread
       enumeration packets, and wherever else is appropriate.

     - The stub shall use the current general thread (as given by the
       'Hg' packet) to determine which process's memory to access, as
       it does now to determine which thread's registers to access.
       GDB shall send 'Hg' packets as necessary before memory accesses,
       as it does now for register accesses.


P7) Non-stop multi-threaded multi-process native debugging

     This project allows GDB to attach to multiple processes
     simultaneously.  This builds on P1 and P6, and addresses C4 and
     C7.

     At the user interface level:

     - MI shall provide a command to stop all the threads in a given
       process, and a command to resume all stopped threads in a given
       process.

     Internally:

     - The remote protocol shall provide a way to tell the stub to
       leave other threads running after reporting an event in one
       thread (non-stop behavior), and a way to tell the stub to stop
       all threads when reporting an event in one thread (stop-all
       behavior).

     - The remote protocol shall allow the stub to respond to
       commands while threads are running, and to report further thread
       events after a thread has stopped.  (This addresses C4.)

     - The remote protocol shall provide ways to stop and start a
       particular thread, and ways to start and stop all the threads in
       a given process.  The mechanisms for stopping threads and
       processes shall allow GDB to behave correctly when a thread
       stops or a process exits simultaneously with GDB sending the
       command.

nathan
-- 
Nathan Sidwell    ::   http://www.codesourcery.com   ::         CodeSourcery

Re: Non-stop multi-threaded debugging

[Nick Roberts]

 > - We're to implement asynchronous interaction with GDB.
 > 
 >    GDB will be responsive to commands while the program is running.
 >    This is mostly a consequence of supporting non-stop multi-threaded
 >    debugging: it's the degenerate case where no threads happen to be
 >    stopped.

I don't expect Jim will cut his holiday short to answer this, so this is
for when he gets back:

Will this use any of the code in the patch that I sent you, or will you use
an entirely different approach?

-- 
Nick                                           http://www.inet.net.nz/~nickrob

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Nick Roberts <nickrob at snap.net.nz> writes:
>  > - We're to implement asynchronous interaction with GDB.
>  > 
>  >    GDB will be responsive to commands while the program is running.
>  >    This is mostly a consequence of supporting non-stop multi-threaded
>  >    debugging: it's the degenerate case where no threads happen to be
>  >    stopped.
>
> I don't expect Jim will cut his holiday short to answer this, so this is
> for when he gets back:
>
> Will this use any of the code in the patch that I sent you, or will you use
> an entirely different approach?

We won't be using an entirely different approach; the existing event
loop seems to be adequate for most purposes.  And we'll certainly need
to do something like what you did with the async_signal_hook.  So it's
likely we'll be using at least portions of your patch.

Re: Non-stop multi-threaded debugging

[Michael Snyder]

Wow -- too long to read in one sitting, but sounds like 
this should be very interesting and challenging work!

I'm sure we'll all look forward to reviewing it, and if
there are issues to be discussed, I for one will look
forward to the discussions ;-)


On Tue, 2007-11-20 at 17:21 +0000, Nathan Sidwell wrote:
> Hi all,
> 
> Jim Blandy prepared this, but is on vacation this week.  So, I'm announcing it 
> in his absence.  Pretend I wrote 'sudo jimb ...'
> 
> A client of CodeSourcery's has contracted with us to implement a
> number of new features in GDB, some of which have been on the
> frequently requested list for quite some time:
> 
> - We're to implement non-stop multi-threaded debugging in GDB.
> 
>    At present, if you are debugging a multi-threaded program, when one
>    thread stops (for a breakpoint, watchpoint, exception, or the like),
>    GDB stops all other threads in the program while you interact with
>    the thread of interest.  When you continue or step a thread, you can
>    allow the other threads to run, or have them remain stopped, but
>    while you inspect any thread's state, all threads stop.
> 
>    In non-stop mode, when one thread stops, other threads can continue
>    to run freely.  You'll be able to treat each thread independently,
>    leaving it stopped or free to run as needed.
> 
>    Non-stop mode will be selectable; the old all-stop behavior will
>    still be available.
> 
> - We're to implement asynchronous interaction with GDB.
> 
>    GDB will be responsive to commands while the program is running.
>    This is mostly a consequence of supporting non-stop multi-threaded
>    debugging: it's the degenerate case where no threads happen to be
>    stopped.
> 
> - We're to implement a limited form of multi-process debugging.
> 
>    Full multi-process debugging would entail changes to
>    1) process management code,
>    2) target interfaces, and
>    3) symbol tables.
> 
>    For our client, however, the case where processes have different
>    memory maps is not (yet) of interest, so they have sponsored us to
>    do 1) and 2), but not 3).  This will yield a GDB that can (for
>    example) follow both parent and child after a fork, but not follow
>    processes across exec or dlopen/dlclose operations.  If a process
>    carries out one of these operations, GDB will ask the user whether
>    to follow that process only, or detach from it and stick with the
>    others.
> 
>    So our goal here is to carry out steps 1) and 2) in such a way that
>    anyone can easily pick up 3) and complete the feature.  In other
>    words, we want the restrictions simply to be a matter of leaving
>    work undone, and not of embedding simplifying assumptions into the
>    code that would make full support difficult.
> 
> Our client would very much like for this work to be incorporated into
> the public GDB sources (although they understand that the decision is
> in the public project's hands), so we'll be posting our design
> thoughts for general discussion.  In particular, I believe the
> multi-process work may overlap with some of the work IBM has done to
> support the Cell processor; we'd very much like to work with IBM to
> ensure that the final model is appropriate for both our client and for
> Cell developers.
> 
> Our client is only interested in the MI interface; they intend to use
> all these facilities via Eclipse.  So we will not be implementing
> command-line support any more than is helpful to us in development.
> But again, we want to do this work in a way that leaves CLI support
> for these features a simple matter of coding, so that our work is
> still forward progress, which anyone can complete.
> 
> Our client is interested in non-stop, multi-process debugging via the
> remote protocol.  However, we will be implementing these for native
> debugging first, in order to break the work into manageable steps.
> 
> The below is taken from a more detailed document we put together
> proposing the work.  It is in two sections:
> 
> - The "Architectural Challenges" section explains limitations of GDB's
>    current architecture that make it difficult to implement non-stop
>    and multi-process debugging at present.
> 
> - The "Projects" section presents a series of well-defined engineering
>    projects which remove limitations or add features to meet one or
>    more of our client's requirements.
> 
> Our intention is to help the list understand why each piece of work is
> needed and what it would accomplish.
> 
> 
> Architectural Challenges
> 
> GDB's present architecture imposes a number of barriers to
> implementing non-stop and multi-process debugging:
> 
> C1) While the user inspects the state of a stopped thread, GDB stops
>      all other threads.  This approach simplifies GDB's user interface,
>      as there is no need to report events taking place in other threads
>      while the user inspects one thread.  However, these
>      simplifications are no longer valid in non-stop debugging
> 
> C2) Stopping all threads also simplifies GDB's execution management
>      code, as GDB can pause all threads, manage interesting events, and
>      then assume the system is quiet.  As above, these simplifications
>      are no longer valid in non-stop debugging.
> 
> C3) Stopping all threads further allows GDB to remove all breakpoints
>      from the program's memory while the program is stopped, and
>      re-insert them only when resuming one or more threads, making it
>      less likely that an abrupt disconnection will abandon a debuggee
>      with breakpoint instructions patched into its code.  However, this
>      behavior is clearly unsuitable if the user wants other threads to
>      continue to execute while she stops one for inspection.
> 
> C4) Finally, stopping all threads simplifies GDB's remote protocol.
>      At present, GDB's remote protocol notifies GDB of exactly one
>      thread's state in response to each 'continue' or 'step' operation,
>      permitting no further packets from the stub until GDB resumes
>      some thread.
> 
> C5) GDB breakpoints are currently per-thread or global.  To satisfy
>      our client's requirements, we must adapt these structures to
>      distinguish per-process and global breakpoints, where 'global'
>      breakpoints are set in all attached processes.
> 
> C6) [Our client elected not to address this issue yet.]
> 
> C7) GDB currently operates on a single process at a time: the list of
>      known threads is global, and the ID of the process being debugged
>      is global.  This conflicts with the needs of multi-process
>      debugging.
> 
> C8) GDB currently maintains a single global map of the address space.
>      It cannot represent multiple processes with code and data
>      appearing at different addresses in different processes.  This is
>      not a problem for our client, because code and variables appear at
>      the same addresses in all processes on their system.  However, it
>      is a requirement for multi-process debugging on Linux.
> 
> C9) GDB will not currently relocate different segments of an
>      executable or shared library by different offsets from the
>      addresses they are assigned in the ELF file.  The client's
>      operating system may relocate each section of a load module by a
>      different amount.
> 
> 
> Projects
> 
> This section breaks down the work necessary into well-defined
> engineering tasks.  For each proposed project, we explain the work
> entailed, the benefits provided, and how it depends on other projects,
> if at all.
> 
> 
> P1) Non-stop multi-threaded native debugging
> 
>      This project allows GDB to stop one thread for inspection on a
>      native system while allowing others to run.
> 
>      To prepare GDB to debug one process while other processes continue
>      to run freely (the feature our client is interested in), we will
>      first implement the ability to debug one thread while other
>      threads in that process continue to run freely.
> 
>      As described in C1, C2, and C3, GDB assumes in its user interface
>      and code that no execution occurs while the user is inspecting a
>      thread's state.  This project removes that simplifying assumption.
> 
>      At the user interface level, GDB's Machine Interface ('MI', the
>      command set used by Eclipse) shall behave as follows:
> 
>      - MI shall provide a command to allow the user to choose between
>        the older 'all-stop' and the new 'non-stop' multi-threaded
>        debugging behaviors.  In all-stop mode, GDB shall behave as it
>        does now.  The following points describe non-stop debugging
>        mode.
> 
>      - GDB shall always prompt for and respond to MI commands,
>        regardless of whether any threads are running or not.
> 
>      - When a thread finishes a command like '-exec-next' or
>        '-exec-finish', hits a breakpoint, or encounters a fault, GDB
>        shall stop that thread, without affecting the other threads in
>        the process.
> 
>      - Execution commands like '-exec-continue' and '-exec-step' shall
>        resume only the selected thread, without affecting the other
>        threads in the process.
> 
>      - The MI '-exec-interrupt' command shall stop all threads.  This
>        will always generate an 'EXEC-ASYNC-OUTPUT' record, even if all
>        threads were already stopped.  (This helps users handle the case
>        where the thread stops of its own accord just as the user sends
>        it an '-exec-interrupt' command.)
> 
>      - The MI '-thread-select' command shall stop the thread selected,
>        if it is running.  The previously selected thread is left in its
>        former state, either stopped or running.  A '-thread-select'
>        command shall always generate an 'EXEC-ASYNC-OUTPUT' record,
>        even if the thread was already stopped.
> 
>      - MI shall provide a command to continue all stopped threads.
> 
>      - GDB shall send 'EXEC-ASYNC-OUTPUT' MI records to notify the user
>        of events that have occurred in threads, even while GDB is
>        waiting for an MI command.  Every thread GDB stops shall be
>        mentioned in some 'EXEC-ASYNC-OUTPUT' record; when GDB stops all
>        threads, the EXEC-ASYNC-OUTPUT record shall include a
>        'thread-id="all"' result.
> 
>      - The MI '-thread-info' and '-thread-list-all-threads' commands
>        shall be implemented.  Their output shall indicate whether each
>        thread listed is currently stopped by GDB, or whether it is
>        allowed to run.
> 
>      - GDB shall use 'EXEC-ASYNC-OUTPUT' MI records to report thread
>        creation and termination.  These records shall include the GDB
>        thread number as a result.  After sending a thread termination
>        record, GDB shall not include the thread in the output of
>        '-thread-list-ids' or '-thread-list-all-threads'.
> 
>      (Adapting GDB's command-line interface to non-stop debugging is
>      more involved; whereas MI need only be accurate and sufficient,
>      the command-line interface must also respect human interface
>      issues.  Since GDB's command-line interface is of limited interest
>      to our client, we have not included it here.)
> 
>      To implement the behavior described above, a number of areas
>      within GDB will need modification:
> 
>      - GDB's event loop must be responsive to user input and thread
>        events from the debuggee simultaneously.
> 
>      - GDB's execution control code must avoid stopping all threads
>        when one reports an event, and must make the processing of
>        thread stops independent of resumption: it must no longer assume
>        that events only arrive after resumptions, and resumptions only
>        happen after events.
> 
>      - GDB must insert breakpoints into code being executed by live
>        threads in a manner supported by the target architecture.
> 
>      - GDB's breakpoint support code must leave breakpoints inserted at
>        all times.  Even while GDB steps a thread past a breakpoint,
>        the breakpoint must remain in effect for all other threads.
> 
>      These are each reasonably substantial pieces of work, the design
>      of which should be discussed on the public GDB list to ensure that
>      the work will be acceptable for inclusion in the public sources
>      when it is complete.
> 
> 
> P4) Stub for client's OS
> 
>      This project will mostly be non-GDB work.  However, there are some
>      changes to the remote protocol we would like to introduce at this
>      point:
> 
>      The remote protocol presently leaves the process to be debugged
>      implicit; users generally specify it when they start the stub.
>      However, to satisfy our client's requirements, we must be able to
>      connect to a system, list the processes present, and attach to one
>      of them.  This entails making some straightforward extensions to
>      the GDB remote protocol, and thus to GDB as well.
> 
>      The stub for our client should use the 'library' stop reply
>      packets and the 'qXfer:libraries:read' packet to report load
>      module events.  However, because the client's OS may bring each
>      section of a load module into memory at a different offset from
>      the VMA given in the ELF file, we will need to extend the format
>      of the library list the latter packet returns, as it currently
>      assumes that each library needs only one offset, and extend GDB to
>      allow each segment to appear at a different offset (C9).
> 
> 
> P6) Multi-threaded limited-multi-process native debugging
> 
>      This project provides multi-threaded debugging of multiple
>      processes simultaneously.  The debugger stops all threads in all
>      attached processes while the user inspects the state of any
>      thread.  This work is independent of P1; we combine P1 and P6 in
>      the next project, P7.
> 
>      At the user interface level:
> 
>      - MI shall provide new commands to attach and detach a process;
>        unlike GDB's existing 'attach' and 'detach' commands, the new
>        'attach' command will not require GDB to detach from any
>        currently attached processes.
> 
>      - MI shall provide a command to list all currently attached
>        processes.
> 
>      - MI shall provide a command to list all the threads in a given
>        attached process.
> 
>      - The output of the MI '-thread-info' and
>        '-thread-list-all-threads' commands shall include the process ID
>        of each thread listed.  The process ID shall be a separate MI
>        'result' from the string provided by the
>        'target_extra_thread_info' function, so that Eclipse can access
>        it reliably.
> 
>      - GDB shall stop all threads in all attached processes while
>        interacting with the user.  Attaching to a process shall stop
>        all threads in that process.  Detaching from a process shall
>        allow its threads to run again.
> 
>      - MI shall report faults encountered by threads in any attached
>        process.
> 
>      - MI shall report the termination of any attached process.  After
>        such a report, GDB will no longer be attached to the process.
> 
>      - MI's '-thread-select' command shall be able to select any thread
>        in any attached process.
> 
>      - MI's existing breakpoint commands shall set breakpoints global
>        to all attached processes.
> 
>      To support those facilities, we need the following changes:
> 
>      - GDB shall maintain a table of attached processes.  The remote
>        protocol shall provide packets directing the stub to attach to a
>        new process, and to detach from a currently attached process.
> 
>      - The remote protocol shall carry process IDs as well as thread
>        IDs in stop reply packets, thread selection packets, thread
>        enumeration packets, and wherever else is appropriate.
> 
>      - The stub shall use the current general thread (as given by the
>        'Hg' packet) to determine which process's memory to access, as
>        it does now to determine which thread's registers to access.
>        GDB shall send 'Hg' packets as necessary before memory accesses,
>        as it does now for register accesses.
> 
> 
> P7) Non-stop multi-threaded multi-process native debugging
> 
>      This project allows GDB to attach to multiple processes
>      simultaneously.  This builds on P1 and P6, and addresses C4 and
>      C7.
> 
>      At the user interface level:
> 
>      - MI shall provide a command to stop all the threads in a given
>        process, and a command to resume all stopped threads in a given
>        process.
> 
>      Internally:
> 
>      - The remote protocol shall provide a way to tell the stub to
>        leave other threads running after reporting an event in one
>        thread (non-stop behavior), and a way to tell the stub to stop
>        all threads when reporting an event in one thread (stop-all
>        behavior).
> 
>      - The remote protocol shall allow the stub to respond to
>        commands while threads are running, and to report further thread
>        events after a thread has stopped.  (This addresses C4.)
> 
>      - The remote protocol shall provide ways to stop and start a
>        particular thread, and ways to start and stop all the threads in
>        a given process.  The mechanisms for stopping threads and
>        processes shall allow GDB to behave correctly when a thread
>        stops or a process exits simultaneously with GDB sending the
>        command.
> 
> nathan

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Nathan Sidwell <nathan at codesourcery.com> writes:
> A client of CodeSourcery's has contracted with us to implement a
> number of new features in GDB, some of which have been on the
> frequently requested list for quite some time:

We now have permission from our client to credit them.  Ericsson is
sponsoring our work on this project.

Re: Non-stop multi-threaded debugging

[Ulrich Weigand]

Nathan Sidwell wrote:

> A client of CodeSourcery's has contracted with us to implement a
> number of new features in GDB, some of which have been on the
> frequently requested list for quite some time:

Thanks, these are indeed very interesting and useful features.

> - We're to implement a limited form of multi-process debugging.
> 
>    Full multi-process debugging would entail changes to
>    1) process management code,
>    2) target interfaces, and
>    3) symbol tables.

> Our client would very much like for this work to be incorporated into
> the public GDB sources (although they understand that the decision is
> in the public project's hands), so we'll be posting our design
> thoughts for general discussion.  In particular, I believe the
> multi-process work may overlap with some of the work IBM has done to
> support the Cell processor; we'd very much like to work with IBM to
> ensure that the final model is appropriate for both our client and for
> Cell developers.

It looks like the points 1) and 2) you are planning to address 
actually do *not* overlap with the work we've been doing on the
Cell combined debugger.  The Cell programming model our debugger
supports is bascially a regular multi-threaded model, except that
some threads may execute either PowerPC or SPU architecture code
at any time.  Multi-process support is not required.

[ However, of course multi-process support would be a nice *extension*
to the Cell combined debugger, allowing to debug multiple processes,
each of which follows the normal multi-threaded Cell model. ]

Point 3) above would be relevant to the Cell combined debugger as well.
You do not address this point in more detail in your document, but I
assume this would imply introducing the notion of "address spaces" and
associating symbol tables to a particular address space.  Something
similar is already needed for the Cell combined debugger, as we have
multiple address spaces even within a single process.  I'm definitely
interested in working on adding this support to mainline GDB -- if
that then also helps the multi-process work, that's an additional
benefit.


Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: Non-stop multi-threaded debugging

[Thiago Jung Bauermann]

Hi,

On Tue, 2007-11-20 at 17:21 +0000, Nathan Sidwell wrote:
> - We're to implement non-stop multi-threaded debugging in GDB.

[snip]

> Our client is only interested in the MI interface; they intend to use
> all these facilities via Eclipse.  So we will not be implementing
> command-line support any more than is helpful to us in development.
> But again, we want to do this work in a way that leaves CLI support
> for these features a simple matter of coding, so that our work is
> still forward progress, which anyone can complete.

That's great news.
I'm interested in seeing non-stop multi-thread debugging support working
in Linux and in having it usable via the command line interface, so I'm
willing to help in those areas (not sure how much work/test you'll be
doing in Linux, since you mention your focus is the client's OS).

I'll keep an eye on the design discussions, and especially the "it would
be nice if someone would implement this" parts for Linux/CLI..
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Thiago Jung Bauermann <bauerman at br.ibm.com> writes:
> That's great news.
> I'm interested in seeing non-stop multi-thread debugging support working
> in Linux and in having it usable via the command line interface, so I'm
> willing to help in those areas (not sure how much work/test you'll be
> doing in Linux, since you mention your focus is the client's OS).
>
> I'll keep an eye on the design discussions, and especially the "it would
> be nice if someone would implement this" parts for Linux/CLI..

That sounds great!

Here is my sketch of what would be needed for a CLI interface.  Since
it wasn't something Ericsson was interested in, I wasn't as careful to
keep it up to date as the rest of the document changed, but hopefully
it will still be useful.

As with the MI commands, folks should read this as an attempt to
mention the issues that need to be addressed, not as a fully
worked-out design.  If there are things that seem clumsy, and you
think you have a better idea, you're probably right.


P10) Non-stop debugging via the command-line interface

    This project makes non-stop multi-threaded debugging
    available via the GDB command-line interface.  This was not a
    stated Ericsson requirement.  It builds on P1.

    An IDE using a graphical user interface can support non-stop
    multi-threaded debugging in a natural way.  A command-line
    interface, however, encounters some challenges:

    - One of the attractive characteristics of a non-stop debugger is
      that the debugger is always ready to respond to the user,
      regardless of what the debuggee is doing.  However, commands
      like 'stop', 'next' and 'finish' traditionally indicate that the
      operation has completed simply by withholding the command prompt
      until that point.

    - In non-stop debugging, events of interest to the user may occur
      in the debuggee's running threads while the user is inspecting a
      stopped thread.  Because a command-line interface cannot
      segregate event notifications and user interaction in separate
      areas of the display the way a GUI can, a flood of events in the
      debuggee can obscure the results of prior commands and make it
      difficult for the user to make headway.

    The following changes would allow GDB to support non-stop
    debugging on the command line in a way that addresses those
    problems:

    - The CLI shall provide commands to switch between all-stop mode
      (the current behavior) and non-stop mode.

    - The CLI 'info threads' command shall indicate, for each thread,
      whether it is currently stopped or running.  The information
      displayed for running threads may be limited.

    - Selecting any thread shall cause that thread to stop.

    - When a thread encounters a breakpoint or a fault, GDB shall stop
      only that thread.  Any other running threads shall continue to
      execute.

    - Resumption commands like 'step' and 'finish' shall affect only
      the selected thread; any other stopped threads will remain
      stopped.  The CLI shall not prompt the user for a new command
      until the operation completes, or an event occurs in any thread,
      or the user sends an interrupt to the terminal.  (This allows
      the presence of the prompt to continue to indicate when a
      command has completed, as it does in all-stop mode.)

    - The CLI's 'continue' command shall print a message indicating
      that threads are running, and then immediately prompt the user
      for the next command.

    - At a given command prompt, the CLI shall display only a limited
      number of lines of event notifications.  If the limit has been
      reached and more events occur, the CLI will display a final
      message indicating that further events are available to report,
      and explain briefly what the user must do to see them.
    
    - Sending an interrupt to the terminal at any time shall cause the
      CLI to prompt the user for a new command, but shall not stop any
      threads.  The CLI shall print a message to this effect,
      explaining briefly how the user may stop all threads if needed.

    - The CLI shall provide commands to stop all threads, and to
      continue all threads.

Re: Non-stop multi-threaded debugging

[Fabian Cenedese]

>   - Selecting any thread shall cause that thread to stop.

Does gdb always have to have a selected thread? I'm wondering how
I could watch a global variable while all threads are running. So either
gdb can inspect variables without thread or selecting a thread shouldn't
stop it.

>    - When a thread encounters a breakpoint or a fault, GDB shall stop
>      only that thread.  Any other running threads shall continue to
>      execute.

I assume this is for both global and thread breakpoints.

Thanks

bye  Fabi

Re: Non-stop multi-threaded debugging

[Nigel Stephens]

Nathan Sidwell wrote:
> 
> Hi all,
> 
> Jim Blandy prepared this, but is on vacation this week.  So, I'm
> announcing it 
> in his absence.  Pretend I wrote 'sudo jimb ...'
> 
> A client of CodeSourcery's has contracted with us to implement a
> number of new features in GDB, some of which have been on the
> frequently requested list for quite some time:
> 

Cool. We'll be very interested in this work, particularly as it applies to
our CPUs with h/w multi-threading.

Nigel
-- 
View this message in context: http://www.nabble.com/Non-stop-multi-threaded-debugging-tf4844837.html#a14179465
Sent from the Sourceware - gdb list mailing list archive at Nabble.com.