Porting gdb to Cyclops64

Porting gdb to Cyclops64

[Brian Heilig]

Dear list,

My name is Brian Heilig and I am porting gdb 7.1-90 to the Cyclops64
architecture. Cyclops64 has 80 cores and each core has 2 thread units
for a total of 160 threads units, all running in parallel. Each thread
unit executes the same code and can access shared memory or its own
thread memory (http://en.wikipedia.org/wiki/SPMD). A traditional thread
model is also supported where SPMD threads can spawn other threads.

I have integrated a gdb server into our kernel and integrated the target
architecture into the gdb client. Right now I can remote debug a program
on the target but it doesn't handle multiple threads well.

When multiple threads are stopped at a breakpoint (or stopped for some
other reason) I should be able to step through the current thread. When
I do gdb is commanding the target to step the current thread and
continue all others like "$vCont;s:1;c".

The plan is to step the current thread and leave all others stopped
until I tell it to continue. If multiple threads stop at a breakpoint
then the target will queue these events. When the target is told to
continue it will notify the client that the next thread has stopped at a
breakpoint. If the user desires they can switch to another thread that
is stopped and step it.

Does gdb support the operation 'step the current thread and leave all
other threads stopped'? Is there some mode or variable I need to set in
the target architecture description to enable this? What is gdb's
behavior now for multi-core architectures?

Thanks,
Brian

Re: Porting gdb to Cyclops64

[Michael Snyder]

Brian Heilig wrote:
> Dear list,
> 
> My name is Brian Heilig and I am porting gdb 7.1-90 to the Cyclops64
> architecture. Cyclops64 has 80 cores and each core has 2 thread units
> for a total of 160 threads units, all running in parallel. Each thread
> unit executes the same code and can access shared memory or its own
> thread memory (http://en.wikipedia.org/wiki/SPMD). A traditional thread
> model is also supported where SPMD threads can spawn other threads.
> 
> I have integrated a gdb server into our kernel and integrated the target
> architecture into the gdb client. Right now I can remote debug a program
> on the target but it doesn't handle multiple threads well.
> 
> When multiple threads are stopped at a breakpoint (or stopped for some
> other reason) I should be able to step through the current thread. When
> I do gdb is commanding the target to step the current thread and
> continue all others like "$vCont;s:1;c".
> 
> The plan is to step the current thread and leave all others stopped
> until I tell it to continue. If multiple threads stop at a breakpoint
> then the target will queue these events. When the target is told to
> continue it will notify the client that the next thread has stopped at a
> breakpoint. If the user desires they can switch to another thread that
> is stopped and step it.
> 
> Does gdb support the operation 'step the current thread and leave all
> other threads stopped'? Is there some mode or variable I need to set in
> the target architecture description to enable this? What is gdb's
> behavior now for multi-core architectures?
> 
> Thanks,
> Brian

Yes, see "set scheduler-locking".

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gdb/thread-stops.html

Re: Porting gdb to Cyclops64

[Brian Heilig]

On Fri, 2010-07-30 at 10:26 -0700, Michael Snyder wrote:
> Brian Heilig wrote:
> > 
> > Does gdb support the operation 'step the current thread and leave all
> > other threads stopped'?
> 
> Yes, see "set scheduler-locking".
> 
> http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gdb/thread-stops.html
> 

Thank you.

When I attach to my remote target, if the target is running (not stopped
at a breakpoint) gdb currently doesn't know it and presents the gdb
prompt as if the inferior were stopped. Clearly I've not implemented the
protocol correctly in this case.

Should I stop the inferior when the client is attached? Or should I send
a response from my target that will indicate to gdb that it is currently
running?

Is there a reference for the most recent remote protocol? I'm sure I've
only implemented a small subset of the protocol, and that probably isn't
%100 correct.

Thanks again,
Brian

Re: Porting gdb to Cyclops64

[Michael Snyder]

Brian Heilig wrote:
> On Fri, 2010-07-30 at 10:26 -0700, Michael Snyder wrote:
>> Brian Heilig wrote:
>>> Does gdb support the operation 'step the current thread and leave all
>>> other threads stopped'?
>> Yes, see "set scheduler-locking".
>>
>> http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gdb/thread-stops.html
>>
> 
> Thank you.
> 
> When I attach to my remote target, if the target is running (not stopped
> at a breakpoint) gdb currently doesn't know it and presents the gdb
> prompt as if the inferior were stopped. Clearly I've not implemented the
> protocol correctly in this case.
> 
> Should I stop the inferior when the client is attached? Or should I send
> a response from my target that will indicate to gdb that it is currently
> running?

The default mode is synchronous debugging, which means that any time
you are looking at the gdb prompt, the target is stopped.

It would probably be easiest to implement that first and get it working,
before attempting asynchronous mode.

So yes, attaching with gdb should stop the target.

> Is there a reference for the most recent remote protocol? I'm sure I've
> only implemented a small subset of the protocol, and that probably isn't
> %100 correct.

You certainly don't have to implement everything all at once.

Try this for a reference:


http://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Protocol.html#Remote-Protocol

Re: Porting gdb to Cyclops64

[Brian Heilig]

On Fri, 2010-07-30 at 11:57 -0700, Michael Snyder wrote:
> The default mode is synchronous debugging, which means that any time
> you are looking at the gdb prompt, the target is stopped.
> 
> It would probably be easiest to implement that first and get it working,
> before attempting asynchronous mode.
> 
> So yes, attaching with gdb should stop the target.

Thanks again Michael.

Since all threads execute in parallel, more than one thread can hit a
breakpoint. In stop mode I should stop all threads when the first thread
hits a breakpoint. While stopping all threads other threads might hit a
breakpoint. My intention is to queue these "events" and report them in
sequential order to gdb. So that, in stop mode, when gdb tells the
target to continue it will actually dequeue the next event. If there are
no events then it will continue. Does this sound correct?

I guess I will have to implement non-stop mode as well since many
threads can hit a breakpoint. It would be very inconvenient for my user
to have to continue each thread individually. In this case I will not
queue events but rather just send them to gdb and let gdb sort them out.
Is this correct as well?

Regards,
Brian

Re: Porting gdb to Cyclops64

[Michael Snyder]

Brian Heilig wrote:
> On Fri, 2010-07-30 at 11:57 -0700, Michael Snyder wrote:
>> The default mode is synchronous debugging, which means that any time
>> you are looking at the gdb prompt, the target is stopped.
>>
>> It would probably be easiest to implement that first and get it working,
>> before attempting asynchronous mode.
>>
>> So yes, attaching with gdb should stop the target.
> 
> Thanks again Michael.
> 
> Since all threads execute in parallel, more than one thread can hit a
> breakpoint. In stop mode I should stop all threads when the first thread
> hits a breakpoint. While stopping all threads other threads might hit a
> breakpoint. My intention is to queue these "events" and report them in
> sequential order to gdb. So that, in stop mode, when gdb tells the
> target to continue it will actually dequeue the next event. If there are
> no events then it will continue. Does this sound correct?

That's very similar to the way it works on native Linux.
Not because of true parallelism, but because we can't really
stop all the threads and start them again simultaneously.

> I guess I will have to implement non-stop mode as well since many
> threads can hit a breakpoint. It would be very inconvenient for my user
> to have to continue each thread individually. In this case I will not
> queue events but rather just send them to gdb and let gdb sort them out.
> Is this correct as well?

I'm not sure -- maybe someone else will care to jump in.

Re: Porting gdb to Cyclops64

[Pedro Alves]

On Friday 30 July 2010 21:28:53, Brian Heilig wrote:
> Since all threads execute in parallel, more than one thread can hit a
> breakpoint. In stop mode I should stop all threads when the first thread
> hits a breakpoint. While stopping all threads other threads might hit a
> breakpoint. My intention is to queue these "events" and report them in
> sequential order to gdb. So that, in stop mode, when gdb tells the
> target to continue it will actually dequeue the next event. If there are
> no events then it will continue. Does this sound correct?

It does.

Only a little care should be taken to dequeue events for
threads that gdb is actually interested in.  Say, while stepping all
threads, some of the other threads hit breakpoints.  The next time the
user continues, gdb will want to step over the breakpoint that the
thread had reported being hit.  So, gdb removes this breakpoint and tells
that single thread to single-step over this location.  At this point, gdb
hadn't told any of the other threads to resume, so you shouldn't
dequeue any pending event for them yet.  (the same needs considering if
the user switches on scheduler-locking and resumes only one
thread --- your target should not report pending events for threads
that hadn't been resumed)

(this is what the lwp->resumed flag in gdb/linux-nat.c or
'struct thread_info'->last_resume_kind in gdb/gdbserver/server.c
implement)

> I guess I will have to implement non-stop mode as well since many
> threads can hit a breakpoint. It would be very inconvenient for my user
> to have to continue each thread individually. In this case I will not
> queue events but rather just send them to gdb and let gdb sort them out.
> Is this correct as well?

Not sure what you mean --- the user would still have to resume each
thread individually.  But yes, in non-stop mode, your target should 
let the other threads that hadn't hit any breakpoint continue running
free.

-- 
Pedro Alves

Re: Porting gdb to Cyclops64

[Brian Heilig]

On Sat, 2010-07-31 at 00:17 +0100, Pedro Alves wrote:
> Not sure what you mean --- the user would still have to resume each
> thread individually.  But yes, in non-stop mode, your target should 
> let the other threads that hadn't hit any breakpoint continue running
> free.

My understanding is that in non-stop mode I can "continue all", whereas
in stop mode I can only continue the current thread. This would mean
that, in stop mode, each thread would have to be continued in
succession.

Brian

Re: Porting gdb to Cyclops64

[Pedro Alves]

On Monday 02 August 2010 14:24:54, Brian Heilig wrote:
> On Sat, 2010-07-31 at 00:17 +0100, Pedro Alves wrote:
> > Not sure what you mean --- the user would still have to resume each
> > thread individually.  But yes, in non-stop mode, your target should 
> > let the other threads that hadn't hit any breakpoint continue running
> > free.
> 
> My understanding is that in non-stop mode I can "continue all", whereas
> in stop mode I can only continue the current thread. This would mean
> that, in stop mode, each thread would have to be continued in
> succession.

In all-stop mode, what's resumed depends on the "set scheduler-locking"
setting.  By default, gdb tells the target to continue all
threads (vCont;c).

-- 
Pedro Alves

Re: Porting gdb to Cyclops64

[Brian Heilig]

Guys, thanks again for all your help. My target is reporting events in a
sane order in stop mode now but I've run into a snag.

While debugging the following program:

#include <stdio.h>                      /* line 1  */
void foo(int *a) {                      /* line 2  */
  *a = 9;                               /* line 3  */
}                                       /* line 4  */
int main() {                            /* line 5  */
  int a=0;                              /* line 6  */
  foo(&a);                              /* line 7  */
  printf("%d\n",a);                     /* line 8  */
  return 0;                             /* line 9  */
}

I set a breakpoint at line 7. All threads hit that breakpoint. I stepped
thread 1 and continued the other threads.

[Switching to Thread 1]

Breakpoint 1, main () at step.c:7
7	  foo(&a);                              /* line 7  */
(gdb) s
[Switching to Thread 2]

Breakpoint 1, main () at step.c:7
7	  foo(&a);                              /* line 7  */
(gdb) c
Continuing.
[Switching to Thread 3]

... etc.

(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 1]
0x0000000080800274 in main () at step.c:7
7	  foo(&a);                              /* line 7  */

I switched back to thread 1 and step again:

(gdb) s
foo (a=0x41103ae8) at step.c:3
3	  *a = 9;                               /* line 3  */
(gdb) 

I step one more time and gdb switches to thread 2. Why is that?

(gdb) s

Program received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 2]
foo (a=0x41803ae8) at step.c:3
3	  *a = 9;                               /* line 3  */

It turns out that when I stepped to line 3 (the second step) gdb
actually set a breakpoint at line 3. This might be because 'foo' has
been inlined. Then gdb told my target to continue all "vCont;c". All
threads hit the breakpoint, but only thread 1 should have.

I don't know how it's supposed to work, but gdb could have sent
"vCont;c:1". Or gdb could have single stepped until it reached the
correct address (obviously not the preferred method). How can I correct
this?

Thanks,
Brian

Re: Porting gdb to Cyclops64

[Brian Heilig]

Dear list,

In my previous message I described a problem I encountered while
stepping a multi-threaded program. Here is a general description of
what's happening.

All threads execute in parallel (on their own core) on the Cyclops64
chip. I implemented a queue on the gdb server to serialize events. This
makes Cyclops64 look like a cooperatively scheduled architecture to gdb
in stop mode.

While single stepping a single thread (let's call it thread 5) gdb
decided to set a breakpoint at some address and continue all threads.
All threads hit this internal breakpoint including thread 5. The
breakpoint events for threads 1-4 are reported and ignored by gdb (gdb
knows it doesn't care about these threads). Then thread 5 is reported
and gdb removes the breakpoint. However the event for threads 6-n are
still in the queue. The next time the user continues these events are
reported. Since gdb removed the internal breakpoint it assumes these
events must be caused by something else and doesn't filter them out.

Is there anything I can do about this? Is there a facility to set a
breakpoint for a single thread and filter them on the server? Or is this
just a consequence of gdb not knowing how to deal with an SPMD
architecture yet?

Thank You,
Brian

Re: Porting gdb to Cyclops64

[Pedro Alves]

On Thursday 05 August 2010 14:56:48, Brian Heilig wrote:

> In my previous message I described a problem I encountered while
> stepping a multi-threaded program. Here is a general description of
> what's happening.
> 
> All threads execute in parallel (on their own core) on the Cyclops64
> chip. I implemented a queue on the gdb server to serialize events. This
> makes Cyclops64 look like a cooperatively scheduled architecture to gdb
> in stop mode.
> 
> While single stepping a single thread (let's call it thread 5) gdb
> decided to set a breakpoint at some address and continue all threads.
> All threads hit this internal breakpoint including thread 5. The
> breakpoint events for threads 1-4 are reported and ignored by gdb (gdb
> knows it doesn't care about these threads). Then thread 5 is reported
> and gdb removes the breakpoint. However the event for threads 6-n are
> still in the queue. The next time the user continues these events are
> reported. Since gdb removed the internal breakpoint it assumes these
> events must be caused by something else and doesn't filter them out.
> 
> Is there anything I can do about this?

Yes.  I'll suggest two alternatives.

 #1 - before reporting a delayed/queued breakpoint hit event, check
   whether there's still a breakpoint still inserted where the
   thread is stopped.  If there isn't, then just resume the thread
   without telling gdb about the stop.

 #2 - don't leave breakpoint hit events queued (all other events, yes,
   leave them).  When you've stopped all threads to report a breakpoint
   hit to gdb, go through all threads, and those that have stopped
   due to a breakpoint, simply forget their breakpoint hits.  The
   next time gdb resumes those threads, they'll immediately hit
   the same breakpoint again.  You'll most likely want to
   add a bit of randomness to which of the threads amongs those
   that have hit a breakpoint you're going to report to gdb.

#2 is what both native linux gdb and linux gdbserver implement
nowadays.  gdbserver used to implement #1 until recently.
#1 has the advantage that it's a bit more efficient since it
avoids resuming the target when not strictly necessary.  The
disadvantage is that it makes managing the PC adjustment after
breakpoint traps more complicated.  E.g., the stub needs extra
care to present to GDB already adjusted PCs for all those
threads that still have pending breakpoint hits to report to gdb
in the event queue, but, to unadjust them when finally report
the queued breakpoint hit, so that gdb itself does the necessary
adjustment.  This may not be a concern on Cyclops64, I don't know.

> Is there a facility to set a
> breakpoint for a single thread and filter them on the server? 

Nope, it's one of those IWBN things that nobody ever got around to
working on, AFAIK.

> Or is this
> just a consequence of gdb not knowing how to deal with an SPMD
> architecture yet?

Nope.

-- 
Pedro Alves

Problems with non-stop - Porting gdb to Cyclops64

[Brian Heilig]

Dear list,

I am having trouble making gdb happy in non-stop mode. When I connect to
my target in non-stop mode gdb queries for the list of active threads.
It then sends a stop request for the first thread in the list,
"vCont;t:1".

The documentation for this feature states: "A stop reply should be
generated for any affected thread not already stopped. When a thread is
stopped by means of a `t' action, the corresponding stop reply should
indicate that the thread has stopped with signal `0', regardless of
whether the target uses some other signal as an implementation detail."

So I send back my stop response: "$T00;thread:1;". gdb responds with
"Stopping Thread 1 failed: T00;thread:1;" and disconnects from the
target.

I tried this same test on my Linux desktop using version 7.1-ubuntu and
received a very different response.

Server: gdbserver :0 a.out
Client: gdb a.out

Set non-stop on the client and then connect with "target
remote :<port>".

Client sends "vCont;t:<thread>"

Server responds with "$OK". The gdb prompt never appears and (as far as
I can tell) nothing can be done with the gdb session.

Am I doing something wrong?

Thanks,
Brian

Cyclops64 Multi-Process

[Brian Heilig]

Dear list,

I have successfully ported gdb 7.1 to Cyclops64 and it is quite stable.
My next task is to implement multi-processor debugging for use in a
super computer. There can be many processors connected through some
interface, each processor having many parallel threads, and all of them
executing the same program.

It would be easy for me to allow gdb to assign a thread id to each
thread on every processor as the stop event arrives. But there can be
100,000+ threads and users tend to think of each thread in terms of the
node that it executes on. Therefore the multiprocess extensions of gdb
seemed like a nice fit because of the 'process.thread' syntax.

The gdb client accepts the stop events for the first pid, but any stop
event thereafter causes the following error: "internal-error: Can't
determine the current address space of thread Thread 2.1"

Do you think shoehorning multiprocess extensions into my multi processor
system is a good idea? Is there something I can manipulate in the
protocol to make the gdb client happy?

Thank you,
Brian