Thread Specific Breakpoints in Remote Targets

Thread Specific Breakpoints in Remote Targets

[Josh Watt]

Hello,

I'm working on implementing an GDB remote stub for a project, and I
had a question
about the behavior of GDB.

For starters, I'm running GDB 7.3a compiled from vanilla source using
MinGW on Windows
XP with the following configure options:
--target=arm-eabi

The target device runs a custom RTOS, and I was able to get the remote
stub to correctly
report the OS threads back to GDB, thus allowing GDB to examine the
individual threads.

My question in particular is related to Thread Specific breakpoints.
Our remote stub has
the ability to internally implement thread specific breakpoints which
saves a lot of overhead
when the breakpoints are in frequently visited locations, since it
means that we don't have
to talk over the slow serial link to GDB just for it to tells us we
can continue execution.

However, it appears that GDB is doing some weird things with regard to
the thread selection.
See the log below:

(gdb) info threads
  Id   Target Id         Frame
  41   Thread 1040 ("Thread 1040") at ...
  40   Thread 1039 ("Thread 1039") at ...
<output truncated for brevity>
  4    Thread 1004 ("Thread 1004") at ...
  3    Thread 1003 ("Thread 1003") at ...
  2    Thread 1002 ("Thread 1002") at ...
* 1    Thread 1000 ("Thread 1000") at ...
(gdb) break *$pc thread 1
Breakpoint 1 at 0x80232ce4: file ..., line 134.
(gdb) set debug remote 1
(gdb) cont
Continuing.
Sending packet: $vCont?#49...Ack
Packet received:
Packet vCont (verbose-resume) is NOT supported
Sending packet: $Hc3e8#7b...Ack
Packet received: OK
Sending packet: $s#73...Ack
Packet received: T05thread:3e8;0:d05da48c;1:02000000;2:01000000;3:00000000;4:000
00000;5:00000000;6:00000000;7:00000000;8:00000000;9:00000000;A:00000000;B:000000
00;C:00000000;D:00a9b782;E:69ec4080;F:e42c2380;19:3f000000;
Sending packet: $Hg410#44...Ack
Packet received: OK
Sending packet: $g#67...Ack
Packet received: 000000000000000000000000000000000100000000000000000000000000000
00000000000000000000000000000000023280900086d6c87c1ef238068f04080
Sending packet: $Z0,80232ce2,2#0d...Ack
Packet received: OK
Sending packet: $Hg3e8#7f...Ack
Packet received: OK
Sending packet: $g#67...Ack
Packet received: d05da48c0200000001000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000a9b78269ec4080e42c2380
Sending packet: $Hc0#db...Ack
Packet received: OK
Sending packet: $c#63...Ack
Packet received: T05thread:3e8;0:e0302380;1:02000000;2:01000000;3:00000000;4:000
00000;5:00000000;6:00000000;7:00000000;8:00000000;9:00000000;A:00000000;B:000000
00;C:00000000;D:00a9b782;E:69ec4080;F:e22c2380;19:3f000000;
Sending packet: $Hg410#44...Ack
Packet received: OK
Sending packet: $g#67...Ack
Packet received: 000000000000000000000000000000000100000000000000000000000000000
00000000000000000000000000000000023280900086d6c87c1ef238068f04080
Sending packet: $z0,80232ce2,2#2d...Ack
Packet received: OK
Sending packet: $Hg3e8#7f...Ack
Packet received: OK
Sending packet: $g#67...Ack
Packet received: e03023800200000001000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000a9b78269ec4080e22c2380

Breakpoint 1, Sending packet: $m80232ce2,4#c6...Ack
Packet received: ff4803c8
0x80232ce2 in ...
(gdb)


As can been seen from the log, the stub is sending a message to switch
to thread 1040 ($Hg410#44)
right before setting the breakpoint (and again before deleting it). In
subsequent operation, it is apparent
that it is always switching to this thread when setting and clearing a
breakpoint. Because of this,
our remote stub cannot rely on the currently selected thread as the
target thread for a given breakpoint
and must communicate with GDB every time a breakpoint is hit.

I also find it interesting that the selected thread is number 1040,
which is the last thread reported to GDB
by the remote stub.

I looked through some of the code, and I think it is due to the following:

In breakpoint.c there are calls to switch_to_program_space_and_thread() at
lines 1895 and 2662 which must be selecting the wrong thread, but I'm not sure
how to make it select the correct thread (if it is even supposed to).


Thanks,
--
~JPEW

Re: Thread Specific Breakpoints in Remote Targets

[Tom Tromey]

>>>>> "Josh" == Josh Watt <jpewdev@gmail.com> writes:

Josh> As can been seen from the log, the stub is sending a message to
Josh> switch to thread 1040 ($Hg410#44) right before setting the
Josh> breakpoint (and again before deleting it). In subsequent
Josh> operation, it is apparent that it is always switching to this
Josh> thread when setting and clearing a breakpoint.

FWIW I found your note very clear, thanks for the dump and background
info.

Josh> Because of this, our remote stub cannot rely on the currently
Josh> selected thread as the target thread for a given breakpoint and
Josh> must communicate with GDB every time a breakpoint is hit.

I did not understand this though.

It sounds like you are making breakpoints on the target thread-specific
based on the current thread.  But I thought we didn't (yet) have a way
to inform the target that a given breakpoint was thread-specific (but I
don't know this area extremely well -- if I'm wrong I'd like to know
about it).

Josh> I looked through some of the code, and I think it is due to the
Josh> following:

Josh> In breakpoint.c there are calls to
Josh> switch_to_program_space_and_thread() at lines 1895 and 2662 which
Josh> must be selecting the wrong thread, but I'm not sure how to make
Josh> it select the correct thread (if it is even supposed to).

I think you are correct.  switch_to_program_space_and_thread selects the
first (meaning first in gdb's internal table) live thread of the
first inferior bound to that program space.

I am not sure whether this choice particularly matters to all callers of
switch_to_program_space_and_thread.  However, I think it probably does not
matter to callers from breakpoint.c; I think those could safely
short-circuit switching if the current program space is already correct
and if the current thread is live.

This, I think, will fix your immediate problem but I can't claim the
result will really be correct.  I think it would be preferable to
implement real target support for thread-specific breakpoints.

Tom

Re: Thread Specific Breakpoints in Remote Targets

[Pedro Alves]

On Wednesday 31 August 2011 15:47:32, Tom Tromey wrote:
> >>>>> "Josh" == Josh Watt <jpewdev@gmail.com> writes:
> 
> Josh> As can been seen from the log, the stub is sending a message to
> Josh> switch to thread 1040 ($Hg410#44) right before setting the
> Josh> breakpoint (and again before deleting it). In subsequent
> Josh> operation, it is apparent that it is always switching to this
> Josh> thread when setting and clearing a breakpoint.
> 
> FWIW I found your note very clear, thanks for the dump and background
> info.
> 
> Josh> Because of this, our remote stub cannot rely on the currently
> Josh> selected thread as the target thread for a given breakpoint and
> Josh> must communicate with GDB every time a breakpoint is hit.
> 
> I did not understand this though.
> 
> It sounds like you are making breakpoints on the target thread-specific
> based on the current thread.  But I thought we didn't (yet) have a way
> to inform the target that a given breakpoint was thread-specific (but I
> don't know this area extremely well -- if I'm wrong I'd like to know
> about it).

You're right, we don't.

> I think it would be preferable to
> implement real target support for thread-specific breakpoints.

Very much.

Also:

> Sending packet: $vCont?#49...Ack
> Packet received:
> Packet vCont (verbose-resume) is NOT supported
...
> Sending packet: $s#73...Ack
...

Please implement vCont support in your stub.  s and c
are deprecated on multi-threaded targets.  There's no
way to make them work correctly in some cases.

-- 
Pedro Alves

Re: Thread Specific Breakpoints in Remote Targets

[Josh Watt]

>> It sounds like you are making breakpoints on the target thread-specific
>> based on the current thread.  But I thought we didn't (yet) have a way
>> to inform the target that a given breakpoint was thread-specific (but I
>> don't know this area extremely well -- if I'm wrong I'd like to know
>> about it).
>
> You're right, we don't.
>
>> I think it would be preferable to
>> implement real target support for thread-specific breakpoints.
>
> Very much.

Is there a particular reason why the select thread packet (Hg) cannot
also control which thread a breakpoint is targeted for?

--
~JPEW

Re: Thread Specific Breakpoints in Remote Targets

[Pedro Alves]

On Wednesday 31 August 2011 19:29:44, Josh Watt wrote:
> >> It sounds like you are making breakpoints on the target thread-specific
> >> based on the current thread.  But I thought we didn't (yet) have a way
> >> to inform the target that a given breakpoint was thread-specific (but I
> >> don't know this area extremely well -- if I'm wrong I'd like to know
> >> about it).
> >
> > You're right, we don't.
> >
> >> I think it would be preferable to
> >> implement real target support for thread-specific breakpoints.
> >
> > Very much.
> 
> Is there a particular reason why the select thread packet (Hg) cannot
> also control which thread a breakpoint is targeted for?

There's that backwards compatibility thing.  You'd have to come up with
a way to get the current behavior of "breakpoint applies to all threads
of this process".  You'd need to add some new meaning to something
like Hg0/HgPID.0, which is close, but that's not what it means
today, so without other changes, you'd be left with the target not
knowing what this particular gdb is trying to say (because you don't
know whether the connecting gdb understands "thread specific" or not).

-- 
Pedro Alves

Re: Thread Specific Breakpoints in Remote Targets

[Raphael Zulliger]

On 31.08.2011 20:09, Pedro Alves wrote:
> On Wednesday 31 August 2011 15:47:32, Tom Tromey wrote:
>>>>>>> "Josh" == Josh Watt<jpewdev@gmail.com>  writes:
>> Josh>  As can been seen from the log, the stub is sending a message to
>> Josh>  switch to thread 1040 ($Hg410#44) right before setting the
>> Josh>  breakpoint (and again before deleting it). In subsequent
>> Josh>  operation, it is apparent that it is always switching to this
>> Josh>  thread when setting and clearing a breakpoint.
>>
>> FWIW I found your note very clear, thanks for the dump and background
>> info.
>>
>> Josh>  Because of this, our remote stub cannot rely on the currently
>> Josh>  selected thread as the target thread for a given breakpoint and
>> Josh>  must communicate with GDB every time a breakpoint is hit.
>>
>> I did not understand this though.
>>
>> It sounds like you are making breakpoints on the target thread-specific
>> based on the current thread.  But I thought we didn't (yet) have a way
>> to inform the target that a given breakpoint was thread-specific (but I
>> don't know this area extremely well -- if I'm wrong I'd like to know
>> about it).
> You're right, we don't.
Just to mention that: My company would be very interested in (optional) 
'thread specific breakpoints' support for remote targets. gdb could ask 
a gdbstub whether it supports this feature (by the qSupported packet).

In our case, our proprietary real-time OS already offers support for 
'thread specific breakpoints' and it is definitely not an option for our 
system to use the 'thread specific breakpoint emulation' performed by 
the gdb frontend today as it would disrupt real-time behavior. The lack 
of this feature causes major troubles for us during single-stepping, 
where temporary (global) breakpoints are set by the gdb frontend... to 
circumvent this "problem", I had to slightly extend the remote protocol 
for real 'thread specific breakpoints' (I just added a new breakpoint 
type (5) that also passes the Pid/Tid to the stub and which is used 
during step operations). I really hope we can make this feature (however 
it'll be implemented) part of the original gdb sometime in the future!
>
>> I think it would be preferable to
>> implement real target support for thread-specific breakpoints.
> Very much.
>
> Also:
>
>> Sending packet: $vCont?#49...Ack
>> Packet received:
>> Packet vCont (verbose-resume) is NOT supported
> ...
>> Sending packet: $s#73...Ack
> ...
>
> Please implement vCont support in your stub.  s and c
> are deprecated on multi-threaded targets.  There's no
> way to make them work correctly in some cases.
>

Re: Thread Specific Breakpoints in Remote Targets

[Josh Watt]

> There's that backwards compatibility thing.  You'd have to come up with
> a way to get the current behavior of "breakpoint applies to all threads
> of this process".  You'd need to add some new meaning to something
> like Hg0/HgPID.0, which is close, but that's not what it means
> today, so without other changes, you'd be left with the target not
> knowing what this particular gdb is trying to say (because you don't
> know whether the connecting gdb understands "thread specific" or not).

Makes sense.

Unless I'm missing something, I don't believe this would be too
difficult to implement. Most of the code would live in remote.c and
would require a new packet for handling thread breakpoints.

Would something like this be OK, or did you have something else in mind?
vThreadBreak;addr;kind;thread-id

The only other question I currently have is how should the thread id
be communicated to remote.c? Should the global inferior_ptid be used,
or would we add another member to the struct bp_target_info that
contains the thread ID?

--
~JPEW

Re: Thread Specific Breakpoints in Remote Targets

[Petr Hluzín]

On 1 September 2011 15:23, Raphael Zulliger <zulliger@indel.ch> wrote:
> On 31.08.2011 20:09, Pedro Alves wrote:
>>
>> On Wednesday 31 August 2011 15:47:32, Tom Tromey wrote:
>>> It sounds like you are making breakpoints on the target thread-specific
>>> based on the current thread.  But I thought we didn't (yet) have a way
>>> to inform the target that a given breakpoint was thread-specific (but I
>>> don't know this area extremely well -- if I'm wrong I'd like to know
>>> about it).
>>
>> You're right, we don't.
>
> Just to mention that: My company would be very interested in (optional)
> 'thread specific breakpoints' support for remote targets. gdb could ask a
> gdbstub whether it supports this feature (by the qSupported packet).
>
> In our case, our proprietary real-time OS already offers support for 'thread
> specific breakpoints' and it is definitely not an option for our system to
> use the 'thread specific breakpoint emulation' performed by the gdb frontend
> today as it would disrupt real-time behavior. The lack of this feature
> causes major troubles for us during single-stepping, ...

I think GDB's thread-specific breakpoints do something different than
you expect: if user sets breakpoint specific to thread 5 then the
other threads do not trigger the breakpoint (so far so good). However
when the thread 5 arrives at the breakpoint then GDB stops _all_
threads even if the breakpoint was thread-specific.

If you want to break only the thread which arrived at the breakpoint
location and have the other threads continue running, then implement
GDB's Non-Stop Mode [1], [2].

I think the thread-specific breakpoints will be almost unnecessary
when you implement the non-stop mode. Users usually debug their own
code which may live in multiple threads and specifying even one thread
condition is slowdown. Of course if users place a breakpoint in a
library used by multiple threads, then the thread-specific breakpoints
will be useful.

Still, the ability of thread-specific breakpoints would be useful to
other remote stubs.

By the way: when user does "next", "step", or "finish" GDB places a
temporary internal breakpoint. Is the breakpoint thread-specific?

I think it would be confusing to do "next" and wake up a different
thread. Some IDE do not notify the user about the different thread.
Furthermore the different thread may have identical stack (but
processing unrelated data).
Anyway it is difficult for the user to return to the correct thread
since the frame of interest will be buried under sub-procedures the
user was next-ing over and he may not even know the thread ID. Plus
the next "next" may do the nastiness again. It would be useful if GDB
placed thread-specific breakpoints for the commands.

[1] http://sourceware.org/gdb/current/onlinedocs/gdb/Non_002dStop-Mode.html
[2] http://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Non_002dStop.html

-- 
Petr Hluzin

Re: Thread Specific Breakpoints in Remote Targets

[Pedro Alves]

On Thursday 01 September 2011 22:34:32, Petr Hluzín wrote:
> By the way: when user does "next", "step", or "finish" GDB places a
> temporary internal breakpoint. Is the breakpoint thread-specific?

It is.

-- 
Pedro Alves

Re: Thread Specific Breakpoints in Remote Targets

[Raphael Zulliger]

On 01.09.2011 23:34, Petr Hluzín wrote:
> On 1 September 2011 15:23, Raphael Zulliger<zulliger@indel.ch>  wrote:
>> right, we don't.
>> Just to mention that: My company would be very interested in (optional)
>> 'thread specific breakpoints' support for remote targets. gdb could ask a
>> gdbstub whether it supports this feature (by the qSupported packet).
>>
>> In our case, our proprietary real-time OS already offers support for 'thread
>> specific breakpoints' and it is definitely not an option for our system to
>> use the 'thread specific breakpoint emulation' performed by the gdb frontend
>> today as it would disrupt real-time behavior. The lack of this feature
>> causes major troubles for us during single-stepping, ...
> I think GDB's thread-specific breakpoints do something different than
> you expect: if user sets breakpoint specific to thread 5 then the
> other threads do not trigger the breakpoint (so far so good).
AFAIK: From point of view of the GDB stub, the breakpoint specific to 
thread 5 is actually a "global breakpoint". I think this is the case due 
to the fact that I couldn't find a 'thread specific breakpoint' facility 
within the gdb remote protocol. Moreover, when such a breakpoint is set 
within the gdb frontend, my gdbstub just receives the command for 
setting a breakpoint along with an address but now pid/tid information...

AFAIK: What finfally makes a "thread specific breakpoint" thread 
specific is the way how the gdb frontend handles a breakpoint hit of an 
unrelated thread: GDB will silently continue that unrelated thread. This 
mechanism leaves the user of gdb under the impression that a breakpoint 
is thread specific - but technically, the breakpoint which has been 
inserted in the inferior is global and will be hit by every thread! 
This, of course, adds a major (and undeterministic) delay to each 
unrelated thread that hits the breakpoint. This is not acceptable in a 
real-time system (at least not in ours).
> However
> when the thread 5 arrives at the breakpoint then GDB stops _all_
> threads even if the breakpoint was thread-specific.
>
> If you want to break only the thread which arrived at the breakpoint
> location and have the other threads continue running, then implement
> GDB's Non-Stop Mode [1], [2].
No, my problem is different. I'm actually using non-stop debugging 
("multiprocess+;QStartNoAckMode+;QNonStop+;qXfer:threads:read+;PacketSize=1EE;qXfer:osdata:read+"). 
And therefore the 'user visible behavior' is, as stated above, 
absolutely ok - but the real-time behavior is not.
>
> I think the thread-specific breakpoints will be almost unnecessary
> when you implement the non-stop mode. Users usually debug their own
> code which may live in multiple threads and specifying even one thread
> condition is slowdown. Of course if users place a breakpoint in a
> library used by multiple threads, then the thread-specific breakpoints
> will be useful.
You may judges the following a bad design/architecture - but our 
real-time os lives in the very same address space as the complete user 
code (mainly because of efficiency reasons). Moreover, high priority 
real-time threads are typically using the very same classes/functions as 
others do. Therefore, a typical debugging session will indeed cause 
unrelated threads hit breakpoints when single stepping.

In short: I partially agree with your point - but for the our system it 
is indeed "a real world problem" (not just a nice to have)...
>
> Still, the ability of thread-specific breakpoints would be useful to
> other remote stubs.
>
> By the way: when user does "next", "step", or "finish" GDB places a
> temporary internal breakpoint. Is the breakpoint thread-specific?
Yes, but I believe these "thread-specific"-ness is handled as stated 
above: by the GDB frontend.
>
> I think it would be confusing to do "next" and wake up a different
> thread. Some IDE do not notify the user about the different thread.
> Furthermore the different thread may have identical stack (but
> processing unrelated data).
> Anyway it is difficult for the user to return to the correct thread
> since the frame of interest will be buried under sub-procedures the
> user was next-ing over and he may not even know the thread ID. Plus
> the next "next" may do the nastiness again. It would be useful if GDB
> placed thread-specific breakpoints for the commands.
>
> [1] http://sourceware.org/gdb/current/onlinedocs/gdb/Non_002dStop-Mode.html
> [2] http://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Non_002dStop.html
>

Re: Thread Specific Breakpoints in Remote Targets

[Petr Hluzín]

On 2 September 2011 07:13, Raphael Zulliger <zulliger@indel.ch> wrote:
> On 01.09.2011 23:34, Petr Hluzín wrote:
>> I think GDB's thread-specific breakpoints do something different than
>> you expect: if user sets breakpoint specific to thread 5 then the
>> other threads do not trigger the breakpoint (so far so good).
> ...
>
> AFAIK: What finfally makes a "thread specific breakpoint" thread specific is
> the way how the gdb frontend handles a breakpoint hit of an unrelated
> thread: GDB will silently continue that unrelated thread. This mechanism
> leaves the user of gdb under the impression that a breakpoint is thread
> specific - but technically, the breakpoint which has been inserted in the
> inferior is global and will be hit by every thread! This, of course, adds a
> major (and undeterministic) delay to each unrelated thread that hits the
> breakpoint. This is not acceptable in a real-time system (at least not in
> ours).
> ...
>> If you want to break only the thread which arrived at the breakpoint
>> location and have the other threads continue running, then implement
>> GDB's Non-Stop Mode [1], [2].
>
> No, my problem is different. I'm actually using non-stop debugging
> ("multiprocess+;QStartNoAckMode+;QNonStop+;qXfer:threads:read+;PacketSize=1EE;qXfer:osdata:read+").
> And therefore the 'user visible behavior' is, as stated above, absolutely ok
> - but the real-time behavior is not.

I was not aware that your system requires more real-time behavior.
(Most systems can tolerate the ~30ms delay when GDB checks current
thread.)
Because you already implement the non-stop mode my suggestion is irrelevant.

> You may judges the following a bad design/architecture - but our real-time
> os lives in the very same address space as the complete user code (mainly
> because of efficiency reasons).

Actually your design/architecture are fine - debug stubs are often
done this way on embedded stuff. One can get breakpoint hit delay
under 10 us. Plus you do not need JTAG adapter etc.

-- 
Petr Hluzin

Re: Thread Specific Breakpoints in Remote Targets

[Tom Tromey]

>>>>> "Josh" == Josh Watt <jpewdev@gmail.com> writes:

[ thread-specific breakpoints on the target ]

Josh> Unless I'm missing something, I don't believe this would be too
Josh> difficult to implement. Most of the code would live in remote.c and
Josh> would require a new packet for handling thread breakpoints.

I didn't see a response to this.

Josh> Would something like this be OK, or did you have something else in mind?
Josh> vThreadBreak;addr;kind;thread-id

Josh> The only other question I currently have is how should the thread id
Josh> be communicated to remote.c? Should the global inferior_ptid be used,
Josh> or would we add another member to the struct bp_target_info that
Josh> contains the thread ID?

I don't know the details of the remote protocol, but yes, something like
this would be good to have.  On the gdb side, I would say adding a field
to bp_target_info would be much better than a global.

Tom