Faster stepping amidst breakpoints

Faster stepping amidst breakpoints

[Maxim Grigoriev]

Hello GDB community,

Has anyone tried to optimize the GDB protocol by letting
the remote agent plant and unplant breakpoints when the
target resumes and stops, rather than having GDB do it
over the wire with round-trip latency on every single
breakpoint plant&  unplant request?

Certain GDB operations involve a lot of single-stepping,
which can be really slow on certain targets (especially
embedded targets) because of that latency.

In principle, GDB can query the remote agent to decide
whether to apply this optimization.  And the remote agent
must be careful not to plant any breakpoint at the current
PC when single-stepping.  There are likely other
considerations, but at a glance, it seems doable.
So am curious whether there's any particular reason
it wasn't done (that we could see).

Thanks,
-- Maxim

Re: Faster stepping amidst breakpoints

[Daniel Jacobowitz]

On Fri, Jan 21, 2011 at 03:05:49PM -0800, Maxim Grigoriev wrote:
> Hello GDB community,
> 
> Has anyone tried to optimize the GDB protocol by letting
> the remote agent plant and unplant breakpoints when the
> target resumes and stops, rather than having GDB do it
> over the wire with round-trip latency on every single
> breakpoint plant&  unplant request?
> 
> Certain GDB operations involve a lot of single-stepping,
> which can be really slow on certain targets (especially
> embedded targets) because of that latency.

Consider "set breakpoint always-inserted".

I've been wondering lately if we should flip the default.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Faster stepping amidst breakpoints

[Maxim Grigoriev]

Thanks much for your fast reply, Daniel.

I will experiment with  "set breakpoint always-inserted"
and see, if this is exactly what we need.

-- Maxim

On 01/22/2011 04:14 PM, Daniel Jacobowitz wrote:
> On Fri, Jan 21, 2011 at 03:05:49PM -0800, Maxim Grigoriev wrote:
>    
>> Hello GDB community,
>>
>> Has anyone tried to optimize the GDB protocol by letting
>> the remote agent plant and unplant breakpoints when the
>> target resumes and stops, rather than having GDB do it
>> over the wire with round-trip latency on every single
>> breakpoint plant&   unplant request?
>>
>> Certain GDB operations involve a lot of single-stepping,
>> which can be really slow on certain targets (especially
>> embedded targets) because of that latency.
>>      
> Consider "set breakpoint always-inserted".
>
> I've been wondering lately if we should flip the default.
>
>    

Re: Faster stepping amidst breakpoints

[Joel Brobecker]

> Consider "set breakpoint always-inserted".
> I've been wondering lately if we should flip the default.

I like the idea of changing the default.

Do you know what the risks would be?  I looked at the code, and
there isn't something obvious/delicate, it seems.  Perhaps we might
find ourselves forgetting to re-insert breakpoints, or inserting
them twice? I think you guys have more experience than we do?

In terms of when, perhaps a good time to switch would be either
now (about 3 weeks away from planned 7.3 branching), or in 3 weeks
from now, right after branching.

-- 
Joel

Re: Faster stepping amidst breakpoints

[Daniel Jacobowitz]

On Mon, Jan 31, 2011 at 08:49:51AM +0400, Joel Brobecker wrote:
> > Consider "set breakpoint always-inserted".
> > I've been wondering lately if we should flip the default.
> 
> I like the idea of changing the default.
> 
> Do you know what the risks would be?  I looked at the code, and
> there isn't something obvious/delicate, it seems.  Perhaps we might
> find ourselves forgetting to re-insert breakpoints, or inserting
> them twice? I think you guys have more experience than we do?

As far as I can remember (you know how much GDB development I do
nowadays), the only risks were if GDB crashed and left the application
with breakpoints inserted.  Of course, I'm in favor of GDB not
crashing.

Pedro, Stan, any thoughts?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Faster stepping amidst breakpoints

[Joel Brobecker]

> As far as I can remember (you know how much GDB development I do
> nowadays), the only risks were if GDB crashed and left the application
> with breakpoints inserted.  Of course, I'm in favor of GDB not
> crashing.

I see this as an acceptable outcome of a debugger crash.  I am guessing
that in most cases, people debug their program, and then want it killed,
so leaving the inferior behind with breakpoints inserted won't matter
much (or, not as much as having their session brutally interrupted by
the crash itself).  For the small number of those for whom it matters,
they can change the switch back...

I was more worried about possible issues during normal operations.
For instance, has this been tested with software-single-step, for
instance? What I'm going to do, to give it a little more exposure,
is flip the switch in AdaCore's HEAD build, and monitor the testsuite
results.  Have you flipped the switch in CS's builds too?

-- 
Joel

Re: Faster stepping amidst breakpoints

[Pedro Alves]

On Tuesday 01 February 2011 15:12:35, Daniel Jacobowitz wrote:
> On Mon, Jan 31, 2011 at 08:49:51AM +0400, Joel Brobecker wrote:
> > > Consider "set breakpoint always-inserted".
> > > I've been wondering lately if we should flip the default.
> > 
> > I like the idea of changing the default.
> > 
> > Do you know what the risks would be?  I looked at the code, and
> > there isn't something obvious/delicate, it seems.  Perhaps we might
> > find ourselves forgetting to re-insert breakpoints, or inserting
> > them twice? I think you guys have more experience than we do?
> 
> As far as I can remember (you know how much GDB development I do
> nowadays), the only risks were if GDB crashed and left the application
> with breakpoints inserted.  Of course, I'm in favor of GDB not
> crashing.
> 
> Pedro, Stan, any thoughts?

Regarding the OP's:

>Certain GDB operations involve a lot of single-stepping,
>which can be really slow on certain targets (especially
>embedded targets) because of that latency."

and the $subject:

> Faster stepping amidst breakpoints

... scenario, I'm not certain always-inserted alone helps that
much, because what always-inserted mainly affects, is whether
GDB removes breakpoints before giving the prompt to the
user, after the whole command finishes; and whether a
"break" or "delete" command inserts/removes breakpoints
in the target immediately.  In all-stop mode, without
displaced stepping enabled, when stepping amidst
breakpoints, even with "breakpoint always-inserted on",
GDB _will still_ remove/insert all breakpoints to single-step
over a location that has a breakpoint.

It sounds to me that what you need to optimize that
use case, is to make GDB smarter into only
remove/inserting the breakpoint being stepped over,
instead of always remove/inserting _all_ breakpoints.

-- 
Pedro Alves

Re: Faster stepping amidst breakpoints

[Michael Snyder]

Joel Brobecker wrote:
>> As far as I can remember (you know how much GDB development I do
>> nowadays), the only risks were if GDB crashed and left the application
>> with breakpoints inserted.  Of course, I'm in favor of GDB not
>> crashing.
> 
> I see this as an acceptable outcome of a debugger crash.  I am guessing
> that in most cases, people debug their program, and then want it killed,
> so leaving the inferior behind with breakpoints inserted won't matter
> much 

It would matter if you were debugging a live real-world app...

Re: Faster stepping amidst breakpoints

[Daniel Jacobowitz]

On Tue, Feb 01, 2011 at 07:23:38AM +0400, Joel Brobecker wrote:
> Have you flipped the switch in CS's builds too?

Not yet.  I've considered it, though.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Faster stepping amidst breakpoints

[Joel Brobecker]

> > Have you flipped the switch in CS's builds too?
> 
> Not yet.  I've considered it, though.

I did it a couple of days ago (just setting the default to on,
instead of auto).  I was waiting for the result from our nightly
testing, but unfortunately, the results didn't come out, so I'm
still waiting for some results.

-- 
Joel

Re: Faster stepping amidst breakpoints

[Joel Brobecker]

> >I see this as an acceptable outcome of a debugger crash.  I am guessing
> >that in most cases, people debug their program, and then want it killed,
> >so leaving the inferior behind with breakpoints inserted won't matter
> >much
> 
> It would matter if you were debugging a live real-world app...

Of course, that's why I was careful to say that it does matter sometimes.
The change is not going to be beneficial in 100% of the cases.  But if
it is an improvement in the majority of the cases, then that's still
progress. Those of us who are occasionally in the situation where it
isn't have the option of switching the setting back.

-- 
Joel

Re: Faster stepping amidst breakpoints

[Maxim Grigoriev]

On 01/30/2011 08:49 PM, Joel Brobecker wrote:
>> Consider "set breakpoint always-inserted".
>> I've been wondering lately if we should flip the default.
>>      
> I like the idea of changing the default.
>
> Do you know what the risks would be?  I looked at the code, and
> there isn't something obvious/delicate, it seems.  Perhaps we might
> find ourselves forgetting to re-insert breakpoints, or inserting
> them twice? I think you guys have more experience than we do?
>
> In terms of when, perhaps a good time to switch would be either
> now (about 3 weeks away from planned 7.3 branching), or in 3 weeks
> from now, right after branching.
>    
I have Three observations.

1) I think changing the default is a good idea, when it's proven safe.

I tested it on Xtensa GDB 7.1, which is probably not
as valuable as testing results you are waiting for.

With the default changed to always-inserted == on
and the target understanding z/Z-packets, I observe
a regression :

(gdb) FAIL: gdb.base/break.exp: finish from called function
(gdb) FAIL: gdb.base/callfuncs.exp: finish from call dummy breakpoint 
returns correct value
FAIL: gdb.base/callfuncs.exp: finish after stop in call dummy preserves 
register contents
FAIL: gdb.base/callfuncs.exp: return after stop in call dummy preserves 
register contents
(gdb) FAIL: gdb.base/callfuncs.exp: Finish from nested call level 4
(gdb) FAIL: gdb.base/callfuncs.exp: backtrace after finish from nested 
call level 4
(gdb) FAIL: gdb.base/callfuncs.exp: Finish from nested call level 3
(gdb) FAIL: gdb.base/callfuncs.exp: backtrace after finish from nested 
call level 3
(gdb) FAIL: gdb.base/callfuncs.exp: Finish from nested call level 2
(gdb) FAIL: gdb.base/callfuncs.exp: backtrace after finish from nested 
call level 2
(gdb) FAIL: gdb.base/callfuncs.exp: Finish from nested call level 1
FAIL: gdb.base/callfuncs.exp: nested call dummies preserve register contents
(gdb) FAIL: gdb.base/sepdebug.exp: finish from called function

All new failures were caused by attempting to remove
non-existent breakpoint.


2) I think in the embedded-system world it does matter
     when crashing / detaching GDB leaves target memory
     and/or registers changed.


3) In my original question I was talking about possible GDB
     protocol extensions introducing a concept of a "smart"
     target agent ( if such a term is appropriate here ).

What I meant was a target agent, which can

     -- realize it's about to single-step over an inserted
        breakpoint and then handle it properly ;

     -- watch out for shutting-down GDB communications,
        while counting time-outs, and then return target to the
        reliable state essentially making GDB non-intrusive.

If such functionality existed GDB would be able to discover
that the target agent is "smart" and then safely switch to the
breakpoint-always-inserted==on mode even when it's used
in the embedded system environment.

-- Maxim

Re: Faster stepping amidst breakpoints

[Tom Tromey]

>>>>> "Maxim" == Maxim Grigoriev <maxim@tensilica.com> writes:

Maxim> With the default changed to always-inserted == on
Maxim> and the target understanding z/Z-packets, I observe
Maxim> a regression :

Could you file this in bugzilla?

Maxim> 3) In my original question I was talking about possible GDB
Maxim>     protocol extensions introducing a concept of a "smart"
Maxim>     target agent ( if such a term is appropriate here ).

I think this would be interesting, too.
It sounds like a way we could use uprobes, at least IIUC.

Tom

Re: Faster stepping amidst breakpoints

[Daniel Jacobowitz]

On Thu, Feb 03, 2011 at 02:53:40PM -0800, Maxim Grigoriev wrote:
> 2) I think in the embedded-system world it does matter
>     when crashing / detaching GDB leaves target memory
>     and/or registers changed.

Yes - if your agent does breakpoint management using z/Z, then it has
the opportunity to clean up as long as the agent doesn't crash.  IMO
that's a reasonable expectation; if the agent crashes, your target is
probably in an unrecoverable state anyway.

> What I meant was a target agent, which can
> 
>     -- realize it's about to single-step over an inserted
>        breakpoint and then handle it properly ;
> 
>     -- watch out for shutting-down GDB communications,
>        while counting time-outs, and then return target to the
>        reliable state essentially making GDB non-intrusive.

I suggest separating these and dealing with them as separate concerns.

A related issue is watchpoints; different agents (and SoCs) handle
the current instruction after hitting a watchpoint differently.  For
instance, GDB assumes all MIPS targets have nonsteppable watchpoints
and we have one where that's clearly not true...

-- 
Daniel Jacobowitz
CodeSourcery

Re: Faster stepping amidst breakpoints

[Maxim Grigoriev]

On 02/04/2011 08:06 AM, Tom Tromey wrote:
>>>>>> "Maxim" == Maxim Grigoriev<maxim@tensilica.com>  writes:
>>>>>>              
> Maxim>  With the default changed to always-inserted == on
> Maxim>  and the target understanding z/Z-packets, I observe
> Maxim>  a regression :
>
> Could you file this in bugzilla?
>    
I filed a PR :

http://sourceware.org/bugzilla/show_bug.cgi?id=12466

Let me know, please, if I can assist you on resolving this issue.

Thanks,
-- Maxim

Re: Faster stepping amidst breakpoints

[Mark Kettenis]

> Date: Tue, 1 Feb 2011 07:23:38 +0400
> From: Joel Brobecker <brobecker@adacore.com>
> 
> > As far as I can remember (you know how much GDB development I do
> > nowadays), the only risks were if GDB crashed and left the application
> > with breakpoints inserted.  Of course, I'm in favor of GDB not
> > crashing.
> 
> I see this as an acceptable outcome of a debugger crash.  I am guessing
> that in most cases, people debug their program, and then want it killed,
> so leaving the inferior behind with breakpoints inserted won't matter
> much (or, not as much as having their session brutally interrupted by
> the crash itself).  For the small number of those for whom it matters,
> they can change the switch back...

You're probably right for the case where people start the program from
GDB, but I guess that when people attach to an already running process
they typically don't want to kill it when they're done with debugging.

That said, it is probably a bad idea to have different behaviour for
"run" and "attach" in this case.  Let's throw the switch after the 7.3
release and see if people even notice.

Re: Faster stepping amidst breakpoints

[Joel Brobecker]

> Let's throw the switch after the 7.3 release and see if people even
> notice.

Maxim Grigoriev actually noticed a regression, and I also noticed
something on VxWorks.  From what I've seen from Maxim's message,
and the regressions we've seen on VxWorks, they might be the same
bug - but the power supply for the VxWorks board I'm using keeps
dying on me :-(.

-- 
Joel

Re: Faster stepping amidst breakpoints

[Joel Brobecker]

> Maxim Grigoriev actually noticed a regression, and I also noticed
> something on VxWorks.

The problem on VxWorks is more or less my own fault - I'm doing
something pretty shady: Inserting a breakpoint on VxWorks is done
via a WTX request which returns an "eventpoint ID".  To remove that
breakpoint, we use the associated routine with the eventpoint ID
of our breakpoint.  As a result, I need to keep a mapping between
breakpoint location and evenpoints IDs.  I was more or less using
the bp_location address as the key to my mapping (this is the shady
part), hoping that it would never be reallocated. Unfortunately, this
is not true at all, or at least not anymore: See breakpoint_re_set_one,
which calls update_breakpoint_locations... The approach seems to be:
unless all locations, recompute new locations, and then purge the old
locations (update_global_location_list).

The reason why this works with the current default is that the
breakpoints get removed before they are re-set, and so the map
gets emptied before the new locations are created.

I'm thinking of adding a target_private_data field in struct
bp_target_info. I think we can get away with memory management
by allocating the private data at breakpoint insertion, and
deallocating it at removal.  That way, I get rid of the map
entirely.

-- 
Joel