Keeping breakpoints inserted

Keeping breakpoints inserted

[Vladimir Prus]

One of the infrastructure bits necessary for the non-stop threads
debugging is always-inserted-breakpoints mode. If GDB has stopped
one thread, and other threads are running, we want those other threads
to still hit breakpoints and watchpoints. However, current GDB removes
all breakpoints from the target before giving user a prompt, and this
has to change.

I've spend quite time examining breakpoint.c and infrun.c and 
cleaning/localizing the decisions as to when breakpoints are 
inserted/removed, and I believe I now have a fully workable plan 
to make breakpoints always inserted. We need to:

1. Generally, whenever a breakpoint is added, changed, or 
removed, immediately communicate this change to the target.

2. In order to do that, we need to have a single function called on each
breakpoint change. That function will be responsible to update 
bp_location_chain, using locations of existing breakpoint, removing
breakpoint locations that are no longer necessary, and inserting newly 
added locations. Since we can have several breakpoints at the same address, 
the function should take care not to remove location belonging to a 
deleted breakpoint, if there's location at the same address belonging 
to a still-present breakpoint.

Specifically, the function will walk over current bp_location_chain. 
For each location:

        - if no breakpoint has a location with the same address, 
	we remove the location from target
        - if no breakpoint has this bp_location* object, we free it.

All locations present in some breakpoint but not yet present in 
bp_location_chain are added to bp_location_chain and inserted to target.

3. The functions that read memory should be modified to restore memory 
content overwritten by breakpoints. I believe there's a 
single function that is in all memory-reading code paths, so this change is
easy.

4. infrun.c should be modified to avoid removing breakpoints when 
we stop -- most probably under control of a new variable.

5. Sometimes, infrun.c still need to remove breakpoints -- in particular 
when stepping over them. We'll implement smart code that allows to 
step over breakpoint without removing it -- Jim will separately post details
on this. For watchpoints, we still have to remove them when stepping 
over watchpoint (for the case of non-continuable non-steppable watchpoints).

6. It's possible that we get an error when inserting breakpoint. 
In present GDB, such an error will be printed when we try to resume
target, and the target won't be resumed. With always-inserted mode, 
the error will be printed when we add or change the breakpoint. However, 
resuming target with not-inserted breakpoint still seems  wrong behaviour, 
so we'll try to insert breakpoints again when resuming, and if that fails,
stop.

Anybody has comments on this approach?

- Volodya

Re: Keeping breakpoints inserted

[Michael Snyder]

On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
> One of the infrastructure bits necessary for the non-stop threads
> debugging is always-inserted-breakpoints mode. If GDB has stopped
> one thread, and other threads are running, we want those other threads
> to still hit breakpoints and watchpoints. However, current GDB removes
> all breakpoints from the target before giving user a prompt, and this
> has to change.
> 
> I've spend quite time examining breakpoint.c and infrun.c and 
> cleaning/localizing the decisions as to when breakpoints are 
> inserted/removed, and I believe I now have a fully workable plan 
> to make breakpoints always inserted. 

> [...]
> 
> Anybody has comments on this approach?

Might there be a user preference, under some circumstances, 
to NOT have them inserted while some threads run and some
are stopped?

Re: Keeping breakpoints inserted

[Vladimir Prus]

On Friday 30 November 2007 04:12:14 you wrote:
> On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
> > One of the infrastructure bits necessary for the non-stop threads
> > debugging is always-inserted-breakpoints mode. If GDB has stopped
> > one thread, and other threads are running, we want those other threads
> > to still hit breakpoints and watchpoints. However, current GDB removes
> > all breakpoints from the target before giving user a prompt, and this
> > has to change.
> >
> > I've spend quite time examining breakpoint.c and infrun.c and
> > cleaning/localizing the decisions as to when breakpoints are
> > inserted/removed, and I believe I now have a fully workable plan
> > to make breakpoints always inserted.
> >
> > [...]
> >
> > Anybody has comments on this approach?
>
> Might there be a user preference, under some circumstances,
> to NOT have them inserted while some threads run and some
> are stopped?

Honestly, originally I planned to have breakpoints always inserted,
unconditionally. Unfortunately, that does not work with remote targets --
since the Z0 packet may use memory breakpoints, and remote protocol does
not requires the memory read packet to return original memory content,
without breakpoints, we'd need to explicitly read the memory ourself --
which is not necessary without non-stop mode.

At the same time, I don't yet know any scenario when 
always-inserted-breakpoints mode should not be used in non-stop mode -- 
as non-stop mode needs some support from remote side anyway, we might
as well demand "right" behaviour from Z0/m.

- Volodya

Re: Keeping breakpoints inserted

[Thiago Jung Bauermann]

Hi,

On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
> Anybody has comments on this approach?

When adding and removing breakpoints, will GDB stop all threads or just
the one under inspection?

On first thought, I think that if adding a breakpoint can be done
atomically (i.e., trap instruction is 1 word wide), then it wouldn't be
necessary to stop other threads. Am I being too naive here? 
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: Keeping breakpoints inserted

[Michael Snyder]

On Fri, 2007-11-30 at 19:03 -0200, Thiago Jung Bauermann wrote:
> Hi,
> 
> On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
> > Anybody has comments on this approach?
> 
> When adding and removing breakpoints, will GDB stop all threads or just
> the one under inspection?
> 
> On first thought, I think that if adding a breakpoint can be done
> atomically (i.e., trap instruction is 1 word wide), then it wouldn't be
> necessary to stop other threads. Am I being too naive here? 

There will be some as-it-were non-deterministic behavior, 
it seems to me.  If threads are running while we are inserting
breakpoints, then there will (at some point) be breakpoint
events while we are inserting breakpoints, and the order of
these events (and of running-threads stopping) will depend
on the order in which we insert them, as well as on what 
the running threads happen to be doing at the time.

I should think that this would be more intrusive (in the 
sense of changing the behavior of the target program) than
we already are.


We could see a deadlock develop during the time it takes us
to insert all the breakpoints.

Re: Keeping breakpoints inserted

[Jim Blandy]

Thiago Jung Bauermann <bauerman at br.ibm.com> writes:
> On Thu, 2007-11-29 at 22:24 +0300, Vladimir Prus wrote:
>> Anybody has comments on this approach?
>
> When adding and removing breakpoints, will GDB stop all threads or just
> the one under inspection?
>
> On first thought, I think that if adding a breakpoint can be done
> atomically (i.e., trap instruction is 1 word wide), then it wouldn't be
> necessary to stop other threads. Am I being too naive here? 

The Linux kernel has a facility called 'kprobes' (which is extremely
cool, BTW) that can insert breakpoint instructions into kernel code
while other processors are executing it.  This interface has been
implemented on many different architectures.  We're using it as our
model, since it does essentially what we need to do, in an even more
challenging context.  I haven't seen one where it needs to halt all
other processors while inserting the breakpoint.

On the i386, you pretty much just save the underlying instruction and
drop the trap in.  There are processor errata (AH 33 in
ftp://download.intel.com/design/mobile/SPECUPDT/31407915.pdf) that
warn against the dangers of one processor making changes in another's
code stream, but it turns out that the 'int3' instruction --- the one
used for breakpoints --- is specifically excluded from those errata;
so we're off the hook.

Re: Keeping breakpoints inserted

[Jim Blandy]

Michael Snyder <msnyder at specifix.com> writes:
> There will be some as-it-were non-deterministic behavior, 
> it seems to me.  If threads are running while we are inserting
> breakpoints, then there will (at some point) be breakpoint
> events while we are inserting breakpoints, and the order of
> these events (and of running-threads stopping) will depend
> on the order in which we insert them, as well as on what 
> the running threads happen to be doing at the time.
>
> I should think that this would be more intrusive (in the 
> sense of changing the behavior of the target program) than
> we already are.
>
> We could see a deadlock develop during the time it takes us
> to insert all the breakpoints.

Since breakpoints only make the threads that hit them execute "very
slowly", I don't see how they could introduce a deadlock in code that
didn't already have the potential for deadlock in the absence of a
debugger.

It's true that the breakpoints won't all appear in the address space
simultaneously.  It might be useful to be able to insert breakpoints
disabled, and then enable them as a group.  Even in all-stop mode, GDB
isn't able to stop all threads simulataneously; it has to obtain the
list of threads and send each one a signal.  So it seems to me that
some degree of 'play' in breakpoint insertion isn't so different from
what we have now.

Inserting breakpoints into code that's currently being executed
inevitably entails race conditions.  But I expect one would only do
such a thing in circumstances where it's meaningful and controllable.

For example, you might know that you'll reach the breakpoint only in
response to a certain input, which you'll send to the program after
you've set the breakpoint.

Or, it might not matter which particular execution of the code you
catch first, you just want to interrupt it.

Non-stop debugging is more invasive than all-stop debugging, if you
assess invasiveness by looking at how the threads progress relative to
each other.  Non-stop debugging can make one thread run 'extremely
slowly' while other threads progress at a normal rate.  All-stop
debugging stops all threads roughly equally (but even then not
perfectly so).

However, non-stop debugging can be less invasive than all-stop
debugging, if you assess invasiveness by looking at how responsive the
system remains to outside events.  With non-stop debugging, you can
debug one particular thread out of the entire system while the others
continue to work normally.

Deciding which mode to use depends on the developer's knowledge of the
system they're debugging.

Re: Keeping breakpoints inserted

[Michael Snyder]

On Fri, 2007-11-30 at 16:08 -0800, Jim Blandy wrote:
> Michael Snyder <msnyder at specifix.com> writes:
> > There will be some as-it-were non-deterministic behavior, 
> > it seems to me.  If threads are running while we are inserting
> > breakpoints, then there will (at some point) be breakpoint
> > events while we are inserting breakpoints, and the order of
> > these events (and of running-threads stopping) will depend
> > on the order in which we insert them, as well as on what 
> > the running threads happen to be doing at the time.
> >
> > I should think that this would be more intrusive (in the 
> > sense of changing the behavior of the target program) than
> > we already are.
> >
> > We could see a deadlock develop during the time it takes us
> > to insert all the breakpoints.
> 
> Since breakpoints only make the threads that hit them execute "very
> slowly", I don't see how they could introduce a deadlock in code that
> didn't already have the potential for deadlock in the absence of a
> debugger.

Well, just 'cause you've changed the timing.  Thread A holds
a resource that thread B needs; thread A hits a breakpoint 
but thread B gets to keep running (for a while) because you
haven't finished inserting breakpoints.

> It's true that the breakpoints won't all appear in the address space
> simultaneously.  It might be useful to be able to insert breakpoints
> disabled, and then enable them as a group.  Even in all-stop mode, GDB
> isn't able to stop all threads simulataneously; it has to obtain the
> list of threads and send each one a signal.  So it seems to me that
> some degree of 'play' in breakpoint insertion isn't so different from
> what we have now.

No, it's just more of what is already an intrusive thing.
It's worse in degree, not in kind.

Re: Keeping breakpoints inserted

[Jim Blandy]

On Nov 30, 2007 5:30 PM, Michael Snyder <msnyder@specifix.com> wrote:
> > Since breakpoints only make the threads that hit them execute "very
> > slowly", I don't see how they could introduce a deadlock in code that
> > didn't already have the potential for deadlock in the absence of a
> > debugger.
>
> Well, just 'cause you've changed the timing.  Thread A holds
> a resource that thread B needs; thread A hits a breakpoint
> but thread B gets to keep running (for a while) because you
> haven't finished inserting breakpoints.

Non-stop debugging can affect the relative timing of threads more than
all-stop debugging.  And even if the developer only stops one thread,
others may stop too, because they're waiting for the developer's
thread to do something.  But I don't think what you've described is
deadlock: B is blocked waiting for A, which may take a long time
(until the developer lets go of it, to be exact) to get through what
it's doing.  When A resumes, both will resume.

The original concern you raised was that non-stop debugging is "more
intrusive than we already are".  But clearly all-stop debugging on a
live system is maximally intrusive to the system's users; non-stop
debugging has the potential to be much less intrusive, when used with
knowledge of the interactions between the system's threads.

I think what's bothering you is that the degree of effective
interference depends on the developer knowing stuff about the system
under debug; it's not just a flat-out guarantee made by the debugger:
"no matter what you do, we promise X, or at least roughly X."  But
there are reasonable debugger use cases that are simply not possible
at all with all-stop debugging.  We want to support those, in addition
to the ones we support now.

Re: Keeping breakpoints inserted

[Thiago Jung Bauermann]

On Sat, 2007-12-01 at 09:52 -0800, Jim Blandy wrote:
> On Nov 30, 2007 5:30 PM, Michael Snyder <msnyder@specifix.com> wrote:
> The original concern you raised was that non-stop debugging is "more
> intrusive than we already are".  But clearly all-stop debugging on a
> live system is maximally intrusive to the system's users; non-stop
> debugging has the potential to be much less intrusive, when used with
> knowledge of the interactions between the system's threads.

There are cases when a developer will want to use non-stop debugging but
minimize change of relative timing of threads. Suppose that a developer
is trying to debug a deadlock situation in a program with 3 threads. A
and B are deadlocking, and C is a "supporting" thread without which the
other two can't run. He can't use all-stop debugging because while
inspecting A and B, C needs to be running. In this case, relative timing
of threads is important in order to have better chance at reproducing
the deadlock.
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: Keeping breakpoints inserted

[Jim Blandy]

Thiago Jung Bauermann <bauerman at br.ibm.com> writes:
> On Sat, 2007-12-01 at 09:52 -0800, Jim Blandy wrote:
>> On Nov 30, 2007 5:30 PM, Michael Snyder <msnyder@specifix.com> wrote:
>> The original concern you raised was that non-stop debugging is "more
>> intrusive than we already are".  But clearly all-stop debugging on a
>> live system is maximally intrusive to the system's users; non-stop
>> debugging has the potential to be much less intrusive, when used with
>> knowledge of the interactions between the system's threads.
>
> There are cases when a developer will want to use non-stop debugging but
> minimize change of relative timing of threads. Suppose that a developer
> is trying to debug a deadlock situation in a program with 3 threads. A
> and B are deadlocking, and C is a "supporting" thread without which the
> other two can't run. He can't use all-stop debugging because while
> inspecting A and B, C needs to be running. In this case, relative timing
> of threads is important in order to have better chance at reproducing
> the deadlock.

Something that would be nice would be the ability to define "thread
groups", that you could stop and start as a group, restrict
breakpoints to, and so on.  You could put A and B in a thread group,
and leave C out of it.  That kind of feature would be straightforward
to implement in terms of the non-stop debugging we're doing now.