RE: user kprobes vs debuggers

RE: user kprobes vs debuggers

[Stone, Joshua I]

Vara Prasad wrote:
> Well come to think of  this problem i am thinking this is very similar
> to the problem between Kprobes and kernel debuggers.
> I am thinking we can have a common interface that ptrace and userspace
> probes can use to register break points similar to the notifiers for
> kernel probes. The common interface stores the information about all
> the break points including the ones from the ptrace interface.  There
> is a tag on the breakpoint that specifies whether the breakpoint is
> inserted by the userspace probes or ptrace. Similar to kernel
> multiple probe handlers at a probe point, we could have multiple
> probe handlers for user space probes as well. One of the probe
> handler among the multiple could be to pass it to the debugger for
> the probes registered by ptrace. When the break point is hit
> userspace probes gets the control and checks if there is a userspace
> probe point at this location, if yes, runs the userspace probes
> handler and if there is a ptrace registered probe point at the same
> location pass it debugger as well.  Unregistering probes of both
> types requires updating the global registery of the probe points. 

Who will single-step the original instruction in this scenario?  It
seems that the only feasible answer is that the debugger will do it.
But, in the case of a probe inserted sooner than the debugger
breakpoint, the debugger doesn't know the original instruction.  And if
the debugger removes its breakpoint, the probe-management would have to
start single-stepping.

Someone mentioned solving this by presenting the debugger with a
virtualized address-space (where the probe doesn't exist).  This may be
possible, but in the keep-it-simple spirit I think it would be best to
just reject the second-comer.  At least with a common interface we can
detect the conflict, so I think it's fine to just disallow the
situation.


Josh

Re: user kprobes vs debuggers

[Vara Prasad]

Stone, Joshua I wrote:

>Vara Prasad wrote:
>  
>
> [...]
>
>Who will single-step the original instruction in this scenario?  It
>seems that the only feasible answer is that the debugger will do it.
>But, in the case of a probe inserted sooner than the debugger
>breakpoint, the debugger doesn't know the original instruction.  And if
>the debugger removes its breakpoint, the probe-management would have to
>start single-stepping.
>  
>
Let us say if there is no user space probes involved debugger remembers 
the original instruction before replacing it is with breakpoint 
instruction. It single steps the original instruction that is stored in 
the userspace. Let us say if there is no debugger and there is only 
userspace probes userspace pobes does the same single stepping but in 
the kernel space. Another important distinction to remember here is each 
of them have their own handlers to run when the break point is hit.
Based on the above background there is a global registry of the 
breakpoints in the kernel that is only used to notify who all would like 
to handle this breakpoint but it is up to each of the owners to run 
their own handlers and as well as handle single stepping.

One complication in this problems is the applications that have compiled 
breakpoint instructions, but i am not sure how common that occurrence is 
and like i mentioned in my earlier reply we may be able to handle them 
as well.
Does this answer your question?

>Someone mentioned solving this by presenting the debugger with a
>virtualized address-space (where the probe doesn't exist).  This may be
>possible, but in the keep-it-simple spirit I think it would be best to
>just reject the second-comer.  At least with a common interface we can
>detect the conflict, so I think it's fine to just disallow the
>situation.
>  
>
I think it would have been o.k to disallow but due to common uses like 
strace people might object to userspace probes  feature.

>
>Josh
>
>  
>

RE: user kprobes vs debuggers

[Stone, Joshua I]

Vara Prasad wrote:
> Let us say if there is no user space probes involved debugger
> remembers the original instruction before replacing it is with
> breakpoint instruction. It single steps the original instruction that
> is stored in the userspace. Let us say if there is no debugger and
> there is only userspace probes userspace pobes does the same single
> stepping but in the kernel space. Another important distinction to
> remember here is each of them have their own handlers to run when the
> break point is hit. Based on the above background there is a global
> registry of the breakpoints in the kernel that is only used to notify
> who all would like to handle this breakpoint but it is up to each of
> the owners to run their own handlers and as well as handle single
> stepping. 

Sure, I see no issue when only a debugger or a userspace probe
instruments a given address - they just single-step it themselves.  The
issue I was referring to is when a debugger and a userspace probe try to
instrument the same address.  In that scenario it's not clear who should
single-step the original instruction and how.

It's made worse if probes & debuggers are registered & removed in an
interleaved fashion - e.g. probe A registers; debugger B registers; A is
removed; what happens to B?

> I think it would have been o.k to disallow but due to common uses like
> strace people might object to userspace probes  feature.

You only have to disallow probes & debugger breakpoints coexisting at
the same address - if they're at different addresses then there's no
problem.  If that's still too restrictive, then we need to resolve the
issues above.

Josh

Re: user kprobes vs debuggers

[James Dickens]

On 2/3/06, Stone, Joshua I <joshua.i.stone@intel.com> wrote:
> Vara Prasad wrote:
> > Let us say if there is no user space probes involved debugger
> > remembers the original instruction before replacing it is with
> > breakpoint instruction. It single steps the original instruction that
> > is stored in the userspace. Let us say if there is no debugger and
> > there is only userspace probes userspace pobes does the same single
> > stepping but in the kernel space. Another important distinction to
> > remember here is each of them have their own handlers to run when the
> > break point is hit. Based on the above background there is a global
> > registry of the breakpoints in the kernel that is only used to notify
> > who all would like to handle this breakpoint but it is up to each of
> > the owners to run their own handlers and as well as handle single
> > stepping.
>
> Sure, I see no issue when only a debugger or a userspace probe
> instruments a given address - they just single-step it themselves.  The
> issue I was referring to is when a debugger and a userspace probe try to
> instrument the same address.  In that scenario it's not clear who should
> single-step the original instruction and how.
>
> It's made worse if probes & debuggers are registered & removed in an
> interleaved fashion - e.g. probe A registers; debugger B registers; A is
> removed; what happens to B?
>
> > I think it would have been o.k to disallow but due to common uses like
> > strace people might object to userspace probes  feature.
>
> You only have to disallow probes & debugger breakpoints coexisting at
> the same address - if they're at different addresses then there's no
> problem.  If that's still too restrictive, then we need to resolve the
> issues above.
>

what happens in the case of  debugger put a breakpoint in a private
copy of a shared library? how do we probe and monitor this? and also
all calls to function XXX of a non private library so can monitor all 
malloc's done by userland apps for instance.


James Dickens
uadmin.blogspot.com

> Josh
>

Re: user kprobes vs debuggers

[Vara Prasad]

Stone, Joshua I wrote:

>Vara Prasad wrote:
>  
>
>>Let us say if there is no user space probes involved debugger
>>remembers the original instruction before replacing it is with
>>breakpoint instruction. It single steps the original instruction that
>>is stored in the userspace. Let us say if there is no debugger and
>>there is only userspace probes userspace pobes does the same single
>>stepping but in the kernel space. Another important distinction to
>>remember here is each of them have their own handlers to run when the
>>break point is hit. Based on the above background there is a global
>>registry of the breakpoints in the kernel that is only used to notify
>>who all would like to handle this breakpoint but it is up to each of
>>the owners to run their own handlers and as well as handle single
>>stepping. 
>>    
>>
>
>Sure, I see no issue when only a debugger or a userspace probe
>instruments a given address - they just single-step it themselves.  The
>issue I was referring to is when a debugger and a userspace probe try to
>instrument the same address.  In that scenario it's not clear who should
>single-step the original instruction and how.
>  
>
I didn't do a good job of explaining above, if there is a breakpoint 
installed at a given address by both probes and debuggers, we only need 
to single step the original instruction ones and that single stepping 
needs to be done by the debugger as it gets to handle the breakpoint the 
last. I am assuming for this discussion we only support one debugger 
debugging the program which is the current Linux model anyway.
In other words the sequence when we have a uprobe and debugger 
breakpoint at the same address is
break point is hit
Uprobes pre-handler is executed (no single stepping)
pass the control to the debugger
debugger runs the pre handler (which is gives the control to the user)
single steps the original instruction

>It's made worse if probes & debuggers are registered & removed in an
>interleaved fashion - e.g. probe A registers; debugger B registers; A is
>removed; what happens to B?
>
>  
>
Answer to your above scenario is global registry will still have 
breakpoint at the address with single handler for  the debugger after 
uprobe is removed.
I think order in which who placed the breakpoint doesn't matter as long 
as the global registry has the knowledge if there are multiple handlers 
or single handler for the given breakpoint. If there are multiple 
handlers protocol is always userspace probes first and then debugger. 

In our current implementation user space probes do allow pre-handler and 
post-handler to my knowledge only pre-handler is the common usage. If we 
have to support post-handler as well then it become bit tricky but can 
be handled. We need to then register a notifier to get the control back 
to user space probes code so that we can run post-handler. We may be 
able to do some optimization and avoid this over head if there are no 
mulitple handlers at a given address.

>>I think it would have been o.k to disallow but due to common uses like
>>strace people might object to userspace probes  feature.
>>    
>>
>
>You only have to disallow probes & debugger breakpoints coexisting at
>the same address - if they're at different addresses then there's no
>problem.  If that's still too restrictive, then we need to resolve the
>issues above.
>  
>

Like i said due to common usage of strace kind of utilities we can not 
really do that. If we allow break points if there is no collision but 
not allow when there is no collision, strace could potentially give you 
wrong results as some of its breakpoints are disallowed or it may fail 
to run.

There is one aspect of the problem that Jim reminded me that i didn't 
address which is copy on write (COW).  Usually when multiple people 
wants to run the same program text segments are shared, but if one of 
the person wants to debug the program, OS makes a copy on write for the 
debugger and removes the write protection on that copy. This poses some 
challenge with our current implementation of userspace probes which are 
for the program not process hence we don't do the copy on write unlike a 
debugger.  As you can imagine if we have put one user space probes in a 
process, multiple copies of which are being run. Then comes along a user 
who wants to debug the program and put one break point due to COW that 
copy might end up with two break points now, one from userspace probes 
and one from the debugger that is bad.

We may have to rethink our our program based approach vs process based 
approach for user space probes. If we go the route of program based 
approach we could do a COW of our own and debugger could come along 
later and do a cow of its own which makes this problem go away 
completely as each of them have their own copies. I have not looked at 
the kernel code to see if you already have a text page with write 
permission will it make another write copy version or not. I need to 
think a bit more about this issue.

>Josh
>
>  
>

user kprobes vs debuggers

[Frank Ch. Eigler]

Hi -

During the teleconference earlier today, we discussed the issue of
coexistence of user-mode kprobes (along the favoured #4 path) with
debuggers, manipulating the same tasks.

The core issue is that both systems insert breakpoints into pages
of the target text.  Ideally, we would like both systems to operate
independently, unaware of each other.  But:

Without synchronization over "ownership" of the text pages, two
systems may perform the insertion or removal interleaved in an
inconvenient way.  It may be possible to lose breakpoints, or even to
create spontaneous ones.  To perform sufficient synchronization, we
may need to (a) detect possible conflicts after the fact, (b) bluntly
block one system when the other is active, (c) hook user-kprobes into
ptrace and /proc/mem code paths to intercept debuggers' operations
and/or (d) provide a virtualization facility where the user-space
tools only see a kprobe-less image of the real text page.

A related problem is handling of breakpoints once triggered.  Clearly
user-kprobes get to run first.  The system needs to know whether user
space has also set a breakpoint at the same spot, so a subsequent
ptrace signal can be propagated to the debugger.  Some peculiar
applications may put breakpoints into themselves even without a
debugger present, expecting to catch SIGTRAP.  Ideally, user kprobes
should work with these too.

- FChE

Re: user kprobes vs debuggers

[Vara Prasad]

Well come to think of  this problem i am thinking this is very similar 
to the problem between Kprobes and kernel debuggers.
I am thinking we can have a common interface that ptrace and userspace 
probes can use to register break points similar to the notifiers for 
kernel probes. The common interface stores the information about all the 
break points including the ones from the ptrace interface.  There is a 
tag on the breakpoint that specifies whether the breakpoint is inserted 
by the userspace probes or ptrace. Similar to kernel multiple probe 
handlers at a probe point, we could have multiple probe handlers for 
user space probes as well. One of the probe handler among the multiple 
could be to pass it to the debugger for the probes registered by ptrace. 
When the break point is hit userspace probes gets the control and checks 
if there is a userspace probe point at this location, if yes, runs the 
userspace probes handler and if there is a ptrace registered probe point 
at the same location pass it debugger as well.  Unregistering probes of 
both types requires updating the global registery of the probe points. 

In the above discussion i didn't differentiate between /proc/pid/mem and 
ptrace() as they both go through the same interface in the kernel they 
are equivalent for this discussion.

I think this mechanism should handle applications that have breakpoints 
builtin as well because i am thinking they are going to use ptrace 
interface as well to register the breakpoint, is that correct.

Did i not consider something in the above design?

Frank Ch. Eigler wrote:

>Hi -
>
>During the teleconference earlier today, we discussed the issue of
>coexistence of user-mode kprobes (along the favoured #4 path) with
>debuggers, manipulating the same tasks.
>
>The core issue is that both systems insert breakpoints into pages
>of the target text.  Ideally, we would like both systems to operate
>independently, unaware of each other.  But:
>
>Without synchronization over "ownership" of the text pages, two
>systems may perform the insertion or removal interleaved in an
>inconvenient way.  It may be possible to lose breakpoints, or even to
>create spontaneous ones.  To perform sufficient synchronization, we
>may need to (a) detect possible conflicts after the fact, (b) bluntly
>block one system when the other is active, (c) hook user-kprobes into
>ptrace and /proc/mem code paths to intercept debuggers' operations
>and/or (d) provide a virtualization facility where the user-space
>tools only see a kprobe-less image of the real text page.
>
>A related problem is handling of breakpoints once triggered.  Clearly
>user-kprobes get to run first.  The system needs to know whether user
>space has also set a breakpoint at the same spot, so a subsequent
>ptrace signal can be propagated to the debugger.  Some peculiar
>applications may put breakpoints into themselves even without a
>debugger present, expecting to catch SIGTRAP.  Ideally, user kprobes
>should work with these too.
>
>- FChE
>
>  
>

Re: user kprobes vs debuggers

[Mathieu Lacage]

On Thu, 2006-02-02 at 22:37 -0800, Vara Prasad wrote:

> I think this mechanism should handle applications that have breakpoints 
> builtin as well because i am thinking they are going to use ptrace 
> interface as well to register the breakpoint, is that correct.

No, such applications (at least, mine) are going to insert int3 by hand
in the code (yes, I know, this is evil but it is possible).

If you are interested, it is some sort of profiler/tracer available from
there: http://www-sop.inria.fr/dream/personnel/Mathieu.Lacage/bozo-
profiler/

regards,
Mathieu
-- 

Re: user kprobes vs debuggers

[Vara Prasad]

Mathieu Lacage wrote:

>On Thu, 2006-02-02 at 22:37 -0800, Vara Prasad wrote:
>
>  
>
>>I think this mechanism should handle applications that have breakpoints 
>>builtin as well because i am thinking they are going to use ptrace 
>>interface as well to register the breakpoint, is that correct.
>>    
>>
>
>No, such applications (at least, mine) are going to insert int3 by hand
>in the code (yes, I know, this is evil but it is possible).
>  
>
I did a cursory glance of the code that handles the signals looks like 
in this case OS thinks there is no process tracing and there will be no 
probes hence code will work as of today which is to deliver the signal 
to the process who put the int3.

The only complication is if a user wants to add a userspace probe at 
this address then we need to check if the instruction being replaced is 
int3 in which case we don't really need to replace the instruction 
instead we should just record there are multiple breakpoints one for 
userspace probes and one for userspace application. After we run the 
userspace probes handler we need to deliver the signal to the 
application, there may be some complications in making a clean code to 
achieve this but i have not spent enough time to address this at this time.

Mathieu, do you know what behavior you get today if you try to put a breakpoint using gdb at the same address you have added int3?


>If you are interested, it is some sort of profiler/tracer available from
>there: http://www-sop.inria.fr/dream/personnel/Mathieu.Lacage/bozo-
>profiler/
>
>regards,
>Mathieu
>  
>

Re: user kprobes vs debuggers

[Ananth N Mavinakayanahalli]

On Thu, Feb 02, 2006 at 02:22:31PM -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> During the teleconference earlier today, we discussed the issue of
> coexistence of user-mode kprobes (along the favoured #4 path) with
> debuggers, manipulating the same tasks.
> 
> The core issue is that both systems insert breakpoints into pages
> of the target text.  Ideally, we would like both systems to operate
> independently, unaware of each other.  But:
> 
> Without synchronization over "ownership" of the text pages, two
> systems may perform the insertion or removal interleaved in an
> inconvenient way.  It may be possible to lose breakpoints, or even to
> create spontaneous ones.  To perform sufficient synchronization, we
> may need to (a) detect possible conflicts after the fact, (b) bluntly
> block one system when the other is active, (c) hook user-kprobes into
> ptrace and /proc/mem code paths to intercept debuggers' operations
> and/or (d) provide a virtualization facility where the user-space
> tools only see a kprobe-less image of the real text page.
> 
> A related problem is handling of breakpoints once triggered.  Clearly
> user-kprobes get to run first.  The system needs to know whether user
> space has also set a breakpoint at the same spot, so a subsequent
> ptrace signal can be propagated to the debugger.  Some peculiar
> applications may put breakpoints into themselves even without a
> debugger present, expecting to catch SIGTRAP.  Ideally, user kprobes
> should work with these too.

Just as a datapoint, atleast on PowerPC, kprobes (and xmon) use a
different "BREAKPOINT" opcode from the one used by GDB so, handling most
cases above should be trivial. In fact, we already handle the
possibility that a different debugger/trace tool may have inserted a
breakpoint at a given location (ref: is_trap() in kernel sources).

Breakpoints at the same address in userspace is a more tricky issue
though.

Ananth

Re: user kprobes vs debuggers

[Richard J Moore]

I honestly think the simplest and only possible general solution is not to
allow or support a probe to be placed on any pre-existing breakpoint.

The problem we have in general is that we cannot say what the expected
outcome of a breakpoint should be.

In some cases it will be to terminate the program i.e. exit to system.
In others it will be to enter a debugger with the option of resuming at the
same or some other point or indeed exiting.
In the case of a probepoint, which is more than a mere breakpoint, the
processing is complex and automated.


Concerns about who handles the debug exception (IA32) apply also to who
handles or intercepts any exception generated by a given instruction.

To support multiple uses of a breakpoint we would need to establish a
protocol that all users would adhere to and a set of APIs to support all
users and multiple concurrent users of APIs. Unless someone can point to a
real need to multi-user breakpoints I suggest we shelve this with the
caveat that probes placed on breakpoints and breakpoints placed on probes
are not supported. We can protect ourself from trouble by putting a test in
kprobes (1) on probe insertion to make sure that we don't install a probe
on a breakpoint and possibly (2) at single-step to make sure that we don't
SS a breakpoint. ptrace should probably make a similar test on breakpoint
insertion.

- -
Richard J Moore
IBM Advanced Linux Response Team - Linux Technology Centre
MOBEX: 264807; Mobile (+44) (0)7739-875237
Office: (+44) (0)1962-817072

systemtap-owner@sourceware.org wrote on 06/02/2006 09:58:35:

> On Thu, Feb 02, 2006 at 02:22:31PM -0500, Frank Ch. Eigler wrote:
> > Hi -
> >
> > During the teleconference earlier today, we discussed the issue of
> > coexistence of user-mode kprobes (along the favoured #4 path) with
> > debuggers, manipulating the same tasks.
> >
> > The core issue is that both systems insert breakpoints into pages
> > of the target text.  Ideally, we would like both systems to operate
> > independently, unaware of each other.  But:
> >
> > Without synchronization over "ownership" of the text pages, two
> > systems may perform the insertion or removal interleaved in an
> > inconvenient way.  It may be possible to lose breakpoints, or even to
> > create spontaneous ones.  To perform sufficient synchronization, we
> > may need to (a) detect possible conflicts after the fact, (b) bluntly
> > block one system when the other is active, (c) hook user-kprobes into
> > ptrace and /proc/mem code paths to intercept debuggers' operations
> > and/or (d) provide a virtualization facility where the user-space
> > tools only see a kprobe-less image of the real text page.
> >
> > A related problem is handling of breakpoints once triggered.  Clearly
> > user-kprobes get to run first.  The system needs to know whether user
> > space has also set a breakpoint at the same spot, so a subsequent
> > ptrace signal can be propagated to the debugger.  Some peculiar
> > applications may put breakpoints into themselves even without a
> > debugger present, expecting to catch SIGTRAP.  Ideally, user kprobes
> > should work with these too.
>
> Just as a datapoint, atleast on PowerPC, kprobes (and xmon) use a
> different "BREAKPOINT" opcode from the one used by GDB so, handling most
> cases above should be trivial. In fact, we already handle the
> possibility that a different debugger/trace tool may have inserted a
> breakpoint at a given location (ref: is_trap() in kernel sources).
>
> Breakpoints at the same address in userspace is a more tricky issue
> though.
>
> Ananth