thoughts about exception-handling requirements for kprobes

thoughts about exception-handling requirements for kprobes

[Richard J Moore]

I've been thinking about the need for exception-handling and how the
current implementation has become a little muddled.

These are the categories we need to think about:


1) Expected exception, user pre-handler.
here the pre-handler code knows it's going to do something dodgy and
doesn't want to cause a  big fuss. If the memory it accesses causes a fault
then the pre-handler continues with something else.
Where could this occur: on  gathering data for a trace record from various
pageable locations. Under such a circumstance one would not want to have
the pre-handler cancelled (i.e. terminated prematurely)  because one or
more items were not available.

2) Unexpected exception, user pre-handler.

here the pre-handler has either a bug or is debugging a badly damaged
environment. Let's not forget that the latter environment is very import to
cater for as best we can.
The response should at the very least be to quietly cancel the pre-handler.
It's also conceivable that one might want to intercept this to free off any
locks and put out an explanatory message.

3) exception of probed instruction during single-step.

kprobes needs to know this has happened so that the usual clean-up can be
done following single-step. In addition kprobes need to fix-up certain
processor flags appropriately if the exception was on a flag altering
instruction. The user post-handler generally doesn't want to be called
since that results in duplicated trace records, one for each retry of the
probed instruction if retry is attempted. There are two cases where the
user post handler doesn't want to know about the single-step  generated an
unexpected exception.

a) where the exception on the probed instruction in not retryable - e.g. it
traps rather than faults. In this case we will never get the chance to
record information about that event.
b) where we are interested specifically in the number of times an
instruction is retried.

4) Expected exception, user post-handler.
same consideration as pre-handler.

5) unexpected exception, user post-handler
same consideration as pre-handler.



It is possible that systemtap might not yet want to exploit all of these 5
categories. However, I can see circumstances where natively written probe
handlers would. And I can see that systemtap might well want to do this is
due course.


Cases 1 & 4
The expected exceptions in user pre- and post-handlers are easy to deal
with in a way that performs well: this is via a setjmp/longjmp mechanism.
setjmp is trivial to implement; examples can be seen in kgdb and xmon and
arguably setjmp ought to be a common kernel routine for certain categories
of use.

So a user pre-hander might do something like this:


if (setjmp(jbuf) == 0) {

   do something dodgy;

}
else printf("yes, that was dodgy");


From a fault-handler the user would do:

if (setjmp-buf-set-up) {
   longjmp(buf,1);
}



However, it seems pointless to call the pagefault handler just to have it
execute the longjmp. Also the page-fault handler needs to be able to
determine unequivocally that the fault it was entered for was fenced by a
setjmp.

It would seem better to have kprobes put a wrapper around setjmp (ksetjmp)
and issue the longjmp directly. kprobes could also maintain a maxfault
counter for the probe so that we can exit recursive fault situations.

setjmp is truly trivial to implement and an exceedingly low overhead. I
can't see a good reason for not implementing this.  Could we exploit if
from systemtap?
How about using a try-catch semantic:

try {

     do some dodgy stuff;
}
catch {

     phew();
}

next();
.
.

Cases 2 & 5:

Here we need to allow the user-probe to clean up from unexpected
exceptions. The page-fault handler as currently specified would seem to be
ideal. There would be no need to surface this at the systemtap scripting
level.


Case 3:

I understand the many users don't have a need for this so the
implementation should not impact those users. In this case I suggest that
either
a) a duplicate post-handler hook be allowed by kprobes that would only be
called if the user registers such a handler and the single-step results in
an exception.

b) we allow the user to specify that the post-handler should be called for
both the exception and non-exception cases then have the call to the post
handler indicate which has happened.

In either case (a or b), on return from the user handler, kprobes would do
the normal back-end probepoint handling.


Since systemtap currently hides pre- and post-handling from the scripting
level, it seems unlikely that this would be exploited without expanding the
specification of a probepoint in some messy way (e.g.probe pre syscall.open
vs probe post syscall.open vs probe fault syscall.open).  But it might be
nice to see if there's a way to support case 3a implicitly. I am strongly
of the opinion that we shouldn't handle case 3 though the page-fault
handler as we attempted to in earlier versions of kprobes - there are just
too many diverse potential reasons for the page-fault handler to be called.


- -
Richard J Moore
IBM Advanced Linux Response Team - Linux Technology Centre
MOBEX: 264807; Mobile (+44) (0)7739-875237
Office: (+44) (0)1962-817072

Re: thoughts about exception-handling requirements for kprobes

[Frank Ch. Eigler]

richardj_moore wrote:

> I've been thinking about the need for exception-handling [...]
> Cases 1 & 4
> The expected exceptions in user pre- and post-handlers are easy to deal
> with in a way that performs well: this is via a setjmp/longjmp mechanism.
> [...]

Right.

> It would seem better to have kprobes put a wrapper around setjmp
> (ksetjmp) and issue the longjmp directly. kprobes could also
> maintain a maxfault counter for the probe so that we can exit
> recursive fault situations.

Sure.  Perhaps the "kprobe" struct could get a jmpbuf field that could
be initialized using a call from within the pre/post handler.  The
default kprobes fault handler could check whether this field was set,
and if so, just longjmp to it.

> [...] Could we exploit if from systemtap?

Certainly - the standard probe prologue/epilogue could hook into it
just inside the lock/unlock sequence.  Given the data in the context
(such as last_stmt), it could synthesize an error message to at least
place blame at the right source-level point.

> How about using a try-catch semantic [...]

If necessary, some nested structure like that could be put also into
the scripting language.  But if all we want is general crash
protection, then the single implicit try/catch around the entire
handler block may well be enough.


- FChE

Re: thoughts about exception-handling requirements for kprobes

[Keshavamurthy Anil S]

On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> 
>    I've been thinking about the need for exception-handling and how the
>    current implementation has become a little muddled.

Here is my thinking on this kprobe fault handling...
Ideally we want the ability to recover from all 
the page faults happening from either pre-handler 
or happening from post-handler transparently in the 
same way as the normal kernel would recover from 
do_page_fault() function. In order for this to happen, 
I think we should not be calling pre-handler/post-handler
by disabling preempt which is a major design change.
Also in the current code if fixup_exception() fails to
fixup the exception then falling back on the normal
do_page_fault() is a bad thing with preempt disabled.

I was thinking on this issue for the past several days 
and I believe that currently we are disabling preempt 
before calling pre/post handler, because we don;t 
want the process to get migrated to different CPU 
and we don't want another process to be scheduled 
while we are servicing kprobe as the newly scheduled
process might trigger another probe and we don;t 
have space to save the kprobe control block(kprobe_ctlbk) 
info, because we save kprobe_ctlbk in the per cpu structure.

If we move this saving kprobe_ctlbk to task struct then
I think we will have the ability to call pre/post-handler
without having to disable preempt and their by any faults
happening from either pre/post handler can recover transparently
in the same way as the normal kernel would recover.


Any comments on the above idea? I still don;t have a good
solution for unregisteration of kprobes if we move to above
design.


>    2) Unexpected exception, user pre-handler.
> 
>    here the pre-handler has either a bug or is debugging a badly damaged
>    environment.  Let's  not  forget  that  the latter environment is very
>    import to
>    cater for as best we can.
>    The  response  should  at  the  very  least  be  to quietly cancel the
>    pre-handler.
>    It's  also  conceivable  that one might want to intercept this to free
>    off any
>    locks and put out an explanatory message.

Yes, it would be nice to recover even from a buggy pre/post_handler and 
disable the buggy probe completly with some explanatory message.


Thanks,
Anil

Re: thoughts about exception-handling requirements for kprobes

[Prasanna S Panchamukhi]

On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> > 
> >    I've been thinking about the need for exception-handling and how the
> >    current implementation has become a little muddled.
> 
> Here is my thinking on this kprobe fault handling...
> Ideally we want the ability to recover from all 
> the page faults happening from either pre-handler 
> or happening from post-handler transparently in the 
> same way as the normal kernel would recover from 
> do_page_fault() function. In order for this to happen, 
> I think we should not be calling pre-handler/post-handler
> by disabling preempt which is a major design change.
> Also in the current code if fixup_exception() fails to
> fixup the exception then falling back on the normal
> do_page_fault() is a bad thing with preempt disabled.
> 
> I was thinking on this issue for the past several days 
> and I believe that currently we are disabling preempt 
> before calling pre/post handler, because we don;t 
> want the process to get migrated to different CPU 
> and we don't want another process to be scheduled 
> while we are servicing kprobe as the newly scheduled
> process might trigger another probe and we don;t 
> have space to save the kprobe control block(kprobe_ctlbk) 
> info, because we save kprobe_ctlbk in the per cpu structure.
> 
> If we move this saving kprobe_ctlbk to task struct then
> I think we will have the ability to call pre/post-handler
> without having to disable preempt and their by any faults
> happening from either pre/post handler can recover transparently
> in the same way as the normal kernel would recover.
> 

Kprobes user-specified pre/post handler are called within
the interrupt context and if we allow page faults while within
user-specified pre/post handler, then it might sleep.
Is is ok to sleep while within the interrupt handler?

Thanks
Prasanna
> 
> 
> >    2) Unexpected exception, user pre-handler.
> > 
> >    here the pre-handler has either a bug or is debugging a badly damaged
> >    environment.  Let's  not  forget  that  the latter environment is very
> >    import to
> >    cater for as best we can.
> >    The  response  should  at  the  very  least  be  to quietly cancel the
> >    pre-handler.
> >    It's  also  conceivable  that one might want to intercept this to free
> >    off any
> >    locks and put out an explanatory message.
> 
> Yes, it would be nice to recover even from a buggy pre/post_handler and 
> disable the buggy probe completly with some explanatory message.
> 
> 
> Thanks,
> Anil

-- 
Prasanna S Panchamukhi
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-51776329

Re: thoughts about exception-handling requirements for kprobes

[Richard J Moore]

systemtap-owner@sourceware.org wrote on 19/03/2006 17:24:54:

> On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> > >
> > >    I've been thinking about the need for exception-handling and how
the
> > >    current implementation has become a little muddled.
> >
> > Here is my thinking on this kprobe fault handling...
> > Ideally we want the ability to recover from all
> > the page faults happening from either pre-handler
> > or happening from post-handler transparently in the
> > same way as the normal kernel would recover from
> > do_page_fault() function. In order for this to happen,
> > I think we should not be calling pre-handler/post-handler
> > by disabling preempt which is a major design change.
> > Also in the current code if fixup_exception() fails to
> > fixup the exception then falling back on the normal
> > do_page_fault() is a bad thing with preempt disabled.
> >
> > I was thinking on this issue for the past several days
> > and I believe that currently we are disabling preempt
> > before calling pre/post handler, because we don;t
> > want the process to get migrated to different CPU
> > and we don't want another process to be scheduled
> > while we are servicing kprobe as the newly scheduled
> > process might trigger another probe and we don;t
> > have space to save the kprobe control block(kprobe_ctlbk)
> > info, because we save kprobe_ctlbk in the per cpu structure.
> >
> > If we move this saving kprobe_ctlbk to task struct then
> > I think we will have the ability to call pre/post-handler
> > without having to disable preempt and their by any faults
> > happening from either pre/post handler can recover transparently
> > in the same way as the normal kernel would recover.
> >
>
> Kprobes user-specified pre/post handler are called within
> the interrupt context and if we allow page faults while within

Clarify what you mean by "allow"

> user-specified pre/post handler, then it might sleep.

Clarify what you mean by "it"


Richard

.
.


> --
> Prasanna S Panchamukhi
> Linux Technology Center
> India Software Labs, IBM Bangalore
> Email: prasanna@in.ibm.com
> Ph: 91-80-51776329

Re: thoughts about exception-handling requirements for kprobes

[Prasanna S Panchamukhi]

On Mon, Mar 20, 2006 at 08:44:00AM +0000, Richard J Moore wrote:
> 
> 
> 
> 
> systemtap-owner@sourceware.org wrote on 19/03/2006 17:24:54:
> 
> > On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> > > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> > > >
> > > >    I've been thinking about the need for exception-handling and how
> the
> > > >    current implementation has become a little muddled.
> > >
> > > Here is my thinking on this kprobe fault handling...
> > > Ideally we want the ability to recover from all
> > > the page faults happening from either pre-handler
> > > or happening from post-handler transparently in the
> > > same way as the normal kernel would recover from
> > > do_page_fault() function. In order for this to happen,
> > > I think we should not be calling pre-handler/post-handler
> > > by disabling preempt which is a major design change.
> > > Also in the current code if fixup_exception() fails to
> > > fixup the exception then falling back on the normal
> > > do_page_fault() is a bad thing with preempt disabled.
> > >
> > > I was thinking on this issue for the past several days
> > > and I believe that currently we are disabling preempt
> > > before calling pre/post handler, because we don;t
> > > want the process to get migrated to different CPU
> > > and we don't want another process to be scheduled
> > > while we are servicing kprobe as the newly scheduled
> > > process might trigger another probe and we don;t
> > > have space to save the kprobe control block(kprobe_ctlbk)
> > > info, because we save kprobe_ctlbk in the per cpu structure.
> > >
> > > If we move this saving kprobe_ctlbk to task struct then
> > > I think we will have the ability to call pre/post-handler
> > > without having to disable preempt and their by any faults
> > > happening from either pre/post handler can recover transparently
> > > in the same way as the normal kernel would recover.
> > >
> >
> > Kprobes user-specified pre/post handler are called within
> > the interrupt context and if we allow page faults while within
> 
> Clarify what you mean by "allow"
> 
> > user-specified pre/post handler, then it might sleep.
> 
> Clarify what you mean by "it"
> 

In simple words, system do_page_fault() can sleep and if we
allow page faults while within user-specified kprobes pre/post
handler, then the pre/post handler might sleep. So can the
user-specifed kprobes per/post handler sleep while within
interrupt context?

Thanks
Prasanna
-- 
Prasanna S Panchamukhi
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-51776329

Re: thoughts about exception-handling requirements for kprobes

[Richard J Moore]

- -
Richard J Moore
IBM Advanced Linux Response Team - Linux Technology Centre
MOBEX: 264807; Mobile (+44) (0)7739-875237
Office: (+44) (0)1962-817072

systemtap-owner@sourceware.org wrote on 20/03/2006 08:44:00:

>
>
>
>
> systemtap-owner@sourceware.org wrote on 19/03/2006 17:24:54:
>
> > On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> > > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> > > >
> > > >    I've been thinking about the need for exception-handling and how
> the
> > > >    current implementation has become a little muddled.
> > >
> > > Here is my thinking on this kprobe fault handling...
> > > Ideally we want the ability to recover from all
> > > the page faults happening from either pre-handler
> > > or happening from post-handler transparently in the
> > > same way as the normal kernel would recover from
> > > do_page_fault() function. In order for this to happen,
> > > I think we should not be calling pre-handler/post-handler
> > > by disabling preempt which is a major design change.
> > > Also in the current code if fixup_exception() fails to
> > > fixup the exception then falling back on the normal
> > > do_page_fault() is a bad thing with preempt disabled.
> > >
> > > I was thinking on this issue for the past several days
> > > and I believe that currently we are disabling preempt
> > > before calling pre/post handler, because we don;t
> > > want the process to get migrated to different CPU
> > > and we don't want another process to be scheduled
> > > while we are servicing kprobe as the newly scheduled
> > > process might trigger another probe and we don;t
> > > have space to save the kprobe control block(kprobe_ctlbk)
> > > info, because we save kprobe_ctlbk in the per cpu structure.
> > >
> > > If we move this saving kprobe_ctlbk to task struct then
> > > I think we will have the ability to call pre/post-handler
> > > without having to disable preempt and their by any faults
> > > happening from either pre/post handler can recover transparently
> > > in the same way as the normal kernel would recover.
> > >
> >
> > Kprobes user-specified pre/post handler are called within
> > the interrupt context and if we allow page faults while within
>
> Clarify what you mean by "allow"
>
> > user-specified pre/post handler, then it might sleep.
>
> Clarify what you mean by "it"
>


No I understand the context to this remark (thanks Prasanna).

So yes, one cannot allow slwwping in a probe handler in general.
However it might be possible to introduce a scheme to allow page-in under
limited circumstances.
I've seen kernel debuggers do this.
For example,  if we took a probepoint in user-space then the probe handler
doesn't interrupt kernel processing
so in theory doen't recurse back into the kernel and cause lock contention.
Under this circumstace it might be possible to read the swap space
synchronously but that would prohibit having any probes in the code path
that did the page-in.
I have seen kernel debuggers do the following for the case where we do take
a breakpoint inside the kernel:
They effect a page-in by scheduling a kernel thread do it and then have
that thread breakin when the page-in is complete.
But essentially this is having two beak-ins to handle a page-in and we
might as well allow the normal page mgmt to do its stuff and when the
probed insruction is retired the same probe handler will fire and the
memory will now be present.

So, the only saving we could make is if we can identfy that the probe is in
a code path that does not own any kernel locks: user-space and some of
those "no mans land" areas of kernel space. Under these circumstances we
might be able to effect a synchronous page-in,  but that depends very much
on the design of the Linux page/swap manager. We might have to make a
significant modification to that component to support doing this. We might
also have to modify some kernel locks - at least those which when held
would will prohibit recursion into page/swp mgmt - to allow them to be
identified as being currently owned.
It s possible, but is it practicalble?

Richard


>
> Richard
>
> .
> .
>
>
> > --
> > Prasanna S Panchamukhi
> > Linux Technology Center
> > India Software Labs, IBM Bangalore
> > Email: prasanna@in.ibm.com
> > Ph: 91-80-51776329
>

Re: thoughts about exception-handling requirements for kprobes

[Frank Ch. Eigler]

Prasanna S Panchamukhi <prasanna@in.ibm.com> writes:

> [...]  In simple words, system do_page_fault() can sleep [...]  So
> can the user-specifed kprobes per/post handler sleep while within
> interrupt context?

The *must* not.  If they *can*, we have a problem.

- FChE

Re: thoughts about exception-handling requirements for kprobes

[Keshavamurthy Anil S]

On Sun, Mar 19, 2006 at 09:24:54AM -0800, Prasanna S Panchamukhi wrote:
> 
>    On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
>    > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
>    > >
>    >  >     I've  been thinking about the need for exception-handling and
>    how the
>    > >    current implementation has become a little muddled.
>    >
>    > Here is my thinking on this kprobe fault handling...
>    > Ideally we want the ability to recover from all
>    > the page faults happening from either pre-handler
>    > or happening from post-handler transparently in the
>    > same way as the normal kernel would recover from
>    > do_page_fault() function. In order for this to happen,
>    > I think we should not be calling pre-handler/post-handler
>    > by disabling preempt which is a major design change.
>    > Also in the current code if fixup_exception() fails to
>    > fixup the exception then falling back on the normal
>    > do_page_fault() is a bad thing with preempt disabled.
>    >
>    > I was thinking on this issue for the past several days
>    > and I believe that currently we are disabling preempt
>    > before calling pre/post handler, because we don;t
>    > want the process to get migrated to different CPU
>    > and we don't want another process to be scheduled
>    > while we are servicing kprobe as the newly scheduled
>    > process might trigger another probe and we don;t
>    > have space to save the kprobe control block(kprobe_ctlbk)
>    > info, because we save kprobe_ctlbk in the per cpu structure.
>    >
>    > If we move this saving kprobe_ctlbk to task struct then
>    > I think we will have the ability to call pre/post-handler
>    > without having to disable preempt and their by any faults
>    > happening from either pre/post handler can recover transparently
>    > in the same way as the normal kernel would recover.
>    >
> 
>    Kprobes user-specified pre/post handler are called within
>    the interrupt context and if we allow page faults while within
>    user-specified pre/post handler, then it might sleep.
>    Is is ok to sleep while within the interrupt handler?
Prasanna,
	I am not getting what you are asking here, if you are
asking is it okay to sleep while within the interrupt handler,
then it is BIG NO.

What I am saying is that we should look into kprobes to see
if we can support calling users pre/post handlers
without having to disable preempt.

Currenlty we are calling users pre_handler() and post_handler()
with preempt disabled. If the user has put a probes on 
syscalls, then when his pre/post handlers are called he is
bound to call copy_from_user(), which has a check might_sleep().
The might_sleep() calls in_atomic() function which checks preempt_count()
and if preempt_count() is greater than zero( in our case it indeed greater
than zero, since we are calling pre/post handlers with preempt disabled)
 the kernel prints a error message
printk(KERN_ERR "Debug: sleeping function called from invalid"
                                " context at %s:%d\n", file, line);

Also if we want to fallback on do_page_fault() function in kprobe_fault_handler() to 
recover the page, then we should not be in preempt_disabled() state.

Let me know if you more questions.

-Anil




	

Re: thoughts about exception-handling requirements for kprobes

[Richard J Moore]

systemtap-owner@sourceware.org wrote on 20/03/2006 18:39:51:

> On Sun, Mar 19, 2006 at 09:24:54AM -0800, Prasanna S Panchamukhi wrote:
> >
> >    On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S
wrote:
> >    > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> >    > >
> >    >  >     I've  been thinking about the need for exception-handling
and
> >    how the
> >    > >    current implementation has become a little muddled.
> >    >
> >    > Here is my thinking on this kprobe fault handling...
> >    > Ideally we want the ability to recover from all
> >    > the page faults happening from either pre-handler
> >    > or happening from post-handler transparently in the
> >    > same way as the normal kernel would recover from
> >    > do_page_fault() function. In order for this to happen,
> >    > I think we should not be calling pre-handler/post-handler
> >    > by disabling preempt which is a major design change.
> >    > Also in the current code if fixup_exception() fails to
> >    > fixup the exception then falling back on the normal
> >    > do_page_fault() is a bad thing with preempt disabled.
> >    >
> >    > I was thinking on this issue for the past several days
> >    > and I believe that currently we are disabling preempt
> >    > before calling pre/post handler, because we don;t
> >    > want the process to get migrated to different CPU
> >    > and we don't want another process to be scheduled
> >    > while we are servicing kprobe as the newly scheduled
> >    > process might trigger another probe and we don;t
> >    > have space to save the kprobe control block(kprobe_ctlbk)
> >    > info, because we save kprobe_ctlbk in the per cpu structure.
> >    >
> >    > If we move this saving kprobe_ctlbk to task struct then
> >    > I think we will have the ability to call pre/post-handler
> >    > without having to disable preempt and their by any faults
> >    > happening from either pre/post handler can recover transparently
> >    > in the same way as the normal kernel would recover.
> >    >
> >
> >    Kprobes user-specified pre/post handler are called within
> >    the interrupt context and if we allow page faults while within
> >    user-specified pre/post handler, then it might sleep.
> >    Is is ok to sleep while within the interrupt handler?
> Prasanna,
>    I am not getting what you are asking here, if you are
> asking is it okay to sleep while within the interrupt handler,
> then it is BIG NO.
>
> What I am saying is that we should look into kprobes to see
> if we can support calling users pre/post handlers
> without having to disable preempt.

Anil, I have already explained the difficulty caused by allowing
pre-emption, namely that the ordering of  pre and post single-step events
is not necessarily preserved.
I explained how we can overcome this by using a hand-crafted per-thread
single-step stack. See my append a week or so ago on single-stepping with
interrupts enabled.


>
> Currenlty we are calling users pre_handler() and post_handler()
> with preempt disabled. If the user has put a probes on
> syscalls, then when his pre/post handlers are called he is
> bound to call copy_from_user(), which has a check might_sleep().
> The might_sleep() calls in_atomic() function which checks preempt_count()
> and if preempt_count() is greater than zero( in our case it indeed
greater
> than zero, since we are calling pre/post handlers with preempt disabled)
>  the kernel prints a error message
> printk(KERN_ERR "Debug: sleeping function called from invalid"
>                                 " context at %s:%d\n", file, line);
>
> Also if we want to fallback on do_page_fault() function in
> kprobe_fault_handler() to
> recover the page, then we should not be in preempt_disabled() state.
>
> Let me know if you more questions.
>
> -Anil
>
>
>
>
>

Re: thoughts about exception-handling requirements for kprobes

[Keshavamurthy Anil S]

On Tue, Mar 21, 2006 at 12:30:38AM +0000, Richard J Moore wrote:
> > What I am saying is that we should look into kprobes to see
> > if we can support calling users pre/post handlers
> > without having to disable preempt.
> 
> Anil, I have already explained the difficulty caused by allowing
> pre-emption, namely that the ordering of  pre and post single-step events
> is not necessarily preserved.
> I explained how we can overcome this by using a hand-crafted per-thread
> single-step stack. See my append a week or so ago on single-stepping with
> interrupts enabled.
Sure, I will take a look at this and will get back to you.
thanks,
Anil

Re: thoughts about exception-handling requirements for kprobes

[Ananth N Mavinakayanahalli]

On Mon, Mar 20, 2006 at 10:39:51AM -0800, Keshavamurthy Anil S wrote:
> On Sun, Mar 19, 2006 at 09:24:54AM -0800, Prasanna S Panchamukhi wrote:
> > 
> >    On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> >    > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> >    > >
> >    >  >     I've  been thinking about the need for exception-handling and
> >    how the
> >    > >    current implementation has become a little muddled.
> >    >
> >    > Here is my thinking on this kprobe fault handling...
> >    > Ideally we want the ability to recover from all
> >    > the page faults happening from either pre-handler
> >    > or happening from post-handler transparently in the
> >    > same way as the normal kernel would recover from
> >    > do_page_fault() function. In order for this to happen,
> >    > I think we should not be calling pre-handler/post-handler
> >    > by disabling preempt which is a major design change.
> >    > Also in the current code if fixup_exception() fails to
> >    > fixup the exception then falling back on the normal
> >    > do_page_fault() is a bad thing with preempt disabled.
> >    >
> >    > I was thinking on this issue for the past several days
> >    > and I believe that currently we are disabling preempt
> >    > before calling pre/post handler, because we don;t
> >    > want the process to get migrated to different CPU
> >    > and we don't want another process to be scheduled
> >    > while we are servicing kprobe as the newly scheduled
> >    > process might trigger another probe and we don;t
> >    > have space to save the kprobe control block(kprobe_ctlbk)
> >    > info, because we save kprobe_ctlbk in the per cpu structure.
> >    >
> >    > If we move this saving kprobe_ctlbk to task struct then
> >    > I think we will have the ability to call pre/post-handler
> >    > without having to disable preempt and their by any faults
> >    > happening from either pre/post handler can recover transparently
> >    > in the same way as the normal kernel would recover.
> >    >
> > 
> >    Kprobes user-specified pre/post handler are called within
> >    the interrupt context and if we allow page faults while within
> >    user-specified pre/post handler, then it might sleep.
> >    Is is ok to sleep while within the interrupt handler?
> Prasanna,
> 	I am not getting what you are asking here, if you are
> asking is it okay to sleep while within the interrupt handler,
> then it is BIG NO.
> 
> What I am saying is that we should look into kprobes to see
> if we can support calling users pre/post handlers
> without having to disable preempt.

The fundamental assumption that probes run with preempt disabled is what
allowed introduction of RCU for lockless execution. If this is to be
changed, we also need a thorough relook at if we can use RCU at all post
the change.

Ananth

Re: thoughts about exception-handling requirements for kprobes

[Prasanna S Panchamukhi]

On Mon, Mar 20, 2006 at 10:39:51AM -0800, Keshavamurthy Anil S wrote:
> On Sun, Mar 19, 2006 at 09:24:54AM -0800, Prasanna S Panchamukhi wrote:
> > 
> >    On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> >    > On Thu, Mar 09, 2006 at 07:57:18AM -0800, Richard J Moore wrote:
> >    > >
> >    >  >     I've  been thinking about the need for exception-handling and
> >    how the
> >    > >    current implementation has become a little muddled.
> >    >
> >    > Here is my thinking on this kprobe fault handling...
> >    > Ideally we want the ability to recover from all
> >    > the page faults happening from either pre-handler
> >    > or happening from post-handler transparently in the
> >    > same way as the normal kernel would recover from
> >    > do_page_fault() function. In order for this to happen,
> >    > I think we should not be calling pre-handler/post-handler
> >    > by disabling preempt which is a major design change.
> >    > Also in the current code if fixup_exception() fails to
> >    > fixup the exception then falling back on the normal
> >    > do_page_fault() is a bad thing with preempt disabled.
> >    >
> >    > I was thinking on this issue for the past several days
> >    > and I believe that currently we are disabling preempt
> >    > before calling pre/post handler, because we don;t
> >    > want the process to get migrated to different CPU
> >    > and we don't want another process to be scheduled
> >    > while we are servicing kprobe as the newly scheduled
> >    > process might trigger another probe and we don;t
> >    > have space to save the kprobe control block(kprobe_ctlbk)
> >    > info, because we save kprobe_ctlbk in the per cpu structure.
> >    >
> >    > If we move this saving kprobe_ctlbk to task struct then
> >    > I think we will have the ability to call pre/post-handler
> >    > without having to disable preempt and their by any faults
> >    > happening from either pre/post handler can recover transparently
> >    > in the same way as the normal kernel would recover.
> >    >
> > 
> >    Kprobes user-specified pre/post handler are called within
> >    the interrupt context and if we allow page faults while within
> >    user-specified pre/post handler, then it might sleep.
> >    Is is ok to sleep while within the interrupt handler?
> Prasanna,
> 	I am not getting what you are asking here, if you are
> asking is it okay to sleep while within the interrupt handler,
> then it is BIG NO.

Anil,

> 
> What I am saying is that we should look into kprobes to see
> if we can support calling users pre/post handlers
> without having to disable preempt.
> 
> Currenlty we are calling users pre_handler() and post_handler()
> with preempt disabled. If the user has put a probes on 
> syscalls, then when his pre/post handlers are called he is
> bound to call copy_from_user(), which has a check might_sleep().
> The might_sleep() calls in_atomic() function which checks preempt_count()
> and if preempt_count() is greater than zero( in our case it indeed greater
> than zero, since we are calling pre/post handlers with preempt disabled)
>  the kernel prints a error message
> printk(KERN_ERR "Debug: sleeping function called from invalid"
>                                 " context at %s:%d\n", file, line);

Are you trying to tell here that by allowing preemption() in the
kprobes handler, the above debug message log can be avoided?

> 
> Also if we want to fallback on do_page_fault() function in kprobe_fault_handler() to 
> recover the page, then we should not be in preempt_disabled() state.

We actually do not want to fall back on system do_page_fault() because,
it might sleep. When pre/post handler page faults, we can just try 
calling fixup_exception() (non-ia64 architectures) and try to avoid actual
do_page_fault() to be called because it might sleep().

Thanks
Prasanna
-- 
Prasanna S Panchamukhi
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-51776329

Re: thoughts about exception-handling requirements for kprobes

[Keshavamurthy Anil S]

On Tue, Mar 21, 2006 at 12:27:19AM +0530, Prasanna S Panchamukhi wrote:
> On Mon, Mar 20, 2006 at 10:39:51AM -0800, Keshavamurthy Anil S wrote:
> > On Sun, Mar 19, 2006 at 09:24:54AM -0800, Prasanna S Panchamukhi wrote:
> > > 
> > >    On Fri, Mar 17, 2006 at 01:50:57PM -0800, Keshavamurthy Anil S wrote:
> > What I am saying is that we should look into kprobes to see
> > if we can support calling users pre/post handlers
> > without having to disable preempt.
> > 
> > Currenlty we are calling users pre_handler() and post_handler()
> > with preempt disabled. If the user has put a probes on 
> > syscalls, then when his pre/post handlers are called he is
> > bound to call copy_from_user(), which has a check might_sleep().
> > The might_sleep() calls in_atomic() function which checks preempt_count()
> > and if preempt_count() is greater than zero( in our case it indeed greater
> > than zero, since we are calling pre/post handlers with preempt disabled)
> >  the kernel prints a error message
> > printk(KERN_ERR "Debug: sleeping function called from invalid"
> >                                 " context at %s:%d\n", file, line);
> 
> Are you trying to tell here that by allowing preemption() in the
> kprobes handler, the above debug message log can be avoided?
Not only that, I am asking to relook at kprobes to see if we
can reliably support calling pre/post handlers without having to
disable preemption. (I am aware that currently the lock-free 
RCU changes depends on the fact that handlers are run with preempt disabled)
If we can acheive calling pre/post handlers without having to disable preempt
then the benifit of this work is huge.

> 
> > 
> > Also if we want to fallback on do_page_fault() function in kprobe_fault_handler() to 
> > recover the page, then we should not be in preempt_disabled() state.
> 
> We actually do not want to fall back on system do_page_fault() because,
> it might sleep. When pre/post handler page faults, we can just try 
> calling fixup_exception() (non-ia64 architectures) and try to avoid actual
> do_page_fault() to be called because it might sleep().
Yes, by fixing up the exceptions(fixup_exception() call) you are avoiding
falling back to do_page_fault(), but the copy_from_user() which generated
the exception will fail and we can not 100% reliably support copy from
user from pre/post handlers even for valid address.  Also on a system 
where memory presure is high we might see lots user address
reference from pre/post handlers failing.
Is this acceptable solution for Systemtap?  Do we know how reliable is 
Dtrace in this respect?

Thanks,
Anil

Re: thoughts about exception-handling requirements for kprobes

[Frank Ch. Eigler]

anil.s.keshavamurthy wrote:

> [...]  Yes, by fixing up the exceptions(fixup_exception() call) you
> are avoiding falling back to do_page_fault(), but the
> copy_from_user() which generated the exception will fail and we can
> not 100% reliably support copy from user from pre/post handlers even
> for valid address.  [...]

I think that's an acceptable trade-off.  It may be possible to let a
systemtap user bias the system for success by arranging to page-in or
even "mlock" selected processes.

> Do we know how reliable is Dtrace in this respect?

IIRC the documentation says that user data that is paged out at the moment
of a probe will cause an error if accessed.  If so, we're "tied".

- FChE