Kprobes Support for ARM arch

Kprobes Support for ARM arch

[rsmadhvesh]

Hi All,

Recently we completed the kprobes support for ARM architecture
targetted at 2.6.16 kernel. I have uploaded these patches in the 
below CELF wiki page along with few test programs.
This implementation is tested using 2.6.16-24 kernel for 
OMAP5912 OSK reference platform. 

The patch is available in the below CELF wiki page
http://tree.celinuxforum.org/CelfPubWiki/PatchArchive

The patch can be downloaded directly from here
http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz

This implementation supports only kprobes and doesnot
support any other variants like jprobes, kretprobes etc.
Also it doesnot support branch/jump instructions probing.

I noticed some discussions regarding ARM kprobes in the
archive. We are open to suggestions and how this implementation can
be improved. 


Best Regards
Madhvesh

=============================
Madhvesh Sulibhavi
Sony India Software Centre
Bangalore
=============================

Note: My official mail id is there in the CC list. Since "disclaimer notice" in the mail
footer could not be rectified, i am making this announcement using above mail
id. You can reach me from that mail id also.

Re: Kprobes Support for ARM arch

[Ananth N Mavinakayanahalli]

On Thu, Jan 11, 2007 at 07:58:07PM +0500, rsmadhvesh@vsnl.net wrote:

Hi Madhvesh,

> Hi All,
> 
> Recently we completed the kprobes support for ARM architecture
> targetted at 2.6.16 kernel. I have uploaded these patches in the
> below CELF wiki page along with few test programs.
> This implementation is tested using 2.6.16-24 kernel for
> OMAP5912 OSK reference platform.

I am not familiar with the ARM architecture details... the following
comments are from a quick glance at the patch.

> The patch is available in the below CELF wiki page
> http://tree.celinuxforum.org/CelfPubWiki/PatchArchive
> 
> The patch can be downloaded directly from here
> http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz
> 
> This implementation supports only kprobes and doesnot
> support any other variants like jprobes, kretprobes etc.
> Also it doesnot support branch/jump instructions probing.

How are you ensuring that a kprobe register request on a branch/jump
instruction fails? I don't see any profiling done to verify what the
underlying instruction is.

See arch_prepare_kprobe() from the powerpc port as to how we fail kprobe
registration on certain instructions.

> I noticed some discussions regarding ARM kprobes in the
> archive. We are open to suggestions and how this implementation can
> be improved.

Given that quite a bit of the kprobes code for any platform is
architecture specific, its important that your code gets reviewed by the
ARM kernel gurus/maintainers, if you intend that the patch be included
in the mainline Linux kernels.

Also, you may want to port the patch to the latest kernel. There have
been some interface changes:
- kprobe modules are more portable with the addition of in-kernel symbol
  lookup.
- The page fault case gets its own notifier so there isn't a penalty paid
  when kprobes aren't in use.

Good work!

Ananth

Re: Kprobes Support for ARM arch

[rsmadhvesh]

Dear Ananth,

Ananth N Mavinakayanahalli wrote on Friday, January 12, 2007 11:02 am
> On Thu, Jan 11, 2007 at 07:58:07PM +0500, rsmadhvesh@vsnl.net wrote:
> 
> Hi Madhvesh,
> 
> > Hi All,
> > 
> > Recently we completed the kprobes support for ARM architecture
> > targetted at 2.6.16 kernel. I have uploaded these patches in the
> > below CELF wiki page along with few test programs.
> > This implementation is tested using 2.6.16-24 kernel for
> > OMAP5912 OSK reference platform.
> 
> I am not familiar with the ARM architecture details... the following
> comments are from a quick glance at the patch.
> 
> > The patch is available in the below CELF wiki page
> > http://tree.celinuxforum.org/CelfPubWiki/PatchArchive
> > 
> > The patch can be downloaded directly from here
> > 
> http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz
> > 
> > This implementation supports only kprobes and doesnot
> > support any other variants like jprobes, kretprobes etc.
> > Also it doesnot support branch/jump instructions probing.
> 
> How are you ensuring that a kprobe register request on a branch/jump
> instruction fails? I don't see any profiling done to verify what the
> underlying instruction is.
We are working out a way to check for branch/jump instructions
which can be used to take care of runtime crash if probe is placed
for those instructions.

> 
> See arch_prepare_kprobe() from the powerpc port as to how we fail 
> kproberegistration on certain instructions.

Thank you for the pointers, we are trying to address in same fashion.

> 
> > I noticed some discussions regarding ARM kprobes in the
> > archive. We are open to suggestions and how this implementation can
> > be improved.
> 
> Given that quite a bit of the kprobes code for any platform is
> architecture specific, its important that your code gets reviewed 
> by the
> ARM kernel gurus/maintainers, if you intend that the patch be included
> in the mainline Linux kernels.
> 
> Also, you may want to port the patch to the latest kernel. There have
> been some interface changes:
> - kprobe modules are more portable with the addition of in-kernel 
> symbol  lookup.
> - The page fault case gets its own notifier so there isn't a 
> penalty paid
>  when kprobes aren't in use.

It will be interesting to merge to main line kernel, i will ask
you if i find any difficulty in porting

Thank you very much for your quick feedback

Best Regards
Madhvesh

Re: Kprobes Support for ARM arch

[Quentin Barnes]

>Hi All,
>
>Recently we completed the kprobes support for ARM architecture
>targetted at 2.6.16 kernel. I have uploaded these patches in the 
>below CELF wiki page along with few test programs.
>This implementation is tested using 2.6.16-24 kernel for 
>OMAP5912 OSK reference platform. 
>
>The patch is available in the below CELF wiki page
>http://tree.celinuxforum.org/CelfPubWiki/PatchArchive
>
>The patch can be downloaded directly from here
>http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz
>
>This implementation supports only kprobes and doesnot
>support any other variants like jprobes, kretprobes etc.
>Also it doesnot support branch/jump instructions probing.
>
>I noticed some discussions regarding ARM kprobes in the
>archive. We are open to suggestions and how this implementation can
>be improved. 

Abhishek Sagar and I have been working together on a joint ARM
kprobes effort for some time.

We have kprobes, jprobes, kretprobes, and boosting support(*)
hobbling along now for ARM.  The port is also SMP clean and
preemption safe.  It is ported against 2.6.19.1.  We're pretty much
code complete and doing cleanup, some manual optimizations, and
testing now.  We're planning on releasing our patch probably about
the end of this month.

If you like, I can send you our work-in-progress for your review.

Quentin


*) The boosting model I've implemented is not the standard boosting
   model, but it effectively does the same thing my eliminating the
   second breakpoint exception.  This approach works in all cases,
   not for just some instructions, so the second breakpoint is never
   necessary.

>Best Regards
>Madhvesh
>
>=============================
>Madhvesh Sulibhavi
>Sony India Software Centre
>Bangalore
>=============================
>
>Note: My official mail id is there in the CC list. Since "disclaimer notice" in the mail
>footer could not be rectified, i am making this announcement using above mail
>id. You can reach me from that mail id also.

Re: Kprobes Support for ARM arch

[rsmadhvesh]

Quentin Barnes wrote on Saturday, January 13, 2007 1:32 am
> >Hi All,
> >
> >Recently we completed the kprobes support for ARM architecture
> >targetted at 2.6.16 kernel. I have uploaded these patches in the 
> >below CELF wiki page along with few test programs.
> >This implementation is tested using 2.6.16-24 kernel for 
> >OMAP5912 OSK reference platform. 
> >
> >The patch is available in the below CELF wiki page
> >http://tree.celinuxforum.org/CelfPubWiki/PatchArchive
> >
> >The patch can be downloaded directly from here
> >http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz
> >
> >This implementation supports only kprobes and doesnot
> >support any other variants like jprobes, kretprobes etc.
> >Also it doesnot support branch/jump instructions probing.
> >
> >I noticed some discussions regarding ARM kprobes in the
> >archive. We are open to suggestions and how this implementation can
> >be improved. 
> 
> Abhishek Sagar and I have been working together on a joint ARM
> kprobes effort for some time.
> 
> We have kprobes, jprobes, kretprobes, and boosting support(*)
> hobbling along now for ARM.  The port is also SMP clean and
> preemption safe.  It is ported against 2.6.19.1.  We're pretty much
> code complete and doing cleanup, some manual optimizations, and
> testing now.  We're planning on releasing our patch probably about
> the end of this month.
> 
> If you like, I can send you our work-in-progress for your review.
Quite interesting, it may be good idea to combine the kprobes
ARM code and start discussions on how it can be pushed
to main line kernel. Ananth in his previous mail pointed out 
few interface changes in latest kernel for kprobes. Our kprobes
support was immediately required for 2.6.16-x kernel, so we started
with that kernel. Now i start checking for main line kernel changes and
we can merge and get it reviewed by ARM experts in arm-mail 
list and then it can be pushed for main line kernel.

> *) The boosting model I've implemented is not the standard boosting
>   model, but it effectively does the same thing my eliminating the
>   second breakpoint exception.  This approach works in all cases,
>   not for just some instructions, so the second breakpoint is never
>   necessary.
> 

-Madhvesh

Re: Kprobes Support for ARM arch

[rsmadhvesh]

Hi All,

The attached patch addresses the branch instructions
check for ARM kprobes last week release and avoids 
unexpected crash when probe is placed for branch 
type instructions. If any one finds issues, please let 
me know.

Regards
Madhvesh

----- Original Message -----
From: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Date: Friday, January 12, 2007 11:02 am
Subject: Re: Kprobes Support for ARM arch
To: rsmadhvesh@vsnl.net
Cc: systemtap@sources.redhat.com, madhvesh.s@ap.sony.com, lubna@ap.sony.com, satish.v@ap.sony.com

> On Thu, Jan 11, 2007 at 07:58:07PM +0500, rsmadhvesh@vsnl.net wrote:
> 
> Hi Madhvesh,
> 
> > Hi All,
> > 
> > Recently we completed the kprobes support for ARM architecture
> > targetted at 2.6.16 kernel. I have uploaded these patches in the
> > below CELF wiki page along with few test programs.
> > This implementation is tested using 2.6.16-24 kernel for
> > OMAP5912 OSK reference platform.
> 
> I am not familiar with the ARM architecture details... the following
> comments are from a quick glance at the patch.
> 
> > The patch is available in the below CELF wiki page
> > http://tree.celinuxforum.org/CelfPubWiki/PatchArchive
> > 
> > The patch can be downloaded directly from here
> > 
> http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz
> > 
> > This implementation supports only kprobes and doesnot
> > support any other variants like jprobes, kretprobes etc.
> > Also it doesnot support branch/jump instructions probing.
> 
> How are you ensuring that a kprobe register request on a branch/jump
> instruction fails? I don't see any profiling done to verify what the
> underlying instruction is.
> 
> See arch_prepare_kprobe() from the powerpc port as to how we fail 
> kproberegistration on certain instructions.
> 
> > I noticed some discussions regarding ARM kprobes in the
> > archive. We are open to suggestions and how this implementation can
> > be improved.
> 
> Given that quite a bit of the kprobes code for any platform is
> architecture specific, its important that your code gets reviewed 
> by the
> ARM kernel gurus/maintainers, if you intend that the patch be included
> in the mainline Linux kernels.
> 
> Also, you may want to port the patch to the latest kernel. There have
> been some interface changes:
> - kprobe modules are more portable with the addition of in-kernel 
> symbol  lookup.
> - The page fault case gets its own notifier so there isn't a 
> penalty paid
>  when kprobes aren't in use.
> 
> Good work!
> 
> Ananth
> 

Re: Kprobes Support for ARM arch

[rsmadhvesh]

[-- Attachment #1: Type: text/plain, Size: 2972 bytes --]

Patch is attached now..

----- Original Message -----
From: rsmadhvesh@vsnl.net
Date: Thursday, January 18, 2007 2:22 pm
Subject: Re: Kprobes Support for ARM arch
To: systemtap@sources.redhat.com
Cc: madhvesh.s@ap.sony.com, satish.v@ap.sony.com

> Hi All,
> 
> The attached patch addresses the branch instructions
> check for ARM kprobes last week release and avoids 
> unexpected crash when probe is placed for branch 
> type instructions. If any one finds issues, please let 
> me know.
> 
> Regards
> Madhvesh
> 
> ----- Original Message -----
> From: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
> Date: Friday, January 12, 2007 11:02 am
> Subject: Re: Kprobes Support for ARM arch
> To: rsmadhvesh@vsnl.net
> Cc: systemtap@sources.redhat.com, madhvesh.s@ap.sony.com, 
> lubna@ap.sony.com, satish.v@ap.sony.com
> 
> > On Thu, Jan 11, 2007 at 07:58:07PM +0500, rsmadhvesh@vsnl.net wrote:
> > 
> > Hi Madhvesh,
> > 
> > > Hi All,
> > > 
> > > Recently we completed the kprobes support for ARM architecture
> > > targetted at 2.6.16 kernel. I have uploaded these patches in the
> > > below CELF wiki page along with few test programs.
> > > This implementation is tested using 2.6.16-24 kernel for
> > > OMAP5912 OSK reference platform.
> > 
> > I am not familiar with the ARM architecture details... the following
> > comments are from a quick glance at the patch.
> > 
> > > The patch is available in the below CELF wiki page
> > > http://tree.celinuxforum.org/CelfPubWiki/PatchArchive
> > > 
> > > The patch can be downloaded directly from here
> > > 
> > 
> http://tree.celinuxforum.org/CelfPubWiki/PatchArchive?action=AttachFile&do=get&target=kprobes-arm-patches-2.6.16.24.tgz
> > > 
> > > This implementation supports only kprobes and doesnot
> > > support any other variants like jprobes, kretprobes etc.
> > > Also it doesnot support branch/jump instructions probing.
> > 
> > How are you ensuring that a kprobe register request on a branch/jump
> > instruction fails? I don't see any profiling done to verify what the
> > underlying instruction is.
> > 
> > See arch_prepare_kprobe() from the powerpc port as to how we fail 
> > kproberegistration on certain instructions.
> > 
> > > I noticed some discussions regarding ARM kprobes in the
> > > archive. We are open to suggestions and how this implementation 
> can> > be improved.
> > 
> > Given that quite a bit of the kprobes code for any platform is
> > architecture specific, its important that your code gets reviewed 
> > by the
> > ARM kernel gurus/maintainers, if you intend that the patch be 
> included> in the mainline Linux kernels.
> > 
> > Also, you may want to port the patch to the latest kernel. There 
> have> been some interface changes:
> > - kprobe modules are more portable with the addition of in-kernel 
> > symbol  lookup.
> > - The page fault case gets its own notifier so there isn't a 
> > penalty paid
> >  when kprobes aren't in use.
> > 
> > Good work!
> > 
> > Ananth
> > 
> 

[-- Attachment #2: kprobes-arm-2.6.16.24-1-branch-check.patch --]
[-- Type: TEXT/PLAIN, Size: 4031 bytes --]

kprobes-arm-2.6.16.24-1-branch-check.patch:

This is arm kprobes additional patch that contains validation
of instructions which modify PC. This avoids unexpected crash
when probe is placed for branch type instructions 

Signed-off-by: Madhvesh Sulibhavi <madhvesh.s@ap.sony.com>
Signed-off-by: Lubna Badkar <lubna@ap.sony.com>

Index: linux-2.6.16.24/arch/arm/kernel/kprobes.c
===================================================================
--- linux-2.6.16.24.orig/arch/arm/kernel/kprobes.c	2007-01-11 19:04:48.000000000 +0530
+++ linux-2.6.16.24/arch/arm/kernel/kprobes.c	2007-01-18 13:08:51.493841440 +0530
@@ -28,6 +28,25 @@
 DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL;
 DEFINE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk);
 
+/* Following macros define opcode values for data processing
+ * instructions which should be checked if Rd happens to be 
+ * R15(PC) and currently kprobes are not supported for such
+ * instructions which directly modify PC
+ */ 
+#define OP_MASK 0x01e00000
+#define OP_AND  0x00000000
+#define OP_EOR  0x00200000
+#define OP_SUB  0x00400000
+#define OP_RSB  0x00600000
+#define OP_ADD  0x00800000
+#define OP_ADC  0x00a00000
+#define OP_SBC  0x00c00000
+#define OP_RSC  0x00e00000
+#define OP_ORR  0x01800000
+#define OP_MOV  0x01a00000
+#define OP_BIC  0x01c00000
+#define OP_MVN  0x01e00000
+
 /* To avoid the SMP problems that can occur when we temporarily 
  * put back the original opcode to single-step, we single-step
  * a copy of the instruction. The address of this copy is present
@@ -35,16 +54,112 @@
  */
 int __kprobes arch_prepare_kprobe(struct kprobe *p)
 {
+	unsigned long insn;
+	int ret = 0;
+
+	if ((unsigned long)p->addr & 0x03) {
+		printk("Attempt to register kprobe at an unaligned address\n");
+		ret = -EINVAL;
+		goto kprobe_out;
+	}
+
+	insn = *p->addr;
+
+	/* Check 25-27 bits for specific instruction types */  
+	switch (insn & 0x0e000000) {
+	/* Below case covers..
+	 * branch/link with exchange types
+	 * data processing immediate/register shift if Rd=PC
+	 */
+	case 0x00000000:
+	case 0x02000000:{
+			if ((insn & 0x0fffffd0) == 0x012fff10) {
+				/*
+				 * bx or blx
+				 */
+				printk("Kprobe for branch/link with exchange instruction type is not supported\n");
+				ret = -EINVAL;
+				goto kprobe_out;
+			}
+
+			/* If Rd is not PC, then break and kprobe is possible */
+			if ((insn & 0xf000) != 0xf000)
+				break;
+
+
+			/* If Rd happens to be PC for below insn types */
+			switch (insn & OP_MASK) {
+			case OP_AND:
+			case OP_EOR:
+			case OP_SUB:
+			case OP_RSB:
+			case OP_ADD:
+			case OP_ADC:
+			case OP_SBC:
+			case OP_RSC:
+			case OP_ORR:
+			case OP_MOV:
+			case OP_BIC:
+			case OP_MVN:
+				printk("Kprobe for data processing instructions with Rd=PC not supported\n");
+				ret = -EINVAL;
+				goto kprobe_out;
+			}
+			break;
+		}
+
+	/* Below case covers..
+	 * load immediate offset
+	 * load register offset
+	 */
+	case 0x04000000:
+	case 0x06000000:
+		/*
+		 * ldr
+		 */
+		if ((insn & 0x0010f000) == 0x0010f000) {
+			printk("Kprobe for load instructions with Rd=PC not supported\n");
+			ret = -EINVAL;
+			goto kprobe_out;
+		}
+		break;
+
+	/* Check for load multiple types with PC */
+	case 0x08000000:
+		/*
+		 * ldm
+		 */
+		if ((insn & 0x00108000) == 0x00108000) {
+			printk("Kprobe for multiple load type instructions not supported\n");
+			ret = -EINVAL;
+			goto kprobe_out;
+		}
+		break;
+
+	/* Check for branch and branch with link */
+	case 0x0a000000:
+		/*
+		 * bl or b
+		 */
+		printk("Kprobe for branch and branch with link type instructions not supported\n");
+		ret = -EINVAL;
+		goto kprobe_out;
+
+	default:
+		break;
+	}
+
 	/* insn: must be on special executable page on arm. */
 	p->ainsn.insn = get_insn_slot();
 	if (!p->ainsn.insn)
-		return -ENOMEM;
+		ret = -ENOMEM;
 
 	p->ainsn.insn[0] = *p->addr;
 	p->ainsn.insn[1] = BREAKPOINT_INSTRUCTION_2;
 	p->opcode = *p->addr;
 
-	return 0;
+kprobe_out:
+	return ret;
 }
 
 void __kprobes arch_arm_kprobe(struct kprobe *p)

Re: Kprobes Support for ARM arch

[Abhishek Sagar]

> > The attached patch addresses the branch instructions
> > check for ARM kprobes last week release and avoids
> > unexpected crash when probe is placed for branch
> > type instructions. If any one finds issues, please let
> > me know.
> >
> > Regards
> > Madhvesh

I would be a good defensive check to include bxj in the first switch
case. The goto's can also be replaces with returns in case you only
intend to do a 'return ret' in the 'kprobe_out' label.

- Abhishek Sagar

Re: Kprobes Support for ARM arch

[Abhishek Sagar]

> > > The attached patch addresses the branch instructions
> > > check for ARM kprobes last week release and avoids
> > > unexpected crash when probe is placed for branch
> > > type instructions. If any one finds issues, please let
> > > me know.
> > >
> > > Regards
> > > Madhvesh
>
> I would be a good defensive check to include bxj in the first switch
> case. The goto's can also be replaces with returns in case you only
> intend to do a 'return ret' in the 'kprobe_out' label.
>
> - Abhishek Sagar

A cpl of typos there :-\

...I wanted to add that if you plan to further the switch-case blocks
for more instructions like msr, cps etc., and do more with in the
kprobe_out label, then keeping the goto's seem fine. For more
instructions to add to your switch cases, kindly refer to the patch
sent by Quentin earlier on this thread. If you have some more
instructions to add that we might have missed in that patch, please
let us know.

Thanks & Regards
Abhishek Sagar