[PATH 1/3] User space probes-take4

[PATH 1/3] User space probes-take4

[Prasanna S Panchamukhi]

Hi All,

Thanks to Bibo for testing out these patches. This patch set
fixes most of the bugs reported by Bibo. Also now user-space
probes code is moved into a separate file.
I will check these patches into the cvs soon.

Thanks
Prasanna

This kprobes patch adds support for userspace probes.  It adds a new
struct, uprobe, to the kprobes API, along with register_uprobe and
unregister_uprobe functions.  The implementation uses another new
struct, uprobe_module.

Objects
--------
struct uprobe		- Allocated per probe by the user.
struct uprobe_module	- Allocated per application by the userspace probe
			  mechanism.
struct uprobe {
	/* pointer to the pathname of the application */
	char *pathname;
	/* kprobe structure with user specified handlers */
	struct kprobe kp;
	/* hlist of all the userspace probes per application */
	struct hlist_node ulist;
	/* inode of the probed application */
	struct inode *inode;
	/* probe offset within the file */
	unsigned long offset;
};

struct uprobe_module {
	/* hlist head of all userspace probes per application */
	struct hlist_head ulist_head;
	/* list of all uprobe_module for probed application */
	struct list_head mlist;
	/* to hold path/dentry etc. */
	struct nameidata nd;
	/* original readpage operations */
	struct address_space_operations *ori_a_ops;
	/* readpage hooks added operations */
	struct address_space_operations user_a_ops;
};

Explanation of struct members:

Before calling register_uprobe, the user sets the following members
of struct uprobe:
pathname - the pathname of the probed application's executable/library file
kp - the kprobe object that specifies the handler(s) to run when the
	probe is hit, and the virtual address (kp.addr) at which to
	place the probe.
offset - is the absolute offset of the probe point from the beginning
	 of the probed executable/library file.

The remaining members are for internal use.

uprobe members inode and offset uniquely identify each probe, where:
inode  - is the inode of the probed application.
	  of the file.

	When the probe is hit, get_uprobe() walks the kprobes hash table
	to find the uprobe structure with the matching inode and offset.
	This is more efficient than searching for the application's
	uprobe_module and then walking that uprobe_module's list of
	uprobes.

ulist_head and ulist - holds all uprobes for an executable/library file.
	During readpage() operations, it walks the per-executable/library file
	probe list and inserts the probes.

mlist - list of all the probed executable/library files. During readpage()
	operations, the module list is used to find the matching probed
	file based on the inode. This list is protected by uprobe_mutex.

nd -	holds the path of the probed executable/library file until all the
	inserted probes are removed for that executable/library file.

ori_a_ops and user_a_ops - are used to hold the readpage pointers and readpage()
	hooks.

Interfaces :

1. register_uprobe(struct uprobe *uprobe) : accepts a pointer to uprobe.
User has to allocate the uprobes structure and initialize following
elements:
	pathname	- points to the application's pathname
	offset		- offset of the probe from the file beginning;
[It's still the case that the user has to specify the offset as well
as the address (see TODO list)]
			  In case of library calls, the offset is the
			  relative offset from the beginning of the of
			  the mapped library.
	kp.addr		- virtual address within the executable.
	kp.pre_handler	- handler to be executed when probe is fired.
	kp.post_handler	- handler to be executed after single stepping
			  the original instruction.
	kp.fault_handler- handler to be executed if fault occurs while
                          executing the original instruction or the
                          handlers.

As with a kprobe, the user should not modify the uprobe while it is
registered. This routine returns zero on successful registeration.

2. unregister_uprobe(struct uprobe *uprobe) : accepts a pointer to uprobe.

Usage:
Usage is similar to kprobe.
	/* Allocate a uprobe structure */
	struct uprobe p;

	/* Define pre handler */
	int handler_pre(struct kprobe *p, struct pt_regs *regs)
	{
		<.............collect useful data..............>
		return 0;
	}

	void handler_post(struct kprobe *p, struct pt_regs *regs,
							unsigned long flags)
	{
		<.............collect useful data..............>
	}

	int handler_fault(struct kprobe *p, struct pt_regs *regs, int trapnr)
	{
		<.............collect useful data..............>
		[return ... what?]
	}

	Before inserting the probe, specify the pathname of the application
	on which the probe is to be inserted.

	/*pointer to the pathname of the application */
	p.pathname = "/home/prasanna/bin/myapp";
	p.kp.pre_handler=handler_pre;
	p.kp.post_handler=handler_post;
	p.kp.fault_handler=handler_fault;

	/* Secify the probe address */
	/* $nm appln |grep func1 */
	p.kp.addr = (kprobe_opcode_t *)0x080484d4;
	/* Specify the offset within the application/executable*/
	p.offset = (unsigned long)0x4d4;
	/* Now register the userspace probe */
	if (ret = register_uprobe(&p))
		printk("register_uprobe: unsuccessful ret= %d\n", ret);

	/* To unregister the registered probed, just call..*/
	unregister_uprobe(&p);

Signed-off-by : Prasanna S Panchamukhi <prasanna@in.ibm.com>



 arch/i386/kernel/uprobes.c |   70 +++++
 fs/namei.c                 |   11 
 include/linux/kprobes.h    |   49 +++
 include/linux/namei.h      |    1 
 kernel/Makefile            |    2 
 kernel/kprobes.c           |    2 
 kernel/uprobes.c           |  591 +++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 722 insertions(+), 4 deletions(-)

diff -puN fs/namei.c~kprobes_userspace_probes-base-interface fs/namei.c
--- linux-2.6.16-rc6-mm1/fs/namei.c~kprobes_userspace_probes-base-interface	2006-03-15 10:06:15.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/fs/namei.c	2006-03-15 10:06:15.000000000 +0530
@@ -322,10 +322,8 @@ int get_write_access(struct inode * inod
 	return 0;
 }
 
-int deny_write_access(struct file * file)
+int deny_write_access_to_inode(struct inode *inode)
 {
-	struct inode *inode = file->f_dentry->d_inode;
-
 	spin_lock(&inode->i_lock);
 	if (atomic_read(&inode->i_writecount) > 0) {
 		spin_unlock(&inode->i_lock);
@@ -337,6 +335,13 @@ int deny_write_access(struct file * file
 	return 0;
 }
 
+int deny_write_access(struct file * file)
+{
+	struct inode *inode = file->f_dentry->d_inode;
+
+	return deny_write_access_to_inode(inode);
+}
+
 void path_release(struct nameidata *nd)
 {
 	dput(nd->dentry);
diff -puN include/linux/kprobes.h~kprobes_userspace_probes-base-interface include/linux/kprobes.h
--- linux-2.6.16-rc6-mm1/include/linux/kprobes.h~kprobes_userspace_probes-base-interface	2006-03-15 10:06:15.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/include/linux/kprobes.h	2006-03-15 11:15:16.000000000 +0530
@@ -37,6 +37,10 @@
 #include <linux/spinlock.h>
 #include <linux/rcupdate.h>
 #include <linux/mutex.h>
+#include <linux/mm.h>
+#include <linux/dcache.h>
+#include <linux/namei.h>
+#include <linux/pagemap.h>
 
 #ifdef CONFIG_KPROBES
 #include <asm/kprobes.h>
@@ -54,6 +58,7 @@ struct kprobe;
 struct pt_regs;
 struct kretprobe;
 struct kretprobe_instance;
+extern struct uprobe *current_uprobe;
 typedef int (*kprobe_pre_handler_t) (struct kprobe *, struct pt_regs *);
 typedef int (*kprobe_break_handler_t) (struct kprobe *, struct pt_regs *);
 typedef void (*kprobe_post_handler_t) (struct kprobe *, struct pt_regs *,
@@ -117,6 +122,32 @@ struct jprobe {
 DECLARE_PER_CPU(struct kprobe *, current_kprobe);
 DECLARE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk);
 
+struct uprobe {
+	/* pointer to the pathname of the application */
+	char *pathname;
+	/* kprobe structure with user specified handlers */
+	struct kprobe kp;
+	/* hlist of all the userspace probes per application */
+	struct hlist_node ulist;
+	/* inode of the probed application */
+	struct inode *inode;
+	/* probe offset within the file */
+	unsigned long offset;
+};
+
+struct uprobe_module {
+	/* hlist head of all userspace probes per application */
+	struct hlist_head ulist_head;
+	/* list of all uprobe_module for probed application */
+	struct list_head mlist;
+	/* to hold path/dentry etc. */
+	struct nameidata nd;
+	/* original readpage operations */
+	struct address_space_operations *ori_a_ops;
+	/* readpage hooks added operations */
+	struct address_space_operations user_a_ops;
+};
+
 #ifdef ARCH_SUPPORTS_KRETPROBES
 extern void arch_prepare_kretprobe(struct kretprobe *rp, struct pt_regs *regs);
 #else /* ARCH_SUPPORTS_KRETPROBES */
@@ -162,9 +193,14 @@ extern void show_registers(struct pt_reg
 extern kprobe_opcode_t *get_insn_slot(void);
 extern void free_insn_slot(kprobe_opcode_t *slot);
 extern void kprobes_inc_nmissed_count(struct kprobe *p);
+extern int arch_copy_uprobe(struct kprobe *p, kprobe_opcode_t *address);
+extern void arch_arm_uprobe(kprobe_opcode_t *address);
+extern void arch_disarm_uprobe(struct kprobe *p, kprobe_opcode_t *address);
 
 /* Get the kprobe at this addr (if any) - called with preemption disabled */
 struct kprobe *get_kprobe(void *addr);
+struct kprobe *get_uprobe(void *addr);
+extern int arch_alloc_insn(struct kprobe *p);
 struct hlist_head * kretprobe_inst_table_head(struct task_struct *tsk);
 
 /* kprobe_running() will just return the current_kprobe on this CPU */
@@ -183,6 +219,16 @@ static inline struct kprobe_ctlblk *get_
 	return (&__get_cpu_var(kprobe_ctlblk));
 }
 
+static inline void set_uprobe_instance(struct kprobe *p)
+{
+	current_uprobe = container_of(p, struct uprobe, kp);
+}
+
+static inline void reset_uprobe_instance(void)
+{
+	current_uprobe = NULL;
+}
+
 int register_kprobe(struct kprobe *p);
 void unregister_kprobe(struct kprobe *p);
 int setjmp_pre_handler(struct kprobe *, struct pt_regs *);
@@ -194,6 +240,9 @@ void jprobe_return(void);
 int register_kretprobe(struct kretprobe *rp);
 void unregister_kretprobe(struct kretprobe *rp);
 
+int register_uprobe(struct uprobe *uprobe);
+void unregister_uprobe(struct uprobe *uprobe);
+
 struct kretprobe_instance *get_free_rp_inst(struct kretprobe *rp);
 void add_rp_inst(struct kretprobe_instance *ri);
 void kprobe_flush_task(struct task_struct *tk);
diff -puN include/linux/namei.h~kprobes_userspace_probes-base-interface include/linux/namei.h
--- linux-2.6.16-rc6-mm1/include/linux/namei.h~kprobes_userspace_probes-base-interface	2006-03-15 10:06:15.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/include/linux/namei.h	2006-03-15 10:06:15.000000000 +0530
@@ -81,6 +81,7 @@ extern int follow_up(struct vfsmount **,
 
 extern struct dentry *lock_rename(struct dentry *, struct dentry *);
 extern void unlock_rename(struct dentry *, struct dentry *);
+extern int deny_write_access_to_inode(struct inode *inode);
 
 static inline void nd_set_link(struct nameidata *nd, char *path)
 {
diff -puN kernel/kprobes.c~kprobes_userspace_probes-base-interface kernel/kprobes.c
--- linux-2.6.16-rc6-mm1/kernel/kprobes.c~kprobes_userspace_probes-base-interface	2006-03-15 10:06:15.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/kernel/kprobes.c	2006-03-15 10:06:15.000000000 +0530
@@ -51,6 +51,7 @@ static struct hlist_head kretprobe_inst_
 DEFINE_MUTEX(kprobe_mutex);		/* Protects kprobe_table */
 DEFINE_SPINLOCK(kretprobe_lock);	/* Protects kretprobe_inst_table */
 static DEFINE_PER_CPU(struct kprobe *, kprobe_instance) = NULL;
+extern void init_uprobes(void);
 
 #ifdef __ARCH_WANT_KPROBES_INSN_SLOT
 /*
@@ -650,6 +651,7 @@ static int __init init_kprobes(void)
 		INIT_HLIST_HEAD(&kretprobe_inst_table[i]);
 	}
 
+	init_uprobes();
 	err = arch_init_kprobes();
 	if (!err)
 		err = register_die_notifier(&kprobe_exceptions_nb);
diff -puN /dev/null kernel/uprobes.c
--- /dev/null	2004-06-24 23:34:38.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/kernel/uprobes.c	2006-03-15 11:15:18.000000000 +0530
@@ -0,0 +1,591 @@
+/*
+ *  User-space Probes (UProbes)
+ *  kernel/uprobes.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright (C) IBM Corporation, 2006
+ *
+ * 2006-Mar	Created by Prasanna S Panchamukhi <prasanna@in.ibm.com>
+ *		User-space probes initial implementation.
+ */
+#include <linux/kprobes.h>
+#include <linux/hash.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+#include <linux/moduleloader.h>
+#include <asm-generic/sections.h>
+#include <asm/cacheflush.h>
+#include <asm/errno.h>
+#include <asm/kdebug.h>
+
+#define UPROBE_HASH_BITS 6
+#define UPROBE_TABLE_SIZE (1 << UPROBE_HASH_BITS)
+
+/* user space probes lists */
+static struct list_head uprobe_module_list;
+static struct hlist_head uprobe_table[UPROBE_TABLE_SIZE];
+DEFINE_SPINLOCK(uprobe_lock);		/* Protects uprobe_table*/
+DEFINE_MUTEX(uprobe_mutex);		/* Protects uprobe_module_table */
+
+/*
+ * Aggregate handlers for multiple uprobes support - these handlers
+ * take care of invoking the individual uprobe handlers on p->list
+ */
+static int __kprobes aggr_user_pre_handler(struct kprobe *p,
+							 struct pt_regs *regs)
+{
+	struct kprobe *kp;
+
+	list_for_each_entry(kp, &p->list, list) {
+		if (kp->pre_handler) {
+			set_uprobe_instance(kp);
+			if (kp->pre_handler(kp, regs))
+				return 1;
+		}
+	}
+	return 0;
+}
+
+static void __kprobes aggr_user_post_handler(struct kprobe *p,
+				struct pt_regs *regs, unsigned long flags)
+{
+	struct kprobe *kp;
+
+	list_for_each_entry(kp, &p->list, list) {
+		if (kp->post_handler) {
+			set_uprobe_instance(kp);
+			kp->post_handler(kp, regs, flags);
+		}
+	}
+}
+
+static int __kprobes aggr_user_fault_handler(struct kprobe *p,
+					struct pt_regs *regs, int trapnr)
+{
+	struct kprobe *cur;
+
+	/*
+	 * if we faulted "during" the execution of a user specified
+	 * probe handler, invoke just that probe's fault handler
+	 */
+	cur = &current_uprobe->kp;
+	if (cur && cur->fault_handler)
+		if (cur->fault_handler(cur, regs, trapnr))
+			return 1;
+	return 0;
+}
+
+/**
+ * This routine looks for an existing uprobe at the given offset and inode.
+ * If it's found, returns the corresponding kprobe pointer.
+ * This should be called with uprobe_lock held.
+ */
+static struct kprobe __kprobes *get_kprobe_user(struct inode *inode,
+							unsigned long offset)
+{
+	struct hlist_head *head;
+	struct hlist_node *node;
+	struct kprobe *p, *kpr;
+	struct uprobe *uprobe;
+
+	head = &uprobe_table[hash_ptr((kprobe_opcode_t *)
+			(((unsigned long)inode) * offset), UPROBE_HASH_BITS)];
+
+	hlist_for_each_entry(p, node, head, hlist) {
+		if (p->pre_handler == aggr_user_pre_handler) {
+			kpr = list_entry(p->list.next, typeof(*kpr), list);
+			uprobe = container_of(kpr, struct uprobe, kp);
+		} else
+			uprobe = container_of(p, struct uprobe, kp);
+
+		if ((uprobe->inode == inode) && (uprobe->offset == offset))
+			return p;
+	}
+
+	return NULL;
+}
+
+/**
+ * Finds a uprobe at the specified user-space address in the current task.
+ * Points current_uprobe at that uprobe and returns the corresponding kprobe.
+ */
+struct kprobe __kprobes *get_uprobe(void *addr)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma;
+	struct inode *inode;
+	unsigned long offset;
+	struct kprobe *p, *kpr;
+	struct uprobe *uprobe;
+
+	vma = find_vma(mm, (unsigned long)addr);
+
+	BUG_ON(!vma);	/* this should not happen, not in our memory map */
+
+	offset = (unsigned long)addr - vma->vm_start +
+						(vma->vm_pgoff << PAGE_SHIFT);
+	if (!vma->vm_file)
+		return NULL;
+
+	inode = vma->vm_file->f_dentry->d_inode;
+
+	p = get_kprobe_user(inode, offset);
+	if (!p)
+		return NULL;
+
+	if (p->pre_handler == aggr_user_pre_handler) {
+		/*
+		 * Walk the uprobe aggrigate list and return firt
+		 * element on aggrigate list.
+		 */
+		kpr = list_entry((p)->list.next, typeof(*kpr), list);
+		uprobe = container_of(kpr, struct uprobe, kp);
+	} else
+		uprobe = container_of(p, struct uprobe, kp);
+
+	if (uprobe)
+		current_uprobe = uprobe;
+
+	return p;
+}
+
+/*
+ * Keep all fields in the kprobe consistent
+ */
+static inline void copy_uprobe(struct kprobe *old_p, struct kprobe *p)
+{
+	memcpy(&p->opcode, &old_p->opcode, sizeof(kprobe_opcode_t));
+	memcpy(&p->ainsn, &old_p->ainsn, sizeof(struct arch_specific_insn));
+}
+
+/*
+ * Fill in the required fields of the "manager uprobe". Replace the
+ * earlier kprobe in the hlist with the manager uprobe
+ */
+static inline void add_aggr_uprobe(struct kprobe *ap, struct kprobe *p)
+{
+	copy_uprobe(p, ap);
+	ap->addr = p->addr;
+	ap->pre_handler = aggr_user_pre_handler;
+	ap->post_handler = aggr_user_post_handler;
+	ap->fault_handler = aggr_user_fault_handler;
+
+	INIT_LIST_HEAD(&ap->list);
+	list_add(&p->list, &ap->list);
+
+	hlist_replace_rcu(&p->hlist, &ap->hlist);
+}
+
+/*
+ * This is the second or subsequent uprobe at the address - handle
+ * the intricacies
+ */
+static int __kprobes register_aggr_uprobe(struct kprobe *old_p,
+					  struct kprobe *p)
+{
+	int ret = 0;
+	struct kprobe *ap;
+
+	if (old_p->pre_handler == aggr_user_pre_handler) {
+		copy_uprobe(old_p, p);
+		list_add(&p->list, &old_p->list);
+	} else {
+		ap = kzalloc(sizeof(struct kprobe), GFP_ATOMIC);
+		if (!ap)
+			return -ENOMEM;
+		add_aggr_uprobe(ap, old_p);
+		copy_uprobe(ap, p);
+		list_add(&p->list, &old_p->list);
+	}
+	return ret;
+}
+
+typedef int (*process_uprobe_func_t)(struct uprobe *uprobe,
+				kprobe_opcode_t *address);
+
+/**
+ * Saves the original instruction in the uprobe structure and
+ * inserts the breakpoint at the given address.
+ */
+int __kprobes insert_kprobe_user(struct uprobe *uprobe,
+				kprobe_opcode_t *address)
+{
+	int ret = 0;
+
+	ret = arch_copy_uprobe(&uprobe->kp, address);
+	if (ret) {
+		printk("Breakpoint already present\n");
+		return ret;
+	}
+	arch_arm_uprobe(address);
+
+	return 0;
+}
+
+/**
+ * Wait for the page to be unlocked if someone else had locked it,
+ * then map the page and insert or remove the breakpoint.
+ */
+static int __kprobes map_uprobe_page(struct page *page, struct uprobe *uprobe,
+				     process_uprobe_func_t process_kprobe_user)
+{
+	int ret = 0;
+	kprobe_opcode_t *uprobe_address;
+
+	if (!page)
+		return -EINVAL; /* TODO: more suitable errno */
+
+	wait_on_page_locked(page);
+	/* could probably retry readpage here. */
+	if (!PageUptodate(page))
+		return -EINVAL; /* TODO: more suitable errno */
+
+	lock_page(page);
+
+	uprobe_address = (kprobe_opcode_t *)kmap(page);
+	uprobe_address = (kprobe_opcode_t *)((unsigned long)uprobe_address +
+						(uprobe->offset & ~PAGE_MASK));
+	ret = (*process_kprobe_user)(uprobe, uprobe_address);
+	kunmap(page);
+
+	unlock_page(page);
+
+	return ret;
+}
+
+/**
+ * flush_vma walks through the list of process private mappings,
+ * gets the vma containing the offset and flush all the vma's
+ * containing the probed page.
+ */
+static void __kprobes flush_vma(struct address_space *mapping,
+				struct page *page, struct uprobe *uprobe)
+{
+	struct vm_area_struct *vma = NULL;
+	struct prio_tree_iter iter;
+	struct prio_tree_root *head = &mapping->i_mmap;
+	struct mm_struct *mm;
+	unsigned long start, end, offset = uprobe->offset;
+
+	spin_lock(&mapping->i_mmap_lock);
+	vma_prio_tree_foreach(vma, &iter, head, offset, offset) {
+		mm = vma->vm_mm;
+		start = vma->vm_start - (vma->vm_pgoff << PAGE_SHIFT);
+		end = vma->vm_end - (vma->vm_pgoff << PAGE_SHIFT);
+
+		if ((start + offset) < end)
+			flush_icache_user_range(vma, page,
+					(unsigned long)uprobe->kp.addr,
+						sizeof(kprobe_opcode_t));
+	}
+	spin_unlock(&mapping->i_mmap_lock);
+}
+
+/**
+ * Walk the uprobe_module_list and return the uprobe module with matching
+ * inode.
+ */
+static struct uprobe_module __kprobes *get_module_by_inode(struct inode *inode)
+{
+	struct uprobe_module *umodule;
+
+	list_for_each_entry(umodule, &uprobe_module_list, mlist) {
+		if (umodule->nd.dentry->d_inode == inode)
+			return umodule;
+	}
+
+	return NULL;
+}
+
+/**
+ * Gets exclusive write access to the given inode to ensure that the file
+ * on which probes are currently applied does not change. Use the function,
+ * deny_write_access_to_inode() we added in fs/namei.c.
+ */
+static inline int ex_write_lock(struct inode *inode)
+{
+	return deny_write_access_to_inode(inode);
+}
+
+/**
+ * Called when removing user space probes to release the write lock on the
+ * inode.
+ */
+static inline int ex_write_unlock(struct inode *inode)
+{
+	atomic_inc(&inode->i_writecount);
+	return 0;
+}
+
+/**
+ * Add uprobe and uprobe_module to the appropriate hash list.
+ */
+static void __kprobes get_inode_ops(struct uprobe *uprobe,
+				   struct uprobe_module *umodule)
+{
+	INIT_HLIST_HEAD(&umodule->ulist_head);
+	hlist_add_head(&uprobe->ulist, &umodule->ulist_head);
+	list_add(&umodule->mlist, &uprobe_module_list);
+}
+
+/*
+ * Removes the specified uprobe from either aggrigate uprobe list
+ * or individual uprobe hash table.
+ */
+
+static int __kprobes remove_uprobe(struct uprobe *uprobe)
+{
+	struct kprobe *old_p, *list_p, *p;
+	int ret = 0;
+
+	p = &uprobe->kp;
+	old_p = get_kprobe_user(uprobe->inode, uprobe->offset);
+	if (unlikely(!old_p))
+		return 0;
+
+	if (p != old_p) {
+		list_for_each_entry(list_p, &old_p->list, list)
+			if (list_p == p)
+			/* kprobe p is a valid probe */
+				goto valid_p;
+		return 0;
+	}
+
+valid_p:
+	if ((old_p == p) ||
+			((old_p->pre_handler == aggr_user_pre_handler) &&
+			(p->list.next == &old_p->list) &&
+			(p->list.prev == &old_p->list))) {
+		/* Only probe on the hash list */
+			ret = 1;
+		hlist_del(&old_p->hlist);
+		if (p != old_p) {
+			list_del(&p->list);
+			kfree(old_p);
+		}
+	} else
+		list_del(&p->list);
+
+	return ret;
+}
+
+/*
+ * Disarms the probe and frees the corresponding instruction slot.
+ */
+static int __kprobes remove_kprobe_user(struct uprobe *uprobe,
+				kprobe_opcode_t *address)
+{
+	struct kprobe *p = &uprobe->kp;
+
+	arch_disarm_uprobe(p, address);
+	arch_remove_kprobe(p);
+
+	return 0;
+}
+
+/*
+ * Adds the given uprobe to the uprobe_hash table if it is
+ * the first probe to be inserted at the given address else
+ * adds to the aggrigate uprobe's list.
+ */
+static int __kprobes insert_uprobe(struct uprobe *uprobe)
+{
+	struct kprobe *old_p;
+	int ret = 0;
+	unsigned long offset = uprobe->offset;
+	unsigned long inode = (unsigned long) uprobe->inode;
+	struct hlist_head *head;
+	unsigned long flags;
+
+	spin_lock_irqsave(&uprobe_lock, flags);
+	uprobe->kp.nmissed = 0;
+
+	old_p = get_kprobe_user(uprobe->inode, uprobe->offset);
+
+	if (old_p)
+		register_aggr_uprobe(old_p, &uprobe->kp);
+	else {
+		head = &uprobe_table[hash_ptr((kprobe_opcode_t *)
+					(offset * inode), UPROBE_HASH_BITS)];
+		INIT_HLIST_NODE(&uprobe->kp.hlist);
+		hlist_add_head(&uprobe->kp.hlist, head);
+		ret = 1;
+	}
+
+	spin_unlock_irqrestore(&uprobe_lock, flags);
+
+	return ret;
+}
+
+/**
+ * unregister_uprobe: Disarms the probe, removes the uprobe
+ * pointers from the hash list and unhooks readpage routines.
+ */
+void __kprobes unregister_uprobe(struct uprobe *uprobe)
+{
+	struct address_space *mapping;
+	struct uprobe_module *umodule;
+	struct page *page;
+	unsigned long flags;
+	int ret = 0;
+
+	if (!uprobe->inode)
+		return;
+
+	mapping = uprobe->inode->i_mapping;
+
+	page = find_get_page(mapping, uprobe->offset >> PAGE_CACHE_SHIFT);
+
+	spin_lock_irqsave(&uprobe_lock, flags);
+	ret = remove_uprobe(uprobe);
+	spin_unlock_irqrestore(&uprobe_lock, flags);
+
+	mutex_lock(&uprobe_mutex);
+	if (!(umodule = get_module_by_inode(uprobe->inode)))
+		goto out;
+
+	hlist_del(&uprobe->ulist);
+	if (hlist_empty(&umodule->ulist_head)) {
+		list_del(&umodule->mlist);
+		ex_write_unlock(uprobe->inode);
+		path_release(&umodule->nd);
+		kfree(umodule);
+	}
+
+out:
+	mutex_unlock(&uprobe_mutex);
+	if (ret)
+		ret = map_uprobe_page(page, uprobe, remove_kprobe_user);
+
+	if (ret == -EINVAL)
+		return;
+	/*
+	 * TODO: unregister_uprobe should not fail, need to handle
+	 * if it fails.
+	 */
+	flush_vma(mapping, page, uprobe);
+
+	if (page)
+		page_cache_release(page);
+}
+
+/**
+ * register_uprobe(): combination of inode and offset is used to
+ * identify each probe uniquely. Each uprobe can be found from the
+ * uprobes_hash table by using inode and offset. register_uprobe(),
+ * inserts the breakpoint at the given address by locating and mapping
+ * the page. return 0 on success and error on failure.
+ */
+int __kprobes register_uprobe(struct uprobe *uprobe)
+{
+	struct address_space *mapping;
+	struct uprobe_module *umodule = NULL;
+	struct inode *inode;
+	struct nameidata nd;
+	struct page *page;
+	int error = 0;
+
+	INIT_HLIST_NODE(&uprobe->ulist);
+
+	/*
+	 * TODO: Need to calculate the absolute file offset for dynamic
+	 * shared libraries.
+	 */
+	if ((error = path_lookup(uprobe->pathname, LOOKUP_FOLLOW, &nd)))
+		return error;
+
+	mutex_lock(&uprobe_mutex);
+
+	inode = nd.dentry->d_inode;
+	error = ex_write_lock(inode);
+	if (error)
+		goto out;
+
+	/*
+	 * Check if there are probes already on this application and
+	 * add the corresponding uprobe to per application probe's list.
+	 */
+	umodule = get_module_by_inode(inode);
+	if (!umodule) {
+
+		error = arch_alloc_insn(&uprobe->kp);
+		if (error)
+			goto out;
+
+		/*
+		 * Allocate a uprobe_module structure for this
+		 * application if not allocated before.
+		 */
+		umodule = kzalloc(sizeof(struct uprobe_module), GFP_KERNEL);
+		if (!umodule) {
+			error = -ENOMEM;
+			ex_write_unlock(inode);
+			arch_remove_kprobe(&uprobe->kp);
+			goto out;
+		}
+		memcpy(&umodule->nd, &nd, sizeof(struct nameidata));
+		get_inode_ops(uprobe, umodule);
+	} else {
+		path_release(&nd);
+		ex_write_unlock(inode);
+		hlist_add_head(&uprobe->ulist, &umodule->ulist_head);
+	}
+	mutex_unlock(&uprobe_mutex);
+
+	uprobe->inode = inode;
+	mapping = inode->i_mapping;
+	page = find_get_page(mapping, (uprobe->offset >> PAGE_CACHE_SHIFT));
+
+	if (insert_uprobe(uprobe))
+		error = map_uprobe_page(page, uprobe, insert_kprobe_user);
+
+	/*
+	 * If error == -EINVAL, return success, probes will inserted by
+	 * readpage hooks.
+	 * TODO: Use a more suitable errno?
+	 */
+	if (error == -EINVAL)
+		error = 0;
+	flush_vma(mapping, page, uprobe);
+
+	if (page)
+		page_cache_release(page);
+
+	return error;
+out:
+	path_release(&nd);
+	mutex_unlock(&uprobe_mutex);
+
+	return error;
+}
+
+void init_uprobes(void)
+{
+	int i;
+
+	/* FIXME allocate the probe table, currently defined statically */
+	/* initialize all list heads */
+	for (i = 0; i < UPROBE_TABLE_SIZE; i++)
+		INIT_HLIST_HEAD(&uprobe_table[i]);
+
+	INIT_LIST_HEAD(&uprobe_module_list);
+}
+
+EXPORT_SYMBOL_GPL(register_uprobe);
+EXPORT_SYMBOL_GPL(unregister_uprobe);
+
+
diff -puN /dev/null arch/i386/kernel/uprobes.c
--- /dev/null	2004-06-24 23:34:38.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/arch/i386/kernel/uprobes.c	2006-03-15 11:16:45.000000000 +0530
@@ -0,0 +1,70 @@
+/*
+ *  User-space Probes (UProbes)
+ *  arch/i386/kernel/uprobes.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright (C) IBM Corporation, 2006.
+ *
+ * 2006-Mar	Created by Prasanna S Panchamukhi <prasanna@in.ibm.com>
+ *		User-space probes initial implementation.
+ */
+
+#include <linux/config.h>
+#include <linux/kprobes.h>
+#include <linux/ptrace.h>
+#include <linux/preempt.h>
+#include <asm/cacheflush.h>
+#include <asm/kdebug.h>
+#include <asm/desc.h>
+
+int __kprobes arch_alloc_insn(struct kprobe *p)
+{
+	mutex_lock(&kprobe_mutex);
+	p->ainsn.insn = get_insn_slot();
+	mutex_unlock(&kprobe_mutex);
+
+	if (!p->ainsn.insn)
+		return -ENOMEM;
+
+	return 0;
+}
+
+void __kprobes arch_disarm_uprobe(struct kprobe *p, kprobe_opcode_t *address)
+{
+	if (p->opcode != BREAKPOINT_INSTRUCTION)
+		*address = p->opcode;
+}
+
+void __kprobes arch_arm_uprobe(kprobe_opcode_t *address)
+{
+	*address = BREAKPOINT_INSTRUCTION;
+}
+
+int __kprobes arch_copy_uprobe(struct kprobe *p, kprobe_opcode_t *address)
+{
+	int ret = 1;
+
+	/*
+	 * TODO: Check if the given address is a valid to access user memory.
+	 */
+	if (*address != BREAKPOINT_INSTRUCTION) {
+		memcpy(p->ainsn.insn, address, MAX_INSN_SIZE * sizeof(kprobe_opcode_t));
+		ret = 0;
+	}
+	p->opcode = *(kprobe_opcode_t *)address;
+
+	return ret;
+}
diff -puN kernel/Makefile~kprobes_userspace_probes-base-interface kernel/Makefile
--- linux-2.6.16-rc6-mm1/kernel/Makefile~kprobes_userspace_probes-base-interface	2006-03-15 10:06:15.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/kernel/Makefile	2006-03-15 10:06:15.000000000 +0530
@@ -32,7 +32,7 @@ obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
 obj-$(CONFIG_AUDITSYSCALL) += auditsc.o
-obj-$(CONFIG_KPROBES) += kprobes.o
+obj-$(CONFIG_KPROBES) += kprobes.o uprobes.o
 obj-$(CONFIG_SYSFS) += ksysfs.o
 obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o
 obj-$(CONFIG_GENERIC_HARDIRQS) += irq/

_
-- 
Prasanna S Panchamukhi
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-51776329

Re: [PATH 2/3] User space probes-readpage-hooks-take4

[Prasanna S Panchamukhi]

This patch provides the feature of inserting probes on pages that are
not present in the memory during registration.

To add readpage and readpages() hooks, two new elements are added to
the uprobe_module object:
	struct address_space_operations *ori_a_ops;
	struct address_space_operations user_a_ops;

User-space probes allows probes to be inserted even in pages that are
not present in the memory at the time of registration. This is done
by adding hooks to the readpage and readpages routines. During
registration, the address space operation object is modified by
substituting user-space probes's specific readpage and readpages
routines. When the pages are read into memory through the readpage and
readpages address space operations, any associated probes are
automatically inserted into those pages. These user-space probes
readpage and readpages routines internally call the original
readpage() and readpages() routines, and then check whether probes are
to be added to these pages, inserting probes as necessary. The
overhead of adding these hooks is limited to the application on which
the probes are inserted.

During unregistration, care should be taken to replace the readpage and
readpages hooks with the original routines if no probes remain on that
application.

Signed-of-by: Prasanna S Panchamukhi <prasanna@in.ibm.com>


 kernel/uprobes.c |  121 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 121 insertions(+)

diff -puN kernel/uprobes.c~kprobes_userspace_probes-hook-readpage kernel/uprobes.c
--- linux-2.6.16-rc6-mm1/kernel/uprobes.c~kprobes_userspace_probes-hook-readpage	2006-03-15 11:16:53.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/kernel/uprobes.c	2006-03-15 11:16:53.000000000 +0530
@@ -311,6 +311,115 @@ static struct uprobe_module __kprobes *g
 	return NULL;
 }
 
+static inline void insert_readpage_uprobe(struct page *page,
+	struct address_space *mapping, struct uprobe *uprobe)
+{
+	unsigned long page_start = page->index << PAGE_CACHE_SHIFT;
+	unsigned long page_end = page_start + PAGE_SIZE;
+
+	if ((uprobe->offset >= page_start) && (uprobe->offset < page_end)) {
+		map_uprobe_page(page, uprobe, insert_kprobe_user);
+		flush_vma(mapping, page, uprobe);
+	}
+}
+
+/**
+ *  This function hooks the readpages() of all modules that have active
+ *  probes on them. The original readpages() is called for the given
+ *  inode/address_space to actually read the pages into the memory.
+ *  Then all probes that are specified on these pages are inserted.
+ */
+static int __kprobes uprobe_readpages(struct file *file,
+				struct address_space *mapping,
+				struct list_head *pages, unsigned nr_pages)
+{
+	int retval = 0;
+	struct page *page;
+	struct uprobe_module *umodule;
+	struct uprobe *uprobe = NULL;
+	struct hlist_node *node;
+
+	mutex_lock(&uprobe_mutex);
+
+	umodule = get_module_by_inode(file->f_dentry->d_inode);
+	if (!umodule) {
+		/*
+		 * No module associated with this file, call the
+		 * original readpages().
+		 */
+		retval = mapping->a_ops->readpages(file, mapping,
+							pages, nr_pages);
+		goto out;
+	}
+
+	/* call original readpages() */
+	retval = umodule->ori_a_ops->readpages(file, mapping, pages, nr_pages);
+	if (retval < 0)
+		goto out;
+
+	/*
+	 * TODO: Walk through readpages page list and get
+	 * pages with probes instead of find_get_page().
+	 */
+	hlist_for_each_entry(uprobe, node, &umodule->ulist_head, ulist) {
+		page = find_get_page(mapping,
+				uprobe->offset >> PAGE_CACHE_SHIFT);
+		if (!page)
+			continue;
+
+		if (!uprobe->kp.opcode)
+			insert_readpage_uprobe(page, mapping, uprobe);
+		page_cache_release(page);
+	}
+
+out:
+	mutex_unlock(&uprobe_mutex);
+
+	return retval;
+}
+
+/**
+ *  This function hooks the readpage() of all modules that have active
+ *  probes on them. The original readpage() is called for the given
+ *  inode/address_space to actually read the pages into the memory.
+ *  Then all probes that are specified on this page are inserted.
+ */
+int __kprobes uprobe_readpage(struct file *file, struct page *page)
+{
+	int retval = 0;
+	struct uprobe_module *umodule;
+	struct uprobe *uprobe = NULL;
+	struct hlist_node *node;
+	struct address_space *mapping = file->f_dentry->d_inode->i_mapping;
+
+	mutex_lock(&uprobe_mutex);
+
+	umodule = get_module_by_inode(file->f_dentry->d_inode);
+	if (!umodule) {
+		/*
+		 * No module associated with this file, call the
+		 * original readpage().
+		 */
+		retval = mapping->a_ops->readpage(file, page);
+		goto out;
+	}
+
+	/* call original readpage() */
+	retval = umodule->ori_a_ops->readpage(file, page);
+	if (retval < 0)
+		goto out;
+
+	hlist_for_each_entry(uprobe, node, &umodule->ulist_head, ulist) {
+		if (!uprobe->kp.opcode)
+			insert_readpage_uprobe(page, mapping, uprobe);
+	}
+
+out:
+	mutex_unlock(&uprobe_mutex);
+
+	return retval;
+}
+
 /**
  * Gets exclusive write access to the given inode to ensure that the file
  * on which probes are currently applied does not change. Use the function,
@@ -333,13 +442,23 @@ static inline int ex_write_unlock(struct
 
 /**
  * Add uprobe and uprobe_module to the appropriate hash list.
+ * Also swithces i_op to hooks into readpage and readpages().
  */
 static void __kprobes get_inode_ops(struct uprobe *uprobe,
 				   struct uprobe_module *umodule)
 {
+	struct address_space *as;
+
 	INIT_HLIST_HEAD(&umodule->ulist_head);
 	hlist_add_head(&uprobe->ulist, &umodule->ulist_head);
 	list_add(&umodule->mlist, &uprobe_module_list);
+	as = umodule->nd.dentry->d_inode->i_mapping;
+	umodule->ori_a_ops = as->a_ops;
+	umodule->user_a_ops = *as->a_ops;
+	umodule->user_a_ops.readpage = uprobe_readpage;
+	umodule->user_a_ops.readpages = uprobe_readpages;
+	as->a_ops = &umodule->user_a_ops;
+
 }
 
 /*
@@ -461,6 +580,8 @@ void __kprobes unregister_uprobe(struct 
 	hlist_del(&uprobe->ulist);
 	if (hlist_empty(&umodule->ulist_head)) {
 		list_del(&umodule->mlist);
+		umodule->nd.dentry->d_inode->i_mapping->a_ops =
+							umodule->ori_a_ops;
 		ex_write_unlock(uprobe->inode);
 		path_release(&umodule->nd);
 		kfree(umodule);

_
-- 
Prasanna S Panchamukhi
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-51776329

Re: [PATH 3/3] User space probes-single-stepping-out-of-line-take4

[Prasanna S Panchamukhi]

This patch provides a mechanism for probe handling and
executing the user-specified handlers.

Each userspace probe is uniquely identified by the combination of
inode and offset, hence during registeration the inode and offset
combination is added to uprobes hash table. Initially when
breakpoint instruction is hit, the uprobes hash table is looked up
for matching inode and offset. The pre_handlers are called in
sequence if multiple probes are registered. Similar to kprobes,
uprobes also adopts to single step out-of-line, so that probe miss in
SMP environment can be avoided. But for userspace probes, instruction
copied into kernel address space cannot be single stepped, hence the
instruction must be copied to user address space. The solution is to
find free space in the current process address space and then copy the
original instruction and single step that instruction.

User processes use stack space to store local variables, agruments and
return values. Normally the stack space either below or above the
stack pointer indicates the free stack space.

The instruction to be single stepped can modify the stack space,
hence before using the free stack space, sufficient stack space should
be left. The instruction is copied to the bottom of the page and check
is made such that the copied instruction does not cross the page
boundry. The copied instruction is then single stepped. Several
architectures does not allow the instruction to be executed from the
stack location, since no-exec bit is set for the stack pages. In those
architectures, the page table entry corresponding to the stack page is
identified and the no-exec bit is unset making the instruction on that
stack page to be executed.

There are situations where even the free stack space is not enough for
the user instruction to be copied and single stepped. In such
situations, the virtual memory area(vma) can be expanded beyond the
current stack vma. This expaneded stack can be used to copy the
original instruction and single step out-of-line.

Even if the vma cannot be extended then the instruction much be
executed inline, by replacing the breakpoint instruction with original
instruction.

TODO list
--------
1. Execution of probe handlers are serialized using a uprobe_lock,
   need to make them scalable.

2. Provide setjmp/longjmp mechansim to allow the handlers run in user
   mode.

3. Insert probes on copy-on-write pages. Tracks all COW pages for the
   page containing the specified probe point and inserts/removes all
   the probe points for that page.

4. Optimize the insertion of probes through readpage hooks. Identify
   all the probes to be inserted on the read page and insert them at
   once.

5. A wrapper routines to calculate the offset from the probed file
   beginning. In case of dynamic shared library, the offset is
   calculated by subtracting the address of the probe point from the
   beginning of the file mapped address.

6. Probes are visiable if a copy of the probed executable is made,
   remove probes from the new copied image.

7. Make user-space probes coexist with other debuggers like gdb etc.

Signed-off-by: Prasanna S Panchamukhi <prasanna@in.ibm.com>


 arch/i386/kernel/Makefile  |    2 
 arch/i386/kernel/kprobes.c |    4 
 arch/i386/kernel/uprobes.c |  534 +++++++++++++++++++++++++++++++++++++++++++++
 arch/i386/mm/fault.c       |    3 
 include/asm-i386/kprobes.h |   19 +
 include/linux/kprobes.h    |    8 
 6 files changed, 565 insertions(+), 5 deletions(-)

diff -puN arch/i386/kernel/kprobes.c~kprobes_userspace_probes-ss-out-of-line arch/i386/kernel/kprobes.c
--- linux-2.6.16-rc6-mm1/arch/i386/kernel/kprobes.c~kprobes_userspace_probes-ss-out-of-line	2006-03-15 11:16:56.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/arch/i386/kernel/kprobes.c	2006-03-15 11:16:56.000000000 +0530
@@ -87,7 +87,7 @@ static inline int can_boost(kprobe_opcod
 /*
  * returns non-zero if opcode modifies the interrupt flag.
  */
-static inline int is_IF_modifier(kprobe_opcode_t opcode)
+int is_IF_modifier(kprobe_opcode_t opcode)
 {
 	switch (opcode) {
 	case 0xfa:		/* cli */
@@ -567,7 +567,7 @@ int __kprobes kprobe_exceptions_notify(s
 	int ret = NOTIFY_DONE;
 
 	if (args->regs && user_mode(args->regs))
-		return ret;
+		return uprobe_exceptions_notify(self, val, data);
 
 	switch (val) {
 	case DIE_INT3:
diff -puN arch/i386/mm/fault.c~kprobes_userspace_probes-ss-out-of-line arch/i386/mm/fault.c
--- linux-2.6.16-rc6-mm1/arch/i386/mm/fault.c~kprobes_userspace_probes-ss-out-of-line	2006-03-15 11:16:56.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/arch/i386/mm/fault.c	2006-03-15 11:16:56.000000000 +0530
@@ -71,8 +71,7 @@ void bust_spinlocks(int yes)
  * 
  * This is slow, but is very rarely executed.
  */
-static inline unsigned long get_segment_eip(struct pt_regs *regs,
-					    unsigned long *eip_limit)
+unsigned long get_segment_eip(struct pt_regs *regs, unsigned long *eip_limit)
 {
 	unsigned long eip = regs->eip;
 	unsigned seg = regs->xcs & 0xffff;
diff -puN include/asm-i386/kprobes.h~kprobes_userspace_probes-ss-out-of-line include/asm-i386/kprobes.h
--- linux-2.6.16-rc6-mm1/include/asm-i386/kprobes.h~kprobes_userspace_probes-ss-out-of-line	2006-03-15 11:16:56.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/include/asm-i386/kprobes.h	2006-03-15 11:16:56.000000000 +0530
@@ -26,6 +26,7 @@
  */
 #include <linux/types.h>
 #include <linux/ptrace.h>
+#include <asm/cacheflush.h>
 
 #define  __ARCH_WANT_KPROBES_INSN_SLOT
 
@@ -77,6 +78,19 @@ struct kprobe_ctlblk {
 	struct prev_kprobe prev_kprobe;
 };
 
+/* per user probe control block */
+struct uprobe_ctlblk {
+	unsigned long uprobe_status;
+	unsigned long uprobe_saved_eflags;
+	unsigned long uprobe_old_eflags;
+	unsigned long singlestep_addr;
+	unsigned long flags;
+	struct kprobe *curr_p;
+	pte_t *upte;
+	struct page *upage;
+	struct task_struct *tsk;
+};
+
 /* trap3/1 are intr gates for kprobes.  So, restore the status of IF,
  * if necessary, before executing the original int3/1 (trap) handler.
  */
@@ -88,4 +102,9 @@ static inline void restore_interrupts(st
 
 extern int kprobe_exceptions_notify(struct notifier_block *self,
 				    unsigned long val, void *data);
+extern int uprobe_exceptions_notify(struct notifier_block *self,
+						unsigned long val, void *data);
+extern unsigned long get_segment_eip(struct pt_regs *regs,
+						unsigned long *eip_limit);
+
 #endif				/* _ASM_KPROBES_H */
diff -puN include/linux/kprobes.h~kprobes_userspace_probes-ss-out-of-line include/linux/kprobes.h
--- linux-2.6.16-rc6-mm1/include/linux/kprobes.h~kprobes_userspace_probes-ss-out-of-line	2006-03-15 11:16:56.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/include/linux/kprobes.h	2006-03-15 11:16:56.000000000 +0530
@@ -51,6 +51,13 @@
 #define KPROBE_REENTER		0x00000004
 #define KPROBE_HIT_SSDONE	0x00000008
 
+/* uprobe_status settings */
+#define UPROBE_HIT_ACTIVE	0x00000001
+#define UPROBE_HIT_SS		0x00000002
+#define UPROBE_HIT_SSDONE	0x00000004
+#define UPROBE_SS_INLINE	0x00000008
+#define UPROBE_SSDONE_INLINE	0x00000010
+
 /* Attach to insert probes on any functions which should be ignored*/
 #define __kprobes	__attribute__((__section__(".kprobes.text")))
 
@@ -183,6 +190,7 @@ struct kretprobe_instance {
 	struct task_struct *task;
 };
 
+extern spinlock_t uprobe_lock;
 extern spinlock_t kretprobe_lock;
 extern struct mutex kprobe_mutex;
 extern int arch_prepare_kprobe(struct kprobe *p);
diff -puN arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line arch/i386/kernel/uprobes.c
--- linux-2.6.16-rc6-mm1/arch/i386/kernel/uprobes.c~kprobes_userspace_probes-ss-out-of-line	2006-03-15 11:16:56.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/arch/i386/kernel/uprobes.c	2006-03-15 11:16:56.000000000 +0530
@@ -29,6 +29,11 @@
 #include <asm/cacheflush.h>
 #include <asm/kdebug.h>
 #include <asm/desc.h>
+#include <asm/uaccess.h>
+
+static struct uprobe_ctlblk uprobe_ctlblk;
+struct uprobe *current_uprobe;
+extern int is_IF_modifier(kprobe_opcode_t opcode);
 
 int __kprobes arch_alloc_insn(struct kprobe *p)
 {
@@ -68,3 +73,532 @@ int __kprobes arch_copy_uprobe(struct kp
 
 	return ret;
 }
+
+/**
+ * This routines get the pte of the page containing the specified address.
+ */
+static pte_t  __kprobes *get_uprobe_pte(unsigned long address)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+	pmd_t *pmd;
+	pte_t *pte = NULL;
+
+	pgd = pgd_offset(current->mm, address);
+	if (!pgd)
+		goto out;
+
+	pud = pud_offset(pgd, address);
+	if (!pud)
+		goto out;
+
+	pmd = pmd_offset(pud, address);
+	if (!pmd)
+		goto out;
+
+	pte = pte_alloc_map(current->mm, pmd, address);
+
+out:
+	return pte;
+}
+
+/**
+ *  This routine check for space in the current process's stack
+ *  address space. If enough address space is found, copy the original
+ *  instruction on that page for single stepping out-of-line.
+ */
+static int __kprobes copy_insn_on_new_page(struct uprobe *uprobe ,
+			struct pt_regs *regs, struct vm_area_struct *vma)
+{
+	unsigned long addr, stack_addr = regs->esp;
+	int size = MAX_INSN_SIZE * sizeof(kprobe_opcode_t);
+
+	if (vma->vm_flags & VM_GROWSDOWN) {
+		if (((stack_addr - sizeof(long long))) <
+						(vma->vm_start + size))
+			return -ENOMEM;
+		addr = vma->vm_start;
+	} else if (vma->vm_flags & VM_GROWSUP) {
+		if ((vma->vm_end - size) < (stack_addr + sizeof(long long)))
+			return -ENOMEM;
+		addr = vma->vm_end - size;
+	} else
+		return -EFAULT;
+
+	vma->vm_flags |= VM_LOCKED;
+
+	if (__copy_to_user_inatomic((unsigned long *)addr,
+				(unsigned long *)uprobe->kp.ainsn.insn, size))
+		return -EFAULT;
+
+	regs->eip = addr;
+
+	return 0;
+}
+
+/**
+ * This routine expands the stack beyond the present process address
+ * space and copies the instruction to that location, so that
+ * processor can single step out-of-line.
+ */
+static int __kprobes copy_insn_onexpstack(struct uprobe *uprobe,
+			struct pt_regs *regs, struct vm_area_struct *vma)
+{
+	unsigned long addr, vm_addr;
+	int size = MAX_INSN_SIZE * sizeof(kprobe_opcode_t);
+	struct vm_area_struct *new_vma;
+	struct mm_struct *mm = current->mm;
+
+
+	 if (!down_read_trylock(&current->mm->mmap_sem))
+		 return -ENOMEM;
+
+	if (vma->vm_flags & VM_GROWSDOWN)
+		vm_addr = vma->vm_start - size;
+	else if (vma->vm_flags & VM_GROWSUP)
+		vm_addr = vma->vm_end + size;
+	else {
+		up_read(&current->mm->mmap_sem);
+		return -EFAULT;
+	}
+
+	new_vma = find_extend_vma(mm, vm_addr);
+	if (!new_vma) {
+		up_read(&current->mm->mmap_sem);
+		return -ENOMEM;
+	}
+
+	if (new_vma->vm_flags & VM_GROWSDOWN)
+		addr = new_vma->vm_start;
+	else
+		addr = new_vma->vm_end - size;
+
+	new_vma->vm_flags |= VM_LOCKED;
+	up_read(&current->mm->mmap_sem);
+
+	if (__copy_to_user_inatomic((unsigned long *)addr,
+				(unsigned long *)uprobe->kp.ainsn.insn, size))
+		return -EFAULT;
+
+	regs->eip = addr;
+
+	return  0;
+}
+
+/**
+ * This routine checks for stack free space below the stack pointer
+ * and then copies the instructions at that location so that the
+ * processor can single step out-of-line. If there is not enough stack
+ * space or if copy_to_user fails or if the vma is invalid, it returns
+ * error.
+ */
+static int __kprobes copy_insn_onstack(struct uprobe *uprobe,
+			struct pt_regs *regs, unsigned long flags)
+{
+	unsigned long page_addr, stack_addr = regs->esp;
+	int  size = MAX_INSN_SIZE * sizeof(kprobe_opcode_t);
+	unsigned long *source = (unsigned long *)uprobe->kp.ainsn.insn;
+
+	if (flags & VM_GROWSDOWN) {
+		page_addr = stack_addr & PAGE_MASK;
+
+		if (((stack_addr - sizeof(long long))) < (page_addr + size))
+			return -ENOMEM;
+
+		if (__copy_to_user_inatomic((unsigned long *)page_addr,
+								source, size))
+			return -EFAULT;
+
+		regs->eip = page_addr;
+	} else if (flags & VM_GROWSUP) {
+		page_addr = stack_addr & PAGE_MASK;
+
+		if (page_addr == stack_addr)
+			return -ENOMEM;
+		else
+			page_addr += PAGE_SIZE;
+
+		if ((page_addr - size) < (stack_addr + sizeof(long long)))
+			return -ENOMEM;
+
+		if (__copy_to_user_inatomic(
+			(unsigned long *)(page_addr - size), source, size))
+			return -EFAULT;
+
+		regs->eip = page_addr - size;
+	} else
+		return -EINVAL;
+
+	return 0;
+}
+
+/**
+ * This routines get the page containing the probe, maps it and
+ * replaced the instruction at the probed address with specified
+ * opcode.
+ */
+void __kprobes replace_original_insn(struct uprobe *uprobe,
+				struct pt_regs *regs, kprobe_opcode_t opcode)
+{
+	kprobe_opcode_t *addr;
+	struct page *page;
+
+	page = find_get_page(uprobe->inode->i_mapping,
+					uprobe->offset >> PAGE_CACHE_SHIFT);
+	BUG_ON(!page);
+
+	__lock_page(page);
+
+	addr = (kprobe_opcode_t *)kmap_atomic(page, KM_USER1);
+	addr = (kprobe_opcode_t *)((unsigned long)addr +
+				 (unsigned long)(uprobe->offset & ~PAGE_MASK));
+	*addr = opcode;
+	/*TODO: flush vma ? */
+	kunmap_atomic(addr, KM_USER1);
+
+	unlock_page(page);
+
+	if (page)
+		page_cache_release(page);
+	regs->eip = (unsigned long)uprobe->kp.addr;
+}
+
+/**
+ * This routine provides the functionality of single stepping
+ * out-of-line. If single stepping out-of-line cannot be achieved,
+ * it replaces with the original instruction allowing it to single
+ * step inline.
+ */
+static inline int prepare_singlestep_uprobe(struct uprobe *uprobe,
+				struct uprobe_ctlblk *ucb, struct pt_regs *regs)
+{
+	unsigned long stack_addr = regs->esp, flags;
+	struct vm_area_struct *vma = NULL;
+	int err = 0;
+
+	vma = find_vma(current->mm, (stack_addr & PAGE_MASK));
+	if (!vma) {
+		/* TODO: Need better error reporting? */
+		goto no_vma;
+	}
+	flags = vma->vm_flags;
+
+	regs->eflags |= TF_MASK;
+	regs->eflags &= ~IF_MASK;
+
+	/*
+	 * Copy_insn_on_stack tries to find some room for the instruction slot
+	 * in the same page as the current esp.
+	 */
+	err = copy_insn_onstack(uprobe, regs, flags);
+
+	/*
+	 * If copy_insn_on_stack() fails, copy_insn_on_new_page() is called to
+	 * try to find some room in the next pages below the current esp;
+	 */
+	if (err)
+		err = copy_insn_on_new_page(uprobe, regs, vma);
+	/*
+	 * If copy_insn_on_new_pagek() fails, copy_insn_on_expstack() is called to
+	 * try to grow the stack's VM area by one page.
+	 */
+	if (err)
+		err = copy_insn_onexpstack(uprobe, regs, vma);
+
+	ucb->uprobe_status = UPROBE_HIT_SS;
+
+	if (!err) {
+		ucb->upte = get_uprobe_pte(regs->eip);
+		if (!ucb->upte)
+			goto no_vma;
+		ucb->upage = pte_page(*ucb->upte);
+		__lock_page(ucb->upage);
+	}
+no_vma:
+	if (err) {
+		replace_original_insn(uprobe, regs, uprobe->kp.opcode);
+		ucb->uprobe_status = UPROBE_SS_INLINE;
+	}
+
+	ucb->singlestep_addr = regs->eip;
+
+	return 0;
+}
+
+/*
+ * uprobe_handler() executes the user specified handler and setup for
+ * single stepping the original instruction either out-of-line or inline.
+ */
+static int __kprobes uprobe_handler(struct pt_regs *regs)
+{
+	struct kprobe *p;
+	int ret = 0;
+	kprobe_opcode_t *addr = NULL;
+	struct uprobe_ctlblk *ucb = &uprobe_ctlblk;
+	unsigned long limit;
+
+	spin_lock_irqsave(&uprobe_lock, ucb->flags);
+	/* preemption is disabled, remains disabled
+	 * untill we single step on original instruction.
+	 */
+	preempt_disable();
+
+	addr = (kprobe_opcode_t *)(get_segment_eip(regs, &limit) - 1);
+
+	p = get_uprobe(addr);
+	if (!p) {
+
+		if (*addr != BREAKPOINT_INSTRUCTION) {
+			/*
+			 * The breakpoint instruction was removed right
+			 * after we hit it.  Another cpu has removed
+			 * either a probepoint or a debugger breakpoint
+			 * at this address.  In either case, no further
+			 * handling of this interrupt is appropriate.
+			 * Back up over the (now missing) int3 and run
+			 * the original instruction.
+			 */
+			regs->eip -= sizeof(kprobe_opcode_t);
+			ret = 1;
+		}
+		/* Not one of ours: let kernel handle it */
+		goto no_uprobe;
+	}
+
+	if (p->opcode == BREAKPOINT_INSTRUCTION) {
+		/*
+		 * Breakpoint was already present even before the probe
+		 * was inserted, this might break some compatability with
+		 * other debuggers like gdb etc. We dont handle such probes.
+		 */
+		current_uprobe = NULL;
+		goto no_uprobe;
+	}
+
+	ucb->curr_p = p;
+	ucb->tsk = current;
+	ucb->uprobe_status = UPROBE_HIT_ACTIVE;
+	ucb->uprobe_saved_eflags = (regs->eflags & (TF_MASK | IF_MASK));
+	ucb->uprobe_old_eflags = (regs->eflags & (TF_MASK | IF_MASK));
+
+	if (p->pre_handler && p->pre_handler(p, regs))
+		/* handler has already set things up, so skip ss setup */
+		return 1;
+
+	prepare_singlestep_uprobe(current_uprobe, ucb, regs);
+	/*
+	 * Avoid scheduling the current while returning from
+	 * kernel to user mode.
+	 */
+	clear_need_resched();
+	return 1;
+
+no_uprobe:
+	spin_unlock_irqrestore(&uprobe_lock, ucb->flags);
+	preempt_enable_no_resched();
+
+	return ret;
+}
+
+/*
+ * Called after single-stepping.  p->addr is the address of the
+ * instruction whose first byte has been replaced by the "int 3"
+ * instruction.  To avoid the SMP problems that can occur when we
+ * temporarily put back the original opcode to single-step, we
+ * single-stepped a copy of the instruction.  The address of this
+ * copy is p->ainsn.insn.
+ *
+ * This function prepares to return from the post-single-step
+ * interrupt.  We have to fix up the stack as follows:
+ *
+ * 0) Typically, the new eip is relative to the copied instruction.  We
+ * need to make it relative to the original instruction.  Exceptions are
+ * return instructions and absolute or indirect jump or call instructions.
+ *
+ * 1) If the single-stepped instruction was pushfl, then the TF and IF
+ * flags are set in the just-pushed eflags, and may need to be cleared.
+ *
+ * 2) If the single-stepped instruction was a call, the return address
+ * that is atop the stack is the address following the copied instruction.
+ * We need to make it the address following the original instruction.
+ */
+static void __kprobes resume_execution_user(struct kprobe *p,
+		struct pt_regs *regs, struct uprobe_ctlblk *ucb)
+{
+	unsigned long *tos = (unsigned long *)regs->esp;
+	unsigned long next_eip = 0;
+	unsigned long copy_eip = ucb->singlestep_addr;
+	unsigned long orig_eip = (unsigned long)p->addr;
+
+	switch (p->ainsn.insn[0]) {
+	case 0x9c:		/* pushfl */
+		*tos &= ~(TF_MASK | IF_MASK);
+		*tos |= ucb->uprobe_old_eflags;
+		break;
+	case 0xc3:		/* ret/lret */
+	case 0xcb:
+	case 0xc2:
+	case 0xca:
+		regs->eflags &= ~TF_MASK;
+		next_eip = regs->eip;
+		/* eip is already adjusted, no more changes required*/
+		return;
+		break;
+	case 0xe8:		/* call relative - Fix return addr */
+		*tos = orig_eip + (*tos - copy_eip);
+		break;
+	case 0xff:
+		if ((p->ainsn.insn[1] & 0x30) == 0x10) {
+			/* call absolute, indirect */
+			/* Fix return addr; eip is correct. */
+			next_eip = regs->eip;
+			*tos = orig_eip + (*tos - copy_eip);
+		} else if (((p->ainsn.insn[1] & 0x31) == 0x20) ||
+			   ((p->ainsn.insn[1] & 0x31) == 0x21)) {
+			/* jmp near or jmp far  absolute indirect */
+			/* eip is correct. */
+			next_eip = regs->eip;
+		}
+		break;
+	case 0xea:		/* jmp absolute -- eip is correct */
+		next_eip = regs->eip;
+		break;
+	default:
+		break;
+	}
+
+	regs->eflags &= ~TF_MASK;
+	if (next_eip)
+		regs->eip = next_eip;
+	else
+		regs->eip = orig_eip + (regs->eip - copy_eip);
+}
+
+/*
+ * post_uprobe_handler(), executes the user specified handlers and
+ * resumes with the normal execution.
+ */
+static inline int post_uprobe_handler(struct pt_regs *regs)
+{
+	struct kprobe *cur;
+	struct uprobe_ctlblk *ucb;
+
+	if (!current_uprobe)
+		return 0;
+
+	ucb = &uprobe_ctlblk;
+	cur = ucb->curr_p;
+
+	if (!cur || ucb->tsk != current)
+		return 0;
+
+	if (cur->post_handler) {
+		if (ucb->uprobe_status == UPROBE_SS_INLINE)
+			ucb->uprobe_status = UPROBE_SSDONE_INLINE;
+		else
+			ucb->uprobe_status = UPROBE_HIT_SSDONE;
+		cur->post_handler(cur, regs, 0);
+	}
+
+	resume_execution_user(cur, regs, ucb);
+	regs->eflags |= ucb->uprobe_saved_eflags;
+
+	if (ucb->uprobe_status == UPROBE_SSDONE_INLINE)
+		replace_original_insn(current_uprobe, regs,
+						BREAKPOINT_INSTRUCTION);
+	else {
+		unlock_page(ucb->upage);
+		pte_unmap(ucb->upte);
+	}
+	current_uprobe = NULL;
+	spin_unlock_irqrestore(&uprobe_lock, ucb->flags);
+	preempt_enable_no_resched();
+	/*
+	 * if somebody else is singlestepping across a probe point, eflags
+	 * will have TF set, in which case, continue the remaining processing
+	 * of do_debug, as if this is not a probe hit.
+	 */
+	if (regs->eflags & TF_MASK)
+		return 0;
+
+	return 1;
+}
+
+static inline int uprobe_fault_handler(struct pt_regs *regs, int trapnr)
+{
+	struct kprobe *cur;
+	struct uprobe_ctlblk *ucb;
+	int ret = 0;
+
+	ucb = &uprobe_ctlblk;
+	cur = ucb->curr_p;
+
+	if (ucb->tsk != current || !cur)
+		return 0;
+
+	switch(ucb->uprobe_status) {
+	case UPROBE_HIT_SS:
+		unlock_page(ucb->upage);
+		pte_unmap(ucb->upte);
+		/* TODO: All acceptable number of faults before disabling */
+		replace_original_insn(current_uprobe, regs, cur->opcode);
+		/* Now reset the current probe */
+	case UPROBE_SS_INLINE:
+		regs->eip = (unsigned long)cur->addr;
+		regs->eflags |= ucb->uprobe_old_eflags;
+		regs->eflags &= ~TF_MASK;
+		current_uprobe = NULL;
+		ret = 1;
+		spin_unlock_irqrestore(&uprobe_lock, ucb->flags);
+		preempt_enable_no_resched();
+		break;
+	case UPROBE_HIT_ACTIVE:
+	case UPROBE_SSDONE_INLINE:
+	case UPROBE_HIT_SSDONE:
+		if (cur->fault_handler && cur->fault_handler(cur, regs, trapnr))
+			return 1;
+
+		if (fixup_exception(regs))
+			return 1;
+		return 1; /* The system may crash */
+		break;
+	default:
+		break;
+	}
+	return ret;
+}
+
+/*
+ * Wrapper routine to for handling exceptions.
+ */
+int __kprobes uprobe_exceptions_notify(struct notifier_block *self,
+				       unsigned long val, void *data)
+{
+	struct die_args *args = (struct die_args *)data;
+	int ret = NOTIFY_DONE;
+
+	if (args->regs->eflags & VM_MASK) {
+		/* We are in virtual-8086 mode. Return NOTIFY_DONE */
+		return ret;
+	}
+
+	switch (val) {
+	case DIE_INT3:
+		if (uprobe_handler(args->regs))
+			ret = NOTIFY_STOP;
+		break;
+	case DIE_DEBUG:
+		if (post_uprobe_handler(args->regs))
+			ret = NOTIFY_STOP;
+		break;
+	case DIE_GPF:
+	case DIE_PAGE_FAULT:
+		if (current_uprobe &&
+		    uprobe_fault_handler(args->regs, args->trapnr))
+			ret = NOTIFY_STOP;
+		break;
+	default:
+		break;
+	}
+	return ret;
+}
diff -puN arch/i386/kernel/Makefile~kprobes_userspace_probes-ss-out-of-line arch/i386/kernel/Makefile
--- linux-2.6.16-rc6-mm1/arch/i386/kernel/Makefile~kprobes_userspace_probes-ss-out-of-line	2006-03-15 11:16:56.000000000 +0530
+++ linux-2.6.16-rc6-mm1-prasanna/arch/i386/kernel/Makefile	2006-03-15 11:16:56.000000000 +0530
@@ -28,7 +28,7 @@ obj-$(CONFIG_KEXEC)		+= machine_kexec.o 
 obj-$(CONFIG_CRASH_DUMP)	+= crash_dump.o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq.o
 obj-$(CONFIG_X86_SUMMIT_NUMA)	+= summit.o
-obj-$(CONFIG_KPROBES)		+= kprobes.o
+obj-$(CONFIG_KPROBES)		+= kprobes.o uprobes.o
 obj-$(CONFIG_MODULES)		+= module.o
 obj-y				+= sysenter.o vsyscall.o
 obj-$(CONFIG_ACPI_SRAT) 	+= srat.o

_
-- 
Prasanna S Panchamukhi
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-51776329

Re: [PATH 3/3] User space probes-single-stepping-out-of-line-take4

[bibo mao]

Prasanna,
 > +	spin_lock_irqsave(&uprobe_lock, ucb->flags);
 > +	/* preemption is disabled, remains disabled
 > +	 * untill we single step on original instruction.
 > +	 */
 > +	preempt_disable();
when single step executing probed application instruction, local irq and 
  preempt are both disabled, I have two questions:
1) If probe point is about execv/exit, which will change VMA space of 
probed process, then what is next IP pointer after single stepping.
2) Some system call like read/wait will sleep by itself, but irq/preempt 
    are both disabled, there will be problem if application sleep.

thanks
bibo,mao

Prasanna S Panchamukhi wrote:
> This patch provides a mechanism for probe handling and
> executing the user-specified handlers.
> 
> Each userspace probe is uniquely identified by the combination of
> inode and offset, hence during registeration the inode and offset
> combination is added to uprobes hash table. Initially when
> breakpoint instruction is hit, the uprobes hash table is looked up
> for matching inode and offset. The pre_handlers are called in
> sequence if multiple probes are registered. Similar to kprobes,
> uprobes also adopts to single step out-of-line, so that probe miss in
> SMP environment can be avoided. But for userspace probes, instruction
> copied into kernel address space cannot be single stepped, hence the
> instruction must be copied to user address space. The solution is to
> find free space in the current process address space and then copy the
> original instruction and single step that instruction.
> 
> User processes use stack space to store local variables, agruments and
> return values. Normally the stack space either below or above the
> stack pointer indicates the free stack space.