[SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

Hi,

This is a v2 version of -rt aware systemtap patchset, tested for 3.14.12-rt9 and for 3.10.40-rt38 kernel version.
For v1->v2 related details refer [1]. Patchset based on stap upstream link [2] master branch.

Change summary;

v1->v2 :

- added Locking helper api. tested for -rt and voluntary mode, works fine.
- reverted v1 change of rd/wr lock with rcu to raw_spinlock in this patch, that
  is beacuse for rcu to effectively get in use in stap modules like utrace.c
and task_finder.c, require to make desing change in general. However it would
improve the performance,
- Removed v1's adder_map patch set, as it wasn't troubling -rt mode.

Test script used for testing :
/usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/netdev.stp
/usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/tcpdumplike.stp

Few other test example script used :
cat ../test-indent.stp
probe kernel.function("*@net/socket.c").call
{
                  printf ("%s -> %s\n", thread_indent(1), probefunc())
}
probe kernel.function("*@net/socket.c").return
{
                  printf ("%s <- %s\n", thread_indent(-1), probefunc())
}


Like know feedback, comment on patch set. also Does it make sense to maintain
systemtap -rt version in upstream.. do we care?

[1] http://sourceware-org.1504.n7.nabble.com/SYSTEMTAP-PATCH-0-4-RT-aware-systemtap-patch-set-td282463.html
[2] git://sourceware.org/git/systemtap.git; branch master 


Santosh Shukla (6):
  stp: rt: locking helper api
  stp: rt: replace __stp_tf_map_lock rd/wr lock with stp style raw lock
  stp: rt: replace __stp_tf_vma_lock rd/wr lock with stp style of raw
    lock
  stp: rt: replace utrace->lock with stp style raw lock
  stp: rt: replace utrace_struct lock to stp style raw lock
  stp: rt: replace __stp_tf_task_work_list_lock to stp raw

 runtime/linux/task_finder2.c    |  14 +++---
 runtime/linux/task_finder_map.c |  27 ++++++-----
 runtime/stp_helper_lock.h       |  80 +++++++++++++++++++++++++++++++
 runtime/stp_utrace.c            | 103 ++++++++++++++++++++--------------------
 runtime/task_finder_vma.c       |  33 ++++++-------
 5 files changed, 170 insertions(+), 87 deletions(-)
 create mode 100644 runtime/stp_helper_lock.h

-- 
1.8.3.1

[SYSTEMTAP/PATCH v2 6/6] stp: rt: replace __stp_tf_task_work_list_lock to stp raw

[Santosh Shukla]

Fixes this bug_on for -rt case :

[ 2184.284672]  [<ffffffff81602329>] dump_stack+0x4e/0x7a
[ 2184.284679]  [<ffffffff815fcaed>] __schedule_bug+0x9f/0xad
[ 2184.284686]  [<ffffffff816057f7>] __schedule+0x627/0x6a0
[ 2184.284694]  [<ffffffff810be5fb>] ? task_blocks_on_rt_mutex+0x19b/0x220
[ 2184.284699]  [<ffffffff816058a0>] schedule+0x30/0xa0
[ 2184.284707]  [<ffffffff8160727d>] rt_spin_lock_slowlock+0xbd/0x1f0
[ 2184.284714]  [<ffffffff81607e25>] rt_spin_lock+0x25/0x30
[ 2184.284727]  [<ffffffffa08ae573>] __stp_tf_alloc_task_work+0x43/0x90 [stap_63e05c06fe2b0c2d17f8d8e096a4ee8a__1700]
[ 2184.284737]  [<ffffffffa08aff8b>] __stp_utrace_task_finder_target_syscall_exit+0xdb/0x350 [stap_63e05c06fe2b0c2d17f8d8e096a4ee8a__1700]
[ 2184.284747]  [<ffffffffa08a91d5>] utrace_report_syscall_exit+0xc5/0x110 [stap_63e05c06fe2b0c2d17f8d8e096a4ee8a__1700]
[ 2184.284753]  [<ffffffff81023ce0>] syscall_trace_leave+0x100/0x130
[ 2184.284758]  [<ffffffff8161090b>] int_check_syscall_exit_work+0x34/0x3d

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
 runtime/linux/task_finder2.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/runtime/linux/task_finder2.c b/runtime/linux/task_finder2.c
index 64f2633..6195c7c 100644
--- a/runtime/linux/task_finder2.c
+++ b/runtime/linux/task_finder2.c
@@ -95,7 +95,7 @@ struct stap_task_finder_target {
 };
 
 static LIST_HEAD(__stp_tf_task_work_list);
-static DEFINE_SPINLOCK(__stp_tf_task_work_list_lock);
+static STP_DEFINE_SPINLOCK(__stp_tf_task_work_list_lock);
 struct __stp_tf_task_work {
 	struct list_head list;
 	struct task_struct *task;
@@ -132,9 +132,9 @@ __stp_tf_alloc_task_work(void *data)
 	// list for easier lookup, but as short as the list should be
 	// (and as short lived as these items are) the extra overhead
 	// probably isn't worth the effort.
-	spin_lock_irqsave(&__stp_tf_task_work_list_lock, flags);
+	stp_spin_lock_irqsave(&__stp_tf_task_work_list_lock, flags);
 	list_add(&tf_work->list, &__stp_tf_task_work_list);
-	spin_unlock_irqrestore(&__stp_tf_task_work_list_lock, flags);
+	stp_spin_unlock_irqrestore(&__stp_tf_task_work_list_lock, flags);
 
 	return &tf_work->work;
 }
@@ -150,14 +150,14 @@ static void __stp_tf_free_task_work(struct task_work *work)
 	tf_work = container_of(work, struct __stp_tf_task_work, work);
 
 	// Remove the item from the list.
-	spin_lock_irqsave(&__stp_tf_task_work_list_lock, flags);
+	stp_spin_lock_irqsave(&__stp_tf_task_work_list_lock, flags);
 	list_for_each_entry(node, &__stp_tf_task_work_list, list) {
 		if (tf_work == node) {
 			list_del(&tf_work->list);
 			break;
 		}
 	}
-	spin_unlock_irqrestore(&__stp_tf_task_work_list_lock, flags);
+	stp_spin_unlock_irqrestore(&__stp_tf_task_work_list_lock, flags);
 
 	// Actually free the data.
 	_stp_kfree(tf_work);
@@ -173,14 +173,14 @@ static void __stp_tf_cancel_task_work(void)
 	unsigned long flags;
 
 	// Cancel all remaining requests.
-	spin_lock_irqsave(&__stp_tf_task_work_list_lock, flags);
+	stp_spin_lock_irqsave(&__stp_tf_task_work_list_lock, flags);
 	list_for_each_entry_safe(node, tmp, &__stp_tf_task_work_list, list) {
 	    // Remove the item from the list, cancel it, then free it.
 	    list_del(&node->list);
 	    stp_task_work_cancel(node->task, node->work.func);
 	    _stp_kfree(node);
 	}
-	spin_unlock_irqrestore(&__stp_tf_task_work_list_lock, flags);
+	stp_spin_unlock_irqrestore(&__stp_tf_task_work_list_lock, flags);
 }
 
 static u32
-- 
1.8.3.1

[SYSTEMTAP/PATCH v2 3/6] stp: rt: replace __stp_tf_vma_lock rd/wr lock with stp style of raw lock

[Santosh Shukla]

To make it -rt aware. Description similar to previous patch
commit, RCU is better substitute for rd/wr lock [todo]

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
 runtime/task_finder_vma.c | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/runtime/task_finder_vma.c b/runtime/task_finder_vma.c
index f826982..a09093d 100644
--- a/runtime/task_finder_vma.c
+++ b/runtime/task_finder_vma.c
@@ -3,11 +3,12 @@
 
 #include <linux/list.h>
 #include <linux/jhash.h>
-#include <linux/spinlock.h>
 
 #include <linux/fs.h>
 #include <linux/dcache.h>
 
+#include "stp_helper_lock.h"
+
 // __stp_tf_vma_lock protects the hash table.
 // Documentation/spinlocks.txt suggest we can be a bit more clever
 // if we guarantee that in interrupt context we only read, not write
@@ -15,7 +16,7 @@
 // contents in interrupt context (which should only ever call 
 // stap_find_vma_map_info for getting stored vma info). So we might
 // want to look into that if this seems a bottleneck.
-static DEFINE_RWLOCK(__stp_tf_vma_lock);
+static STP_DEFINE_RWLOCK(__stp_tf_vma_lock);
 
 #define __STP_TF_HASH_BITS 4
 #define __STP_TF_TABLE_SIZE (1 << __STP_TF_HASH_BITS)
@@ -180,17 +181,17 @@ stap_add_vma_map_info(struct task_struct *tsk,
 	// Take a write lock, since we are most likely going to write
 	// after reading. But reserve a new entry first outside the lock.
 	new_entry = __stp_tf_vma_new_entry();
-	write_lock_irqsave(&__stp_tf_vma_lock, flags);
+	stp_write_lock_irqsave(&__stp_tf_vma_lock, flags);
 	entry = __stp_tf_get_vma_map_entry_internal(tsk, vm_start);
 	if (entry != NULL) {
-		write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+		stp_write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 		if (new_entry)
 			__stp_tf_vma_release_entry(new_entry);
 		return -EBUSY;	/* Already there */
 	}
 
 	if (!new_entry) {
-		write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+		stp_write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 		return -ENOMEM;
 	}
 
@@ -213,7 +214,7 @@ stap_add_vma_map_info(struct task_struct *tsk,
 
 	head = &__stp_tf_vma_map[__stp_tf_vma_map_hash(tsk)];
 	hlist_add_head(&entry->hlist, head);
-	write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+	stp_write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 	return 0;
 }
 
@@ -234,13 +235,13 @@ stap_extend_vma_map_info(struct task_struct *tsk,
 
 	// Take a write lock, since we are most likely going to write
 	// to the entry after reading, if its vm_end matches our vm_start.
-	write_lock_irqsave(&__stp_tf_vma_lock, flags);
+	stp_write_lock_irqsave(&__stp_tf_vma_lock, flags);
 	entry = __stp_tf_get_vma_map_entry_end_internal(tsk, vm_start);
 	if (entry != NULL) {
 		entry->vm_end = vm_end;
 		res = 0;
 	}
-	write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+	stp_write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 	return res;
 }
 
@@ -258,14 +259,14 @@ stap_remove_vma_map_info(struct task_struct *tsk, unsigned long vm_start)
 	// Take a write lock since we are most likely going to delete
 	// after reading.
 	unsigned long flags;
-	write_lock_irqsave(&__stp_tf_vma_lock, flags);
+	stp_write_lock_irqsave(&__stp_tf_vma_lock, flags);
 	entry = __stp_tf_get_vma_map_entry_internal(tsk, vm_start);
 	if (entry != NULL) {
 		hlist_del(&entry->hlist);
 		__stp_tf_vma_release_entry(entry);
                 rc = 0;
 	}
-	write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+	stp_write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 	return rc;
 }
 
@@ -288,7 +289,7 @@ stap_find_vma_map_info(struct task_struct *tsk, unsigned long addr,
 	if (__stp_tf_vma_map == NULL)
 		return rc;
 
-	read_lock_irqsave(&__stp_tf_vma_lock, flags);
+	stp_read_lock_irqsave(&__stp_tf_vma_lock, flags);
 	head = &__stp_tf_vma_map[__stp_tf_vma_map_hash(tsk)];
 	stap_hlist_for_each_entry(entry, node, head, hlist) {
 		if (tsk->pid == entry->pid
@@ -309,7 +310,7 @@ stap_find_vma_map_info(struct task_struct *tsk, unsigned long addr,
 			*user = found_entry->user;
 		rc = 0;
 	}
-	read_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+	stp_read_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 	return rc;
 }
 
@@ -332,7 +333,7 @@ stap_find_vma_map_info_user(struct task_struct *tsk, void *user,
 	if (__stp_tf_vma_map == NULL)
 		return rc;
 
-	read_lock_irqsave(&__stp_tf_vma_lock, flags);
+	stp_read_lock_irqsave(&__stp_tf_vma_lock, flags);
 	head = &__stp_tf_vma_map[__stp_tf_vma_map_hash(tsk)];
 	stap_hlist_for_each_entry(entry, node, head, hlist) {
 		if (tsk->pid == entry->pid
@@ -350,7 +351,7 @@ stap_find_vma_map_info_user(struct task_struct *tsk, void *user,
 			*path = found_entry->path;
 		rc = 0;
 	}
-	read_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+	stp_read_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 	return rc;
 }
 
@@ -363,7 +364,7 @@ stap_drop_vma_maps(struct task_struct *tsk)
 	struct __stp_tf_vma_entry *entry;
 
 	unsigned long flags;
-	write_lock_irqsave(&__stp_tf_vma_lock, flags);
+	stp_write_lock_irqsave(&__stp_tf_vma_lock, flags);
 	head = &__stp_tf_vma_map[__stp_tf_vma_map_hash(tsk)];
         stap_hlist_for_each_entry_safe(entry, node, n, head, hlist) {
             if (tsk->pid == entry->pid) {
@@ -371,7 +372,7 @@ stap_drop_vma_maps(struct task_struct *tsk)
 		    __stp_tf_vma_release_entry(entry);
             }
         }
-	write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
+	stp_write_unlock_irqrestore(&__stp_tf_vma_lock, flags);
 	return 0;
 }
 
-- 
1.8.3.1

[SYSTEMTAP/PATCH v2 1/6] stp: rt: locking helper api

[Santosh Shukla]

Locking helper api used for replacing -rt and non-rt specific locks with common
helper lock starts with prefix stp_.  -rt version of locking api compatible to
kernel version 3.0 and greater. Older -rt version can be supported by adding
more kernel version checks, ifdefs in header file.. I haven't tested on
those(older) kernel version so not including code for now. Tested for
3.10.40-rt38 and 3.14.12-rt9.

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
 runtime/stp_helper_lock.h | 67 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
 create mode 100644 runtime/stp_helper_lock.h

diff --git a/runtime/stp_helper_lock.h b/runtime/stp_helper_lock.h
new file mode 100644
index 0000000..d1a69b4
--- /dev/null
+++ b/runtime/stp_helper_lock.h
@@ -0,0 +1,67 @@
+/* -*- linux-c -*- 
+ * Locking helper function api to support preempt-rt variant raw locks
+ * and keep legacy locking compatibility intact.
+ *
+ * Author: Santosh Shukla <sshukla@mvista.com>
+ *
+ * Copyright (C) 2014 Red Hat Inc.
+ * 
+ * This file is part of systemtap, and is free software.  You can
+ * redistribute it and/or modify it under the terms of the GNU General
+ * Public License (GPL); either version 2, or (at your option) any
+ * later version.
+ * */
+
+#ifndef _STP_HELPER_LOCK_H_
+#define _STP_HELPER_LOCK_H_
+
+#include <linux/spinlock.h>
+
+#ifdef CONFIG_PREEMPT_RT_FULL
+
+#define STP_DEFINE_SPINLOCK(lock)	DEFINE_RAW_SPINLOCK(lock)
+
+static inline void stp_spin_lock(raw_spinlock_t *lock)		{ raw_spin_lock(lock); }
+static inline void stp_spin_unlock(raw_spinlock_t *lock)	{ raw_spin_unlock(lock); }
+
+#define stp_spin_lock_irqsave(lock, flags)		raw_spin_lock_irqsave(lock, flags)
+#define stp_spin_unlock_irqrestore(lock, flags)		raw_spin_unlock_irqrestore(lock, flags)
+
+
+#define STP_DEFINE_RWLOCK(lock)		DEFINE_RAW_SPINLOCK(lock)
+
+static inline void stp_read_lock(raw_spinlock_t *lock)		{ raw_spin_lock(lock); }
+static inline void stp_read_unlock(raw_spinlock_t *lock)	{ raw_spin_unlock(lock); }
+static inline void stp_write_lock(raw_spinlock_t *lock)		{ raw_spin_lock(lock); }
+static inline void stp_write_unlock(raw_spinlock_t *lock) 	{ raw_spin_unlock(lock); }
+
+#define stp_read_lock_irqsave(lock, flags)		raw_spin_lock_irqsave(lock, flags)
+#define stp_read_unlock_irqrestore(lock, flags)		raw_spin_unlock_irqrestore(lock, flags)
+#define stp_write_lock_irqsave(lock, flags)		raw_spin_lock_irqsave(lock, flags)
+#define stp_write_unlock_irqrestore(lock, flags) 	raw_spin_unlock_irqrestore(lock, flags)
+  
+#else
+#define STP_DEFINE_SPINLOCK(lock)	DEFINE_SPINLOCK(lock)
+
+static inline void stp_spin_lock(spinlock_t *lock)		{ spin_lock(lock); }
+static inline void stp_spin_unlock(spinlock_t *lock)		{ spin_unlock(lock); }
+
+#define stp_spin_lock_irqsave(lock, flags)		spin_lock_irqsave(lock, flags)
+#define stp_spin_unlock_irqrestore(lock, flags)		spin_unlock_irqrestore(lock, flags)
+
+#define STP_DEFINE_RWLOCK(lock)				DEFINE_RWLOCK(lock)
+
+static inline void stp_read_lock(rwlock_t *lock)	{ read_lock(lock); }
+static inline void stp_read_unlock(rwlock_t *lock)	{ read_unlock(lock); }
+static inline void stp_write_lock(rwlock_t *lock)	{ write_lock(lock); }
+static inline void stp_write_unlock(rwlock_t *lock)	{ write_unlock(lock); }
+
+#define stp_read_lock_irqsave(lock, flags)		read_lock_irqsave(lock, flags)
+#define stp_read_unlock_irqrestore(lock, flags)		read_unlock_irqrestore(lock, flags)
+#define stp_write_lock_irqsave(lock, flags)		write_lock_irqsave(lock, flags)
+#define stp_write_unlock_irqrestore(lock, flags) 	write_unlock_irqrestore(lock, flags)
+
+#endif
+
+#endif /* _STP_HELPER_LOCK_H_ */
+
-- 
1.8.3.1

[SYSTEMTAP/PATCH v2 2/6] stp: rt: replace __stp_tf_map_lock rd/wr lock with stp style raw lock

[Santosh Shukla]

noticed that in rt mode - read and write_lock lead to several bug_on :

[ 1033.544821]  ffff880035463d38 ffffffff815fcaed ffff880035463d90 ffffffff816057f7
[ 1033.544826]  ffff880035463fd8 0000000000017e80 0000000000017e80 ffffffff810be5fb
[ 1033.544826] Call Trace:
[ 1033.544830]  [<ffffffff81602329>] dump_stack+0x4e/0x7a
[ 1033.544834]  [<ffffffff815fcaed>] __schedule_bug+0x9f/0xad
[ 1033.544838]  [<ffffffff816057f7>] __schedule+0x627/0x6a0
[ 1033.544842]  [<ffffffff810be5fb>] ? task_blocks_on_rt_mutex+0x19b/0x220
[ 1033.544846]  [<ffffffff816058a0>] schedule+0x30/0xa0
[ 1033.544850]  [<ffffffff8160727d>] rt_spin_lock_slowlock+0xbd/0x1f0
[ 1033.544856]  [<ffffffff81607df5>] __rt_spin_lock+0x25/0x30
[ 1033.544858]  [<ffffffff816080b0>] rt_read_lock+0x30/0x40
[ 1033.544861]  [<ffffffff816080ce>] rt_read_lock_irqsave+0xe/0x20
[ 1033.544867]  [<ffffffffa08a9b89>] __stp_tf_get_map_entry+0x19/0xc0 [stap_e40dcb2c46d7c0a2fb8d70ba343e393a_15235]
[ 1033.544873]  [<ffffffffa08afedd>] __stp_utrace_task_finder_target_syscall_exit+0x5d/0x350 [stap_e40dcb2c46d7c0a2fb8d70ba343e393a_15235]
[ 1033.544889]  [<ffffffffa08a91c5>] utrace_report_syscall_exit+0xc5/0x110 [stap_e40dcb2c46d7c0a2fb8d70ba343e393a_15235]
[ 1033.544893]  [<ffffffff81023ce0>] syscall_trace_leave+0x100/0x130
[ 1033.544896]  [<ffffffff8161090b>] int_check_syscall_exit_work+0x34/0x3d

Replacing rd/wr lock with raw lock supress these bug_on. However more suitable
fix is to replace read/write lock in general with RCU style lock, followed by
use of rcu variant hlist api but current module desing need to go through some
real desing chnages in order of adding and freeing entry to/from free list,
which i choose work upon as separate patch[todo.]

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
 runtime/linux/task_finder_map.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/runtime/linux/task_finder_map.c b/runtime/linux/task_finder_map.c
index d515e9f..010460c 100644
--- a/runtime/linux/task_finder_map.c
+++ b/runtime/linux/task_finder_map.c
@@ -1,6 +1,7 @@
 #include <linux/list.h>
 #include <linux/jhash.h>
-#include <linux/spinlock.h>
+
+#include "stp_helper_lock.h"
 
 // When handling mmap()/munmap()/mprotect() syscall tracing to notice
 // memory map changes, we need to cache syscall entry parameter values
@@ -13,7 +14,7 @@
 // contents in interrupt context (which should only ever call 
 // stap_find_map_map_info for getting stored info). So we might
 // want to look into that if this seems a bottleneck.
-static DEFINE_RWLOCK(__stp_tf_map_lock);
+static STP_DEFINE_RWLOCK(__stp_tf_map_lock);
 
 #define __STP_TF_HASH_BITS 4
 #define __STP_TF_TABLE_SIZE (1 << __STP_TF_HASH_BITS)
@@ -51,11 +52,11 @@ __stp_tf_map_initialize(void)
 	struct hlist_head *head = &__stp_tf_map_free_list[0];
 
 	unsigned long flags;
-	write_lock_irqsave(&__stp_tf_map_lock, flags);
+	stp_write_lock_irqsave(&__stp_tf_map_lock, flags);
 	for (i = 0; i < TASK_FINDER_MAP_ENTRY_ITEMS; i++) {
 		hlist_add_head(&__stp_tf_map_free_list_items[i].hlist, head);
 	}
-	write_unlock_irqrestore(&__stp_tf_map_lock, flags);
+	stp_write_unlock_irqrestore(&__stp_tf_map_lock, flags);
 }
 
 
@@ -109,15 +110,15 @@ __stp_tf_get_map_entry(struct task_struct *tsk)
 	struct __stp_tf_map_entry *entry;
 
 	unsigned long flags;
-	read_lock_irqsave(&__stp_tf_map_lock, flags);
+	stp_read_lock_irqsave(&__stp_tf_map_lock, flags);
 	head = &__stp_tf_map_table[__stp_tf_map_hash(tsk)];
 	stap_hlist_for_each_entry(entry, node, head, hlist) {
 		if (tsk->pid == entry->pid) {
-			read_unlock_irqrestore(&__stp_tf_map_lock, flags);
+			stp_read_unlock_irqrestore(&__stp_tf_map_lock, flags);
 			return entry;
 		}
 	}
-	read_unlock_irqrestore(&__stp_tf_map_lock, flags);
+	stp_read_unlock_irqrestore(&__stp_tf_map_lock, flags);
 	return NULL;
 }
 
@@ -133,14 +134,14 @@ __stp_tf_add_map(struct task_struct *tsk, long syscall_no, unsigned long arg0,
 	struct __stp_tf_map_entry *entry;
 	unsigned long flags;
 
-	write_lock_irqsave(&__stp_tf_map_lock, flags);
+	stp_write_lock_irqsave(&__stp_tf_map_lock, flags);
 	head = &__stp_tf_map_table[__stp_tf_map_hash(tsk)];
 	stap_hlist_for_each_entry(entry, node, head, hlist) {
 		// If we find an existing entry, just increment the
 		// usage count.
 		if (tsk->pid == entry->pid) {
 			entry->usage++;
-			write_unlock_irqrestore(&__stp_tf_map_lock, flags);
+			stp_write_unlock_irqrestore(&__stp_tf_map_lock, flags);
 			return 0;
 		}
 	}
@@ -148,7 +149,7 @@ __stp_tf_add_map(struct task_struct *tsk, long syscall_no, unsigned long arg0,
 	// Get an element from the free list.
 	entry = __stp_tf_map_get_free_entry();
 	if (!entry) {
-		write_unlock_irqrestore(&__stp_tf_map_lock, flags);
+		stp_write_unlock_irqrestore(&__stp_tf_map_lock, flags);
 		return -ENOMEM;
 	}
 	entry->usage = 1;
@@ -158,7 +159,7 @@ __stp_tf_add_map(struct task_struct *tsk, long syscall_no, unsigned long arg0,
 	entry->arg1 = arg1;
 	entry->arg2 = arg2;
 	hlist_add_head(&entry->hlist, head);
-	write_unlock_irqrestore(&__stp_tf_map_lock, flags);
+	stp_write_unlock_irqrestore(&__stp_tf_map_lock, flags);
 	return 0;
 }
 
@@ -174,7 +175,7 @@ __stp_tf_remove_map_entry(struct __stp_tf_map_entry *entry)
 
 	if (entry != NULL) {
 		unsigned long flags;
-		write_lock_irqsave(&__stp_tf_map_lock, flags);
+		stp_write_lock_irqsave(&__stp_tf_map_lock, flags);
 
 		// Decrement the usage count.
 		entry->usage--;
@@ -185,7 +186,7 @@ __stp_tf_remove_map_entry(struct __stp_tf_map_entry *entry)
 			hlist_del(&entry->hlist);
 			__stp_tf_map_put_free_entry(entry);
 		}
-		write_unlock_irqrestore(&__stp_tf_map_lock, flags);
+		stp_write_unlock_irqrestore(&__stp_tf_map_lock, flags);
 	}
 	return 0;
 }
-- 
1.8.3.1

[SYSTEMTAP/PATCH v2 5/6] stp: rt: replace utrace_struct lock to stp style raw lock

[Santosh Shukla]

Patch fixes below bug_on for -rt mode kernel:

[ 3375.430085]  [<ffffffff815de469>] __schedule+0x5a9/0x700
[ 3375.430090]  [<ffffffff815da3a4>] dump_stack+0x19/0x1b
[ 3375.430094]  [<ffffffff815de5ea>] schedule+0x2a/0x90
[ 3375.430098]  [<ffffffff815d49d7>] __schedule_bug+0xa0/0xae
[ 3375.430102]  [<ffffffff815df525>] rt_spin_lock_slowlock+0xe5/0x2e0
[ 3375.430107]  [<ffffffff815de469>] __schedule+0x5a9/0x700
[ 3375.430110]  [<ffffffff815df935>] rt_spin_lock+0x25/0x30
[ 3375.430116]  [<ffffffff815de5ea>] schedule+0x2a/0x90
[ 3375.430125]  [<ffffffffa2f5ed5e>] task_utrace_struct+0x1e/0x40 [stap_eb141ade124ccb17a233482e6996651f_15664]
[ 3375.430131]  [<ffffffff815df525>] rt_spin_lock_slowlock+0xe5/0x2e0
[ 3375.430138]  [<ffffffffa2f6204b>] utrace_report_syscall_exit+0x4b/0x110 [stap_eb141ade124ccb17a233482e6996651f_15664]
[ 3375.430143]  [<ffffffff815df935>] rt_spin_lock+0x25/0x30
[ 3375.430148]  [<ffffffff810ea646>] ? __audit_syscall_exit+0x1f6/0x2a0
[ 3375.430156]  [<ffffffffa2f5ed5e>] task_utrace_struct+0x1e/0x40 [stap_eb141ade124ccb17a233482e6996651f_15664]
[ 3375.430161]  [<ffffffff81021d56>] syscall_trace_leave+0xd6/0xf0
[ 3375.430168]  [<ffffffffa2f6204b>] utrace_report_syscall_exit+0x4b/0x110 [stap_eb141ade124ccb17a233482e6996651f_15664]
[ 3375.430173]  [<ffffffff815e7af0>] int_check_syscall_exit_work+0x34/0x3d
[ 3375.430178]  [<ffffffff810ea646>] ? __audit_syscall_exit+0x1f6/0x2a0
[ 3375.430184]  [<ffffffff81021d56>] syscall_trace_leave+0xd6/0xf0
[ 3375.430191]  [<ffffffff815e7af0>] int_check_syscall_exit_work+0x34/0x3d

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
 runtime/stp_utrace.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/runtime/stp_utrace.c b/runtime/stp_utrace.c
index e5d6d55..c69dec1 100644
--- a/runtime/stp_utrace.c
+++ b/runtime/stp_utrace.c
@@ -85,7 +85,7 @@ struct utrace {
 
 static struct hlist_head task_utrace_table[TASK_UTRACE_TABLE_SIZE];
 //DEFINE_MUTEX(task_utrace_mutex);      /* Protects task_utrace_table */
-static DEFINE_SPINLOCK(task_utrace_lock); /* Protects task_utrace_table */
+static STP_DEFINE_SPINLOCK(task_utrace_lock); /* Protects task_utrace_table */
 
 static struct kmem_cache *utrace_cachep;
 static struct kmem_cache *utrace_engine_cachep;
@@ -468,7 +468,7 @@ static void utrace_shutdown(void)
 #ifdef STP_TF_DEBUG
 	printk(KERN_ERR "%s:%d - freeing task-specific\n", __FUNCTION__, __LINE__);
 #endif
-	spin_lock(&task_utrace_lock);
+	stp_spin_lock(&task_utrace_lock);
 	for (i = 0; i < TASK_UTRACE_TABLE_SIZE; i++) {
 		head = &task_utrace_table[i];
 		stap_hlist_for_each_entry_safe(utrace, node, node2, head,
@@ -477,7 +477,7 @@ static void utrace_shutdown(void)
 			utrace_cleanup(utrace);
 		}
 	}
-	spin_unlock(&task_utrace_lock);
+	stp_spin_unlock(&task_utrace_lock);
 #ifdef STP_TF_DEBUG
 	printk(KERN_ERR "%s:%d - done\n", __FUNCTION__, __LINE__);
 #endif
@@ -524,7 +524,7 @@ static bool utrace_task_alloc(struct task_struct *task)
 	stp_init_task_work(&utrace->work, &utrace_resume);
 	stp_init_task_work(&utrace->report_work, &utrace_report_work);
 
-	spin_lock(&task_utrace_lock);
+	stp_spin_lock(&task_utrace_lock);
 	u = __task_utrace_struct(task);
 	if (u == NULL) {
 		hlist_add_head(&utrace->hlist,
@@ -533,7 +533,7 @@ static bool utrace_task_alloc(struct task_struct *task)
 	else {
 		kmem_cache_free(utrace_cachep, utrace);
 	}
-	spin_unlock(&task_utrace_lock);
+	stp_spin_unlock(&task_utrace_lock);
 
 	return true;
 }
@@ -552,9 +552,9 @@ static void utrace_free(struct utrace *utrace)
 
 	/* Remove this utrace from the mapping list of tasks to
 	 * struct utrace. */
-	spin_lock(&task_utrace_lock);
+	stp_spin_lock(&task_utrace_lock);
 	hlist_del(&utrace->hlist);
-	spin_unlock(&task_utrace_lock);
+	stp_spin_unlock(&task_utrace_lock);
 
 	/* Free the utrace struct. */
 	stp_spin_lock(&utrace->lock);
@@ -595,9 +595,9 @@ static struct utrace *task_utrace_struct(struct task_struct *task)
 {
 	struct utrace *utrace;
 
-	spin_lock(&task_utrace_lock);
+	stp_spin_lock(&task_utrace_lock);
 	utrace = __task_utrace_struct(task);
-	spin_unlock(&task_utrace_lock);
+	stp_spin_unlock(&task_utrace_lock);
 	return utrace;
 }
 
-- 
1.8.3.1

[SYSTEMTAP/PATCH v2 4/6] stp: rt: replace utrace->lock with stp style raw lock

[Santosh Shukla]

In preempt-rt kernel, At the time of launching stap script noticed below
bug_on. Replacing spinlock with Raw fixes this problem.

[  159.433464] Preemption disabled at:[<ffffffff810b74d6>] remove_wait_queue+0x36/0x40

[  159.433466] CPU: 8 PID: 6723 Comm: bash Tainted: GF       W  O 3.14.12-rt-rt9+ #1
[  159.433467] Hardware name: Intel Corporation S2600CP/S2600CP, BIOS SE5C600.86B.02.01.0002.082220131453 08/22/2013
[  159.433471]  ffff88042d917d80 ffff88040f623d90 ffffffff81602b13 ffff8804121e3980
[  159.433474]  ffff88040f623da0 ffffffff815fd2fb ffff88040f623df8 ffffffff81606017
[  159.433478]  ffff88040f623fd8 0000000000017d80 0000000000017d80 ffffffff810bcbcb
[  159.433478] Call Trace:
[  159.433481]  [<ffffffff81602b13>] dump_stack+0x4e/0x7a
[  159.433484]  [<ffffffff815fd2fb>] __schedule_bug+0x9f/0xad
[  159.433486]  [<ffffffff81606017>] __schedule+0x627/0x6a0
[  159.433489]  [<ffffffff810bcbcb>] ? task_blocks_on_rt_mutex+0x19b/0x220
[  159.433492]  [<ffffffff816060c0>] schedule+0x30/0xa0
[  159.433495]  [<ffffffff81607a9d>] rt_spin_lock_slowlock+0xbd/0x1f0
[  159.433498]  [<ffffffff81608645>] rt_spin_lock+0x25/0x30
[  159.433503]  [<ffffffffa076fbf5>] start_report+0x45/0xb0 [stap_c108d00c22143294d42db713b804dbb9_10325]
[  159.433508]  [<ffffffffa0773e38>] utrace_report_syscall_exit+0x88/0x110 [stap_c108d00c22143294d42db713b804dbb9_10325]
[  159.433511]  [<ffffffff81023d30>] syscall_trace_leave+0x100/0x130
[  159.433514]  [<ffffffff8161114b>] int_check_syscall_exit_work+0x34/0x3d

Note : This module afaiu is prime candidate to benifit rcu locking primitives
and same some cycle, should improve overall performance, more scallable.
[This is general desing improvement so keeping those changes out from this
patch. .todo]

Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
 runtime/stp_helper_lock.h | 15 ++++++++-
 runtime/stp_utrace.c      | 85 ++++++++++++++++++++++++-----------------------
 2 files changed, 57 insertions(+), 43 deletions(-)

diff --git a/runtime/stp_helper_lock.h b/runtime/stp_helper_lock.h
index d1a69b4..f95d3c0 100644
--- a/runtime/stp_helper_lock.h
+++ b/runtime/stp_helper_lock.h
@@ -19,11 +19,17 @@
 
 #ifdef CONFIG_PREEMPT_RT_FULL
 
+#define stp_spinlock_t raw_spinlock_t
+
 #define STP_DEFINE_SPINLOCK(lock)	DEFINE_RAW_SPINLOCK(lock)
 
+static inline void stp_spin_lock_init(raw_spinlock_t *lock)	{ raw_spin_lock_init(lock); }
+
 static inline void stp_spin_lock(raw_spinlock_t *lock)		{ raw_spin_lock(lock); }
 static inline void stp_spin_unlock(raw_spinlock_t *lock)	{ raw_spin_unlock(lock); }
 
+static inline void stp_spin_unlock_wait(raw_spinlock_t *lock)	{ raw_spin_unlock_wait(lock); }
+
 #define stp_spin_lock_irqsave(lock, flags)		raw_spin_lock_irqsave(lock, flags)
 #define stp_spin_unlock_irqrestore(lock, flags)		raw_spin_unlock_irqrestore(lock, flags)
 
@@ -33,7 +39,7 @@ static inline void stp_spin_unlock(raw_spinlock_t *lock)	{ raw_spin_unlock(lock)
 static inline void stp_read_lock(raw_spinlock_t *lock)		{ raw_spin_lock(lock); }
 static inline void stp_read_unlock(raw_spinlock_t *lock)	{ raw_spin_unlock(lock); }
 static inline void stp_write_lock(raw_spinlock_t *lock)		{ raw_spin_lock(lock); }
-static inline void stp_write_unlock(raw_spinlock_t *lock) 	{ raw_spin_unlock(lock); }
+static inline void stp_write_unlock(raw_spinlock_t *lock)	{ raw_spin_unlock(lock); }
 
 #define stp_read_lock_irqsave(lock, flags)		raw_spin_lock_irqsave(lock, flags)
 #define stp_read_unlock_irqrestore(lock, flags)		raw_spin_unlock_irqrestore(lock, flags)
@@ -41,11 +47,18 @@ static inline void stp_write_unlock(raw_spinlock_t *lock) 	{ raw_spin_unlock(loc
 #define stp_write_unlock_irqrestore(lock, flags) 	raw_spin_unlock_irqrestore(lock, flags)
   
 #else
+
+#define stp_spinlock_t spinlock_t
+
 #define STP_DEFINE_SPINLOCK(lock)	DEFINE_SPINLOCK(lock)
 
+static inline void stp_spin_lock_init(spinlock_t *lock)		{ spin_lock_init(lock); }
+
 static inline void stp_spin_lock(spinlock_t *lock)		{ spin_lock(lock); }
 static inline void stp_spin_unlock(spinlock_t *lock)		{ spin_unlock(lock); }
 
+static inline void stp_spin_unlock_wait(spinlock_t *lock)	{ spin_unlock_wait(lock); }
+
 #define stp_spin_lock_irqsave(lock, flags)		spin_lock_irqsave(lock, flags)
 #define stp_spin_unlock_irqrestore(lock, flags)		spin_unlock_irqrestore(lock, flags)
 
diff --git a/runtime/stp_utrace.c b/runtime/stp_utrace.c
index acbe936..e5d6d55 100644
--- a/runtime/stp_utrace.c
+++ b/runtime/stp_utrace.c
@@ -22,12 +22,13 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/slab.h>
-#include <linux/spinlock.h>
 #include <trace/events/sched.h>
 #include <trace/events/syscalls.h>
 #include "stp_task_work.c"
 #include "linux/stp_tracepoint.h"
 
+#include "stp_helper_lock.h"
+
 /*
  * Per-thread structure private to utrace implementation.
  * If task_struct.utrace_flags is nonzero, task_struct.utrace
@@ -56,7 +57,7 @@
  * in time to have their callbacks seen.
  */
 struct utrace {
-	spinlock_t lock;
+	stp_spinlock_t lock;
 	struct list_head attached, attaching;
 
 	struct utrace_engine *reporting;
@@ -379,7 +380,7 @@ static void utrace_cleanup(struct utrace *utrace)
 
 	/* Free engines associated with the struct utrace, starting
 	 * with the 'attached' list then doing the 'attaching' list. */
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 	list_for_each_entry_safe(engine, next, &utrace->attached, entry) {
 #ifdef STP_TF_DEBUG
 	    printk(KERN_ERR "%s:%d - removing engine\n",
@@ -420,7 +421,7 @@ static void utrace_cleanup(struct utrace *utrace)
 #endif
 		utrace->report_work_added = 0;
 	}
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	/* Free the struct utrace itself. */
 	kmem_cache_free(utrace_cachep, utrace);
@@ -515,7 +516,7 @@ static bool utrace_task_alloc(struct task_struct *task)
 
 	if (unlikely(!utrace))
 		return false;
-	spin_lock_init(&utrace->lock);
+	stp_spin_lock_init(&utrace->lock);
 	INIT_LIST_HEAD(&utrace->attached);
 	INIT_LIST_HEAD(&utrace->attaching);
 	utrace->resume = UTRACE_RESUME;
@@ -556,7 +557,7 @@ static void utrace_free(struct utrace *utrace)
 	spin_unlock(&task_utrace_lock);
 
 	/* Free the utrace struct. */
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 #ifdef STP_TF_DEBUG
 	if (unlikely(utrace->reporting)
 	    || unlikely(!list_empty(&utrace->attached))
@@ -585,7 +586,7 @@ static void utrace_free(struct utrace *utrace)
 				: "UNKNOWN"));
 		utrace->report_work_added = 0;
 	}
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	kmem_cache_free(utrace_cachep, utrace);
 }
@@ -661,7 +662,7 @@ static int utrace_add_engine(struct task_struct *target,
 {
 	int ret;
 
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 
 	ret = -EEXIST;
 	if ((flags & UTRACE_ATTACH_EXCLUSIVE) &&
@@ -709,7 +710,7 @@ static int utrace_add_engine(struct task_struct *target,
 	utrace_engine_get(engine);
 	ret = 0;
 unlock:
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	return ret;
 }
@@ -758,11 +759,11 @@ static struct utrace_engine *utrace_attach_task(
 	if (!(flags & UTRACE_ATTACH_CREATE)) {
 		if (unlikely(!utrace))
 			return ERR_PTR(-ENOENT);
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		engine = find_matching_engine(utrace, flags, ops, data);
 		if (engine)
 			utrace_engine_get(engine);
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 		return engine ?: ERR_PTR(-ENOENT);
 	}
 
@@ -878,14 +879,14 @@ static struct utrace *get_utrace_lock(struct task_struct *target,
 	}
 
 	utrace = task_utrace_struct(target);
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 	if (unlikely(utrace->reap) || unlikely(!engine->ops) ||
 	    unlikely(engine->ops == &utrace_detached_ops)) {
 		/*
 		 * By the time we got the utrace lock,
 		 * it had been reaped or detached already.
 		 */
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 		utrace = ERR_PTR(-ESRCH);
 		if (!attached && engine->ops == &utrace_detached_ops)
 			utrace = ERR_PTR(-ERESTARTSYS);
@@ -1051,7 +1052,7 @@ static int utrace_set_events(struct task_struct *target,
 			ret = -EINPROGRESS;
 	}
 unlock:
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	return ret;
 }
@@ -1181,7 +1182,7 @@ static bool utrace_reset(struct task_struct *task, struct utrace *utrace)
 	 */
 	rcu_read_lock();
 	utrace->utrace_flags = flags;
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 	rcu_read_unlock();
 
 	put_detached_list(&detached);
@@ -1197,7 +1198,7 @@ static void utrace_finish_stop(void)
 	 */
 	if (unlikely(__fatal_signal_pending(current))) {
 		struct utrace *utrace = task_utrace_struct(current);
-		spin_unlock_wait(&utrace->lock);
+		stp_spin_unlock_wait(&utrace->lock);
 	}
 }
 
@@ -1211,7 +1212,7 @@ static void utrace_stop(struct task_struct *task, struct utrace *utrace,
 			enum utrace_resume_action action)
 {
 relock:
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 
 	if (action < utrace->resume) {
 		/*
@@ -1247,7 +1248,7 @@ relock:
 
 	if (unlikely(__fatal_signal_pending(task))) {
 		spin_unlock_irq(&task->sighand->siglock);
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 		return;
 	}
 
@@ -1262,7 +1263,7 @@ relock:
 		task->signal->flags = SIGNAL_STOP_STOPPED;
 
 	spin_unlock_irq(&task->sighand->siglock);
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	schedule();
 
@@ -1298,7 +1299,7 @@ static void utrace_maybe_reap(struct task_struct *target, struct utrace *utrace,
 	struct utrace_engine *engine, *next;
 	struct list_head attached;
 
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 
 	if (reap) {
 		/*
@@ -1312,7 +1313,7 @@ static void utrace_maybe_reap(struct task_struct *target, struct utrace *utrace,
 		utrace->reap = 1;
 
 		if (utrace->utrace_flags & _UTRACE_DEATH_EVENTS) {
-			spin_unlock(&utrace->lock);
+			stp_spin_unlock(&utrace->lock);
 			return;
 		}
 	} else {
@@ -1350,7 +1351,7 @@ static void utrace_maybe_reap(struct task_struct *target, struct utrace *utrace,
 	list_replace_init(&utrace->attached, &attached);
 	list_splice_tail_init(&utrace->attaching, &attached);
 
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	list_for_each_entry_safe(engine, next, &attached, entry) {
 		if (engine->flags & UTRACE_EVENT(REAP))
@@ -1525,7 +1526,7 @@ static int utrace_control(struct task_struct *target,
 	if (unlikely(target->exit_state)) {
 		ret = utrace_control_dead(target, utrace, action);
 		if (ret) {
-			spin_unlock(&utrace->lock);
+			stp_spin_unlock(&utrace->lock);
 			return ret;
 		}
 		reset = true;
@@ -1623,7 +1624,7 @@ static int utrace_control(struct task_struct *target,
 	if (reset)
 		utrace_reset(target, utrace);
 	else
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 
 	return ret;
 }
@@ -1682,7 +1683,7 @@ static int utrace_barrier(struct task_struct *target,
 			 */
 			if (utrace->reporting != engine)
 				ret = 0;
-			spin_unlock(&utrace->lock);
+			stp_spin_unlock(&utrace->lock);
 			if (!ret)
 				break;
 		}
@@ -1720,12 +1721,12 @@ static enum utrace_resume_action start_report(struct utrace *utrace)
 	enum utrace_resume_action resume = utrace->resume;
 	if (utrace->pending_attach ||
 	    (resume > UTRACE_STOP && resume < UTRACE_RESUME)) {
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		splice_attaching(utrace);
 		resume = utrace->resume;
 		if (resume > UTRACE_STOP)
 			utrace->resume = UTRACE_RESUME;
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 	}
 	return resume;
 }
@@ -1735,7 +1736,7 @@ static inline void finish_report_reset(struct task_struct *task,
 				       struct utrace_report *report)
 {
 	if (unlikely(report->spurious || report->detaches)) {
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		if (utrace_reset(task, utrace))
 			report->action = UTRACE_RESUME;
 	}
@@ -1758,12 +1759,12 @@ static void finish_report(struct task_struct *task, struct utrace *utrace,
 		resume = will_not_stop ? UTRACE_REPORT : UTRACE_RESUME;
 
 	if (resume < utrace->resume) {
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		utrace->resume = resume;
 		stp_task_notify_resume(task, utrace);
 		if (resume == UTRACE_INTERRUPT)
 			set_tsk_thread_flag(task, TIF_SIGPENDING);
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 	}
 
 	finish_report_reset(task, utrace, report);
@@ -1784,9 +1785,9 @@ static void finish_callback_report(struct task_struct *task,
 		 * This way, a 0 return is an unambiguous indicator that any
 		 * callback returning UTRACE_DETACH has indeed caused detach.
 		 */
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		engine->ops = &utrace_detached_ops;
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 	}
 
 	/*
@@ -1805,16 +1806,16 @@ static void finish_callback_report(struct task_struct *task,
 			report->resume_action = action;
 
 		if (engine_wants_stop(engine)) {
-			spin_lock(&utrace->lock);
+			stp_spin_lock(&utrace->lock);
 			clear_engine_wants_stop(engine);
-			spin_unlock(&utrace->lock);
+			stp_spin_unlock(&utrace->lock);
 		}
 
 		return;
 	}
 
 	if (!engine_wants_stop(engine)) {
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		/*
 		 * If utrace_control() came in and detached us
 		 * before we got the lock, we must not stop now.
@@ -1823,7 +1824,7 @@ static void finish_callback_report(struct task_struct *task,
 			report->detaches = true;
 		else
 			mark_engine_wants_stop(utrace, engine);
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 	}
 }
 
@@ -2201,9 +2202,9 @@ static void utrace_finish_vfork(struct task_struct *task)
 	struct utrace *utrace = task_utrace_struct(task);
 
 	if (utrace->vfork_stop) {
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		utrace->vfork_stop = 0;
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 		utrace_stop(task, utrace, UTRACE_RESUME); /* XXX */
 	}
 }
@@ -2278,12 +2279,12 @@ static void utrace_report_death(void *cb_data __attribute__ ((unused)),
 		}
 	}
 	else {
-		spin_lock(&utrace->lock);
+		stp_spin_lock(&utrace->lock);
 		BUG_ON(utrace->death);
 		utrace->death = 1;
 		utrace->resume = UTRACE_RESUME;
 		splice_attaching(utrace);
-		spin_unlock(&utrace->lock);
+		stp_spin_unlock(&utrace->lock);
 
 		REPORT_CALLBACKS(, task, utrace, &report, UTRACE_EVENT(DEATH),
 				 report_death, engine, -1/*group_dead*/,
@@ -2436,12 +2437,12 @@ static void utrace_report_work(struct task_work *work)
 	might_sleep();
 	utrace->report_work_added = 0;
 
-	spin_lock(&utrace->lock);
+	stp_spin_lock(&utrace->lock);
 	BUG_ON(utrace->death);
 	utrace->death = 1;
 	utrace->resume = UTRACE_RESUME;
 	splice_attaching(utrace);
-	spin_unlock(&utrace->lock);
+	stp_spin_unlock(&utrace->lock);
 
 	REPORT_CALLBACKS(, task, utrace, &report, UTRACE_EVENT(DEATH),
 			 report_death, engine, -1/*group_dead*/,
-- 
1.8.3.1

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Frank Ch. Eigler]

Santosh Shukla <sshukla@mvista.com> writes:

> [...]  This is a v2 version of -rt aware systemtap patchset, tested
> for 3.14.12-rt9 and for 3.10.40-rt38 kernel version. [...]

Thanks!

> [...]
> Test script used for testing :
> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/netdev.stp
> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/tcpdumplike.stp
> [...]
> Like know feedback, comment on patch set. also Does it make sense to maintain
> systemtap -rt version in upstream.. do we care?

Thanks to Josh for looking into this in more detail.  I only have the
general observation that this patchset would be desirable in mainline
stap, but only if it is tested by at least the whole testsuite (rather
than just those two samples) on your -rt kernel, *and* if the patched
code also runs the whole testsuite on a non-rt kernel.  (We don't
expect 100% PASS, but also no smoke attributable to the patches.)

- FChE

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

On Tue, Sep 16, 2014 at 8:10 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Santosh Shukla <sshukla@mvista.com> writes:
>
>> [...]  This is a v2 version of -rt aware systemtap patchset, tested
>> for 3.14.12-rt9 and for 3.10.40-rt38 kernel version. [...]
>
> Thanks!
>
>> [...]
>> Test script used for testing :
>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/netdev.stp
>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/tcpdumplike.stp
>> [...]
>> Like know feedback, comment on patch set. also Does it make sense to maintain
>> systemtap -rt version in upstream.. do we care?
>
> Thanks to Josh for looking into this in more detail.  I only have the
> general observation that this patchset would be desirable in mainline
> stap, but only if it is tested by at least the whole testsuite (rather
> than just those two samples) on your -rt kernel, *and* if the patched
> code also runs the whole testsuite on a non-rt kernel.  (We don't
> expect 100% PASS, but also no smoke attributable to the patches.)
>

Agree, Could you point me testsuite to trigger and test.. I looked
perhaps not so much keenly.. so it would be good to know right
pointers about testing details .. I can run test and present the
report. let me know. Also Any performance specific testsuite too? I
did asked for in v1 patchset.
Thanks.
> - FChE

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

Frank suggested me to run "make installcheck" on -rt and not-rt mode
which I tried and got build failure saying "./execrc: line 4: runtest:
command not found". Anyone knows how to fix this.

I am building/testing vanilla kernel version 3.14.12-rt9 on x86
Ivybridge box [12 cores HT].


On Tue, Sep 16, 2014 at 11:18 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> On Tue, Sep 16, 2014 at 8:10 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
>> Santosh Shukla <sshukla@mvista.com> writes:
>>
>>> [...]  This is a v2 version of -rt aware systemtap patchset, tested
>>> for 3.14.12-rt9 and for 3.10.40-rt38 kernel version. [...]
>>
>> Thanks!
>>
>>> [...]
>>> Test script used for testing :
>>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/netdev.stp
>>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/tcpdumplike.stp
>>> [...]
>>> Like know feedback, comment on patch set. also Does it make sense to maintain
>>> systemtap -rt version in upstream.. do we care?
>>
>> Thanks to Josh for looking into this in more detail.  I only have the
>> general observation that this patchset would be desirable in mainline
>> stap, but only if it is tested by at least the whole testsuite (rather
>> than just those two samples) on your -rt kernel, *and* if the patched
>> code also runs the whole testsuite on a non-rt kernel.  (We don't
>> expect 100% PASS, but also no smoke attributable to the patches.)
>>
>
> Agree, Could you point me testsuite to trigger and test.. I looked
> perhaps not so much keenly.. so it would be good to know right
> pointers about testing details .. I can run test and present the
> report. let me know. Also Any performance specific testsuite too? I
> did asked for in v1 patchset.
> Thanks.
>> - FChE

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

Installing dejagnu solves the problem.. Pl. ignore. Sorry for noise. Thanks.

On Wed, Sep 17, 2014 at 2:04 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> Frank suggested me to run "make installcheck" on -rt and not-rt mode
> which I tried and got build failure saying "./execrc: line 4: runtest:
> command not found". Anyone knows how to fix this.
>
> I am building/testing vanilla kernel version 3.14.12-rt9 on x86
> Ivybridge box [12 cores HT].
>
>
> On Tue, Sep 16, 2014 at 11:18 PM, Santosh Shukla <sshukla@mvista.com> wrote:
>> On Tue, Sep 16, 2014 at 8:10 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
>>> Santosh Shukla <sshukla@mvista.com> writes:
>>>
>>>> [...]  This is a v2 version of -rt aware systemtap patchset, tested
>>>> for 3.14.12-rt9 and for 3.10.40-rt38 kernel version. [...]
>>>
>>> Thanks!
>>>
>>>> [...]
>>>> Test script used for testing :
>>>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/netdev.stp
>>>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/tcpdumplike.stp
>>>> [...]
>>>> Like know feedback, comment on patch set. also Does it make sense to maintain
>>>> systemtap -rt version in upstream.. do we care?
>>>
>>> Thanks to Josh for looking into this in more detail.  I only have the
>>> general observation that this patchset would be desirable in mainline
>>> stap, but only if it is tested by at least the whole testsuite (rather
>>> than just those two samples) on your -rt kernel, *and* if the patched
>>> code also runs the whole testsuite on a non-rt kernel.  (We don't
>>> expect 100% PASS, but also no smoke attributable to the patches.)
>>>
>>
>> Agree, Could you point me testsuite to trigger and test.. I looked
>> perhaps not so much keenly.. so it would be good to know right
>> pointers about testing details .. I can run test and present the
>> report. let me know. Also Any performance specific testsuite too? I
>> did asked for in v1 patchset.
>> Thanks.
>>> - FChE

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

Some update on regression -rt stap test that It looks like I fixed
most of deadlock specific bugs in stap specific -rt mode.. with
obvious addition of more -rt stap patch to show up in v3 version
soon.. Running one last time regression stuff with more load on 12
core Ivybridge machine before flashing the patches.

Thanks Frank for autotest pointers.

On Wed, Sep 17, 2014 at 2:04 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> Frank suggested me to run "make installcheck" on -rt and not-rt mode
> which I tried and got build failure saying "./execrc: line 4: runtest:
> command not found". Anyone knows how to fix this.
>
> I am building/testing vanilla kernel version 3.14.12-rt9 on x86
> Ivybridge box [12 cores HT].
>
>
> On Tue, Sep 16, 2014 at 11:18 PM, Santosh Shukla <sshukla@mvista.com> wrote:
>> On Tue, Sep 16, 2014 at 8:10 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
>>> Santosh Shukla <sshukla@mvista.com> writes:
>>>
>>>> [...]  This is a v2 version of -rt aware systemtap patchset, tested
>>>> for 3.14.12-rt9 and for 3.10.40-rt38 kernel version. [...]
>>>
>>> Thanks!
>>>
>>>> [...]
>>>> Test script used for testing :
>>>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/netdev.stp
>>>> /usr/local/stap/bin/stap -v testsuite/systemtap.examples/network/tcpdumplike.stp
>>>> [...]
>>>> Like know feedback, comment on patch set. also Does it make sense to maintain
>>>> systemtap -rt version in upstream.. do we care?
>>>
>>> Thanks to Josh for looking into this in more detail.  I only have the
>>> general observation that this patchset would be desirable in mainline
>>> stap, but only if it is tested by at least the whole testsuite (rather
>>> than just those two samples) on your -rt kernel, *and* if the patched
>>> code also runs the whole testsuite on a non-rt kernel.  (We don't
>>> expect 100% PASS, but also no smoke attributable to the patches.)
>>>
>>
>> Agree, Could you point me testsuite to trigger and test.. I looked
>> perhaps not so much keenly.. so it would be good to know right
>> pointers about testing details .. I can run test and present the
>> report. let me know. Also Any performance specific testsuite too? I
>> did asked for in v1 patchset.
>> Thanks.
>>> - FChE

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[David Smith]

On 09/16/2014 07:49 AM, Santosh Shukla wrote:
> Hi,
> 
> This is a v2 version of -rt aware systemtap patchset, tested for 3.14.12-rt9 and for 3.10.40-rt38 kernel version.
> For v1->v2 related details refer [1]. Patchset based on stap upstream link [2] master branch.

I did a bit of testing on these patches.

First I tested plain systemtap code on 3.10.0-153.el7.x86_64 to get a
baseline. I then applied the 6 patches, and re-ran the entire testsuite
on that kernel. No changes. Great.

I then put 3.10.0-155.rt56.79.el7rt.x86_64 on that same system and ran
the testsuite. Unfortunately the system crashes somewhere in check.exp.
So, we've still got a ways to go.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

On Fri, Sep 19, 2014 at 8:16 PM, David Smith <dsmith@redhat.com> wrote:
> On 09/16/2014 07:49 AM, Santosh Shukla wrote:
>> Hi,
>>
>> This is a v2 version of -rt aware systemtap patchset, tested for 3.14.12-rt9 and for 3.10.40-rt38 kernel version.
>> For v1->v2 related details refer [1]. Patchset based on stap upstream link [2] master branch.
>
> I did a bit of testing on these patches.
>
> First I tested plain systemtap code on 3.10.0-153.el7.x86_64 to get a
> baseline. I then applied the 6 patches, and re-ran the entire testsuite
> on that kernel. No changes. Great.
>
> I then put 3.10.0-155.rt56.79.el7rt.x86_64 on that same system and ran
> the testsuite. Unfortunately the system crashes somewhere in check.exp.
> So, we've still got a ways to go.
>

Right, I fixed those.. still seeing few deadlock. I'll update v3 patch
set passing installcheck testsuite soon. Thanks.

> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

Hi,

On Fri, Sep 19, 2014 at 11:12 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> On Fri, Sep 19, 2014 at 8:16 PM, David Smith <dsmith@redhat.com> wrote:
>> On 09/16/2014 07:49 AM, Santosh Shukla wrote:
>>> Hi,
>>>
>>> This is a v2 version of -rt aware systemtap patchset, tested for 3.14.12-rt9 and for 3.10.40-rt38 kernel version.
>>> For v1->v2 related details refer [1]. Patchset based on stap upstream link [2] master branch.
>>
>> I did a bit of testing on these patches.
>>
>> First I tested plain systemtap code on 3.10.0-153.el7.x86_64 to get a
>> baseline. I then applied the 6 patches, and re-ran the entire testsuite
>> on that kernel. No changes. Great.
>>
>> I then put 3.10.0-155.rt56.79.el7rt.x86_64 on that same system and ran
>> the testsuite. Unfortunately the system crashes somewhere in check.exp.
>> So, we've still got a ways to go.
>>
>
> Right, I fixed those.. still seeing few deadlock. I'll update v3 patch
> set passing installcheck testsuite soon. Thanks.

Test result summary for make installcheck using el7 config on -rt and
non-rt mode tested on vanilla 3.14.12 (non-rt) and on 3.14.12-rt9
kernel below :

1) -rt mode:


=== systemtap Summary ===

# of expected passes 5060
# of unexpected failures 131
# of unexpected successes 1
# of expected failures 315
# of known failures 68
# of untested testcases 275
# of unsupported tests 4



2) non-rt mode:

                === systemtap Summary ===

# of expected passes            4988
# of unexpected failures        181
# of unexpected successes       1
# of expected failures          315
# of known failures             68
# of untested testcases         285
# of unsupported tests          4

Let me know if you like to know more details, I can upload full test
log on ant share-able site.

I will be posting v3 patches soon. Thanks.

>
>> --
>> David Smith
>> dsmith@redhat.com
>> Red Hat
>> http://www.redhat.com
>> 256.217.0141 (direct)
>> 256.837.0057 (fax)

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[David Smith]

On 09/20/2014 08:12 AM, Santosh Shukla wrote:
> Hi,
> 
> On Fri, Sep 19, 2014 at 11:12 PM, Santosh Shukla <sshukla@mvista.com> wrote:
>> On Fri, Sep 19, 2014 at 8:16 PM, David Smith <dsmith@redhat.com> wrote:
>>> On 09/16/2014 07:49 AM, Santosh Shukla wrote:
>>>> Hi,
>>>>
>>>> This is a v2 version of -rt aware systemtap patchset, tested for 3.14.12-rt9 and for 3.10.40-rt38 kernel version.
>>>> For v1->v2 related details refer [1]. Patchset based on stap upstream link [2] master branch.
>>>
>>> I did a bit of testing on these patches.
>>>
>>> First I tested plain systemtap code on 3.10.0-153.el7.x86_64 to get a
>>> baseline. I then applied the 6 patches, and re-ran the entire testsuite
>>> on that kernel. No changes. Great.
>>>
>>> I then put 3.10.0-155.rt56.79.el7rt.x86_64 on that same system and ran
>>> the testsuite. Unfortunately the system crashes somewhere in check.exp.
>>> So, we've still got a ways to go.
>>>
>>
>> Right, I fixed those.. still seeing few deadlock. I'll update v3 patch
>> set passing installcheck testsuite soon. Thanks.
> 
> Test result summary for make installcheck using el7 config on -rt and
> non-rt mode tested on vanilla 3.14.12 (non-rt) and on 3.14.12-rt9
> kernel below :

Let me see if I understand this:

-rt mode: This is your patched systemtap running on a rt kernel

non-rt mode: This is your patches systemtap running on a regular,
non-rt, kernel.

Is the above correct?

> 1) -rt mode:
> 
> 
> === systemtap Summary ===
> 
> # of expected passes 5060
> # of unexpected failures 131
> # of unexpected successes 1
> # of expected failures 315
> # of known failures 68
> # of untested testcases 275
> # of unsupported tests 4
> 
> 
> 
> 2) non-rt mode:
> 
>                 === systemtap Summary ===
> 
> # of expected passes            4988
> # of unexpected failures        181
> # of unexpected successes       1
> # of expected failures          315
> # of known failures             68
> # of untested testcases         285
> # of unsupported tests          4

What's odd here is that the number of unexpected failures is higher on
the non-rt kernel. I would have expected the other way around.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: [SYSTEMTAP/PATCH v2 0/6] RT aware systemtap patch set

[Santosh Shukla]

On Mon, Sep 22, 2014 at 7:20 PM, David Smith <dsmith@redhat.com> wrote:
> On 09/20/2014 08:12 AM, Santosh Shukla wrote:
>> Hi,
>>
>> On Fri, Sep 19, 2014 at 11:12 PM, Santosh Shukla <sshukla@mvista.com> wrote:
>>> On Fri, Sep 19, 2014 at 8:16 PM, David Smith <dsmith@redhat.com> wrote:
>>>> On 09/16/2014 07:49 AM, Santosh Shukla wrote:
>>>>> Hi,
>>>>>
>>>>> This is a v2 version of -rt aware systemtap patchset, tested for 3.14.12-rt9 and for 3.10.40-rt38 kernel version.
>>>>> For v1->v2 related details refer [1]. Patchset based on stap upstream link [2] master branch.
>>>>
>>>> I did a bit of testing on these patches.
>>>>
>>>> First I tested plain systemtap code on 3.10.0-153.el7.x86_64 to get a
>>>> baseline. I then applied the 6 patches, and re-ran the entire testsuite
>>>> on that kernel. No changes. Great.
>>>>
>>>> I then put 3.10.0-155.rt56.79.el7rt.x86_64 on that same system and ran
>>>> the testsuite. Unfortunately the system crashes somewhere in check.exp.
>>>> So, we've still got a ways to go.
>>>>
>>>
>>> Right, I fixed those.. still seeing few deadlock. I'll update v3 patch
>>> set passing installcheck testsuite soon. Thanks.
>>
>> Test result summary for make installcheck using el7 config on -rt and
>> non-rt mode tested on vanilla 3.14.12 (non-rt) and on 3.14.12-rt9
>> kernel below :
>
> Let me see if I understand this:
>
> -rt mode: This is your patched systemtap running on a rt kernel
>
> non-rt mode: This is your patches systemtap running on a regular,
> non-rt, kernel.
>
> Is the above correct?
No. V3 patch set which I sent out today. It is tested for -rt and
non-rt mode. I could send you the systemtap.sum for both the cases if
you suspect on test results, Let me know, thanks.

>
>> 1) -rt mode:
>>
>>
>> === systemtap Summary ===
>>
>> # of expected passes 5060
>> # of unexpected failures 131
>> # of unexpected successes 1
>> # of expected failures 315
>> # of known failures 68
>> # of untested testcases 275
>> # of unsupported tests 4
>>
>>
>>
>> 2) non-rt mode:
>>
>>                 === systemtap Summary ===
>>
>> # of expected passes            4988
>> # of unexpected failures        181
>> # of unexpected successes       1
>> # of expected failures          315
>> # of known failures             68
>> # of untested testcases         285
>> # of unsupported tests          4
>
> What's odd here is that the number of unexpected failures is higher on
> the non-rt kernel. I would have expected the other way around.

I am really not sure :) You are asking me to fish out and answer out
of 4000+ test, which one failed on -rt and on non-rt. You may try
running v3 patchset for -rt and non-rt mode.

>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)