Improving SystemTap ARM Architecture Support

Improving SystemTap ARM Architecture Support

[William Cohen]

With the ARM architecture becoming a more common architecture for
Linux SystemTap should be improve to better support the ARM
architecture.  There are currently Ubuntu and Fedora images available
for the ARM architecture.

For this work has been concentrating on getting things working on the
Fedora 13 release candidate with SystemTap running in an a self hosted
manner.  The variations in the ARM platforms make it difficult to have
a single binary kernel RPM for all the arm platforms (kernel-debuginfo
and kernel-devel rpms are rare).  This work has been using a locally
built linux kernel to make all the kernel debuginfo and devel
information available.

The goal is to be able to run the systemtap testsuite to find to what
tests work and fail on the ARM architecture. In the past people have
just cross compiled an individual script on one machine and transfered
them over to the target ARM machine. The testsuite is not designed to
run in cross-compiled manner.

Reviewing early test results and the code the following is an
incomplete list of improvements:

-created aux_syscalls.stp (FIXED)
  "make installcheck" uses debuginfo smoke test
  needs _arch_ptrace_argstr(), aux_syscalls.stp
-map arm variants (armv7l and armv5tel) to arm (PR4186)
-provide tapset/arm/registers.stp
  -map register name to/from sruct
  -functions for nd probe argument access
-add any arm specific syscalls in tapset/arm/syscalls.stp
-provide high-resolution timer for gettimeofday_ns()
  -this may be complicated by the variety of arm high-res timer mechanisms

Right now the "make installcheck" tests are running. The ARM machine
is rather slow, but once the results are available will mail them to
dejazilla so have a reference point for the ARM architecture.

-Will

Re: Improving SystemTap ARM Architecture Support

[Mark Wielaard]

On Tue, 2011-07-19 at 17:48 -0400, William Cohen wrote:
> Reviewing early test results and the code the following is an
> incomplete list of improvements:
> 
> -created aux_syscalls.stp (FIXED)
>   "make installcheck" uses debuginfo smoke test
>   needs _arch_ptrace_argstr(), aux_syscalls.stp
> -map arm variants (armv7l and armv5tel) to arm (PR4186)
> -provide tapset/arm/registers.stp
>   -map register name to/from sruct
>   -functions for nd probe argument access
> -add any arm specific syscalls in tapset/arm/syscalls.stp
> -provide high-resolution timer for gettimeofday_ns()
>   -this may be complicated by the variety of arm high-res timer mechanisms

For sdt probes one also needs a fixed binutils with the following fix in
it: http://sourceware.org/ml/binutils/2011-07/msg00118.html
Fedora already has source packages, but I don't know if they have been
build for arm yet:
http://koji.fedoraproject.org/koji/buildinfo?buildID=253553

For user space probing one needs some general kernel cleanups, utrace
and uprobes fixes as described in this thread:
https://www.redhat.com/archives/utrace-devel/2011-April/msg00005.html

Cheers,

Mark

Re: Improving SystemTap ARM Architecture Support

[William Cohen]

On 07/20/2011 04:35 AM, Mark Wielaard wrote:
> On Tue, 2011-07-19 at 17:48 -0400, William Cohen wrote:
>> Reviewing early test results and the code the following is an
>> incomplete list of improvements:
>>
>> -created aux_syscalls.stp (FIXED)
>>   "make installcheck" uses debuginfo smoke test
>>   needs _arch_ptrace_argstr(), aux_syscalls.stp
>> -map arm variants (armv7l and armv5tel) to arm (PR4186)
>> -provide tapset/arm/registers.stp
>>   -map register name to/from sruct
>>   -functions for nd probe argument access
>> -add any arm specific syscalls in tapset/arm/syscalls.stp
>> -provide high-resolution timer for gettimeofday_ns()
>>   -this may be complicated by the variety of arm high-res timer mechanisms
> 
> For sdt probes one also needs a fixed binutils with the following fix in
> it: http://sourceware.org/ml/binutils/2011-07/msg00118.html
> Fedora already has source packages, but I don't know if they have been
> build for arm yet:
> http://koji.fedoraproject.org/koji/buildinfo?buildID=253553
> 
> For user space probing one needs some general kernel cleanups, utrace
> and uprobes fixes as described in this thread:
> https://www.redhat.com/archives/utrace-devel/2011-April/msg00005.html
> 
> Cheers,
> 
> Mark



The test results are posted on dejazilla:

http://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%3C4E26D449.7070005%40redhat.com%3E%27

		=== systemtap Summary ===

# of expected passes		1006
# of unexpected failures	190
# of unexpected successes	9
# of expected failures		235
# of known failures		2
# of untested testcases		189
# of unsupported tests		5


Looks like all the nd_syscall tests (37) fail because there is no code to get the arguments. systemtap.example/general/eventcount also fails because of missing int_arg().
There are a couple failures due to arch armv7l not being mapped to arm in systemtap.base/cmd_parse.exp and preprocessor.exp.
A number of the examples failed because of problems finding members fields. I think that this might be due to the native fc13 arm compiler, because I did not see this problem on the same example disktop.stp when cross compiling.

-Will

-Will

Re: Improving SystemTap ARM Architecture Support

[William Cohen]

On 07/20/2011 04:35 AM, Mark Wielaard wrote:
> On Tue, 2011-07-19 at 17:48 -0400, William Cohen wrote:
>> Reviewing early test results and the code the following is an
>> incomplete list of improvements:
>>
>> -created aux_syscalls.stp (FIXED)
>>   "make installcheck" uses debuginfo smoke test
>>   needs _arch_ptrace_argstr(), aux_syscalls.stp
>> -map arm variants (armv7l and armv5tel) to arm (PR4186)
>> -provide tapset/arm/registers.stp
>>   -map register name to/from sruct
>>   -functions for nd probe argument access
>> -add any arm specific syscalls in tapset/arm/syscalls.stp
>> -provide high-resolution timer for gettimeofday_ns()
>>   -this may be complicated by the variety of arm high-res timer mechanisms
> 
> For sdt probes one also needs a fixed binutils with the following fix in
> it: http://sourceware.org/ml/binutils/2011-07/msg00118.html
> Fedora already has source packages, but I don't know if they have been
> build for arm yet:
> http://koji.fedoraproject.org/koji/buildinfo?buildID=253553

There is a similar but separate koji system for arm:

http://arm.koji.fedoraproject.org/

Looks like the newest binutils rpms for arm, still don't have that:
.
http://arm.koji.fedoraproject.org/koji/packageinfo?packageID=702

> For user space probing one needs some general kernel cleanups, utrace
> and uprobes fixes as described in this thread:
> https://www.redhat.com/archives/utrace-devel/2011-April/msg00005.html

Right now the arm machine isn't running a fedora kernel. It is running a locally built kernel from sources at:

git://gitorious.org/efikamx/linux-kernel.git

I haven't moved over to the fedora kernels yet, because I am afraid that is going to take some time to find the appropriate settings for the config file (every ARM seems to be special) and there appears to be some patches in the current efika mx git tree that might be needed.

-Will

Re: Improving SystemTap ARM Architecture Support

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 2704 bytes --]

On 07/19/2011 05:48 PM, William Cohen wrote:
> With the ARM architecture becoming a more common architecture for
> Linux SystemTap should be improve to better support the ARM
> architecture.  There are currently Ubuntu and Fedora images available
> for the ARM architecture.
> 
> For this work has been concentrating on getting things working on the
> Fedora 13 release candidate with SystemTap running in an a self hosted
> manner.  The variations in the ARM platforms make it difficult to have
> a single binary kernel RPM for all the arm platforms (kernel-debuginfo
> and kernel-devel rpms are rare).  This work has been using a locally
> built linux kernel to make all the kernel debuginfo and devel
> information available.
> 
> The goal is to be able to run the systemtap testsuite to find to what
> tests work and fail on the ARM architecture. In the past people have
> just cross compiled an individual script on one machine and transfered
> them over to the target ARM machine. The testsuite is not designed to
> run in cross-compiled manner.
> 
> Reviewing early test results and the code the following is an
> incomplete list of improvements:
> 
> -created aux_syscalls.stp (FIXED)
>   "make installcheck" uses debuginfo smoke test
>   needs _arch_ptrace_argstr(), aux_syscalls.stp
> -map arm variants (armv7l and armv5tel) to arm (PR4186)
> -provide tapset/arm/registers.stp
>   -map register name to/from sruct
>   -functions for nd probe argument access
> -add any arm specific syscalls in tapset/arm/syscalls.stp
> -provide high-resolution timer for gettimeofday_ns()
>   -this may be complicated by the variety of arm high-res timer mechanisms
> 
> Right now the "make installcheck" tests are running. The ARM machine
> is rather slow, but once the results are available will mail them to
> dejazilla so have a reference point for the ARM architecture.
> 
> -Will

I have been working on improving the test results for the arm. I found that loc2c-runtime.h had some problem on arm:

-arm misses handling 64-bit get, so has the same restrictions as 32-bit i386
-asm statement for __stp_put_user_asm_dword() appeared to be wrong (wondering if can eliminate entirely)

I have taken an attempt at making a tapset/arm/registers.stp file. It still needs some work. The first 4 args are passed in registers on arm, but it doesn't handle arg 5 or 6. Feedback on it would be appreciated.

These changes improved the test results. Went from 190 failures down to 114:

		=== systemtap Summary ===

# of expected passes		1107
# of unexpected failures	114
# of unexpected successes	9
# of expected failures		232
# of known failures		2
# of untested testcases		187
# of unsupported tests		5



-Will



[-- Attachment #2: loc2-runtime-arm.diff --]
[-- Type: text/x-patch, Size: 833 bytes --]

diff --git a/runtime/loc2c-runtime.h b/runtime/loc2c-runtime.h
index 968045f..f8ed19a 100644
--- a/runtime/loc2c-runtime.h
+++ b/runtime/loc2c-runtime.h
@@ -789,8 +789,8 @@ extern void __store_deref_bad(void);
 	"	.long	1b, 4b\n"				\
 	"	.long	2b, 4b\n"				\
 	"	.previous"					\
-	: "+r" (err), "+r" (__pu_addr)				\
-	: "r" (x), "i" (-EFAULT)				\
+	: "+r" (err)						\
+	: "r" (x), "r" (__pu_addr), "i" (-EFAULT)		\
 	: "cc")
 
 #define deref(size, addr)						\
@@ -939,9 +939,9 @@ extern void __store_deref_bad(void);
 #endif /* (s390) || (s390x) */
 
 
-#if defined __i386__
+#if defined (__i386__) || defined (__arm__)
 
-/* x86 can't do 8-byte put/get_user_asm, so we have to split it */
+/* x86 and arm can't do 8-byte put/get_user_asm, so we have to split it */
 
 #define kread(ptr)					\
   ((sizeof(*(ptr)) == 8) ?				\

[-- Attachment #3: registers.stp --]
[-- Type: text/plain, Size: 3973 bytes --]

/* Dwarfless register access for arm */

global _reg_offsets, _stp_regs_registered

function _stp_register_regs() {

	/* Same order as pt_regs */
	_reg_offsets["r0"] =  0		_reg_offsets["a1"] =  0
	_reg_offsets["r1"] =  4		_reg_offsets["a2"] =  4
	_reg_offsets["r2"] =  8		_reg_offsets["a3"] =  8
	_reg_offsets["r3"] = 12		_reg_offsets["a4"] =  12
	_reg_offsets["r4"] = 16		_reg_offsets["v1"] =  16
	_reg_offsets["r5"] = 20		_reg_offsets["v2"] =  20
	_reg_offsets["r6"] = 24		_reg_offsets["v3"] =  24
	_reg_offsets["r7"] = 28		_reg_offsets["v4"] =  28
	_reg_offsets["r8"] = 32		_reg_offsets["v5"] =  32
	_reg_offsets["r9"] = 36  	_reg_offsets["v6"] = 36
	_reg_offsets["r10"] = 40	_reg_offsets["v7"] =  40
	_reg_offsets["fp"] = 44		_reg_offsets["v8"] =  44
	_reg_offsets["ip"] = 48
	_reg_offsets["sp"] = 52
	_reg_offsets["lr"] = 56
	_reg_offsets["pc"] = 60
	_reg_offsets["cpsr"] = 64
	_reg_offsets["orig_r0"] = 68

	_stp_regs_registered = 1
}

function _stp_get_register_by_offset:long (offset:long) %{ /* pure */
	long value;
	if (!CONTEXT->regs) {
		CONTEXT->last_error = "No registers available in this context";
		return;
	}
	if (THIS->offset < 0 || THIS->offset > sizeof(struct pt_regs) - sizeof(long)) {
		snprintf(CONTEXT->error_buffer, sizeof(CONTEXT->error_buffer),
				"Bad register offset: %lld", THIS->offset);
		CONTEXT->last_error = CONTEXT->error_buffer;
		return;
	}
	memcpy(&value, ((char *)CONTEXT->regs) + THIS->offset, sizeof(value));
	THIS->__retvalue = value;
%}

function _stp_probing_kernel:long () %{ /* pure */
	THIS->__retvalue = !user_mode(CONTEXT->regs);
%}

/* Return the named register value as a signed value. */
function register:long (name:string) {
	if (!registers_valid()) {
		error("cannot access CPU registers in this context")
		return 0
	}
	if (!_stp_regs_registered)
		_stp_register_regs()
	offset = _reg_offsets[name]
	if (offset == 0 && !(name in _reg_offsets)) {
		error("Unknown register: " . name)
		return 0
	}
	return _stp_get_register_by_offset(offset)
}

/*
 * Return the named register value as an unsigned value.  Specifically,
 * don't sign-extend the register value when promoting it to 64 bits.
 */
function u_register:long (name:string) {
	return register(name) & 0xffffffff;
}

/* Return the value of function arg #argnum (1=first arg) as a signed value. */
function _stp_arg:long (argnum:long) {
	val = 0
	if (argnum < 1 || argnum > 4) {
		error(sprintf("Cannot access arg(%d)", argnum))
		return 0
	}

	if (argnum == 1)
		val = u_register("r0")
	else if (argnum == 2)
		val = u_register("r1")
	else if (argnum == 3)
		val = u_register("r2")
	else if (argnum == 4)
		val = u_register("r3")

	return val;
}

/* Return the value of function arg #argnum as a signed int. */
function int_arg:long (argnum:long) {
	return _stp_arg(argnum)
}

/* Return the value of function arg #argnum as an unsigned int. */
function uint_arg:long (argnum:long) {
	return _stp_arg(argnum) & 0xffffffff;
}

function long_arg:long (argnum:long) {
	return int_arg(argnum)
}

function ulong_arg:long (argnum:long) {
	return uint_arg(argnum)
}

function longlong_arg:long (argnum:long) {
	/*
	 * TODO: If argnum == nr_regarg, gcc puts the whole 64-bit arg
	 * on the stack.
	 */
	lowbits = uint_arg(argnum)
	highbits = uint_arg(argnum+1)
	return ((highbits << 32) | lowbits)
}

function ulonglong_arg:long (argnum:long) {
	return longlong_arg(argnum)
}

function pointer_arg:long (argnum:long) {
	return ulong_arg(argnum)
}

function s32_arg:long (argnum:long) {
	return int_arg(argnum)
}

function u32_arg:long (argnum:long) {
	return uint_arg(argnum)
}

function s64_arg:long (argnum:long) {
	return longlong_arg(argnum)
}

function u64_arg:long (argnum:long) {
	return ulonglong_arg(argnum)
}

function asmlinkage() %{ /* pure */ %}

function fastcall() %{ /* pure */ %}

function regparm() %{
	snprintf(CONTEXT->error_buffer, sizeof(CONTEXT->error_buffer),
		"regparm is invalid on arm.");
	CONTEXT->last_error = CONTEXT->error_buffer;
%}