systemtap ARM port status

systemtap ARM port status

[Anderson Lizardo]

Hi Quentin/all,

After a quick look on the systemtap archives, I saw some references to
an ongoing systemtap ARM port, e.g. [1]. We have some interest on such
port, but we don't want to dupplicate work. So I have some questions:

- Is there any patches available we could test? We have some OMAP
boards available so we do some test on them.

- How are you running the stap scripts on the platform? We are doing
something like:

1) On host: stap -p 4 -k script.stap
2) Copy generated .ko to target
3) On target: insmod stap-module.ko

[1] http://sourceware.org/ml/systemtap/2007-q2/msg00231.html

Regards,
-- 
Anderson Lizardo
Open Source Mobile Research Center - OSMRC
Nokia Institute of Technology - INdT
Manaus - Brazil

Re: systemtap ARM port status

[Quentin Barnes]

>Hi Quentin/all,
>
>After a quick look on the systemtap archives, I saw some references to
>an ongoing systemtap ARM port, e.g. [1]. We have some interest on such
>port, but we don't want to dupplicate work. So I have some questions:
>
>- Is there any patches available we could test? We have some OMAP
>boards available so we do some test on them.

Recently I've given a patch to support ARM in the Systemtap runtime
to Martin Hunt.  He's doing a careful review of it for me.  The
patch isn't completely there yet, but it is very, very close to
being complete.  If the ARM port work doesn't make it into the next
weekly snapshot, I'll see about a more direct access.

In the last few days I have found a couple of additional changes to
the ARM patch I sent Martin.  I'm probably going to send them to
him later today. I also am having trouble understanding a couple of
things in the runtime that's causing me trouble.  I wrote them up
and asked him about them.  I think once those things are resolved,
the ARM port of the runtime will be basically 100% there.

Note though that the ARM port work doesn't yet include all the
changes to get the testsuite to run on an ARM platform.  I'm still
working on those and will be submitting them to the list soon.

>- How are you running the stap scripts on the platform? We are doing
>something like:
>
>1) On host: stap -p 4 -k script.stap
>2) Copy generated .ko to target
>3) On target: insmod stap-module.ko

Well, for now I'm running completely native.  Yes, that's very, very
painful to run gcc and g++ on such a board with an NFS mounted file
system and NFS mounted swap file over loopback.  It can take four
hours of wall time to compile and link the "stap" binary and the
better part of a day to run the full testsuite.

I've had so many problems to solve getting Systemtap to support
ARM, I didn't want to have to also deal with problems introduced
from trying to get it to also support cross-builds simultaneously.
I decided to work on one problem at a time.

There is another group who's adding cross-target support to Systemtap:
http://tree.celinuxforum.org/CelfPubWiki/ELC2007TechnicalShowcase?action=AttachFile&do=get&target=SystemTap_Poster.ppt

I have had only limited contact with them so far.  I would like to
have more and come up to speed on their work.


To back up a bit, where are you getting the ARM Kprobes port from?
Is this based on Sagar and my work I posted to the ARM Linux kernel
mailing list recently, or some other ARM Kprobes port?

>[1] http://sourceware.org/ml/systemtap/2007-q2/msg00231.html
>
>Regards,
>-- 
>Anderson Lizardo
>Open Source Mobile Research Center - OSMRC
>Nokia Institute of Technology - INdT
>Manaus - Brazil

Quentin

Re: systemtap ARM port status

[Anderson Lizardo]

On 6/1/07, Quentin Barnes <qbarnes@urbana.css.mot.com> wrote:
> Note though that the ARM port work doesn't yet include all the
> changes to get the testsuite to run on an ARM platform.  I'm still
> working on those and will be submitting them to the list soon.

Nice! Have you already tested some of the example .stp scripts from
the systemtap package on ARM?

> Well, for now I'm running completely native.  Yes, that's very, very
> painful to run gcc and g++ on such a board with an NFS mounted file
> system and NFS mounted swap file over loopback.  It can take four
> hours of wall time to compile and link the "stap" binary and the
> better part of a day to run the full testsuite.

How do you solve the problem of having the kernel and the modules
compiled with the same   toolchain? Do you also compile the kernel on
ARM?

> There is another group who's adding cross-target support to Systemtap:
> http://tree.celinuxforum.org/CelfPubWiki/ELC2007TechnicalShowcase?action=AttachFile&do=get&target=SystemTap_Poster.ppt
>
> I have had only limited contact with them so far.  I would like to
> have more and come up to speed on their work.

Thanks for the link. We will look into it once we have some
implementation to work on :)

> To back up a bit, where are you getting the ARM Kprobes port from?
> Is this based on Sagar and my work I posted to the ARM Linux kernel
> mailing list recently, or some other ARM Kprobes port?

We are using the patches you sent to LKML a while ago. Any changes to
those patches since then?

BTW, we had a couple of problems getting them to compile for OMAP and
QEMU (emulated Versatile platform):

- errors about undefined "__asm__" macros on kprobes-decode.c. To
workaround, we changed all occurrences to "asm".
- an error about undefined "__LINUX_ARM__ARCH___" (note the additional
underlines) on kprobes-decode.c. We changed it to
"__LINUX_ARM_ARCH__". Maybe it was a typo?
- we removed the "___attribute___((naked))" atributte from
kretprobe_trampoline_holder() since our compiler (CodeSourcery
2006q3-27) didn't seem to like it)

After patching these issues, we could get some simple tests from
kprobes.txt to run.

Regards,
-- 
Anderson Lizardo
Open Source Mobile Research Center - OSMRC
Nokia Institute of Technology - INdT
Manaus - Brazil

Re: systemtap ARM port status

[Quentin Barnes]

On Fri, Jun 01, 2007 at 04:12:58PM -0400, Anderson Lizardo wrote:
>On 6/1/07, Quentin Barnes <qbarnes@urbana.css.mot.com> wrote:
>>Note though that the ARM port work doesn't yet include all the
>>changes to get the testsuite to run on an ARM platform.  I'm still
>>working on those and will be submitting them to the list soon.
>
>Nice! Have you already tested some of the example .stp scripts from
>the systemtap package on ARM?

No, I've been concentrating on getting the full testsuite to pass
first.  I've just made some changes today that I think will get
virtually all the tests to pass that are expected to pass on the
kernel version and configuration I'm using.

However, one of the changes I made to loc2c-runtime.h shouldn't be
necessary, if anything, it should cause things to fail, but instead
it makes some tests now pass.  I sent some mail to Martin about it
to see if he has any ideas as to what's going on.

>>Well, for now I'm running completely native.  Yes, that's very, very
>>painful to run gcc and g++ on such a board with an NFS mounted file
>>system and NFS mounted swap file over loopback.  It can take four
>>hours of wall time to compile and link the "stap" binary and the
>>better part of a day to run the full testsuite.
>
>How do you solve the problem of having the kernel and the modules
>compiled with the same toolchain? Do you also compile the kernel on
>ARM?

I do cross-build the kernel for all my development.

You don't have to use the identical toolchain for both.  You just
have to configure the native and cross kernels identically, which I
do.

On my cross environment, I'm using CodeSourcery's ARM Sourcery G++
2006q3-26.  Natively I'm using Debian 4.1.1-21.

I push my devel kernel tree over to the native platform then rebuild
enough of the kernel tools to build modules.  Basically, I find
all objects that are i386 native in the "O" directory of the kernel
tree, rm them, then natively "make" to just build the necessary
pieces natively.

>>To back up a bit, where are you getting the ARM Kprobes port from?
>>Is this based on Sagar and my work I posted to the ARM Linux kernel
>>mailing list recently, or some other ARM Kprobes port?
>
>We are using the patches you sent to LKML a while ago. Any changes to
>those patches since then?

I never posted anything to the LKML mailing list.  I've only posted
to Sytemtap (this list) and the ARM Kernel mailing list.  You should
definitely use the work from my post to the latter on May 17.

>BTW, we had a couple of problems getting them to compile for OMAP and
>QEMU (emulated Versatile platform):
>
>- errors about undefined "__asm__" macros on kprobes-decode.c. To
>workaround, we changed all occurrences to "asm".
>- an error about undefined "__LINUX_ARM__ARCH___" (note the additional
>underlines) on kprobes-decode.c. We changed it to
>"__LINUX_ARM_ARCH__". Maybe it was a typo?
>- we removed the "___attribute___((naked))" atributte from
>kretprobe_trampoline_holder() since our compiler (CodeSourcery
>2006q3-27) didn't seem to like it)

These are all very strange problems you shouldn't have had.  First,
we're using virtually identical compilers (CodeSourcery's 2006q3-26
vs. 2006q3-27).

The keyword "__asm__" has been in all versions of gcc I'm personally
aware of going all the way back to gcc 2.95.3.  You can find it on
the gcc.gnu.org documentation even for that version.

In the patch sets I posted to both mailing lists, the macro is
"__LINUX_ARM_ARCH__", not "__LINUX_ARM__ARCH___" or anything else.
Sounds like you have a damaged or strangely hacked patch.

The attribute "__attribute__((naked))" has been in the ARM port a
very long time, at least since gcc 3.2.3, and I'm positive it is in
2006q3-27.

Where did you get your CodeSourcery compiler from?  Are you running
the binary release, or did you build it yourself from the source
release?  Or could someone have hacked the default compiler flags in
your arch/arm/Makefile?

Sounds like you've got both a bad patch and a misconfigured GCC
compiler.

>After patching these issues, we could get some simple tests from
>kprobes.txt to run.
>
>Regards,
>-- 
>Anderson Lizardo
>Open Source Mobile Research Center - OSMRC
>Nokia Institute of Technology - INdT
>Manaus - Brazil

Quentin

Re: systemtap ARM port status

[Roland McGrath]

> However, one of the changes I made to loc2c-runtime.h shouldn't be
> necessary, if anything, it should cause things to fail, but instead
> it makes some tests now pass.  I sent some mail to Martin about it
> to see if he has any ideas as to what's going on.

Please use this mailing list for such discussion.


Thanks,
Roland

Re: systemtap ARM port status

[Quentin Barnes]

On Fri, Jun 01, 2007 at 01:49:51PM -0700, Roland McGrath wrote:
>> However, one of the changes I made to loc2c-runtime.h shouldn't be
>> necessary, if anything, it should cause things to fail, but instead
>> it makes some tests now pass.  I sent some mail to Martin about it
>> to see if he has any ideas as to what's going on.
>
>Please use this mailing list for such discussion.

Hi Roland!  I don't think we've swapped mail in some time, maybe
even going back to the gmake alpha devel mailing list back in the
'90s.

Well, gee, I think that's a real first for me.  I'm often told to
take a discussion off a mailing list to e-mail, not the other way
around!

Okay, you ask for it; you got it.  The two mails I sent to Martin
about these issues are below.

>Thanks,
>Roland

Quentin

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Date: Fri, 1 Jun 2007 00:17:24 -0500
From: Quentin Barnes <qbarnes@urbana.css.mot.com>
To: Martin Hunt <hunt@redhat.com>
Subject: Strange failures with syscall testsuite

I've got most all of the testsuite tests passing (or at least
understand why they fail) with one big, notable exception, all
the tests under systemtap.syscall fail.  I've been working on
understanding the failures, but am starting to hit a wall due to
several anomalies I've encountered.

Here's a sample failure:
=====
Testing 32-bit access
FAIL: 32-bit access
access FAILED. output of "stap -c ../access /usr/src/systemtap-20070519/testsuite/systemtap.syscall/sys.stp" was:
------------------------------------------
staprun: getpid () = N/A
staprun: getrlimit (RLIMIT_STACK, 0xbec5caf0) = N/A
staprun: rt_sigaction (32, 0xbec5ca70, 0x00000000, 8) = N/A
staprun: rt_sigaction (33, 0xbec5ca70, 0x00000000, 8) = N/A
staprun: rt_sigaction (34, 0xbec5ca70, 0x00000000, 8) = N/A
WARNING: Number of errors: 1, skipped probes: 0
ERROR: kernel string copy fault at 0xc1068000 near identifier '$filename' at /usr/local/share/systemtap/tapset/syscalls.stp:49:48
------------------------------------------
RESULTS: ('*' = MATCHED EXPECTED)
--------- EXPECTED and NOT MATCHED ----------
access: access \("foobar1", F_OK\) = 0
access: access \("foobar1", R_OK\) = 0
access: access \("foobar1", W_OK\) = 0
access: access \("foobar1", X_OK\) = -[\-0-9]+ \(EACCES\)
access: access \("foobar1", W_OK \|R_OK\) = 0
access: access \("foobar1", X_OK \|W_OK \|R_OK\) = -[\-0-9]+ \(EACCES\)
=====

0xc1068000 is a legit kernel memory address on ARM.

When I look at line 49 in syscalls.stp, it is:
        argstr = sprintf("%s, %s", user_string_quoted($filename), mode_str)

$filename is the first argument to sys_access() which has type
"const char __user *filename".  I dumped the regs as passed into
enter_kprobe_probe().  R0 matches the address (i.e. 0xc1068000),
so that seems fine, sort of.

The first anomaly is 0xc1068000 is a _kernel_ memory address.
Shouldn't the address passed into sys_access() be a user-space
address due to the "__user" qualifier?  I don't know Linux kernel
programming nuances, but I thought that tag meant it was expected to
be a user-space address only.

I assume the error as reported is from expanding the
"user_string_quoted($filename)" expression.  Having a user space
string routine expand a kernel address would certainly be fatal!
ARM uses an "ldrbt" instruction to do the copy -- that would fault
on trying to copy from a kernel memory address.

But then there's anomaly number two.  The error message is
"ERROR: kernel string copy fault at ...". This is a message from
function_kernel_string().  This function is expecting to copy a
string at a kernel address.  How did it get invoked from a service
called "user_string_quoted"?

In looking at the implementation of function_kernel_string(),
it calls deref_string(), a macro in loc2c-runtime.h.  deref_string()
invokes deref().  For byte operations, deref() invokes a platform
specific call.  In the case of i386, it is called "__get_user_asm()".
Now I would expect such a routine to perform user-space only loading
of data and accessing kernel data would fault.  Anomaly number three
is why doesn't calling such a function from function_kernel_string()
fault on kernel memory access on other platforms like it does on ARM?

As a hack, I decided to modify ARM's call of __get_user_asm_byte()
by deref() to just do "*(char *)addr".  Suddenly, all the test cases
that were previously failing with the "ERROR: kernel string copy
fault at ..." diagnostic are now making it past that point and much
further!  (Most of the tests might actually even be passing now,
but debug messages I put into the stap output are throwing off the
Expect scripts.)

Could ARM be doing something unusual (or the systemtap test suite)
with providing a kernel data address to sys_access() and the other 
system calls?

What am I not understanding?  Any words of wisdom on what's going on
and how to solve this?

Quentin


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Date: Fri, 1 Jun 2007 15:04:31 -0500
From: Quentin Barnes<qlb01+stap@qbarnes.org>
To: Martin Hunt<hunt@redhat.com>
Subject: Re: Basic ARM support for Systemtap

At the end of this mail are three additional files for ARM on top of
the ones I sent you.  They are pretty straight forward.  The patch
fixes the "N/A" in the output I had sent you (and one cast).

In the mean time, I hacked the deref macro in loc2c-runtime.h to
be this for ARM:
=====
#define deref(size, addr)                             \
  ({                                                  \
     int _bad = 0;                                    \
     intptr_t _v=0;                                   \
      switch (size){                                  \
      case 1: _v = *(char *)addr; break;              \
      case 2: _v = *(short *)addr; break;             \
      case 4: _v = *(int *)addr; break;               \
      default: __get_user_bad(); break;               \
      }                                               \
    if (_bad)                                         \
      goto deref_fault;                               \
     _v;                                              \
   }) 
=====

This hackery above and the patch below, for the first time ever, I'm
getting the "systemtap.syscall" testsuite to pass!  I don't know how
many will pass.  It will take another several hours for the suite to
finish running that section on my devel board, but I am very curious
to see.

If you can help me understand what's going on here with why the
above hackery "works" at all, it would be a big help.  (My questions
are in the previous mail I sent in the wee hours earlier today).

Quentin




Index: runtime/stack-arm.c
===================================================================
--- runtime/stack-arm.c	(revision 195)
+++ runtime/stack-arm.c	(working copy)
@@ -59,7 +59,7 @@ static void __stp_stack_print (struct pt
 			_stp_symbol_print((unsigned long)pc);
 			_stp_print_char('\n');
 		} else {
-			_stp_printf("%08lx ", pc);
+			_stp_printf("%08lx ", (unsigned long)pc);
 		}
 
 		/* Sanity check the next_fp. */
Index: runtime/regs.c
===================================================================
--- runtime/regs.c	(revision 195)
+++ runtime/regs.c	(working copy)
@@ -46,6 +46,8 @@ unsigned long _stp_ret_addr (struct pt_r
 	return regs->b0;
 #elif defined (__s390__) || defined (__s390x__)
 	return regs->gprs[14];
+#elif defined (__arm__)
+	return regs->ARM_r0;
 #else
 	#error Unimplemented architecture
 #endif
Index: tapset/errno.stp
===================================================================
--- tapset/errno.stp	(revision 194)
+++ tapset/errno.stp	(working copy)
@@ -370,6 +370,8 @@ function returnstr:string (returnp:long)
 		ret = CONTEXT->regs->u_regs[UREG_RETPC];
 #elif defined (__s390x__)
 		ret = CONTEXT->regs->gprs[2];
+#elif defined (__arm__)
+		ret = CONTEXT->regs->ARM_r0;
 #else
 		goto no_ret;
 #endif

Re: systemtap ARM port status

[Roland McGrath]

> Hi Roland!  I don't think we've swapped mail in some time, maybe
> even going back to the gmake alpha devel mailing list back in the
> '90s.

:-)  You'll forgive me if I don't recall.  I've always been lousy at even
noticing who it is I'm conversing with in email.  And I've been actively
repressing memories associated with make for far over a decade now.  ;-)

> Well, gee, I think that's a real first for me.  I'm often told to
> take a discussion off a mailing list to e-mail, not the other way
> around!

Technical details almost always belong on a mailing list.  Then everything
is archived for future reference.  Moreover, anyone with something to
contribute can help.  It's rarely the case that exactly one person is the
one and only person who can figure out any particular problem most easily,
let alone that one can guess a priori who that person is.


The loc2c-runtime.h macros deref and store_deref are intended for dealing
with kernel addresses only.  Any appearance of "user" in their definitions
is a machine-specific implementation detail that should not concern you.
The specified use of these macros is for kernel-space addresses, with
unspecified behavior when given a user-space address.  The implementations
we have do not check that the addresses passed are kernel-space addresses,
since they don't have to.  In almost all cases, they could check (and
perhaps it would be wise for bug-catching).  But in the general case, there
is not necessarily any such check that can be made.  (The actual instance of
that case is the "4G/4G" kernel, which exists in RHEL4 for i386.)  

On machines systemtap has worked on so far, in almost all configurations
(all but 4g/4g), the relationship between user and kernel address spaces is
such that using the deref macro works the same for either kind of address.
A particular implementation of the runtime's user_* functions for a
particular machine and configuration might rely on this fact and use those
macros for some reason.  Martin will have to answer about those particulars.
I have only been directly involved with the loc2c-runtime.h code.  I would
have thought that user address access in the runtime would just use the
vanilla uaccess.h calls (get_user et al), but perhaps there is some
__might_sleep issue in kprobes handlers.

For a new machine, the deref/store_deref macros can be implemented however
works best or is easiest for you to write and maintain, and need only
support access to a kernel-space address.  What it must do is gracefully
goto deref_fault for any kind address access that does not work.  On
configurations where kernel and user address values are disjoint, it is
certainly kosher for it to goto deref_fault for a user address.


Thanks,
Roland

Re: systemtap ARM port status

[Frank Ch. Eigler]

Roland McGrath <roland@redhat.com> writes:

> [...]  In almost all cases, they could check [for kernel- vs
> user-space pointers].  But in the general case, there is not
> necessarily any such check that can be made.  (The actual instance
> of that case is the "4G/4G" kernel, which exists in RHEL4 for i386.)

I suspect that we will care little about preserving full function on
4G/4G if in exchange we can make systemtap more robust everywhere
else.  For one thing, I expect not to see utrace and uprobes on RHEL4,
so it will fall behind anyway.

> I would have thought that user address access in the runtime would
> just use the vanilla uaccess.h calls (get_user et al), but perhaps
> there is some __might_sleep issue in kprobes handlers.

That's exactly it.  General kprobe handlers need to be atomic.

> [...]  What it must do is gracefully goto deref_fault for any kind
> address access that does not work.  [...]

This includes detection of problems such as unaligned, unmapped,
I/O-sensitive pointer values.  Assume only that the pointer fits into
64 bits, but otherwise it might be random.

- FChE

Re: systemtap ARM port status

[Roland McGrath]

> I suspect that we will care little about preserving full function on
> 4G/4G if in exchange we can make systemtap more robust everywhere
> else.  For one thing, I expect not to see utrace and uprobes on RHEL4,
> so it will fall behind anyway.

It's easily checked by an #ifdef CONFIG_something.  So there is no reason
the macros couldn't reject user addresses on every other flavor of kernel.


Thanks,
Roland

Re: systemtap ARM port status

[Quentin Barnes]

>> Hi Roland!  I don't think we've swapped mail in some time, maybe
>> even going back to the gmake alpha devel mailing list back in the
>> '90s.
>
>:-)  You'll forgive me if I don't recall.  I've always been lousy at even
>noticing who it is I'm conversing with in email.  And I've been actively
>repressing memories associated with make for far over a decade now.  ;-)

Oh, I don't fault you at all.  I'm the same way with names, just
absolutely horrible, and with e-mail too.

I think though if I dug up some of our good discussions from back
then, I'm pretty sure you'd recall one or two.

>> Well, gee, I think that's a real first for me.  I'm often told to
>> take a discussion off a mailing list to e-mail, not the other way
>> around!
>
>Technical details almost always belong on a mailing list.  Then
>everything is archived for future reference.  Moreover, anyone
>with something to contribute can help.  It's rarely the case that
>exactly one person is the one and only person who can figure out
>any particular problem most easily, let alone that one can guess a
>priori who that person is.

If Martin didn't know the answer off the top of his head, I would
have certainly taken the technical issues to the list.

>The loc2c-runtime.h macros deref and store_deref are intended for
>dealing with kernel addresses only.  Any appearance of "user" in
>their definitions is a machine-specific implementation detail that
>should not concern you.

I does concern me if I'm trying to understand and port the code!  :-)

>The specified use of these macros is for kernel-space addresses,
>with unspecified behavior when given a user-space address.

Ah, ha!  Some pieces are starting to come together.  I get this
comment block now from loc2c-runtime.h:

   On most machines, the asm/uaccess.h macros __get_user_asm and
   __put_user_asm do exactly the low-level work we need to access
   memory with fault handling, and are not actually specific to
   user-address access at all.  Each machine's definition of deref
   and deref_store here must work right for kernel addresses, and
   can use whatever existing machine-specific kernel macros are
   convenient.

As I discovered the hard way, this assertion about the
__{get|put}_user_asm*() macros is apparently not true on ARM, hence
my problem.

Only knowing the ARM port to date, and from their names, I would not
have guessed that the opposite is true in the general case, that
those macros could safely be used on kernel addresses on most other
architectures.  (And the whole bit with __user and such was a good
red herring.)  Sigh, another case of inappropriately named kernel
services confusing their purpose.  I suppose it's considered an
implementation detail which way they happen to work though.

I would think then that the ARM version of __get_user_asm_byte()
macro is being (overly?) pedantic.  There's no need for it to use
the user-only space byte load instruction, "ldrbt".  It could have
used the general byte load "ldrb" instruction.  As I now understand
it, all user-space addresses prior to accessing by the kernel must
be checked with access_ok() anyways to make sure the address is in
the range that a user space process can legally access.

So I think all I have to do to properly port the deref() macro
for ARM is to copy the ARM's __{get|put}_user_asm_{byte|half|word}()
macros to loc2c-runtime.h, rename them to the _stp_* variety, and
have them use the "ldrb", "strb", "ldr", and "str" instructions.


Aside from the Sytemtap issue, I would like to understand why
fs/open.c:sys_access() "filename" parameter is tagged with "__user",
but it seems to be passed a kernel address.  I thought for sure this
always meant that such a qualified pointer is expected to be holding
a user-space only address.  Is my understanding of this qualifier in
Linux kernel programming errant, or is the ARM port broken in this
way?


[...]

>Thanks, Roland

Quentin

Re: systemtap ARM port status

[Quentin Barnes]

On Fri, Jun 01, 2007 at 10:29:37PM -0500, Quentin Barnes wrote:
[...]
>So I think all I have to do to properly port the deref() macro
>for ARM is to copy the ARM's __{get|put}_user_asm_{byte|half|word}()
>macros to loc2c-runtime.h, rename them to the _stp_* variety, and
>have them use the "ldrb", "strb", "ldr", and "str" instructions.

Overnight testing revealed that works.  Attached is the ARM version
that seems to do the right thing:


Index: loc2c-runtime.h
===================================================================
--- loc2c-runtime.h	(.../vendor/usr/src/systemtap-20070526/runtime/loc2c-runtime.h)	(revision 206)
+++ loc2c-runtime.h	(.../branches/kprobes/usr/src/systemtap-20070526/runtime/loc2c-runtime.h)	(revision 206)
@@ -112,6 +112,13 @@
 #define fetch_register(regno) ((intptr_t) c->regs->gpr[regno])
 #define store_register(regno) (c->regs->gpr[regno] = (value))
 
+#elif defined (__arm__)
+
+#undef fetch_register
+#undef store_register
+#define fetch_register(regno) ((long) c->regs->uregs[regno])
+#define store_register(regno) (c->regs->uregs[regno] = (value))
+
 #elif defined (__s390__) || defined (__s390x__)
 #undef fetch_register
 #undef store_register
@@ -294,6 +301,141 @@
       goto deref_fault;							      \
   })
 
+#elif defined (__arm__)
+
+#define __stp_get_user_asm_byte(x,addr,err)			\
+	__asm__ __volatile__(					\
+	"1:	ldrb	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	mov	%1, #0\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err), "=&r" (x)					\
+	: "r" (addr), "i" (-EFAULT)				\
+	: "cc")
+
+#ifndef __ARMEB__
+#define __stp_get_user_asm_half(x,__gu_addr,err)		\
+({								\
+	unsigned long __b1, __b2;				\
+	__stp_get_user_asm_byte(__b1, __gu_addr, err);		\
+	__stp_get_user_asm_byte(__b2, __gu_addr + 1, err);	\
+	(x) = __b1 | (__b2 << 8);				\
+})
+#else
+#define __stp_get_user_asm_half(x,__gu_addr,err)		\
+({								\
+	unsigned long __b1, __b2;				\
+	__stp_get_user_asm_byte(__b1, __gu_addr, err);		\
+	__stp_get_user_asm_byte(__b2, __gu_addr + 1, err);	\
+	(x) = (__b1 << 8) | __b2;				\
+})
+#endif
+
+#define __stp_get_user_asm_word(x,addr,err)			\
+	__asm__ __volatile__(					\
+	"1:	ldr	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	mov	%1, #0\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err), "=&r" (x)					\
+	: "r" (addr), "i" (-EFAULT)				\
+	: "cc")
+
+#define __stp_put_user_asm_byte(x,__pu_addr,err)		\
+	__asm__ __volatile__(					\
+	"1:	strb	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err)						\
+	: "r" (x), "r" (__pu_addr), "i" (-EFAULT)		\
+	: "cc")
+
+#ifndef __ARMEB__
+#define __stp_put_user_asm_half(x,__pu_addr,err)			\
+({									\
+	unsigned long __temp = (unsigned long)(x);			\
+	__stp_put_user_asm_byte(__temp, __pu_addr, err);		\
+	__stp_put_user_asm_byte(__temp >> 8, __pu_addr + 1, err);	\
+})
+#else
+#define __stp_put_user_asm_half(x,__pu_addr,err)			\
+({									\
+	unsigned long __temp = (unsigned long)(x);			\
+	__stp_put_user_asm_byte(__temp >> 8, __pu_addr, err);		\
+	__stp_put_user_asm_byte(__temp, __pu_addr + 1, err);		\
+})
+#endif
+
+#define __stp_put_user_asm_word(x,__pu_addr,err)		\
+	__asm__ __volatile__(					\
+	"1:	str	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err)						\
+	: "r" (x), "r" (__pu_addr), "i" (-EFAULT)		\
+	: "cc")
+
+#define deref(size, addr)						\
+  ({									\
+     int _bad = 0;							\
+     intptr_t _v=0;							\
+	switch (size){							\
+	case 1: __stp_get_user_asm_byte(_v, addr, _bad); break; 	\
+	case 2: __stp_get_user_asm_half(_v, addr, _bad); break; 	\
+	case 4: __stp_get_user_asm_word(_v, addr, _bad); break; 	\
+	default: __get_user_bad(); break;				\
+	}								\
+    if (_bad)  								\
+	goto deref_fault;						\
+     _v;								\
+   })
+
+#define store_deref(size, addr, value)					\
+  ({									\
+    int _bad=0;								\
+	switch (size){							\
+	case 1: __stp_put_user_asm_byte(value, addr, _bad); break;	\
+	case 2: __stp_put_user_asm_half(value, addr, _bad); break;	\
+	case 4: __stp_put_user_asm_word(value, addr, _bad); break;	\
+	case 8: __stp_put_user_asm_dword(value, addr, _bad); break;	\
+	default: __put_user_bad(); break;				\
+	}								\
+    if (_bad)								\
+	   goto deref_fault;						\
+   })
+
 #elif defined (__s390__) || defined (__s390x__)
 
 #ifndef EX_TABLE

Re: systemtap ARM port status

[Roland McGrath]

Despite past questionable precedent, I would not use "user" in those macro
names.  The only reason such names were used in the first place was because
reusing existing asm/uaccess.h macros worked on some machines.


Thanks,
Roland

Re: systemtap ARM port status

[Martin Hunt]

On Sat, 2007-06-02 at 09:11 -0500, Quentin Barnes wrote:
> On Fri, Jun 01, 2007 at 10:29:37PM -0500, Quentin Barnes wrote:
> [...]
> >So I think all I have to do to properly port the deref() macro
> >for ARM is to copy the ARM's __{get|put}_user_asm_{byte|half|word}()
> >macros to loc2c-runtime.h, rename them to the _stp_* variety, and
> >have them use the "ldrb", "strb", "ldr", and "str" instructions.
> 
> Overnight testing revealed that works.  Attached is the ARM version
> that seems to do the right thing:

Sorry I was unable to respond to this sooner; I was out on Friday. 

Do you want me to check this latest version in, or  are you still
testing?

Martin

Re: systemtap ARM port status

[Anderson Lizardo]

On 6/1/07, Quentin Barnes <qbarnes@urbana.css.mot.com> wrote:
> On Fri, Jun 01, 2007 at 04:12:58PM -0400, Anderson Lizardo wrote:
> >We are using the patches you sent to LKML a while ago. Any changes to
> >those patches since then?
>
> I never posted anything to the LKML mailing list.  I've only posted
> to Sytemtap (this list) and the ARM Kernel mailing list.  You should
> definitely use the work from my post to the latter on May 17.

My mistake. I subscribe to both LKML and linux-arm, and I was under
the impression it was sent to the former.

> >BTW, we had a couple of problems getting them to compile for OMAP and
> >QEMU (emulated Versatile platform):
> [...]

I just recompiled my kernel with the patches from linux-arm and all
went ok. I have originally taken the patches from gmane archives, and
they somehow became corrupted.

> Where did you get your CodeSourcery compiler from?  Are you running
> the binary release, or did you build it yourself from the source
> release?  Or could someone have hacked the default compiler flags in
> your arch/arm/Makefile?
>
> Sounds like you've got both a bad patch and a misconfigured GCC
> compiler.

The compiler was the binary distribution from CodeSourcery. It seems
all reported errors were from corrupted patches. We're now testing the
latest 20070602 systemtap snapshot which seems to include ARM support
(from what I've read on its Changelog).

Thanks for your help! Regards,
-- 
Anderson Lizardo
Open Source Mobile Research Center - OSMRC
Nokia Institute of Technology - INdT
Manaus - Brazil

Re: systemtap ARM port status

[Quentin Barnes]

On Mon, Jun 04, 2007 at 04:09:28PM -0400, Anderson Lizardo wrote:
>On 6/1/07, Quentin Barnes <qbarnes@urbana.css.mot.com> wrote:
>>Sounds like you've got both a bad patch and a misconfigured GCC
>>compiler.
>
>The compiler was the binary distribution from CodeSourcery. It seems
>all reported errors were from corrupted patches. We're now testing the
>latest 20070602 systemtap snapshot which seems to include ARM support
>(from what I've read on its Changelog).

I just checked my current work against the 20070602 snapshot.  The
stuff I had given Martin is all there, just the very latest fixes
are missing that's come up in the last week.  They are all trivial
except for the changes to runtime/loc2c-runtime.h.  Make sure you
pick up my patch to runtime/loc2c-runtime.h that I mailed out on
June 2.  I'll be sending out a patch summarizing the latest changes
against 20070602 for ARM later tonight or tomorrow.

Note that you can't run the testsuite on ARM yet even with the
latest changes.  I'll have a patch set for those files probably
tomorrow after I can run the testsuite through once on the new
release.

>Thanks for your help! Regards,
>-- 
>Anderson Lizardo
>Open Source Mobile Research Center - OSMRC
>Nokia Institute of Technology - INdT
>Manaus - Brazil

Quentin

Re: systemtap ARM port status

[Quentin Barnes]

On Mon, Jun 04, 2007 at 10:35:47AM -0400, Martin Hunt wrote:
>On Sat, 2007-06-02 at 09:11 -0500, Quentin Barnes wrote:
>> On Fri, Jun 01, 2007 at 10:29:37PM -0500, Quentin Barnes wrote:
>> [...]
>> >So I think all I have to do to properly port the deref() macro
>> >for ARM is to copy the ARM's __{get|put}_user_asm_{byte|half|word}()
>> >macros to loc2c-runtime.h, rename them to the _stp_* variety, and
>> >have them use the "ldrb", "strb", "ldr", and "str" instructions.
>> 
>> Overnight testing revealed that works.  Attached is the ARM version
>> that seems to do the right thing:
>
>Sorry I was unable to respond to this sooner; I was out on Friday. 
>
>Do you want me to check this latest version in, or  are you still
>testing?

The 20070602 testsuite is still running with this patch, but it's
made it through the critical parts now.  The patch below should
bring the runtime up to fully operational with ARM (at least as far
as I'm currently aware!).  Please inspect it over carefully and
check it in if it is acceptable.

>Martin

Quentin


Index: systemtap-20070602/runtime/stack-arm.c
===================================================================
--- systemtap-20070602/runtime/stack-arm.c	(.../vendor/usr/src)	(revision 214)
+++ systemtap-20070602/runtime/stack-arm.c	(.../branches/kprobes/usr/src)	(revision 214)
@@ -59,7 +59,7 @@ static void __stp_stack_print (struct pt
 			_stp_symbol_print((unsigned long)pc);
 			_stp_print_char('\n');
 		} else {
-			_stp_printf("%08lx ", pc);
+			_stp_printf("%08lx ", (unsigned long)pc);
 		}
 
 		/* Sanity check the next_fp. */
Index: systemtap-20070602/runtime/loc2c-runtime.h
===================================================================
--- systemtap-20070602/runtime/loc2c-runtime.h	(.../vendor/usr/src)	(revision 214)
+++ systemtap-20070602/runtime/loc2c-runtime.h	(.../branches/kprobes/usr/src)	(revision 214)
@@ -303,14 +303,118 @@
 
 #elif defined (__arm__)
 
+#define __stp_get_user_asm_byte(x,addr,err)			\
+	__asm__ __volatile__(					\
+	"1:	ldrb	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	mov	%1, #0\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err), "=&r" (x)					\
+	: "r" (addr), "i" (-EFAULT)				\
+	: "cc")
+
+#ifndef __ARMEB__
+#define __stp_get_user_asm_half(x,__gu_addr,err)		\
+({								\
+	unsigned long __b1, __b2;				\
+	__stp_get_user_asm_byte(__b1, __gu_addr, err);		\
+	__stp_get_user_asm_byte(__b2, __gu_addr + 1, err);	\
+	(x) = __b1 | (__b2 << 8);				\
+})
+#else
+#define __stp_get_user_asm_half(x,__gu_addr,err)		\
+({								\
+	unsigned long __b1, __b2;				\
+	__stp_get_user_asm_byte(__b1, __gu_addr, err);		\
+	__stp_get_user_asm_byte(__b2, __gu_addr + 1, err);	\
+	(x) = (__b1 << 8) | __b2;				\
+})
+#endif
+
+#define __stp_get_user_asm_word(x,addr,err)			\
+	__asm__ __volatile__(					\
+	"1:	ldr	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	mov	%1, #0\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err), "=&r" (x)					\
+	: "r" (addr), "i" (-EFAULT)				\
+	: "cc")
+
+#define __stp_put_user_asm_byte(x,__pu_addr,err)		\
+	__asm__ __volatile__(					\
+	"1:	strb	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err)						\
+	: "r" (x), "r" (__pu_addr), "i" (-EFAULT)		\
+	: "cc")
+
+#ifndef __ARMEB__
+#define __stp_put_user_asm_half(x,__pu_addr,err)			\
+({									\
+	unsigned long __temp = (unsigned long)(x);			\
+	__stp_put_user_asm_byte(__temp, __pu_addr, err);		\
+	__stp_put_user_asm_byte(__temp >> 8, __pu_addr + 1, err);	\
+})
+#else
+#define __stp_put_user_asm_half(x,__pu_addr,err)			\
+({									\
+	unsigned long __temp = (unsigned long)(x);			\
+	__stp_put_user_asm_byte(__temp >> 8, __pu_addr, err);		\
+	__stp_put_user_asm_byte(__temp, __pu_addr + 1, err);		\
+})
+#endif
+
+#define __stp_put_user_asm_word(x,__pu_addr,err)		\
+	__asm__ __volatile__(					\
+	"1:	str	%1,[%2],#0\n"				\
+	"2:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"3:	mov	%0, %3\n"				\
+	"	b	2b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 3b\n"				\
+	"	.previous"					\
+	: "+r" (err)						\
+	: "r" (x), "r" (__pu_addr), "i" (-EFAULT)		\
+	: "cc")
+
 #define deref(size, addr)						\
   ({									\
      int _bad = 0;							\
      intptr_t _v=0;							\
 	switch (size){							\
-	case 1: __get_user_asm_byte(_v, addr, _bad); break; 		\
-	case 2: __get_user_asm_half(_v, addr, _bad); break;  		\
-	case 4: __get_user_asm_word(_v, addr, _bad); break;  		\
+	case 1: __stp_get_user_asm_byte(_v, addr, _bad); break; 	\
+	case 2: __stp_get_user_asm_half(_v, addr, _bad); break; 	\
+	case 4: __stp_get_user_asm_word(_v, addr, _bad); break; 	\
 	default: __get_user_bad(); break;				\
 	}								\
     if (_bad)  								\
@@ -322,10 +426,10 @@
   ({									\
     int _bad=0;								\
 	switch (size){							\
-	case 1: __put_user_asm_byte(value, addr, _bad); break;		\
-	case 2: __put_user_asm_half(value, addr, _bad); break;		\
-	case 4: __put_user_asm_word(value, addr, _bad); break;		\
-	case 8: __put_user_asm_dword(value, addr, _bad); break;		\
+	case 1: __stp_put_user_asm_byte(value, addr, _bad); break;	\
+	case 2: __stp_put_user_asm_half(value, addr, _bad); break;	\
+	case 4: __stp_put_user_asm_word(value, addr, _bad); break;	\
+	case 8: __stp_put_user_asm_dword(value, addr, _bad); break;	\
 	default: __put_user_bad(); break;				\
 	}								\
     if (_bad)								\
Index: systemtap-20070602/runtime/regs.c
===================================================================
--- systemtap-20070602/runtime/regs.c	(.../vendor/usr/src)	(revision 214)
+++ systemtap-20070602/runtime/regs.c	(.../branches/kprobes/usr/src)	(revision 214)
@@ -47,6 +47,8 @@ unsigned long _stp_ret_addr (struct pt_r
 	return regs->b0;
 #elif defined (__s390__) || defined (__s390x__)
 	return regs->gprs[14];
+#elif defined (__arm__)
+	return regs->ARM_r0;
 #else
 	#error Unimplemented architecture
 #endif
Index: systemtap-20070602/tapset/errno.stp
===================================================================
--- systemtap-20070602/tapset/errno.stp	(.../vendor/usr/src)	(revision 214)
+++ systemtap-20070602/tapset/errno.stp	(.../branches/kprobes/usr/src)	(revision 214)
@@ -370,6 +370,8 @@ function returnstr:string (returnp:long)
 		ret = CONTEXT->regs->u_regs[UREG_RETPC];
 #elif defined (__s390x__)
 		ret = CONTEXT->regs->gprs[2];
+#elif defined (__arm__)
+		ret = CONTEXT->regs->ARM_r0;
 #else
 		goto no_ret;
 #endif

Re: systemtap ARM port status [PATCH]

[Quentin Barnes]

I've finished analyzing the remaining testsuite failures on ARM.

This posting are the failures that required code changes, either
to the runtime or to the test itself to fix.  The other failures
are more complicated to address.  I'll write them up in a later
mail.

The first patch below fixes the "buildok/twenty.stp" and
"buildok/twentythree.stp" failures.  I had forgotten the
__stp_put_user_asm_dword() function.  (Note: this patch is
on top of the patch I sent out on June 5.)

The second patch below fixes the "prologues -P", "prologues no-P".
More problems with expect scripts missing "-timout" parameters.

Quentin


Index: systemtap-20070602/runtime/loc2c-runtime.h
===================================================================
--- systemtap-20070602/runtime/loc2c-runtime.h	(revision 218)
+++ systemtap-20070602/runtime/loc2c-runtime.h	(working copy)
@@ -407,6 +407,33 @@
 	: "r" (x), "r" (__pu_addr), "i" (-EFAULT)		\
 	: "cc")
 
+#ifndef __ARMEB__
+#define	__reg_oper0	"%R2"
+#define	__reg_oper1	"%Q2"
+#else
+#define	__reg_oper0	"%Q2"
+#define	__reg_oper1	"%R2"
+#endif
+
+#define __stp_put_user_asm_dword(x,__pu_addr,err)		\
+	__asm__ __volatile__(					\
+	"1:	str	" __reg_oper1 ", [%1], #4\n"		\
+	"2:	str	" __reg_oper0 ", [%1], #0\n"		\
+	"3:\n"							\
+	"	.section .fixup,\"ax\"\n"			\
+	"	.align	2\n"					\
+	"4:	mov	%0, %3\n"				\
+	"	b	3b\n"					\
+	"	.previous\n"					\
+	"	.section __ex_table,\"a\"\n"			\
+	"	.align	3\n"					\
+	"	.long	1b, 4b\n"				\
+	"	.long	2b, 4b\n"				\
+	"	.previous"					\
+	: "+r" (err), "+r" (__pu_addr)				\
+	: "r" (x), "i" (-EFAULT)				\
+	: "cc")
+
 #define deref(size, addr)						\
   ({									\
      int _bad = 0;							\

Index: systemtap-20070602/testsuite/systemtap.base/prologues.exp
===================================================================
--- systemtap-20070602/testsuite/systemtap.base/prologues.exp	(revision 219)
+++ systemtap-20070602/testsuite/systemtap.base/prologues.exp	(working copy)
@@ -3,8 +3,10 @@
     set ok 0
     spawn stap -P $srcdir/$subdir/prologues.stp
     expect {
+        -timeout 240
         -re {read[^\r\n]*\r\n} { incr ok; exp_continue }
         -re {write[^\r\n]*\r\n} { incr ok; exp_continue }
+        timeout { fail "$test (timeout)" }
         eof
     }
     wait
@@ -16,8 +18,10 @@
     set ok 0
     spawn stap $srcdir/$subdir/prologues.stp
     expect {
+        -timeout 240
         -re {read[^\r\n]*\r\n} { incr ok; exp_continue }
         -re {write[^\r\n]*\r\n} { incr ok; exp_continue }
+        timeout { fail "$test (timeout)" }
         eof
     }
     wait

Re: systemtap ARM port status

[Eugene Teo]

Quentin Barnes wrote:
[...]
>> 1) On host: stap -p 4 -k script.stap
>> 2) Copy generated .ko to target
>> 3) On target: insmod stap-module.ko
> 
> Well, for now I'm running completely native.  Yes, that's very, very
> painful to run gcc and g++ on such a board with an NFS mounted file
> system and NFS mounted swap file over loopback.  It can take four
> hours of wall time to compile and link the "stap" binary and the
> better part of a day to run the full testsuite.

Yes, it's very painful. I didn't run it completely native though. I ran
it in scratchbox, but it keeps on complaining that it is unable to find
the kernel debuginfo, even though it's there. I noticed that strace on
arm eabi seems to be buggy, and that didn't help. A simple hello world
stp script worked though. That is, step 1 and 2 (above), and step 3 using
staprun instead of insmod.

> I've had so many problems to solve getting Systemtap to support
> ARM, I didn't want to have to also deal with problems introduced
> from trying to get it to also support cross-builds simultaneously.
> I decided to work on one problem at a time.

Did you install it to /usr/local? I am wondering if you can tar it up,
and pass me a copy, so that I don't have to download compilers, etc onto
the device to install systemtap natively..

Eugene