Regarding systemtap support for AArch64

Regarding systemtap support for AArch64

[Sandeepa Prabhu]

Hi,

At Linaro, we are developing support for kprobes (and uprobes) for ARM
v8 platform, and interested in running systemtap for validating our
work. I wanted to check if the current version of  systemtap support
AArch64? Is there a systemtap version I can use to verify kprobes
mechanism?

Thanks,
Sandeepa

Re: Regarding systemtap support for AArch64

[Mark Wielaard]

On Tue, 2013-09-24 at 08:43 +0530, Sandeepa Prabhu wrote:
> At Linaro, we are developing support for kprobes (and uprobes) for ARM
> v8 platform, and interested in running systemtap for validating our
> work. I wanted to check if the current version of  systemtap support
> AArch64? Is there a systemtap version I can use to verify kprobes
> mechanism?

There is a fedora bug tracking support work items:
https://bugzilla.redhat.com/show_bug.cgi?id=926602
For elfutils you might want to try the pmachata/aarch64 in upstream git.
https://git.fedorahosted.org/cgit/elfutils.git/log/?h=pmachata/aarch64
It isn't fully ready yet, but I am sure Petr would like some extra
testing help.

Cheers,

Mark

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 24 September 2013 13:57, Mark Wielaard <mjw@redhat.com> wrote:
> On Tue, 2013-09-24 at 08:43 +0530, Sandeepa Prabhu wrote:
>> At Linaro, we are developing support for kprobes (and uprobes) for ARM
>> v8 platform, and interested in running systemtap for validating our
>> work. I wanted to check if the current version of  systemtap support
>> AArch64? Is there a systemtap version I can use to verify kprobes
>> mechanism?
>
> There is a fedora bug tracking support work items:
> https://bugzilla.redhat.com/show_bug.cgi?id=926602
> For elfutils you might want to try the pmachata/aarch64 in upstream git.
> https://git.fedorahosted.org/cgit/elfutils.git/log/?h=pmachata/aarch64
> It isn't fully ready yet, but I am sure Petr would like some extra
> testing help.
Thanks, So to start with, I will try to build systemtap for aarch64
with above combination and let know how does it proceed.

Cheers,
Sandeepa

>
> Cheers,
>
> Mark
>

Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

Hi Sandeepa,

I'm also interested in your work. Would you work on some public repository?
And would you test it with linux/samples/kprobes/* ?

Thank you,

(2013/09/24 12:13), Sandeepa Prabhu wrote:
> Hi,
> 
> At Linaro, we are developing support for kprobes (and uprobes) for ARM
> v8 platform, and interested in running systemtap for validating our
> work. I wanted to check if the current version of  systemtap support
> AArch64? Is there a systemtap version I can use to verify kprobes
> mechanism?
> 
> Thanks,
> Sandeepa
> 


-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Regarding systemtap support for AArch64

[William Cohen]

On 09/24/2013 04:27 AM, Mark Wielaard wrote:
> On Tue, 2013-09-24 at 08:43 +0530, Sandeepa Prabhu wrote:
>> At Linaro, we are developing support for kprobes (and uprobes) for ARM
>> v8 platform, and interested in running systemtap for validating our
>> work. I wanted to check if the current version of  systemtap support
>> AArch64? Is there a systemtap version I can use to verify kprobes
>> mechanism?
> 
> There is a fedora bug tracking support work items:
> https://bugzilla.redhat.com/show_bug.cgi?id=926602
> For elfutils you might want to try the pmachata/aarch64 in upstream git.
> https://git.fedorahosted.org/cgit/elfutils.git/log/?h=pmachata/aarch64
> It isn't fully ready yet, but I am sure Petr would like some extra
> testing help.
> 
> Cheers,
> 
> Mark
> 

Hi,

My aarch64 simulator is running an old 3.9+ kernel and I don't have all the other kernel-devel related stuff to instrument that kernel. As a work around I have been using a 3.12-rc0 rpm.  Also trying to avoid needing the elfutils at the moment, so I am running some thing like the following command to see what breaks for a really simple "hello world" script:

sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x1.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'

The software simulator is really slow and it looks like the testsuite is probably going to time out for the tests.

-Will

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

Hi Will, Masami,

Nice to hear from you, I am using ARM fast model/Foundation model with
ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
testing with loadable modules for now (so don't have dependency on
elfutils or GCC autoconf etc)

Would you be interested to use Linaro kprobe work as a base for
development and validation of systemtap-aarch64?  We are happy to
share a public git repo where 'upstream' kernel can be built with
kprobes support, which systemtap team can use for verification.  I can
do this soon as I have most things working locally, except for
kretprobes and 'boosting' support(systemtap can be run without these I
believe).

Looking forward for a collaboration :-)

Cheers,
Sandeepa

On 26 September 2013 00:15, William Cohen <wcohen@redhat.com> wrote:
> On 09/24/2013 04:27 AM, Mark Wielaard wrote:
>> On Tue, 2013-09-24 at 08:43 +0530, Sandeepa Prabhu wrote:
>>> At Linaro, we are developing support for kprobes (and uprobes) for ARM
>>> v8 platform, and interested in running systemtap for validating our
>>> work. I wanted to check if the current version of  systemtap support
>>> AArch64? Is there a systemtap version I can use to verify kprobes
>>> mechanism?
>>
>> There is a fedora bug tracking support work items:
>> https://bugzilla.redhat.com/show_bug.cgi?id=926602
>> For elfutils you might want to try the pmachata/aarch64 in upstream git.
>> https://git.fedorahosted.org/cgit/elfutils.git/log/?h=pmachata/aarch64
>> It isn't fully ready yet, but I am sure Petr would like some extra
>> testing help.
>>
>> Cheers,
>>
>> Mark
>>
>
> Hi,
>
> My aarch64 simulator is running an old 3.9+ kernel and I don't have all the other kernel-devel related stuff to instrument that kernel. As a work around I have been using a 3.12-rc0 rpm.  Also trying to avoid needing the elfutils at the moment, so I am running some thing like the following command to see what breaks for a really simple "hello world" script:
>
> sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x1.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
>
> The software simulator is really slow and it looks like the testsuite is probably going to time out for the tests.
>
> -Will

Re: Regarding systemtap support for AArch64

[William Cohen]

On 09/25/2013 11:13 PM, Sandeepa Prabhu wrote:
> Hi Will, Masami,
> 
> Nice to hear from you, I am using ARM fast model/Foundation model with
> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
> testing with loadable modules for now (so don't have dependency on
> elfutils or GCC autoconf etc)
> 
> Would you be interested to use Linaro kprobe work as a base for
> development and validation of systemtap-aarch64?  We are happy to
> share a public git repo where 'upstream' kernel can be built with
> kprobes support, which systemtap team can use for verification.  I can
> do this soon as I have most things working locally, except for
> kretprobes and 'boosting' support(systemtap can be run without these I
> believe).
> 
> Looking forward for a collaboration :-)
> 
> Cheers,
> Sandeepa

Hi Sandeepa,

It is very nice to hear that Linaro has been working on aarch64 kprobes.  Is there a git repository or kernel available that has the aarch64 patches?  Also is there a recomended config file and instructions on how to get a new kernel running on the simulator?

We would like to get systemtap working on aarch64.  There are a number architecture specific things that need to be adjusted in systemtap to support aarch64.  A few of the changes have been pushed into the systemtap git repo, but more are needed in the runtime and some more in the tapset.

Locally I have a ARM fast model/Foundation model set up with Fedora 19.  It can build systemtap from source, but it is pretty slow. Running the systemtap testsuite will probably require access to actual hardware. Otherwise tests will timeout and just fail. 

-Will


> 
> On 26 September 2013 00:15, William Cohen <wcohen@redhat.com> wrote:
>> On 09/24/2013 04:27 AM, Mark Wielaard wrote:
>>> On Tue, 2013-09-24 at 08:43 +0530, Sandeepa Prabhu wrote:
>>>> At Linaro, we are developing support for kprobes (and uprobes) for ARM
>>>> v8 platform, and interested in running systemtap for validating our
>>>> work. I wanted to check if the current version of  systemtap support
>>>> AArch64? Is there a systemtap version I can use to verify kprobes
>>>> mechanism?
>>>
>>> There is a fedora bug tracking support work items:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=926602
>>> For elfutils you might want to try the pmachata/aarch64 in upstream git.
>>> https://git.fedorahosted.org/cgit/elfutils.git/log/?h=pmachata/aarch64
>>> It isn't fully ready yet, but I am sure Petr would like some extra
>>> testing help.
>>>
>>> Cheers,
>>>
>>> Mark
>>>
>>
>> Hi,
>>
>> My aarch64 simulator is running an old 3.9+ kernel and I don't have all the other kernel-devel related stuff to instrument that kernel. As a work around I have been using a 3.12-rc0 rpm.  Also trying to avoid needing the elfutils at the moment, so I am running some thing like the following command to see what breaks for a really simple "hello world" script:
>>
>> sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x1.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
>>
>> The software simulator is really slow and it looks like the testsuite is probably going to time out for the tests.
>>
>> -Will

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 26 September 2013 20:05, William Cohen <wcohen@redhat.com> wrote:
> On 09/25/2013 11:13 PM, Sandeepa Prabhu wrote:
>> Hi Will, Masami,
>>
>> Nice to hear from you, I am using ARM fast model/Foundation model with
>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>> testing with loadable modules for now (so don't have dependency on
>> elfutils or GCC autoconf etc)
>>
>> Would you be interested to use Linaro kprobe work as a base for
>> development and validation of systemtap-aarch64?  We are happy to
>> share a public git repo where 'upstream' kernel can be built with
>> kprobes support, which systemtap team can use for verification.  I can
>> do this soon as I have most things working locally, except for
>> kretprobes and 'boosting' support(systemtap can be run without these I
>> believe).
>>
>> Looking forward for a collaboration :-)
>>
>> Cheers,
>> Sandeepa
>
> Hi Sandeepa,
>
> It is very nice to hear that Linaro has been working on aarch64 kprobes.  Is there a git repository or kernel available that has the aarch64 patches?  Also is there a recommend config file and instructions on how to get a new kernel running on the simulator?
>
> We would like to get systemtap working on aarch64.  There are a number architecture specific things that need to be adjusted in systemtap to support aarch64.  A few of the changes have been pushed into the systemtap git repo, but more are needed in the runtime and some more in the tapset.
>
> Locally I have a ARM fast model/Foundation model set up with Fedora 19.  It can build systemtap from source, but it is pretty slow. Running the systemtap testsuite will probably require access to actual hardware. Otherwise tests will timeout and just fail.

Hi Will,

I would create a linaro git repository and upload all the patches and
config file that are needed to build the kernel with kprobes support,
doing it sometimes early next week and share it on systemtap mailing
list.

As I said, I am using minimal setup (busybox based) and using only
sample kernel modules to verify, have not used sophisticated rootfs
like fedora or ubuntu on the models since any GUI will be run very
slow.  Alternately, do you consider using lightweight command-line
system like LAMP stack (available with linaro public builds) and
cross-compile the systemtap and dependent tools on open-embedded
environment. I understand that minimal system is far away from real
environment where systemtap will be used (like fedora, enterprise
servers) but it should serve as temporary setup until the real
hardware is available.

Thanks,
Sandeepa

>
> -Will
>
>
>>
>> On 26 September 2013 00:15, William Cohen <wcohen@redhat.com> wrote:
>>> On 09/24/2013 04:27 AM, Mark Wielaard wrote:
>>>> On Tue, 2013-09-24 at 08:43 +0530, Sandeepa Prabhu wrote:
>>>>> At Linaro, we are developing support for kprobes (and uprobes) for ARM
>>>>> v8 platform, and interested in running systemtap for validating our
>>>>> work. I wanted to check if the current version of  systemtap support
>>>>> AArch64? Is there a systemtap version I can use to verify kprobes
>>>>> mechanism?
>>>>
>>>> There is a fedora bug tracking support work items:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=926602
>>>> For elfutils you might want to try the pmachata/aarch64 in upstream git.
>>>> https://git.fedorahosted.org/cgit/elfutils.git/log/?h=pmachata/aarch64
>>>> It isn't fully ready yet, but I am sure Petr would like some extra
>>>> testing help.
>>>>
>>>> Cheers,
>>>>
>>>> Mark
>>>>
>>>
>>> Hi,
>>>
>>> My aarch64 simulator is running an old 3.9+ kernel and I don't have all the other kernel-devel related stuff to instrument that kernel. As a work around I have been using a 3.12-rc0 rpm.  Also trying to avoid needing the elfutils at the moment, so I am running some thing like the following command to see what breaks for a really simple "hello world" script:
>>>
>>> sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x1.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
>>>
>>> The software simulator is really slow and it looks like the testsuite is probably going to time out for the tests.
>>>
>>> -Will
>

Re: Regarding systemtap support for AArch64

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 3984 bytes --]

On 09/26/2013 10:57 AM, Sandeepa Prabhu wrote:
> On 26 September 2013 20:05, William Cohen <wcohen@redhat.com> wrote:
>> On 09/25/2013 11:13 PM, Sandeepa Prabhu wrote:
>>> Hi Will, Masami,
>>>
>>> Nice to hear from you, I am using ARM fast model/Foundation model with
>>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>>> testing with loadable modules for now (so don't have dependency on
>>> elfutils or GCC autoconf etc)
>>>
>>> Would you be interested to use Linaro kprobe work as a base for
>>> development and validation of systemtap-aarch64?  We are happy to
>>> share a public git repo where 'upstream' kernel can be built with
>>> kprobes support, which systemtap team can use for verification.  I can
>>> do this soon as I have most things working locally, except for
>>> kretprobes and 'boosting' support(systemtap can be run without these I
>>> believe).
>>>
>>> Looking forward for a collaboration :-)
>>>
>>> Cheers,
>>> Sandeepa
>>
>> Hi Sandeepa,
>>
>> It is very nice to hear that Linaro has been working on aarch64 kprobes.  Is there a git repository or kernel available that has the aarch64 patches?  Also is there a recommend config file and instructions on how to get a new kernel running on the simulator?
>>
>> We would like to get systemtap working on aarch64.  There are a number architecture specific things that need to be adjusted in systemtap to support aarch64.  A few of the changes have been pushed into the systemtap git repo, but more are needed in the runtime and some more in the tapset.
>>
>> Locally I have a ARM fast model/Foundation model set up with Fedora 19.  It can build systemtap from source, but it is pretty slow. Running the systemtap testsuite will probably require access to actual hardware. Otherwise tests will timeout and just fail.
> 
> Hi Will,
> 
> I would create a linaro git repository and upload all the patches and
> config file that are needed to build the kernel with kprobes support,
> doing it sometimes early next week and share it on systemtap mailing
> list.

Hi Sandeepa,

I went though and made a minimal patch (aarch64_changes.patch) that allows systemtap to translate a simple hello world script into c code.  The kernel on my software armv8 simulator is a cross compiled version and some of the scripts programs in there don't work because they are ARM-64 executable so it failed when doing the actual compile of module:  

sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x2.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
Pass 1: parsed user script and 92 library script(s) using 38784virt/30016res/550
4shr/23360data kb, in 4700usr/90sys/4805real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) usin
g 39296virt/31936res/5952shr/23872data kb, in 90usr/0sys/81real ms.
Pass 3: translated to C into "/tmp/stapy7iaug/hello_src.c" using 39296virt/31936
res/5952shr/23872data kb, in 0usr/0sys/7real ms.
/bin/sh: scripts/basic/fixdep: cannot execute binary file
make[1]: *** [/tmp/stapy7iaug/hello_src.o] Error 126
make: *** [_module_/tmp/stapy7iaug] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "hello.ko" in 110730usr/3270sys/115966real ms.
Pass 4: compilation failed.  [man error::pass4]
Keeping temporary directory "/tmp/stapy7iaug"

-Will

> As I said, I am using minimal setup (busybox based) and using only
> sample kernel modules to verify, have not used sophisticated rootfs
> like fedora or ubuntu on the models since any GUI will be run very
> slow.  Alternately, do you consider using lightweight command-line
> system like LAMP stack (available with linaro public builds) and
> cross-compile the systemtap and dependent tools on open-embedded
> environment. I understand that minimal system is far away from real
> environment where systemtap will be used (like fedora, enterprise
> servers) but it should serve as temporary setup until the real
> hardware is available.
> 
> Thanks,
> Sandeepa


[-- Attachment #2: aarch64_changes.patch --]
[-- Type: text/x-patch, Size: 6248 bytes --]

diff --git a/runtime/linux/copy.c b/runtime/linux/copy.c
index fc730d2..2bdb52a 100644
--- a/runtime/linux/copy.c
+++ b/runtime/linux/copy.c
@@ -24,9 +24,6 @@
  * @{
  */
 
-
-static long __stp_strncpy_from_user(char *dst, const char __user *src, long count);
-
 #ifdef CONFIG_GENERIC_STRNCPY_FROM_USER
 #define __stp_strncpy_from_user(dst,src,count,res) \
 	do { res = strncpy_from_user(dst, src, count); } while(0)
@@ -91,7 +88,7 @@ do {									   \
 #define __stp_strncpy_from_user(dst,src,count,res) \
 	do { res = __strncpy_from_user(dst, src, count); } while(0)
 
-#elif defined (__s390__) || defined (__s390x__)
+#elif defined (__s390__) || defined (__s390x__)|| defined (__aarch64__)
 #define __stp_strncpy_from_user(dst,src,count,res) \
 	do { res = strncpy_from_user(dst, src, count); } while(0)
 #elif defined (__ia64__)
diff --git a/runtime/linux/loc2c-runtime.h b/runtime/linux/loc2c-runtime.h
index e01b7f4..060b8c3 100644
--- a/runtime/linux/loc2c-runtime.h
+++ b/runtime/linux/loc2c-runtime.h
@@ -563,6 +563,56 @@ extern void __store_deref_bad(void);
       STORE_DEREF_FAULT(addr);						      \
   })
 
+#elif defined (__aarch64__)
+
+#define _stp_deref(size, addr, seg)                                           \
+  ({									      \
+    int _bad = 0;							      \
+    intptr_t _v = 0;							      \
+    mm_segment_t _oldfs = get_fs();                                           \
+    set_fs(seg);                                                              \
+    pagefault_disable();                                                      \
+    if (lookup_bad_addr((unsigned long)addr, size))			      \
+      _bad = 1;                                                               \
+    else                                                                      \
+      switch (size)                                                           \
+        {                                                                     \
+	case 1: __get_user_asm("ldrb", "%w", _v, (unsigned long)addr, _bad); break;\
+	case 2: __get_user_asm("ldrh", "%w",_v, (unsigned long)addr, _bad); break;\
+	case 4: __get_user_asm("ldr", "%w",_v,  (unsigned long)addr, _bad); break;\
+	case 8: __get_user_asm("ldr", "%",_v,  (unsigned long)addr, _bad); break;\
+        default: BUILD_BUG();			                              \
+        }                                                                     \
+    pagefault_enable();                                                       \
+    set_fs(_oldfs);                                                           \
+    if (_bad)								      \
+      DEREF_FAULT(addr);						      \
+    _v;									      \
+  })
+
+#define _stp_store_deref(size, addr, value, seg)                              \
+  ({									      \
+    int _bad = 0;							      \
+    mm_segment_t _oldfs = get_fs();                                           \
+    set_fs(seg);                                                              \
+    pagefault_disable();                                                      \
+    if (lookup_bad_addr((unsigned long)addr, size))			      \
+      _bad = 1;                                                               \
+    else                                                                      \
+      switch (size)                                                           \
+        {                                                                     \
+	case 1: __put_user_asm("strb", "%w", ((u8)(value)), addr, _bad); break;\
+	case 2: __put_user_asm("strh", "%w", ((u16)(value)), addr, _bad); break;\
+	case 4: __put_user_asm("str", "%w", ((u32)(value)), addr, _bad); break;\
+	case 8: __put_user_asm("str", "%", ((u64)(value)), addr, _bad); break;\
+        default: BUILD_BUG();                                                 \
+        }                                                                     \
+    pagefault_enable();                                                       \
+    set_fs(_oldfs);                                                           \
+    if (_bad)								      \
+      STORE_DEREF_FAULT(addr);						      \
+  })
+
 #elif defined (__arm__)
 
 /* Macros for ARM lifted from 2.6.21.1's linux/include/asm-arm/uaccess.h
diff --git a/runtime/loc2c-runtime.h b/runtime/loc2c-runtime.h
index ca9ad11..bd13341 100644
--- a/runtime/loc2c-runtime.h
+++ b/runtime/loc2c-runtime.h
@@ -154,6 +154,15 @@
 #define pt_regs_store_register(pt_regs,regno,value) \
   (pt_regs->gpr[regno] = (value))
 
+#elif defined (__aarch64__)
+
+#undef pt_regs_fetch_register
+#undef pt_regs_store_register
+#define pt_regs_fetch_register(pt_regs,regno) \
+  ((long) pt_regs->regs[regno])
+#define pt_regs_store_register(pt_regs,regno,value) \
+  (pt_regs->regs[regno] = (value))
+
 #elif defined (__arm__)
 
 #undef pt_regs_fetch_register
diff --git a/runtime/regs.h b/runtime/regs.h
index dbe41ab..2f027ad 100644
--- a/runtime/regs.h
+++ b/runtime/regs.h
@@ -42,6 +42,12 @@
 #define REG_SP(regs) regs->gpr[1]
 #define REG_LINK(regs) regs->link
 
+#elif defined (__aarch64__)
+
+#define REG_IP(regs) regs->pc
+#define REG_SP(regs) regs->sp
+#define REG_LINK(regs) regs->regs[30]
+
 #elif defined (__arm__)
 
 #define REG_IP(regs) regs->ARM_pc
diff --git a/runtime/time.c b/runtime/time.c
index 068db0a..7ae07af 100644
--- a/runtime/time.c
+++ b/runtime/time.c
@@ -75,7 +75,7 @@ __stp_get_freq(void)
     // If we can get the actual frequency of HW counter, we use it.
 #if defined (__ia64__)
     return local_cpu_data->itc_freq / 1000;
-#elif defined (__s390__) || defined (__s390x__) || defined (__arm__)
+#elif defined (__s390__) || defined (__s390x__) || defined (__arm__) || defined (__aarch64__)
     // We don't need to find the cpu freq on s390 as the 
     // TOD clock is always a fix freq. (see: POO pg 4-36.)
     return 0;
@@ -83,7 +83,7 @@ __stp_get_freq(void)
     return tsc_khz;
 #elif (defined (__i386__) || defined (__x86_64__)) && defined(STAPCONF_CPU_KHZ)
     return cpu_khz;
-#else /* __i386__ || __x86_64__ || __ia64__ || __s390__ || __s390x__ || __arm__*/
+#else /* __i386__ || __x86_64__ */
     // If we don't know the actual frequency, we estimate it.
     cycles_t beg, mid, end;
     beg = get_cycles(); barrier();

Re: Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/09/26 12:13), Sandeepa Prabhu wrote:
> Hi Will, Masami,
> 
> Nice to hear from you, I am using ARM fast model/Foundation model with
> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
> testing with loadable modules for now (so don't have dependency on
> elfutils or GCC autoconf etc)
> 
> Would you be interested to use Linaro kprobe work as a base for
> development and validation of systemtap-aarch64?  We are happy to
> share a public git repo where 'upstream' kernel can be built with
> kprobes support, which systemtap team can use for verification.  I can
> do this soon as I have most things working locally, except for
> kretprobes and 'boosting' support(systemtap can be run without these I
> believe).

Of course I'm interested in that. Actually I've tried to boot up Linaro's
Aarch64 kernel on a simulator which ARM is distributing.
As far as I could see (at that time), aarch64 branch already has singlestep
implementation for debugging, but no primitive kprobes. And we need to know
the aarch64 ISA for decoding the binary to find the instructions which will
be affected by the out-of-line execution, like relative jump, call etc.

> Looking forward for a collaboration :-)

Me too ;)

Thanks!
-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 30 September 2013 08:06, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> (2013/09/26 12:13), Sandeepa Prabhu wrote:
>> Hi Will, Masami,
>>
>> Nice to hear from you, I am using ARM fast model/Foundation model with
>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>> testing with loadable modules for now (so don't have dependency on
>> elfutils or GCC autoconf etc)
>>
>> Would you be interested to use Linaro kprobe work as a base for
>> development and validation of systemtap-aarch64?  We are happy to
>> share a public git repo where 'upstream' kernel can be built with
>> kprobes support, which systemtap team can use for verification.  I can
>> do this soon as I have most things working locally, except for
>> kretprobes and 'boosting' support(systemtap can be run without these I
>> believe).
>
> Of course I'm interested in that. Actually I've tried to boot up Linaro's
> Aarch64 kernel on a simulator which ARM is distributing.
> As far as I could see (at that time), aarch64 branch already has singlestep
> implementation for debugging, but no primitive kprobes. And we need to know
> the aarch64 ISA for decoding the binary to find the instructions which will
> be affected by the out-of-line execution, like relative jump, call etc.
Hi Masami,

I am almost done with my first set, will upload and expose my
development branch in couple of days (on linaro public git server) and
then share the info, alongside publishing RFC/RFT on mailing lists.
As for v8 ISA, the public version of ARM ARM is out couple of weeks
back, on silver arm:
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0487a/index.html.
 [Although, have developed basic support for handling out-of-line
single stepping as well as simulating literal access & branch, there
are few instructions types that are not supported in kprobes right
now, like exclusive monito, FP/AdvSIMD load/store from symbolic
litreals]

As long as if interested in using sample loadable modules for running,
you would only need a busybox setup, but for systemtap or other tools,
you need a more appropriate file system like fc19 or a open-embedded.

Cheers!
Sandeepa

>
>> Looking forward for a collaboration :-)
>
> Me too ;)
>
> Thanks!
> --
> Masami HIRAMATSU
> IT Management Research Dept. Linux Technology Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>
>

Re: Regarding systemtap support for AArch64

[William Cohen]

On 09/29/2013 10:36 PM, Masami Hiramatsu wrote:
> (2013/09/26 12:13), Sandeepa Prabhu wrote:
>> Hi Will, Masami,
>>
>> Nice to hear from you, I am using ARM fast model/Foundation model with
>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>> testing with loadable modules for now (so don't have dependency on
>> elfutils or GCC autoconf etc)
>>
>> Would you be interested to use Linaro kprobe work as a base for
>> development and validation of systemtap-aarch64?  We are happy to
>> share a public git repo where 'upstream' kernel can be built with
>> kprobes support, which systemtap team can use for verification.  I can
>> do this soon as I have most things working locally, except for
>> kretprobes and 'boosting' support(systemtap can be run without these I
>> believe).
> 
> Of course I'm interested in that. Actually I've tried to boot up Linaro's
> Aarch64 kernel on a simulator which ARM is distributing.
> As far as I could see (at that time), aarch64 branch already has singlestep
> implementation for debugging, but no primitive kprobes. And we need to know
> the aarch64 ISA for decoding the binary to find the instructions which will
> be affected by the out-of-line execution, like relative jump, call etc.
> 
>> Looking forward for a collaboration :-)
> 
> Me too ;)
> 
> Thanks!
> 

Hi All,

I am still trying to get a better kernel running on the aarch64 simulator.  Currently the simulator is running and Fedora 19 image and Linux 3.9+ kernel, but I don't have any of the needed kernel-devel stuff for that kernel. As a work around I am using the aarch64 kernel rpms and attempting to build systemtap instrumentation modules using the following rpms:

kernel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
kernel-devel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64

These kernel rpms are cross compiled so some script executables are for AMD64 rather than aarch64.  I had to go through and recompile the script executables.  However, I was able to get an instrumentation modules with the current git checkout for simple hello world:

$ sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x2.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'

Pass 1: parsed user script and 92 library script(s) using 38784virt/30080res/5504shr/23360data kb, in 4730usr/30sys/4763real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 39296virt/32000res/5952shr/23872data kb, in 80usr/0sys/81real ms.
Pass 3: translated to C into "/tmp/stapzSdKrS/hello_src.c" using 39296virt/32000res/5952shr/23872data kb, in 10usr/0sys/7real ms.
hello.ko
Pass 4: compiled C into "hello.ko" in 114710usr/3620sys/120572real ms.
Keeping temporary directory "/tmp/stapzSdKrS"

There are still other things that need to be fixed in systemtap for aarch64.  However, if you are feeling adventurous you might try the current systemtap git checkout on aarch64.

-Will

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

Hi all,

I have uploaded ARM64 kprobes work on Linaro public git:
git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
kprobes_devel_v8.  Patches are published on LAKML too.  This is based
on v8 upstream kernel (3.12-rc1) right now, and works with linaro
boot-wrapper and fast model setup, though, not sure what it takes to
build for fedora.

Will,

Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
model, are there instructions about how to get the packages and
build/run them?

Thanks,
Sandeepa



On 30 September 2013 17:40, William Cohen <wcohen@redhat.com> wrote:
> On 09/29/2013 10:36 PM, Masami Hiramatsu wrote:
>> (2013/09/26 12:13), Sandeepa Prabhu wrote:
>>> Hi Will, Masami,
>>>
>>> Nice to hear from you, I am using ARM fast model/Foundation model with
>>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>>> testing with loadable modules for now (so don't have dependency on
>>> elfutils or GCC autoconf etc)
>>>
>>> Would you be interested to use Linaro kprobe work as a base for
>>> development and validation of systemtap-aarch64?  We are happy to
>>> share a public git repo where 'upstream' kernel can be built with
>>> kprobes support, which systemtap team can use for verification.  I can
>>> do this soon as I have most things working locally, except for
>>> kretprobes and 'boosting' support(systemtap can be run without these I
>>> believe).
>>
>> Of course I'm interested in that. Actually I've tried to boot up Linaro's
>> Aarch64 kernel on a simulator which ARM is distributing.
>> As far as I could see (at that time), aarch64 branch already has singlestep
>> implementation for debugging, but no primitive kprobes. And we need to know
>> the aarch64 ISA for decoding the binary to find the instructions which will
>> be affected by the out-of-line execution, like relative jump, call etc.
>>
>>> Looking forward for a collaboration :-)
>>
>> Me too ;)
>>
>> Thanks!
>>
>
> Hi All,
>
> I am still trying to get a better kernel running on the aarch64 simulator.  Currently the simulator is running and Fedora 19 image and Linux 3.9+ kernel, but I don't have any of the needed kernel-devel stuff for that kernel. As a work around I am using the aarch64 kernel rpms and attempting to build systemtap instrumentation modules using the following rpms:
>
> kernel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
> kernel-devel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
>
> These kernel rpms are cross compiled so some script executables are for AMD64 rather than aarch64.  I had to go through and recompile the script executables.  However, I was able to get an instrumentation modules with the current git checkout for simple hello world:
>
> $ sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x2.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
>
> Pass 1: parsed user script and 92 library script(s) using 38784virt/30080res/5504shr/23360data kb, in 4730usr/30sys/4763real ms.
> Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 39296virt/32000res/5952shr/23872data kb, in 80usr/0sys/81real ms.
> Pass 3: translated to C into "/tmp/stapzSdKrS/hello_src.c" using 39296virt/32000res/5952shr/23872data kb, in 10usr/0sys/7real ms.
> hello.ko
> Pass 4: compiled C into "hello.ko" in 114710usr/3620sys/120572real ms.
> Keeping temporary directory "/tmp/stapzSdKrS"
>
> There are still other things that need to be fixed in systemtap for aarch64.  However, if you are feeling adventurous you might try the current systemtap git checkout on aarch64.
>
> -Will

Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/10/02 13:17), Sandeepa Prabhu wrote:
> Hi all,
> 
> I have uploaded ARM64 kprobes work on Linaro public git:
> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
> boot-wrapper and fast model setup, though, not sure what it takes to
> build for fedora.

Thank you for the great work! :)

At a glance, I have some comments (almost formatting issues) on it.
 - kprobes def config patch should be moved after the
   kprobe support.
 - Is it really need to use spinlock to protect break_hook?
   Why can't we use rcu_lock as step_hook?
 - patch_text should check the alignment in itself and return an error.
 - For ease of bisect debugging, I'd recommend you to split decoder and
   simulator, and to reorder it as below.
1. decoder patch
2. kprobes basic patch (this doesn't support insns which need simulator)
3. simulator & kprobes enhancement patch
 - probes-*.c is not good name for the simulator. those should have
   better name.
 - kprobes-arm64.c is also not good name. I think we can have
   arch/arm64/kernel/kprobes/ directory on arm64 as same as x86.
   we may have core.c and insn_map.c.
 - probes-common.c looks only for kprobes. This should be go into kprobes.c.
 - It seems that a part of kretprobe logic is in the kprobes' patch.

IMHO, the simulator/decoder will be useful for other cases (KVM, mmiotrace etc.)
So I recommend you to keep it away from kprobes as far as possible.

And please CC the series to me next time
(since I don't subscribe LAKML...).

Thank you again!


> 
> Will,
> 
> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
> model, are there instructions about how to get the packages and
> build/run them?
> 
> Thanks,
> Sandeepa
> 
> 
> 
> On 30 September 2013 17:40, William Cohen <wcohen@redhat.com> wrote:
>> On 09/29/2013 10:36 PM, Masami Hiramatsu wrote:
>>> (2013/09/26 12:13), Sandeepa Prabhu wrote:
>>>> Hi Will, Masami,
>>>>
>>>> Nice to hear from you, I am using ARM fast model/Foundation model with
>>>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>>>> testing with loadable modules for now (so don't have dependency on
>>>> elfutils or GCC autoconf etc)
>>>>
>>>> Would you be interested to use Linaro kprobe work as a base for
>>>> development and validation of systemtap-aarch64?  We are happy to
>>>> share a public git repo where 'upstream' kernel can be built with
>>>> kprobes support, which systemtap team can use for verification.  I can
>>>> do this soon as I have most things working locally, except for
>>>> kretprobes and 'boosting' support(systemtap can be run without these I
>>>> believe).
>>>
>>> Of course I'm interested in that. Actually I've tried to boot up Linaro's
>>> Aarch64 kernel on a simulator which ARM is distributing.
>>> As far as I could see (at that time), aarch64 branch already has singlestep
>>> implementation for debugging, but no primitive kprobes. And we need to know
>>> the aarch64 ISA for decoding the binary to find the instructions which will
>>> be affected by the out-of-line execution, like relative jump, call etc.
>>>
>>>> Looking forward for a collaboration :-)
>>>
>>> Me too ;)
>>>
>>> Thanks!
>>>
>>
>> Hi All,
>>
>> I am still trying to get a better kernel running on the aarch64 simulator.  Currently the simulator is running and Fedora 19 image and Linux 3.9+ kernel, but I don't have any of the needed kernel-devel stuff for that kernel. As a work around I am using the aarch64 kernel rpms and attempting to build systemtap instrumentation modules using the following rpms:
>>
>> kernel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
>> kernel-devel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
>>
>> These kernel rpms are cross compiled so some script executables are for AMD64 rather than aarch64.  I had to go through and recompile the script executables.  However, I was able to get an instrumentation modules with the current git checkout for simple hello world:
>>
>> $ sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x2.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
>>
>> Pass 1: parsed user script and 92 library script(s) using 38784virt/30080res/5504shr/23360data kb, in 4730usr/30sys/4763real ms.
>> Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 39296virt/32000res/5952shr/23872data kb, in 80usr/0sys/81real ms.
>> Pass 3: translated to C into "/tmp/stapzSdKrS/hello_src.c" using 39296virt/32000res/5952shr/23872data kb, in 10usr/0sys/7real ms.
>> hello.ko
>> Pass 4: compiled C into "hello.ko" in 114710usr/3620sys/120572real ms.
>> Keeping temporary directory "/tmp/stapzSdKrS"
>>
>> There are still other things that need to be fixed in systemtap for aarch64.  However, if you are feeling adventurous you might try the current systemtap git checkout on aarch64.
>>
>> -Will
> 
> 


-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 2 October 2013 16:54, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> (2013/10/02 13:17), Sandeepa Prabhu wrote:
>> Hi all,
>>
>> I have uploaded ARM64 kprobes work on Linaro public git:
>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>> boot-wrapper and fast model setup, though, not sure what it takes to
>> build for fedora.
>
> Thank you for the great work! :)
>
> At a glance, I have some comments (almost formatting issues) on it.
>  - kprobes def config patch should be moved after the
>    kprobe support.
Yes ideally, but it is not part of the patchset, I only added this new
config in my repo for development testing (for sample modules), actual
changes should go in in arm/arm64/defconfig or any other configs that
can enable it.

>  - Is it really need to use spinlock to protect break_hook?
Any cpu can remove breakpoint hooks right, and traversal happen in
debug exception context so mutex are not safe (can sleep/schedule out)
in debug exception.
>    Why can't we use rcu_lock as step_hook?
Need to check, but we need protection for list traversal only (if hook
removed from another cpu)

>  - patch_text should check the alignment in itself and return an error.
Yes, perhaps kprobe also should check the alignment before handling,
if debug addr (ELR_EL1) is not 32-bit aligned, it's an error and
should be a BUG() since this it cannot be handled, can fix this in
next version.

>  - For ease of bisect debugging, I'd recommend you to split decoder and
>    simulator, and to reorder it as below.
> 1. decoder patch
> 2. kprobes basic patch (this doesn't support insns which need simulator)
> 3. simulator & kprobes enhancement patch
Good point, I will refine in next set of patches, I thought it is
difficult to split that way since we have unified decode table. One
basic question with this way of splitting,  "is it expected that  1.
and 2. (without simulate patch-3) can build and execute?"

>  - probes-*.c is not good name for the simulator. those should have
>    better name.
May be decode-arm64.c? Originally I had decode-* but the logic is
limited to kprobes and uprobes only so renamed that way.  Other cases
like jump_labels, use different decoding, and may not share same code.

>  - kprobes-arm64.c is also not good name. I think we can have
>    arch/arm64/kernel/kprobes/ directory on arm64 as same as x86.
>    we may have core.c and insn_map.c.
Hmmm, with our design uprobes should share probes-* or decode-* files,
then uprobes code also reside in arch/arm64/kernel/kprobes/ folder
itself.  Please suggest if this is fine? folder name may be
misleading.

>  - probes-common.c looks only for kprobes. This should be go into kprobes.c.
No, this file is common for kprobes 64-bit kernel, uprobes 32-bit
userspace, and uprobes 64-bit userspace, so needs to be in separate
file.

>  - It seems that a part of kretprobe logic is in the kprobes' patch.
Hmm I can refine it in next patchset, may be jprobes support too go in
as separate patch?

>
> IMHO, the simulator/decoder will be useful for other cases (KVM, mmiotrace etc.)
> So I recommend you to keep it away from kprobes as far as possible.
>
> And please CC the series to me next time
> (since I don't subscribe LAKML...).
Sorry, I did not CC you last time, I will add you from v2.

Masami,
Couple of questions:
1. My changes reside in arm/arm64 only right now, does it make sense
to publish on LKML so that core kprobes developers can review?
2. Wanted to change sample/kprobes/kprobe_example.c to check for the
missing case of ARM & ARM64 and print the relevant info. Can this
change be as part of same series (if going on LKML)?

Thanks a lot for your review, I would also need help on validation for
my work, please let me know if our repo holds good for systemtap
validation. I am interested in using fedora on v8 model, please
provide me some instructions to get the packages.

Cheers,
~Sandeepa
>
> Thank you again!
>
>
>>
>> Will,
>>
>> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
>> model, are there instructions about how to get the packages and
>> build/run them?
>>
>> Thanks,
>> Sandeepa
>>
>>
>>
>> On 30 September 2013 17:40, William Cohen <wcohen@redhat.com> wrote:
>>> On 09/29/2013 10:36 PM, Masami Hiramatsu wrote:
>>>> (2013/09/26 12:13), Sandeepa Prabhu wrote:
>>>>> Hi Will, Masami,
>>>>>
>>>>> Nice to hear from you, I am using ARM fast model/Foundation model with
>>>>> ARM v8 upstream kernel and a Linaro minimal busybox based ramdisk,
>>>>> testing with loadable modules for now (so don't have dependency on
>>>>> elfutils or GCC autoconf etc)
>>>>>
>>>>> Would you be interested to use Linaro kprobe work as a base for
>>>>> development and validation of systemtap-aarch64?  We are happy to
>>>>> share a public git repo where 'upstream' kernel can be built with
>>>>> kprobes support, which systemtap team can use for verification.  I can
>>>>> do this soon as I have most things working locally, except for
>>>>> kretprobes and 'boosting' support(systemtap can be run without these I
>>>>> believe).
>>>>
>>>> Of course I'm interested in that. Actually I've tried to boot up Linaro's
>>>> Aarch64 kernel on a simulator which ARM is distributing.
>>>> As far as I could see (at that time), aarch64 branch already has singlestep
>>>> implementation for debugging, but no primitive kprobes. And we need to know
>>>> the aarch64 ISA for decoding the binary to find the instructions which will
>>>> be affected by the out-of-line execution, like relative jump, call etc.
>>>>
>>>>> Looking forward for a collaboration :-)
>>>>
>>>> Me too ;)
>>>>
>>>> Thanks!
>>>>
>>>
>>> Hi All,
>>>
>>> I am still trying to get a better kernel running on the aarch64 simulator.  Currently the simulator is running and Fedora 19 image and Linux 3.9+ kernel, but I don't have any of the needed kernel-devel stuff for that kernel. As a work around I am using the aarch64 kernel rpms and attempting to build systemtap instrumentation modules using the following rpms:
>>>
>>> kernel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
>>> kernel-devel-3.12.0-0.rc0.git20.1.x2.fc19.aarch64
>>>
>>> These kernel rpms are cross compiled so some script executables are for AMD64 rather than aarch64.  I had to go through and recompile the script executables.  However, I was able to get an instrumentation modules with the current git checkout for simple hello world:
>>>
>>> $ sudo ../install/bin/stap  -v -m hello -r 3.12.0-0.rc0.git20.1.x2.fc19.aarch64 -p4 -k -e 'probe begin {printf("hello world\n")}'
>>>
>>> Pass 1: parsed user script and 92 library script(s) using 38784virt/30080res/5504shr/23360data kb, in 4730usr/30sys/4763real ms.
>>> Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 39296virt/32000res/5952shr/23872data kb, in 80usr/0sys/81real ms.
>>> Pass 3: translated to C into "/tmp/stapzSdKrS/hello_src.c" using 39296virt/32000res/5952shr/23872data kb, in 10usr/0sys/7real ms.
>>> hello.ko
>>> Pass 4: compiled C into "hello.ko" in 114710usr/3620sys/120572real ms.
>>> Keeping temporary directory "/tmp/stapzSdKrS"
>>>
>>> There are still other things that need to be fixed in systemtap for aarch64.  However, if you are feeling adventurous you might try the current systemtap git checkout on aarch64.
>>>
>>> -Will
>>
>>
>
>
> --
> Masami HIRAMATSU
> IT Management Research Dept. Linux Technology Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>
>

Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/10/03 12:12), Sandeepa Prabhu wrote:
> On 2 October 2013 16:54, Masami Hiramatsu
> <masami.hiramatsu.pt@hitachi.com> wrote:
>> (2013/10/02 13:17), Sandeepa Prabhu wrote:
>>> Hi all,
>>>
>>> I have uploaded ARM64 kprobes work on Linaro public git:
>>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>>> boot-wrapper and fast model setup, though, not sure what it takes to
>>> build for fedora.
>>
>> Thank you for the great work! :)
>>
>> At a glance, I have some comments (almost formatting issues) on it.
>>  - kprobes def config patch should be moved after the
>>    kprobe support.
> Yes ideally, but it is not part of the patchset, I only added this new
> config in my repo for development testing (for sample modules), actual
> changes should go in in arm/arm64/defconfig or any other configs that
> can enable it.

Ah, I see. :)

>>  - Is it really need to use spinlock to protect break_hook?
> Any cpu can remove breakpoint hooks right, and traversal happen in
> debug exception context so mutex are not safe (can sleep/schedule out)
> in debug exception.

I don't think we need to remove the breakpoint hooks after starting
up the kernel. If we use the spinlock there, we'll pay a big cost
because of the lock contention.

>>    Why can't we use rcu_lock as step_hook?
> Need to check, but we need protection for list traversal only (if hook
> removed from another cpu)
> 
>>  - patch_text should check the alignment in itself and return an error.
> Yes, perhaps kprobe also should check the alignment before handling,
> if debug addr (ELR_EL1) is not 32-bit aligned, it's an error and
> should be a BUG() since this it cannot be handled, can fix this in
> next version.

Right, for the similar illegal address, we already use -EILSEQ in
arch_prepare_kprobe() on x86. You can do same thing on arm64.

>>  - For ease of bisect debugging, I'd recommend you to split decoder and
>>    simulator, and to reorder it as below.
>> 1. decoder patch
>> 2. kprobes basic patch (this doesn't support insns which need simulator)
>> 3. simulator & kprobes enhancement patch
> Good point, I will refine in next set of patches, I thought it is
> difficult to split that way since we have unified decode table. One
> basic question with this way of splitting,  "is it expected that  1.
> and 2. (without simulate patch-3) can build and execute?"

Yes, we can just reject the instructions which require simulation at
that point ;) Patch 3. enables kprobes to support such instructions.

>>  - probes-*.c is not good name for the simulator. those should have
>>    better name.
> May be decode-arm64.c? Originally I had decode-* but the logic is
> limited to kprobes and uprobes only so renamed that way.  Other cases
> like jump_labels, use different decoding, and may not share same code.

The decoding table will be different from other usecase, but I think
simulator code can be shared (and must be). It may just get the address,
instruction, and registers, not the kprobe.

>>  - kprobes-arm64.c is also not good name. I think we can have
>>    arch/arm64/kernel/kprobes/ directory on arm64 as same as x86.
>>    we may have core.c and insn_map.c.
> Hmmm, with our design uprobes should share probes-* or decode-* files,
> then uprobes code also reside in arch/arm64/kernel/kprobes/ folder
> itself.  Please suggest if this is fine? folder name may be
> misleading.

Ah, I see. On x86, both uprobes and kprobes have own decode(actually
for classifying instructions) table. Those are a bit different.
At this point, I think you don't need to consider so much about
uprobes. Just focus on kprobes ;) We can evolve the code when we need.

>>  - probes-common.c looks only for kprobes. This should be go into kprobes.c.
> No, this file is common for kprobes 64-bit kernel, uprobes 32-bit
> userspace, and uprobes 64-bit userspace, so needs to be in separate
> file.

Right, I misunderstood. This is a part of the simulator.

>>  - It seems that a part of kretprobe logic is in the kprobes' patch.
> Hmm I can refine it in next patchset, may be jprobes support too go in
> as separate patch?

Yeah, feel free to add jprobes support too. However, jprobe is not common
feature in kernel now, I mean it is not high priority.

>> IMHO, the simulator/decoder will be useful for other cases (KVM, mmiotrace etc.)
>> So I recommend you to keep it away from kprobes as far as possible.
>>
>> And please CC the series to me next time
>> (since I don't subscribe LAKML...).
> Sorry, I did not CC you last time, I will add you from v2.

Thanks ;)

> Masami,
> Couple of questions:
> 1. My changes reside in arm/arm64 only right now, does it make sense
> to publish on LKML so that core kprobes developers can review?

Of course, it is always helpful for us :)

> 2. Wanted to change sample/kprobes/kprobe_example.c to check for the
> missing case of ARM & ARM64 and print the relevant info. Can this
> change be as part of same series (if going on LKML)?

Yeah, it is better to update it within the series.

> Thanks a lot for your review, I would also need help on validation for
> my work, please let me know if our repo holds good for systemtap
> validation. I am interested in using fedora on v8 model, please
> provide me some instructions to get the packages.

Thank you !



-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 3 October 2013 18:30, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> (2013/10/03 12:12), Sandeepa Prabhu wrote:
>> On 2 October 2013 16:54, Masami Hiramatsu
>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>> (2013/10/02 13:17), Sandeepa Prabhu wrote:
>>>> Hi all,
>>>>
>>>> I have uploaded ARM64 kprobes work on Linaro public git:
>>>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>>>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>>>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>>>> boot-wrapper and fast model setup, though, not sure what it takes to
>>>> build for fedora.
>>>
>>> Thank you for the great work! :)
>>>
>>> At a glance, I have some comments (almost formatting issues) on it.
>>>  - kprobes def config patch should be moved after the
>>>    kprobe support.
>> Yes ideally, but it is not part of the patchset, I only added this new
>> config in my repo for development testing (for sample modules), actual
>> changes should go in in arm/arm64/defconfig or any other configs that
>> can enable it.
>
> Ah, I see. :)
>
>>>  - Is it really need to use spinlock to protect break_hook?
>> Any cpu can remove breakpoint hooks right, and traversal happen in
>> debug exception context so mutex are not safe (can sleep/schedule out)
>> in debug exception.
>
> I don't think we need to remove the breakpoint hooks after starting
> up the kernel. If we use the spinlock there, we'll pay a big cost
> because of the lock contention.
Not in kprobes. But kgdb can remove breakpoint handler and use same
API. or atleast while providing an api we should not assume race
cannot happen right?
And there wont be much lock contention, i'ts only if the debug
framework (like kgdb) is wrapping-up, not is normal use-case.

>
>>>    Why can't we use rcu_lock as step_hook?
>> Need to check, but we need protection for list traversal only (if hook
>> removed from another cpu)
>>
>>>  - patch_text should check the alignment in itself and return an error.
>> Yes, perhaps kprobe also should check the alignment before handling,
>> if debug addr (ELR_EL1) is not 32-bit aligned, it's an error and
>> should be a BUG() since this it cannot be handled, can fix this in
>> next version.
>
> Right, for the similar illegal address, we already use -EILSEQ in
> arch_prepare_kprobe() on x86. You can do same thing on arm64.
Hmm, I will change this part.

>
>>>  - For ease of bisect debugging, I'd recommend you to split decoder and
>>>    simulator, and to reorder it as below.
>>> 1. decoder patch
>>> 2. kprobes basic patch (this doesn't support insns which need simulator)
>>> 3. simulator & kprobes enhancement patch
>> Good point, I will refine in next set of patches, I thought it is
>> difficult to split that way since we have unified decode table. One
>> basic question with this way of splitting,  "is it expected that  1.
>> and 2. (without simulate patch-3) can build and execute?"
>
> Yes, we can just reject the instructions which require simulation at
> that point ;) Patch 3. enables kprobes to support such instructions.
Got it, and can be done cleanly :-)

>
>>>  - probes-*.c is not good name for the simulator. those should have
>>>    better name.
>> May be decode-arm64.c? Originally I had decode-* but the logic is
>> limited to kprobes and uprobes only so renamed that way.  Other cases
>> like jump_labels, use different decoding, and may not share same code.
>
> The decoding table will be different from other usecase, but I think
> simulator code can be shared (and must be). It may just get the address,
> instruction, and registers, not the kprobe.
When we wrote at Linaro, our plan was to share simulation calls
between kprobes and uprobes, and planned to use 'struct kprobes' for
both the frameworks, this is how it was done on "arch/arm/kernel/"
effectively.
If more frameworks can use it (as it seems) I change it to accept
opcode, pc value and saved pt_regs and avoid kprobe struct altogether.
Also, we are starting on uprobes at Linaro, so it won't be too long
before we start thinking about that too ;-)

>
>>>  - kprobes-arm64.c is also not good name. I think we can have
>>>    arch/arm64/kernel/kprobes/ directory on arm64 as same as x86.
>>>    we may have core.c and insn_map.c.
>> Hmmm, with our design uprobes should share probes-* or decode-* files,
>> then uprobes code also reside in arch/arm64/kernel/kprobes/ folder
>> itself.  Please suggest if this is fine? folder name may be
>> misleading.
>
> Ah, I see. On x86, both uprobes and kprobes have own decode(actually
> for classifying instructions) table. Those are a bit different.
> At this point, I think you don't need to consider so much about
> uprobes. Just focus on kprobes ;) We can evolve the code when we need.
>
>>>  - probes-common.c looks only for kprobes. This should be go into kprobes.c.
>> No, this file is common for kprobes 64-bit kernel, uprobes 32-bit
>> userspace, and uprobes 64-bit userspace, so needs to be in separate
>> file.
>
> Right, I misunderstood. This is a part of the simulator.
>
>>>  - It seems that a part of kretprobe logic is in the kprobes' patch.
>> Hmm I can refine it in next patchset, may be jprobes support too go in
>> as separate patch?
>
> Yeah, feel free to add jprobes support too. However, jprobe is not common
> feature in kernel now, I mean it is not high priority.
>
>>> IMHO, the simulator/decoder will be useful for other cases (KVM, mmiotrace etc.)
>>> So I recommend you to keep it away from kprobes as far as possible.
>>>
>>> And please CC the series to me next time
>>> (since I don't subscribe LAKML...).
>> Sorry, I did not CC you last time, I will add you from v2.
>
> Thanks ;)
>
>> Masami,
>> Couple of questions:
>> 1. My changes reside in arm/arm64 only right now, does it make sense
>> to publish on LKML so that core kprobes developers can review?
>
> Of course, it is always helpful for us :)
>
>> 2. Wanted to change sample/kprobes/kprobe_example.c to check for the
>> missing case of ARM & ARM64 and print the relevant info. Can this
>> change be as part of same series (if going on LKML)?
>
> Yeah, it is better to update it within the series.
OK, I will do that, and CC LKML from v2.

>
>> Thanks a lot for your review, I would also need help on validation for
>> my work, please let me know if our repo holds good for systemtap
>> validation. I am interested in using fedora on v8 model, please
>> provide me some instructions to get the packages.
>
> Thank you !

Question:
I am working on v2 patchset based on comments, for next week to post,
do you have basic aarch64 setup (fast-model/hardware), ARM v8-ARM etc?
I mean, how about sharing some efforts with me(Linaro) going further?
Most work shall go through LAKML so you may have to subscribe to that
;),  but do you mind working on Linaro hosted public git ? we can lay
out a plan then. After kprobes, we have much work on uprobes in queue
(both 32-bit and 64-bit user-space) and your insights help us, since
you are one of the maintainers of both subsystems.

Cheers,
Sandeepa
>
>
>
> --
> Masami HIRAMATSU
> IT Management Research Dept. Linux Technology Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>
>

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/02/2013 12:17 AM, Sandeepa Prabhu wrote:
> Hi all,
> 
> I have uploaded ARM64 kprobes work on Linaro public git:
> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
> boot-wrapper and fast model setup, though, not sure what it takes to
> build for fedora.
> 
> Will,
> 
> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
> model, are there instructions about how to get the packages and
> build/run them?
> 
> Thanks,
> Sandeepa
> 

Hi Sandeepa,

The fedora 19 work is publicly available.  The following page talks about setting things up:

http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart

-Will

Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/10/04 12:24), Sandeepa Prabhu wrote:
>>>>  - Is it really need to use spinlock to protect break_hook?
>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>> in debug exception.
>>
>> I don't think we need to remove the breakpoint hooks after starting
>> up the kernel. If we use the spinlock there, we'll pay a big cost
>> because of the lock contention.
> Not in kprobes. But kgdb can remove breakpoint handler and use same
> API. or atleast while providing an api we should not assume race
> cannot happen right?

In that case, we'd better add a wrapper handler for kgdb so that
the list isn't updated even if the kgdb removes its handler.

> And there wont be much lock contention, i'ts only if the debug
> framework (like kgdb) is wrapping-up, not is normal use-case.

Hmm, it seems that the spinlock is locked while handling a breakpoint.
This will cause a bad performance issue when we put many kprobes
on SMP system.

[...]
>>>>  - probes-*.c is not good name for the simulator. those should have
>>>>    better name.
>>> May be decode-arm64.c? Originally I had decode-* but the logic is
>>> limited to kprobes and uprobes only so renamed that way.  Other cases
>>> like jump_labels, use different decoding, and may not share same code.
>>
>> The decoding table will be different from other usecase, but I think
>> simulator code can be shared (and must be). It may just get the address,
>> instruction, and registers, not the kprobe.
> When we wrote at Linaro, our plan was to share simulation calls
> between kprobes and uprobes, and planned to use 'struct kprobes' for
> both the frameworks, this is how it was done on "arch/arm/kernel/"
> effectively.

Uh, I should review arm32 again...

> If more frameworks can use it (as it seems) I change it to accept
> opcode, pc value and saved pt_regs and avoid kprobe struct altogether.
> Also, we are starting on uprobes at Linaro, so it won't be too long
> before we start thinking about that too ;-)

Since kprobes data structure includes many information which is not related
to the simulation, I'd like to keep it away from that.

[...]
>>> Masami,
>>> Couple of questions:
>>> 1. My changes reside in arm/arm64 only right now, does it make sense
>>> to publish on LKML so that core kprobes developers can review?
>>
>> Of course, it is always helpful for us :)
>>
>>> 2. Wanted to change sample/kprobes/kprobe_example.c to check for the
>>> missing case of ARM & ARM64 and print the relevant info. Can this
>>> change be as part of same series (if going on LKML)?
>>
>> Yeah, it is better to update it within the series.
> OK, I will do that, and CC LKML from v2.

thanks :)

>>> Thanks a lot for your review, I would also need help on validation for
>>> my work, please let me know if our repo holds good for systemtap
>>> validation. I am interested in using fedora on v8 model, please
>>> provide me some instructions to get the packages.
>>
>> Thank you !
> 
> Question:
> I am working on v2 patchset based on comments, for next week to post,
> do you have basic aarch64 setup (fast-model/hardware), ARM v8-ARM etc?
> I mean, how about sharing some efforts with me(Linaro) going further?

Yeah, I have a foundation-model simulator, I just need to set it up.

> Most work shall go through LAKML so you may have to subscribe to that
> ;),  but do you mind working on Linaro hosted public git ? we can lay
> out a plan then. After kprobes, we have much work on uprobes in queue
> (both 32-bit and 64-bit user-space) and your insights help us, since
> you are one of the maintainers of both subsystems.

Oh, OK. I'll subscribe it, and, yeah, linaro public git should be
better place to work with :)

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 4 October 2013 21:26, William Cohen <wcohen@redhat.com> wrote:
> On 10/02/2013 12:17 AM, Sandeepa Prabhu wrote:
>> Hi all,
>>
>> I have uploaded ARM64 kprobes work on Linaro public git:
>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>> boot-wrapper and fast model setup, though, not sure what it takes to
>> build for fedora.
>>
>> Will,
>>
>> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
>> model, are there instructions about how to get the packages and
>> build/run them?
>>
>> Thanks,
>> Sandeepa
>>
>
> Hi Sandeepa,
>
> The fedora 19 work is publicly available.  The following page talks about setting things up:
>
> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart

Thanks, I will try this setup.
>
> -Will

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 5 October 2013 08:54, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> (2013/10/04 12:24), Sandeepa Prabhu wrote:
>>>>>  - Is it really need to use spinlock to protect break_hook?
>>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>>> in debug exception.
>>>
>>> I don't think we need to remove the breakpoint hooks after starting
>>> up the kernel. If we use the spinlock there, we'll pay a big cost
>>> because of the lock contention.
>> Not in kprobes. But kgdb can remove breakpoint handler and use same
>> API. or atleast while providing an api we should not assume race
>> cannot happen right?
>
> In that case, we'd better add a wrapper handler for kgdb so that
> the list isn't updated even if the kgdb removes its handler.
>
>> And there wont be much lock contention, i'ts only if the debug
>> framework (like kgdb) is wrapping-up, not is normal use-case.
>
> Hmm, it seems that the spinlock is locked while handling a breakpoint.
> This will cause a bad performance issue when we put many kprobes
> on SMP system.
arm maintainers prefer a reader/writer spin-locks, so there wont be
lock contention in debug path, each instance of kprobe hook trap (on
any CPU) would be a reader, not blocking.

>
> [...]
>>>>>  - probes-*.c is not good name for the simulator. those should have
>>>>>    better name.
>>>> May be decode-arm64.c? Originally I had decode-* but the logic is
>>>> limited to kprobes and uprobes only so renamed that way.  Other cases
>>>> like jump_labels, use different decoding, and may not share same code.
>>>
>>> The decoding table will be different from other usecase, but I think
>>> simulator code can be shared (and must be). It may just get the address,
>>> instruction, and registers, not the kprobe.
>> When we wrote at Linaro, our plan was to share simulation calls
>> between kprobes and uprobes, and planned to use 'struct kprobes' for
>> both the frameworks, this is how it was done on "arch/arm/kernel/"
>> effectively.
>
> Uh, I should review arm32 again...
>
>> If more frameworks can use it (as it seems) I change it to accept
>> opcode, pc value and saved pt_regs and avoid kprobe struct altogether.
>> Also, we are starting on uprobes at Linaro, so it won't be too long
>> before we start thinking about that too ;-)
>
> Since kprobes data structure includes many information which is not related
> to the simulation, I'd like to keep it away from that.
Yup, can simulate without that. I will avoid kprobe struct from
simulators and keep it cleaner :-)
>
> [...]
>>>> Masami,
>>>> Couple of questions:
>>>> 1. My changes reside in arm/arm64 only right now, does it make sense
>>>> to publish on LKML so that core kprobes developers can review?
>>>
>>> Of course, it is always helpful for us :)
>>>
>>>> 2. Wanted to change sample/kprobes/kprobe_example.c to check for the
>>>> missing case of ARM & ARM64 and print the relevant info. Can this
>>>> change be as part of same series (if going on LKML)?
>>>
>>> Yeah, it is better to update it within the series.
>> OK, I will do that, and CC LKML from v2.
>
> thanks :)
>
>>>> Thanks a lot for your review, I would also need help on validation for
>>>> my work, please let me know if our repo holds good for systemtap
>>>> validation. I am interested in using fedora on v8 model, please
>>>> provide me some instructions to get the packages.
>>>
>>> Thank you !
>>
>> Question:
>> I am working on v2 patchset based on comments, for next week to post,
>> do you have basic aarch64 setup (fast-model/hardware), ARM v8-ARM etc?
>> I mean, how about sharing some efforts with me(Linaro) going further?
>
> Yeah, I have a foundation-model simulator, I just need to set it up.
>
>> Most work shall go through LAKML so you may have to subscribe to that
>> ;),  but do you mind working on Linaro hosted public git ? we can lay
>> out a plan then. After kprobes, we have much work on uprobes in queue
>> (both 32-bit and 64-bit user-space) and your insights help us, since
>> you are one of the maintainers of both subsystems.
>
> Oh, OK. I'll subscribe it, and, yeah, linaro public git should be
> better place to work with :)
Thanks!!  I will reach out to leads in Linaro regarding read/write
access and stuff (guess it's read-only right now)
You can have a look at following Linaro cards (agile projecting tracking):
   https://cards.linaro.org/browse/KWG-13
   https://cards.linaro.org/browse/CARD-564
If you find them interesting, you may subscribe to these by creating a
login, and clicking on  more -> 'watch this issue'.

Thanks,
Sandeepa
>
> Thank you,
>
> --
> Masami HIRAMATSU
> IT Management Research Dept. Linux Technology Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>
>

Re: Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/10/07 18:50), Sandeepa Prabhu wrote:
> On 5 October 2013 08:54, Masami Hiramatsu
> <masami.hiramatsu.pt@hitachi.com> wrote:
>> (2013/10/04 12:24), Sandeepa Prabhu wrote:
>>>>>>  - Is it really need to use spinlock to protect break_hook?
>>>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>>>> in debug exception.
>>>>
>>>> I don't think we need to remove the breakpoint hooks after starting
>>>> up the kernel. If we use the spinlock there, we'll pay a big cost
>>>> because of the lock contention.
>>> Not in kprobes. But kgdb can remove breakpoint handler and use same
>>> API. or atleast while providing an api we should not assume race
>>> cannot happen right?
>>
>> In that case, we'd better add a wrapper handler for kgdb so that
>> the list isn't updated even if the kgdb removes its handler.
>>
>>> And there wont be much lock contention, i'ts only if the debug
>>> framework (like kgdb) is wrapping-up, not is normal use-case.
>>
>> Hmm, it seems that the spinlock is locked while handling a breakpoint.
>> This will cause a bad performance issue when we put many kprobes
>> on SMP system.
> arm maintainers prefer a reader/writer spin-locks, so there wont be
> lock contention in debug path, each instance of kprobe hook trap (on
> any CPU) would be a reader, not blocking.

OK for the first step, and it eventually should be fixed to lockless.
(depends on the performance improvement)

>> [...]
>>>>>>  - probes-*.c is not good name for the simulator. those should have
>>>>>>    better name.
>>>>> May be decode-arm64.c? Originally I had decode-* but the logic is
>>>>> limited to kprobes and uprobes only so renamed that way.  Other cases
>>>>> like jump_labels, use different decoding, and may not share same code.
>>>>
>>>> The decoding table will be different from other usecase, but I think
>>>> simulator code can be shared (and must be). It may just get the address,
>>>> instruction, and registers, not the kprobe.
>>> When we wrote at Linaro, our plan was to share simulation calls
>>> between kprobes and uprobes, and planned to use 'struct kprobes' for
>>> both the frameworks, this is how it was done on "arch/arm/kernel/"
>>> effectively.
>>
>> Uh, I should review arm32 again...
>>
>>> If more frameworks can use it (as it seems) I change it to accept
>>> opcode, pc value and saved pt_regs and avoid kprobe struct altogether.
>>> Also, we are starting on uprobes at Linaro, so it won't be too long
>>> before we start thinking about that too ;-)
>>
>> Since kprobes data structure includes many information which is not related
>> to the simulation, I'd like to keep it away from that.
> Yup, can simulate without that. I will avoid kprobe struct from
> simulators and keep it cleaner :-)

Good ;)

[...]
>>> Question:
>>> I am working on v2 patchset based on comments, for next week to post,
>>> do you have basic aarch64 setup (fast-model/hardware), ARM v8-ARM etc?
>>> I mean, how about sharing some efforts with me(Linaro) going further?
>>
>> Yeah, I have a foundation-model simulator, I just need to set it up.
>>
>>> Most work shall go through LAKML so you may have to subscribe to that
>>> ;),  but do you mind working on Linaro hosted public git ? we can lay
>>> out a plan then. After kprobes, we have much work on uprobes in queue
>>> (both 32-bit and 64-bit user-space) and your insights help us, since
>>> you are one of the maintainers of both subsystems.
>>
>> Oh, OK. I'll subscribe it, and, yeah, linaro public git should be
>> better place to work with :)
> Thanks!!  I will reach out to leads in Linaro regarding read/write
> access and stuff (guess it's read-only right now)
> You can have a look at following Linaro cards (agile projecting tracking):
>    https://cards.linaro.org/browse/KWG-13
>    https://cards.linaro.org/browse/CARD-564
> If you find them interesting, you may subscribe to these by creating a
> login, and clicking on  more -> 'watch this issue'.

Looks nice :) I'll do that.

Thank you!


-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 7 October 2013 15:41, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> (2013/10/07 18:50), Sandeepa Prabhu wrote:
>> On 5 October 2013 08:54, Masami Hiramatsu
>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>> (2013/10/04 12:24), Sandeepa Prabhu wrote:
>>>>>>>  - Is it really need to use spinlock to protect break_hook?
>>>>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>>>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>>>>> in debug exception.
>>>>>
>>>>> I don't think we need to remove the breakpoint hooks after starting
>>>>> up the kernel. If we use the spinlock there, we'll pay a big cost
>>>>> because of the lock contention.
>>>> Not in kprobes. But kgdb can remove breakpoint handler and use same
>>>> API. or atleast while providing an api we should not assume race
>>>> cannot happen right?
>>>
>>> In that case, we'd better add a wrapper handler for kgdb so that
>>> the list isn't updated even if the kgdb removes its handler.
>>>
>>>> And there wont be much lock contention, i'ts only if the debug
>>>> framework (like kgdb) is wrapping-up, not is normal use-case.
>>>
>>> Hmm, it seems that the spinlock is locked while handling a breakpoint.
>>> This will cause a bad performance issue when we put many kprobes
>>> on SMP system.
>> arm maintainers prefer a reader/writer spin-locks, so there wont be
>> lock contention in debug path, each instance of kprobe hook trap (on
>> any CPU) would be a reader, not blocking.
>
> OK for the first step, and it eventually should be fixed to lockless.
> (depends on the performance improvement)
Hmm, is there a performance requirement for systemtap or perf? -like
how much time each test suite should consume etc?
Want to know the acceptance criteria for systemtap or perf to say
'kprobes/uprobes on an architecture' is complaint and good enough for
tracing?

>
>>> [...]
>>>>>>>  - probes-*.c is not good name for the simulator. those should have
>>>>>>>    better name.
>>>>>> May be decode-arm64.c? Originally I had decode-* but the logic is
>>>>>> limited to kprobes and uprobes only so renamed that way.  Other cases
>>>>>> like jump_labels, use different decoding, and may not share same code.
>>>>>
>>>>> The decoding table will be different from other usecase, but I think
>>>>> simulator code can be shared (and must be). It may just get the address,
>>>>> instruction, and registers, not the kprobe.
>>>> When we wrote at Linaro, our plan was to share simulation calls
>>>> between kprobes and uprobes, and planned to use 'struct kprobes' for
>>>> both the frameworks, this is how it was done on "arch/arm/kernel/"
>>>> effectively.
>>>
>>> Uh, I should review arm32 again...
>>>
>>>> If more frameworks can use it (as it seems) I change it to accept
>>>> opcode, pc value and saved pt_regs and avoid kprobe struct altogether.
>>>> Also, we are starting on uprobes at Linaro, so it won't be too long
>>>> before we start thinking about that too ;-)
>>>
>>> Since kprobes data structure includes many information which is not related
>>> to the simulation, I'd like to keep it away from that.
>> Yup, can simulate without that. I will avoid kprobe struct from
>> simulators and keep it cleaner :-)
>
> Good ;)
>
> [...]
>>>> Question:
>>>> I am working on v2 patchset based on comments, for next week to post,
>>>> do you have basic aarch64 setup (fast-model/hardware), ARM v8-ARM etc?
>>>> I mean, how about sharing some efforts with me(Linaro) going further?
>>>
>>> Yeah, I have a foundation-model simulator, I just need to set it up.
>>>
>>>> Most work shall go through LAKML so you may have to subscribe to that
>>>> ;),  but do you mind working on Linaro hosted public git ? we can lay
>>>> out a plan then. After kprobes, we have much work on uprobes in queue
>>>> (both 32-bit and 64-bit user-space) and your insights help us, since
>>>> you are one of the maintainers of both subsystems.
>>>
>>> Oh, OK. I'll subscribe it, and, yeah, linaro public git should be
>>> better place to work with :)
>> Thanks!!  I will reach out to leads in Linaro regarding read/write
>> access and stuff (guess it's read-only right now)
>> You can have a look at following Linaro cards (agile projecting tracking):
>>    https://cards.linaro.org/browse/KWG-13
>>    https://cards.linaro.org/browse/CARD-564
>> If you find them interesting, you may subscribe to these by creating a
>> login, and clicking on  more -> 'watch this issue'.
>
> Looks nice :) I'll do that.
>
> Thank you!
>
>
> --
> Masami HIRAMATSU
> IT Management Research Dept. Linux Technology Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>
>

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

>
> Hi Sandeepa,
>
> The fedora 19 work is publicly available.  The following page talks about setting things up:
>
> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>
> -Will
Hi Will,

Thanks, I could get the console, but very slow.  Where can I look for
kernel sources and the systemtap packages? Wanted to try my kernel
changes on this and experiment systemtap, I can clone and create
kernel devel branch on linaro git for kprobes, if this work out well.

Thanks,
Sandeepa

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 8 October 2013 09:58, Sandeepa Prabhu <sandeepa.prabhu@linaro.org> wrote:
>>
>> Hi Sandeepa,
>>
>> The fedora 19 work is publicly available.  The following page talks about setting things up:
>>
>> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>>
>> -Will
> Hi Will,
>
> Thanks, I could get the console, but very slow.  Where can I look for
> kernel sources and the systemtap packages? Wanted to try my kernel
> changes on this and experiment systemtap, I can clone and create
> kernel devel branch on linaro git for kprobes, if this work out well.
>
This is single CPU version, has it been tested on 4-core models
(RTSM)? Responsiveness might improve with 4-core model [atleast LAMP
stack run faster]

> Thanks,
> Sandeepa

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/08/2013 12:39 AM, Sandeepa Prabhu wrote:
> On 8 October 2013 09:58, Sandeepa Prabhu <sandeepa.prabhu@linaro.org> wrote:
>>>
>>> Hi Sandeepa,
>>>
>>> The fedora 19 work is publicly available.  The following page talks about setting things up:
>>>
>>> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>>>
>>> -Will
>> Hi Will,
>>
>> Thanks, I could get the console, but very slow.  Where can I look for
>> kernel sources and the systemtap packages? Wanted to try my kernel
>> changes on this and experiment systemtap, I can clone and create
>> kernel devel branch on linaro git for kprobes, if this work out well.
>>
> This is single CPU version, has it been tested on 4-core models
> (RTSM)? Responsiveness might improve with 4-core model [atleast LAMP
> stack run faster]
> 
>> Thanks,
>> Sandeepa

Hi Sandeepa,

Yes, the Foundation model is very slow.  I have read that there has been some work to get aarch64 support in qemu, but I have not personally tried it:

http://news.opensuse.org/2013/10/01/suse-speeds-up-building-aarch64-software-in-qemu/

For systemtap your best bet is to use a git clone which has the current aarch64 patches:

git clone git://sourceware.org/git/systemtap.git

Then build (you may need to do a "yum-builddep systemtap" as root to pull in dependencies that systemtap needs to build):

cd systemtap
./configure --disable-docs
make
make install

At this point should have a systemtap that will build stuff for aarch64

For the current state of aarch64 package you can look at what is in http://arm-temp.ausil.us/pub/fedora-arm/stage4/
Each of the packages is in a subdirectory.  You should be able to do something like the following to get the sources:

wget http://arm-temp.ausil.us/pub/fedora-arm/stage4/kernel-3.12.0-0.rc4.git0.1.x2.fc19/kernel-3.12.0-0.rc4.git0.1.x2.fc19.src.rpm
rpm -Uvh kernel*.src.rpm
cd rpmbuild/SPECS
yum-builddep kernel
rpmbuild -bp kernel.spec

Generally when fedora builds kernels it produces a /boot/config-`uname -r` file describing the configure used to build the kernel.  This is in the kernel rpm.  I don't recall if the kernel rpm is installed in the basic aarch64 image.  You may need to do a "yum install kernel" to get that installed on the machine.

-Will

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/14/2013 12:38 PM, William Cohen wrote:
> On 10/08/2013 12:39 AM, Sandeepa Prabhu wrote:
>> On 8 October 2013 09:58, Sandeepa Prabhu <sandeepa.prabhu@linaro.org> wrote:
>>>>
>>>> Hi Sandeepa,
>>>>
>>>> The fedora 19 work is publicly available.  The following page talks about setting things up:
>>>>
>>>> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>>>>
>>>> -Will
>>> Hi Will,
>>>
>>> Thanks, I could get the console, but very slow.  Where can I look for
>>> kernel sources and the systemtap packages? Wanted to try my kernel
>>> changes on this and experiment systemtap, I can clone and create
>>> kernel devel branch on linaro git for kprobes, if this work out well.
>>>
>> This is single CPU version, has it been tested on 4-core models
>> (RTSM)? Responsiveness might improve with 4-core model [atleast LAMP
>> stack run faster]
>>
>>> Thanks,
>>> Sandeepa

Hi Sandeepa,

There is a new fedora image which uses uefi to boot just announce today.  This might be a bit easier to use use that the older image.  The following URL mentions it:

https://lists.fedoraproject.org/pipermail/arm/2013-October/006964.html

-Will
> 
> Hi Sandeepa,
> 
> Yes, the Foundation model is very slow.  I have read that there has been some work to get aarch64 support in qemu, but I have not personally tried it:
> 
> http://news.opensuse.org/2013/10/01/suse-speeds-up-building-aarch64-software-in-qemu/
> 
> For systemtap your best bet is to use a git clone which has the current aarch64 patches:
> 
> git clone git://sourceware.org/git/systemtap.git
> 
> Then build (you may need to do a "yum-builddep systemtap" as root to pull in dependencies that systemtap needs to build):
> 
> cd systemtap
> ./configure --disable-docs
> make
> make install
> 
> At this point should have a systemtap that will build stuff for aarch64
> 
> For the current state of aarch64 package you can look at what is in http://arm-temp.ausil.us/pub/fedora-arm/stage4/
> Each of the packages is in a subdirectory.  You should be able to do something like the following to get the sources:
> 
> wget http://arm-temp.ausil.us/pub/fedora-arm/stage4/kernel-3.12.0-0.rc4.git0.1.x2.fc19/kernel-3.12.0-0.rc4.git0.1.x2.fc19.src.rpm
> rpm -Uvh kernel*.src.rpm
> cd rpmbuild/SPECS
> yum-builddep kernel
> rpmbuild -bp kernel.spec
> 
> Generally when fedora builds kernels it produces a /boot/config-`uname -r` file describing the configure used to build the kernel.  This is in the kernel rpm.  I don't recall if the kernel rpm is installed in the basic aarch64 image.  You may need to do a "yum install kernel" to get that installed on the machine.
> 
> -Will
> 

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

Hi Will,

Thanks for the details, so we have 2 more options (qemu and uefi
image) to try out. I would get back to you once tried out the options,
and then decide what's best for kprobes <-> systemtap validation.
Bit of challenge for me is that I have Ubuntu host - running yum upon
that!,  not sure of any cross-development issues, but as for aarch64
native building(on fc19-aarch64 model itself) this should be okay
though its very slow.

Thanks,
Sandeepa



On 15 October 2013 02:51, William Cohen <wcohen@redhat.com> wrote:
> On 10/14/2013 12:38 PM, William Cohen wrote:
>> On 10/08/2013 12:39 AM, Sandeepa Prabhu wrote:
>>> On 8 October 2013 09:58, Sandeepa Prabhu <sandeepa.prabhu@linaro.org> wrote:
>>>>>
>>>>> Hi Sandeepa,
>>>>>
>>>>> The fedora 19 work is publicly available.  The following page talks about setting things up:
>>>>>
>>>>> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>>>>>
>>>>> -Will
>>>> Hi Will,
>>>>
>>>> Thanks, I could get the console, but very slow.  Where can I look for
>>>> kernel sources and the systemtap packages? Wanted to try my kernel
>>>> changes on this and experiment systemtap, I can clone and create
>>>> kernel devel branch on linaro git for kprobes, if this work out well.
>>>>
>>> This is single CPU version, has it been tested on 4-core models
>>> (RTSM)? Responsiveness might improve with 4-core model [atleast LAMP
>>> stack run faster]
>>>
>>>> Thanks,
>>>> Sandeepa
>
> Hi Sandeepa,
>
> There is a new fedora image which uses uefi to boot just announce today.  This might be a bit easier to use use that the older image.  The following URL mentions it:
>
> https://lists.fedoraproject.org/pipermail/arm/2013-October/006964.html
>
> -Will
>>
>> Hi Sandeepa,
>>
>> Yes, the Foundation model is very slow.  I have read that there has been some work to get aarch64 support in qemu, but I have not personally tried it:
>>
>> http://news.opensuse.org/2013/10/01/suse-speeds-up-building-aarch64-software-in-qemu/
>>
>> For systemtap your best bet is to use a git clone which has the current aarch64 patches:
>>
>> git clone git://sourceware.org/git/systemtap.git
>>
>> Then build (you may need to do a "yum-builddep systemtap" as root to pull in dependencies that systemtap needs to build):
>>
>> cd systemtap
>> ./configure --disable-docs
>> make
>> make install
>>
>> At this point should have a systemtap that will build stuff for aarch64
>>
>> For the current state of aarch64 package you can look at what is in http://arm-temp.ausil.us/pub/fedora-arm/stage4/
>> Each of the packages is in a subdirectory.  You should be able to do something like the following to get the sources:
>>
>> wget http://arm-temp.ausil.us/pub/fedora-arm/stage4/kernel-3.12.0-0.rc4.git0.1.x2.fc19/kernel-3.12.0-0.rc4.git0.1.x2.fc19.src.rpm
>> rpm -Uvh kernel*.src.rpm
>> cd rpmbuild/SPECS
>> yum-builddep kernel
>> rpmbuild -bp kernel.spec
>>
>> Generally when fedora builds kernels it produces a /boot/config-`uname -r` file describing the configure used to build the kernel.  This is in the kernel rpm.  I don't recall if the kernel rpm is installed in the basic aarch64 image.  You may need to do a "yum install kernel" to get that installed on the machine.
>>
>> -Will
>>
>

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/14/2013 10:29 PM, Sandeepa Prabhu wrote:
> Hi Will,
> 
> Thanks for the details, so we have 2 more options (qemu and uefi
> image) to try out. I would get back to you once tried out the options,
> and then decide what's best for kprobes <-> systemtap validation.
> Bit of challenge for me is that I have Ubuntu host - running yum upon
> that!,  not sure of any cross-development issues, but as for aarch64
> native building(on fc19-aarch64 model itself) this should be okay
> though its very slow.
> 
> Thanks,
> Sandeepa

Hi Sandeepa,

If you have things on ubuntu image, the information on the wiki might be helpful in setting up systemtap development build:

https://sourceware.org/systemtap/wiki/SystemtapOnUbuntu

The following is a general description of building systemtap:

https://sourceware.org/git/?p=systemtap.git;a=blob_plain;f=README;hb=HEAD

Yes, building systemtap on the foundation model is very slow.  One thing to avoid slowing things down further is to disable documentation building. Here is what I am using on my image to configure things:

 ./configure --disable-docs --prefix=/home/wcohen/systemtap_write/install

One other caveat that may (or may not) be an issue is whether the native compiler include a neon header file. The lack of neon header caused a problem with some of the native kernel builds and there were some patches that had to be put in. I beleive that the following directory has a source rpm with the needed patch:

http://arm-temp.ausil.us/pub/fedora-arm/stage4/gcc-4.8.1-10.x1.fc19/

The following is the related fedora bugzilla entry for it:

https://bugzilla.redhat.com/show_bug.cgi?id=1007490

-Will

Re: Re: Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/10/07 20:12), Sandeepa Prabhu wrote:
>>>>>>>>  - Is it really need to use spinlock to protect break_hook?
>>>>>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>>>>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>>>>>> in debug exception.
>>>>>>
>>>>>> I don't think we need to remove the breakpoint hooks after starting
>>>>>> up the kernel. If we use the spinlock there, we'll pay a big cost
>>>>>> because of the lock contention.
>>>>> Not in kprobes. But kgdb can remove breakpoint handler and use same
>>>>> API. or atleast while providing an api we should not assume race
>>>>> cannot happen right?
>>>>
>>>> In that case, we'd better add a wrapper handler for kgdb so that
>>>> the list isn't updated even if the kgdb removes its handler.
>>>>
>>>>> And there wont be much lock contention, i'ts only if the debug
>>>>> framework (like kgdb) is wrapping-up, not is normal use-case.
>>>>
>>>> Hmm, it seems that the spinlock is locked while handling a breakpoint.
>>>> This will cause a bad performance issue when we put many kprobes
>>>> on SMP system.
>>> arm maintainers prefer a reader/writer spin-locks, so there wont be
>>> lock contention in debug path, each instance of kprobe hook trap (on
>>> any CPU) would be a reader, not blocking.
>>
>> OK for the first step, and it eventually should be fixed to lockless.
>> (depends on the performance improvement)
> Hmm, is there a performance requirement for systemtap or perf? -like
> how much time each test suite should consume etc?

Basically, for the enterprise use, we aims to get less than 5% loss
of runtime performance. Of course it depends on the configuration.
This requirement comes from the usage of tracing, it's usually used
as a "flight-recorder" in such system. For analyzing the root cause
of the trouble, some fundamental events are always recorded into a
memory buffer. When encountering a trouble, the buffer will be dumped,
and trouble shooting team analyzes it.

Thus, I'd like to make the performance overhead of tracing as
small as possible.

However, for debugging use, the performance degradation is not
so important.

> Want to know the acceptance criteria for systemtap or perf to say
> 'kprobes/uprobes on an architecture' is complaint and good enough for
> tracing?

I think there is no such criteria. The overhead problem depends on the
use-cases as I said above. If it is functional, it's enough to use by
perf/ftrace ;) Performance optimization can be done afterwords.

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/14/2013 05:21 PM, William Cohen wrote:
> On 10/14/2013 12:38 PM, William Cohen wrote:
>> On 10/08/2013 12:39 AM, Sandeepa Prabhu wrote:
>>> On 8 October 2013 09:58, Sandeepa Prabhu <sandeepa.prabhu@linaro.org> wrote:
>>>>>
>>>>> Hi Sandeepa,
>>>>>
>>>>> The fedora 19 work is publicly available.  The following page talks about setting things up:
>>>>>
>>>>> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>>>>>
>>>>> -Will
>>>> Hi Will,
>>>>
>>>> Thanks, I could get the console, but very slow.  Where can I look for
>>>> kernel sources and the systemtap packages? Wanted to try my kernel
>>>> changes on this and experiment systemtap, I can clone and create
>>>> kernel devel branch on linaro git for kprobes, if this work out well.
>>>>
>>> This is single CPU version, has it been tested on 4-core models
>>> (RTSM)? Responsiveness might improve with 4-core model [atleast LAMP
>>> stack run faster]
>>>
>>>> Thanks,
>>>> Sandeepa
> 
> Hi Sandeepa,
> 
> There is a new fedora image which uses uefi to boot just announce today.  This might be a bit easier to use use that the older image.  The following URL mentions it:
> 
> https://lists.fedoraproject.org/pipermail/arm/2013-October/006964.html
> 
> -Will

Hi Sandeepa,

I have that new aarch64 image set up with a kernel-3.12.0-0.rc4.git0.1.x2.fc19.aarch64 and all the other related software to natively build systemtap and have systemtap build instrumentation natively.  This evening it showed signs of life.  A trivial hello world on the aarch64 foundation simulator:

$ sudo ../install/bin/stap  -v -k -e 'probe begin {printf("hello world\n"); exit()}'
Pass 1: parsed user script and 92 library script(s) using 140944virt/25340res/2736shr/23292data kb, in 4760usr/140sys/5313real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 141468virt/26264res/2952shr/23816data kb, in 80usr/10sys/90real ms.
Pass 3: translated to C into "/tmp/stapUPFYHe/stap_14454_src.c" using 141468virt/26264res/2952shr/23816data kb, in 10usr/0sys/9real ms.
Pass 4: compiled C into "stap_14454.ko" in 34870usr/3060sys/42182real ms.
Pass 5: starting run.
[62727.772309] stap_14598: systemtap: 2.4/0.156, base: ffffffbffc0a4000, memory: 13data/18text/0ctx/2058net/11alloc kb, probes: 1
hello world
Pass 5: run completed in 80usr/90sys/501real ms.
Keeping temporary directory "/tmp/stapUPFYHe"

Yes, it is really, really slow as seen in the times that each of the passes took. Pass 1 took 5.3 seconds and pass 4 took 42 seconds for this example that didn't have to search any debuginfo information and only had two statements in it.  It is unlikely to be practical for the simulator to run "make installcheck"

-Will

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/14/2013 12:38 PM, William Cohen wrote:
> On 10/08/2013 12:39 AM, Sandeepa Prabhu wrote:
>> On 8 October 2013 09:58, Sandeepa Prabhu <sandeepa.prabhu@linaro.org> wrote:
>>>>
>>>> Hi Sandeepa,
>>>>
>>>> The fedora 19 work is publicly available.  The following page talks about setting things up:
>>>>
>>>> http://fedoraproject.org/wiki/Architectures/ARM/AArch64/QuickStart
>>>>
>>>> -Will
>>> Hi Will,
>>>
>>> Thanks, I could get the console, but very slow.  Where can I look for
>>> kernel sources and the systemtap packages? Wanted to try my kernel
>>> changes on this and experiment systemtap, I can clone and create
>>> kernel devel branch on linaro git for kprobes, if this work out well.
>>>
>> This is single CPU version, has it been tested on 4-core models
>> (RTSM)? Responsiveness might improve with 4-core model [atleast LAMP
>> stack run faster]
>>
>>> Thanks,
>>> Sandeepa
> 
> Hi Sandeepa,
> 
> Yes, the Foundation model is very slow.  I have read that there has been some work to get aarch64 support in qemu, but I have not personally tried it:
> 
> http://news.opensuse.org/2013/10/01/suse-speeds-up-building-aarch64-software-in-qemu/
> 
> For systemtap your best bet is to use a git clone which has the current aarch64 patches:
> 
> git clone git://sourceware.org/git/systemtap.git
> 
> Then build (you may need to do a "yum-builddep systemtap" as root to pull in dependencies that systemtap needs to build):
> 
> cd systemtap
> ./configure --disable-docs
> make
> make install
> 
> At this point should have a systemtap that will build stuff for aarch64
> 
> For the current state of aarch64 package you can look at what is in http://arm-temp.ausil.us/pub/fedora-arm/stage4/
> Each of the packages is in a subdirectory.  You should be able to do something like the following to get the sources:
> 
> wget http://arm-temp.ausil.us/pub/fedora-arm/stage4/kernel-3.12.0-0.rc4.git0.1.x2.fc19/kernel-3.12.0-0.rc4.git0.1.x2.fc19.src.rpm
> rpm -Uvh kernel*.src.rpm
> cd rpmbuild/SPECS
> yum-builddep kernel
> rpmbuild -bp kernel.spec
> 
> Generally when fedora builds kernels it produces a /boot/config-`uname -r` file describing the configure used to build the kernel.  This is in the kernel rpm.  I don't recall if the kernel rpm is installed in the basic aarch64 image.  You may need to do a "yum install kernel" to get that installed on the machine.
> 
> -Will
> 


Hi Sandeepa,

I haven't personally tried it but some of the work on 32-bit arm to cross compile systemtap scripts might be adapted to build code for aarch64.  This is described on:

http://omappedia.org/wiki/Systemtap

-Will

Re: Regarding systemtap support for AArch64

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

On 10/02/2013 12:17 AM, Sandeepa Prabhu wrote:
> Hi all,
> 
> I have uploaded ARM64 kprobes work on Linaro public git:
> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
> boot-wrapper and fast model setup, though, not sure what it takes to
> build for fedora.
> 
> Will,
> 
> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
> model, are there instructions about how to get the packages and
> build/run them?
> 
> Thanks,
> Sandeepa

Hi Sandeepa,

I finally got a locally built aarch64 kernel with the kprobe patches
built and running on the simulator.  The SystemTap tapsets.cxx needed to be
patched to understand the aarch64 (the attached patch). With that patch the
SystemTap Beginner's guide smoke test example showed signs of life!
However, it took minutes for it to compile and run.

[wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
[sudo] password for wcohen: 
Pass 1: parsed user script and 92 library script(s) using 140528virt/24952res/2744shr/22868data kb, in 4740usr/160sys/5319real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 326804virt/106856res/3568shr/103752data kb, in 33860usr/14520sys/52942real ms.
Pass 3: translated to C into "/tmp/stap6Cbu2d/stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393_src.c" using 326804virt/109236res/5948shr/103752data kb, in 120usr/30sys/163real ms.
Pass 4: compiled C into "stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393.ko" in 117620usr/10440sys/141829real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 100usr/90sys/573real ms.
[wcohen@localhost systemtap]$ uname -a
Linux localhost 3.12.0-rc5+ #3 SMP Wed Oct 23 15:03:27 EDT 2013 aarch64 aarch64 aarch64 GNU/Linux

-Will

[-- Attachment #2: systemtap-aarch64-kprobes.patch --]
[-- Type: text/x-patch, Size: 2316 bytes --]

diff --git a/tapsets.cxx b/tapsets.cxx
index f729c1c..79ac6c5 100644
--- a/tapsets.cxx
+++ b/tapsets.cxx
@@ -2002,6 +2002,7 @@ validate_module_elf (Dwfl_Module *mod, const char *name,  base_query *q)
     case EM_S390: expect_machine = "s390"; break;
     case EM_IA_64: expect_machine = "ia64"; break;
     case EM_ARM: expect_machine = "arm*"; break;
+    case EM_AARCH64: expect_machine = "arm64"; break;
       // XXX: fill in some more of these
     default: expect_machine = "?"; break;
     }
@@ -5609,6 +5610,39 @@ struct sdt_uprobe_var_expanding_visitor: public var_expanding_visitor
       DRI ("sp", 13, SI);
       DRI ("lr", 14, SI);
       DRI ("pc", 15, SI);
+    } else if (elf_machine == EM_AARCH64) {
+      DRI ("x0", 0, DI); DRI ("w0", 0, SI);
+      DRI ("x1", 1, DI); DRI ("w1", 1, SI);
+      DRI ("x2", 2, DI); DRI ("w2", 2, SI);
+      DRI ("x3", 3, DI); DRI ("w3", 3, SI);
+      DRI ("x4", 4, DI); DRI ("w4", 4, SI);
+      DRI ("x5", 5, DI); DRI ("w5", 5, SI);
+      DRI ("x6", 6, DI); DRI ("w6", 6, SI);
+      DRI ("x7", 7, DI); DRI ("w7", 7, SI);
+      DRI ("x8", 8, DI); DRI ("w8", 8, SI);
+      DRI ("x9", 9, DI); DRI ("w9", 9, SI);
+      DRI ("x10", 10, DI); DRI ("w10", 10, SI);
+      DRI ("x11", 11, DI); DRI ("w11", 11, SI);
+      DRI ("x12", 12, DI); DRI ("w12", 12, SI);
+      DRI ("x13", 13, DI); DRI ("w13", 13, SI);
+      DRI ("x14", 14, DI); DRI ("w14", 14, SI);
+      DRI ("x15", 15, DI); DRI ("w15", 15, SI);
+      DRI ("x16", 16, DI); DRI ("w16", 16, SI);
+      DRI ("x17", 17, DI); DRI ("w17", 17, SI);
+      DRI ("x18", 18, DI); DRI ("w18", 18, SI);
+      DRI ("x19", 19, DI); DRI ("w19", 19, SI);
+      DRI ("x20", 20, DI); DRI ("w20", 20, SI);
+      DRI ("x21", 21, DI); DRI ("w21", 21, SI);
+      DRI ("x22", 22, DI); DRI ("w22", 22, SI);
+      DRI ("x23", 23, DI); DRI ("w23", 23, SI);
+      DRI ("x24", 24, DI); DRI ("w24", 24, SI);
+      DRI ("x25", 25, DI); DRI ("w25", 25, SI);
+      DRI ("x26", 26, DI); DRI ("w26", 26, SI);
+      DRI ("x27", 27, DI); DRI ("w27", 27, SI);
+      DRI ("x28", 28, DI); DRI ("w28", 28, SI);
+      DRI ("x29", 29, DI); DRI ("w29", 29, SI);
+      DRI ("x30", 30, DI); DRI ("w30", 30, SI);
+      DRI ("sp", 31, DI);
     } else if (arg_count) {
       /* permit this case; just fall back to dwarf */
     }

Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 24 October 2013 07:20, William Cohen <wcohen@redhat.com> wrote:
> On 10/02/2013 12:17 AM, Sandeepa Prabhu wrote:
>> Hi all,
>>
>> I have uploaded ARM64 kprobes work on Linaro public git:
>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>> boot-wrapper and fast model setup, though, not sure what it takes to
>> build for fedora.
>>
>> Will,
>>
>> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
>> model, are there instructions about how to get the packages and
>> build/run them?
>>
>> Thanks,
>> Sandeepa
>
> Hi Sandeepa,
>
> I finally got a locally built aarch64 kernel with the kprobe patches
> built and running on the simulator.  The SystemTap tapsets.cxx needed to be
> patched to understand the aarch64 (the attached patch). With that patch the
> SystemTap Beginner's guide smoke test example showed signs of life!
> However, it took minutes for it to compile and run.
>
> [wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
> [sudo] password for wcohen:
> Pass 1: parsed user script and 92 library script(s) using 140528virt/24952res/2744shr/22868data kb, in 4740usr/160sys/5319real ms.
> Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 326804virt/106856res/3568shr/103752data kb, in 33860usr/14520sys/52942real ms.
> Pass 3: translated to C into "/tmp/stap6Cbu2d/stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393_src.c" using 326804virt/109236res/5948shr/103752data kb, in 120usr/30sys/163real ms.
> Pass 4: compiled C into "stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393.ko" in 117620usr/10440sys/141829real ms.
> Pass 5: starting run.
> read performed
> Pass 5: run completed in 100usr/90sys/573real ms.
Hi Will,

Great to know this!, Thanks for the patch. Hope you have taken v2
version of kprobe patches, found here:
https://git.linaro.org/gitweb?p=people/sandeepa.prabhu/linux-aarch64.git;a=shortlog;h=refs/heads/arm64-kprobes-v2
v2 is cleaner, and implements recursive kprobes (like probing printk)
and also few fixes in missed-kprobe handling.
So, do you think we can measure kprobes performance running on v8 fast
models (slow model though :))? the milliseconds it shows for
completion will be accurate only if have real hardware right?

From this weekend thru next week, we are mostly under travel to Linaro
connect @Santa Clara, so may not spend more time on this, but we would
like to start validating the kprobes using systemtap and may get more
hands within Linaro to work with systemtap community. If anyone is
attending LCU this time, ping me and we can meet.

Cheers,
Sandeepa

> [wcohen@localhost systemtap]$ uname -a
> Linux localhost 3.12.0-rc5+ #3 SMP Wed Oct 23 15:03:27 EDT 2013 aarch64 aarch64 aarch64 GNU/Linux
>
> -Will

Re: Re: Re: Regarding systemtap support for AArch64

[Sandeepa Prabhu]

On 15 October 2013 15:09, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
> (2013/10/07 20:12), Sandeepa Prabhu wrote:
>>>>>>>>>  - Is it really need to use spinlock to protect break_hook?
>>>>>>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>>>>>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>>>>>>> in debug exception.
>>>>>>>
>>>>>>> I don't think we need to remove the breakpoint hooks after starting
>>>>>>> up the kernel. If we use the spinlock there, we'll pay a big cost
>>>>>>> because of the lock contention.
>>>>>> Not in kprobes. But kgdb can remove breakpoint handler and use same
>>>>>> API. or atleast while providing an api we should not assume race
>>>>>> cannot happen right?
>>>>>
>>>>> In that case, we'd better add a wrapper handler for kgdb so that
>>>>> the list isn't updated even if the kgdb removes its handler.
>>>>>
>>>>>> And there wont be much lock contention, i'ts only if the debug
>>>>>> framework (like kgdb) is wrapping-up, not is normal use-case.
>>>>>
>>>>> Hmm, it seems that the spinlock is locked while handling a breakpoint.
>>>>> This will cause a bad performance issue when we put many kprobes
>>>>> on SMP system.
>>>> arm maintainers prefer a reader/writer spin-locks, so there wont be
>>>> lock contention in debug path, each instance of kprobe hook trap (on
>>>> any CPU) would be a reader, not blocking.
>>>
>>> OK for the first step, and it eventually should be fixed to lockless.
>>> (depends on the performance improvement)
>> Hmm, is there a performance requirement for systemtap or perf? -like
>> how much time each test suite should consume etc?
>
> Basically, for the enterprise use, we aims to get less than 5% loss
> of runtime performance. Of course it depends on the configuration.
> This requirement comes from the usage of tracing, it's usually used
> as a "flight-recorder" in such system. For analyzing the root cause
> of the trouble, some fundamental events are always recorded into a
> memory buffer. When encountering a trouble, the buffer will be dumped,
> and trouble shooting team analyzes it.
>
> Thus, I'd like to make the performance overhead of tracing as
> small as possible.
Hmm, my worry is whether we can really measure and improve performance
or not -running on foundation model, do not have real hardware access
right now :(

>
> However, for debugging use, the performance degradation is not
> so important.
>
>> Want to know the acceptance criteria for systemtap or perf to say
>> 'kprobes/uprobes on an architecture' is complaint and good enough for
>> tracing?
>
> I think there is no such criteria. The overhead problem depends on the
> use-cases as I said above. If it is functional, it's enough to use by
> perf/ftrace ;) Performance optimization can be done afterwords.
>
> Thank you,
>
> --
> Masami HIRAMATSU
> IT Management Research Dept. Linux Technology Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>
>

Re: Regarding systemtap support for AArch64

[Masami Hiramatsu]

(2013/10/24 13:26), Sandeepa Prabhu wrote:
> On 15 October 2013 15:09, Masami Hiramatsu
> <masami.hiramatsu.pt@hitachi.com> wrote:
>> (2013/10/07 20:12), Sandeepa Prabhu wrote:
>>>>>>>>>>  - Is it really need to use spinlock to protect break_hook?
>>>>>>>>> Any cpu can remove breakpoint hooks right, and traversal happen in
>>>>>>>>> debug exception context so mutex are not safe (can sleep/schedule out)
>>>>>>>>> in debug exception.
>>>>>>>>
>>>>>>>> I don't think we need to remove the breakpoint hooks after starting
>>>>>>>> up the kernel. If we use the spinlock there, we'll pay a big cost
>>>>>>>> because of the lock contention.
>>>>>>> Not in kprobes. But kgdb can remove breakpoint handler and use same
>>>>>>> API. or atleast while providing an api we should not assume race
>>>>>>> cannot happen right?
>>>>>>
>>>>>> In that case, we'd better add a wrapper handler for kgdb so that
>>>>>> the list isn't updated even if the kgdb removes its handler.
>>>>>>
>>>>>>> And there wont be much lock contention, i'ts only if the debug
>>>>>>> framework (like kgdb) is wrapping-up, not is normal use-case.
>>>>>>
>>>>>> Hmm, it seems that the spinlock is locked while handling a breakpoint.
>>>>>> This will cause a bad performance issue when we put many kprobes
>>>>>> on SMP system.
>>>>> arm maintainers prefer a reader/writer spin-locks, so there wont be
>>>>> lock contention in debug path, each instance of kprobe hook trap (on
>>>>> any CPU) would be a reader, not blocking.
>>>>
>>>> OK for the first step, and it eventually should be fixed to lockless.
>>>> (depends on the performance improvement)
>>> Hmm, is there a performance requirement for systemtap or perf? -like
>>> how much time each test suite should consume etc?
>>
>> Basically, for the enterprise use, we aims to get less than 5% loss
>> of runtime performance. Of course it depends on the configuration.
>> This requirement comes from the usage of tracing, it's usually used
>> as a "flight-recorder" in such system. For analyzing the root cause
>> of the trouble, some fundamental events are always recorded into a
>> memory buffer. When encountering a trouble, the buffer will be dumped,
>> and trouble shooting team analyzes it.
>>
>> Thus, I'd like to make the performance overhead of tracing as
>> small as possible.
> Hmm, my worry is whether we can really measure and improve performance
> or not -running on foundation model, do not have real hardware access
> right now :(

Ah, right! :)
OK, so I think this should good to be done after real hardware is out.

Thank you!


-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/24/2013 12:19 AM, Sandeepa Prabhu wrote:
> On 24 October 2013 07:20, William Cohen <wcohen@redhat.com> wrote:
>> On 10/02/2013 12:17 AM, Sandeepa Prabhu wrote:
>>> Hi all,
>>>
>>> I have uploaded ARM64 kprobes work on Linaro public git:
>>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>>> boot-wrapper and fast model setup, though, not sure what it takes to
>>> build for fedora.
>>>
>>> Will,
>>>
>>> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
>>> model, are there instructions about how to get the packages and
>>> build/run them?
>>>
>>> Thanks,
>>> Sandeepa
>>
>> Hi Sandeepa,
>>
>> I finally got a locally built aarch64 kernel with the kprobe patches
>> built and running on the simulator.  The SystemTap tapsets.cxx needed to be
>> patched to understand the aarch64 (the attached patch). With that patch the
>> SystemTap Beginner's guide smoke test example showed signs of life!
>> However, it took minutes for it to compile and run.
>>
>> [wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
>> [sudo] password for wcohen:
>> Pass 1: parsed user script and 92 library script(s) using 140528virt/24952res/2744shr/22868data kb, in 4740usr/160sys/5319real ms.
>> Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 326804virt/106856res/3568shr/103752data kb, in 33860usr/14520sys/52942real ms.
>> Pass 3: translated to C into "/tmp/stap6Cbu2d/stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393_src.c" using 326804virt/109236res/5948shr/103752data kb, in 120usr/30sys/163real ms.
>> Pass 4: compiled C into "stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393.ko" in 117620usr/10440sys/141829real ms.
>> Pass 5: starting run.
>> read performed
>> Pass 5: run completed in 100usr/90sys/573real ms.
> Hi Will,
> 
> Great to know this!, Thanks for the patch. Hope you have taken v2
> version of kprobe patches, found here:
> https://git.linaro.org/gitweb?p=people/sandeepa.prabhu/linux-aarch64.git;a=shortlog;h=refs/heads/arm64-kprobes-v2
> v2 is cleaner, and implements recursive kprobes (like probing printk)
> and also few fixes in missed-kprobe handling.
> So, do you think we can measure kprobes performance running on v8 fast
> models (slow model though :))? the milliseconds it shows for
> completion will be accurate only if have real hardware right?
> 
> From this weekend thru next week, we are mostly under travel to Linaro
> connect @Santa Clara, so may not spend more time on this, but we would
> like to start validating the kprobes using systemtap and may get more
> hands within Linaro to work with systemtap community. If anyone is
> attending LCU this time, ping me and we can meet.
> 
> Cheers,
> Sandeepa

Hi Sandeepa,

The run above was with the old armv8 kprobes patches.  I just switched my kernel sources to the arm64-kprobes-v2 branch this morning and building a new kernel.  It is building with the minimal modules needed for the armv8 simulator, but it will take a while because I am building it on the armv8 simulator, so that all the kernel-devel support is available for systemtap.

The fedora 19 image I am using uses uefi to boot up.  There are additional patches that I needed to pull in from  https://github.com/mosalter/linux the armv8-uefi-latest branch to make a kernel that boot on uefi.  

It does appear to be able to read parameter values, but the $return isn't available:

[wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -e 'probe vfs.read {printf("read performed %s\n", $$parms$); exit()}'
Pass 1: parsed user script and 92 library script(s) using 140532virt/24956res/2744shr/22872data kb, in 4790usr/110sys/5315real ms.
Pass 2: analyzed script: 1 probe(s), 21 function(s), 3 embed(s), 0 global(s) using 326840virt/106908res/3580shr/103788data kb, in 34110usr/8180sys/45930real ms.
Pass 3: translated to C into "/tmp/stapso4oWX/stap_165197dc0524f0f70a7a3b68cb1cb26f_12311_src.c" using 326840virt/109304res/5976shr/103788data kb, in 180usr/10sys/200real ms.
Pass 4: compiled C into "stap_165197dc0524f0f70a7a3b68cb1cb26f_12311.ko" in 49890usr/3230sys/58657real ms.
Pass 5: starting run.
read performed file={.f_u={...}, .f_path={...}, .f_inode=0xffffffc04f5ff600, .f_op=0xffffffbffc0c05b8, .f_lock={...}, .f_sb_list_cpu=0, .f_count={...}, .f_flags=133122, .f_mode=31, .f_pos=0, .f_owner={...}, .f_cred=0xffffffc05113d9c0, .f_ra={...}, .f_version=0, .f_security=0xffffffc051158900, .private_data=0x0, .f_ep_links={...}, .f_tfile_llink={...}, .f_mapping=0xffffffc04f5ff750} buf="\x04" count=8196 pos=-273515643216
Pass 5: run completed in 120usr/70sys/508real ms.

However, $return doesn't work:

[wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -e 'probe vfs.read.return {printf("read performed %d\n", $return); exit()}'
[sudo] password for wcohen: 
Pass 1: parsed user script and 92 library script(s) using 140532virt/24956res/2744shr/22872data kb, in 4830usr/70sys/5316real ms.
semantic error: failed to retrieve return value location for vfs_read [man error::dwarf] (fs/read_write.c): identifier '$return' at <input>:1:54
        source: probe vfs.read.return {printf("read performed %d\n", $return); exit()}
                                                                     ^

Pass 2: analyzed script: 2 probe(s), 6 function(s), 6 embed(s), 0 global(s) using 326832virt/106912res/3584shr/103780data kb, in 34190usr/8060sys/45880real ms.
Pass 2: analysis failed.  [man error::pass2]

-Will



> 
>> [wcohen@localhost systemtap]$ uname -a
>> Linux localhost 3.12.0-rc5+ #3 SMP Wed Oct 23 15:03:27 EDT 2013 aarch64 aarch64 aarch64 GNU/Linux
>>
>> -Will

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/24/2013 12:19 AM, Sandeepa Prabhu wrote:
> On 24 October 2013 07:20, William Cohen <wcohen@redhat.com> wrote:
>> On 10/02/2013 12:17 AM, Sandeepa Prabhu wrote:
>>> Hi all,
>>>
>>> I have uploaded ARM64 kprobes work on Linaro public git:
>>> git://git.linaro.org/people/sandeepa.prabhu/linux-aarch64.git  Branch:
>>> kprobes_devel_v8.  Patches are published on LAKML too.  This is based
>>> on v8 upstream kernel (3.12-rc1) right now, and works with linaro
>>> boot-wrapper and fast model setup, though, not sure what it takes to
>>> build for fedora.
>>>
>>> Will,
>>>
>>> Is aarch64 fc19 port  public? I am interested in using fc on v8 fast
>>> model, are there instructions about how to get the packages and
>>> build/run them?
>>>
>>> Thanks,
>>> Sandeepa
>>
>> Hi Sandeepa,
>>
>> I finally got a locally built aarch64 kernel with the kprobe patches
>> built and running on the simulator.  The SystemTap tapsets.cxx needed to be
>> patched to understand the aarch64 (the attached patch). With that patch the
>> SystemTap Beginner's guide smoke test example showed signs of life!
>> However, it took minutes for it to compile and run.
>>
>> [wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
>> [sudo] password for wcohen:
>> Pass 1: parsed user script and 92 library script(s) using 140528virt/24952res/2744shr/22868data kb, in 4740usr/160sys/5319real ms.
>> Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 326804virt/106856res/3568shr/103752data kb, in 33860usr/14520sys/52942real ms.
>> Pass 3: translated to C into "/tmp/stap6Cbu2d/stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393_src.c" using 326804virt/109236res/5948shr/103752data kb, in 120usr/30sys/163real ms.
>> Pass 4: compiled C into "stap_3a0ef010f6cdfb2bdc3cac691a0f3c0e_1393.ko" in 117620usr/10440sys/141829real ms.
>> Pass 5: starting run.
>> read performed
>> Pass 5: run completed in 100usr/90sys/573real ms.
> Hi Will,
> 
> Great to know this!, Thanks for the patch. Hope you have taken v2
> version of kprobe patches, found here:
> https://git.linaro.org/gitweb?p=people/sandeepa.prabhu/linux-aarch64.git;a=shortlog;h=refs/heads/arm64-kprobes-v2
> v2 is cleaner, and implements recursive kprobes (like probing printk)
> and also few fixes in missed-kprobe handling.
> So, do you think we can measure kprobes performance running on v8 fast
> models (slow model though :))? the milliseconds it shows for
> completion will be accurate only if have real hardware right?
> 
> From this weekend thru next week, we are mostly under travel to Linaro
> connect @Santa Clara, so may not spend more time on this, but we would
> like to start validating the kprobes using systemtap and may get more
> hands within Linaro to work with systemtap community. If anyone is
> attending LCU this time, ping me and we can meet.
> 
> Cheers,
> Sandeepa
> 
>> [wcohen@localhost systemtap]$ uname -a
>> Linux localhost 3.12.0-rc5+ #3 SMP Wed Oct 23 15:03:27 EDT 2013 aarch64 aarch64 aarch64 GNU/Linux
>>
>> -Will


Hi Sandeepa,

I got the v2 versions of the kprobe patches and built and booted the new kernel. It functions and gives the same same results as the earlier version of the kprobe patches.  I was looking around to see why the $return doesn't work and came across the a similar thread for the arm from a couple years ago:

http://comments.gmane.org/gmane.linux.systemtap/17986

elfutils still need to have some additional aarch64 support included so that systemtap can find the location of the return value.

-Will

Re: Regarding systemtap support for AArch64

[William Cohen]

On 10/28/2013 10:02 AM, William Cohen wrote:

> Hi Sandeepa,
> 
> I got the v2 versions of the kprobe patches and built and booted the new kernel. It functions and gives the same same results as the earlier version of the kprobe patches.  I was looking around to see why the $return doesn't work and came across the a similar thread for the arm from a couple years ago:
> 
> http://comments.gmane.org/gmane.linux.systemtap/17986
> 
> elfutils still need to have some additional aarch64 support included so that systemtap can find the location of the return value.
> 
> -Will
> 

Hi Sandeepa,

With the helpful tips from Mark and Frank, I have a built systemtap on
aarch64 using the elfutils with aarch64 support from the elfutils git
repo.  Systemtap scripts using the $return compiles but the $return
value looks wrong.  For example below $return shouldn't be some huge
negative number:

$ sudo ../install/bin/stap -m return -k -v -e 'probe vfs.read.return {printf("read performed 0x%x\n", $return); exit()}'
Pass 1: parsed user script and 92 library script(s) using 140728virt/25036res/2784shr/22952data kb, in 4750usr/150sys/10805real ms.
Pass 2: analyzed script: 2 probe(s), 7 function(s), 6 embed(s), 0 global(s) using 323588virt/103604res/3676shr/100440data kb, in 40150usr/7910sys/105966real ms.
Pass 3: translated to C into "/tmp/stapmZsmxy/return_src.c" using 323588virt/105924res/5996shr/100440data kb, in 150usr/10sys/357real ms.
Pass 4: compiled C into "return.ko" in 41820usr/2890sys/100357real ms.
Pass 5: starting run.
[358010.304872] return: systemtap: 2.4/0.157, base: ffffffbffc0e8000, memory: 15data/21text/0ctx/2058net/11alloc kb, probes: 2
read performed 0xffffffc077a9a100
Pass 5: run completed in 40usr/150sys/3714real ms.
Keeping temporary directory "/tmp/stapmZsmxy"

The aarch64 call abi is described in:

http://infocenter.arm.com/help/topic/com.arm.doc.ihi0055b/IHI0055B_aapcs64.pdf

According to the call abi the simple integer value should be in
register "x0".  It looks in the generated c code for the module that
it is trying to access the return value with:

    { int64_t value = fetch_register (0); STAP_RETVALUE = value; }


As a sanity check I ran a "make -s check" on the locally built aarch64
elfutils to see if there were any issues with it.

FAIL: run-elflint-self.sh

Probably should implement the print_regs() to make debugging this easier.

-Will

An attempt for systemtap "make installcheck" AArch64

[William Cohen]

Hi Sandeepa,

Over the Long US Thankgiving weekend I tried the latest git checkout of systemtap on aarch64 to see how far the "make installcheck" would get on aarch64.  Yes, it was slow and didn't complete.  The results are posted on dejazilla:

https://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%3C529C9AC1.2060706%40redhat.com%3E%27

Some of the tests timed out probably because the simulator is so slow:

FAIL: at_var_tracepoint startup (timeout)
KFAIL: backtrace-unwindsyms (timeout) (PRMS: 10739)
FAIL: beginenderror (timeout)
FAIL: bz6503 (timeout)
FAIL: cmd_parse8: unexpected timeout
FAIL: cmd_parse15: timeout
FAIL: debugpath-bad (timeout1)
FAIL: debugpath-good (timeout2)

Tests that depend on back traces or time support are not expected to work right now.
KFAIL: backtrace (0 0) (PRMS: 10739)
FAIL: gtod (0)

The kernel crashed when running kallsyms_expand_symbol.stp or something soon after it.  Unfortunately, I did not take a look at the console before rebooting to see what caused the crash.

-Will

Re: An attempt for systemtap "make installcheck" AArch64

[Sandeepa Prabhu]

Hi Will,

I am still struggling with getting 'kernel and systemtap' builds on
fc19 EFI based image, so I can test only after succeed with builds.  I
had updated yum to use new "koji" server, but seeing that 'yum install
gcc' got me different version of toolchain: " gcc version 4.8.1
20130920 (Red Hat 4.8.1-10) (GCC)", and systemtap (+elfutils) did not
build as ./configure ... fails.  Moreover, kernel build just hangs the
model ssh/console indefinitely!. Any pointers from you would help me,
I am not sure of the dependencies between systemtap/elfutils, gcc,
kernel version, efi changes (I merged msalter "armv8-uefi-latest"
branch).


On 2 December 2013 21:14, William Cohen <wcohen@redhat.com> wrote:
> Hi Sandeepa,
>
> Over the Long US Thankgiving weekend I tried the latest git checkout of systemtap on aarch64 to see how far the "make installcheck" would get on aarch64.  Yes, it was slow and didn't complete.  The results are posted on dejazilla:
>
> https://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%3C529C9AC1.2060706%40redhat.com%3E%27
>
> Some of the tests timed out probably because the simulator is so slow:
>
> FAIL: at_var_tracepoint startup (timeout)
> KFAIL: backtrace-unwindsyms (timeout) (PRMS: 10739)
> FAIL: beginenderror (timeout)
> FAIL: bz6503 (timeout)
> FAIL: cmd_parse8: unexpected timeout
> FAIL: cmd_parse15: timeout
> FAIL: debugpath-bad (timeout1)
> FAIL: debugpath-good (timeout2)
Can't we have a test version with timeouts increased substantially
(20-times?) for running specially on foundation-model? Does it make
sense?

>
> Tests that depend on back traces or time support are not expected to work right now.
> KFAIL: backtrace (0 0) (PRMS: 10739)
> FAIL: gtod (0)
>
> The kernel crashed when running kallsyms_expand_symbol.stp or something soon after it.  Unfortunately, I did not take a look at the console before rebooting to see what caused the crash.
Hmm, I did not find kernel logs on dejazilla, we are aware of a
loop-hole in my implementation where kernel can end-up in crash,
especially in recursive exception and/or re-enter kprobe (though not
sure your crash is related to that). I am on the way to fix it would
provide new set of patches once the issue is resolved.

Thanks,
Sandeepa
>
> -Will

Re: An attempt for systemtap "make installcheck" AArch64

[William Cohen]

On 12/03/2013 12:23 AM, Sandeepa Prabhu wrote:
> Hi Will,
> 
> I am still struggling with getting 'kernel and systemtap' builds on
> fc19 EFI based image, so I can test only after succeed with builds.  I
> had updated yum to use new "koji" server, but seeing that 'yum install
> gcc' got me different version of toolchain: " gcc version 4.8.1
> 20130920 (Red Hat 4.8.1-10) (GCC)", and systemtap (+elfutils) did not
> build as ./configure ... fails.  Moreover, kernel build just hangs the
> model ssh/console indefinitely!. Any pointers from you would help me,
> I am not sure of the dependencies between systemtap/elfutils, gcc,
> kernel version, efi changes (I merged msalter "armv8-uefi-latest"
> branch).

Hi Sandeepa,


I am cc'ing msalter on this email because he can probably give you more/better information on building aarch64 kernel and which branch is best to pull things from to get a working aarch64 kernel.

I am currently using a UEFI based aarch64 Fedora 19 image.  There is a newer UEFI aarch64 fedora 19 image available at http://dmarlin.fedorapeople.org/fedora-arm/aarch64/F19-aarch64-efi.tar.xz. Using this might make the environment a bit closer to what the Red Hat aarch64 developers are currently using. It should also be easier to install the kernel.  Just need to write the Image and initramfs.img files into /boot/efi rather than copying them to the host.

I built my kernel November 7, so it has been a while since I have pull in changes. You could look through and see if there are any significant differences between the kernel configs. Below is a URL for config file that I used to build my local kernel:

http://people.redhat.com/wcohen/aarch64/aarch64_config

The version of gcc that I am using in my aarch64 image is:

gcc-4.8.1-10.x1.fc19.aarch64

I suspect the problem is if you checked out elfutils from a git repository you will need to go in to the elfutils source directory and do the following to create a ./configure file:

 autoreconf -f -v -i

Then when building systemtap you will need to include something like the following option the systemtap ./configure:

--with-elfutils=/home/wcohen/elfutil


-Will

Re: An attempt for systemtap "make installcheck" AArch64

[William Cohen]

On 12/03/2013 10:21 AM, William Cohen wrote:
> On 12/03/2013 12:23 AM, Sandeepa Prabhu wrote:
>> Hi Will,
>>
>> I am still struggling with getting 'kernel and systemtap' builds on
>> fc19 EFI based image, so I can test only after succeed with builds.  I
>> had updated yum to use new "koji" server, but seeing that 'yum install
>> gcc' got me different version of toolchain: " gcc version 4.8.1
>> 20130920 (Red Hat 4.8.1-10) (GCC)", and systemtap (+elfutils) did not
>> build as ./configure ... fails.  Moreover, kernel build just hangs the
>> model ssh/console indefinitely!. Any pointers from you would help me,
>> I am not sure of the dependencies between systemtap/elfutils, gcc,
>> kernel version, efi changes (I merged msalter "armv8-uefi-latest"
>> branch).
> 
> Hi Sandeepa,
> 
> 
> I am cc'ing msalter on this email because he can probably give you more/better information on building aarch64 kernel and which branch is best to pull things from to get a working aarch64 kernel.

Mark Salter suggested that armv8-uefi-v3.13rc is the prefered branch out of his git repo to work on.

I tried to do a merge between armv8-uefi-v3.13rc and the arm64-kprobes-v2 branches and there are a couple of files that conflict:

CONFLICT (content): Merge conflict in arch/arm64/kernel/module.c
CONFLICT (content): Merge conflict in arch/arm64/kernel/Makefile

-Will

Re: An attempt for systemtap "make installcheck" AArch64

[William Cohen]

On 12/03/2013 10:21 AM, William Cohen wrote:
> On 12/03/2013 12:23 AM, Sandeepa Prabhu wrote:
>> Hi Will,
>>
>> I am still struggling with getting 'kernel and systemtap' builds on
>> fc19 EFI based image, so I can test only after succeed with builds.  I
>> had updated yum to use new "koji" server, but seeing that 'yum install
>> gcc' got me different version of toolchain: " gcc version 4.8.1
>> 20130920 (Red Hat 4.8.1-10) (GCC)", and systemtap (+elfutils) did not
>> build as ./configure ... fails.  Moreover, kernel build just hangs the
>> model ssh/console indefinitely!. Any pointers from you would help me,
>> I am not sure of the dependencies between systemtap/elfutils, gcc,
>> kernel version, efi changes (I merged msalter "armv8-uefi-latest"
>> branch).
> 
> Hi Sandeepa,
> 
> 
> I am cc'ing msalter on this email because he can probably give you more/better information on building aarch64 kernel and which branch is best to pull things from to get a working aarch64 kernel.
> 
> I am currently using a UEFI based aarch64 Fedora 19 image.  There is a newer UEFI aarch64 fedora 19 image available at http://dmarlin.fedorapeople.org/fedora-arm/aarch64/F19-aarch64-efi.tar.xz. Using this might make the environment a bit closer to what the Red Hat aarch64 developers are currently using. It should also be easier to install the kernel.  Just need to write the Image and initramfs.img files into /boot/efi rather than copying them to the host.
> 

There arenot any instructions in the F19-aarch64-efi.tar.xz.  However it should be fairly simple setup/run.  Assuming that the Foundation simulator is unpacked in the same directory as F19-aarch64-efi.tar.xz:

xzcat < F19-aarch64-efi.tar.xz |tar xvf -
sudo ./efi-aarch64.sh

-Will

Re: An attempt for systemtap "make installcheck" AArch64

[William Cohen]

On 12/03/2013 11:36 AM, William Cohen wrote:
> On 12/03/2013 10:21 AM, William Cohen wrote:
>> On 12/03/2013 12:23 AM, Sandeepa Prabhu wrote:
>>> Hi Will,
>>>
>>> I am still struggling with getting 'kernel and systemtap' builds on
>>> fc19 EFI based image, so I can test only after succeed with builds.  I
>>> had updated yum to use new "koji" server, but seeing that 'yum install
>>> gcc' got me different version of toolchain: " gcc version 4.8.1
>>> 20130920 (Red Hat 4.8.1-10) (GCC)", and systemtap (+elfutils) did not
>>> build as ./configure ... fails.  Moreover, kernel build just hangs the
>>> model ssh/console indefinitely!. Any pointers from you would help me,
>>> I am not sure of the dependencies between systemtap/elfutils, gcc,
>>> kernel version, efi changes (I merged msalter "armv8-uefi-latest"
>>> branch).
>>
>> Hi Sandeepa,
>>
>>
>> I am cc'ing msalter on this email because he can probably give you more/better information on building aarch64 kernel and which branch is best to pull things from to get a working aarch64 kernel.
> 
> Mark Salter suggested that armv8-uefi-v3.13rc is the prefered branch out of his git repo to work on.
> 
> I tried to do a merge between armv8-uefi-v3.13rc and the arm64-kprobes-v2 branches and there are a couple of files that conflict:
> 
> CONFLICT (content): Merge conflict in arch/arm64/kernel/module.c
> CONFLICT (content): Merge conflict in arch/arm64/kernel/Makefile
> 
> -Will
> 


Hi Sandeepa,

Do you have a set of aarch64 kprobe patches that would apply cleanly to msalter's armv8-uefi-v3.13rc branch?

-Will

Re: An attempt for systemtap "make installcheck" AArch64

[Sandeepa Prabhu]

On 10 December 2013 02:05, William Cohen <wcohen@redhat.com> wrote:
> On 12/03/2013 11:36 AM, William Cohen wrote:
>> On 12/03/2013 10:21 AM, William Cohen wrote:
>>> On 12/03/2013 12:23 AM, Sandeepa Prabhu wrote:
>>>> Hi Will,
>>>>
>>>> I am still struggling with getting 'kernel and systemtap' builds on
>>>> fc19 EFI based image, so I can test only after succeed with builds.  I
>>>> had updated yum to use new "koji" server, but seeing that 'yum install
>>>> gcc' got me different version of toolchain: " gcc version 4.8.1
>>>> 20130920 (Red Hat 4.8.1-10) (GCC)", and systemtap (+elfutils) did not
>>>> build as ./configure ... fails.  Moreover, kernel build just hangs the
>>>> model ssh/console indefinitely!. Any pointers from you would help me,
>>>> I am not sure of the dependencies between systemtap/elfutils, gcc,
>>>> kernel version, efi changes (I merged msalter "armv8-uefi-latest"
>>>> branch).
>>>
>>> Hi Sandeepa,
>>>
>>>
>>> I am cc'ing msalter on this email because he can probably give you more/better information on building aarch64 kernel and which branch is best to pull things from to get a working aarch64 kernel.
>>
>> Mark Salter suggested that armv8-uefi-v3.13rc is the prefered branch out of his git repo to work on.
>>
>> I tried to do a merge between armv8-uefi-v3.13rc and the arm64-kprobes-v2 branches and there are a couple of files that conflict:
>>
>> CONFLICT (content): Merge conflict in arch/arm64/kernel/module.c
>> CONFLICT (content): Merge conflict in arch/arm64/kernel/Makefile
>>
>> -Will
>>
>
>
> Hi Sandeepa,
>
> Do you have a set of aarch64 kprobe patches that would apply cleanly to msalter's armv8-uefi-v3.13rc branch?
Hi Will,

Sorry for late reply - just back from vacation.  Have not migrated to
'armv8-uefi-v3.13rc' yet, and also am reworking kprobes patches to fix
some concurrency bugs. I would update you once have my new patches on
linaro git.

Thanks,
Sandeepa

>
> -Will

Re: An attempt for systemtap "make installcheck" AArch64

[William Cohen]

On 12/16/2013 01:06 AM, Sandeepa Prabhu wrote:
> On 10 December 2013 02:05, William Cohen <wcohen@redhat.com> wrote:

>> Hi Sandeepa,
>>
>> Do you have a set of aarch64 kprobe patches that would apply cleanly to msalter's armv8-uefi-v3.13rc branch?
> Hi Will,
> 
> Sorry for late reply - just back from vacation.  Have not migrated to
> 'armv8-uefi-v3.13rc' yet, and also am reworking kprobes patches to fix
> some concurrency bugs. I would update you once have my new patches on
> linaro git.
> 
> Thanks,
> Sandeepa
> 
>>
>> -Will

Hi Sandeepa,

Thanks so much for working on the revised set of patches.  I will be on the look out for the patches.  Our office has a shutdown December 24-Jan 1 and I will be out of the office then.

-Will