aarch64 systemtap testsuite results

aarch64 systemtap testsuite results

[William Cohen]

Some actual aarch64 hardware is now available and the testsuite can be run in a ressonable amount of time. The test results show signs of life, but there is definitely room for improvement:

 https://web.elastic.org/~dejazilla/viewsummary.php?variant=%3D%27aarch64-unknown-linux-gnu%27

There are going to be a lot of failures in these tests because the kernel did not have kprobe patches in it. There is also no uprobes support in the kernel.

Some other things observed when reviewing the results:

-a number of tests failed to build because of missing kernel header hypervisor.h. The kernel-devel rpm appeared to omit some needed header files

-tracepoints probe locations did not work.  The probe points are visible in "perf list" but "stap -L 'kernel.trace("*")'" returns nothing.

-aarch64 stack backtraces are not yet supported.

-Can't find  @cast(sock, "inet_sock", "kernel<net/ip.h>") for
 stap -v ./systemtap.base/ipaddr1.stp

-The preprocessor sets arch to "arm64" rather than "aarch64" "preprocessor basic ops" fails

-The nd_syscall probes points are not available

-SystemTap could not find locations of $prev or $next arguments for kernel.function("context_switch")




-Will

Re: aarch64 systemtap testsuite results

[Martin Cermak]

Hi Jakub,

At the moment, I believe, the most important issue is correct kprobes support, tracked as bug 1074123 in Red Hat bugzilla. Important points:

> There are two issues:
> (1) kprobes needs a 'blacklist' of kernel functions that can't be probed, b/c kprobe uses them -- infinite loop.  upstream has a V10 of a 7-patch series for this, so hopefully its close for us to use in near term.
> (2) Linaro kprobe developer agrees there is a bug in the multi-kprobe implementation.  I have hooked him up w/Will Cohen to get systemtap sources to use for their duplication, then debug & test efforts.

It'd be great to move on with this.

Kind Regards, 
Martin

----- Original Message -----
> Hello William,
> 
> many thanks for the update!
> 
> Is any of the reported issues something that Linaro should work on fixing?
> Here at Linaro we want to get Systemtap running in our automated testing
> loop for ARMv8 and we hope to fare better in pass-rates as we have ARM64
> kprobes implementation included in Linaro kernel already.
> 
> 
> On 7 May 2014 16:15, William Cohen <wcohen@redhat.com> wrote:
> 
> > Some actual aarch64 hardware is now available and the testsuite can be run
> > in a ressonable amount of time. The test results show signs of life, but
> > there is definitely room for improvement:
> >
> >
> > https://web.elastic.org/~dejazilla/viewsummary.php?variant=%3D%27aarch64-unknown-linux-gnu%27
> >
> > There are going to be a lot of failures in these tests because the kernel
> > did not have kprobe patches in it. There is also no uprobes support in the
> > kernel.
> >
> > Some other things observed when reviewing the results:
> >
> > -a number of tests failed to build because of missing kernel header
> > hypervisor.h. The kernel-devel rpm appeared to omit some needed header
> > files
> >
> > -tracepoints probe locations did not work.  The probe points are visible
> > in "perf list" but "stap -L 'kernel.trace("*")'" returns nothing.
> >
> > -aarch64 stack backtraces are not yet supported.
> >
> > -Can't find  @cast(sock, "inet_sock", "kernel<net/ip.h>") for
> >  stap -v ./systemtap.base/ipaddr1.stp
> >
> > -The preprocessor sets arch to "arm64" rather than "aarch64" "preprocessor
> > basic ops" fails
> >
> > -The nd_syscall probes points are not available
> >
> > -SystemTap could not find locations of $prev or $next arguments for
> > kernel.function("context_switch")
> >
> >
> >
> >
> > -Will
> >
> 
> 
> 
> --
> With best regards,
> 
> Jakub Pavelek
> 
> Kernel working group and Android engineering project manager
> Linaro.org│ Open source software for ARM SoCs
> 

Re: aarch64 systemtap testsuite results

[Sandeepa Prabhu]

On 14 May 2014 14:44, Martin Cermak <mcermak@redhat.com> wrote:
> Hi Jakub,
>
> At the moment, I believe, the most important issue is correct kprobes support, tracked as bug 1074123 in Red Hat bugzilla. Important points:
>
>> There are two issues:
>> (1) kprobes needs a 'blacklist' of kernel functions that can't be probed, b/c kprobe uses them -- infinite loop.  upstream has a V10 of a 7-patch series for this, so hopefully its close for us to use in near term.
Those patches are close to upstream, and expect  changes in arm64
(arch code) kprobes code as well to adapt these.

>> (2) Linaro kprobe developer agrees there is a bug in the multi-kprobe implementation.  I have hooked him up w/Will Cohen to get systemtap sources to use for their duplication, then debug & test efforts.
Yes, mainly since the failure summary reported hang while probing
read* syscalls, and I suppose none of the read system calls should be
in blacklist! as those calls wouldn't be invoked from kprobes or debug
exceptions. So I suspect a bug in my kprobes implementation, even
though I had tested for multi- and recursive probes on models, I have
not covered exhaustive tests like systemtap on real hw.

I am working with Will and Don on another e-mail thread, hope to
root-cause the issue sometimes soon :-)

Cheers,
~Sandeepa
>
> It'd be great to move on with this.
>
> Kind Regards,
> Martin
>
> ----- Original Message -----
>> Hello William,
>>
>> many thanks for the update!
>>
>> Is any of the reported issues something that Linaro should work on fixing?
>> Here at Linaro we want to get Systemtap running in our automated testing
>> loop for ARMv8 and we hope to fare better in pass-rates as we have ARM64
>> kprobes implementation included in Linaro kernel already.
>>
>>
>> On 7 May 2014 16:15, William Cohen <wcohen@redhat.com> wrote:
>>
>> > Some actual aarch64 hardware is now available and the testsuite can be run
>> > in a ressonable amount of time. The test results show signs of life, but
>> > there is definitely room for improvement:
>> >
>> >
>> > https://web.elastic.org/~dejazilla/viewsummary.php?variant=%3D%27aarch64-unknown-linux-gnu%27
>> >
>> > There are going to be a lot of failures in these tests because the kernel
>> > did not have kprobe patches in it. There is also no uprobes support in the
>> > kernel.
>> >
>> > Some other things observed when reviewing the results:
>> >
>> > -a number of tests failed to build because of missing kernel header
>> > hypervisor.h. The kernel-devel rpm appeared to omit some needed header
>> > files
>> >
>> > -tracepoints probe locations did not work.  The probe points are visible
>> > in "perf list" but "stap -L 'kernel.trace("*")'" returns nothing.
>> >
>> > -aarch64 stack backtraces are not yet supported.
>> >
>> > -Can't find  @cast(sock, "inet_sock", "kernel<net/ip.h>") for
>> >  stap -v ./systemtap.base/ipaddr1.stp
>> >
>> > -The preprocessor sets arch to "arm64" rather than "aarch64" "preprocessor
>> > basic ops" fails
>> >
>> > -The nd_syscall probes points are not available
>> >
>> > -SystemTap could not find locations of $prev or $next arguments for
>> > kernel.function("context_switch")
>> >
>> >
>> >
>> >
>> > -Will
>> >
>>
>>
>>
>> --
>> With best regards,
>>
>> Jakub Pavelek
>>
>> Kernel working group and Android engineering project manager
>> Linaro.org│ Open source software for ARM SoCs
>>

Re: aarch64 systemtap testsuite results

[William Cohen]

On 05/14/2014 03:24 AM, Jakub Pavelek wrote:
> Hello William,
> 
> many thanks for the update! 
> 
> Is any of the reported issues something that Linaro should work on fixing? Here at Linaro we want to get Systemtap running in our automated testing loop for ARMv8 and we hope to fare better in pass-rates as we have ARM64 kprobes implementation included in Linaro kernel already.
> 
> 

Hi Jakub,

We have a kernel with kprobe patches running on a machine.  However, it seems pretty easy to crash the machine with scripts using the kprobes.  I have a list scripts to try exercise things in different ways to get a better idea how things are failing.  So far it seems that a kprobe firing once works.  When a single probe fires multiple times with an empty probe handler there isn't a problem either.  However, if the probe that does printing in the probe handler then things go wrong.  This might be a problem in the code that systemtap is generating or the systemtap runtime.  Below are the script suggestions to exercise kprobes.

-Will



Start with the very simple single probe:

  stap -kve 'probe syscall.read {printf("hey\n"); exit()}'

Have the same probe point fire twice:

  stap -kve 'global x; probe syscall.read {x++; printf("hey %d\n", x); if (x>1) exit()}'

Have the same probe point fire multiple times with empty body:

  stap -kve 'probe syscall.read {} probe timer.s(1){exit()}'

Have the same probe point fire multiple times:

  stap -kve 'probe syscall.read {printf("hey\n")} probe timer.s(1){exit()}'

Have two probes points fire multiple times with empty body:

  stap -kve 'probe syscall.read, syscall.write {} probe timer.s(1){exit()}'

Have two probes points fire multiple times:

  stap -kve 'probe syscall.read, syscall.write {printf("%s\n",pn())} probe timer.s(1){exit()}'

Have many probe points fire multiple times with empty body:

  stap -kve 'probe syscall.* {} probe timer.s(1){exit()}'


> On 7 May 2014 16:15, William Cohen <wcohen@redhat.com <mailto:wcohen@redhat.com>> wrote:
> 
>     Some actual aarch64 hardware is now available and the testsuite can be run in a ressonable amount of time. The test results show signs of life, but there is definitely room for improvement:
> 
>      https://web.elastic.org/~dejazilla/viewsummary.php?variant=%3D%27aarch64-unknown-linux-gnu%27
> 
>     There are going to be a lot of failures in these tests because the kernel did not have kprobe patches in it. There is also no uprobes support in the kernel.
> 
>     Some other things observed when reviewing the results:
> 
>     -a number of tests failed to build because of missing kernel header hypervisor.h. The kernel-devel rpm appeared to omit some needed header files
> 
>     -tracepoints probe locations did not work.  The probe points are visible in "perf list" but "stap -L 'kernel.trace("*")'" returns nothing.
> 
>     -aarch64 stack backtraces are not yet supported.
> 
>     -Can't find  @cast(sock, "inet_sock", "kernel<net/ip.h>") for
>      stap -v ./systemtap.base/ipaddr1.stp
> 
>     -The preprocessor sets arch to "arm64" rather than "aarch64" "preprocessor basic ops" fails
> 
>     -The nd_syscall probes points are not available
> 
>     -SystemTap could not find locations of $prev or $next arguments for kernel.function("context_switch")
> 
> 
> 
> 
>     -Will
> 
> 
> 
> 
> -- 
> With best regards,
> 
> Jakub Pavelek
> 
> Kernel working group and Android engineering project manager
> Linaro.org│ Open source software for ARM SoCs
> 

Re: aarch64 systemtap testsuite results

[Sandeepa Prabhu]

On 15 May 2014 00:14, William Cohen <wcohen@redhat.com> wrote:
> On 05/14/2014 03:24 AM, Jakub Pavelek wrote:
>> Hello William,
>>
>> many thanks for the update!
>>
>> Is any of the reported issues something that Linaro should work on fixing? Here at Linaro we want to get Systemtap running in our automated testing loop for ARMv8 and we hope to fare better in pass-rates as we have ARM64 kprobes implementation included in Linaro kernel already.
>>
>>
>
> Hi Jakub,
>
> We have a kernel with kprobe patches running on a machine.  However, it seems pretty easy to crash the machine with scripts using the kprobes.  I have a list scripts to try exercise things in different ways to get a better idea how things are failing.  So far it seems that a kprobe firing once works.  When a single probe fires multiple times with an empty probe handler there isn't a problem either.  However, if the probe that does printing in the probe handler then things go wrong.  This might be a problem in the code that systemtap is generating or the systemtap runtime.  Below are the script suggestions to exercise kprobes.
>
> -Will
>
>
>
> Start with the very simple single probe:
>
>   stap -kve 'probe syscall.read {printf("hey\n"); exit()}'
>
> Have the same probe point fire twice:
>
>   stap -kve 'global x; probe syscall.read {x++; printf("hey %d\n", x); if (x>1) exit()}'
>
> Have the same probe point fire multiple times with empty body:
>
>   stap -kve 'probe syscall.read {} probe timer.s(1){exit()}'
>
> Have the same probe point fire multiple times:
>
>   stap -kve 'probe syscall.read {printf("hey\n")} probe timer.s(1){exit()}'
>
> Have two probes points fire multiple times with empty body:
>
>   stap -kve 'probe syscall.read, syscall.write {} probe timer.s(1){exit()}'
>
> Have two probes points fire multiple times:
>
>   stap -kve 'probe syscall.read, syscall.write {printf("%s\n",pn())} probe timer.s(1){exit()}'
>
> Have many probe points fire multiple times with empty body:
>
>   stap -kve 'probe syscall.* {} probe timer.s(1){exit()}'
>
Hi Will,

Thanks for the steps, most of these cases had been tested with
loadable kernel modules, like multiple probes and recursive probe hits
etc. I had not tested more than 3 simultaneous probe points though!

~Sandeepa

>
>> On 7 May 2014 16:15, William Cohen <wcohen@redhat.com <mailto:wcohen@redhat.com>> wrote:
>>
>>     Some actual aarch64 hardware is now available and the testsuite can be run in a ressonable amount of time. The test results show signs of life, but there is definitely room for improvement:
>>
>>      https://web.elastic.org/~dejazilla/viewsummary.php?variant=%3D%27aarch64-unknown-linux-gnu%27
>>
>>     There are going to be a lot of failures in these tests because the kernel did not have kprobe patches in it. There is also no uprobes support in the kernel.
>>
>>     Some other things observed when reviewing the results:
>>
>>     -a number of tests failed to build because of missing kernel header hypervisor.h. The kernel-devel rpm appeared to omit some needed header files
>>
>>     -tracepoints probe locations did not work.  The probe points are visible in "perf list" but "stap -L 'kernel.trace("*")'" returns nothing.
>>
>>     -aarch64 stack backtraces are not yet supported.
>>
>>     -Can't find  @cast(sock, "inet_sock", "kernel<net/ip.h>") for
>>      stap -v ./systemtap.base/ipaddr1.stp
>>
>>     -The preprocessor sets arch to "arm64" rather than "aarch64" "preprocessor basic ops" fails
>>
>>     -The nd_syscall probes points are not available
>>
>>     -SystemTap could not find locations of $prev or $next arguments for kernel.function("context_switch")
>>
>>
>>
>>
>>     -Will
>>
>>
>>
>>
>> --
>> With best regards,
>>
>> Jakub Pavelek
>>
>> Kernel working group and Android engineering project manager
>> Linaro.org│ Open source software for ARM SoCs
>>
>

Re: aarch64 systemtap testsuite results

[William Cohen]

On 05/15/2014 01:32 AM, Sandeepa Prabhu wrote:
> On 15 May 2014 00:14, William Cohen <wcohen@redhat.com> wrote:
>> On 05/14/2014 03:24 AM, Jakub Pavelek wrote:
>>> Hello William,
>>>
>>> many thanks for the update!
>>>
>>> Is any of the reported issues something that Linaro should work on fixing? Here at Linaro we want to get Systemtap running in our automated testing loop for ARMv8 and we hope to fare better in pass-rates as we have ARM64 kprobes implementation included in Linaro kernel already.
>>>
>>>
>>
>> Hi Jakub,
>>
>> We have a kernel with kprobe patches running on a machine.  However, it seems pretty easy to crash the machine with scripts using the kprobes.  I have a list scripts to try exercise things in different ways to get a better idea how things are failing.  So far it seems that a kprobe firing once works.  When a single probe fires multiple times with an empty probe handler there isn't a problem either.  However, if the probe that does printing in the probe handler then things go wrong.  This might be a problem in the code that systemtap is generating or the systemtap runtime.  Below are the script suggestions to exercise kprobes.
>>
>> -Will
>>
>>
>>
>> Start with the very simple single probe:
>>
>>   stap -kve 'probe syscall.read {printf("hey\n"); exit()}'
>>
>> Have the same probe point fire twice:
>>
>>   stap -kve 'global x; probe syscall.read {x++; printf("hey %d\n", x); if (x>1) exit()}'
>>
>> Have the same probe point fire multiple times with empty body:
>>
>>   stap -kve 'probe syscall.read {} probe timer.s(1){exit()}'
>>
>> Have the same probe point fire multiple times:
>>
>>   stap -kve 'probe syscall.read {printf("hey\n")} probe timer.s(1){exit()}'
>>
>> Have two probes points fire multiple times with empty body:
>>
>>   stap -kve 'probe syscall.read, syscall.write {} probe timer.s(1){exit()}'
>>
>> Have two probes points fire multiple times:
>>
>>   stap -kve 'probe syscall.read, syscall.write {printf("%s\n",pn())} probe timer.s(1){exit()}'
>>
>> Have many probe points fire multiple times with empty body:
>>
>>   stap -kve 'probe syscall.* {} probe timer.s(1){exit()}'
>>
> Hi Will,
> 
> Thanks for the steps, most of these cases had been tested with
> loadable kernel modules, like multiple probes and recursive probe hits
> etc. I had not tested more than 3 simultaneous probe points though!
> 
> ~Sandeepa
> 

Hi Sandeepa,

Is the aarch64 kprobes testing using the linux kernel modules built with  CONFIG_KPROBES_SANITY_TEST=y?  Or is there some other test code?  If it is some other code base could you provide a pointer to where it is located? Having non-systemtap based exercise of the kprobes would be good. It is possible there is something going on in the code that systemtap is generating that is causing the problems.  There is some arch specific code in the systemtap runtime and it is possible that it is incorrect.

-Will

Re: aarch64 systemtap testsuite results

[Sandeepa Prabhu]

[-- Attachment #1: Type: text/plain, Size: 3390 bytes --]

On 15 May 2014 19:14, William Cohen <wcohen@redhat.com> wrote:
> On 05/15/2014 01:32 AM, Sandeepa Prabhu wrote:
>> On 15 May 2014 00:14, William Cohen <wcohen@redhat.com> wrote:
>>> On 05/14/2014 03:24 AM, Jakub Pavelek wrote:
>>>> Hello William,
>>>>
>>>> many thanks for the update!
>>>>
>>>> Is any of the reported issues something that Linaro should work on fixing? Here at Linaro we want to get Systemtap running in our automated testing loop for ARMv8 and we hope to fare better in pass-rates as we have ARM64 kprobes implementation included in Linaro kernel already.
>>>>
>>>>
>>>
>>> Hi Jakub,
>>>
>>> We have a kernel with kprobe patches running on a machine.  However, it seems pretty easy to crash the machine with scripts using the kprobes.  I have a list scripts to try exercise things in different ways to get a better idea how things are failing.  So far it seems that a kprobe firing once works.  When a single probe fires multiple times with an empty probe handler there isn't a problem either.  However, if the probe that does printing in the probe handler then things go wrong.  This might be a problem in the code that systemtap is generating or the systemtap runtime.  Below are the script suggestions to exercise kprobes.
>>>
>>> -Will
>>>
>>>
>>>
>>> Start with the very simple single probe:
>>>
>>>   stap -kve 'probe syscall.read {printf("hey\n"); exit()}'
>>>
>>> Have the same probe point fire twice:
>>>
>>>   stap -kve 'global x; probe syscall.read {x++; printf("hey %d\n", x); if (x>1) exit()}'
>>>
>>> Have the same probe point fire multiple times with empty body:
>>>
>>>   stap -kve 'probe syscall.read {} probe timer.s(1){exit()}'
>>>
>>> Have the same probe point fire multiple times:
>>>
>>>   stap -kve 'probe syscall.read {printf("hey\n")} probe timer.s(1){exit()}'
>>>
>>> Have two probes points fire multiple times with empty body:
>>>
>>>   stap -kve 'probe syscall.read, syscall.write {} probe timer.s(1){exit()}'
>>>
>>> Have two probes points fire multiple times:
>>>
>>>   stap -kve 'probe syscall.read, syscall.write {printf("%s\n",pn())} probe timer.s(1){exit()}'
>>>
>>> Have many probe points fire multiple times with empty body:
>>>
>>>   stap -kve 'probe syscall.* {} probe timer.s(1){exit()}'
>>>
>> Hi Will,
>>
>> Thanks for the steps, most of these cases had been tested with
>> loadable kernel modules, like multiple probes and recursive probe hits
>> etc. I had not tested more than 3 simultaneous probe points though!
>>
>> ~Sandeepa
>>
>
> Hi Sandeepa,
>
> Is the aarch64 kprobes testing using the linux kernel modules built with  CONFIG_KPROBES_SANITY_TEST=y?  Or is there some other test code?  If it is some other code base could you provide a pointer to where it is located? Having non-systemtap based exercise of the kprobes would be good. It is possible there is something going on in the code that systemtap is generating that is causing the problems.  There is some arch specific code in the systemtap runtime and it is possible that it is incorrect.
>
Hi Will,

No, I am using sample modules from "KERNELDIR/samples/kprobes/" and I
modified a bit. Attached the modules sources since it's not anywhere
on public git. Compared to systemtap tests, these tests are pretty
basic and doesn't cover much. Let me know if it useful.

Thanks,
Sandeepa

> -Will

[-- Attachment #2: kprobes_simple_test.tar.gz --]
[-- Type: application/x-gzip, Size: 1928 bytes --]