Re: Recent aarch64 kprobes and uprobes patch systemtap testing

[David Long <dave.long@linaro.org>]

On 12/10/2015 11:19 PM, Pratyush Anand wrote:
> On 10/12/2015:04:12:30 PM, David Long wrote:
>> On 12/10/2015 03:24 PM, William Cohen wrote:
>>> Hi All,
>>>
>>> Dave Long and Pratyush Anand have been working on kprobe and uprobe patches for aarch64.  I have built a local version the uprobe/upstream_arm64_devel branch of https://github.com/pratyushanand/linux which includes those patches in a linux-4.4.0-rc3 kernel.
>>>
>>> The tests seemed to run fairly well and the results have been uploaded to dejazilla:
>>>
>>> https://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%3C56698DCC.3090207%40redhat.com%3E%27
>>>
>>> 		=== systemtap Summary ===
>>>
>>> # of expected passes		6096
>>> # of unexpected failures	111
>>> # of unexpected successes	2
>>> # of expected failures		333
>>> # of unknown successes		2
>>> # of known failures		89
>>> # of untested testcases		97
>>> # of unsupported tests		27
>>> runtest completed at Thu Dec 10 05:54:32 2015
>>>
>>>
>>> There are still some areas needing for for aarch64 such as stack backtrace support.
>>>
>>> The following failure looks suspect because the child process died:
>>>
>>>
>>> spawn stap -g ./systemtap.examples/process/threadstacks.stp -Gsize=65536 -c /root/systemtap_write/systemtap/testsuite/pthread_stacks.x 1024 0 -d /root/systemtap_write/systemtap/testsuite/pthread_stacks.x
>>>
>>> pthread_stacks.x: ./systemtap.base/pthread_stacks.c:67: main: Assertion `rc == 0' failed.
>>>
>>> WARNING: Child process exited with signal 6 (Aborted)
>>>
>>> pthread_stacks.[3567] overwrote __default_stacksize@0x3ffb3be4338 (8388608->65536)
>>>
>>> WARNING: /root/systemtap_write/install/bin/staprun exited with status: 1
>>>
>>> Pass 5: run failed.  [man error::pass5]
>>>
>>> FAIL: pthread_stacks -Gsize (0 0)
>>>
>>> The fslatency-nd and fsslower-nd tests need further investigation:
>>>
>>> PASS: ./systemtap.examples/lwtools/fslatency-nd build
>>> meta taglines 'test_installcheck: stap fslatency-nd.stp 1 1' tag 'test_installcheck' value 'stap fslatency-nd.stp 1 1'
>>> attempting command stap fslatency-nd.stp 1 1
>>> OUT ERROR: read fault [man error::fault] at 0x0000000000000034 (addr) near operator '@cast' at fslatency-nd.stp:66:15
>>> Tracing FS sync reads and writes... Output every 1 secs.
>>> WARNING: Number of errors: 1, skipped probes: 1
>>> WARNING: /root/systemtap_write/install/bin/staprun exited with status: 1
>>> Pass 5: run failed.  [man error::pass5]
>>> child process exited abnormally
>>> RC 1
>>> FAIL: ./systemtap.examples/lwtools/fslatency-nd run
>>>
>>> PASS: ./systemtap.examples/lwtools/fsslower-nd build
>>> meta taglines 'test_installcheck: stap fsslower-nd.stp -c "sleep 1"' tag 'test_installcheck' value 'stap fsslower-nd.stp -c "sleep 1"'
>>> attempting command stap fsslower-nd.stp -c "sleep 1"
>>> OUT ERROR: read fault [man error::fault] at 0x0000000000000034 (addr) near operator '@cast' at fsslower-nd.stp:68:15
>>> Tracing FS sync reads and writes slower than 10 ms... Hit Ctrl-C to end.
>>> TIME     PID    COMM             FUNC           SIZE     LAT(ms)
>>> WARNING: Number of errors: 1, skipped probes: 1
>>> WARNING: /root/systemtap_write/install/bin/staprun exited with status: 1
>>> Pass 5: run failed.  [man error::pass5]
>>> child process exited abnormally
>>> RC 1
>>> FAIL: ./systemtap.examples/lwtools/fsslower-nd run
>>>
>>>
>>> Also a number of network tests failed like the following
>>>
>>>
>>> TEST PWD=/root/systemtap_write/systemtap/testsuite/systemtap.examples/network
>>> meta taglines 'test_check: stap -g -p4 netfilter_drop.stp TCP 1' tag 'test_check' value 'stap -g -p4 netfilter_drop.stp TCP 1'
>>> attempting command stap -g -p4 netfilter_drop.stp TCP 1
>>> OUT /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2731:1: error: initialization from incompatible pointer type [-Werror]
>>>   .hook = enter_netfilter_probe_0,
>>>   ^
>>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2731:1: error: (near initialization for 'netfilter_opts_0.hook') [-Werror]
>>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2732:1: error: unknown field 'owner' specified in initializer
>>>   .owner = THIS_MODULE,
>>>   ^
>>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2732:1: error: initialization from incompatible pointer type [-Werror]
>>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2732:1: error: (near initialization for 'netfilter_opts_0.dev') [-Werror]
>>> cc1: all warnings being treated as errors
>>> make[4]: *** [/tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.o] Error 1
>>> make[3]: *** [_module_/tmp/stapbIEqFl] Error 2
>>> WARNING: kbuild exited with status: 2
>>> Pass 4: compilation failed.  [man error::pass4]
>>> child process exited abnormally
>>> RC 1
>>> FAIL: ./systemtap.examples/network/netfilter_drop build
>>>
>>> -Will
>>
>>
>> Cool. Wish I could make sense of systemtap error messages.
>>
>>   At Will Deacon's suggested I tested probing the instruction in
>> __copy_to_user that can cause a captured kernel exception when an
>> application passes in a bad buffer address.  Unfortunately the result was a
>> hang.  So copy_to/from user is going to have to be blacklisted for now,
>> unless there turns out to be a simple fix. I'm worried there might be other
>> places in the kernel where an otherwise probeable instruction might be
>> expected to generate an exception.
>
> There are many arm64 specific functions which need blacklisting. I have them
> here.
>
> https://github.com/pratyushanand/linux/commit/4098b5ad2c67bf4c375981fc68793f44af005eb9
> https://github.com/pratyushanand/linux/commit/df3e76cbf70a8e1af42951d4b30587f022d25938
>
> I think uprobe_pre/post_sstep_notifier() should also be blacklisted.
>
> https://github.com/pratyushanand/linux/commit/99c89512931a46582d2f026b7288c895b8ef320c
>
> Certainly, there could be some more functions which need kprobe blacklisting.
> Because I remember, I had kprobe at every function of kallsyms on a x86
> platform, and it had crashed.
>
> ~Pratyush
>

OK, it sounds like this has been just a "best effort" on x86 then?  I'll 
blacklist the copy functions and move on.

-dl