Re: Recent aarch64 kprobes and uprobes patch systemtap testing

[William Cohen <wcohen@redhat.com>]

On 12/16/2015 06:55 AM, Pratyush Anand wrote:
> On 11/12/2015:03:59:53 PM, William Cohen wrote:

>> Hi Dave and Pratyush,
>>
>> I did some more experimentation with the fslatency-nd and fsslow-nd tests to see what is going on.  The problem seems to be related to the return probes.  I have a small reproducer attached which runs fine on x86_64 machine.  However on aarch64 it has the bogus read because some of the argument registers have changed value
>>
>> # ../install/bin/stap ./aarch64_retkprobe_issue2.stp 
>> ERROR: read fault [man error::fault] at 0x0000000000000034 (addr) near operator '@cast' at ./aarch64_retkprobe_issue2.stp:13:7
>> pc : [<fffffe000021e37c>] lr : [<fffffe000021eb64>] pstate: 80000145
>> sp : fffffe00bad7be30
>> x29: fffffe00bad7be30 x28: fffffe00bad78000 
>> x27: fffffe0000912000 x26: 000000000000003f 
>> x25: 000000000000011d x24: 0000000000000015 
>> x23: 0000000080000000 x22: 000003fff82b9760 
>> x21: fffffe00bad7bec8 x20: 0000000000002004 
>> x19: fffffe01b716e100 x18: 000003fff82b8160 
>> x17: 000003ff849bf0a0 x16: fffffe000021f4a0 
>> x15: 0000000000000004 x14: 000003fff82bb910 
>> x13: 0000000000000001 x12: 000003ff7d75f200 
>> x11: 00000000003d0f00 x10: 000003ff849b7af4 
>> x9 : 0000000000000028 x8 : 0000000000000020 
>> x7 : fffffe00bc5c3600 x6 : 0000000000000000 
>> x5 : 0000000000000000 x4 : 0000000000000000 
>> x3 : fffffe00bad7bec8 x2 : 0000000000002004 
>> x1 : 000003fff82b9760 x0 : fffffe01b716e100 
>>
>> pc : [<fffffe000021e37c>] lr : [<fffffe000009fbe0>] pstate: 60000145
>> sp : fffffe00bad7be30
>> x29: fffffe00bad7be30 x28: fffffe00bad78000 
>> x27: fffffe0000912000 x26: 000000000000003f 
>> x25: 000000000000011d x24: 0000000000000015 
>> x23: 0000000080000000 x22: 000003fff82b9760 
>> x21: fffffe00bad7bec8 x20: 0000000000002004 
>> x19: fffffe01b716e100 x18: 000003fff82b8160 
>> x17: 000003ff849bf0a0 x16: fffffe000021f4a0 
>> x15: 0000000000000004 x14: 000003fff82bb910 
>> x13: 0000000000000001 x12: 000003ff7d75f200 
>> x11: 00000000003d0f00 x10: 000003ff849b7af4 
>> x9 : 0000000000000028 x8 : 0000000000000020 
>> x7 : fffffe00bc5c3600 x6 : 000003fff82b976c 
>> x5 : 000003fff82b976c x4 : 0000000000000000 
>> x3 : 0000000000000000 x2 : 0000000000000000 
>> x1 : 0000000000000000 x0 : 000000000000000c 
> 
> Although I am not sure, but this is what it seems to me:
> 
> First argument (file) is in x0, and which is 0xC in case of kretprobe. But, can
> x0 really be considered as 1st arg in case of kretprobe? 
> I think, x0 should have return value of __vfs_read() in case of kretprobe. So,
> 0xC could be the number of bytes read.
> 
> With perf I see:
> 
> # perf probe -k vmlinux __vfs_read_exit=__vfs_read%return file
> Semantic error :You can't specify local variable for kretprobe.
> 
> So, I am not sure what mechanism systemtap uses to get local variable in case of
> kretprobe.
> 
> Moreover, on x86 I see that loop exits after the 1st print_regs() only. So it
> means there was valid file->f_op->read() for the 1st file itself. If I comment
> "kprobe.function("__vfs_read")", then there is no print at all. It means, we are
> not hitting a case on x86 when callback was called for kretprobe and we had
> nonzero 1st argument.
> 
> ~Pratyush
> 

Hi Pratyush,

I found that the problem was that the systemtap scripts assumed x86_64 behavior where the arguments passed into the function are in memory and are still around on return.  For aarch64 (and other architectures that pass arguments via registers) the registers holding the values get clobbered.  I checked in a fix into the exampls to address this problem:

https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commit;h=3d0c2f452f09a64b800aabe68508f8f0183f0ea1

I also filed a systemtap bug to look for this issue in other places in systemtap:

https://sourceware.org/bugzilla/show_bug.cgi?id=19360

-Will