Re: exercising current aarch64 kprobe support with systemtap

[William Cohen <wcohen@redhat.com>]

On 08/04/2016 10:35 AM, Pratyush Anand wrote:
> Hi Will,
> 
> On 04/08/2016:09:56:45 AM, William Cohen wrote:
>> On 08/04/2016 12:42 AM, Pratyush Anand wrote:
>>> Hi Will,
>>>
>>> On 03/08/2016:01:39:47 PM, William Cohen wrote:
>>>> On 08/03/2016 09:13 AM, Pratyush Anand wrote:
>>>>> On 07/07/2016:03:58:37 PM, Frank Ch. Eigler wrote:
>>>>>> David Long <dave.long@linaro.org> writes:
>>>>>>
>>>>>>> [...]
>>>>>>>> - bug_handler() calls report_bug() which calls __warn()
>>>>>>>> - __warn() does lot of pr_warn()  which invokes print_worker_info()
>>>>>>>> where we have a kprobe instrumented.
>>>>>>>> - Therefore, we are encountering this issue.
>>>>>
>>>>> Hi Will,
>>>>>
>>>>> Can you please cross-check if following branch works perfectly with
>>>>> kprobes_onthefly.exp and other systemtap tests.
>>>>>
>>>>> https://github.com/pratyushanand/linux/tree/uprobe/upstream_arm64_devel_v1.1
>>>>>
>>>>> Following patch in above branch should solve this issue.
>>>>> https://github.com/pratyushanand/linux/commit/d0dcc6477f1279ab0bd99aefc30efdecb16c586e
>>>>>
>>>>> However, I am not yet sure that above modification could be the best solution,
>>>>> so discussing at arm kernel list.
>>>>>
>>>>> ~Pratyush
>>>>>
>>>>
>>>> I have an AMD seattle machine set up Fedora24, the upstream_arm64_devel_v1.1 branch kernel locally built, and a locally built checkout of systemtap (systemtap rpm in fc24 doesn't generate models for linux 4.6 and newer kernels).  Tried to run the systemtap tests with:
>>>>
>>>>  make installcheck RUNTESTFLAGS="--debug systemtap.onthefly/kprobes_onthefly.exp"
>>>>
>>>> However at some point the kernel starts having problems:
>>>>
>>>> http://paste.stg.fedoraproject.org/5375/
>>>
>>> Yes, this is what you were getting with earlier code as well, but now it is not
>>> going to infinite unexpected EL1, so at least proposed kprobe improvement seems
>>> fine to me.
>>>
>>> In this failing test we are getting oom and the oom_killer is called. So,
>>> I think, this is another point of investigation that why this OOM occurs. 
>>>
>>> ~Pratyush
>>>
>>
>> Hi,
>>
>> The OOM errors came before the otf_stress_hard_iter_5000 test that previous triggered the infinite unexpected EL1, so can't really say that the proposed patch has fixed the problem.
> 
> Yes, yes, previously also we were getting OOM, and then that OOM was triggering
> infinite unexpected EL1, because OOM message uses WARN_ON() to print, and
> WARN_ON() uses "BRK BUG_BRK_IMM". Now when it is printing though BRK, we were
> hitting kprobe at print_worker_info() which was resulting in unexpected EL1.
> 
> Proposed patch fixes kprobe tracing within none kprobe BRK context such as
> uprobe or WARN_ON() breakpoint handler etc. So, now a kprobe at
> print_worker_info() will work while printing message of WARN_ON().
> 

Okay, I didn't realize that the EL1 issue was hiding the OOM issue. So the patch is helping things.

> 
>>
>> Any thoughts on how to track down the oom issue?  Are you able to replicate it running the systemtap onthefly/kprobes_onthefly.exp tests?
> 
> Sure, will look into. Have reserved a seattle.

Thanks so much.

-Will