From: William Cohen <wcohen@redhat.com>
To: Pratyush Anand <panand@redhat.com>
Cc: systemtap@sourceware.org, Dave Long <dave.long@linaro.org>,
Mark Brown <broonie@linaro.org>,
Jeremy Linton <jlinton@redhat.com>
Subject: Re: exercising current aarch64 kprobe support with systemtap
Date: Mon, 13 Jun 2016 13:42:00 -0000 [thread overview]
Message-ID: <156bb7aa-1542-80a4-5585-6a5cec12f97f@redhat.com> (raw)
In-Reply-To: <20160613042758.GB6344@dhcppc6>
[-- Attachment #1: Type: text/plain, Size: 2964 bytes --]
On 06/13/2016 12:27 AM, Pratyush Anand wrote:
> Hi Will,
>
> On 10/06/2016:05:28:36 PM, William Cohen wrote:
>> On 06/09/2016 12:17 PM, William Cohen wrote:
>>> I have been exercising the current kprobes and uprobe patches for
>>> arm64 that are in the test_upstream_arm64_devel branch of
>>> https://github.com/pratyushanand/linux with systemtap. There are a
>>> two issues that I have seen on this kernel with systemtap. There are
>>> some cases where kprobes fail to register at places that appear to be
>>> reasonable places for a kprobe. The other issue is that kernel starts
>>> having soft lockups when the hw_watch_addr.stp tests runs. To get
>>> systemtap with the newer kernels need the attached hack because of
>>> changes in the aarch64 macro args.
>> ...
>>> Soft Lookup for the hw_watch_addr.stp
>>>
>>> When running the hw_watch_addr.stp tests the machine gets a number of
>>> processes using a lot of sys time and eventually the kernel reports
>>> soft lockup:
>>>
>>> http://paste.stg.fedoraproject.org/5323/
>>>
>>> The systemtap.base/overload.exp tests all pass, but maybe there is
>>> much work being done to generate the backtraces for hw_watch_addr.stp
>>> and that is triggering the problem.
>>
>> I can reliably reproduce the soft lockup running a single test with:
>>
>> /root/systemtap_write/install/bin/stap --all-modules \
>> /root/systemtap_write/systemtap/testsuite/systemtap.examples/memory/hw_watch_addr.stp \
>> 0x`grep "vm_dirty_ratio" /proc/kallsyms | awk '{print $1}'` -T 5 > /dev/null
>>
>> paste of output and soft lockup at: http://paste.stg.fedoraproject.org/5324/
>>
>> One of the things that Jeremy Linton pointed to was:
>>
>> https://lkml.org/lkml/2016/3/21/198
>
> Now we have following in arch_within_kprobe_blacklist(). So above issue should
> not bite us.
>
> + !!search_exception_tables(addr))
> + return true;
>
>>
>> Could the aarch64 hardware watchpoint handler have an issue that is causing this problem with the soft lockup?
>> Or spending too much time doing the stack backtrace?
>
> Not sure, could be the locked up CPU waiting for a lock (spinlock), which is not
> being released. Just noticed that, backtrace of all active CPUs (`echo l >
> /proc/sysrq-trigger`) is not working for arm64. Probably because, we do not have
> arch_trigger_all_cpu_backtrace() defined for aarch64. May be we can have one,
> like that of arm. Backtrace of CPUs in this state might give us some input.
>
> ~Pratyush
>
Hi Pratyush,
I did some additional experimentation this weekend. The version of systemtap script with an empty body (the attached hw_watch_addr_null2.stp) still caused the system to have soft lockup. However, the equivalent perf use of the hardware watchpoint worked fine (it got counts and no soft lookup):
perf stat -a -e mem:0x`grep "vm_dirty_ratio" /proc/kallsyms | awk '{print $1}'`/1 bash
So it looks like the issue might lie with something in systemtap.
-Will
[-- Attachment #2: hw_watch_addr_null2.stp --]
[-- Type: text/plain, Size: 111 bytes --]
#! /usr/bin/env stap
%( CONFIG_HAVE_HW_BREAKPOINT == "y" %?
probe kernel.data($1).rw
{
}
%:
probe never {}
%)
next prev parent reply other threads:[~2016-06-13 13:42 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-09 16:17 William Cohen
2016-06-09 19:52 ` William Cohen
2016-06-10 3:42 ` David Long
2016-06-10 5:49 ` David Long
2016-06-10 13:43 ` Pratyush Anand
2016-06-10 14:03 ` William Cohen
2016-06-10 14:37 ` David Long
2016-06-10 15:27 ` William Cohen
2016-06-10 14:20 ` David Long
2016-06-10 15:11 ` William Cohen
2016-06-10 17:07 ` Pratyush Anand
2016-07-12 14:33 ` William Cohen
2016-07-13 18:26 ` David Long
2016-07-13 18:47 ` Pratyush Anand
2016-07-13 19:45 ` William Cohen
2016-06-10 21:28 ` William Cohen
2016-06-10 21:37 ` William Cohen
2016-06-13 4:28 ` Pratyush Anand
2016-06-13 13:42 ` William Cohen [this message]
2016-06-22 20:24 ` William Cohen
2016-06-23 3:19 ` David Long
2016-06-23 13:42 ` William Cohen
2016-06-23 13:47 ` David Smith
2016-06-23 15:49 ` William Cohen
2016-06-23 18:26 ` David Long
2016-06-23 19:22 ` William Cohen
2016-06-27 2:57 ` David Long
2016-06-27 14:18 ` Pratyush Anand
2016-06-28 3:20 ` William Cohen
2016-07-04 12:46 ` Pratyush Anand
2016-07-07 19:05 ` David Long
2016-07-07 19:58 ` Frank Ch. Eigler
2016-08-03 13:13 ` Pratyush Anand
2016-08-03 14:51 ` William Cohen
2016-08-03 15:11 ` David Long
2016-08-03 17:40 ` William Cohen
2016-08-03 20:00 ` Lastest kprobes64 patch David Long
2016-08-03 20:01 ` Frank Ch. Eigler
2016-08-03 20:08 ` David Long
2016-08-04 5:03 ` Pratyush Anand
2016-08-04 13:07 ` David Long
2016-08-04 4:42 ` exercising current aarch64 kprobe support with systemtap Pratyush Anand
2016-08-04 13:57 ` William Cohen
2016-08-04 14:36 ` Pratyush Anand
2016-08-04 14:50 ` William Cohen
2016-08-04 20:51 ` William Cohen
2016-08-17 14:36 ` William Cohen
2016-08-17 18:04 ` David Smith
2016-08-17 18:28 ` William Cohen
2016-08-18 15:07 ` David Smith
2016-08-18 15:16 ` William Cohen
2016-08-18 15:39 ` David Smith
2016-08-18 14:55 ` Pratyush Anand
2016-06-13 16:11 ` William Cohen
2016-06-13 16:15 ` William Cohen
2016-06-14 4:27 ` Pratyush Anand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=156bb7aa-1542-80a4-5585-6a5cec12f97f@redhat.com \
--to=wcohen@redhat.com \
--cc=broonie@linaro.org \
--cc=dave.long@linaro.org \
--cc=jlinton@redhat.com \
--cc=panand@redhat.com \
--cc=systemtap@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).