From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 103868 invoked by alias); 10 Jun 2016 21:37:39 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 103847 invoked by uid 89); 10 Jun 2016 21:37:38 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=H*i:sk:d03cd7b, H*f:sk:d03cd7b, Hx-languages-length:1992, H*MI:sk:d03cd7b X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Fri, 10 Jun 2016 21:37:28 +0000 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8A51980E4A; Fri, 10 Jun 2016 21:37:27 +0000 (UTC) Received: from [10.13.129.159] (dhcp129-159.rdu.redhat.com [10.13.129.159]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u5ALbQbI000593; Fri, 10 Jun 2016 17:37:27 -0400 Subject: Re: exercising current aarch64 kprobe support with systemtap To: systemtap@sourceware.org, Dave Long , Pratyush Anand , Mark Brown References: Cc: Jeremy Linton From: William Cohen Message-ID: <7d2d8cae-b448-4599-55c9-0864af24ba4e@redhat.com> Date: Fri, 10 Jun 2016 21:37:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2016-q2/txt/msg00216.txt.bz2 On 06/10/2016 05:28 PM, William Cohen wrote: > On 06/09/2016 12:17 PM, William Cohen wrote: >> I have been exercising the current kprobes and uprobe patches for >> arm64 that are in the test_upstream_arm64_devel branch of >> https://github.com/pratyushanand/linux with systemtap. There are a >> two issues that I have seen on this kernel with systemtap. There are >> some cases where kprobes fail to register at places that appear to be >> reasonable places for a kprobe. The other issue is that kernel starts >> having soft lockups when the hw_watch_addr.stp tests runs. To get >> systemtap with the newer kernels need the attached hack because of >> changes in the aarch64 macro args. > ... >> Soft Lookup for the hw_watch_addr.stp >> >> When running the hw_watch_addr.stp tests the machine gets a number of >> processes using a lot of sys time and eventually the kernel reports >> soft lockup: >> >> http://paste.stg.fedoraproject.org/5323/ >> >> The systemtap.base/overload.exp tests all pass, but maybe there is >> much work being done to generate the backtraces for hw_watch_addr.stp >> and that is triggering the problem. > > I can reliably reproduce the soft lockup running a single test with: > > /root/systemtap_write/install/bin/stap --all-modules \ > /root/systemtap_write/systemtap/testsuite/systemtap.examples/memory/hw_watch_addr.stp \ > 0x`grep "vm_dirty_ratio" /proc/kallsyms | awk '{print $1}'` -T 5 > /dev/null > > paste of output and soft lockup at: http://paste.stg.fedoraproject.org/5324/ > > One of the things that Jeremy Linton pointed to was: > > https://lkml.org/lkml/2016/3/21/198 > > Could the aarch64 hardware watchpoint handler have an issue that is causing this problem with the soft lockup? > Or spending too much time doing the stack backtrace? > > -Will > The soft lockup remains even if when the stabk backtrace in systemtap script is disabled: http://paste.stg.fedoraproject.org/5325/ -Will