From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 73937 invoked by alias); 9 Jun 2016 19:52:38 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 73925 invoked by uid 89); 9 Jun 2016 19:52:37 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-3.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=Hx-languages-length:4500, H*MI:sk:befacf5, H*i:sk:befacf5, H*f:sk:befacf5 X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Thu, 09 Jun 2016 19:52:27 +0000 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 650A5835; Thu, 9 Jun 2016 19:52:26 +0000 (UTC) Received: from [10.13.129.159] (dhcp129-159.rdu.redhat.com [10.13.129.159]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u59JqPTj027295; Thu, 9 Jun 2016 15:52:25 -0400 Subject: Re: exercising current aarch64 kprobe support with systemtap To: systemtap@sourceware.org, Dave Long , Pratyush Anand , Mark Brown References: From: William Cohen Message-ID: <599229e0-49ad-1c8e-1055-81e38692e5ec@redhat.com> Date: Thu, 09 Jun 2016 19:52:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2016-q2/txt/msg00204.txt.bz2 On 06/09/2016 12:17 PM, William Cohen wrote: > I have been exercising the current kprobes and uprobe patches for > arm64 that are in the test_upstream_arm64_devel branch of > https://github.com/pratyushanand/linux with systemtap. There are a > two issues that I have seen on this kernel with systemtap. There are > some cases where kprobes fail to register at places that appear to be > reasonable places for a kprobe. The other issue is that kernel starts > having soft lockups when the hw_watch_addr.stp tests runs. To get > systemtap with the newer kernels need the attached hack because of > changes in the aarch64 macro args. > > EINVAL for seemingly valid kprobe registration > > Below shows the bz1027459.stp failing because of the some of the kprobes not registering. > > # make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp" > ... > > > spawn stap /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.stp > WARNING: probe kernel.function("SyS_set_tid_address@kernel/fork.c:1236").call (address 0xfffffc00080c9578) registration error (rc -22) > WARNING: probe kernel.function("SyS_sched_setaffinity@kernel/sched/core.c:4690").call (address 0xfffffc0008104d58) registration error (rc -22) > WARNING: probe kernel.function("SyS_sched_get_priority_min@kernel/sched/core.c:5013").call (address 0xfffffc0008105250) registration error (rc -22) > WARNING: probe kernel.function("SyS_sched_get_priority_max@kernel/sched/core.c:4986").call (address 0xfffffc00081051e8) registration error (rc -22) > hi > FAIL: bz1027459 -p5 (0) > > area around Sys_set_tid_address > > fffffc00080c956c: d503201f nop > fffffc00080c9570: 08dc4c80 .word 0x08dc4c80 > fffffc00080c9574: fffffc00 .word 0xfffffc00 > > fffffc00080c9578 : > fffffc00080c9578: a9be7bfd stp x29, x30, [sp,#-32]! > fffffc00080c957c: 910003fd mov x29, sp > > area around SyS_sched_setaffiniity > > fffffc0008104d4c: 17ffff73 b fffffc0008104b18 > fffffc0008104d50: 08dd9d80 .word 0x08dd9d80 > fffffc0008104d54: fffffc00 .word 0xfffffc00 > > fffffc0008104d58 : > fffffc0008104d58: a9bb7bfd stp x29, x30, [sp,#-80]! > fffffc0008104d5c: 910003fd mov x29, sp > > area around SyS_sched_get_priority_min > > fffffc0008105244: f9400bf3 ldr x19, [sp,#16] > fffffc0008105248: a8c27bfd ldp x29, x30, [sp],#32 > fffffc000810524c: d65f03c0 ret > > fffffc0008105250 : > fffffc0008105250: a9be7bfd stp x29, x30, [sp,#-32]! > fffffc0008105254: 910003fd mov x29, sp > > > area around SyS_sched_get_priority_max > > > fffffc00081051dc: 17ffffe8 b fffffc000810517c > fffffc00081051e0: 08dd9d80 .word 0x08dd9d80 > fffffc00081051e4: fffffc00 .word 0xfffffc00 > > fffffc00081051e8 : > fffffc00081051e8: a9be7bfd stp x29, x30, [sp,#-32]! > fffffc00081051ec: 910003fd mov x29, sp > > > The stp (store pair) instructions at the beginning of these functions > should be fine to instrument. One thing that I could think of causing > a problem is the test to make sure that the instruction is not inside > a load exclusive/store exclusive region. The test might be mistaking > some of the data before the start of the function as load exclusive > instructions. I verified that the cause of kprobes not being registered is the scan backward for load exclusive instructions. For one example have: ... fffffc00080c98cc: d503201f nop fffffc00080c98d0: 08dc4c80 .word 0x08dc4c80 fffffc00080c98d4: fffffc00 .word 0xfffffc00 fffffc00080c98d8 : fffffc00080c98d8: a9be7bfd stp x29, x30, [sp,#-32]! fffffc00080c98dc: 910003fd mov x29, sp The previous function has 0xfffffc0008dc4c80 as data at the end of the function. The scan backwards from the beginning of the current function Sys_set_tid_address stumbles into that data and interprets the 0x08dc4c80 as load exclusive instructions. This causes the kprobe registration to fail. Disabled the is_probed_address_atomic() scan for atomic instructions allows the test to work: make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp" ... Running target unix Running /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.exp ... PASS: bz1027459 -p5 === systemtap Summary === # of expected passes 1 Somehow the is_probed_address_atomic and arm_kprobe_decode_insn functions need to avoid scanning past the beginning of a function. -Will