From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 47649 invoked by alias); 10 Jun 2016 03:42:36 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 47629 invoked by uid 89); 10 Jun 2016 03:42:34 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 spammy=H*MI:sk:599229e, H*i:sk:599229e, H*f:sk:599229e X-HELO: mail-qk0-f171.google.com Received: from mail-qk0-f171.google.com (HELO mail-qk0-f171.google.com) (209.85.220.171) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Fri, 10 Jun 2016 03:42:24 +0000 Received: by mail-qk0-f171.google.com with SMTP id c73so5906530qkg.2 for ; Thu, 09 Jun 2016 20:42:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=NxuCjF4acP7NroLLV/C1uSeEiMesgelzwfGPYwkuTbY=; b=FX5BwMjiadrmR+9gvQPUlhdZ9AYZN7IkKf4Yj8cM6oGLGi1f9E6Hs6qHNXIT5IBy6+ kYpIHMuiVfqVXhIg/qQ7YFtQG9GWWLDa/KRpn39pSZqAzpj9XC1t/qFojvyU4YZk4o2T kC17lubnwLeAWiLTWoHKYrb8dc1rovekOZuQQfsRkQQP3kgCAX2ZKQrYPuseUEQlRz8O d3e4f3Xh5m/aai6gnDFaPc+MNkX8cq1gDnn3jGhY9gH2VM6d5g4i9w50fIrmr5Nw5Xdy eZABvdqq4D0ciI0L3le9nMPqpiisccB34uW80WTtPGovoc6Ur6WF9WAmTD/Ff15DfJDB mxww== X-Gm-Message-State: ALyK8tINTUsOTzydFEDTHM4KO353UVIGNYH9f25lZzr2JbB2ti7aESP5BO8Gf+YkhqE1RM8Q X-Received: by 10.55.139.135 with SMTP id n129mr13763479qkd.56.1465530142389; Thu, 09 Jun 2016 20:42:22 -0700 (PDT) Received: from [192.168.1.116] (pool-72-71-243-181.cncdnh.fast00.myfairpoint.net. [72.71.243.181]) by smtp.googlemail.com with ESMTPSA id 90sm2585967qgh.28.2016.06.09.20.42.21 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 09 Jun 2016 20:42:21 -0700 (PDT) Subject: Re: exercising current aarch64 kprobe support with systemtap To: William Cohen , systemtap@sourceware.org, Pratyush Anand , Mark Brown References: <599229e0-49ad-1c8e-1055-81e38692e5ec@redhat.com> From: David Long Message-ID: <575A3713.9080303@linaro.org> Date: Fri, 10 Jun 2016 03:42:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <599229e0-49ad-1c8e-1055-81e38692e5ec@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-SW-Source: 2016-q2/txt/msg00205.txt.bz2 On 06/09/2016 03:52 PM, William Cohen wrote: > On 06/09/2016 12:17 PM, William Cohen wrote: >> I have been exercising the current kprobes and uprobe patches for >> arm64 that are in the test_upstream_arm64_devel branch of >> https://github.com/pratyushanand/linux with systemtap. There are a >> two issues that I have seen on this kernel with systemtap. There are >> some cases where kprobes fail to register at places that appear to be >> reasonable places for a kprobe. The other issue is that kernel starts >> having soft lockups when the hw_watch_addr.stp tests runs. To get >> systemtap with the newer kernels need the attached hack because of >> changes in the aarch64 macro args. >> >> EINVAL for seemingly valid kprobe registration >> >> Below shows the bz1027459.stp failing because of the some of the kprobes not registering. >> >> # make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp" >> ... >> >> >> spawn stap /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.stp >> WARNING: probe kernel.function("SyS_set_tid_address@kernel/fork.c:1236").call (address 0xfffffc00080c9578) registration error (rc -22) >> WARNING: probe kernel.function("SyS_sched_setaffinity@kernel/sched/core.c:4690").call (address 0xfffffc0008104d58) registration error (rc -22) >> WARNING: probe kernel.function("SyS_sched_get_priority_min@kernel/sched/core.c:5013").call (address 0xfffffc0008105250) registration error (rc -22) >> WARNING: probe kernel.function("SyS_sched_get_priority_max@kernel/sched/core.c:4986").call (address 0xfffffc00081051e8) registration error (rc -22) >> hi >> FAIL: bz1027459 -p5 (0) >> >> area around Sys_set_tid_address >> >> fffffc00080c956c: d503201f nop >> fffffc00080c9570: 08dc4c80 .word 0x08dc4c80 >> fffffc00080c9574: fffffc00 .word 0xfffffc00 >> >> fffffc00080c9578 : >> fffffc00080c9578: a9be7bfd stp x29, x30, [sp,#-32]! >> fffffc00080c957c: 910003fd mov x29, sp >> >> area around SyS_sched_setaffiniity >> >> fffffc0008104d4c: 17ffff73 b fffffc0008104b18 >> fffffc0008104d50: 08dd9d80 .word 0x08dd9d80 >> fffffc0008104d54: fffffc00 .word 0xfffffc00 >> >> fffffc0008104d58 : >> fffffc0008104d58: a9bb7bfd stp x29, x30, [sp,#-80]! >> fffffc0008104d5c: 910003fd mov x29, sp >> >> area around SyS_sched_get_priority_min >> >> fffffc0008105244: f9400bf3 ldr x19, [sp,#16] >> fffffc0008105248: a8c27bfd ldp x29, x30, [sp],#32 >> fffffc000810524c: d65f03c0 ret >> >> fffffc0008105250 : >> fffffc0008105250: a9be7bfd stp x29, x30, [sp,#-32]! >> fffffc0008105254: 910003fd mov x29, sp >> >> >> area around SyS_sched_get_priority_max >> >> >> fffffc00081051dc: 17ffffe8 b fffffc000810517c >> fffffc00081051e0: 08dd9d80 .word 0x08dd9d80 >> fffffc00081051e4: fffffc00 .word 0xfffffc00 >> >> fffffc00081051e8 : >> fffffc00081051e8: a9be7bfd stp x29, x30, [sp,#-32]! >> fffffc00081051ec: 910003fd mov x29, sp >> >> >> The stp (store pair) instructions at the beginning of these functions >> should be fine to instrument. One thing that I could think of causing >> a problem is the test to make sure that the instruction is not inside >> a load exclusive/store exclusive region. The test might be mistaking >> some of the data before the start of the function as load exclusive >> instructions. > > > I verified that the cause of kprobes not being registered is the scan > backward for load exclusive instructions. For one example have: > > ... > fffffc00080c98cc: d503201f nop > fffffc00080c98d0: 08dc4c80 .word 0x08dc4c80 > fffffc00080c98d4: fffffc00 .word 0xfffffc00 > > fffffc00080c98d8 : > fffffc00080c98d8: a9be7bfd stp x29, x30, [sp,#-32]! > fffffc00080c98dc: 910003fd mov x29, sp > > The previous function has 0xfffffc0008dc4c80 as data at the end of the > function. The scan backwards from the beginning of the current > function Sys_set_tid_address stumbles into that data and interprets > the 0x08dc4c80 as load exclusive instructions. This causes the kprobe > registration to fail. > > Disabled the is_probed_address_atomic() scan for atomic instructions > allows the test to work: > > make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp" > ... > Running target unix > Running /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.exp ... > PASS: bz1027459 -p5 > > === systemtap Summary === > > # of expected passes 1 > Interesting coincidence. Thanks for isolating the cause of that problem. > > Somehow the is_probed_address_atomic and arm_kprobe_decode_insn > functions need to avoid scanning past the beginning of a function. > > -Will > > I'm not sure we're going to be able to do that. We could interpret a "ret" to end the scan but that's probably going to be on the other side of the data. And can we be certain the compiler only puts literals between functions? Maybe the "stp x29,x30,[sp,...]" instruction is definitive enough to use as a flag for the beginning of functions. This is the approach I'm now thinking might be the best fix. -dl