From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 112184 invoked by alias); 27 Jun 2016 14:18:58 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 112166 invoked by uid 89); 27 Jun 2016 14:18:57 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.2 spammy=exercises, Dave X-HELO: mail-qk0-f169.google.com Received: from mail-qk0-f169.google.com (HELO mail-qk0-f169.google.com) (209.85.220.169) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Mon, 27 Jun 2016 14:18:47 +0000 Received: by mail-qk0-f169.google.com with SMTP id t127so206884428qkf.1 for ; Mon, 27 Jun 2016 07:18:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=m07E/zZU60zbq5vgR/wFMI5WjdxdODHqWp+mCkIaMl8=; b=C33xKdRYac7Yg39PhKuM/OAGLmD8X6OcMeiHf60xztUfVQw9GdR1NscuAtPD8BK1aG 5zwgjkvo4UqPbB/Bl7A9DW0nFHej+ZzZlpxlfe38l04+JMQY3Z5oKZ9ladlCuOMEWbqF biCVs/aFi5qiPXSd2akXJHWQGGreW6wCVs9rJT19jnpclVnl4t+geR2qE0IGQWVrr73U RRpv1ZQ3j/PkCCdwVqhptQ9/6+32w12R1LUQpvhOr72OWzPzyU+xZAJaFlFL37RKce1m 0B1J3/zK2KNbwM43VSOUJAJkGuy4jxfIGgJPF0MfuZ4bxzsk0GY+5trnUbVNgJ11lN2G fXMA== X-Gm-Message-State: ALyK8tKXOqlQidKoZx6lYCNi31WXnBVQwv8/pqtQbXBikA93gluYxzxL5NxlDGjVxmf1/nE6 X-Received: by 10.55.186.193 with SMTP id k184mr24170977qkf.184.1467037124940; Mon, 27 Jun 2016 07:18:44 -0700 (PDT) Received: from localhost ([122.177.254.203]) by smtp.gmail.com with ESMTPSA id v12sm3514720qtc.3.2016.06.27.07.18.42 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Jun 2016 07:18:43 -0700 (PDT) Date: Mon, 27 Jun 2016 14:18:00 -0000 From: Pratyush Anand To: William Cohen Cc: David Long , systemtap@sourceware.org, Mark Brown , Jeremy Linton , David Smith Subject: Re: exercising current aarch64 kprobe support with systemtap Message-ID: <20160627141840.GB8139@dhcppc9> References: <8f40d0b9-5550-92f9-d1c5-8769f52304c0@redhat.com> <576B5501.1030106@linaro.org> <576C29E1.8060805@linaro.org> <0a594132-796b-779d-b473-a06c0f3e8ae8@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0a594132-796b-779d-b473-a06c0f3e8ae8@redhat.com> User-Agent: Mutt/1.6.1 (2016-04-27) X-IsSubscribed: yes X-SW-Source: 2016-q2/txt/msg00293.txt.bz2 Hi Will, On 23/06/2016:03:22:44 PM, William Cohen wrote: > On 06/23/2016 02:26 PM, David Long wrote: > > On 06/23/2016 11:49 AM, William Cohen wrote: > >> On 06/22/2016 11:18 PM, David Long wrote: > >>> On 06/22/2016 04:24 PM, William Cohen wrote: > >>>> Hi all, > >>>> > >>>> When running the current systemtap checked out from the git repository > >>>> and a locally built kernel with the kprobes64-v13 patches (the > >>>> test_upstream_arm64_devel branch of > >>>> https://github.com/pratyushanand/linux) on Fedora 23 machine one of > >>>> the kprobes_onthefly.exp tests is causing the machine to get in a > >>>> state that requires rebooting to fix. This can be triggered by running a > >>>> portion of the systemtap tests with: > >>>> > >>>> make installcheck RUNTESTFLAGS="--debug systemtap.onthefly/kprobes_onthefly.exp" > >>>> > >>>> When it gets to the kprobes_onthefly - otf_stress_max_iter_5000 test the > >>>> console starts spewing the following and needs to be rebooted: > >>>> > >>>> [23394.036860] Unexpected kernel single-step exception at EL1 > >>>> [23394.042434] Unexpected kernel single-step exception at EL1 > >>>> [23394.048008] Unexpected kernel single-step exception at EL1 > >>>> [23394.053541] Unexpected kernel single-step exception at EL1 > >>>> [23394.059053] Unexpected kernel single-step exception at EL1 > >>>> [23394.064545] Unexpected kernel single-step exception at EL1 > >>>> > >>>> Sorry I don't have the start of the failure it scrolled off the screen very quickly. > >>>> > >>>> -Will > >>>> > >>>> > >>> > >>> I'll take a look and see what I can figure out. > >>> > >>> In the meantime I did just push a v14 branch. I'm doubtful that it will address the above problem even though it contains a few bug fixes. > >>> > >>> -dl > >>> > >> > >> Hi Dave and Pratyush, > >> > >> I tried the kprobes64-v13 kernel and it also seems to work, so it lookw like the problem might be in the the > >> test_upstream_arm64_devel branch of https://github.com/pratyushanand/linux . > >> > >> -Will > >> > > > > I'm going to interpret that as meaning you know of no problem in the kprobes v14 patch that would give me pause to email it upstream. Do you disagree? > > > > -dl > > > > Hi Dave, > > Yes, the problem only seems to be in that other kernel from https://github.com/pratyushanand/linux with the kprobe and uprobe patches, so the arm64 patches do not appear to be the problem. I don't know what is causing the problem maybe there is something going on with the porting of the patches to that kernel or the patches included in there (uprobes/kexec) in there. Just to update: I confirm that problem arises after uprobe patches only, but not yet sure that actual culprit is uprobe code. I can see that kprobes_onthefly.exp also exercises uprobes in the test. It seems, when problem happens, there was a kprobe at print_worker_info(). Most likely re-entrant kprobe is called when kprobe is instrumented at print_worker_info(). I guessed it could be show_regs() from arm64/kprobe code, but commenting show_regs() did not make any difference. Even blacklisting print_worker_info() also did not resolve it, probelem reproduced in a different way after blacklisting. So, still its vague and debugging is continued. If I can clearly understand the systemtap test code, then probably it will be easier to debug. I mean, if I can get the kernel and user space symbols name where this test is instrumenting probes then that would help a lot to zero it down. ~Pratyush