From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 30664 invoked by alias); 4 Aug 2016 14:36:05 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 30656 invoked by uid 89); 4 Aug 2016 14:36:04 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.2 spammy=H*MI:sk:f059377, H*f:sk:f059377, H*i:sk:f059377, reserved X-HELO: mail-pa0-f43.google.com Received: from mail-pa0-f43.google.com (HELO mail-pa0-f43.google.com) (209.85.220.43) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Thu, 04 Aug 2016 14:35:54 +0000 Received: by mail-pa0-f43.google.com with SMTP id fi15so83557944pac.1 for ; Thu, 04 Aug 2016 07:35:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=876Newnlq6+iC1RSXYnhrwqoXeMftrEDzpESdOD88r0=; b=Hf94sRlgrQgAlFJvpEKzUCsrjCBF3/FfKLVLfqWXUoHu6T4WVRYo8imGWj/0AYtlPb A+gFvqi27VKhp61v7Wo2CaxaZFALNCpJp/3J77byIASr8UqU6hzEi7a3tImKPvIPE8QS 3nNmdbMpk3cdoXNjxh+hO3ZcjKlIHnqUM/5TrEv/yf0SIWI71Ae+IJpUWCxd0XjgwYYj tfsSbsRwy9D5s+4JlwvAOGZJgL5PJyf/6vZabOG+/pa7N3zm7wkLVl7NMOhk3kyrvB1F IFmVLNULblX9BvF6byY8mrAuQ0gruHFxKROD8m25u9uW7SfBhgqRLazGqsCK3hQ0yTO/ PkiA== X-Gm-Message-State: AEkoouueVtnIaDDUVLkPv7nj+U/EUCP4cOZNeuSJi3Vd0jC5RUOoelF3Sd6QGt5KPqLWjnVH X-Received: by 10.66.13.200 with SMTP id j8mr125577901pac.150.1470321352169; Thu, 04 Aug 2016 07:35:52 -0700 (PDT) Received: from localhost ([122.180.189.31]) by smtp.gmail.com with ESMTPSA id l81sm20984747pfi.50.2016.08.04.07.35.50 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 04 Aug 2016 07:35:51 -0700 (PDT) Date: Thu, 04 Aug 2016 14:36:00 -0000 From: Pratyush Anand To: William Cohen Cc: David Long , systemtap@sourceware.org, Mark Brown , Jeremy Linton , David Smith , "Frank Ch. Eigler" Subject: Re: exercising current aarch64 kprobe support with systemtap Message-ID: <20160804143549.GF22191@localhost.localdomain> References: <0a594132-796b-779d-b473-a06c0f3e8ae8@redhat.com> <20160627141840.GB8139@dhcppc9> <577EA7EE.2070607@linaro.org> <20160803131302.GC18785@localhost.localdomain> <2947a749-a518-d560-f768-60cc2f2c691e@redhat.com> <20160804044230.GB22191@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-IsSubscribed: yes X-SW-Source: 2016-q3/txt/msg00129.txt.bz2 Hi Will, On 04/08/2016:09:56:45 AM, William Cohen wrote: > On 08/04/2016 12:42 AM, Pratyush Anand wrote: > > Hi Will, > > > > On 03/08/2016:01:39:47 PM, William Cohen wrote: > >> On 08/03/2016 09:13 AM, Pratyush Anand wrote: > >>> On 07/07/2016:03:58:37 PM, Frank Ch. Eigler wrote: > >>>> David Long writes: > >>>> > >>>>> [...] > >>>>>> - bug_handler() calls report_bug() which calls __warn() > >>>>>> - __warn() does lot of pr_warn() which invokes print_worker_info() > >>>>>> where we have a kprobe instrumented. > >>>>>> - Therefore, we are encountering this issue. > >>> > >>> Hi Will, > >>> > >>> Can you please cross-check if following branch works perfectly with > >>> kprobes_onthefly.exp and other systemtap tests. > >>> > >>> https://github.com/pratyushanand/linux/tree/uprobe/upstream_arm64_devel_v1.1 > >>> > >>> Following patch in above branch should solve this issue. > >>> https://github.com/pratyushanand/linux/commit/d0dcc6477f1279ab0bd99aefc30efdecb16c586e > >>> > >>> However, I am not yet sure that above modification could be the best solution, > >>> so discussing at arm kernel list. > >>> > >>> ~Pratyush > >>> > >> > >> I have an AMD seattle machine set up Fedora24, the upstream_arm64_devel_v1.1 branch kernel locally built, and a locally built checkout of systemtap (systemtap rpm in fc24 doesn't generate models for linux 4.6 and newer kernels). Tried to run the systemtap tests with: > >> > >> make installcheck RUNTESTFLAGS="--debug systemtap.onthefly/kprobes_onthefly.exp" > >> > >> However at some point the kernel starts having problems: > >> > >> http://paste.stg.fedoraproject.org/5375/ > > > > Yes, this is what you were getting with earlier code as well, but now it is not > > going to infinite unexpected EL1, so at least proposed kprobe improvement seems > > fine to me. > > > > In this failing test we are getting oom and the oom_killer is called. So, > > I think, this is another point of investigation that why this OOM occurs. > > > > ~Pratyush > > > > Hi, > > The OOM errors came before the otf_stress_hard_iter_5000 test that previous triggered the infinite unexpected EL1, so can't really say that the proposed patch has fixed the problem. Yes, yes, previously also we were getting OOM, and then that OOM was triggering infinite unexpected EL1, because OOM message uses WARN_ON() to print, and WARN_ON() uses "BRK BUG_BRK_IMM". Now when it is printing though BRK, we were hitting kprobe at print_worker_info() which was resulting in unexpected EL1. Proposed patch fixes kprobe tracing within none kprobe BRK context such as uprobe or WARN_ON() breakpoint handler etc. So, now a kprobe at print_worker_info() will work while printing message of WARN_ON(). > > Any thoughts on how to track down the oom issue? Are you able to replicate it running the systemtap onthefly/kprobes_onthefly.exp tests? Sure, will look into. Have reserved a seattle. ~Pratyush