From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 113477 invoked by alias); 12 Jan 2016 14:32:17 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 113331 invoked by uid 89); 12 Jan 2016 14:32:16 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.2 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=recall, UD:io, sk:kernel-, Hx-languages-length:1887 X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Tue, 12 Jan 2016 14:32:15 +0000 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id F1016A19AD; Tue, 12 Jan 2016 14:32:13 +0000 (UTC) Received: from fche.csb (vpn-62-170.rdu2.redhat.com [10.10.62.170]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0CEWDEe030566; Tue, 12 Jan 2016 09:32:13 -0500 Received: by fche.csb (Postfix, from userid 2569) id 7B367586A5; Tue, 12 Jan 2016 09:32:13 -0500 (EST) To: Jinhua Luo Cc: systemtap@sourceware.org Subject: Re: [HELP] ubacktrace() problem References: From: fche@redhat.com (Frank Ch. Eigler) Date: Tue, 12 Jan 2016 14:32:00 -0000 In-Reply-To: (Jinhua Luo's message of "Tue, 12 Jan 2016 13:47:49 +0800") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-SW-Source: 2016-q1/txt/msg00021.txt.bz2 Hi - luajit.io wrote: > [...] > See https://github.com/openresty/nginx-systemtap-toolkit#sample-bt-off-cpu > for the script. It's a perl-generated stap script/invocation: https://github.com/openresty/nginx-systemtap-toolkit/blob/master/sample-bt-off-cpu > It fails to unwind some user backtrace (which is a big fraction of the > total samples), and just show some hex adddress, e.g. 0xb77a5d40. > > I check that address belongs to the vdso mapping. [...] Please check if your kernel dbgsym packages include the vdso debuginfo files. In Fedora for example: % rpm -ql kernel-debuginfo | grep vdso /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-int80.so.debug /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-syscall.so.debug /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso32-sysenter.so.debug /usr/lib/debug/lib/modules/4.2.6-200.fc22.x86_64/vdso/vdso64.so.debug These may be necessary to be able to unwind through the kernel-user transition (if a process is interrupted mid-syscall). I seem to recall that, years ago, the Fedora kernel team had some struggles to ensure that these userspace libraries' debug data was generated & included properly. Maybe the Debian/Ubuntu folks are about to have the same struggle. > And, I am also get confused about the warnings: > > WARNING: Too many CFI instuctions > WARNING: DWARF expression stack underflow in CFI > WARNING: no or bad debug frame hdr > WARNING: No binary search table for eh frame, doing slow linear search > for stap_b9bfb422b7bd73acc448d7f775aaacd_37477 These generally reflect imperfections in the gcc-generated unwind information. These are not something a user can do anything about; we should probably stop warning about them (or give more information like the suspect module name). - FChE