From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 18385 invoked by alias); 4 Dec 2015 09:56:17 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 18371 invoked by uid 89); 4 Dec 2015 09:56:16 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wm0-f68.google.com Received: from mail-wm0-f68.google.com (HELO mail-wm0-f68.google.com) (74.125.82.68) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Fri, 04 Dec 2015 09:56:14 +0000 Received: by wmeo63 with SMTP id o63so9586947wme.2 for ; Fri, 04 Dec 2015 01:56:12 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.28.213.72 with SMTP id m69mr4119729wmg.101.1449222972066; Fri, 04 Dec 2015 01:56:12 -0800 (PST) Received: by 10.194.109.6 with HTTP; Fri, 4 Dec 2015 01:56:11 -0800 (PST) In-Reply-To: <566075FA.40207@redhat.com> References: <564B5A4C.1080302@redhat.com> <564CD3C1.2090900@redhat.com> <564DE376.3020104@redhat.com> <565CC50B.90104@redhat.com> <565DCA83.6040102@redhat.com> <565F65D4.4050005@redhat.com> <566075FA.40207@redhat.com> Date: Fri, 04 Dec 2015 09:56:00 -0000 Message-ID: Subject: Re: How to track the functions in self-written module using SystemTap? From: Nan Xiao To: David Smith Cc: "Frank Ch. Eigler" , systemtap@sourceware.org Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes X-SW-Source: 2015-q4/txt/msg00230.txt.bz2 Hi Dave, > It looks like you've got a couple of options: > 1) Copy your module into the kernel module tree (located at > /lib/modules/`uname -r`/kernel) to use systemtap on it. I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always executes error: # cd /lib/modules/`uname -r`/kernel # pwd /lib/modules/3.10.0-123.el7.x86_64.debug/kernel # ls arch crypto drivers fs kernel kex.ko lib mm net sound # stap -v -e 'probe module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") { printf("%s\n", ppfunc()) }' Pass 1: parsed user script and 103 library script(s) using 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. semantic error: while resolving probe point: identifier 'module' at :1:7 source: probe module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") { printf("%s\n", ppfunc()) } ^ semantic error: no match Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 372028virt/33024res/3412shr/30124data kb, in 20usr/770sys/804real ms. Pass 2: analysis failed. [man error::pass2] # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", ppfunc()) }' Pass 1: parsed user script and 103 library script(s) using 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. semantic error: while resolving probe point: identifier 'module' at :1:7 source: probe module("kex.ko").function("*") { printf("%s\n", ppfunc()) } ^ semantic error: no match Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 372032virt/33016res/3412shr/30128data kb, in 40usr/880sys/929real ms. Pass 2: analysis failed. [man error::pass2] > 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 > system, and the modules_out_of_tree.exp test case passed there.) I install the RHEL 7.2, and it really works! But it can't print actual stack traces: # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { print_backtrace() }' WARNING: no or bad debug frame hdr WARNING: No binary search table for debug frame, doing slow linear search for /root/kernel/105.ops/kex.ko 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] 0xffffffff810020e8 (inexact) 0xffffffff810ed4ae (inexact) 0xffffffff81316880 (inexact) 0xffffffff810e9743 (inexact) 0xffffffff810ede66 (inexact) 0xffffffff81645909 (inexact) 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] 0xffffffff810eb23b (inexact) 0xffffffff81641113 (inexact) 0xffffffff81014b12 (inexact) 0xffffffff81645909 (inexact) So it means the module isn't built with debuginfo, right? Thanks very much for your time & effort! Best Regards Nan Xiao On Fri, Dec 4, 2015 at 1:03 AM, David Smith wrote: > On 12/02/2015 07:13 PM, Nan Xiao wrote: >> Hi David, >> >> (1) >>>>> Hmm. OK, let's try a couple more things: >>>> >>>>> 1) It could be that systemtap is missing the module load somehow. So, >>>>> try this: >>>> >>>>> - load the module >>>>> - run stap >>>>> - exercise the module >>>>> - unload the module >>>>> - kill stap >>>> >>>> Still outputs nothing. >> >>> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap >>> command and show us the output? >> >> After adding '-DDEBUG_KPROBES', the stap command still outputs nothing. > > Hmm. > >> (2) >> [root@localhost testsuite]# make installcheck >> RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" >> >> >> Running ./systemtap.base/kmodule.exp ... >> >> === systemtap Summary === >> >> # of expected passes 7 > > Ah. It ran kmodule.exp, but not modules_out_of_tree.exp. You are running > systemtap 2.4, and that testcase was first present in systemtap 2.6. > > I was testing another bug so I had a RHEL 7.0 systemtap set up and I > installed systemtap 2.4 on it. It looks like that version can't probe > out-of-tree modules. > > It looks like you've got a couple of options: > > 1) Copy your module into the kernel module tree (located at > /lib/modules/`uname -r`/kernel) to use systemtap on it. > > 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 > system, and the modules_out_of_tree.exp test case passed there.) > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax)