From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 74812 invoked by alias); 8 Dec 2015 01:05:41 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 74800 invoked by uid 89); 8 Dec 2015 01:05:40 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wm0-f65.google.com Received: from mail-wm0-f65.google.com (HELO mail-wm0-f65.google.com) (74.125.82.65) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Tue, 08 Dec 2015 01:05:38 +0000 Received: by wmeo63 with SMTP id o63so1728369wme.2 for ; Mon, 07 Dec 2015 17:05:35 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.194.87.170 with SMTP id az10mr669017wjb.144.1449536735295; Mon, 07 Dec 2015 17:05:35 -0800 (PST) Received: by 10.194.109.6 with HTTP; Mon, 7 Dec 2015 17:05:35 -0800 (PST) In-Reply-To: <5665E3CC.6000104@redhat.com> References: <564CD3C1.2090900@redhat.com> <564DE376.3020104@redhat.com> <565CC50B.90104@redhat.com> <565DCA83.6040102@redhat.com> <565F65D4.4050005@redhat.com> <566075FA.40207@redhat.com> <566183E6.9050101@redhat.com> <5665E3CC.6000104@redhat.com> Date: Tue, 08 Dec 2015 01:05:00 -0000 Message-ID: Subject: Re: How to track the functions in self-written module using SystemTap? From: Nan Xiao To: David Smith Cc: "Frank Ch. Eigler" , systemtap@sourceware.org Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes X-SW-Source: 2015-q4/txt/msg00241.txt.bz2 Hi David, > 1) Make sure your kernel was compiled with frame pointer support (which > it should if it is a standard RHEL kernel): > # fgrep FRAME_POINTER /boot/config-`uname -r` > CONFIG_SCHED_OMIT_FRAME_POINTER=y > CONFIG_ARCH_WANT_FRAME_POINTERS=y > CONFIG_FRAME_POINTER=y Yes, all the CONFIG options are "yes". > 2) Let's make sure your module still has debuginfo present. First, run > "file" on your module, making sure it says "not stripped". Yes, it is "not stripped". > Then go a bit deeper and run "readelf --sections" on your module - there should be > several sections that start with '.debug', including one called > '.debug_frame'. Below are all the sections with `.debug`: ...... [19] .debug_info PROGBITS 0000000000000000 000008e0 000000000000b6d9 0000000000000000 0 0 1 [20] .rela.debug_info RELA 0000000000000000 00014fe0 000000000000ed00 0000000000000018 36 19 8 [21] .debug_abbrev PROGBITS 0000000000000000 0000bfb9 00000000000007cc 0000000000000000 0 0 1 [22] .debug_loc PROGBITS 0000000000000000 0000c785 0000000000000324 0000000000000000 0 0 1 [23] .rela.debug_loc RELA 0000000000000000 00023ce0 0000000000000570 0000000000000018 36 22 8 [24] .debug_aranges PROGBITS 0000000000000000 0000caa9 0000000000000070 0000000000000000 0 0 1 [25] .rela.debug_arang RELA 0000000000000000 00024250 0000000000000078 0000000000000018 36 24 8 [26] .debug_ranges PROGBITS 0000000000000000 0000cb19 0000000000000040 0000000000000000 0 0 1 [27] .rela.debug_range RELA 0000000000000000 000242c8 0000000000000090 0000000000000018 36 26 8 [28] .debug_line PROGBITS 0000000000000000 0000cb59 0000000000000a8e 0000000000000000 0 0 1 [29] .rela.debug_line RELA 0000000000000000 00024358 0000000000000048 0000000000000018 36 28 8 [30] .debug_str PROGBITS 0000000000000000 0000d5e7 00000000000069fc 0000000000000001 MS 0 0 1 [31] .comment PROGBITS 0000000000000000 00013fe3 000000000000005a 0000000000000001 MS 0 0 1 [32] .note.GNU-stack PROGBITS 0000000000000000 0001403d 0000000000000000 0000000000000000 0 0 1 [33] .debug_frame PROGBITS 0000000000000000 00014040 00000000000000f0 0000000000000000 0 0 8 [34] .rela.debug_frame RELA 0000000000000000 000243a0 00000000000000f0 0000000000000018 36 33 8 ...... Thanks! Best Regards Nan Xiao On Tue, Dec 8, 2015 at 3:53 AM, David Smith wrote: > On 12/06/2015 02:53 AM, Nan Xiao wrote: >> Hi David, >> >>> (1) >>> I see I didn't explain well here. Once you copied your module to the >>> kernel's module directory, systemtap no longer would need a full path to >>> the module name - just "kex" should have worked. >> >> Yeah, it worked! >> >>> (2) >>> Hmm, it could mean that we just need to make sure that the debuginfo is >>> loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the >>> stap command line. (The '-d' option asks stap to load the symbol/unwind >>> information for a module even if stap doesn't think it needs it.) >> >> The output stills have error: >> >> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe >> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >> print_backtrace()) }' >> Pass 1: parsed user script and 110 library script(s) using >> 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms. >> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 >> global(s) using 221348virt/39612res/3832shr/36156data kb, in >> 20usr/60sys/114real ms. >> Pass 3: translated to C into >> "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c" >> using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real >> ms. >> Pass 4: compiled C into >> "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in >> 2280usr/540sys/5099real ms. >> Pass 5: starting run. >> WARNING: no or bad debug frame hdr >> WARNING: No binary search table for debug frame, doing slow linear >> search for /root/kernel/105.ops/kex.ko >> 0xffffffffa0037000 : kex_init+0x0/0x0 [kex] >> 0xffffffff810020e8 (inexact) >> 0xffffffff810ed4ae (inexact) >> 0xffffffff81316880 (inexact) >> 0xffffffff810e9743 (inexact) >> 0xffffffff810ede66 (inexact) >> 0xffffffff81645909 (inexact) >> >> 0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex] >> 0xffffffff810eb23b (inexact) >> 0xffffffff81641113 (inexact) >> 0xffffffff81014b12 (inexact) >> 0xffffffff81645909 (inexact) > > Hmm. OK, let's make sure of a couple of things: > > 1) Make sure your kernel was compiled with frame pointer support (which > it should if it is a standard RHEL kernel): > > # fgrep FRAME_POINTER /boot/config-`uname -r` > CONFIG_SCHED_OMIT_FRAME_POINTER=y > CONFIG_ARCH_WANT_FRAME_POINTERS=y > CONFIG_FRAME_POINTER=y > > 2) Let's make sure your module still has debuginfo present. First, run > "file" on your module, making sure it says "not stripped". Then go a bit > deeper and run "readelf --sections" on your module - there should be > several sections that start with '.debug', including one called > '.debug_frame'. > >> >> Thanks! >> Best Regards >> Nan Xiao >> >> >> On Fri, Dec 4, 2015 at 8:15 PM, David Smith wrote: >>> On 12/04/2015 03:56 AM, Nan Xiao wrote: >>>> Hi Dave, >>>> >>>>> It looks like you've got a couple of options: >>>> >>>>> 1) Copy your module into the kernel module tree (located at >>>>> /lib/modules/`uname -r`/kernel) to use systemtap on it. >>>> >>>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always >>>> executes error: >>>> >>>> # cd /lib/modules/`uname -r`/kernel >>>> # pwd >>>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel >>>> # ls >>>> arch crypto drivers fs kernel kex.ko lib mm net sound >>>> >>>> # stap -v -e 'probe >>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >>>> { printf("%s\n", ppfunc()) }' >>>> Pass 1: parsed user script and 103 library script(s) using >>>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. >>>> semantic error: while resolving probe point: identifier 'module' at :1:7 >>>> source: probe >>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >>>> { printf("%s\n", ppfunc()) } >>>> ^ >>>> >>>> semantic error: no match >>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >>>> global(s) using 372028virt/33024res/3412shr/30124data kb, in >>>> 20usr/770sys/804real ms. >>>> Pass 2: analysis failed. [man error::pass2] >>>> >>>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", >>>> ppfunc()) }' Pass 1: parsed user >>>> script and 103 library script(s) using >>>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. >>>> semantic error: while resolving probe point: identifier 'module' at :1:7 >>>> source: probe module("kex.ko").function("*") { printf("%s\n", >>>> ppfunc()) } >>>> ^ >>>> >>>> semantic error: no match >>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >>>> global(s) using 372032virt/33016res/3412shr/30128data kb, in >>>> 40usr/880sys/929real ms. >>>> Pass 2: analysis failed. [man error::pass2] >>> >>> I see I didn't explain well here. Once you copied your module to the >>> kernel's module directory, systemtap no longer would need a full path to >>> the module name - just "kex" should have worked. >>> >>>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 >>>>> system, and the modules_out_of_tree.exp test case passed there.) >>>> >>>> I install the RHEL 7.2, and it really works! But it can't print actual >>>> stack traces: >>>> >>>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { >>>> print_backtrace() }' >>>> WARNING: no or bad debug frame hdr >>>> WARNING: No binary search table for debug frame, doing slow linear >>>> search for /root/kernel/105.ops/kex.ko >>>> 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] >>>> 0xffffffff810020e8 (inexact) >>>> 0xffffffff810ed4ae (inexact) >>>> 0xffffffff81316880 (inexact) >>>> 0xffffffff810e9743 (inexact) >>>> 0xffffffff810ede66 (inexact) >>>> 0xffffffff81645909 (inexact) >>>> 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] >>>> 0xffffffff810eb23b (inexact) >>>> 0xffffffff81641113 (inexact) >>>> 0xffffffff81014b12 (inexact) >>>> 0xffffffff81645909 (inexact) >>>> >>>> So it means the module isn't built with debuginfo, right? >>> >>> Hmm, it could mean that we just need to make sure that the debuginfo is >>> loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the >>> stap command line. (The '-d' option asks stap to load the symbol/unwind >>> information for a module even if stap doesn't think it needs it.) >>> >>> -- >>> David Smith >>> dsmith@redhat.com >>> Red Hat >>> http://www.redhat.com >>> 256.217.0141 (direct) >>> 256.837.0057 (fax) > > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax)