* How to track the functions in self-written module using SystemTap? @ 2015-11-17 7:14 Nan Xiao 2015-11-17 16:48 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-11-17 7:14 UTC (permalink / raw) To: systemtap Hi all, I build a simple Linux module (https://github.com/troydhanson/kernel/blob/master/105.ops/kex.c), and want to use SystemTap to track it. But I find there is no available probes: # stap -l 'module("kex").function("*")' # How can I track the functions in module written by myself? Is there any tutorial about this? I try to search on https://sourceware.org/systemtap/documentation.html, but can't find wanted content. Thanks! Best Regards Nan Xiao ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-17 7:14 How to track the functions in self-written module using SystemTap? Nan Xiao @ 2015-11-17 16:48 ` David Smith 2015-11-18 3:39 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-11-17 16:48 UTC (permalink / raw) To: Nan Xiao, systemtap On 11/17/2015 01:14 AM, Nan Xiao wrote: > Hi all, > > I build a simple Linux module > (https://github.com/troydhanson/kernel/blob/master/105.ops/kex.c), > and want to use SystemTap to track it. > > But I find there is no available probes: > # stap -l 'module("kex").function("*")' > # > > How can I track the functions in module written by myself? Is there > any tutorial about this? I try to > search on https://sourceware.org/systemtap/documentation.html, but > can't find wanted content. If your 'kex' module isn't the same directory as the other kernel modules (/lib/modules/`uname -r`/), systemtap won't be able to find it. So, you have to tell systemtap where the module is, like this: # stap -l 'module("FULL_PATH_TO_KEX").function("*")' If that doesn't work, please let me know. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-17 16:48 ` David Smith @ 2015-11-18 3:39 ` Nan Xiao 2015-11-18 19:38 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-11-18 3:39 UTC (permalink / raw) To: David Smith; +Cc: systemtap Hi David, Yes, the "FULL_PATH_TO_KEX" works: ~/Downloads/kernel/105.ops # stap -l 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")' module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21") module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45") module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33") module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72") module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90") module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61") But the "RELATIVE_PATH_TO_KEX" doesn't work: ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")' ~/Downloads/kernel/105.ops # Is it the expected behavior of SystemTap? Thanks! Best Regards Nan Xiao On Wed, Nov 18, 2015 at 12:48 AM, David Smith <dsmith@redhat.com> wrote: > On 11/17/2015 01:14 AM, Nan Xiao wrote: >> Hi all, >> >> I build a simple Linux module >> (https://github.com/troydhanson/kernel/blob/master/105.ops/kex.c), >> and want to use SystemTap to track it. >> >> But I find there is no available probes: >> # stap -l 'module("kex").function("*")' >> # >> >> How can I track the functions in module written by myself? Is there >> any tutorial about this? I try to >> search on https://sourceware.org/systemtap/documentation.html, but >> can't find wanted content. > > If your 'kex' module isn't the same directory as the other kernel > modules (/lib/modules/`uname -r`/), systemtap won't be able to find it. > So, you have to tell systemtap where the module is, like this: > > # stap -l 'module("FULL_PATH_TO_KEX").function("*")' > > If that doesn't work, please let me know. > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-18 3:39 ` Nan Xiao @ 2015-11-18 19:38 ` David Smith 2015-11-19 0:49 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-11-18 19:38 UTC (permalink / raw) To: Nan Xiao; +Cc: systemtap On 11/17/2015 09:38 PM, Nan Xiao wrote: > Hi David, > > Yes, the "FULL_PATH_TO_KEX" works: > > ~/Downloads/kernel/105.ops # stap -l > 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")' > module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21") > module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45") > module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33") > module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72") > module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90") > module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61") > > But the "RELATIVE_PATH_TO_KEX" doesn't work: > ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")' > ~/Downloads/kernel/105.ops # > > Is it the expected behavior of SystemTap? Thanks! Good question. You are correct, using relative paths doesn't seem to work. I've filed PR19265 on this issue. <https://sourceware.org/bugzilla/show_bug.cgi?id=19265> Feel free to add yourself to the CC list on that bug. If you have the time, also feel free to look into this more. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-18 19:38 ` David Smith @ 2015-11-19 0:49 ` Nan Xiao 2015-11-19 8:33 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-11-19 0:49 UTC (permalink / raw) To: David Smith; +Cc: systemtap Hi David, Got it, thanks very much for your time and help! Best Regards Nan Xiao On Thu, Nov 19, 2015 at 3:38 AM, David Smith <dsmith@redhat.com> wrote: > On 11/17/2015 09:38 PM, Nan Xiao wrote: >> Hi David, >> >> Yes, the "FULL_PATH_TO_KEX" works: >> >> ~/Downloads/kernel/105.ops # stap -l >> 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")' >> module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21") >> module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45") >> module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33") >> module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72") >> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90") >> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61") >> >> But the "RELATIVE_PATH_TO_KEX" doesn't work: >> ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")' >> ~/Downloads/kernel/105.ops # >> >> Is it the expected behavior of SystemTap? Thanks! > > Good question. You are correct, using relative paths doesn't seem to > work. I've filed PR19265 on this issue. > > <https://sourceware.org/bugzilla/show_bug.cgi?id=19265> > > Feel free to add yourself to the CC list on that bug. If you have the > time, also feel free to look into this more. > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-19 0:49 ` Nan Xiao @ 2015-11-19 8:33 ` Nan Xiao 2015-11-19 14:58 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-11-19 8:33 UTC (permalink / raw) To: David Smith; +Cc: systemtap Hi David, Another interesting thing: if there is a "-" in path, the SystemTap also fail to parse: # stap -v -l 'module("/root/Downloads/kernel-master/105.ops/kex.ko").function("*")' Pass 1: parsed user script and 102 library script(s) using 78528virt/28740res/2700shr/26724data kb, in 170usr/20sys/183real ms. Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 79536virt/30480res/3164shr/27732data kb, in 0usr/50sys/53real ms. Change kernel-master to kernel: # stap -v -l 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")' Pass 1: parsed user script and 102 library script(s) using 78528virt/28732res/2700shr/26724data kb, in 180usr/10sys/189real ms. module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel-master/105.ops/kex.c:21") module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel-master/105.ops/kex.c:45") module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel-master/105.ops/kex.c:33") module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-6/arch/x86/include/asm/uaccess_64.h:72") module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel-master/105.ops/kex.c:90") module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel-master/105.ops/kex.c:61") Pass 2: analyzed script: 6 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 79548virt/30624res/3304shr/27744data kb, in 20usr/30sys/56real ms. Thanks! Best Regards Nan Xiao On Thu, Nov 19, 2015 at 8:49 AM, Nan Xiao <xiaonan830818@gmail.com> wrote: > Hi David, > > Got it, thanks very much for your time and help! > Best Regards > Nan Xiao > > > On Thu, Nov 19, 2015 at 3:38 AM, David Smith <dsmith@redhat.com> wrote: >> On 11/17/2015 09:38 PM, Nan Xiao wrote: >>> Hi David, >>> >>> Yes, the "FULL_PATH_TO_KEX" works: >>> >>> ~/Downloads/kernel/105.ops # stap -l >>> 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")' >>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21") >>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45") >>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33") >>> module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72") >>> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90") >>> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61") >>> >>> But the "RELATIVE_PATH_TO_KEX" doesn't work: >>> ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")' >>> ~/Downloads/kernel/105.ops # >>> >>> Is it the expected behavior of SystemTap? Thanks! >> >> Good question. You are correct, using relative paths doesn't seem to >> work. I've filed PR19265 on this issue. >> >> <https://sourceware.org/bugzilla/show_bug.cgi?id=19265> >> >> Feel free to add yourself to the CC list on that bug. If you have the >> time, also feel free to look into this more. >> >> -- >> David Smith >> dsmith@redhat.com >> Red Hat >> http://www.redhat.com >> 256.217.0141 (direct) >> 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-19 8:33 ` Nan Xiao @ 2015-11-19 14:58 ` David Smith 2015-11-26 6:10 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-11-19 14:58 UTC (permalink / raw) To: Nan Xiao; +Cc: systemtap On 11/19/2015 02:33 AM, Nan Xiao wrote: > Hi David, > > Another interesting thing: if there is a "-" in path, the SystemTap > also fail to parse: That's odd. I've added that information to the bug. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-19 14:58 ` David Smith @ 2015-11-26 6:10 ` Nan Xiao 2015-11-30 21:52 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-11-26 6:10 UTC (permalink / raw) To: David Smith; +Cc: systemtap Hi David, I do "insmod kex.ko" and "rmmod kex" repeatedly, but It seems the related probes can't be triggered: # stap -l 'module("/root/kernel/105.ops/kex.ko").function("*")' module("/root/kernel/105.ops/kex.ko").function("_open@/root/kernel/105.ops/kex.c:21") module("/root/kernel/105.ops/kex.ko").function("_read@/root/kernel/105.ops/kex.c:45") module("/root/kernel/105.ops/kex.ko").function("_release@/root/kernel/105.ops/kex.c:33") module("/root/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/kernels/3.10.0-123.el7.x86_64.debug/arch/x86/include/asm/uaccess_64.h:72") module("/root/kernel/105.ops/kex.ko").function("kex_cleanup@/root/kernel/105.ops/kex.c:90") module("/root/kernel/105.ops/kex.ko").function("kex_init@/root/kernel/105.ops/kex.c:61") # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("kex_init"), module("/root/kernel/105.ops/kex.ko").function("kex_cleanup") {print_backtrace()}' Could you help to check it? Thanks in advance! Best Regards Nan Xiao On Thu, Nov 19, 2015 at 10:57 PM, David Smith <dsmith@redhat.com> wrote: > On 11/19/2015 02:33 AM, Nan Xiao wrote: >> Hi David, >> >> Another interesting thing: if there is a "-" in path, the SystemTap >> also fail to parse: > > That's odd. I've added that information to the bug. > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-26 6:10 ` Nan Xiao @ 2015-11-30 21:52 ` David Smith 2015-11-30 23:45 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-11-30 21:52 UTC (permalink / raw) To: Nan Xiao; +Cc: systemtap On 11/26/2015 12:10 AM, Nan Xiao wrote: > Hi David, > > I do "insmod kex.ko" and "rmmod kex" repeatedly, but It seems the > related probes can't be triggered: > > # stap -l 'module("/root/kernel/105.ops/kex.ko").function("*")' > module("/root/kernel/105.ops/kex.ko").function("_open@/root/kernel/105.ops/kex.c:21") > module("/root/kernel/105.ops/kex.ko").function("_read@/root/kernel/105.ops/kex.c:45") > module("/root/kernel/105.ops/kex.ko").function("_release@/root/kernel/105.ops/kex.c:33") > module("/root/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/kernels/3.10.0-123.el7.x86_64.debug/arch/x86/include/asm/uaccess_64.h:72") > module("/root/kernel/105.ops/kex.ko").function("kex_cleanup@/root/kernel/105.ops/kex.c:90") > module("/root/kernel/105.ops/kex.ko").function("kex_init@/root/kernel/105.ops/kex.c:61") > # stap -e 'probe > module("/root/kernel/105.ops/kex.ko").function("kex_init"), > module("/root/kernel/105.ops/kex.ko").function("kex_cleanup") > {print_backtrace()}' Hmm. Note that I've done a good bit of work in the area of catching module init functions recently. What version of systemtap are you working with? Let's see if systemtap can catch anything in your module. Try something like: # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", ppfunc()) }' -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-30 21:52 ` David Smith @ 2015-11-30 23:45 ` Nan Xiao 2015-12-01 2:18 ` Frank Ch. Eigler 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-11-30 23:45 UTC (permalink / raw) To: David Smith; +Cc: systemtap Hi David, My OS is RHEL 7.0, but run on VirtualBox: # stap -V Systemtap translator/driver (version 2.4/0.158, rpm 2.4-14.el7) Copyright (C) 2005-2013 Red Hat, Inc. and others This is free software; see the source for copying conditions. enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS DYNINST JAVA LIBVIRT LIBXML2 run "# stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", ppfunc()) }'", but run "insmod kex.ko" & "rmmod kex" still outputs nothing. Best Regards Nan Xiao On Tue, Dec 1, 2015 at 5:52 AM, David Smith <dsmith@redhat.com> wrote: > On 11/26/2015 12:10 AM, Nan Xiao wrote: >> Hi David, >> >> I do "insmod kex.ko" and "rmmod kex" repeatedly, but It seems the >> related probes can't be triggered: >> >> # stap -l 'module("/root/kernel/105.ops/kex.ko").function("*")' >> module("/root/kernel/105.ops/kex.ko").function("_open@/root/kernel/105.ops/kex.c:21") >> module("/root/kernel/105.ops/kex.ko").function("_read@/root/kernel/105.ops/kex.c:45") >> module("/root/kernel/105.ops/kex.ko").function("_release@/root/kernel/105.ops/kex.c:33") >> module("/root/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/kernels/3.10.0-123.el7.x86_64.debug/arch/x86/include/asm/uaccess_64.h:72") >> module("/root/kernel/105.ops/kex.ko").function("kex_cleanup@/root/kernel/105.ops/kex.c:90") >> module("/root/kernel/105.ops/kex.ko").function("kex_init@/root/kernel/105.ops/kex.c:61") >> # stap -e 'probe >> module("/root/kernel/105.ops/kex.ko").function("kex_init"), >> module("/root/kernel/105.ops/kex.ko").function("kex_cleanup") >> {print_backtrace()}' > > Hmm. Note that I've done a good bit of work in the area of catching > module init functions recently. What version of systemtap are you > working with? > > Let's see if systemtap can catch anything in your module. Try something > like: > > # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { > printf("%s\n", ppfunc()) }' > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-11-30 23:45 ` Nan Xiao @ 2015-12-01 2:18 ` Frank Ch. Eigler 2015-12-01 2:57 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: Frank Ch. Eigler @ 2015-12-01 2:18 UTC (permalink / raw) To: Nan Xiao; +Cc: David Smith, systemtap Nan Xiao <xiaonan830818@gmail.com> writes: > [...] > run "# stap -e 'probe > module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", > ppfunc()) }'", but > run "insmod kex.ko" & "rmmod kex" still outputs nothing. Xan you exercise kex.ko beyond just installing & removing it right away, while the systemtap script is probing it? - FChE ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-01 2:18 ` Frank Ch. Eigler @ 2015-12-01 2:57 ` Nan Xiao 2015-12-01 16:27 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-12-01 2:57 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: David Smith, systemtap Hi Frank, I have just tested it: # dd if=kex of=/dev/null bs=101 count=1 0+1 records in 0+1 records out 100 bytes (100 B) copied, 0.00278675 s, 35.9 kB/s But still nothing outputs: # stap -v -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", ppfunc()) }' Pass 1: parsed user script and 100 library script(s) using 210544virt/28008res/3036shr/25484data kb, in 110usr/50sys/162real ms. Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 370776virt/29748res/3596shr/26612data kb, in 30usr/760sys/802real ms. Pass 3: using cached /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.c Pass 4: using cached /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.ko Pass 5: starting run. Thanks! Best Regards Nan Xiao On Tue, Dec 1, 2015 at 10:18 AM, Frank Ch. Eigler <fche@redhat.com> wrote: > Nan Xiao <xiaonan830818@gmail.com> writes: > >> [...] >> run "# stap -e 'probe >> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >> ppfunc()) }'", but >> run "insmod kex.ko" & "rmmod kex" still outputs nothing. > > Xan you exercise kex.ko beyond just installing & removing it right > away, while the systemtap script is probing it? > > - FChE ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-01 2:57 ` Nan Xiao @ 2015-12-01 16:27 ` David Smith 2015-12-02 2:22 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-12-01 16:27 UTC (permalink / raw) To: Nan Xiao, Frank Ch. Eigler; +Cc: systemtap On 11/30/2015 08:57 PM, Nan Xiao wrote: > Hi Frank, > > I have just tested it: > > # dd if=kex of=/dev/null bs=101 count=1 > 0+1 records in > 0+1 records out > 100 bytes (100 B) copied, 0.00278675 s, 35.9 kB/s > > But still nothing outputs: > # stap -v -e 'probe > module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", > ppfunc()) }' > Pass 1: parsed user script and 100 library script(s) using > 210544virt/28008res/3036shr/25484data kb, in 110usr/50sys/162real ms. > Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 > global(s) using 370776virt/29748res/3596shr/26612data kb, in > 30usr/760sys/802real ms. > Pass 3: using cached > /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.c > Pass 4: using cached > /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.ko > Pass 5: starting run. > > Thanks! > Best Regards > Nan Xiao > > > On Tue, Dec 1, 2015 at 10:18 AM, Frank Ch. Eigler <fche@redhat.com> wrote: >> Nan Xiao <xiaonan830818@gmail.com> writes: >> >>> [...] >>> run "# stap -e 'probe >>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >>> ppfunc()) }'", but >>> run "insmod kex.ko" & "rmmod kex" still outputs nothing. >> >> Xan you exercise kex.ko beyond just installing & removing it right >> away, while the systemtap script is probing it? >> >> - FChE Hmm. OK, let's try a couple more things: 1) It could be that systemtap is missing the module load somehow. So, try this: - load the module - run stap - exercise the module - unload the module - kill stap 2) It looks like you are running the rpm version. Can you install the systemtap-testsuite rpm and try the following (as root): # cd /usr/share/systemtap/testsuite # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" That should run 2 testcases that test out-of-tree modules and in-tree modules. -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-01 16:27 ` David Smith @ 2015-12-02 2:22 ` Nan Xiao 2015-12-02 21:42 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-12-02 2:22 UTC (permalink / raw) To: David Smith; +Cc: Frank Ch. Eigler, systemtap > Hmm. OK, let's try a couple more things: > 1) It could be that systemtap is missing the module load somehow. So, > try this: > - load the module > - run stap > - exercise the module > - unload the module > - kill stap Still outputs nothing. > 2) It looks like you are running the rpm version. Can you install the > systemtap-testsuite rpm and try the following (as root): > # cd /usr/share/systemtap/testsuite > # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" > That should run 2 testcases that test out-of-tree modules and in-tree > modules. The output likes this: # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" Making a new site.exp file ... rmmod uprobes 2>/dev/null make: [installcheck] Error 1 (ignored) make check-DEJAGNU RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp --tool_opts \'install \'" make[1]: Entering directory `/usr/share/systemtap/testsuite' srcdir='.'; export srcdir; \ EXPECT=expect; export EXPECT; \ if /bin/sh -c "env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest --version" > /dev/null 2>&1; then \ exit_status=0; l='systemtap'; for tool in $l; do \ if env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest --tool $tool --tool_opts \'\' --srcdir $srcdir modules_out_of_tree.exp kmodule.exp --tool_opts \'install \'; \ then :; else exit_status=1; fi; \ done; \ else echo "WARNING: could not find 'env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest'" 1>&2; :;\ fi; \ exit $exit_status WARNING: Couldn't find the global config file. kernel location: /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64.debug/vmlinux kernel version: 3.10.0-123.el7.x86_64.debug systemtap location: /usr/bin/stap systemtap version: version 2.4/0.158, rpm 2.4-14.el7 gcc location: /usr/bin/gcc gcc version: gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16) spawn -ignore SIGHUP gcc hello.c -g -m64 -lm -o hello-m64 spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32 /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgcc_s.so when searching for -lgcc_s /usr/bin/ld: cannot find -lgcc_s collect2: error: ld returned 1 exit status **** failed gcc m32 smoke test: /usr/bin/ld: cannot find -lgcc_s collect2: error: ld returned 1 exit status Please install libgcc and glibc development packages for m32 **** aborting testing. make[1]: Leaving directory `/usr/share/systemtap/testsuite' if test -n ""; then mail < systemtap.sum; fi Thanks! Best Regards Nan Xiao On Wed, Dec 2, 2015 at 12:27 AM, David Smith <dsmith@redhat.com> wrote: > On 11/30/2015 08:57 PM, Nan Xiao wrote: >> Hi Frank, >> >> I have just tested it: >> >> # dd if=kex of=/dev/null bs=101 count=1 >> 0+1 records in >> 0+1 records out >> 100 bytes (100 B) copied, 0.00278675 s, 35.9 kB/s >> >> But still nothing outputs: >> # stap -v -e 'probe >> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >> ppfunc()) }' >> Pass 1: parsed user script and 100 library script(s) using >> 210544virt/28008res/3036shr/25484data kb, in 110usr/50sys/162real ms. >> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 >> global(s) using 370776virt/29748res/3596shr/26612data kb, in >> 30usr/760sys/802real ms. >> Pass 3: using cached >> /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.c >> Pass 4: using cached >> /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.ko >> Pass 5: starting run. >> >> Thanks! >> Best Regards >> Nan Xiao >> >> >> On Tue, Dec 1, 2015 at 10:18 AM, Frank Ch. Eigler <fche@redhat.com> wrote: >>> Nan Xiao <xiaonan830818@gmail.com> writes: >>> >>>> [...] >>>> run "# stap -e 'probe >>>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >>>> ppfunc()) }'", but >>>> run "insmod kex.ko" & "rmmod kex" still outputs nothing. >>> >>> Xan you exercise kex.ko beyond just installing & removing it right >>> away, while the systemtap script is probing it? >>> >>> - FChE > > Hmm. OK, let's try a couple more things: > > 1) It could be that systemtap is missing the module load somehow. So, > try this: > > - load the module > - run stap > - exercise the module > - unload the module > - kill stap > > 2) It looks like you are running the rpm version. Can you install the > systemtap-testsuite rpm and try the following (as root): > > # cd /usr/share/systemtap/testsuite > # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" > > That should run 2 testcases that test out-of-tree modules and in-tree > modules. > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-02 2:22 ` Nan Xiao @ 2015-12-02 21:42 ` David Smith 2015-12-03 1:13 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-12-02 21:42 UTC (permalink / raw) To: Nan Xiao; +Cc: Frank Ch. Eigler, systemtap On 12/01/2015 08:22 PM, Nan Xiao wrote: >> Hmm. OK, let's try a couple more things: > >> 1) It could be that systemtap is missing the module load somehow. So, >> try this: > >> - load the module >> - run stap >> - exercise the module >> - unload the module >> - kill stap > > Still outputs nothing. Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap command and show us the output? >> 2) It looks like you are running the rpm version. Can you install the >> systemtap-testsuite rpm and try the following (as root): > >> # cd /usr/share/systemtap/testsuite >> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" > >> That should run 2 testcases that test out-of-tree modules and in-tree >> modules. > > The output likes this: > > # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" ... stuff deleted ... > spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32 > /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgcc_s.so when searching for > -lgcc_s > /usr/bin/ld: cannot find -lgcc_s > collect2: error: ld returned 1 exit status > > > > **** failed gcc m32 smoke test: > > /usr/bin/ld: cannot find -lgcc_s > collect2: error: ld returned 1 exit status > > Please install libgcc and glibc development packages for m32 The testsuite checks a few things before starting, one of them being that gcc can create 32-bit executables on an x86_64 system. If you want to proceed further here, you'll have to do the following: # yum install glibc.i686 libgcc.i686 glibc-devel.i686 libstdc++-devel.i686 -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-02 21:42 ` David Smith @ 2015-12-03 1:13 ` Nan Xiao 2015-12-03 17:04 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-12-03 1:13 UTC (permalink / raw) To: David Smith; +Cc: Frank Ch. Eigler, systemtap Hi David, (1) >>> Hmm. OK, let's try a couple more things: >> >>> 1) It could be that systemtap is missing the module load somehow. So, >>> try this: >> >>> - load the module >>> - run stap >>> - exercise the module >>> - unload the module >>> - kill stap >> >> Still outputs nothing. > Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap > command and show us the output? After adding '-DDEBUG_KPROBES', the stap command still outputs nothing. (2) The output of executing "# make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"": # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" make: *** No rule to make target `installcheck'. Stop. [root@localhost yum.repos.d]# cd /usr/share/systemtap/testsuite [root@localhost testsuite]# make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" rmmod uprobes 2>/dev/null make: [installcheck] Error 1 (ignored) make check-DEJAGNU RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp --tool_opts \'install \'" make[1]: Entering directory `/usr/share/systemtap/testsuite' srcdir='.'; export srcdir; \ EXPECT=expect; export EXPECT; \ if /bin/sh -c "env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest --version" > /dev/null 2>&1; then \ exit_status=0; l='systemtap'; for tool in $l; do \ if env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest --tool $tool --tool_opts \'\' --srcdir $srcdir modules_out_of_tree.exp kmodule.exp --tool_opts \'install \'; \ then :; else exit_status=1; fi; \ done; \ else echo "WARNING: could not find 'env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin SYSTEMTAP_INCLUDES=/usr/include PKGLIBDIR=/usr/libexec/systemtap ./execrc runtest'" 1>&2; :;\ fi; \ exit $exit_status WARNING: Couldn't find the global config file. kernel location: /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64.debug/vmlinux kernel version: 3.10.0-123.el7.x86_64.debug systemtap location: /usr/bin/stap systemtap version: version 2.4/0.158, rpm 2.4-14.el7 gcc location: /usr/bin/gcc gcc version: gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16) spawn -ignore SIGHUP gcc hello.c -g -m64 -lm -o hello-m64 spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32 spawn -ignore SIGHUP g++ hello.cxx -g -m64 -lm -o hello-m64 spawn -ignore SIGHUP g++ hello.cxx -g -m32 -lm -o hello-m32 Test Run By root on Wed Dec 2 20:08:12 2015 Native configuration is x86_64-unknown-linux-gnu === systemtap tests === Schedule of variations: unix Running target unix Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target. Using /usr/share/dejagnu/config/unix.exp as generic interface file for target. Using ./config/unix.exp as tool-and-target-specific interface file. Host: Linux localhost.localdomain 3.10.0-123.el7.x86_64.debug #1 SMP Mon May 5 11:24:18 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux Snapshot: version 2.4/0.158, rpm 2.4-14.el7 GCC: 4.8.2 [gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)] Distro: Red Hat Enterprise Linux Server release 7.0 (Maipo) SElinux: Enforcing Running ./systemtap.base/kmodule.exp ... === systemtap Summary === # of expected passes 7 make[1]: Leaving directory `/usr/share/systemtap/testsuite' if test -n ""; then mail < systemtap.sum; fi Thanks! Best Regards Nan Xiao On Thu, Dec 3, 2015 at 5:42 AM, David Smith <dsmith@redhat.com> wrote: > On 12/01/2015 08:22 PM, Nan Xiao wrote: >>> Hmm. OK, let's try a couple more things: >> >>> 1) It could be that systemtap is missing the module load somehow. So, >>> try this: >> >>> - load the module >>> - run stap >>> - exercise the module >>> - unload the module >>> - kill stap >> >> Still outputs nothing. > > Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap > command and show us the output? > >>> 2) It looks like you are running the rpm version. Can you install the >>> systemtap-testsuite rpm and try the following (as root): >> >>> # cd /usr/share/systemtap/testsuite >>> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" >> >>> That should run 2 testcases that test out-of-tree modules and in-tree >>> modules. >> >> The output likes this: >> >> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" > > ... stuff deleted ... > >> spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32 >> /usr/bin/ld: skipping incompatible >> /usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgcc_s.so when searching for >> -lgcc_s >> /usr/bin/ld: cannot find -lgcc_s >> collect2: error: ld returned 1 exit status >> >> >> >> **** failed gcc m32 smoke test: >> >> /usr/bin/ld: cannot find -lgcc_s >> collect2: error: ld returned 1 exit status >> >> Please install libgcc and glibc development packages for m32 > > The testsuite checks a few things before starting, one of them being > that gcc can create 32-bit executables on an x86_64 system. If you want > to proceed further here, you'll have to do the following: > > # yum install glibc.i686 libgcc.i686 glibc-devel.i686 libstdc++-devel.i686 > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-03 1:13 ` Nan Xiao @ 2015-12-03 17:04 ` David Smith 2015-12-04 9:56 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-12-03 17:04 UTC (permalink / raw) To: Nan Xiao; +Cc: Frank Ch. Eigler, systemtap On 12/02/2015 07:13 PM, Nan Xiao wrote: > Hi David, > > (1) >>>> Hmm. OK, let's try a couple more things: >>> >>>> 1) It could be that systemtap is missing the module load somehow. So, >>>> try this: >>> >>>> - load the module >>>> - run stap >>>> - exercise the module >>>> - unload the module >>>> - kill stap >>> >>> Still outputs nothing. > >> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap >> command and show us the output? > > After adding '-DDEBUG_KPROBES', the stap command still outputs nothing. Hmm. > (2) > [root@localhost testsuite]# make installcheck > RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" > > > Running ./systemtap.base/kmodule.exp ... > > === systemtap Summary === > > # of expected passes 7 Ah. It ran kmodule.exp, but not modules_out_of_tree.exp. You are running systemtap 2.4, and that testcase was first present in systemtap 2.6. I was testing another bug so I had a RHEL 7.0 systemtap set up and I installed systemtap 2.4 on it. It looks like that version can't probe out-of-tree modules. It looks like you've got a couple of options: 1) Copy your module into the kernel module tree (located at /lib/modules/`uname -r`/kernel) to use systemtap on it. 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 system, and the modules_out_of_tree.exp test case passed there.) -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-03 17:04 ` David Smith @ 2015-12-04 9:56 ` Nan Xiao 2015-12-04 12:15 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-12-04 9:56 UTC (permalink / raw) To: David Smith; +Cc: Frank Ch. Eigler, systemtap Hi Dave, > It looks like you've got a couple of options: > 1) Copy your module into the kernel module tree (located at > /lib/modules/`uname -r`/kernel) to use systemtap on it. I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always executes error: # cd /lib/modules/`uname -r`/kernel # pwd /lib/modules/3.10.0-123.el7.x86_64.debug/kernel # ls arch crypto drivers fs kernel kex.ko lib mm net sound # stap -v -e 'probe module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") { printf("%s\n", ppfunc()) }' Pass 1: parsed user script and 103 library script(s) using 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. semantic error: while resolving probe point: identifier 'module' at <input>:1:7 source: probe module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") { printf("%s\n", ppfunc()) } ^ semantic error: no match Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 372028virt/33024res/3412shr/30124data kb, in 20usr/770sys/804real ms. Pass 2: analysis failed. [man error::pass2] # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", ppfunc()) }' Pass 1: parsed user script and 103 library script(s) using 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. semantic error: while resolving probe point: identifier 'module' at <input>:1:7 source: probe module("kex.ko").function("*") { printf("%s\n", ppfunc()) } ^ semantic error: no match Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 372032virt/33016res/3412shr/30128data kb, in 40usr/880sys/929real ms. Pass 2: analysis failed. [man error::pass2] > 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 > system, and the modules_out_of_tree.exp test case passed there.) I install the RHEL 7.2, and it really works! But it can't print actual stack traces: # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { print_backtrace() }' WARNING: no or bad debug frame hdr WARNING: No binary search table for debug frame, doing slow linear search for /root/kernel/105.ops/kex.ko 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] 0xffffffff810020e8 (inexact) 0xffffffff810ed4ae (inexact) 0xffffffff81316880 (inexact) 0xffffffff810e9743 (inexact) 0xffffffff810ede66 (inexact) 0xffffffff81645909 (inexact) 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] 0xffffffff810eb23b (inexact) 0xffffffff81641113 (inexact) 0xffffffff81014b12 (inexact) 0xffffffff81645909 (inexact) So it means the module isn't built with debuginfo, right? Thanks very much for your time & effort! Best Regards Nan Xiao On Fri, Dec 4, 2015 at 1:03 AM, David Smith <dsmith@redhat.com> wrote: > On 12/02/2015 07:13 PM, Nan Xiao wrote: >> Hi David, >> >> (1) >>>>> Hmm. OK, let's try a couple more things: >>>> >>>>> 1) It could be that systemtap is missing the module load somehow. So, >>>>> try this: >>>> >>>>> - load the module >>>>> - run stap >>>>> - exercise the module >>>>> - unload the module >>>>> - kill stap >>>> >>>> Still outputs nothing. >> >>> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap >>> command and show us the output? >> >> After adding '-DDEBUG_KPROBES', the stap command still outputs nothing. > > Hmm. > >> (2) >> [root@localhost testsuite]# make installcheck >> RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp" >> >> >> Running ./systemtap.base/kmodule.exp ... >> >> === systemtap Summary === >> >> # of expected passes 7 > > Ah. It ran kmodule.exp, but not modules_out_of_tree.exp. You are running > systemtap 2.4, and that testcase was first present in systemtap 2.6. > > I was testing another bug so I had a RHEL 7.0 systemtap set up and I > installed systemtap 2.4 on it. It looks like that version can't probe > out-of-tree modules. > > It looks like you've got a couple of options: > > 1) Copy your module into the kernel module tree (located at > /lib/modules/`uname -r`/kernel) to use systemtap on it. > > 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 > system, and the modules_out_of_tree.exp test case passed there.) > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-04 9:56 ` Nan Xiao @ 2015-12-04 12:15 ` David Smith 2015-12-06 8:53 ` Nan Xiao 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2015-12-04 12:15 UTC (permalink / raw) To: Nan Xiao; +Cc: Frank Ch. Eigler, systemtap On 12/04/2015 03:56 AM, Nan Xiao wrote: > Hi Dave, > >> It looks like you've got a couple of options: > >> 1) Copy your module into the kernel module tree (located at >> /lib/modules/`uname -r`/kernel) to use systemtap on it. > > I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always > executes error: > > # cd /lib/modules/`uname -r`/kernel > # pwd > /lib/modules/3.10.0-123.el7.x86_64.debug/kernel > # ls > arch crypto drivers fs kernel kex.ko lib mm net sound > > # stap -v -e 'probe > module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") > { printf("%s\n", ppfunc()) }' > Pass 1: parsed user script and 103 library script(s) using > 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. > semantic error: while resolving probe point: identifier 'module' at <input>:1:7 > source: probe > module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") > { printf("%s\n", ppfunc()) } > ^ > > semantic error: no match > Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 > global(s) using 372028virt/33024res/3412shr/30124data kb, in > 20usr/770sys/804real ms. > Pass 2: analysis failed. [man error::pass2] > > # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", > ppfunc()) }' Pass 1: parsed user > script and 103 library script(s) using > 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. > semantic error: while resolving probe point: identifier 'module' at <input>:1:7 > source: probe module("kex.ko").function("*") { printf("%s\n", > ppfunc()) } > ^ > > semantic error: no match > Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 > global(s) using 372032virt/33016res/3412shr/30128data kb, in > 40usr/880sys/929real ms. > Pass 2: analysis failed. [man error::pass2] I see I didn't explain well here. Once you copied your module to the kernel's module directory, systemtap no longer would need a full path to the module name - just "kex" should have worked. >> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 >> system, and the modules_out_of_tree.exp test case passed there.) > > I install the RHEL 7.2, and it really works! But it can't print actual > stack traces: > > # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { > print_backtrace() }' > WARNING: no or bad debug frame hdr > WARNING: No binary search table for debug frame, doing slow linear > search for /root/kernel/105.ops/kex.ko > 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] > 0xffffffff810020e8 (inexact) > 0xffffffff810ed4ae (inexact) > 0xffffffff81316880 (inexact) > 0xffffffff810e9743 (inexact) > 0xffffffff810ede66 (inexact) > 0xffffffff81645909 (inexact) > 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] > 0xffffffff810eb23b (inexact) > 0xffffffff81641113 (inexact) > 0xffffffff81014b12 (inexact) > 0xffffffff81645909 (inexact) > > So it means the module isn't built with debuginfo, right? Hmm, it could mean that we just need to make sure that the debuginfo is loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the stap command line. (The '-d' option asks stap to load the symbol/unwind information for a module even if stap doesn't think it needs it.) -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-04 12:15 ` David Smith @ 2015-12-06 8:53 ` Nan Xiao 2015-12-07 19:53 ` David Smith 0 siblings, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-12-06 8:53 UTC (permalink / raw) To: David Smith; +Cc: Frank Ch. Eigler, systemtap Hi David, > (1) > I see I didn't explain well here. Once you copied your module to the > kernel's module directory, systemtap no longer would need a full path to > the module name - just "kex" should have worked. Yeah, it worked! > (2) > Hmm, it could mean that we just need to make sure that the debuginfo is > loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the > stap command line. (The '-d' option asks stap to load the symbol/unwind > information for a module even if stap doesn't think it needs it.) The output stills have error: # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", print_backtrace()) }' Pass 1: parsed user script and 110 library script(s) using 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms. Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 221348virt/39612res/3832shr/36156data kb, in 20usr/60sys/114real ms. Pass 3: translated to C into "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c" using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real ms. Pass 4: compiled C into "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in 2280usr/540sys/5099real ms. Pass 5: starting run. WARNING: no or bad debug frame hdr WARNING: No binary search table for debug frame, doing slow linear search for /root/kernel/105.ops/kex.ko 0xffffffffa0037000 : kex_init+0x0/0x0 [kex] 0xffffffff810020e8 (inexact) 0xffffffff810ed4ae (inexact) 0xffffffff81316880 (inexact) 0xffffffff810e9743 (inexact) 0xffffffff810ede66 (inexact) 0xffffffff81645909 (inexact) 0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex] 0xffffffff810eb23b (inexact) 0xffffffff81641113 (inexact) 0xffffffff81014b12 (inexact) 0xffffffff81645909 (inexact) Thanks! Best Regards Nan Xiao On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote: > On 12/04/2015 03:56 AM, Nan Xiao wrote: >> Hi Dave, >> >>> It looks like you've got a couple of options: >> >>> 1) Copy your module into the kernel module tree (located at >>> /lib/modules/`uname -r`/kernel) to use systemtap on it. >> >> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always >> executes error: >> >> # cd /lib/modules/`uname -r`/kernel >> # pwd >> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel >> # ls >> arch crypto drivers fs kernel kex.ko lib mm net sound >> >> # stap -v -e 'probe >> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >> { printf("%s\n", ppfunc()) }' >> Pass 1: parsed user script and 103 library script(s) using >> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. >> semantic error: while resolving probe point: identifier 'module' at <input>:1:7 >> source: probe >> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >> { printf("%s\n", ppfunc()) } >> ^ >> >> semantic error: no match >> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >> global(s) using 372028virt/33024res/3412shr/30124data kb, in >> 20usr/770sys/804real ms. >> Pass 2: analysis failed. [man error::pass2] >> >> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", >> ppfunc()) }' Pass 1: parsed user >> script and 103 library script(s) using >> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. >> semantic error: while resolving probe point: identifier 'module' at <input>:1:7 >> source: probe module("kex.ko").function("*") { printf("%s\n", >> ppfunc()) } >> ^ >> >> semantic error: no match >> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >> global(s) using 372032virt/33016res/3412shr/30128data kb, in >> 40usr/880sys/929real ms. >> Pass 2: analysis failed. [man error::pass2] > > I see I didn't explain well here. Once you copied your module to the > kernel's module directory, systemtap no longer would need a full path to > the module name - just "kex" should have worked. > >>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 >>> system, and the modules_out_of_tree.exp test case passed there.) >> >> I install the RHEL 7.2, and it really works! But it can't print actual >> stack traces: >> >> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { >> print_backtrace() }' >> WARNING: no or bad debug frame hdr >> WARNING: No binary search table for debug frame, doing slow linear >> search for /root/kernel/105.ops/kex.ko >> 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] >> 0xffffffff810020e8 (inexact) >> 0xffffffff810ed4ae (inexact) >> 0xffffffff81316880 (inexact) >> 0xffffffff810e9743 (inexact) >> 0xffffffff810ede66 (inexact) >> 0xffffffff81645909 (inexact) >> 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] >> 0xffffffff810eb23b (inexact) >> 0xffffffff81641113 (inexact) >> 0xffffffff81014b12 (inexact) >> 0xffffffff81645909 (inexact) >> >> So it means the module isn't built with debuginfo, right? > > Hmm, it could mean that we just need to make sure that the debuginfo is > loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the > stap command line. (The '-d' option asks stap to load the symbol/unwind > information for a module even if stap doesn't think it needs it.) > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-06 8:53 ` Nan Xiao @ 2015-12-07 19:53 ` David Smith 2015-12-08 1:05 ` Nan Xiao 2016-10-06 22:05 ` Martin Cermak 0 siblings, 2 replies; 26+ messages in thread From: David Smith @ 2015-12-07 19:53 UTC (permalink / raw) To: Nan Xiao; +Cc: Frank Ch. Eigler, systemtap On 12/06/2015 02:53 AM, Nan Xiao wrote: > Hi David, > >> (1) >> I see I didn't explain well here. Once you copied your module to the >> kernel's module directory, systemtap no longer would need a full path to >> the module name - just "kex" should have worked. > > Yeah, it worked! > >> (2) >> Hmm, it could mean that we just need to make sure that the debuginfo is >> loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the >> stap command line. (The '-d' option asks stap to load the symbol/unwind >> information for a module even if stap doesn't think it needs it.) > > The output stills have error: > > # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe > module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", > print_backtrace()) }' > Pass 1: parsed user script and 110 library script(s) using > 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms. > Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 > global(s) using 221348virt/39612res/3832shr/36156data kb, in > 20usr/60sys/114real ms. > Pass 3: translated to C into > "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c" > using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real > ms. > Pass 4: compiled C into > "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in > 2280usr/540sys/5099real ms. > Pass 5: starting run. > WARNING: no or bad debug frame hdr > WARNING: No binary search table for debug frame, doing slow linear > search for /root/kernel/105.ops/kex.ko > 0xffffffffa0037000 : kex_init+0x0/0x0 [kex] > 0xffffffff810020e8 (inexact) > 0xffffffff810ed4ae (inexact) > 0xffffffff81316880 (inexact) > 0xffffffff810e9743 (inexact) > 0xffffffff810ede66 (inexact) > 0xffffffff81645909 (inexact) > > 0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex] > 0xffffffff810eb23b (inexact) > 0xffffffff81641113 (inexact) > 0xffffffff81014b12 (inexact) > 0xffffffff81645909 (inexact) Hmm. OK, let's make sure of a couple of things: 1) Make sure your kernel was compiled with frame pointer support (which it should if it is a standard RHEL kernel): # fgrep FRAME_POINTER /boot/config-`uname -r` CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_ARCH_WANT_FRAME_POINTERS=y CONFIG_FRAME_POINTER=y 2) Let's make sure your module still has debuginfo present. First, run "file" on your module, making sure it says "not stripped". Then go a bit deeper and run "readelf --sections" on your module - there should be several sections that start with '.debug', including one called '.debug_frame'. > > Thanks! > Best Regards > Nan Xiao > > > On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote: >> On 12/04/2015 03:56 AM, Nan Xiao wrote: >>> Hi Dave, >>> >>>> It looks like you've got a couple of options: >>> >>>> 1) Copy your module into the kernel module tree (located at >>>> /lib/modules/`uname -r`/kernel) to use systemtap on it. >>> >>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always >>> executes error: >>> >>> # cd /lib/modules/`uname -r`/kernel >>> # pwd >>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel >>> # ls >>> arch crypto drivers fs kernel kex.ko lib mm net sound >>> >>> # stap -v -e 'probe >>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >>> { printf("%s\n", ppfunc()) }' >>> Pass 1: parsed user script and 103 library script(s) using >>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. >>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7 >>> source: probe >>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >>> { printf("%s\n", ppfunc()) } >>> ^ >>> >>> semantic error: no match >>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >>> global(s) using 372028virt/33024res/3412shr/30124data kb, in >>> 20usr/770sys/804real ms. >>> Pass 2: analysis failed. [man error::pass2] >>> >>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", >>> ppfunc()) }' Pass 1: parsed user >>> script and 103 library script(s) using >>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. >>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7 >>> source: probe module("kex.ko").function("*") { printf("%s\n", >>> ppfunc()) } >>> ^ >>> >>> semantic error: no match >>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >>> global(s) using 372032virt/33016res/3412shr/30128data kb, in >>> 40usr/880sys/929real ms. >>> Pass 2: analysis failed. [man error::pass2] >> >> I see I didn't explain well here. Once you copied your module to the >> kernel's module directory, systemtap no longer would need a full path to >> the module name - just "kex" should have worked. >> >>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 >>>> system, and the modules_out_of_tree.exp test case passed there.) >>> >>> I install the RHEL 7.2, and it really works! But it can't print actual >>> stack traces: >>> >>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { >>> print_backtrace() }' >>> WARNING: no or bad debug frame hdr >>> WARNING: No binary search table for debug frame, doing slow linear >>> search for /root/kernel/105.ops/kex.ko >>> 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] >>> 0xffffffff810020e8 (inexact) >>> 0xffffffff810ed4ae (inexact) >>> 0xffffffff81316880 (inexact) >>> 0xffffffff810e9743 (inexact) >>> 0xffffffff810ede66 (inexact) >>> 0xffffffff81645909 (inexact) >>> 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] >>> 0xffffffff810eb23b (inexact) >>> 0xffffffff81641113 (inexact) >>> 0xffffffff81014b12 (inexact) >>> 0xffffffff81645909 (inexact) >>> >>> So it means the module isn't built with debuginfo, right? >> >> Hmm, it could mean that we just need to make sure that the debuginfo is >> loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the >> stap command line. (The '-d' option asks stap to load the symbol/unwind >> information for a module even if stap doesn't think it needs it.) >> >> -- >> David Smith >> dsmith@redhat.com >> Red Hat >> http://www.redhat.com >> 256.217.0141 (direct) >> 256.837.0057 (fax) -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-07 19:53 ` David Smith @ 2015-12-08 1:05 ` Nan Xiao 2015-12-08 19:41 ` Frank Ch. Eigler 2016-10-06 22:05 ` Martin Cermak 1 sibling, 1 reply; 26+ messages in thread From: Nan Xiao @ 2015-12-08 1:05 UTC (permalink / raw) To: David Smith; +Cc: Frank Ch. Eigler, systemtap Hi David, > 1) Make sure your kernel was compiled with frame pointer support (which > it should if it is a standard RHEL kernel): > # fgrep FRAME_POINTER /boot/config-`uname -r` > CONFIG_SCHED_OMIT_FRAME_POINTER=y > CONFIG_ARCH_WANT_FRAME_POINTERS=y > CONFIG_FRAME_POINTER=y Yes, all the CONFIG options are "yes". > 2) Let's make sure your module still has debuginfo present. First, run > "file" on your module, making sure it says "not stripped". Yes, it is "not stripped". > Then go a bit deeper and run "readelf --sections" on your module - there should be > several sections that start with '.debug', including one called > '.debug_frame'. Below are all the sections with `.debug`: ...... [19] .debug_info PROGBITS 0000000000000000 000008e0 000000000000b6d9 0000000000000000 0 0 1 [20] .rela.debug_info RELA 0000000000000000 00014fe0 000000000000ed00 0000000000000018 36 19 8 [21] .debug_abbrev PROGBITS 0000000000000000 0000bfb9 00000000000007cc 0000000000000000 0 0 1 [22] .debug_loc PROGBITS 0000000000000000 0000c785 0000000000000324 0000000000000000 0 0 1 [23] .rela.debug_loc RELA 0000000000000000 00023ce0 0000000000000570 0000000000000018 36 22 8 [24] .debug_aranges PROGBITS 0000000000000000 0000caa9 0000000000000070 0000000000000000 0 0 1 [25] .rela.debug_arang RELA 0000000000000000 00024250 0000000000000078 0000000000000018 36 24 8 [26] .debug_ranges PROGBITS 0000000000000000 0000cb19 0000000000000040 0000000000000000 0 0 1 [27] .rela.debug_range RELA 0000000000000000 000242c8 0000000000000090 0000000000000018 36 26 8 [28] .debug_line PROGBITS 0000000000000000 0000cb59 0000000000000a8e 0000000000000000 0 0 1 [29] .rela.debug_line RELA 0000000000000000 00024358 0000000000000048 0000000000000018 36 28 8 [30] .debug_str PROGBITS 0000000000000000 0000d5e7 00000000000069fc 0000000000000001 MS 0 0 1 [31] .comment PROGBITS 0000000000000000 00013fe3 000000000000005a 0000000000000001 MS 0 0 1 [32] .note.GNU-stack PROGBITS 0000000000000000 0001403d 0000000000000000 0000000000000000 0 0 1 [33] .debug_frame PROGBITS 0000000000000000 00014040 00000000000000f0 0000000000000000 0 0 8 [34] .rela.debug_frame RELA 0000000000000000 000243a0 00000000000000f0 0000000000000018 36 33 8 ...... Thanks! Best Regards Nan Xiao On Tue, Dec 8, 2015 at 3:53 AM, David Smith <dsmith@redhat.com> wrote: > On 12/06/2015 02:53 AM, Nan Xiao wrote: >> Hi David, >> >>> (1) >>> I see I didn't explain well here. Once you copied your module to the >>> kernel's module directory, systemtap no longer would need a full path to >>> the module name - just "kex" should have worked. >> >> Yeah, it worked! >> >>> (2) >>> Hmm, it could mean that we just need to make sure that the debuginfo is >>> loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the >>> stap command line. (The '-d' option asks stap to load the symbol/unwind >>> information for a module even if stap doesn't think it needs it.) >> >> The output stills have error: >> >> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe >> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >> print_backtrace()) }' >> Pass 1: parsed user script and 110 library script(s) using >> 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms. >> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0 >> global(s) using 221348virt/39612res/3832shr/36156data kb, in >> 20usr/60sys/114real ms. >> Pass 3: translated to C into >> "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c" >> using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real >> ms. >> Pass 4: compiled C into >> "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in >> 2280usr/540sys/5099real ms. >> Pass 5: starting run. >> WARNING: no or bad debug frame hdr >> WARNING: No binary search table for debug frame, doing slow linear >> search for /root/kernel/105.ops/kex.ko >> 0xffffffffa0037000 : kex_init+0x0/0x0 [kex] >> 0xffffffff810020e8 (inexact) >> 0xffffffff810ed4ae (inexact) >> 0xffffffff81316880 (inexact) >> 0xffffffff810e9743 (inexact) >> 0xffffffff810ede66 (inexact) >> 0xffffffff81645909 (inexact) >> >> 0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex] >> 0xffffffff810eb23b (inexact) >> 0xffffffff81641113 (inexact) >> 0xffffffff81014b12 (inexact) >> 0xffffffff81645909 (inexact) > > Hmm. OK, let's make sure of a couple of things: > > 1) Make sure your kernel was compiled with frame pointer support (which > it should if it is a standard RHEL kernel): > > # fgrep FRAME_POINTER /boot/config-`uname -r` > CONFIG_SCHED_OMIT_FRAME_POINTER=y > CONFIG_ARCH_WANT_FRAME_POINTERS=y > CONFIG_FRAME_POINTER=y > > 2) Let's make sure your module still has debuginfo present. First, run > "file" on your module, making sure it says "not stripped". Then go a bit > deeper and run "readelf --sections" on your module - there should be > several sections that start with '.debug', including one called > '.debug_frame'. > >> >> Thanks! >> Best Regards >> Nan Xiao >> >> >> On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote: >>> On 12/04/2015 03:56 AM, Nan Xiao wrote: >>>> Hi Dave, >>>> >>>>> It looks like you've got a couple of options: >>>> >>>>> 1) Copy your module into the kernel module tree (located at >>>>> /lib/modules/`uname -r`/kernel) to use systemtap on it. >>>> >>>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always >>>> executes error: >>>> >>>> # cd /lib/modules/`uname -r`/kernel >>>> # pwd >>>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel >>>> # ls >>>> arch crypto drivers fs kernel kex.ko lib mm net sound >>>> >>>> # stap -v -e 'probe >>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >>>> { printf("%s\n", ppfunc()) }' >>>> Pass 1: parsed user script and 103 library script(s) using >>>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms. >>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7 >>>> source: probe >>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*") >>>> { printf("%s\n", ppfunc()) } >>>> ^ >>>> >>>> semantic error: no match >>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >>>> global(s) using 372028virt/33024res/3412shr/30124data kb, in >>>> 20usr/770sys/804real ms. >>>> Pass 2: analysis failed. [man error::pass2] >>>> >>>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n", >>>> ppfunc()) }' Pass 1: parsed user >>>> script and 103 library script(s) using >>>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms. >>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7 >>>> source: probe module("kex.ko").function("*") { printf("%s\n", >>>> ppfunc()) } >>>> ^ >>>> >>>> semantic error: no match >>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 >>>> global(s) using 372032virt/33016res/3412shr/30128data kb, in >>>> 40usr/880sys/929real ms. >>>> Pass 2: analysis failed. [man error::pass2] >>> >>> I see I didn't explain well here. Once you copied your module to the >>> kernel's module directory, systemtap no longer would need a full path to >>> the module name - just "kex" should have worked. >>> >>>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1 >>>>> system, and the modules_out_of_tree.exp test case passed there.) >>>> >>>> I install the RHEL 7.2, and it really works! But it can't print actual >>>> stack traces: >>>> >>>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") { >>>> print_backtrace() }' >>>> WARNING: no or bad debug frame hdr >>>> WARNING: No binary search table for debug frame, doing slow linear >>>> search for /root/kernel/105.ops/kex.ko >>>> 0xffffffffa006b000 : kex_init+0x0/0x0 [kex] >>>> 0xffffffff810020e8 (inexact) >>>> 0xffffffff810ed4ae (inexact) >>>> 0xffffffff81316880 (inexact) >>>> 0xffffffff810e9743 (inexact) >>>> 0xffffffff810ede66 (inexact) >>>> 0xffffffff81645909 (inexact) >>>> 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex] >>>> 0xffffffff810eb23b (inexact) >>>> 0xffffffff81641113 (inexact) >>>> 0xffffffff81014b12 (inexact) >>>> 0xffffffff81645909 (inexact) >>>> >>>> So it means the module isn't built with debuginfo, right? >>> >>> Hmm, it could mean that we just need to make sure that the debuginfo is >>> loaded. Try adding the '-d /root/kernel/105.ops/kex.ko' option to the >>> stap command line. (The '-d' option asks stap to load the symbol/unwind >>> information for a module even if stap doesn't think it needs it.) >>> >>> -- >>> David Smith >>> dsmith@redhat.com >>> Red Hat >>> http://www.redhat.com >>> 256.217.0141 (direct) >>> 256.837.0057 (fax) > > > -- > David Smith > dsmith@redhat.com > Red Hat > http://www.redhat.com > 256.217.0141 (direct) > 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-08 1:05 ` Nan Xiao @ 2015-12-08 19:41 ` Frank Ch. Eigler 0 siblings, 0 replies; 26+ messages in thread From: Frank Ch. Eigler @ 2015-12-08 19:41 UTC (permalink / raw) To: Nan Xiao; +Cc: David Smith, systemtap Nan Xiao <xiaonan830818@gmail.com> writes: > [...] >>> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe >>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n", >>> print_backtrace()) }' >>> [...] >>> 2280usr/540sys/5099real ms. >>> Pass 5: starting run. >>> WARNING: no or bad debug frame hdr >>> WARNING: No binary search table for debug frame, doing slow linear >>> search for /root/kernel/105.ops/kex.ko I wonder if something is going wrong with the module pathname logic. Could you try a few more things? - installing kex.ko under the kernel install tree (probably /lib/modules/`uname -r`/kernel or similar), so that stap -d kex -e 'probe module("kex").... {}' works (without the path name). - running stap in save-temp-directory (-k) mode, and shipping us a tarball of the /tmp/XXX directory, to see specifically how the stap-symbols.h file looks. (email direct rather than to the mailing list; this file may be big) - running stap in -DDEBUG_UNWIND=2 mode to get lots of backtracing-related data - FChE ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2015-12-07 19:53 ` David Smith 2015-12-08 1:05 ` Nan Xiao @ 2016-10-06 22:05 ` Martin Cermak 2016-10-07 17:59 ` David Smith 1 sibling, 1 reply; 26+ messages in thread From: Martin Cermak @ 2016-10-06 22:05 UTC (permalink / raw) To: David Smith; +Cc: Nan Xiao, Frank Ch. Eigler, systemtap On Mon 2015-12-07 13:53 , David Smith wrote: > 1) Make sure your kernel was compiled with frame pointer support (which > it should if it is a standard RHEL kernel): > > # fgrep FRAME_POINTER /boot/config-`uname -r` > CONFIG_SCHED_OMIT_FRAME_POINTER=y > CONFIG_ARCH_WANT_FRAME_POINTERS=y > CONFIG_FRAME_POINTER=y > > 2) Let's make sure your module still has debuginfo present. First, run > "file" on your module, making sure it says "not stripped". Then go a bit > deeper and run "readelf --sections" on your module - there should be > several sections that start with '.debug', including one called > '.debug_frame'. I'm traying to trace my own stap module too. I'm using stap -B CONFIG_DEBUG_INFO=y to put the dfebuginfo into the module. However, this way I seem to get the '.debug_frame' section on RHEL6, but not on RHEL7 or recent Fedora. Because of this I'm failing to get reasonable backtraces. Any workaround? Martin ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2016-10-06 22:05 ` Martin Cermak @ 2016-10-07 17:59 ` David Smith 2016-10-12 17:02 ` Martin Cermak 0 siblings, 1 reply; 26+ messages in thread From: David Smith @ 2016-10-07 17:59 UTC (permalink / raw) To: Martin Cermak; +Cc: Nan Xiao, Frank Ch. Eigler, systemtap On 10/06/2016 05:05 PM, Martin Cermak wrote: > On Mon 2015-12-07 13:53 , David Smith wrote: >> 1) Make sure your kernel was compiled with frame pointer support (which >> it should if it is a standard RHEL kernel): >> >> # fgrep FRAME_POINTER /boot/config-`uname -r` >> CONFIG_SCHED_OMIT_FRAME_POINTER=y >> CONFIG_ARCH_WANT_FRAME_POINTERS=y >> CONFIG_FRAME_POINTER=y >> >> 2) Let's make sure your module still has debuginfo present. First, run >> "file" on your module, making sure it says "not stripped". Then go a bit >> deeper and run "readelf --sections" on your module - there should be >> several sections that start with '.debug', including one called >> '.debug_frame'. > > I'm traying to trace my own stap module too. I'm using stap -B > CONFIG_DEBUG_INFO=y to put the dfebuginfo into the module. > However, this way I seem to get the '.debug_frame' section on > RHEL6, but not on RHEL7 or recent Fedora. Because of this I'm > failing to get reasonable backtraces. Any workaround? We actually test this in the systemtap.context/context.exp test case. It builds 2 external modules and tries to get backtraces from them (among other things). On RHEL7.2 ppc64, the backtrace tests don't pass, but I think that's really a test case problem. ==== backtrace from module("systemtap_test_module2").function("yyy_func2@/root/rhel7-2.ppc64/testsuite/artifacts/systemtap.context/context/staptestMS19Tt/systemtap_test_module2.c:33"): 0xd000000006fd00a0 : yyy_func2+0x0/0x30 [systemtap_test_module2] 0xd000000006fd00e0 : yyy_func1+0x10/0x0 [systemtap_test_module2] 0xd0000000082c01b8 : stm_write_cmd+0x1b8/0x200 [systemtap_test_module1] 0x0 (inexact) ==== The test case expects the '0x0 (inexact)' line to be from the kernel. This might be a ppc64-specific problem. But, the backtrace from the modules themselves look fine. Does your systemtap.context/context.exp test case output look similar? -- David Smith dsmith@redhat.com Red Hat http://www.redhat.com 256.217.0141 (direct) 256.837.0057 (fax) ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: How to track the functions in self-written module using SystemTap? 2016-10-07 17:59 ` David Smith @ 2016-10-12 17:02 ` Martin Cermak 0 siblings, 0 replies; 26+ messages in thread From: Martin Cermak @ 2016-10-12 17:02 UTC (permalink / raw) To: David Smith; +Cc: Nan Xiao, Frank Ch. Eigler, systemtap On Fri 2016-10-07 12:59 , David Smith wrote: > On 10/06/2016 05:05 PM, Martin Cermak wrote: > > On Mon 2015-12-07 13:53 , David Smith wrote: > >> 1) Make sure your kernel was compiled with frame pointer support (which > >> it should if it is a standard RHEL kernel): > >> > >> # fgrep FRAME_POINTER /boot/config-`uname -r` > >> CONFIG_SCHED_OMIT_FRAME_POINTER=y > >> CONFIG_ARCH_WANT_FRAME_POINTERS=y > >> CONFIG_FRAME_POINTER=y > >> > >> 2) Let's make sure your module still has debuginfo present. First, run > >> "file" on your module, making sure it says "not stripped". Then go a bit > >> deeper and run "readelf --sections" on your module - there should be > >> several sections that start with '.debug', including one called > >> '.debug_frame'. > > > > I'm traying to trace my own stap module too. I'm using stap -B > > CONFIG_DEBUG_INFO=y to put the dfebuginfo into the module. > > However, this way I seem to get the '.debug_frame' section on > > RHEL6, but not on RHEL7 or recent Fedora. Because of this I'm > > failing to get reasonable backtraces. Any workaround? > > We actually test this in the systemtap.context/context.exp test case. It > builds 2 external modules and tries to get backtraces from them (among > other things). On RHEL7.2 ppc64, the backtrace tests don't pass, but I > think that's really a test case problem. > > ==== > backtrace from > module("systemtap_test_module2").function("yyy_func2@/root/rhel7-2.ppc64/testsuite/artifacts/systemtap.context/context/staptestMS19Tt/systemtap_test_module2.c:33"): > 0xd000000006fd00a0 : yyy_func2+0x0/0x30 [systemtap_test_module2] > 0xd000000006fd00e0 : yyy_func1+0x10/0x0 [systemtap_test_module2] > 0xd0000000082c01b8 : stm_write_cmd+0x1b8/0x200 [systemtap_test_module1] > 0x0 (inexact) > ==== > > The test case expects the '0x0 (inexact)' line to be from the kernel. > This might be a ppc64-specific problem. But, the backtrace from the > modules themselves look fine. > > Does your systemtap.context/context.exp test case output look similar? > Yup, it does look similar. My scenario is slightly different, but ITSM that looking at context.exp helped me to narrow my problem down to the GCC invocation. My scenario is to trace a real stap module. OTOH, context.exp traces a module not created by stap. Here is my scenario: ======= $ # compile and run the tracee module $ $ stap -v -m blah -B CONFIG_DEBUG_INFO=y --poison-cache -e 'global x probe timer.s(1) {x<<<1 println(@count(x))}' $ $ # now run the tracer module (in another terminal): $ $ stap -d kernel -e 'probe module("/home/mcermak/blah.ko").function("_stp_stat_add") {print_backtrace() exit()}' WARNING: no or bad debug frame hdr WARNING: No binary search table for eh frame, doing slow linear search for /home/mcermak/blah.ko 0xffffffffc086d628 : probe_2815+0x58/0x4d0 [blah] $ ======= I'd expect richer backtrace above. Comparing the simple Makefile used by context.exp to a relatively complex Makefile of a real stap module made me think that by commenting out some of the EXTRA_CFLAGS defined in buildrun.cxx must fix my problem. And indeed, following update seems to help: ======= // Generate eh_frame for self-backtracing - o << "EXTRA_CFLAGS += -fasynchronous-unwind-tables" << endl; + //o << "EXTRA_CFLAGS += -fasynchronous-unwind-tables" << endl; ======= After applying it, and running whole the test again, I'm getting following output: ======= $ stap -d kernel -e 'probe module("/home/mcermak/blah.ko").function("_stp_stat_add") {print_backtrace() exit()}' 0xffffffffc086d628 : probe_2815+0x58/0x4d0 [blah] 0xffffffffc086c3c3 : _stp_hrtimer_notify_function+0x223/0x3f0 [blah] 0xffffffff81115d0e : __hrtimer_run_queues+0xee/0x270 [kernel] 0xffffffff811164aa : hrtimer_interrupt+0x9a/0x180 [kernel] 0xffffffff81053588 : local_apic_timer_interrupt+0x38/0x60 [kernel] 0xffffffff817e011d : smp_apic_timer_interrupt+0x3d/0x50 [kernel] 0xffffffff817de25c : apic_timer_interrupt+0x8c/0xa0 [kernel] $ ======= So, when the stap module gets compiled with gcc-6.2.1-1.fc26.x86_64 with -fasynchronous-unwind-tables, there is no '.debug_frame' section and no traceback. Removing this option helps me: ======= $ readelf --sections blah.ko | grep '.debug_frame' [37] .debug_frame PROGBITS 0000000000000000 00031620 [38] .rela.debug_frame RELA 0000000000000000 0009a5b0 $ ======= ... but probably breaks something else, since the relevant comment reads: "Generate eh_frame for self-backtracing". Martin ^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2016-10-12 17:02 UTC | newest] Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-11-17 7:14 How to track the functions in self-written module using SystemTap? Nan Xiao 2015-11-17 16:48 ` David Smith 2015-11-18 3:39 ` Nan Xiao 2015-11-18 19:38 ` David Smith 2015-11-19 0:49 ` Nan Xiao 2015-11-19 8:33 ` Nan Xiao 2015-11-19 14:58 ` David Smith 2015-11-26 6:10 ` Nan Xiao 2015-11-30 21:52 ` David Smith 2015-11-30 23:45 ` Nan Xiao 2015-12-01 2:18 ` Frank Ch. Eigler 2015-12-01 2:57 ` Nan Xiao 2015-12-01 16:27 ` David Smith 2015-12-02 2:22 ` Nan Xiao 2015-12-02 21:42 ` David Smith 2015-12-03 1:13 ` Nan Xiao 2015-12-03 17:04 ` David Smith 2015-12-04 9:56 ` Nan Xiao 2015-12-04 12:15 ` David Smith 2015-12-06 8:53 ` Nan Xiao 2015-12-07 19:53 ` David Smith 2015-12-08 1:05 ` Nan Xiao 2015-12-08 19:41 ` Frank Ch. Eigler 2016-10-06 22:05 ` Martin Cermak 2016-10-07 17:59 ` David Smith 2016-10-12 17:02 ` Martin Cermak
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).