How to track the functions in self-written module using SystemTap?

How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi all,

I build a simple Linux module
(https://github.com/troydhanson/kernel/blob/master/105.ops/kex.c),
and want to use SystemTap to track it.

But I find there is no available probes:
# stap -l 'module("kex").function("*")'
#

How can I track the functions in module written by myself? Is there
any tutorial about this? I try to
search on https://sourceware.org/systemtap/documentation.html, but
can't find wanted content.

Thanks!

Best Regards
Nan Xiao

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 11/17/2015 01:14 AM, Nan Xiao wrote:
> Hi all,
> 
> I build a simple Linux module
> (https://github.com/troydhanson/kernel/blob/master/105.ops/kex.c),
> and want to use SystemTap to track it.
> 
> But I find there is no available probes:
> # stap -l 'module("kex").function("*")'
> #
> 
> How can I track the functions in module written by myself? Is there
> any tutorial about this? I try to
> search on https://sourceware.org/systemtap/documentation.html, but
> can't find wanted content.

If your 'kex' module isn't the same directory as the other kernel
modules (/lib/modules/`uname -r`/), systemtap won't be able to find it.
So, you have to tell systemtap where the module is, like this:

# stap -l 'module("FULL_PATH_TO_KEX").function("*")'

If that doesn't work, please let me know.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

Yes, the "FULL_PATH_TO_KEX" works:

~/Downloads/kernel/105.ops # stap -l
'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")'
module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21")
module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45")
module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33")
module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72")
module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90")
module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61")

But the "RELATIVE_PATH_TO_KEX" doesn't work:
~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")'
~/Downloads/kernel/105.ops #

Is it the expected behavior of SystemTap? Thanks!
Best Regards
Nan Xiao


On Wed, Nov 18, 2015 at 12:48 AM, David Smith <dsmith@redhat.com> wrote:
> On 11/17/2015 01:14 AM, Nan Xiao wrote:
>> Hi all,
>>
>> I build a simple Linux module
>> (https://github.com/troydhanson/kernel/blob/master/105.ops/kex.c),
>> and want to use SystemTap to track it.
>>
>> But I find there is no available probes:
>> # stap -l 'module("kex").function("*")'
>> #
>>
>> How can I track the functions in module written by myself? Is there
>> any tutorial about this? I try to
>> search on https://sourceware.org/systemtap/documentation.html, but
>> can't find wanted content.
>
> If your 'kex' module isn't the same directory as the other kernel
> modules (/lib/modules/`uname -r`/), systemtap won't be able to find it.
> So, you have to tell systemtap where the module is, like this:
>
> # stap -l 'module("FULL_PATH_TO_KEX").function("*")'
>
> If that doesn't work, please let me know.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 11/17/2015 09:38 PM, Nan Xiao wrote:
> Hi David,
> 
> Yes, the "FULL_PATH_TO_KEX" works:
> 
> ~/Downloads/kernel/105.ops # stap -l
> 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")'
> module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21")
> module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45")
> module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33")
> module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72")
> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90")
> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61")
> 
> But the "RELATIVE_PATH_TO_KEX" doesn't work:
> ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")'
> ~/Downloads/kernel/105.ops #
> 
> Is it the expected behavior of SystemTap? Thanks!

Good question. You are correct, using relative paths doesn't seem to
work. I've filed PR19265 on this issue.

<https://sourceware.org/bugzilla/show_bug.cgi?id=19265>

Feel free to add yourself to the CC list on that bug. If you have the
time, also feel free to look into this more.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

Got it, thanks very much for your time and help!
Best Regards
Nan Xiao


On Thu, Nov 19, 2015 at 3:38 AM, David Smith <dsmith@redhat.com> wrote:
> On 11/17/2015 09:38 PM, Nan Xiao wrote:
>> Hi David,
>>
>> Yes, the "FULL_PATH_TO_KEX" works:
>>
>> ~/Downloads/kernel/105.ops # stap -l
>> 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")'
>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21")
>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45")
>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33")
>> module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72")
>> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90")
>> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61")
>>
>> But the "RELATIVE_PATH_TO_KEX" doesn't work:
>> ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")'
>> ~/Downloads/kernel/105.ops #
>>
>> Is it the expected behavior of SystemTap? Thanks!
>
> Good question. You are correct, using relative paths doesn't seem to
> work. I've filed PR19265 on this issue.
>
> <https://sourceware.org/bugzilla/show_bug.cgi?id=19265>
>
> Feel free to add yourself to the CC list on that bug. If you have the
> time, also feel free to look into this more.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

Another interesting thing: if there is a "-" in path, the SystemTap
also fail to parse:

# stap -v -l 'module("/root/Downloads/kernel-master/105.ops/kex.ko").function("*")'
Pass 1: parsed user script and 102 library script(s) using
78528virt/28740res/2700shr/26724data kb, in 170usr/20sys/183real ms.
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
global(s) using 79536virt/30480res/3164shr/27732data kb, in
0usr/50sys/53real ms.

Change kernel-master to kernel:

 # stap -v -l 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")'
Pass 1: parsed user script and 102 library script(s) using
78528virt/28732res/2700shr/26724data kb, in 180usr/10sys/189real ms.
module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel-master/105.ops/kex.c:21")
module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel-master/105.ops/kex.c:45")
module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel-master/105.ops/kex.c:33")
module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-6/arch/x86/include/asm/uaccess_64.h:72")
module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel-master/105.ops/kex.c:90")
module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel-master/105.ops/kex.c:61")
Pass 2: analyzed script: 6 probe(s), 0 function(s), 0 embed(s), 0
global(s) using 79548virt/30624res/3304shr/27744data kb, in
20usr/30sys/56real ms.

Thanks!
Best Regards
Nan Xiao


On Thu, Nov 19, 2015 at 8:49 AM, Nan Xiao <xiaonan830818@gmail.com> wrote:
> Hi David,
>
> Got it, thanks very much for your time and help!
> Best Regards
> Nan Xiao
>
>
> On Thu, Nov 19, 2015 at 3:38 AM, David Smith <dsmith@redhat.com> wrote:
>> On 11/17/2015 09:38 PM, Nan Xiao wrote:
>>> Hi David,
>>>
>>> Yes, the "FULL_PATH_TO_KEX" works:
>>>
>>> ~/Downloads/kernel/105.ops # stap -l
>>> 'module("/root/Downloads/kernel/105.ops/kex.ko").function("*")'
>>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_open@/root/Downloads/kernel/105.ops/kex.c:21")
>>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_read@/root/Downloads/kernel/105.ops/kex.c:45")
>>> module("/root/Downloads/kernel/105.ops/kex.ko").function("_release@/root/Downloads/kernel/105.ops/kex.c:33")
>>> module("/root/Downloads/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/linux-3.12.49-3/arch/x86/include/asm/uaccess_64.h:72")
>>> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_cleanup@/root/Downloads/kernel/105.ops/kex.c:90")
>>> module("/root/Downloads/kernel/105.ops/kex.ko").function("kex_init@/root/Downloads/kernel/105.ops/kex.c:61")
>>>
>>> But the "RELATIVE_PATH_TO_KEX" doesn't work:
>>> ~/Downloads/kernel/105.ops # stap -l 'module("./kex.ko").function("*")'
>>> ~/Downloads/kernel/105.ops #
>>>
>>> Is it the expected behavior of SystemTap? Thanks!
>>
>> Good question. You are correct, using relative paths doesn't seem to
>> work. I've filed PR19265 on this issue.
>>
>> <https://sourceware.org/bugzilla/show_bug.cgi?id=19265>
>>
>> Feel free to add yourself to the CC list on that bug. If you have the
>> time, also feel free to look into this more.
>>
>> --
>> David Smith
>> dsmith@redhat.com
>> Red Hat
>> http://www.redhat.com
>> 256.217.0141 (direct)
>> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 11/19/2015 02:33 AM, Nan Xiao wrote:
> Hi David,
> 
> Another interesting thing: if there is a "-" in path, the SystemTap
> also fail to parse:

That's odd. I've added that information to the bug.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

I do "insmod kex.ko" and "rmmod kex" repeatedly, but It seems the
related probes can't be triggered:

# stap -l 'module("/root/kernel/105.ops/kex.ko").function("*")'
module("/root/kernel/105.ops/kex.ko").function("_open@/root/kernel/105.ops/kex.c:21")
module("/root/kernel/105.ops/kex.ko").function("_read@/root/kernel/105.ops/kex.c:45")
module("/root/kernel/105.ops/kex.ko").function("_release@/root/kernel/105.ops/kex.c:33")
module("/root/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/kernels/3.10.0-123.el7.x86_64.debug/arch/x86/include/asm/uaccess_64.h:72")
module("/root/kernel/105.ops/kex.ko").function("kex_cleanup@/root/kernel/105.ops/kex.c:90")
module("/root/kernel/105.ops/kex.ko").function("kex_init@/root/kernel/105.ops/kex.c:61")
# stap -e 'probe
module("/root/kernel/105.ops/kex.ko").function("kex_init"),
module("/root/kernel/105.ops/kex.ko").function("kex_cleanup")
{print_backtrace()}'

Could you help to check it? Thanks in advance!
Best Regards
Nan Xiao


On Thu, Nov 19, 2015 at 10:57 PM, David Smith <dsmith@redhat.com> wrote:
> On 11/19/2015 02:33 AM, Nan Xiao wrote:
>> Hi David,
>>
>> Another interesting thing: if there is a "-" in path, the SystemTap
>> also fail to parse:
>
> That's odd. I've added that information to the bug.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 11/26/2015 12:10 AM, Nan Xiao wrote:
> Hi David,
> 
> I do "insmod kex.ko" and "rmmod kex" repeatedly, but It seems the
> related probes can't be triggered:
> 
> # stap -l 'module("/root/kernel/105.ops/kex.ko").function("*")'
> module("/root/kernel/105.ops/kex.ko").function("_open@/root/kernel/105.ops/kex.c:21")
> module("/root/kernel/105.ops/kex.ko").function("_read@/root/kernel/105.ops/kex.c:45")
> module("/root/kernel/105.ops/kex.ko").function("_release@/root/kernel/105.ops/kex.c:33")
> module("/root/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/kernels/3.10.0-123.el7.x86_64.debug/arch/x86/include/asm/uaccess_64.h:72")
> module("/root/kernel/105.ops/kex.ko").function("kex_cleanup@/root/kernel/105.ops/kex.c:90")
> module("/root/kernel/105.ops/kex.ko").function("kex_init@/root/kernel/105.ops/kex.c:61")
> # stap -e 'probe
> module("/root/kernel/105.ops/kex.ko").function("kex_init"),
> module("/root/kernel/105.ops/kex.ko").function("kex_cleanup")
> {print_backtrace()}'

Hmm. Note that I've done a good bit of work in the area of catching
module init functions recently. What version of systemtap are you
working with?

Let's see if systemtap can catch anything in your module. Try something
like:

# stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
printf("%s\n", ppfunc()) }'

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

My OS is RHEL 7.0, but run on VirtualBox:

# stap -V
Systemtap translator/driver (version 2.4/0.158, rpm 2.4-14.el7)
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR
TR1_UNORDERED_MAP NLS DYNINST JAVA LIBVIRT LIBXML2

run "# stap -e 'probe
module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
ppfunc()) }'", but
run "insmod kex.ko" & "rmmod kex" still outputs nothing.

Best Regards
Nan Xiao


On Tue, Dec 1, 2015 at 5:52 AM, David Smith <dsmith@redhat.com> wrote:
> On 11/26/2015 12:10 AM, Nan Xiao wrote:
>> Hi David,
>>
>> I do "insmod kex.ko" and "rmmod kex" repeatedly, but It seems the
>> related probes can't be triggered:
>>
>> # stap -l 'module("/root/kernel/105.ops/kex.ko").function("*")'
>> module("/root/kernel/105.ops/kex.ko").function("_open@/root/kernel/105.ops/kex.c:21")
>> module("/root/kernel/105.ops/kex.ko").function("_read@/root/kernel/105.ops/kex.c:45")
>> module("/root/kernel/105.ops/kex.ko").function("_release@/root/kernel/105.ops/kex.c:33")
>> module("/root/kernel/105.ops/kex.ko").function("copy_to_user@/usr/src/kernels/3.10.0-123.el7.x86_64.debug/arch/x86/include/asm/uaccess_64.h:72")
>> module("/root/kernel/105.ops/kex.ko").function("kex_cleanup@/root/kernel/105.ops/kex.c:90")
>> module("/root/kernel/105.ops/kex.ko").function("kex_init@/root/kernel/105.ops/kex.c:61")
>> # stap -e 'probe
>> module("/root/kernel/105.ops/kex.ko").function("kex_init"),
>> module("/root/kernel/105.ops/kex.ko").function("kex_cleanup")
>> {print_backtrace()}'
>
> Hmm. Note that I've done a good bit of work in the area of catching
> module init functions recently. What version of systemtap are you
> working with?
>
> Let's see if systemtap can catch anything in your module. Try something
> like:
>
> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
> printf("%s\n", ppfunc()) }'
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Frank Ch. Eigler]

Nan Xiao <xiaonan830818@gmail.com> writes:

> [...]
> run "# stap -e 'probe
> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
> ppfunc()) }'", but
> run "insmod kex.ko" & "rmmod kex" still outputs nothing.

Xan you exercise kex.ko beyond just installing & removing it right
away, while the systemtap script is probing it?

- FChE

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi Frank,

I have just tested it:

# dd if=kex of=/dev/null bs=101 count=1
0+1 records in
0+1 records out
100 bytes (100 B) copied, 0.00278675 s, 35.9 kB/s

But still nothing outputs:
# stap -v -e 'probe
module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
ppfunc()) }'
Pass 1: parsed user script and 100 library script(s) using
210544virt/28008res/3036shr/25484data kb, in 110usr/50sys/162real ms.
Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
global(s) using 370776virt/29748res/3596shr/26612data kb, in
30usr/760sys/802real ms.
Pass 3: using cached
/root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.c
Pass 4: using cached
/root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.ko
Pass 5: starting run.

Thanks!
Best Regards
Nan Xiao


On Tue, Dec 1, 2015 at 10:18 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Nan Xiao <xiaonan830818@gmail.com> writes:
>
>> [...]
>> run "# stap -e 'probe
>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>> ppfunc()) }'", but
>> run "insmod kex.ko" & "rmmod kex" still outputs nothing.
>
> Xan you exercise kex.ko beyond just installing & removing it right
> away, while the systemtap script is probing it?
>
> - FChE

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 11/30/2015 08:57 PM, Nan Xiao wrote:
> Hi Frank,
> 
> I have just tested it:
> 
> # dd if=kex of=/dev/null bs=101 count=1
> 0+1 records in
> 0+1 records out
> 100 bytes (100 B) copied, 0.00278675 s, 35.9 kB/s
> 
> But still nothing outputs:
> # stap -v -e 'probe
> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
> ppfunc()) }'
> Pass 1: parsed user script and 100 library script(s) using
> 210544virt/28008res/3036shr/25484data kb, in 110usr/50sys/162real ms.
> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
> global(s) using 370776virt/29748res/3596shr/26612data kb, in
> 30usr/760sys/802real ms.
> Pass 3: using cached
> /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.c
> Pass 4: using cached
> /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.ko
> Pass 5: starting run.
> 
> Thanks!
> Best Regards
> Nan Xiao
> 
> 
> On Tue, Dec 1, 2015 at 10:18 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
>> Nan Xiao <xiaonan830818@gmail.com> writes:
>>
>>> [...]
>>> run "# stap -e 'probe
>>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>>> ppfunc()) }'", but
>>> run "insmod kex.ko" & "rmmod kex" still outputs nothing.
>>
>> Xan you exercise kex.ko beyond just installing & removing it right
>> away, while the systemtap script is probing it?
>>
>> - FChE

Hmm. OK, let's try a couple more things:

1) It could be that systemtap is missing the module load somehow. So,
try this:

- load the module
- run stap
- exercise the module
- unload the module
- kill stap

2) It looks like you are running the rpm version. Can you install the
systemtap-testsuite rpm and try the following (as root):

# cd /usr/share/systemtap/testsuite
# make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"

That should run 2 testcases that test out-of-tree modules and in-tree
modules.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

> Hmm. OK, let's try a couple more things:

> 1) It could be that systemtap is missing the module load somehow. So,
> try this:

> - load the module
> - run stap
> - exercise the module
> - unload the module
> - kill stap

Still outputs nothing.

> 2) It looks like you are running the rpm version. Can you install the
> systemtap-testsuite rpm and try the following (as root):

> # cd /usr/share/systemtap/testsuite
> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"

> That should run 2 testcases that test out-of-tree modules and in-tree
> modules.

The output likes this:

# make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
Making a new site.exp file ...
rmmod uprobes 2>/dev/null
make: [installcheck] Error 1 (ignored)
make  check-DEJAGNU RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp
--tool_opts \'install \'"
make[1]: Entering directory `/usr/share/systemtap/testsuite'
srcdir='.'; export srcdir; \
EXPECT=expect; export EXPECT; \
if /bin/sh -c "env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C
SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS=
SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime
SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset
LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap
PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin
SYSTEMTAP_INCLUDES=/usr/include  PKGLIBDIR=/usr/libexec/systemtap
./execrc runtest --version" > /dev/null 2>&1; then \
  exit_status=0; l='systemtap'; for tool in $l; do \
    if env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C
SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS=
SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime
SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset
LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap
PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin
SYSTEMTAP_INCLUDES=/usr/include  PKGLIBDIR=/usr/libexec/systemtap
./execrc runtest  --tool $tool --tool_opts \'\' --srcdir $srcdir
modules_out_of_tree.exp kmodule.exp --tool_opts \'install \'; \
    then :; else exit_status=1; fi; \
  done; \
else echo "WARNING: could not find 'env XDG_DATA_DIRS=
SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS=
SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime
SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset
LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap
PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin
SYSTEMTAP_INCLUDES=/usr/include  PKGLIBDIR=/usr/libexec/systemtap
./execrc runtest'" 1>&2; :;\
fi; \
exit $exit_status
WARNING: Couldn't find the global config file.
kernel location: /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64.debug/vmlinux
kernel version: 3.10.0-123.el7.x86_64.debug
systemtap location: /usr/bin/stap
systemtap version: version 2.4/0.158, rpm 2.4-14.el7
gcc location: /usr/bin/gcc
gcc version: gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)
spawn -ignore SIGHUP gcc hello.c -g -m64 -lm -o hello-m64
spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32
/usr/bin/ld: skipping incompatible
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgcc_s.so when searching for
-lgcc_s
/usr/bin/ld: cannot find -lgcc_s
collect2: error: ld returned 1 exit status



**** failed gcc m32 smoke test:

/usr/bin/ld: cannot find -lgcc_s
collect2: error: ld returned 1 exit status

Please install libgcc and glibc development packages for m32


**** aborting testing.

make[1]: Leaving directory `/usr/share/systemtap/testsuite'
if test -n ""; then mail  < systemtap.sum; fi


Thanks!
Best Regards
Nan Xiao


On Wed, Dec 2, 2015 at 12:27 AM, David Smith <dsmith@redhat.com> wrote:
> On 11/30/2015 08:57 PM, Nan Xiao wrote:
>> Hi Frank,
>>
>> I have just tested it:
>>
>> # dd if=kex of=/dev/null bs=101 count=1
>> 0+1 records in
>> 0+1 records out
>> 100 bytes (100 B) copied, 0.00278675 s, 35.9 kB/s
>>
>> But still nothing outputs:
>> # stap -v -e 'probe
>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>> ppfunc()) }'
>> Pass 1: parsed user script and 100 library script(s) using
>> 210544virt/28008res/3036shr/25484data kb, in 110usr/50sys/162real ms.
>> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
>> global(s) using 370776virt/29748res/3596shr/26612data kb, in
>> 30usr/760sys/802real ms.
>> Pass 3: using cached
>> /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.c
>> Pass 4: using cached
>> /root/.systemtap/cache/5c/stap_5cfec0f637bce0fa61c51d4186192dd5_2853.ko
>> Pass 5: starting run.
>>
>> Thanks!
>> Best Regards
>> Nan Xiao
>>
>>
>> On Tue, Dec 1, 2015 at 10:18 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
>>> Nan Xiao <xiaonan830818@gmail.com> writes:
>>>
>>>> [...]
>>>> run "# stap -e 'probe
>>>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>>>> ppfunc()) }'", but
>>>> run "insmod kex.ko" & "rmmod kex" still outputs nothing.
>>>
>>> Xan you exercise kex.ko beyond just installing & removing it right
>>> away, while the systemtap script is probing it?
>>>
>>> - FChE
>
> Hmm. OK, let's try a couple more things:
>
> 1) It could be that systemtap is missing the module load somehow. So,
> try this:
>
> - load the module
> - run stap
> - exercise the module
> - unload the module
> - kill stap
>
> 2) It looks like you are running the rpm version. Can you install the
> systemtap-testsuite rpm and try the following (as root):
>
> # cd /usr/share/systemtap/testsuite
> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
>
> That should run 2 testcases that test out-of-tree modules and in-tree
> modules.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 12/01/2015 08:22 PM, Nan Xiao wrote:
>> Hmm. OK, let's try a couple more things:
> 
>> 1) It could be that systemtap is missing the module load somehow. So,
>> try this:
> 
>> - load the module
>> - run stap
>> - exercise the module
>> - unload the module
>> - kill stap
> 
> Still outputs nothing.

Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap
command and show us the output?

>> 2) It looks like you are running the rpm version. Can you install the
>> systemtap-testsuite rpm and try the following (as root):
> 
>> # cd /usr/share/systemtap/testsuite
>> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
> 
>> That should run 2 testcases that test out-of-tree modules and in-tree
>> modules.
> 
> The output likes this:
> 
> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"

... stuff deleted ...

> spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32
> /usr/bin/ld: skipping incompatible
> /usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgcc_s.so when searching for
> -lgcc_s
> /usr/bin/ld: cannot find -lgcc_s
> collect2: error: ld returned 1 exit status
> 
> 
> 
> **** failed gcc m32 smoke test:
> 
> /usr/bin/ld: cannot find -lgcc_s
> collect2: error: ld returned 1 exit status
> 
> Please install libgcc and glibc development packages for m32

The testsuite checks a few things before starting, one of them being
that gcc can create 32-bit executables on an x86_64 system. If you want
to proceed further here, you'll have to do the following:

# yum install glibc.i686 libgcc.i686 glibc-devel.i686 libstdc++-devel.i686

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

(1)
>>> Hmm. OK, let's try a couple more things:
>>
>>> 1) It could be that systemtap is missing the module load somehow. So,
>>> try this:
>>
>>> - load the module
>>> - run stap
>>> - exercise the module
>>> - unload the module
>>> - kill stap
>>
>> Still outputs nothing.

> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap
> command and show us the output?

After adding '-DDEBUG_KPROBES', the stap command still outputs nothing.

(2)
The output of executing "# make installcheck
RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"":

# make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
make: *** No rule to make target `installcheck'.  Stop.
[root@localhost yum.repos.d]# cd /usr/share/systemtap/testsuite
[root@localhost testsuite]# make installcheck
RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
rmmod uprobes 2>/dev/null
make: [installcheck] Error 1 (ignored)
make  check-DEJAGNU RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp
--tool_opts \'install \'"
make[1]: Entering directory `/usr/share/systemtap/testsuite'
srcdir='.'; export srcdir; \
EXPECT=expect; export EXPECT; \
if /bin/sh -c "env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C
SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS=
SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime
SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset
LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap
PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin
SYSTEMTAP_INCLUDES=/usr/include  PKGLIBDIR=/usr/libexec/systemtap
./execrc runtest --version" > /dev/null 2>&1; then \
  exit_status=0; l='systemtap'; for tool in $l; do \
    if env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C
SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS=
SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime
SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset
LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap
PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin
SYSTEMTAP_INCLUDES=/usr/include  PKGLIBDIR=/usr/libexec/systemtap
./execrc runtest  --tool $tool --tool_opts \'\' --srcdir $srcdir
modules_out_of_tree.exp kmodule.exp --tool_opts \'install \'; \
    then :; else exit_status=1; fi; \
  done; \
else echo "WARNING: could not find 'env XDG_DATA_DIRS=
SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS=
SYSTEMTAP_RUNTIME=/usr/share/systemtap/runtime
SYSTEMTAP_TAPSET=/usr/share/systemtap/tapset
LD_LIBRARY_PATH=/usr/lib64/systemtap CRASH_LIBDIR=/usr/lib64/systemtap
PATH=/usr/bin:$PATH SYSTEMTAP_PATH=/usr/bin
SYSTEMTAP_INCLUDES=/usr/include  PKGLIBDIR=/usr/libexec/systemtap
./execrc runtest'" 1>&2; :;\
fi; \
exit $exit_status
WARNING: Couldn't find the global config file.
kernel location: /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64.debug/vmlinux
kernel version: 3.10.0-123.el7.x86_64.debug
systemtap location: /usr/bin/stap
systemtap version: version 2.4/0.158, rpm 2.4-14.el7
gcc location: /usr/bin/gcc
gcc version: gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)
spawn -ignore SIGHUP gcc hello.c -g -m64 -lm -o hello-m64
spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32
spawn -ignore SIGHUP g++ hello.cxx -g -m64 -lm -o hello-m64
spawn -ignore SIGHUP g++ hello.cxx -g -m32 -lm -o hello-m32
Test Run By root on Wed Dec  2 20:08:12 2015
Native configuration is x86_64-unknown-linux-gnu

                === systemtap tests ===

Schedule of variations:
    unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file
for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using ./config/unix.exp as tool-and-target-specific interface file.

Host: Linux localhost.localdomain 3.10.0-123.el7.x86_64.debug #1 SMP
Mon May 5 11:24:18 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
Snapshot: version 2.4/0.158, rpm 2.4-14.el7
GCC: 4.8.2 [gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)]
Distro: Red Hat Enterprise Linux Server release 7.0 (Maipo)
SElinux: Enforcing


Running ./systemtap.base/kmodule.exp ...

                === systemtap Summary ===

# of expected passes            7
make[1]: Leaving directory `/usr/share/systemtap/testsuite'
if test -n ""; then mail  < systemtap.sum; fi

Thanks!
Best Regards
Nan Xiao


On Thu, Dec 3, 2015 at 5:42 AM, David Smith <dsmith@redhat.com> wrote:
> On 12/01/2015 08:22 PM, Nan Xiao wrote:
>>> Hmm. OK, let's try a couple more things:
>>
>>> 1) It could be that systemtap is missing the module load somehow. So,
>>> try this:
>>
>>> - load the module
>>> - run stap
>>> - exercise the module
>>> - unload the module
>>> - kill stap
>>
>> Still outputs nothing.
>
> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap
> command and show us the output?
>
>>> 2) It looks like you are running the rpm version. Can you install the
>>> systemtap-testsuite rpm and try the following (as root):
>>
>>> # cd /usr/share/systemtap/testsuite
>>> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
>>
>>> That should run 2 testcases that test out-of-tree modules and in-tree
>>> modules.
>>
>> The output likes this:
>>
>> # make installcheck RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
>
> ... stuff deleted ...
>
>> spawn -ignore SIGHUP gcc hello.c -g -m32 -lm -o hello-m32
>> /usr/bin/ld: skipping incompatible
>> /usr/lib/gcc/x86_64-redhat-linux/4.8.2/libgcc_s.so when searching for
>> -lgcc_s
>> /usr/bin/ld: cannot find -lgcc_s
>> collect2: error: ld returned 1 exit status
>>
>>
>>
>> **** failed gcc m32 smoke test:
>>
>> /usr/bin/ld: cannot find -lgcc_s
>> collect2: error: ld returned 1 exit status
>>
>> Please install libgcc and glibc development packages for m32
>
> The testsuite checks a few things before starting, one of them being
> that gcc can create 32-bit executables on an x86_64 system. If you want
> to proceed further here, you'll have to do the following:
>
> # yum install glibc.i686 libgcc.i686 glibc-devel.i686 libstdc++-devel.i686
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 12/02/2015 07:13 PM, Nan Xiao wrote:
> Hi David,
> 
> (1)
>>>> Hmm. OK, let's try a couple more things:
>>>
>>>> 1) It could be that systemtap is missing the module load somehow. So,
>>>> try this:
>>>
>>>> - load the module
>>>> - run stap
>>>> - exercise the module
>>>> - unload the module
>>>> - kill stap
>>>
>>> Still outputs nothing.
> 
>> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap
>> command and show us the output?
> 
> After adding '-DDEBUG_KPROBES', the stap command still outputs nothing.

Hmm.

> (2)
> [root@localhost testsuite]# make installcheck
> RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
> 
> 
> Running ./systemtap.base/kmodule.exp ...
> 
>                 === systemtap Summary ===
> 
> # of expected passes            7

Ah. It ran kmodule.exp, but not modules_out_of_tree.exp. You are running
systemtap 2.4, and that testcase was first present in systemtap 2.6.

I was testing another bug so I had a RHEL 7.0 systemtap set up and I
installed systemtap 2.4 on it. It looks like that version can't probe
out-of-tree modules.

It looks like you've got a couple of options:

1) Copy your module into the kernel module tree (located at
/lib/modules/`uname -r`/kernel) to use systemtap on it.

2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
system, and the modules_out_of_tree.exp test case passed there.)

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi Dave,

> It looks like you've got a couple of options:

> 1) Copy your module into the kernel module tree (located at
> /lib/modules/`uname -r`/kernel) to use systemtap on it.

I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
executes error:

# cd /lib/modules/`uname -r`/kernel
# pwd
/lib/modules/3.10.0-123.el7.x86_64.debug/kernel
# ls
arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound

# stap -v -e 'probe
module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
{ printf("%s\n", ppfunc()) }'
Pass 1: parsed user script and 103 library script(s) using
214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
semantic error: while resolving probe point: identifier 'module' at <input>:1:7
        source: probe
module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
{ printf("%s\n", ppfunc()) }
                      ^

semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
global(s) using 372028virt/33024res/3412shr/30124data kb, in
20usr/770sys/804real ms.
Pass 2: analysis failed.  [man error::pass2]

# stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
ppfunc()) }'                                    Pass 1: parsed user
script and 103 library script(s) using
214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
semantic error: while resolving probe point: identifier 'module' at <input>:1:7
        source: probe module("kex.ko").function("*") { printf("%s\n",
ppfunc()) }
                      ^

semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
global(s) using 372032virt/33016res/3412shr/30128data kb, in
40usr/880sys/929real ms.
Pass 2: analysis failed.  [man error::pass2]

> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
> system, and the modules_out_of_tree.exp test case passed there.)

I install the RHEL 7.2, and it really works! But it can't print actual
stack traces:

# stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
print_backtrace() }'
WARNING: no or bad debug frame hdr
WARNING: No binary search table for debug frame, doing slow linear
search for /root/kernel/105.ops/kex.ko
 0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
 0xffffffff810020e8 (inexact)
 0xffffffff810ed4ae (inexact)
 0xffffffff81316880 (inexact)
 0xffffffff810e9743 (inexact)
 0xffffffff810ede66 (inexact)
 0xffffffff81645909 (inexact)
 0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
 0xffffffff810eb23b (inexact)
 0xffffffff81641113 (inexact)
 0xffffffff81014b12 (inexact)
 0xffffffff81645909 (inexact)

So it means the module isn't built with debuginfo, right?

Thanks very much for your time & effort!
Best Regards
Nan Xiao


On Fri, Dec 4, 2015 at 1:03 AM, David Smith <dsmith@redhat.com> wrote:
> On 12/02/2015 07:13 PM, Nan Xiao wrote:
>> Hi David,
>>
>> (1)
>>>>> Hmm. OK, let's try a couple more things:
>>>>
>>>>> 1) It could be that systemtap is missing the module load somehow. So,
>>>>> try this:
>>>>
>>>>> - load the module
>>>>> - run stap
>>>>> - exercise the module
>>>>> - unload the module
>>>>> - kill stap
>>>>
>>>> Still outputs nothing.
>>
>>> Can you do the same thing here and add '-DDEBUG_KPROBES' to the stap
>>> command and show us the output?
>>
>> After adding '-DDEBUG_KPROBES', the stap command still outputs nothing.
>
> Hmm.
>
>> (2)
>> [root@localhost testsuite]# make installcheck
>> RUNTESTFLAGS="modules_out_of_tree.exp kmodule.exp"
>>
>>
>> Running ./systemtap.base/kmodule.exp ...
>>
>>                 === systemtap Summary ===
>>
>> # of expected passes            7
>
> Ah. It ran kmodule.exp, but not modules_out_of_tree.exp. You are running
> systemtap 2.4, and that testcase was first present in systemtap 2.6.
>
> I was testing another bug so I had a RHEL 7.0 systemtap set up and I
> installed systemtap 2.4 on it. It looks like that version can't probe
> out-of-tree modules.
>
> It looks like you've got a couple of options:
>
> 1) Copy your module into the kernel module tree (located at
> /lib/modules/`uname -r`/kernel) to use systemtap on it.
>
> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
> system, and the modules_out_of_tree.exp test case passed there.)
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 12/04/2015 03:56 AM, Nan Xiao wrote:
> Hi Dave,
> 
>> It looks like you've got a couple of options:
> 
>> 1) Copy your module into the kernel module tree (located at
>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
> 
> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
> executes error:
> 
> # cd /lib/modules/`uname -r`/kernel
> # pwd
> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
> # ls
> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
> 
> # stap -v -e 'probe
> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
> { printf("%s\n", ppfunc()) }'
> Pass 1: parsed user script and 103 library script(s) using
> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>         source: probe
> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
> { printf("%s\n", ppfunc()) }
>                       ^
> 
> semantic error: no match
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
> global(s) using 372028virt/33024res/3412shr/30124data kb, in
> 20usr/770sys/804real ms.
> Pass 2: analysis failed.  [man error::pass2]
> 
> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
> ppfunc()) }'                                    Pass 1: parsed user
> script and 103 library script(s) using
> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>         source: probe module("kex.ko").function("*") { printf("%s\n",
> ppfunc()) }
>                       ^
> 
> semantic error: no match
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
> global(s) using 372032virt/33016res/3412shr/30128data kb, in
> 40usr/880sys/929real ms.
> Pass 2: analysis failed.  [man error::pass2]

I see I didn't explain well here. Once you copied your module to the
kernel's module directory, systemtap no longer would need a full path to
the module name - just "kex" should have worked.

>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>> system, and the modules_out_of_tree.exp test case passed there.)
> 
> I install the RHEL 7.2, and it really works! But it can't print actual
> stack traces:
> 
> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
> print_backtrace() }'
> WARNING: no or bad debug frame hdr
> WARNING: No binary search table for debug frame, doing slow linear
> search for /root/kernel/105.ops/kex.ko
>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>  0xffffffff810020e8 (inexact)
>  0xffffffff810ed4ae (inexact)
>  0xffffffff81316880 (inexact)
>  0xffffffff810e9743 (inexact)
>  0xffffffff810ede66 (inexact)
>  0xffffffff81645909 (inexact)
>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>  0xffffffff810eb23b (inexact)
>  0xffffffff81641113 (inexact)
>  0xffffffff81014b12 (inexact)
>  0xffffffff81645909 (inexact)
> 
> So it means the module isn't built with debuginfo, right?

Hmm, it could mean that we just need to make sure that the debuginfo is
loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
stap command line. (The '-d' option asks stap to load the symbol/unwind
information for a module even if stap doesn't think it needs it.)

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

> (1)
> I see I didn't explain well here. Once you copied your module to the
> kernel's module directory, systemtap no longer would need a full path to
> the module name - just "kex" should have worked.

Yeah, it worked!

> (2)
> Hmm, it could mean that we just need to make sure that the debuginfo is
> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
> stap command line. (The '-d' option asks stap to load the symbol/unwind
> information for a module even if stap doesn't think it needs it.)

The output stills have error:

# stap -d /root/kernel/105.ops/kex.ko -v -e 'probe
module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
print_backtrace()) }'
Pass 1: parsed user script and 110 library script(s) using
220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms.
Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
global(s) using 221348virt/39612res/3832shr/36156data kb, in
20usr/60sys/114real ms.
Pass 3: translated to C into
"/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c"
using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real
ms.
Pass 4: compiled C into
"stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in
2280usr/540sys/5099real ms.
Pass 5: starting run.
WARNING: no or bad debug frame hdr
WARNING: No binary search table for debug frame, doing slow linear
search for /root/kernel/105.ops/kex.ko
 0xffffffffa0037000 : kex_init+0x0/0x0 [kex]
 0xffffffff810020e8 (inexact)
 0xffffffff810ed4ae (inexact)
 0xffffffff81316880 (inexact)
 0xffffffff810e9743 (inexact)
 0xffffffff810ede66 (inexact)
 0xffffffff81645909 (inexact)

 0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex]
 0xffffffff810eb23b (inexact)
 0xffffffff81641113 (inexact)
 0xffffffff81014b12 (inexact)
 0xffffffff81645909 (inexact)

Thanks!
Best Regards
Nan Xiao


On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote:
> On 12/04/2015 03:56 AM, Nan Xiao wrote:
>> Hi Dave,
>>
>>> It looks like you've got a couple of options:
>>
>>> 1) Copy your module into the kernel module tree (located at
>>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
>>
>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
>> executes error:
>>
>> # cd /lib/modules/`uname -r`/kernel
>> # pwd
>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
>> # ls
>> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
>>
>> # stap -v -e 'probe
>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>> { printf("%s\n", ppfunc()) }'
>> Pass 1: parsed user script and 103 library script(s) using
>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>         source: probe
>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>> { printf("%s\n", ppfunc()) }
>>                       ^
>>
>> semantic error: no match
>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>> global(s) using 372028virt/33024res/3412shr/30124data kb, in
>> 20usr/770sys/804real ms.
>> Pass 2: analysis failed.  [man error::pass2]
>>
>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
>> ppfunc()) }'                                    Pass 1: parsed user
>> script and 103 library script(s) using
>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>         source: probe module("kex.ko").function("*") { printf("%s\n",
>> ppfunc()) }
>>                       ^
>>
>> semantic error: no match
>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>> global(s) using 372032virt/33016res/3412shr/30128data kb, in
>> 40usr/880sys/929real ms.
>> Pass 2: analysis failed.  [man error::pass2]
>
> I see I didn't explain well here. Once you copied your module to the
> kernel's module directory, systemtap no longer would need a full path to
> the module name - just "kex" should have worked.
>
>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>>> system, and the modules_out_of_tree.exp test case passed there.)
>>
>> I install the RHEL 7.2, and it really works! But it can't print actual
>> stack traces:
>>
>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
>> print_backtrace() }'
>> WARNING: no or bad debug frame hdr
>> WARNING: No binary search table for debug frame, doing slow linear
>> search for /root/kernel/105.ops/kex.ko
>>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>>  0xffffffff810020e8 (inexact)
>>  0xffffffff810ed4ae (inexact)
>>  0xffffffff81316880 (inexact)
>>  0xffffffff810e9743 (inexact)
>>  0xffffffff810ede66 (inexact)
>>  0xffffffff81645909 (inexact)
>>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>>  0xffffffff810eb23b (inexact)
>>  0xffffffff81641113 (inexact)
>>  0xffffffff81014b12 (inexact)
>>  0xffffffff81645909 (inexact)
>>
>> So it means the module isn't built with debuginfo, right?
>
> Hmm, it could mean that we just need to make sure that the debuginfo is
> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
> stap command line. (The '-d' option asks stap to load the symbol/unwind
> information for a module even if stap doesn't think it needs it.)
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 12/06/2015 02:53 AM, Nan Xiao wrote:
> Hi David,
> 
>> (1)
>> I see I didn't explain well here. Once you copied your module to the
>> kernel's module directory, systemtap no longer would need a full path to
>> the module name - just "kex" should have worked.
> 
> Yeah, it worked!
> 
>> (2)
>> Hmm, it could mean that we just need to make sure that the debuginfo is
>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>> information for a module even if stap doesn't think it needs it.)
> 
> The output stills have error:
> 
> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe
> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
> print_backtrace()) }'
> Pass 1: parsed user script and 110 library script(s) using
> 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms.
> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
> global(s) using 221348virt/39612res/3832shr/36156data kb, in
> 20usr/60sys/114real ms.
> Pass 3: translated to C into
> "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c"
> using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real
> ms.
> Pass 4: compiled C into
> "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in
> 2280usr/540sys/5099real ms.
> Pass 5: starting run.
> WARNING: no or bad debug frame hdr
> WARNING: No binary search table for debug frame, doing slow linear
> search for /root/kernel/105.ops/kex.ko
>  0xffffffffa0037000 : kex_init+0x0/0x0 [kex]
>  0xffffffff810020e8 (inexact)
>  0xffffffff810ed4ae (inexact)
>  0xffffffff81316880 (inexact)
>  0xffffffff810e9743 (inexact)
>  0xffffffff810ede66 (inexact)
>  0xffffffff81645909 (inexact)
> 
>  0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex]
>  0xffffffff810eb23b (inexact)
>  0xffffffff81641113 (inexact)
>  0xffffffff81014b12 (inexact)
>  0xffffffff81645909 (inexact)

Hmm. OK, let's make sure of a couple of things:

1) Make sure your kernel was compiled with frame pointer support (which
it should if it is a standard RHEL kernel):

# fgrep FRAME_POINTER /boot/config-`uname -r`
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y

2) Let's make sure your module still has debuginfo present. First, run
"file" on your module, making sure it says "not stripped". Then go a bit
deeper and run "readelf --sections" on your module - there should be
several sections that start with '.debug', including one called
'.debug_frame'.

> 
> Thanks!
> Best Regards
> Nan Xiao
> 
> 
> On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote:
>> On 12/04/2015 03:56 AM, Nan Xiao wrote:
>>> Hi Dave,
>>>
>>>> It looks like you've got a couple of options:
>>>
>>>> 1) Copy your module into the kernel module tree (located at
>>>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
>>>
>>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
>>> executes error:
>>>
>>> # cd /lib/modules/`uname -r`/kernel
>>> # pwd
>>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
>>> # ls
>>> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
>>>
>>> # stap -v -e 'probe
>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>> { printf("%s\n", ppfunc()) }'
>>> Pass 1: parsed user script and 103 library script(s) using
>>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>         source: probe
>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>> { printf("%s\n", ppfunc()) }
>>>                       ^
>>>
>>> semantic error: no match
>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>> global(s) using 372028virt/33024res/3412shr/30124data kb, in
>>> 20usr/770sys/804real ms.
>>> Pass 2: analysis failed.  [man error::pass2]
>>>
>>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
>>> ppfunc()) }'                                    Pass 1: parsed user
>>> script and 103 library script(s) using
>>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>         source: probe module("kex.ko").function("*") { printf("%s\n",
>>> ppfunc()) }
>>>                       ^
>>>
>>> semantic error: no match
>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>> global(s) using 372032virt/33016res/3412shr/30128data kb, in
>>> 40usr/880sys/929real ms.
>>> Pass 2: analysis failed.  [man error::pass2]
>>
>> I see I didn't explain well here. Once you copied your module to the
>> kernel's module directory, systemtap no longer would need a full path to
>> the module name - just "kex" should have worked.
>>
>>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>>>> system, and the modules_out_of_tree.exp test case passed there.)
>>>
>>> I install the RHEL 7.2, and it really works! But it can't print actual
>>> stack traces:
>>>
>>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
>>> print_backtrace() }'
>>> WARNING: no or bad debug frame hdr
>>> WARNING: No binary search table for debug frame, doing slow linear
>>> search for /root/kernel/105.ops/kex.ko
>>>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>>>  0xffffffff810020e8 (inexact)
>>>  0xffffffff810ed4ae (inexact)
>>>  0xffffffff81316880 (inexact)
>>>  0xffffffff810e9743 (inexact)
>>>  0xffffffff810ede66 (inexact)
>>>  0xffffffff81645909 (inexact)
>>>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>>>  0xffffffff810eb23b (inexact)
>>>  0xffffffff81641113 (inexact)
>>>  0xffffffff81014b12 (inexact)
>>>  0xffffffff81645909 (inexact)
>>>
>>> So it means the module isn't built with debuginfo, right?
>>
>> Hmm, it could mean that we just need to make sure that the debuginfo is
>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>> information for a module even if stap doesn't think it needs it.)
>>
>> --
>> David Smith
>> dsmith@redhat.com
>> Red Hat
>> http://www.redhat.com
>> 256.217.0141 (direct)
>> 256.837.0057 (fax)


-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao]

Hi David,

> 1) Make sure your kernel was compiled with frame pointer support (which
> it should if it is a standard RHEL kernel):

> # fgrep FRAME_POINTER /boot/config-`uname -r`
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y

Yes, all the CONFIG options are "yes".

> 2) Let's make sure your module still has debuginfo present. First, run
> "file" on your module, making sure it says "not stripped".

Yes, it is "not stripped".

> Then go a bit deeper and run "readelf --sections" on your module - there should be
> several sections that start with '.debug', including one called
> '.debug_frame'.

Below are all the sections with `.debug`:

......
[19] .debug_info       PROGBITS         0000000000000000  000008e0
       000000000000b6d9  0000000000000000           0     0     1
  [20] .rela.debug_info  RELA             0000000000000000  00014fe0
       000000000000ed00  0000000000000018          36    19     8
  [21] .debug_abbrev     PROGBITS         0000000000000000  0000bfb9
       00000000000007cc  0000000000000000           0     0     1
  [22] .debug_loc        PROGBITS         0000000000000000  0000c785
       0000000000000324  0000000000000000           0     0     1
  [23] .rela.debug_loc   RELA             0000000000000000  00023ce0
       0000000000000570  0000000000000018          36    22     8
  [24] .debug_aranges    PROGBITS         0000000000000000  0000caa9
       0000000000000070  0000000000000000           0     0     1
  [25] .rela.debug_arang RELA             0000000000000000  00024250
       0000000000000078  0000000000000018          36    24     8
  [26] .debug_ranges     PROGBITS         0000000000000000  0000cb19
       0000000000000040  0000000000000000           0     0     1
  [27] .rela.debug_range RELA             0000000000000000  000242c8
       0000000000000090  0000000000000018          36    26     8
  [28] .debug_line       PROGBITS         0000000000000000  0000cb59
       0000000000000a8e  0000000000000000           0     0     1
  [29] .rela.debug_line  RELA             0000000000000000  00024358
       0000000000000048  0000000000000018          36    28     8
  [30] .debug_str        PROGBITS         0000000000000000  0000d5e7
       00000000000069fc  0000000000000001  MS       0     0     1
  [31] .comment          PROGBITS         0000000000000000  00013fe3
       000000000000005a  0000000000000001  MS       0     0     1
  [32] .note.GNU-stack   PROGBITS         0000000000000000  0001403d
       0000000000000000  0000000000000000           0     0     1
  [33] .debug_frame      PROGBITS         0000000000000000  00014040
       00000000000000f0  0000000000000000           0     0     8
  [34] .rela.debug_frame RELA             0000000000000000  000243a0
       00000000000000f0  0000000000000018          36    33     8
......

Thanks!
Best Regards
Nan Xiao


On Tue, Dec 8, 2015 at 3:53 AM, David Smith <dsmith@redhat.com> wrote:
> On 12/06/2015 02:53 AM, Nan Xiao wrote:
>> Hi David,
>>
>>> (1)
>>> I see I didn't explain well here. Once you copied your module to the
>>> kernel's module directory, systemtap no longer would need a full path to
>>> the module name - just "kex" should have worked.
>>
>> Yeah, it worked!
>>
>>> (2)
>>> Hmm, it could mean that we just need to make sure that the debuginfo is
>>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>>> information for a module even if stap doesn't think it needs it.)
>>
>> The output stills have error:
>>
>> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe
>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>> print_backtrace()) }'
>> Pass 1: parsed user script and 110 library script(s) using
>> 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms.
>> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
>> global(s) using 221348virt/39612res/3832shr/36156data kb, in
>> 20usr/60sys/114real ms.
>> Pass 3: translated to C into
>> "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c"
>> using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real
>> ms.
>> Pass 4: compiled C into
>> "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in
>> 2280usr/540sys/5099real ms.
>> Pass 5: starting run.
>> WARNING: no or bad debug frame hdr
>> WARNING: No binary search table for debug frame, doing slow linear
>> search for /root/kernel/105.ops/kex.ko
>>  0xffffffffa0037000 : kex_init+0x0/0x0 [kex]
>>  0xffffffff810020e8 (inexact)
>>  0xffffffff810ed4ae (inexact)
>>  0xffffffff81316880 (inexact)
>>  0xffffffff810e9743 (inexact)
>>  0xffffffff810ede66 (inexact)
>>  0xffffffff81645909 (inexact)
>>
>>  0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex]
>>  0xffffffff810eb23b (inexact)
>>  0xffffffff81641113 (inexact)
>>  0xffffffff81014b12 (inexact)
>>  0xffffffff81645909 (inexact)
>
> Hmm. OK, let's make sure of a couple of things:
>
> 1) Make sure your kernel was compiled with frame pointer support (which
> it should if it is a standard RHEL kernel):
>
> # fgrep FRAME_POINTER /boot/config-`uname -r`
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
>
> 2) Let's make sure your module still has debuginfo present. First, run
> "file" on your module, making sure it says "not stripped". Then go a bit
> deeper and run "readelf --sections" on your module - there should be
> several sections that start with '.debug', including one called
> '.debug_frame'.
>
>>
>> Thanks!
>> Best Regards
>> Nan Xiao
>>
>>
>> On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote:
>>> On 12/04/2015 03:56 AM, Nan Xiao wrote:
>>>> Hi Dave,
>>>>
>>>>> It looks like you've got a couple of options:
>>>>
>>>>> 1) Copy your module into the kernel module tree (located at
>>>>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
>>>>
>>>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
>>>> executes error:
>>>>
>>>> # cd /lib/modules/`uname -r`/kernel
>>>> # pwd
>>>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
>>>> # ls
>>>> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
>>>>
>>>> # stap -v -e 'probe
>>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>>> { printf("%s\n", ppfunc()) }'
>>>> Pass 1: parsed user script and 103 library script(s) using
>>>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
>>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>>         source: probe
>>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>>> { printf("%s\n", ppfunc()) }
>>>>                       ^
>>>>
>>>> semantic error: no match
>>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>>> global(s) using 372028virt/33024res/3412shr/30124data kb, in
>>>> 20usr/770sys/804real ms.
>>>> Pass 2: analysis failed.  [man error::pass2]
>>>>
>>>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
>>>> ppfunc()) }'                                    Pass 1: parsed user
>>>> script and 103 library script(s) using
>>>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
>>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>>         source: probe module("kex.ko").function("*") { printf("%s\n",
>>>> ppfunc()) }
>>>>                       ^
>>>>
>>>> semantic error: no match
>>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>>> global(s) using 372032virt/33016res/3412shr/30128data kb, in
>>>> 40usr/880sys/929real ms.
>>>> Pass 2: analysis failed.  [man error::pass2]
>>>
>>> I see I didn't explain well here. Once you copied your module to the
>>> kernel's module directory, systemtap no longer would need a full path to
>>> the module name - just "kex" should have worked.
>>>
>>>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>>>>> system, and the modules_out_of_tree.exp test case passed there.)
>>>>
>>>> I install the RHEL 7.2, and it really works! But it can't print actual
>>>> stack traces:
>>>>
>>>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
>>>> print_backtrace() }'
>>>> WARNING: no or bad debug frame hdr
>>>> WARNING: No binary search table for debug frame, doing slow linear
>>>> search for /root/kernel/105.ops/kex.ko
>>>>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>>>>  0xffffffff810020e8 (inexact)
>>>>  0xffffffff810ed4ae (inexact)
>>>>  0xffffffff81316880 (inexact)
>>>>  0xffffffff810e9743 (inexact)
>>>>  0xffffffff810ede66 (inexact)
>>>>  0xffffffff81645909 (inexact)
>>>>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>>>>  0xffffffff810eb23b (inexact)
>>>>  0xffffffff81641113 (inexact)
>>>>  0xffffffff81014b12 (inexact)
>>>>  0xffffffff81645909 (inexact)
>>>>
>>>> So it means the module isn't built with debuginfo, right?
>>>
>>> Hmm, it could mean that we just need to make sure that the debuginfo is
>>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>>> information for a module even if stap doesn't think it needs it.)
>>>
>>> --
>>> David Smith
>>> dsmith@redhat.com
>>> Red Hat
>>> http://www.redhat.com
>>> 256.217.0141 (direct)
>>> 256.837.0057 (fax)
>
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Frank Ch. Eigler]

Nan Xiao <xiaonan830818@gmail.com> writes:

> [...]
>>> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe
>>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>>> print_backtrace()) }'
>>> [...]
>>> 2280usr/540sys/5099real ms.
>>> Pass 5: starting run.
>>> WARNING: no or bad debug frame hdr
>>> WARNING: No binary search table for debug frame, doing slow linear
>>> search for /root/kernel/105.ops/kex.ko

I wonder if something is going wrong with the module pathname logic.
Could you try a few more things?

- installing kex.ko under the kernel install tree (probably
  /lib/modules/`uname -r`/kernel or similar), so that
  stap -d kex -e 'probe module("kex").... {}'  
  works (without the path name).

- running stap in save-temp-directory (-k) mode, and shipping
  us a tarball of the /tmp/XXX directory, to see specifically
  how the stap-symbols.h file looks.  (email direct rather than
  to the mailing list; this file may be big)

- running stap in -DDEBUG_UNWIND=2 mode to get lots of
  backtracing-related data 


- FChE

Re: How to track the functions in self-written module using SystemTap?

[Martin Cermak]

On  Mon  2015-12-07  13:53 , David Smith wrote:
> 1) Make sure your kernel was compiled with frame pointer support (which
> it should if it is a standard RHEL kernel):
> 
> # fgrep FRAME_POINTER /boot/config-`uname -r`
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
> 
> 2) Let's make sure your module still has debuginfo present. First, run
> "file" on your module, making sure it says "not stripped". Then go a bit
> deeper and run "readelf --sections" on your module - there should be
> several sections that start with '.debug', including one called
> '.debug_frame'.

I'm traying to trace my own stap module too.  I'm using stap -B
CONFIG_DEBUG_INFO=y to put the dfebuginfo into the module.
However, this way I seem to get the '.debug_frame' section on
RHEL6, but not on RHEL7 or recent Fedora.  Because of this I'm
failing to get reasonable backtraces.  Any workaround?

Martin

Re: How to track the functions in self-written module using SystemTap?

[David Smith]

On 10/06/2016 05:05 PM, Martin Cermak wrote:
> On  Mon  2015-12-07  13:53 , David Smith wrote:
>> 1) Make sure your kernel was compiled with frame pointer support (which
>> it should if it is a standard RHEL kernel):
>>
>> # fgrep FRAME_POINTER /boot/config-`uname -r`
>> CONFIG_SCHED_OMIT_FRAME_POINTER=y
>> CONFIG_ARCH_WANT_FRAME_POINTERS=y
>> CONFIG_FRAME_POINTER=y
>>
>> 2) Let's make sure your module still has debuginfo present. First, run
>> "file" on your module, making sure it says "not stripped". Then go a bit
>> deeper and run "readelf --sections" on your module - there should be
>> several sections that start with '.debug', including one called
>> '.debug_frame'.
> 
> I'm traying to trace my own stap module too.  I'm using stap -B
> CONFIG_DEBUG_INFO=y to put the dfebuginfo into the module.
> However, this way I seem to get the '.debug_frame' section on
> RHEL6, but not on RHEL7 or recent Fedora.  Because of this I'm
> failing to get reasonable backtraces.  Any workaround?

We actually test this in the systemtap.context/context.exp test case. It
builds 2 external modules and tries to get backtraces from them (among
other things). On RHEL7.2 ppc64, the backtrace tests don't pass, but I
think that's really a test case problem.

====
backtrace from
module("systemtap_test_module2").function("yyy_func2@/root/rhel7-2.ppc64/testsuite/artifacts/systemtap.context/context/staptestMS19Tt/systemtap_test_module2.c:33"):
 0xd000000006fd00a0 : yyy_func2+0x0/0x30 [systemtap_test_module2]
 0xd000000006fd00e0 : yyy_func1+0x10/0x0 [systemtap_test_module2]
 0xd0000000082c01b8 : stm_write_cmd+0x1b8/0x200 [systemtap_test_module1]
 0x0 (inexact)
====

The test case expects the '0x0 (inexact)' line to be from the kernel.
This might be a ppc64-specific problem. But, the backtrace from the
modules themselves look fine.

Does your systemtap.context/context.exp test case output look similar?

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: How to track the functions in self-written module using SystemTap?

[Martin Cermak]

On  Fri  2016-10-07  12:59 , David Smith wrote:
> On 10/06/2016 05:05 PM, Martin Cermak wrote:
> > On  Mon  2015-12-07  13:53 , David Smith wrote:
> >> 1) Make sure your kernel was compiled with frame pointer support (which
> >> it should if it is a standard RHEL kernel):
> >>
> >> # fgrep FRAME_POINTER /boot/config-`uname -r`
> >> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> >> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> >> CONFIG_FRAME_POINTER=y
> >>
> >> 2) Let's make sure your module still has debuginfo present. First, run
> >> "file" on your module, making sure it says "not stripped". Then go a bit
> >> deeper and run "readelf --sections" on your module - there should be
> >> several sections that start with '.debug', including one called
> >> '.debug_frame'.
> > 
> > I'm traying to trace my own stap module too.  I'm using stap -B
> > CONFIG_DEBUG_INFO=y to put the dfebuginfo into the module.
> > However, this way I seem to get the '.debug_frame' section on
> > RHEL6, but not on RHEL7 or recent Fedora.  Because of this I'm
> > failing to get reasonable backtraces.  Any workaround?
> 
> We actually test this in the systemtap.context/context.exp test case. It
> builds 2 external modules and tries to get backtraces from them (among
> other things). On RHEL7.2 ppc64, the backtrace tests don't pass, but I
> think that's really a test case problem.
> 
> ====
> backtrace from
> module("systemtap_test_module2").function("yyy_func2@/root/rhel7-2.ppc64/testsuite/artifacts/systemtap.context/context/staptestMS19Tt/systemtap_test_module2.c:33"):
>  0xd000000006fd00a0 : yyy_func2+0x0/0x30 [systemtap_test_module2]
>  0xd000000006fd00e0 : yyy_func1+0x10/0x0 [systemtap_test_module2]
>  0xd0000000082c01b8 : stm_write_cmd+0x1b8/0x200 [systemtap_test_module1]
>  0x0 (inexact)
> ====
> 
> The test case expects the '0x0 (inexact)' line to be from the kernel.
> This might be a ppc64-specific problem. But, the backtrace from the
> modules themselves look fine.
> 
> Does your systemtap.context/context.exp test case output look similar?
> 


Yup, it does look similar.  My scenario is slightly different,
but ITSM that looking at context.exp helped me to narrow my
problem down to the GCC invocation.

My scenario is to trace a real stap module.  OTOH, context.exp
traces a module not created by stap.  Here is my scenario:

=======
$ # compile and run the tracee module
$
$ stap -v -m blah -B CONFIG_DEBUG_INFO=y --poison-cache -e 'global x probe timer.s(1) {x<<<1 println(@count(x))}'
$
$ # now run the tracer module (in another terminal):
$
$ stap  -d kernel -e 'probe
module("/home/mcermak/blah.ko").function("_stp_stat_add")
{print_backtrace() exit()}'
WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear
search for /home/mcermak/blah.ko
 0xffffffffc086d628 : probe_2815+0x58/0x4d0 [blah]
$
=======

I'd expect richer backtrace above.  Comparing the simple Makefile
used by context.exp to a relatively complex Makefile of a real
stap module made me think that by commenting out some of the
EXTRA_CFLAGS defined in buildrun.cxx must fix my problem.  And
indeed, following update seems to help:

=======
   // Generate eh_frame for self-backtracing
-  o << "EXTRA_CFLAGS += -fasynchronous-unwind-tables" << endl;
+  //o << "EXTRA_CFLAGS += -fasynchronous-unwind-tables" << endl;
=======

After applying it, and running whole the test again, I'm getting
following output:

=======
$ stap  -d kernel -e 'probe module("/home/mcermak/blah.ko").function("_stp_stat_add") {print_backtrace() exit()}'
 0xffffffffc086d628 : probe_2815+0x58/0x4d0 [blah]
 0xffffffffc086c3c3 : _stp_hrtimer_notify_function+0x223/0x3f0 [blah]
 0xffffffff81115d0e : __hrtimer_run_queues+0xee/0x270 [kernel]
 0xffffffff811164aa : hrtimer_interrupt+0x9a/0x180 [kernel]
 0xffffffff81053588 : local_apic_timer_interrupt+0x38/0x60 [kernel]
 0xffffffff817e011d : smp_apic_timer_interrupt+0x3d/0x50 [kernel]
 0xffffffff817de25c : apic_timer_interrupt+0x8c/0xa0 [kernel]
$
=======

So, when the stap module gets compiled with gcc-6.2.1-1.fc26.x86_64 with
-fasynchronous-unwind-tables, there is no '.debug_frame' section and no
traceback.  Removing this option helps me:

=======
$ readelf --sections blah.ko | grep '.debug_frame'
  [37] .debug_frame      PROGBITS         0000000000000000  00031620
  [38] .rela.debug_frame RELA             0000000000000000  0009a5b0
$
=======

... but probably breaks something else, since the relevant
comment reads: "Generate eh_frame for self-backtracing".


Martin