* Why 'stap -L' shows no variables in some modules?
@ 2017-09-14 15:11 Freeman Zhang
2017-09-14 16:05 ` David Smith
0 siblings, 1 reply; 6+ messages in thread
From: Freeman Zhang @ 2017-09-14 15:11 UTC (permalink / raw)
To: systemtap
Hello list!
I'm learning to use systemtap these days. However I find something
stang: I can access local variables and parameters in some modules while
I can't do that in some others. For example:
--1--
stap -e 'probe kernel.function("sys_*") { printf("%s\n", $$parms$) }'
can print a lot but when I try in module nvme:
stap -e 'probe module("nvme").function("*") { printf("%s\n", $$parms$) }'
will print nothing but empty lines.
--2--
When I try to check probe points in module ext4:
stap -L 'module("ext4").function("*")'
This will print available probe points and variables (to be more
accurate, I think they are parms instead of local vars):
module("ext4").function("try_to_extend_transaction@fs/ext4/indirect.c:703") $handle:handle_t* $inode:struct inode*
however when I try it in module nvme:
stap -L 'module("nvme").function("*")'
it only prints probe points, no variables:
module("nvme").function("nvme_resume")
module("nvme").function("nvme_setup_io_queues")
module("nvme").function("nvme_shutdown")
module("nvme").function("nvme_slot_reset")
----
#I wonder why they print differently? What determines whether 'stap -L'
shows local vars or not?
# how can I access the local vars when 'stap -L' gives no local vars?
FYI: The kernel I use is 4.11.3 and is built by myself. I think it's
with debuginfo because I turned on the CONFIG_DEBUG_INFO stuff and got a
159MB vmlinux. GDB also gave me positive response loading symbols from
vmlinux and ko files. And the systemtap was built from the updated git
repo.
Since I am not in the list at present (I haven't received any
subscription confirm email somehow), please reply directly to me or cc
me if you are willing to help.
All the best!
Freeman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Why 'stap -L' shows no variables in some modules?
2017-09-14 15:11 Why 'stap -L' shows no variables in some modules? Freeman Zhang
@ 2017-09-14 16:05 ` David Smith
2017-09-14 16:31 ` Freeman Zhang
0 siblings, 1 reply; 6+ messages in thread
From: David Smith @ 2017-09-14 16:05 UTC (permalink / raw)
To: Freeman Zhang; +Cc: systemtap
On Thu, Sep 14, 2017 at 10:11 AM, Freeman Zhang
<freeman.zhang1992@gmail.com> wrote:
> Hello list!
>
> I'm learning to use systemtap these days. However I find something
> stang: I can access local variables and parameters in some modules while
> I can't do that in some others. For example:
>
> --1--
> stap -e 'probe kernel.function("sys_*") { printf("%s\n", $$parms$) }'
>
> can print a lot but when I try in module nvme:
>
> stap -e 'probe module("nvme").function("*") { printf("%s\n", $$parms$) }'
>
> will print nothing but empty lines.
>
>
> --2--
> When I try to check probe points in module ext4:
>
> stap -L 'module("ext4").function("*")'
>
> This will print available probe points and variables (to be more
> accurate, I think they are parms instead of local vars):
>
> module("ext4").function("try_to_extend_transaction@fs/ext4/indirect.c:703") $handle:handle_t* $inode:struct inode*
>
> however when I try it in module nvme:
>
> stap -L 'module("nvme").function("*")'
>
> it only prints probe points, no variables:
>
> module("nvme").function("nvme_resume")
> module("nvme").function("nvme_setup_io_queues")
> module("nvme").function("nvme_shutdown")
> module("nvme").function("nvme_slot_reset")
> ----
>
> #I wonder why they print differently? What determines whether 'stap -L'
> shows local vars or not?
There are several factors that determine whether stap can find
parameters and local variables. The compiler you use, the quality of
its debuginfo, and the level of optimization are the biggest issues.
In the case of local variables, with a sufficiently high optimization
level, the compiler may have completely optimized the variable away so
it never existed.
In the case of the nvme module itself, on my distro kernel I certainly
see parameters with 'stap -L' (and perhaps local variables too, I
didn't check).
I'd make sure the nvme module has debuginfo. There are probably
several ways of doing this, the easiest might be using 'eu-readelf
-w'.
> # how can I access the local vars when 'stap -L' gives no local vars?
>
> FYI: The kernel I use is 4.11.3 and is built by myself. I think it's
> with debuginfo because I turned on the CONFIG_DEBUG_INFO stuff and got a
> 159MB vmlinux. GDB also gave me positive response loading symbols from
> vmlinux and ko files. And the systemtap was built from the updated git
> repo.
You might show us the output of:
$ grep DEBUG_INFO .config
Make sure CONFIG_DEBUG_INFO_SPLIT isn't on.
--
David Smith
Principal Software Engineer
Red Hat
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Why 'stap -L' shows no variables in some modules?
2017-09-14 16:05 ` David Smith
@ 2017-09-14 16:31 ` Freeman Zhang
2017-09-14 18:06 ` David Smith
0 siblings, 1 reply; 6+ messages in thread
From: Freeman Zhang @ 2017-09-14 16:31 UTC (permalink / raw)
To: David Smith; +Cc: systemtap
Hi David
Thanks for your reply!
On 9/15/17 12:05 AM, David Smith wrote:
>>
>> #I wonder why they print differently? What determines whether 'stap -L'
>> shows local vars or not?
>
> There are several factors that determine whether stap can find
> parameters and local variables. The compiler you use, the quality of
> its debuginfo, and the level of optimization are the biggest issues.
> In the case of local variables, with a sufficiently high optimization
> level, the compiler may have completely optimized the variable away so
> it never existed.
>
> In the case of the nvme module itself, on my distro kernel I certainly
> see parameters with 'stap -L' (and perhaps local variables too, I
> didn't check).
>
> I'd make sure the nvme module has debuginfo. There are probably
> several ways of doing this, the easiest might be using 'eu-readelf
> -w'.
>
I would like to lower the optimization level but I heard there would be
bunches of problems. Does this mean I'll never get access to local
variables?
'eu-readelf -w nvme.ko' print something like:
---
➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | head
DWARF section [32] '.debug_aranges' at offset 0x6ddc:
Table at offset 0:
Length: 92
DWARF version: 2
CU offset: 0
Address size: 8
Segment size: 0
➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | tail
<nvme_queue_rq+0x297>...text+0x00000000000031cf <nvme_queue_rq+0x29f>
[ 1d00] .text+000000000000000000
<nvme_admin_exit_hctx>...text.unlikely+000000000000000000
<dma_unmap_sg_attrs.part.37>
.text.unlikely+000000000000000000
<dma_unmap_sg_attrs.part.37>...init.text+000000000000000000 <init_module>
.init.text+000000000000000000
<init_module>...exit.text+000000000000000000 <cleanup_module>
.exit.text+000000000000000000
<cleanup_module>...exit.text+0x000000000000001e
---
>> # how can I access the local vars when 'stap -L' gives no local vars?
>>
>> FYI: The kernel I use is 4.11.3 and is built by myself. I think it's
>> with debuginfo because I turned on the CONFIG_DEBUG_INFO stuff and got a
>> 159MB vmlinux. GDB also gave me positive response loading symbols from
>> vmlinux and ko files. And the systemtap was built from the updated git
>> repo.
>
> You might show us the output of:
>
> $ grep DEBUG_INFO .config
>
> Make sure CONFIG_DEBUG_INFO_SPLIT isn't on.
>
➜ linux-4.11.3 grep DEBUG_INFO .config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
Is there any problem?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Why 'stap -L' shows no variables in some modules?
2017-09-14 16:31 ` Freeman Zhang
@ 2017-09-14 18:06 ` David Smith
2017-09-15 2:05 ` Freeman Zhang
0 siblings, 1 reply; 6+ messages in thread
From: David Smith @ 2017-09-14 18:06 UTC (permalink / raw)
To: Freeman Zhang; +Cc: systemtap
On Thu, Sep 14, 2017 at 11:31 AM, Freeman Zhang
<freeman.zhang1992@gmail.com> wrote:
> Hi David
>
> Thanks for your reply!
>
> On 9/15/17 12:05 AM, David Smith wrote:
>>>
>>> #I wonder why they print differently? What determines whether 'stap -L'
>>> shows local vars or not?
>>
>> There are several factors that determine whether stap can find
>> parameters and local variables. The compiler you use, the quality of
>> its debuginfo, and the level of optimization are the biggest issues.
>> In the case of local variables, with a sufficiently high optimization
>> level, the compiler may have completely optimized the variable away so
>> it never existed.
>>
>> In the case of the nvme module itself, on my distro kernel I certainly
>> see parameters with 'stap -L' (and perhaps local variables too, I
>> didn't check).
>>
>> I'd make sure the nvme module has debuginfo. There are probably
>> several ways of doing this, the easiest might be using 'eu-readelf
>> -w'.
>>
>
> I would like to lower the optimization level but I heard there would be
> bunches of problems. Does this mean I'll never get access to local
> variables?
It depends on your compiler.
> 'eu-readelf -w nvme.ko' print something like:
>
> ---
>
> ➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | head
> DWARF section [32] '.debug_aranges' at offset 0x6ddc:
> Table at offset 0:
> Length: 92
> DWARF version: 2
> CU offset: 0
> Address size: 8
> Segment size: 0
>
> ➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | tail
> <nvme_queue_rq+0x297>...text+0x00000000000031cf <nvme_queue_rq+0x29f>
> [ 1d00] .text+000000000000000000
> <nvme_admin_exit_hctx>...text.unlikely+000000000000000000
> <dma_unmap_sg_attrs.part.37>
> .text.unlikely+000000000000000000
> <dma_unmap_sg_attrs.part.37>...init.text+000000000000000000 <init_module>
> .init.text+000000000000000000
> <init_module>...exit.text+000000000000000000 <cleanup_module>
> .exit.text+000000000000000000
> <cleanup_module>...exit.text+0x000000000000001e
>
> ---
From what I can see, the above looks fairly reasonable.
>>> # how can I access the local vars when 'stap -L' gives no local vars?
Let's try something else here, try the following:
# stap -L 'module("nvme").function("*").call'
That should ignore inlined function calls, and should give you
parameter information.
>>> FYI: The kernel I use is 4.11.3 and is built by myself. I think it's
>>> with debuginfo because I turned on the CONFIG_DEBUG_INFO stuff and got a
>>> 159MB vmlinux. GDB also gave me positive response loading symbols from
>>> vmlinux and ko files. And the systemtap was built from the updated git
>>> repo.
>>
>> You might show us the output of:
>>
>> $ grep DEBUG_INFO .config
>>
>> Make sure CONFIG_DEBUG_INFO_SPLIT isn't on.
>>
>
>
> ➜ linux-4.11.3 grep DEBUG_INFO .config
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> # CONFIG_DEBUG_INFO_DWARF4 is not set
>
> Is there any problem?
Nope, that looks fairly reasonable.
--
David Smith
Principal Software Engineer
Red Hat
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Why 'stap -L' shows no variables in some modules?
2017-09-14 18:06 ` David Smith
@ 2017-09-15 2:05 ` Freeman Zhang
2017-09-15 2:34 ` Freeman Zhang
0 siblings, 1 reply; 6+ messages in thread
From: Freeman Zhang @ 2017-09-15 2:05 UTC (permalink / raw)
To: David Smith; +Cc: systemtap
On 9/15/17 2:06 AM, David Smith wrote:
> On Thu, Sep 14, 2017 at 11:31 AM, Freeman Zhang
> <freeman.zhang1992@gmail.com> wrote:
>> Hi David
>>
>> Thanks for your reply!
>>
>> On 9/15/17 12:05 AM, David Smith wrote:
>>>>
>>>> #I wonder why they print differently? What determines whether 'stap -L'
>>>> shows local vars or not?
>>>
>>> There are several factors that determine whether stap can find
>>> parameters and local variables. The compiler you use, the quality of
>>> its debuginfo, and the level of optimization are the biggest issues.
>>> In the case of local variables, with a sufficiently high optimization
>>> level, the compiler may have completely optimized the variable away so
>>> it never existed.
>>>
>>> In the case of the nvme module itself, on my distro kernel I certainly
>>> see parameters with 'stap -L' (and perhaps local variables too, I
>>> didn't check).
>>>
>>> I'd make sure the nvme module has debuginfo. There are probably
>>> several ways of doing this, the easiest might be using 'eu-readelf
>>> -w'.
>>>
>>
>> I would like to lower the optimization level but I heard there would be
>> bunches of problems. Does this mean I'll never get access to local
>> variables?
>
> It depends on your compiler.
>
>> 'eu-readelf -w nvme.ko' print something like:
>>
>> ---
>>
>> ➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | head
>> DWARF section [32] '.debug_aranges' at offset 0x6ddc:
>> Table at offset 0:
>> Length: 92
>> DWARF version: 2
>> CU offset: 0
>> Address size: 8
>> Segment size: 0
>>
>> ➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | tail
>> <nvme_queue_rq+0x297>...text+0x00000000000031cf <nvme_queue_rq+0x29f>
>> [ 1d00] .text+000000000000000000
>> <nvme_admin_exit_hctx>...text.unlikely+000000000000000000
>> <dma_unmap_sg_attrs.part.37>
>> .text.unlikely+000000000000000000
>> <dma_unmap_sg_attrs.part.37>...init.text+000000000000000000 <init_module>
>> .init.text+000000000000000000
>> <init_module>...exit.text+000000000000000000 <cleanup_module>
>> .exit.text+000000000000000000
>> <cleanup_module>...exit.text+0x000000000000001e
>>
>> ---
>
> From what I can see, the above looks fairly reasonable.
>
>>>> # how can I access the local vars when 'stap -L' gives no local vars?
>
> Let's try something else here, try the following:
>
> # stap -L 'module("nvme").function("*").call'
>
> That should ignore inlined function calls, and should give you
> parameter information.
>
No, it doesn't work either for nvme module while for other modules it
does strip some function calls like you said.
The problem is that I don't see anything special in nvme codes as well
as its Makefile and output of eu_readelf. If it's for optimization
matter, other modules shouldn't have printed any variable either but in
fact they do -- it's just nvme that refuses to print :-(
>>>> FYI: The kernel I use is 4.11.3 and is built by myself. I think it's
>>>> with debuginfo because I turned on the CONFIG_DEBUG_INFO stuff and got a
>>>> 159MB vmlinux. GDB also gave me positive response loading symbols from
>>>> vmlinux and ko files. And the systemtap was built from the updated git
>>>> repo.
>>>
>>> You might show us the output of:
>>>
>>> $ grep DEBUG_INFO .config
>>>
>>> Make sure CONFIG_DEBUG_INFO_SPLIT isn't on.
>>>
>>
>>
>> ➜ linux-4.11.3 grep DEBUG_INFO .config
>> CONFIG_DEBUG_INFO=y
>> # CONFIG_DEBUG_INFO_REDUCED is not set
>> # CONFIG_DEBUG_INFO_SPLIT is not set
>> # CONFIG_DEBUG_INFO_DWARF4 is not set
>>
>> Is there any problem?
>
> Nope, that looks fairly reasonable.
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Why 'stap -L' shows no variables in some modules?
2017-09-15 2:05 ` Freeman Zhang
@ 2017-09-15 2:34 ` Freeman Zhang
0 siblings, 0 replies; 6+ messages in thread
From: Freeman Zhang @ 2017-09-15 2:34 UTC (permalink / raw)
To: David Smith; +Cc: systemtap
On 9/15/17 10:05 AM, Freeman Zhang wrote:
>
>
> On 9/15/17 2:06 AM, David Smith wrote:
>> On Thu, Sep 14, 2017 at 11:31 AM, Freeman Zhang
>> <freeman.zhang1992@gmail.com> wrote:
>>> Hi David
>>>
>>> Thanks for your reply!
>>>
>>> On 9/15/17 12:05 AM, David Smith wrote:
>>>>>
>>>>> #I wonder why they print differently? What determines whether 'stap -L'
>>>>> shows local vars or not?
>>>>
>>>> There are several factors that determine whether stap can find
>>>> parameters and local variables. The compiler you use, the quality of
>>>> its debuginfo, and the level of optimization are the biggest issues.
>>>> In the case of local variables, with a sufficiently high optimization
>>>> level, the compiler may have completely optimized the variable away so
>>>> it never existed.
>>>>
>>>> In the case of the nvme module itself, on my distro kernel I certainly
>>>> see parameters with 'stap -L' (and perhaps local variables too, I
>>>> didn't check).
>>>>
>>>> I'd make sure the nvme module has debuginfo. There are probably
>>>> several ways of doing this, the easiest might be using 'eu-readelf
>>>> -w'.
>>>>
>>>
>>> I would like to lower the optimization level but I heard there would be
>>> bunches of problems. Does this mean I'll never get access to local
>>> variables?
>>
>> It depends on your compiler.
>>
>>> 'eu-readelf -w nvme.ko' print something like:
>>>
>>> ---
>>>
>>> ➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | head
>>> DWARF section [32] '.debug_aranges' at offset 0x6ddc:
>>> Table at offset 0:
>>> Length: 92
>>> DWARF version: 2
>>> CU offset: 0
>>> Address size: 8
>>> Segment size: 0
>>>
>>> ➜ linux-4.11.3 eu-readelf -w drivers/nvme/host/nvme.ko | tail
>>> <nvme_queue_rq+0x297>...text+0x00000000000031cf <nvme_queue_rq+0x29f>
>>> [ 1d00] .text+000000000000000000
>>> <nvme_admin_exit_hctx>...text.unlikely+000000000000000000
>>> <dma_unmap_sg_attrs.part.37>
>>> .text.unlikely+000000000000000000
>>> <dma_unmap_sg_attrs.part.37>...init.text+000000000000000000 <init_module>
>>> .init.text+000000000000000000
>>> <init_module>...exit.text+000000000000000000 <cleanup_module>
>>> .exit.text+000000000000000000
>>> <cleanup_module>...exit.text+0x000000000000001e
>>>
>>> ---
>>
>> From what I can see, the above looks fairly reasonable.
>>
>>>>> # how can I access the local vars when 'stap -L' gives no local vars?
>>
>> Let's try something else here, try the following:
>>
>> # stap -L 'module("nvme").function("*").call'
>>
>> That should ignore inlined function calls, and should give you
>> parameter information.
>>
>
> No, it doesn't work either for nvme module while for other modules it
> does strip some function calls like you said.
>
> The problem is that I don't see anything special in nvme codes as well
> as its Makefile and output of eu_readelf. If it's for optimization
> matter, other modules shouldn't have printed any variable either but in
> fact they do -- it's just nvme that refuses to print :-(
>
Now I know the unwelcome truth -- the nvme module inserted to the system
isn't the one I built.
[root@200 zhangzy]# modinfo nvme
filename: /lib/modules/4.11.3/extra/mlnx-nvme/host/nvme.ko
[root@200 zhangzy]# du -h
/lib/modules/4.11.3/extra/mlnx-nvme//host/nvme.ko
56K /lib/modules/4.11.3/extra/mlnx-nvme//host/nvme.ko
[root@200 zhangzy]# du -h
/lib/modules/4.11.3/build/drivers/nvme/host/nvme.ko
508K /lib/modules/4.11.3/build/drivers/nvme/host/nvme.ko
After replaceing the out-of-tree modules with in-tree ones, I got
everything and probably want to lower opitimization level for more
information.
Thank you every much for your help!
Freeman
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-09-15 2:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-14 15:11 Why 'stap -L' shows no variables in some modules? Freeman Zhang
2017-09-14 16:05 ` David Smith
2017-09-14 16:31 ` Freeman Zhang
2017-09-14 18:06 ` David Smith
2017-09-15 2:05 ` Freeman Zhang
2017-09-15 2:34 ` Freeman Zhang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).