do_mmap fails to work on RHEL5.1GA

do_mmap fails to work on RHEL5.1GA

[baiwd]

Hello:
    When I test with the newest stp on RHEL5.1GA, I  found that probe 
for do_mmap in memory.stp
cannot work correctly. It reported:
Pass 1: parsed user script and 38 library script(s) in 
306usr/4sys/325real ms.
Pass 2: analyzed script: 14 probe(s), 0 function(s), 0 embed(s), 0 
global(s) in 1076usr/97sys/1172real ms.
Pass 3: translated to C into 
"/tmp/stapLlRNuv/stap_5a545779c7b8d4c8da640eed3f8c78f8_2314.c" in 
1usr/0sys/1real ms.
Pass 4: compiled C into "stap_5a545779c7b8d4c8da640eed3f8c78f8_2314.ko" 
in 4715usr/245sys/4710real ms.
Pass 5: starting run.
ERROR: probe kernel.function("do_mmap@include/linux/mm.h:1001") 
registration error (rc -22)
     By looking through the elf file of linux kernel debug and the 
source codes, I think do_mmap and do_mmap2
are inlined functions, and they maybe inlined in optimization. But I 
find that do_mmap and do_mmap2 in all the
versions use do_mmap_pgoff which is not inlined and use almost the same 
arguments as do_mmap and do_mmap2.
So I wondered whether memory.stp can be changed as following:

--- memory.stp.old 2007-12-12 14:14:46.000000000 +0900
+++ memory.stp 2007-12-12 14:40:52.000000000 +0900
@@ -112,7 +112,7 @@ probe vm.write_shared_copy = kernel.func
* address - the requested address
* length - the length of the memory segment
*/
-probe vm.mmap = kernel.function("do_mmap"), kernel.function("do_mmap2")? {
+probe vm.mmap = kernel.function("do_mmap_pgoff") {
address = $addr
length = $len
}


 From Bai Weidong <baiwd@cn.fujitsu.com>
Best Regards

Re: do_mmap fails to work on RHEL5.1GA

[baiwd]

Frank:
Frank Ch. Eigler wrote:
> baiwd <baiwd@cn.fujitsu.com> writes:
>
>   
>> When I test with the newest stp on RHEL5.1GA, I found that probe for
>> do_mmap in memory.stp cannot work correctly. [...]
>>     
>
> This means that systemtap got addresses it thought plausible, but the
> kernel refused to actually insert a probe there.  I wonder if this is
> the latent systemtap bug awakened by elfutils 0.131.
>   
>
> This would be a good candidate for the recently added optional/sufficient flag:
>
> probe vm.mmap = kernel.function("do_mmap")!,
>                 kernel.function("do_mmap_pgoff")!,
>                 kernel.function("do_mmap2") { ... }
>   
         Thank you for your directions. From the following reports, I 
think the problem is not in
tapset. You are right. The kernel seems to refuse to insert a probe there.

Pass 1: parsed user script and 38 library script(s) in 
306usr/4sys/312real ms.
probe do_brk@mm/mmap.c:1961 kernel section=.text pc=0xa00000010012c760
probe do_mmap@include/linux/mm.h:1001 kernel section=.text 
pc=0xa0000001001afaa0probe do_mmap@include/linux/mm.h:1001 kernel 
section=.text pc=0xa0000001001cddc0probe do_mmap@include/linux/mm.h:1001 
kernel section=.text pc=0xa0000001001cde90probe 
do_mmap@include/linux/mm.h:1001 kernel section=.text 
pc=0xa0000001001ce211probe do_mmap@include/linux/mm.h:1001 kernel 
section=.text pc=0xa0000001001d0940probe do_mmap@include/linux/mm.h:1001 
kernel section=.text pc=0xa00000010022bb40Probe point 
kernel.function("do_mmap")! sufficient, skipped 
kernel.function("do_mmap_pgoff")! kernel.function("do_mmap2")probe 
do_munmap@mm/mmap.c:1874 kernel section=.text pc=0xa00000010012b340
probe __oom_kill_task@mm/oom_kill.c:265 kernel section=.text 
pc=0xa000000100104c40probe __handle_mm_fault@mm/memory.c:2574 kernel 
section=.text pc=0xa000000100123000probe do_wp_page@mm/memory.c:1622 
kernel section=.text pc=0xa00000010011e720
probe __handle_mm_fault@mm/memory.c:2574 kernel section=.text 
pc=0xa000000100123000Pass 2: analyzed script: 12 probe(s), 0 
function(s), 0 embed(s), 0 global(s) in 988usr/102sys/1099real ms.
Pass 3: using cached 
/root/.systemtap/cache/e2/stap_e21b079aada2c0de4695a0eb4bfdf705_2188.cPass 
4: using cached 
/root/.systemtap/cache/e2/stap_e21b079aada2c0de4695a0eb4bfdf705_2188.koPass 
5: starting run.
Running /usr/bin/staprun -v -d 8500 
/tmp/stapQZYd3a/stap_e21b079aada2c0de4695a0eb4bfdf705_2188.koERROR: 
probe kernel.function("do_mmap@include/linux/mm.h:1001") registration 
error (rc -22)stapio:cleanup_and_exit:216 CLEANUP AND EXIT  closed=1
stapio:cleanup_and_exit:229 closing control channel
Pass 5: run completed in 27usr/78sys/128real ms.
Running rm -rf /tmp/stapQZYd3a


Best Regards
Bai Weidong

Re: do_mmap fails to work on RHEL5.1GA

[Frank Ch. Eigler]

baiwd <baiwd@cn.fujitsu.com> writes:

> When I test with the newest stp on RHEL5.1GA, I found that probe for
> do_mmap in memory.stp cannot work correctly. [...]

Thanks for testing!

> ERROR: probe kernel.function("do_mmap@include/linux/mm.h:1001")
> registration error (rc -22)

This means that systemtap got addresses it thought plausible, but the
kernel refused to actually insert a probe there.  I wonder if this is
the latent systemtap bug awakened by elfutils 0.131.

> By looking through the elf file of linux kernel debug and the source
> codes, I think do_mmap and do_mmap2 are inlined functions [...]

That by itself is not a problem.  We can often find reasonable probing
addresses for inlined functions.  (Not being able to get at their
parameters is a whole different (gcc) problem.)


> So I wondered whether memory.stp can be changed as following:
> [...]
> -probe vm.mmap = kernel.function("do_mmap"), kernel.function("do_mmap2")? {
> +probe vm.mmap = kernel.function("do_mmap_pgoff") {

This would be a good candidate for the recently added optional/sufficient flag:

probe vm.mmap = kernel.function("do_mmap")!,
                kernel.function("do_mmap_pgoff")!,
                kernel.function("do_mmap2") { ... }

- FChE

do_mmap fails to work on RHEL5.1GA

[baiwd]

Hello:
     When I test with the newest stp on RHEL5.1GA, I  found that probe 
for do_mmap in memory.stp
cannot work correctly. It reported:
Pass 1: parsed user script and 38 library script(s) in 
306usr/4sys/325real ms.
Pass 2: analyzed script: 14 probe(s), 0 function(s), 0 embed(s), 0 
global(s) in 1076usr/97sys/1172real ms.
Pass 3: translated to C into 
"/tmp/stapLlRNuv/stap_5a545779c7b8d4c8da640eed3f8c78f8_2314.c" in 
1usr/0sys/1real ms.
Pass 4: compiled C into "stap_5a545779c7b8d4c8da640eed3f8c78f8_2314.ko" 
in 4715usr/245sys/4710real ms.
Pass 5: starting run.
ERROR: probe kernel.function("do_mmap@include/linux/mm.h:1001") 
registration error (rc -22)
      By looking through the elf file of linux kernel debug and the 
source codes, I think do_mmap and do_mmap2
are inlined functions, and they maybe inlined in optimization. But I 
find that do_mmap and do_mmap2 in all the
versions use do_mmap_pgoff which is not inlined and use almost the same 
arguments as do_mmap and do_mmap2.
So I wondered whether memory.stp can be changed as following:

--- memory.stp.old 2007-12-12 14:14:46.000000000 +0900
+++ memory.stp 2007-12-12 14:40:52.000000000 +0900
@@ -112,7 +112,7 @@ probe vm.write_shared_copy = kernel.func
* address - the requested address
* length - the length of the memory segment
*/
-probe vm.mmap = kernel.function("do_mmap"), kernel.function("do_mmap2")? {
+probe vm.mmap = kernel.function("do_mmap_pgoff") {
address = $addr
length = $len
}


 From Bai Weidong <baiwd@cn.fujitsu.com>
Best Regards