Stap is translating to functions in __exit sections...and later module load fails

Stap is translating to functions in __exit sections...and later module load fails

[Keshavamurthy, Anil S]

stap -e 'probe kernel.function("*") { log("."); }'
results in
ERROR: dwarf probe 
kernel.function("exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507") 
registration failed, rc=1

Looks like on Ia64, stap is translating to functions which are in
__exit sections, as in the above failure case exit_pfm_fs() function
is prefixed with __exit and when the module is loaded the
kprobe registration fails as the functions in exit sections no longer
exist.


I am running the latest systemtap from the CVS, please let me
know if their is a workaround.

thanks,
Anil Keshavamurthy
Intel Corp.

Re: Stap is translating to functions in __exit sections...and later module load fails

[Keshavamurthy Anil S]

On Tue, Oct 17, 2006 at 06:06:23PM -0700, Stone, Joshua I wrote:
> On Tuesday, October 17, 2006 5:52 PM, Keshavamurthy, Anil S wrote:
> > After commenting (i.e not discarding) *(.exit.text) from
> > arch/ia64/kernel/vmlinux.lds.S file and booting with this new
> > kernel, stap ignores functions in __exit sections and elaborates
> > and translates functions properly.
> 
> The issue in stap/elfutils when __exit is discarded is that it can't
> locate the function properly.  It knows there's a function called
> 'exit_pfm_fs', but since it can't figure the probe address for the
> function (since it was discarded), stap gets an address of 0x0.  Then
> blacklist_p tries to figure out what section that's in, and if that
> section is .exit.*, the function is rejected.  (Rejection in the case of
> function("*") just means it is skipped, not an error.)  In this case the
> section is not .exit.* -- or rather, we don't know what section it's in
> -- so the probe is added to the list.
> 
> Perhaps the simple solution would be to treat a probe address with an
> *unknown* section as blacklisted as well.  If we don't know what it is,
> it probably isn't safe, right?
Yup, I agree with you.

> 
> > Can someone tell me why this section is not discarded on Ia32?
> 
> Not me.  But what is the point of __exit functions if they are always
> discarded?  Wouldn't you think that __exit should be kept around for
> when the kernel exits?
If anything is inbuilt into kernel, then their is no way 
its __exit functions are called, hence it can be safely
discarded in the vmlinux.


thanks,
Anil

RE: Stap is translating to functions in __exit sections...and later module load fails

[Stone, Joshua I]

On Tuesday, October 17, 2006 5:52 PM, Keshavamurthy, Anil S wrote:
> After commenting (i.e not discarding) *(.exit.text) from
> arch/ia64/kernel/vmlinux.lds.S file and booting with this new
> kernel, stap ignores functions in __exit sections and elaborates
> and translates functions properly.

The issue in stap/elfutils when __exit is discarded is that it can't
locate the function properly.  It knows there's a function called
'exit_pfm_fs', but since it can't figure the probe address for the
function (since it was discarded), stap gets an address of 0x0.  Then
blacklist_p tries to figure out what section that's in, and if that
section is .exit.*, the function is rejected.  (Rejection in the case of
function("*") just means it is skipped, not an error.)  In this case the
section is not .exit.* -- or rather, we don't know what section it's in
-- so the probe is added to the list.

Perhaps the simple solution would be to treat a probe address with an
*unknown* section as blacklisted as well.  If we don't know what it is,
it probably isn't safe, right?

> Can someone tell me why this section is not discarded on Ia32?

Not me.  But what is the point of __exit functions if they are always
discarded?  Wouldn't you think that __exit should be kept around for
when the kernel exits?


Josh

RE: Stap is translating to functions in __exit sections...and later module load fails

[Keshavamurthy, Anil S]

After commenting (i.e not discarding) *(.exit.text)
from arch/ia64/kernel/vmlinux.lds.S file and booting with this new kernel, 
stap ignores functions in __exit sections and elaborates and
translates functions properly.

Can someone tell me why this section is not discarded on Ia32?

Thanks,
-Anil Keshavamurthy

-----Original Message-----
From: Mao, Bibo 
Sent: Monday, October 16, 2006 4:27 AM
To: Eugeniy Meshcheryakov
Cc: Stone, Joshua I; Keshavamurthy, Anil S; Systemtap
Subject: Re: Stap is translating to functions in __exit sections...and later module load fails

I do not think it is gcc bug, I think it depends on ld script, this
is part of IA64 linux 2.6.19-rc2 vmlinux.lds.S,
   SECTIONS
   {
     /* Sections to be discarded */
     /DISCARD/ : {
         *(.exit.text)
         *(.exit.data)
         *(.exitcall.exit)
         *(.IA_64.unwind.exit.text)
         *(.IA_64.unwind_info.exit.text)
         }
    .......
it seems that .exit.text is discarded when vmlinux image is linked, and
on IA32 .exit.text segment is not discarded, the script
stap -e  'probe kernel.function("*") { print(".") }' can run on IA32
platform.

it is strange that when exit_pfm_fs is probe there is one such line
	probe exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507 pc=0x0
I do not how function probe-point is parsed, I am not familiar with
Elfutils package :(

And powerpc has the same ld link script, it should has the same problem.

thanks
bibo,mao

Eugeniy Meshcheryakov wrote:
> 13 жовÑ'ня 2006 о 15:28 -0700 Stone, Joshua I написав(-ла):
>> This seems to support the notion that the linker elided that function.
>>
>> So, as Eugeniy wondered, why did the translator pick up that function at
>> all?  I'm also curious what IP it chose for the kprobe, if the function
>> no longer exists... Please try this command:
>>
>> $ stap -p2 -vv -e 'probe kernel.function("exit_pfm_fs"){}' 2>&1 | grep
>> 'pc='
>>
>> I suspect that either elfutils is giving 'stale' debug information, or
>> the translator is misinterpreting its results.
> It loks like a bug in some version of gcc. I tried to compile the
> following program with version from Debian stable and unstable on ia64:
> 
> $ cat test.c
> static void __attribute__((section(".exit.text"))) exit_pfm_fs(void)
> {
>         return;
> }
> $
> 
> With gcc 3.3.5:
> 
> $ readelf -a test.o | grep exit_
>      9: 0000000000000000    16 FUNC    LOCAL  DEFAULT    9 exit_pfm_fs
> $ objdump -W test.o
> ...
> The section .debug_info contains:
> 
>   Compilation Unit @ offset 0x0:
>    Length:        52
>    Version:       2
>    Abbrev Offset: 0
>    Pointer Size:  8
>  <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>      DW_AT_stmt_list   : 0
>      DW_AT_name        : (indirect string, offset: 0x38): test.c
>      DW_AT_comp_dir    : (indirect string, offset: 0x0): /home/eugen
>      DW_AT_producer    : (indirect string, offset: 0xc): GNU C 3.3.5
> (Debian 1:3.3.5-13)
>      DW_AT_language    : 1      (ANSI C)
>  <1><1d>: Abbrev Number: 2 (DW_TAG_subprogram)
>      DW_AT_name        : (indirect string, offset: 0x2c): exit_pfm_fs
>      DW_AT_decl_file   : 1
>      DW_AT_decl_line   : 2
>      DW_AT_prototyped  : 1
>      DW_AT_low_pc      : 0
>      DW_AT_high_pc     : 0x10
>      DW_AT_frame_base  : 1 byte block: 5c       (DW_OP_reg12)
> ...
> $
> 
> With gcc 4.1.2:
> 
> $ readelf -a t.o | grep exit_
> $ objdump -W t.o | grep exit_
> objdump: Error: No comp units in .debug_info section ?objdump: Error: No
> comp units in .debug_info section ?$
> 
> 

Re: Stap is translating to functions in __exit sections...and later module load fails

[bibo,mao]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=windows-1252; format=flowed, Size: 2990 bytes --]

I do not think it is gcc bug, I think it depends on ld script, this
is part of IA64 linux 2.6.19-rc2 vmlinux.lds.S,
   SECTIONS
   {
     /* Sections to be discarded */
     /DISCARD/ : {
         *(.exit.text)
         *(.exit.data)
         *(.exitcall.exit)
         *(.IA_64.unwind.exit.text)
         *(.IA_64.unwind_info.exit.text)
         }
    .......
it seems that .exit.text is discarded when vmlinux image is linked, and
on IA32 .exit.text segment is not discarded, the script
stap -e  'probe kernel.function("*") { print(".") }' can run on IA32
platform.

it is strange that when exit_pfm_fs is probe there is one such line
	probe exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507 pc=0x0
I do not how function probe-point is parsed, I am not familiar with
Elfutils package :(

And powerpc has the same ld link script, it should has the same problem.

thanks
bibo,mao

Eugeniy Meshcheryakov wrote:
> 13 жовтня 2006 о 15:28 -0700 Stone, Joshua I написав(-ла):
>> This seems to support the notion that the linker elided that function.
>>
>> So, as Eugeniy wondered, why did the translator pick up that function at
>> all?  I'm also curious what IP it chose for the kprobe, if the function
>> no longer exists... Please try this command:
>>
>> $ stap -p2 -vv -e 'probe kernel.function("exit_pfm_fs"){}' 2>&1 | grep
>> 'pc='
>>
>> I suspect that either elfutils is giving 'stale' debug information, or
>> the translator is misinterpreting its results.
> It loks like a bug in some version of gcc. I tried to compile the
> following program with version from Debian stable and unstable on ia64:
> 
> $ cat test.c
> static void __attribute__((section(".exit.text"))) exit_pfm_fs(void)
> {
>         return;
> }
> $
> 
> With gcc 3.3.5:
> 
> $ readelf -a test.o | grep exit_
>      9: 0000000000000000    16 FUNC    LOCAL  DEFAULT    9 exit_pfm_fs
> $ objdump -W test.o
> ...
> The section .debug_info contains:
> 
>   Compilation Unit @ offset 0x0:
>    Length:        52
>    Version:       2
>    Abbrev Offset: 0
>    Pointer Size:  8
>  <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>      DW_AT_stmt_list   : 0
>      DW_AT_name        : (indirect string, offset: 0x38): test.c
>      DW_AT_comp_dir    : (indirect string, offset: 0x0): /home/eugen
>      DW_AT_producer    : (indirect string, offset: 0xc): GNU C 3.3.5
> (Debian 1:3.3.5-13)
>      DW_AT_language    : 1      (ANSI C)
>  <1><1d>: Abbrev Number: 2 (DW_TAG_subprogram)
>      DW_AT_name        : (indirect string, offset: 0x2c): exit_pfm_fs
>      DW_AT_decl_file   : 1
>      DW_AT_decl_line   : 2
>      DW_AT_prototyped  : 1
>      DW_AT_low_pc      : 0
>      DW_AT_high_pc     : 0x10
>      DW_AT_frame_base  : 1 byte block: 5c       (DW_OP_reg12)
> ...
> $
> 
> With gcc 4.1.2:
> 
> $ readelf -a t.o | grep exit_
> $ objdump -W t.o | grep exit_
> objdump: Error: No comp units in .debug_info section ?objdump: Error: No
> comp units in .debug_info section ?$
> 
> 

Re: Stap is translating to functions in __exit sections...and later module load fails

[Eugeniy Meshcheryakov]

[-- Attachment #1: Type: text/plain, Size: 2035 bytes --]

13 жовтня 2006 о 15:28 -0700 Stone, Joshua I написав(-ла):
> This seems to support the notion that the linker elided that function.
> 
> So, as Eugeniy wondered, why did the translator pick up that function at
> all?  I'm also curious what IP it chose for the kprobe, if the function
> no longer exists... Please try this command:
> 
> $ stap -p2 -vv -e 'probe kernel.function("exit_pfm_fs"){}' 2>&1 | grep
> 'pc='
> 
> I suspect that either elfutils is giving 'stale' debug information, or
> the translator is misinterpreting its results.
It loks like a bug in some version of gcc. I tried to compile the
following program with version from Debian stable and unstable on ia64:

$ cat test.c
static void __attribute__((section(".exit.text"))) exit_pfm_fs(void)
{
        return;
}
$

With gcc 3.3.5:

$ readelf -a test.o | grep exit_
     9: 0000000000000000    16 FUNC    LOCAL  DEFAULT    9 exit_pfm_fs
$ objdump -W test.o
...
The section .debug_info contains:

  Compilation Unit @ offset 0x0:
   Length:        52
   Version:       2
   Abbrev Offset: 0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
     DW_AT_stmt_list   : 0
     DW_AT_name        : (indirect string, offset: 0x38): test.c
     DW_AT_comp_dir    : (indirect string, offset: 0x0): /home/eugen
     DW_AT_producer    : (indirect string, offset: 0xc): GNU C 3.3.5
(Debian 1:3.3.5-13)
     DW_AT_language    : 1      (ANSI C)
 <1><1d>: Abbrev Number: 2 (DW_TAG_subprogram)
     DW_AT_name        : (indirect string, offset: 0x2c): exit_pfm_fs
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 2
     DW_AT_prototyped  : 1
     DW_AT_low_pc      : 0
     DW_AT_high_pc     : 0x10
     DW_AT_frame_base  : 1 byte block: 5c       (DW_OP_reg12)
...
$

With gcc 4.1.2:

$ readelf -a t.o | grep exit_
$ objdump -W t.o | grep exit_
objdump: Error: No comp units in .debug_info section ?objdump: Error: No
comp units in .debug_info section ?$


-- 
Eugeniy Meshcheryakov

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Stap is translating to functions in __exit sections...and later module load fails

[Keshavamurthy Anil S]

On Fri, Oct 13, 2006 at 03:28:46PM -0700, Stone, Joshua I wrote:
> On Friday, October 13, 2006 2:29 PM, Eugeniy Meshcheryakov wrote:
> $ stap -p2 -vv -e 'probe kernel.function("exit_pfm_fs"){}' 2>&1 | grep
> 'pc='

[root@csdor-tiger1 linux-2.6.19-rc1]# stap -p2 -vv -e 'probe kernel.function("exit_pfm_fs"){}' 2>&1 | grep 'pc='
probe exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507 pc=0x0

-Anil

RE: Stap is translating to functions in __exit sections...and later module load fails

[Stone, Joshua I]

On Friday, October 13, 2006 2:29 PM, Eugeniy Meshcheryakov wrote:
> It is strange that translator sees it at all. The code is
> 
> static void __exit
> exit_pfm_fs(void)
> {
>         unregister_filesystem(&pfm_fs_type);
>         mntput(pfmfs_mnt);
> }
> 
> and this function is never used...

If the function is static and never used, then the linker will probably
remove it...


On Friday, October 13, 2006 2:38 PM, Keshavamurthy, Anil S wrote:
> Looks like this is in .exit.text section
> [root@csdor-tiger1 linux-2.6.19-rc1]# objdump -t
> arch/ia64/kernel/perfmon.o |grep exit_pfm_fs 0000000000000000 l     F
> .exit.text     0000000000000070 exit_pfm_fs 
> 
> However "objdump -t vmlinux| grep exit_pfm_fs" did not show any thing.

This seems to support the notion that the linker elided that function.

So, as Eugeniy wondered, why did the translator pick up that function at
all?  I'm also curious what IP it chose for the kprobe, if the function
no longer exists... Please try this command:

$ stap -p2 -vv -e 'probe kernel.function("exit_pfm_fs"){}' 2>&1 | grep
'pc='

I suspect that either elfutils is giving 'stale' debug information, or
the translator is misinterpreting its results.


Josh

Re: Stap is translating to functions in __exit sections...and later module load fails

[Keshavamurthy Anil S]

On Fri, Oct 13, 2006 at 02:26:44PM -0700, Stone, Joshua I wrote:
> On Friday, October 13, 2006 1:51 PM, Keshavamurthy, Anil S wrote:
> > On Fri, Oct 13, 2006 at 02:02:39PM -0700, Stone, Joshua I wrote:
> >> The translator blacklist tries to avoid such functions by checking
> >> the section name -- anything within a section beginning with
> >> '.exit.' (e.g., .exit.text) is blacklisted, and thus won't match a
> >> function("*"). 
> > Exactly, I expected that the translator blacklists' and does not
> > elaborate such functions. But looks like this is not happening.
> > 
> > See here....
> > #stap -p2 -e 'probe kernel.function("*") {}' | sort | uniq |grep
> > exit_pfm_fs
> > kernel.function("exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507"), 
> 
> Yes, I believe you that the translator is producing a hit for
> exit_pfm_fs.  The question is *why*.  Can you use objdump on your
> vmlinux to find out what section that function is actually in?
> 
> $ objdump -t <...>/vmlinux | grep exit_pfm_fs
> 
> On x86 and x86_64, functions decorated with __exit end up in
> '.exit.text'.  If this isn't the case for IA64, then we'll need a
> different mechanism for detecting such functions from the translator.

Looks like this is in .exit.text section
[root@csdor-tiger1 linux-2.6.19-rc1]# objdump -t arch/ia64/kernel/perfmon.o |grep exit_pfm_fs
0000000000000000 l     F .exit.text     0000000000000070 exit_pfm_fs

However "objdump -t vmlinux| grep exit_pfm_fs" did not show any thing.

-thanks,
Anil Keshavamurthy




> 
> 
> Josh

Re: Stap is translating to functions in __exit sections...and later module load fails

[Eugeniy Meshcheryakov]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

13 жовтня 2006 о 14:02 -0700 Stone, Joshua I написав(-ла):
> The translator blacklist tries to avoid such functions by checking the
> section name -- anything within a section beginning with '.exit.' (e.g.,
> .exit.text) is blacklisted, and thus won't match a function("*").
> 
> Does IA64 do something different with the __exit sections?
It is strange that translator sees it at all. The code is

static void __exit
exit_pfm_fs(void)
{
        unregister_filesystem(&pfm_fs_type);
        mntput(pfmfs_mnt);
}

and this function is never used...

-- 
Eugeniy Meshcheryakov

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: Stap is translating to functions in __exit sections...and later module load fails

[Stone, Joshua I]

On Friday, October 13, 2006 1:51 PM, Keshavamurthy, Anil S wrote:
> On Fri, Oct 13, 2006 at 02:02:39PM -0700, Stone, Joshua I wrote:
>> The translator blacklist tries to avoid such functions by checking
>> the section name -- anything within a section beginning with
>> '.exit.' (e.g., .exit.text) is blacklisted, and thus won't match a
>> function("*"). 
> Exactly, I expected that the translator blacklists' and does not
> elaborate such functions. But looks like this is not happening.
> 
> See here....
> #stap -p2 -e 'probe kernel.function("*") {}' | sort | uniq |grep
> exit_pfm_fs
> kernel.function("exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507"), 

Yes, I believe you that the translator is producing a hit for
exit_pfm_fs.  The question is *why*.  Can you use objdump on your
vmlinux to find out what section that function is actually in?

$ objdump -t <...>/vmlinux | grep exit_pfm_fs

On x86 and x86_64, functions decorated with __exit end up in
'.exit.text'.  If this isn't the case for IA64, then we'll need a
different mechanism for detecting such functions from the translator.


Josh

Re: Stap is translating to functions in __exit sections...and later module load fails

[Keshavamurthy Anil S]

On Fri, Oct 13, 2006 at 02:02:39PM -0700, Stone, Joshua I wrote:
> On Friday, October 13, 2006 1:26 PM, Keshavamurthy, Anil S wrote:
> > stap -e 'probe kernel.function("*") { log("."); }'
> > results in
> > ERROR: dwarf probe
> > kernel.function("exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507")
> > registration failed, rc=1
> > 
> > Looks like on Ia64, stap is translating to functions which are in
> > __exit sections, as in the above failure case exit_pfm_fs() function
> > is prefixed with __exit and when the module is loaded the
> > kprobe registration fails as the functions in exit sections no longer
> > exist.
> 
> The translator blacklist tries to avoid such functions by checking the
> section name -- anything within a section beginning with '.exit.' (e.g.,
> .exit.text) is blacklisted, and thus won't match a function("*").
Exactly, I expected that the translator blacklists' and does not 
elaborate such functions. But looks like this is not happening.

See here....
#stap -p2 -e 'probe kernel.function("*") {}' | sort | uniq |grep exit_pfm_fs
kernel.function("exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507"),

# cat /proc/kallsyms |grep exit_pfm_fs
#
#

> 
> Does IA64 do something different with the __exit sections?
> 
Everything in __exit sections will be discarded just after boot.

-thanks,
Anil

RE: Stap is translating to functions in __exit sections...and later module load fails

[Stone, Joshua I]

On Friday, October 13, 2006 1:26 PM, Keshavamurthy, Anil S wrote:
> stap -e 'probe kernel.function("*") { log("."); }'
> results in
> ERROR: dwarf probe
> kernel.function("exit_pfm_fs@arch/ia64/kernel/perfmon.c:1507")
> registration failed, rc=1
> 
> Looks like on Ia64, stap is translating to functions which are in
> __exit sections, as in the above failure case exit_pfm_fs() function
> is prefixed with __exit and when the module is loaded the
> kprobe registration fails as the functions in exit sections no longer
> exist.

The translator blacklist tries to avoid such functions by checking the
section name -- anything within a section beginning with '.exit.' (e.g.,
.exit.text) is blacklisted, and thus won't match a function("*").

Does IA64 do something different with the __exit sections?


Josh