A 48 byte kernel module/cfi load mystery

A 48 byte kernel module/cfi load mystery

[Mark Wielaard]

Hi Roland,

I was hoping you (or any other systemtap hacker of course) could help me
solve a mystery 48 byte offset I am seeing when using the CFI of kernel
modules. For some reason it looks like the FDE initial location is
always 48 bytes beyond the actual code that it covers. You can also see
this by hand. e.g:

$ eu-objdump --disassemble /lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko
/lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko: elf64-elf_x86_64

Disassembly of section .text:

       0:    55                       push    %rbp
       1:    48 89 e5                 mov     %rsp,%rbp
       4:    e8 00 00 00 00           callq   0x9
       9:    48 8b 87 90 02 00 00     mov     0x290(%rdi),%rax
      10:    49 89 d0                 mov     %rdx,%r8
      13:    48 8b 50 60              mov     0x60(%rax),%rdx
      17:    8b 78 10                 mov     0x10(%rax),%edi
      1a:    8b 52 14                 mov     0x14(%rdx),%edx
      1d:    48 29 d6                 sub     %rdx,%rsi
      20:    48 85 c9                 test    %rcx,%rcx
      23:    74 0a                    je      0x2f
      25:    48 89 f0                 mov     %rsi,%rax
      28:    31 d2                    xor     %edx,%edx
      2a:    48 f7 f7                 div     %rdi
      2d:    89 11                    mov     %edx,(%rcx)
      2f:    4d 85 c0                 test    %r8,%r8
      32:    74 0b                    je      0x3f
      34:    48 89 f0                 mov     %rsi,%rax
      37:    31 d2                    xor     %edx,%edx
      39:    48 f7 f7                 div     %rdi
      3c:    41 89 00                 mov     %eax,(%r8)
      3f:    c9                       leaveq
      40:    c3                       retq

Our first function starts at zero in the .text section and is 0x41 bytes
long. So look at the CFIs:

$ eu-readelf --debug-dump=frames /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko.debug

DWARF section [47] '.debug_frame' at offset 0x475ca8:

 [     0] CIE length=20
   CIE_id:                   18446744073709551615
   version:                  3
   augmentation:             ""
   code_alignment_factor:    1
   data_alignment_factor:    -8
   return_address_register:  16

   Program:
     def_cfa r7 (rsp) at offset 8
     offset_extended_sf r16 (rip) at cfa-8
     nop
     nop
     nop
     nop
     nop

 [    18] FDE length=52 cie=[     0]
   CIE_pointer:              0
   initial_location:         0x30
   address_range:            0x41

   Program:
     advance_loc4 1 to 0x1
     def_cfa_offset 16
     offset_extended_sf r6 (rbp) at cfa-16
     advance_loc4 3 to 0x4
     def_cfa_register r6 (rbp)
     advance_loc4 60 to 0x40
     restore r6 (rbp)
     def_cfa r7 (rsp) at offset 8
     nop
     nop
     nop
     nop
     nop
     nop

So our first FDE covers a location starting at 0x30 (covering these
first 0x41 bytes).

I cannot figure out why the FDE initial_location is shifted by 48 bytes.
If in the stap unwinder I just pretend the kernel module has been loaded
a little earlier:

-    vm_addr = s->static_addr;
+    vm_addr = s->static_addr - 0x30;

all the CFIs line up and I can unwind perfectly through the kernel
modules. But this mysterious magic 48 bytes bothers me. It is the same
on i686 btw. Any idea where they might be coming from?

Thanks,

Mark

Re: A 48 byte kernel module/cfi load mystery

[Mark Wielaard]

On Fri, 2010-07-02 at 13:23 +0200, Mark Wielaard wrote:
> I cannot figure out why the FDE initial_location is shifted by 48 bytes.
> If in the stap unwinder I just pretend the kernel module has been loaded
> a little earlier:
> 
> -    vm_addr = s->static_addr;
> +    vm_addr = s->static_addr - 0x30;
> 
> all the CFIs line up and I can unwind perfectly through the kernel
> modules. But this mysterious magic 48 bytes bothers me. It is the same
> on i686 btw. Any idea where they might be coming from?

Actually it isn't always the same. It was for that kernel on both x86_64
and i686. But with a recent fedora x86_64 kernel
(2.6.33.5-124.fc13.x86_64) the offset is always 0x24.

Cheers,

Mark

Re: A 48 byte kernel module/cfi load mystery

[Roland McGrath]

Kernel modules are relocatable objects, so those addresses you see are as
placed by libdwfl.  If you pass them to dwfl_module_relocate_address you
will get their section-relative offsets.

Let's take an example object:

	$ eu-readelf -S /usr/lib/debug/lib/modules/2.6.33.5-124.fc13.x86_64/kernel/fs/ext2/ext2.ko.debug
	There are 44 section headers, starting at offset 0x245418:

	Section Headers:
	[Nr] Name                 Type         Addr             Off      Size     ES Flags Lk Inf Al
	[ 0]                      NULL         0000000000000000 00000000 00000000  0        0   0  0
*	[ 1] .note.gnu.build-id   NOTE         0000000000000000 00000040 00000024  0 A      0   0  4
*	[ 2] .text                NOBITS       0000000000000000 00000064 00009c8c  0 AX     0   0  4
	<...>

Since the allocated section before it has size 0x24, that will be the
address where .text is placed.  Now let's look at the data:

	$ eu-readelf -r --debug-dump=frames /usr/lib/debug/lib/modules/2.6.33.5-124.fc13.x86_64/kernel/fs/ext2/ext2.ko.debug
	<...>
	Relocation section [33] '.rela.debug_frame' for section [32] '.debug_frame' at offset 0x1aa048 contains 288 entries:
	  Offset              Type            Value               Addend Name
	  0x000000000000001c  X86_64_32       000000000000000000      +0 .debug_frame
*	  0x0000000000000020  X86_64_64       000000000000000000      +0 .text
	  0x0000000000000054  X86_64_32       000000000000000000      +0 .debug_frame
	  0x0000000000000058  X86_64_64       000000000000000000     +93 .text
	<...>
	DWARF section [32] '.debug_frame' at offset 0x1a6fb8:

	 [     0] CIE length=20
	   CIE_id:                   18446744073709551615
	   version:                  3
	   augmentation:             ""
	   code_alignment_factor:    1
	   data_alignment_factor:    -8
	   return_address_register:  16

	   Program:
	     def_cfa r7 (rsp) at offset 8
	     offset_extended_sf r16 (rip) at cfa-8
	     nop
	     nop
	     nop
	     nop
	     nop

	 [    18] FDE length=52 cie=[     0]
	   CIE_pointer:              0
*	   initial_location:         0x24
	   address_range:            0x5d

	   Program:
	     advance_loc4 8 to 0x8
	     def_cfa_offset 16
	     offset_extended_sf r6 (rbp) at cfa-16
	     advance_loc4 3 to 0xb
	     def_cfa_register r6 (rbp)
	     advance_loc4 81 to 0x5c
	     restore r6 (rbp)
	     def_cfa r7 (rsp) at offset 8
	     nop
	     nop
	     nop
	     nop
	     nop
	     nop
	<...>

The decoded initial_location in the FDE at 0x18 is shown as 0x24.  
You can see from the reloc records that there is a reloc right there
(at 0x20, in the FDE header), for 0 bytes into the .text section.

I've just changed eu-readelf (for the eventual 0.149) to print FDE
addresses like it does others from DWARF, so (without -N) it shows:

	 [    18] FDE length=52 cie=[     0]
	   CIE_pointer:              0
	   initial_location:         .text+000000000000000000 <ext2_bg_has_super>
	   address_range:            0x5d

I hope that helps avoid future confusion about this.


Thanks,
Roland

Re: A 48 byte kernel module/cfi load mystery

[Mark Wielaard]

On Fri, 2010-07-02 at 11:19 -0700, Roland McGrath wrote:
> Kernel modules are relocatable objects, so those addresses you see are as
> placed by libdwfl.  If you pass them to dwfl_module_relocate_address you
> will get their section-relative offsets.

Thanks for the explanation. In retrospect it is all so obvious :)

> Let's take an example object:
> 
> 	$ eu-readelf -S /usr/lib/debug/lib/modules/2.6.33.5-124.fc13.x86_64/kernel/fs/ext2/ext2.ko.debug
> 	There are 44 section headers, starting at offset 0x245418:
> 
> 	Section Headers:
> 	[Nr] Name                 Type         Addr             Off      Size     ES Flags Lk Inf Al
> 	[ 0]                      NULL         0000000000000000 00000000 00000000  0        0   0  0
> *	[ 1] .note.gnu.build-id   NOTE         0000000000000000 00000040 00000024  0 A      0   0  4
> *	[ 2] .text                NOBITS       0000000000000000 00000064 00009c8c  0 AX     0   0  4
> 	<...>
> 
> Since the allocated section before it has size 0x24, that will be the
> address where .text is placed.

And the 0x48 on x86_64 was similar, but needed a bit more because of
address alignements.

> I've just changed eu-readelf (for the eventual 0.149) to print FDE
> addresses like it does others from DWARF, so (without -N) it shows:
> 
> 	 [    18] FDE length=52 cie=[     0]
> 	   CIE_pointer:              0
> 	   initial_location:         .text+000000000000000000 <ext2_bg_has_super>
> 	   address_range:            0x5d
> 
> I hope that helps avoid future confusion about this.

Yes, that would have made the issue pretty clear. Thanks.

I changed the systemtap translator to keep the .debug_frame FDE initial
locations (the synthetic .debug_frame_hdr) on a per section basis,
including a section load offset that the runtime unwinder uses to arrive
at the correct relative address.

For now, commit 4d83bd, only for the "magic sections", but will change
to track them accurately for all sections covered.

Cheers,

Mark