* A 48 byte kernel module/cfi load mystery
@ 2010-07-02 11:23 Mark Wielaard
2010-07-02 14:14 ` Mark Wielaard
0 siblings, 1 reply; 4+ messages in thread
From: Mark Wielaard @ 2010-07-02 11:23 UTC (permalink / raw)
To: roland; +Cc: systemtap
Hi Roland,
I was hoping you (or any other systemtap hacker of course) could help me
solve a mystery 48 byte offset I am seeing when using the CFI of kernel
modules. For some reason it looks like the FDE initial location is
always 48 bytes beyond the actual code that it covers. You can also see
this by hand. e.g:
$ eu-objdump --disassemble /lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko
/lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko: elf64-elf_x86_64
Disassembly of section .text:
0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: e8 00 00 00 00 callq 0x9
9: 48 8b 87 90 02 00 00 mov 0x290(%rdi),%rax
10: 49 89 d0 mov %rdx,%r8
13: 48 8b 50 60 mov 0x60(%rax),%rdx
17: 8b 78 10 mov 0x10(%rax),%edi
1a: 8b 52 14 mov 0x14(%rdx),%edx
1d: 48 29 d6 sub %rdx,%rsi
20: 48 85 c9 test %rcx,%rcx
23: 74 0a je 0x2f
25: 48 89 f0 mov %rsi,%rax
28: 31 d2 xor %edx,%edx
2a: 48 f7 f7 div %rdi
2d: 89 11 mov %edx,(%rcx)
2f: 4d 85 c0 test %r8,%r8
32: 74 0b je 0x3f
34: 48 89 f0 mov %rsi,%rax
37: 31 d2 xor %edx,%edx
39: 48 f7 f7 div %rdi
3c: 41 89 00 mov %eax,(%r8)
3f: c9 leaveq
40: c3 retq
Our first function starts at zero in the .text section and is 0x41 bytes
long. So look at the CFIs:
$ eu-readelf --debug-dump=frames /usr/lib/debug/lib/modules/2.6.32-37.el6.x86_64/kernel/fs/ext4/ext4.ko.debug
DWARF section [47] '.debug_frame' at offset 0x475ca8:
[ 0] CIE length=20
CIE_id: 18446744073709551615
version: 3
augmentation: ""
code_alignment_factor: 1
data_alignment_factor: -8
return_address_register: 16
Program:
def_cfa r7 (rsp) at offset 8
offset_extended_sf r16 (rip) at cfa-8
nop
nop
nop
nop
nop
[ 18] FDE length=52 cie=[ 0]
CIE_pointer: 0
initial_location: 0x30
address_range: 0x41
Program:
advance_loc4 1 to 0x1
def_cfa_offset 16
offset_extended_sf r6 (rbp) at cfa-16
advance_loc4 3 to 0x4
def_cfa_register r6 (rbp)
advance_loc4 60 to 0x40
restore r6 (rbp)
def_cfa r7 (rsp) at offset 8
nop
nop
nop
nop
nop
nop
So our first FDE covers a location starting at 0x30 (covering these
first 0x41 bytes).
I cannot figure out why the FDE initial_location is shifted by 48 bytes.
If in the stap unwinder I just pretend the kernel module has been loaded
a little earlier:
- vm_addr = s->static_addr;
+ vm_addr = s->static_addr - 0x30;
all the CFIs line up and I can unwind perfectly through the kernel
modules. But this mysterious magic 48 bytes bothers me. It is the same
on i686 btw. Any idea where they might be coming from?
Thanks,
Mark
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: A 48 byte kernel module/cfi load mystery
2010-07-02 11:23 A 48 byte kernel module/cfi load mystery Mark Wielaard
@ 2010-07-02 14:14 ` Mark Wielaard
2010-07-02 18:19 ` Roland McGrath
0 siblings, 1 reply; 4+ messages in thread
From: Mark Wielaard @ 2010-07-02 14:14 UTC (permalink / raw)
To: roland; +Cc: systemtap
On Fri, 2010-07-02 at 13:23 +0200, Mark Wielaard wrote:
> I cannot figure out why the FDE initial_location is shifted by 48 bytes.
> If in the stap unwinder I just pretend the kernel module has been loaded
> a little earlier:
>
> - vm_addr = s->static_addr;
> + vm_addr = s->static_addr - 0x30;
>
> all the CFIs line up and I can unwind perfectly through the kernel
> modules. But this mysterious magic 48 bytes bothers me. It is the same
> on i686 btw. Any idea where they might be coming from?
Actually it isn't always the same. It was for that kernel on both x86_64
and i686. But with a recent fedora x86_64 kernel
(2.6.33.5-124.fc13.x86_64) the offset is always 0x24.
Cheers,
Mark
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: A 48 byte kernel module/cfi load mystery
2010-07-02 14:14 ` Mark Wielaard
@ 2010-07-02 18:19 ` Roland McGrath
2010-07-05 19:35 ` Mark Wielaard
0 siblings, 1 reply; 4+ messages in thread
From: Roland McGrath @ 2010-07-02 18:19 UTC (permalink / raw)
To: Mark Wielaard; +Cc: systemtap
Kernel modules are relocatable objects, so those addresses you see are as
placed by libdwfl. If you pass them to dwfl_module_relocate_address you
will get their section-relative offsets.
Let's take an example object:
$ eu-readelf -S /usr/lib/debug/lib/modules/2.6.33.5-124.fc13.x86_64/kernel/fs/ext2/ext2.ko.debug
There are 44 section headers, starting at offset 0x245418:
Section Headers:
[Nr] Name Type Addr Off Size ES Flags Lk Inf Al
[ 0] NULL 0000000000000000 00000000 00000000 0 0 0 0
* [ 1] .note.gnu.build-id NOTE 0000000000000000 00000040 00000024 0 A 0 0 4
* [ 2] .text NOBITS 0000000000000000 00000064 00009c8c 0 AX 0 0 4
<...>
Since the allocated section before it has size 0x24, that will be the
address where .text is placed. Now let's look at the data:
$ eu-readelf -r --debug-dump=frames /usr/lib/debug/lib/modules/2.6.33.5-124.fc13.x86_64/kernel/fs/ext2/ext2.ko.debug
<...>
Relocation section [33] '.rela.debug_frame' for section [32] '.debug_frame' at offset 0x1aa048 contains 288 entries:
Offset Type Value Addend Name
0x000000000000001c X86_64_32 000000000000000000 +0 .debug_frame
* 0x0000000000000020 X86_64_64 000000000000000000 +0 .text
0x0000000000000054 X86_64_32 000000000000000000 +0 .debug_frame
0x0000000000000058 X86_64_64 000000000000000000 +93 .text
<...>
DWARF section [32] '.debug_frame' at offset 0x1a6fb8:
[ 0] CIE length=20
CIE_id: 18446744073709551615
version: 3
augmentation: ""
code_alignment_factor: 1
data_alignment_factor: -8
return_address_register: 16
Program:
def_cfa r7 (rsp) at offset 8
offset_extended_sf r16 (rip) at cfa-8
nop
nop
nop
nop
nop
[ 18] FDE length=52 cie=[ 0]
CIE_pointer: 0
* initial_location: 0x24
address_range: 0x5d
Program:
advance_loc4 8 to 0x8
def_cfa_offset 16
offset_extended_sf r6 (rbp) at cfa-16
advance_loc4 3 to 0xb
def_cfa_register r6 (rbp)
advance_loc4 81 to 0x5c
restore r6 (rbp)
def_cfa r7 (rsp) at offset 8
nop
nop
nop
nop
nop
nop
<...>
The decoded initial_location in the FDE at 0x18 is shown as 0x24.
You can see from the reloc records that there is a reloc right there
(at 0x20, in the FDE header), for 0 bytes into the .text section.
I've just changed eu-readelf (for the eventual 0.149) to print FDE
addresses like it does others from DWARF, so (without -N) it shows:
[ 18] FDE length=52 cie=[ 0]
CIE_pointer: 0
initial_location: .text+000000000000000000 <ext2_bg_has_super>
address_range: 0x5d
I hope that helps avoid future confusion about this.
Thanks,
Roland
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: A 48 byte kernel module/cfi load mystery
2010-07-02 18:19 ` Roland McGrath
@ 2010-07-05 19:35 ` Mark Wielaard
0 siblings, 0 replies; 4+ messages in thread
From: Mark Wielaard @ 2010-07-05 19:35 UTC (permalink / raw)
To: Roland McGrath; +Cc: systemtap
On Fri, 2010-07-02 at 11:19 -0700, Roland McGrath wrote:
> Kernel modules are relocatable objects, so those addresses you see are as
> placed by libdwfl. If you pass them to dwfl_module_relocate_address you
> will get their section-relative offsets.
Thanks for the explanation. In retrospect it is all so obvious :)
> Let's take an example object:
>
> $ eu-readelf -S /usr/lib/debug/lib/modules/2.6.33.5-124.fc13.x86_64/kernel/fs/ext2/ext2.ko.debug
> There are 44 section headers, starting at offset 0x245418:
>
> Section Headers:
> [Nr] Name Type Addr Off Size ES Flags Lk Inf Al
> [ 0] NULL 0000000000000000 00000000 00000000 0 0 0 0
> * [ 1] .note.gnu.build-id NOTE 0000000000000000 00000040 00000024 0 A 0 0 4
> * [ 2] .text NOBITS 0000000000000000 00000064 00009c8c 0 AX 0 0 4
> <...>
>
> Since the allocated section before it has size 0x24, that will be the
> address where .text is placed.
And the 0x48 on x86_64 was similar, but needed a bit more because of
address alignements.
> I've just changed eu-readelf (for the eventual 0.149) to print FDE
> addresses like it does others from DWARF, so (without -N) it shows:
>
> [ 18] FDE length=52 cie=[ 0]
> CIE_pointer: 0
> initial_location: .text+000000000000000000 <ext2_bg_has_super>
> address_range: 0x5d
>
> I hope that helps avoid future confusion about this.
Yes, that would have made the issue pretty clear. Thanks.
I changed the systemtap translator to keep the .debug_frame FDE initial
locations (the synthetic .debug_frame_hdr) on a per section basis,
including a section load offset that the runtime unwinder uses to arrive
at the correct relative address.
For now, commit 4d83bd, only for the "magic sections", but will change
to track them accurately for all sections covered.
Cheers,
Mark
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-07-05 19:35 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-07-02 11:23 A 48 byte kernel module/cfi load mystery Mark Wielaard
2010-07-02 14:14 ` Mark Wielaard
2010-07-02 18:19 ` Roland McGrath
2010-07-05 19:35 ` Mark Wielaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).