Question about kernel module debug

Question about kernel module debug

[Yuxin_Zhuang]

Hi,Systemtap users:

I've installed systemtap 0.5.8 on RHEL4 with kernel 2.6.9-42. I wanna
use systemtap to debug a kernel module which is not built into the
kernel and I got some questions.

1. Is the 'debuginfo' of the module needed by systemtap is just the
stuff created by gcc with option '-g'? Will an additional '-O2' make a
lot of difference?

2. I've put the '.ko' file which is built with options '-g -O2' under
the proper search path and created a symbol link with the suffix
'.ko.debug' to it. Before the systemtap is run, the module has been
loaded and works well. But systemtap reported that it can not find the
module's debuginfo. 
The output is: 

Pass 1: parsed user script and 25 library script(s) in
80usr/10sys/136real ms.
semantic error: cannot find module my_module debuginfo: r_offset is
bogus
semantic error: no match for probe point
while: resolving probe point module("my_module").function("*")
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 global(s) in
0usr/30sys/43real ms.
Pass 2: analysis failed.  Try again with more '-v' (verbose) options.

And I straced the stap, the output before printing the error is as
follows :

......
open("/usr/lib/systemtap/../lib/systemtap/libebl_i386.so", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/usr/lib/systemtap/libebl_i386.so", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`\6\0\000"...,
512) = 512
fstat64(5, {st_mode=S_IFREG|0755, st_size=7724, ...}) = 0
old_mmap(NULL, 10596, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5,
0) = 0xcf5000
old_mmap(0xcf7000, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x1000) = 0xcf7000
close(5)                                = 0
open("/lib/modules/2.6.9-42.ELsmp/kernel/fs/my_module/my_module.ko.debug
", O_RDONLY|O_LARGEFILE) = 5
fcntl64(5, F_GETFL)                     = 0x8000 (flags
O_RDONLY|O_LARGEFILE)
fstat64(5, {st_mode=S_IFREG|0755, st_size=2734531, ...}) = 0
mmap2(NULL, 2734531, PROT_READ|PROT_WRITE, MAP_PRIVATE, 5, 0) =
0xb589f000
open("/sys/module/my_module/sections/.text", O_RDONLY) = 6
fstat64(6, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb589e000
read(6, "0xd0e0c000\n", 4096)           = 11
read(6, "", 4096)                       = 0
close(6)                                = 0
munmap(0xb589e000, 4096)                = 0
open("/sys/module/my_module/sections/.rodata", O_RDONLY) = 6
fstat64(6, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb589e000
read(6, "0xd0e29c90\n", 4096)           = 11
read(6, "", 4096)                       = 0
close(6)                                = 0
munmap(0xb589e000, 4096)                = 0
write(2, "semantic error: ", 16semantic error: )        = 16
...... 

What can be the reason for the error?? 
Any suggestions will be appreciated, thanks a lot!


ZHUANG Yuxin

RE: Question about kernel module debug

[Yuxin_Zhuang]

Sorry, the module is compiled with just '-g', no '-O2'.

> _____________________________________________ 
> From: 	Yuxin, Zhuang  
> Sent:	2007年10月16日 11:09
> To:	systemtap@sources.redhat.com
> Cc:	Yuxin, Zhuang
> Subject:	Question about kernel module debug
> 
> Hi,Systemtap users:
> 
> I've installed systemtap 0.5.8 on RHEL4 with kernel 2.6.9-42. I wanna use systemtap to debug a kernel module which is not built into the kernel and I got some questions.
> 
> 1. Is the 'debuginfo' of the module needed by systemtap is just the stuff created by gcc with option '-g'? Will an additional '-O2' make a lot of difference?
> 
> 2. I've put the '.ko' file which is built with options '-g -O2' under the proper search path and created a symbol link with the suffix '.ko.debug' to it. Before the systemtap is run, the module has been loaded and works well. But systemtap reported that it can not find the module's debuginfo. 
> The output is: 
> 
> Pass 1: parsed user script and 25 library script(s) in 80usr/10sys/136real ms.
> semantic error: cannot find module my_module debuginfo: r_offset is bogus
> semantic error: no match for probe point
> while: resolving probe point module("my_module").function("*")
> Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 global(s) in 0usr/30sys/43real ms.
> Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
> 
> And I straced the stap, the output before printing the error is as follows :
> 
> …...
> open("/usr/lib/systemtap/../lib/systemtap/libebl_i386.so", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/usr/lib/systemtap/libebl_i386.so", O_RDONLY) = 5
> read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`\6\0\000"..., 512) = 512
> fstat64(5, {st_mode=S_IFREG|0755, st_size=7724, ...}) = 0
> old_mmap(NULL, 10596, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0xcf5000
> old_mmap(0xcf7000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x1000) = 0xcf7000
> close(5)                                = 0
> open("/lib/modules/2.6.9-42.ELsmp/kernel/fs/my_module/my_module.ko.debug", O_RDONLY|O_LARGEFILE) = 5
> fcntl64(5, F_GETFL)                     = 0x8000 (flags O_RDONLY|O_LARGEFILE)
> fstat64(5, {st_mode=S_IFREG|0755, st_size=2734531, ...}) = 0
> mmap2(NULL, 2734531, PROT_READ|PROT_WRITE, MAP_PRIVATE, 5, 0) = 0xb589f000
> open("/sys/module/my_module/sections/.text", O_RDONLY) = 6
> fstat64(6, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb589e000
> read(6, "0xd0e0c000\n", 4096)           = 11
> read(6, "", 4096)                       = 0
> close(6)                                = 0
> munmap(0xb589e000, 4096)                = 0
> open("/sys/module/my_module/sections/.rodata", O_RDONLY) = 6
> fstat64(6, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb589e000
> read(6, "0xd0e29c90\n", 4096)           = 11
> read(6, "", 4096)                       = 0
> close(6)                                = 0
> munmap(0xb589e000, 4096)                = 0
> write(2, "semantic error: ", 16semantic error: )        = 16
> …… 
> 
> What can be the reason for the error?? 
> Any suggestions will be appreciated, thanks a lot!
> 
> 
> ZHUANG Yuxin

Re: Question about kernel module debug

[Frank Ch. Eigler]

Yuxin_Zhuang@emc.com writes:

> I've installed systemtap 0.5.8 on RHEL4 with kernel 2.6.9-42. 

This is over a year old.  If you can, please upgrade to a later
systemtap build.  RHN should have 0.5.12 or even 0.5.14 by now.

> [...]  1. Is the 'debuginfo' of the module needed by systemtap is
> just the stuff created by gcc with option '-g'?

Yes.

> Will an additional '-O2' make a lot of difference?

It will make the debugging information somewhat less precise, giving
you fewer useful probe points.

> 2. I've put the '.ko' file which is built with options '-g -O2' under
> the proper search path and created a symbol link with the suffix
> '.ko.debug' to it. [...]

There should be no need for the .ko.debug file if the basic .ko file
was not stripped.  What does "readelf -S module.ko" say about .debug_*
sections?

- FChE

RE: Question about kernel module debug

[Yuxin_Zhuang]

Thanks for your reply !

The result of 'readelf -S'  is as follows and the .debug_* sections are already there:

****************************
There are 52 section headers, starting at offset 0x2084ec:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        00000000 000040 01db74 00  AX  0   0 16
  [ 2] .rel.text         REL             00000000 208d0c 00b6d0 08     50   1  4
  [ 3] .fixup            PROGBITS        00000000 01dbb4 00004a 00  AX  0   0  1
  [ 4] .rel.fixup        REL             00000000 2143dc 000038 08     50   3  4
  [ 5] .altinstr_replace PROGBITS        00000000 01dbfe 00007b 00  AX  0   0  1
  [ 6] .exit.text        PROGBITS        00000000 01dc7c 000053 00  AX  0   0  4
  [ 7] .rel.exit.text    REL             00000000 214414 000070 08     50   6  4
  [ 8] .init.text        PROGBITS        00000000 01dcd0 000250 00  AX  0   0  4
  [ 9] .rel.init.text    REL             00000000 214484 0001c8 08     50   8  4
  [10] .rodata           PROGBITS        00000000 01df20 001059 00   A  0   0  4
  [11] .rel.rodata       REL             00000000 21464c 0003c0 08     50  10  4
  [12] .rodata.str1.1    PROGBITS        00000000 01ef79 00182f 01 AMS  0   0  1
  [13] .rodata.str1.4    PROGBITS        00000000 0207a8 006015 01 AMS  0   0  4
  [14] __ex_table        PROGBITS        00000000 0267c0 000038 00   A  0   0  4
  [15] .rel__ex_table    REL             00000000 214a0c 000070 08     50  14  4
  [16] .altinstructions  PROGBITS        00000000 0267f8 0001eb 00   A  0   0  4
  [17] .rel.altinstructi REL             00000000 214a7c 000290 08     50  16  4
  [18] __param           PROGBITS        00000000 0269e4 000154 00   A  0   0  4
  [19] .rel__param       REL             00000000 214d0c 000220 08     50  18  4
  [20] .modinfo          PROGBITS        00000000 026b40 000602 00   A  0   0 32
  [21] __versions        PROGBITS        00000000 027144 000000 00   A  0   0  4
  [22] .data             PROGBITS        00000000 027160 000af0 00  WA  0   0 32
  [23] .rel.data         REL             00000000 214f2c 000698 08     50  22  4
  [24] .gnu.linkonce.thi PROGBITS        00000000 027c80 001180 00  WA  0   0 128
  [25] .rel.gnu.linkonce REL             00000000 2155c4 000010 08     50  24  4
  [26] .bss              NOBITS          00000000 028e00 041ad0 00  WA  0   0 32
  [27] .stab             PROGBITS        00000000 028e00 001338 0c     29   0  4
  [28] .rel.stab         REL             00000000 2155d4 000028 08     50  27  4
  [29] .stabstr          STRTAB          00000000 02a138 00438c 00      0   0  1
  [30] .comment          PROGBITS        00000000 02e4c4 00057f 00      0   0  1
  [31] .debug_aranges    PROGBITS        00000000 02ea43 000370 00      0   0  1
  [32] .rel.debug_arange REL             00000000 2155fc 0001c0 08     50  31  4
  [33] .debug_pubnames   PROGBITS        00000000 02edb3 001c3c 00      0   0  1
  [34] .rel.debug_pubnam REL             00000000 2157bc 0000e8 08     50  33  4
  [35] .debug_info       PROGBITS        00000000 0309ef 105d1d 00      0   0  1
  [36] .rel.debug_info   REL             00000000 2158a4 07a810 08     50  35  4
  [37] .debug_abbrev     PROGBITS        00000000 13670c 0077fd 00      0   0  1
  [38] .debug_line       PROGBITS        00000000 13df09 011d32 00      0   0  1
  [39] .rel.debug_line   REL             00000000 2900b4 0000f0 08     50  38  4
  [40] .debug_frame      PROGBITS        00000000 14fc3c 00ae0c 00      0   0  4
  [41] .rel.debug_frame  REL             00000000 2901a4 002150 08     50  40  4
  [42] .debug_str        PROGBITS        00000000 15aa48 07a8d7 01  MS  0   0  1
  [43] .debug_loc        PROGBITS        00000000 1d531f 02f82d 00      0   0  1
  [44] .rel.debug_loc    REL             00000000 2922f4 000ca0 08     50  43  4
  [45] .note             NOTE            00000000 204b4c 000014 00      0   0  1
  [46] .debug_ranges     PROGBITS        00000000 204b60 0037c8 00      0   0  1
  [47] .rel.debug_ranges REL             00000000 292f94 000200 08     50  46  4
  [48] .note.GNU-stack   PROGBITS        00000000 208328 000000 00   X  0   0  1
  [49] .shstrtab         STRTAB          00000000 208328 0001c3 00      0   0  1
  [50] .symtab           SYMTAB          00000000 293194 004ad0 10     51 710  4
  [51] .strtab           STRTAB          00000000 297c64 003d5f 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)
********************************

But the output of systemtap says:

*********************************
Pass 1: parsed user script and 25 library script(s) in
80usr/10sys/136real ms.
semantic error: cannot find module my_module debuginfo: r_offset is
bogus
semantic error: no match for probe point
while: resolving probe point module("my_module").function("*")
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 global(s) in
0usr/30sys/43real ms.
Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
*********************************

What can be the reason ?



-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org] On Behalf Of Frank Ch. Eigler
Sent: 2007年10月16日 23:58
To: Yuxin, Zhuang
Cc: systemtap@sources.redhat.com
Subject: Re: Question about kernel module debug

Yuxin_Zhuang@emc.com writes:

> I've installed systemtap 0.5.8 on RHEL4 with kernel 2.6.9-42. 

This is over a year old.  If you can, please upgrade to a later
systemtap build.  RHN should have 0.5.12 or even 0.5.14 by now.

> [...]  1. Is the 'debuginfo' of the module needed by systemtap is
> just the stuff created by gcc with option '-g'?

Yes.

> Will an additional '-O2' make a lot of difference?

It will make the debugging information somewhat less precise, giving
you fewer useful probe points.

> 2. I've put the '.ko' file which is built with options '-g -O2' under
> the proper search path and created a symbol link with the suffix
> '.ko.debug' to it. [...]

There should be no need for the .ko.debug file if the basic .ko file
was not stripped.  What does "readelf -S module.ko" say about .debug_*
sections?

- FChE

RE: Question about kernel module debug

[Yuxin_Zhuang]

What do you mean by " There should be no need for the .ko.debug file if the basic .ko file was not stripped. "??
Does it means that systemtap will get the debug information from the .ko file and will not look for the .ko.debug file on the three search paths??
But when the .ko.debug symlink is removed, the result of strace shows that without finding the '.ko.debug' file on the three search paths systemtap just fails. Saying 'r_offset is bogus'.

-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org] On Behalf Of Frank Ch. Eigler
Sent: 2007年10月16日 23:58
To: Yuxin, Zhuang
Cc: systemtap@sources.redhat.com
Subject: Re: Question about kernel module debug

Yuxin_Zhuang@emc.com writes:

> I've installed systemtap 0.5.8 on RHEL4 with kernel 2.6.9-42. 

This is over a year old.  If you can, please upgrade to a later
systemtap build.  RHN should have 0.5.12 or even 0.5.14 by now.

> [...]  1. Is the 'debuginfo' of the module needed by systemtap is
> just the stuff created by gcc with option '-g'?

Yes.

> Will an additional '-O2' make a lot of difference?

It will make the debugging information somewhat less precise, giving
you fewer useful probe points.

> 2. I've put the '.ko' file which is built with options '-g -O2' under
> the proper search path and created a symbol link with the suffix
> '.ko.debug' to it. [...]

There should be no need for the .ko.debug file if the basic .ko file
was not stripped.  What does "readelf -S module.ko" say about .debug_*
sections?

- FChE

Re: Question about kernel module debug

[Frank Ch. Eigler]

Yuxin_Zhuang@emc.com writes:

> What do you mean by " There should be no need for the .ko.debug file
> if the basic .ko file was not stripped. "??  Does it means that
> systemtap will get the debug information from the .ko file and will
> not look for the .ko.debug file on the three search paths??

That's right.

> But when the .ko.debug symlink is removed, the result of strace
> shows that without finding the '.ko.debug' file on the three search
> paths systemtap just fails. Saying 'r_offset is bogus'.

That's not right (elfutils error DWFL_E_BADRELOFF).  It's as if
something has corrupted the data.  Please describe the toolchain
version, architecture, and anything else that may be unusual about the
build process.

- FChE

RE: Question about kernel module debug

[Yuxin_Zhuang]

Configuration of my environment:
Machine architecture:	i686
OS: 	RHEL4 with kernel 2.6.9-42.Elsmp
Systemtap version:	0.6
Elfutils :	0.129
GCC:	3.4.6

The content of the script
**************************
probe module("my_module").function("*")
{
	printf( "In the module\n");
}
**************************

The output is as follows:
**************************
Pass 1: parsed user script and 25 library script(s) in 80usr/10sys/136real ms.
semantic error: cannot find module my_module debuginfo: r_offset is bogus
semantic error: no match for probe point
while: resolving probe point module("my_module").function("*")
Pass 2: analyzed script: 2 probe(s), 0 function(s), 0 global(s) in 0usr/30sys/43real ms.
Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
**************************

And the functions in kernel and modules built with the kernel can be probed with no problem.
The module 'my_module' has been built with the debug information not stripped. But systemtap is still looking for a '.ko.debug' file.

Thanks !
ZHUANG Yuxin

-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org] On Behalf Of Frank Ch. Eigler
Sent: 2007年10月19日 2:22
To: Yuxin, Zhuang
Cc: systemtap@sources.redhat.com
Subject: Re: Question about kernel module debug

Yuxin_Zhuang@emc.com writes:

> What do you mean by " There should be no need for the .ko.debug file
> if the basic .ko file was not stripped. "??  Does it means that
> systemtap will get the debug information from the .ko file and will
> not look for the .ko.debug file on the three search paths??

That's right.

> But when the .ko.debug symlink is removed, the result of strace
> shows that without finding the '.ko.debug' file on the three search
> paths systemtap just fails. Saying 'r_offset is bogus'.

That's not right (elfutils error DWFL_E_BADRELOFF).  It's as if
something has corrupted the data.  Please describe the toolchain
version, architecture, and anything else that may be unusual about the
build process.

- FChE

RE: Question about kernel module debug

[Yuxin_Zhuang]

I've also got a newer version of systemtap ( 0.6) with elfutils version
0.129. But it doesn't work either.