Location of modules with debug info

Location of modules with debug info

[Roberto Jung Drebes]

Hello there,

I'm trying to run a simple probe in Ununtu 9.04. I already installed  
the linux-image-debug package with the debug information, so I do have  
the vmlinux and modules with debug info on /usr/lib/debug/lib/modules/ 
2.6.28-15-generic. I also made a symbolic link for the vmlinux at /boot:

[root@rerun modules]# ls -l /boot/vmlinux
lrwxrwxrwx 1 root root 45 2009-09-28 16:28 /boot/vmlinux -> /usr/lib/ 
debug/boot/vmlinux-2.6.28-15-generic

The probe I want to run just prints a message on every function entry  
and exit of a kernel module:

probe module("snd").function("*") {
	printf ("%s -> %s\n", thread_indent(1), probefunc())
}
probe module("snd").function("*").return {
	printf ("%s -> %s\n", thread_indent(-1), probefunc())
}

But when I try to run it, it complains that it can't find the debug  
info for the snd.ko module:

WARNING: cannot find module snd debuginfo: No such file or directory
semantic error: no match while resolving probe point module 
("snd").function("*")
WARNING: cannot find module snd debuginfo: No such file or directory
semantic error: no match while resolving probe point module 
("snd").function("*").return
semantic error: no probes found

When running it with the --vp 03 option, I see that it's trying to use  
the modules at /lib/modules/2.6.28-15-generic (which are the system  
modules), not the debug ones in (/usr/lib/debug/lib/modules/2.6.28-15- 
generic).

How can I indicate the location of the debug modules?

Thanks in advance,

-- 
Roberto Jung Drebes - Tokyo, Japan
<drebes@gmail.com>
http://japao.drebes.org/

Re: Location of modules with debug info

[Frank Ch. Eigler]

Roberto Jung Drebes <drebes@gmail.com> writes:

> [...]
> I'm trying to run a simple probe in Ununtu 9.04. I already installed
> the linux-image-debug package with the debug information, so I do have
> the vmlinux and modules with debug info on /usr/lib/debug/lib/modules/
> 2.6.28-15-generic. [...]

Where did you get that one?  It doesn't appear to be in the standard
9.04 repositories.
(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/289087)


> The probe I want to run just prints a message on every function entry
> and exit of a kernel module:
>
> probe module("snd").function("*") {
> 	printf ("%s -> %s\n", thread_indent(1), probefunc())
> }

(You will want to add a ".call" at the end of that probe point.)

> probe module("snd").function("*").return {
> 	printf ("%s -> %s\n", thread_indent(-1), probefunc())
> }

Right.

> WARNING: cannot find module snd debuginfo: No such file or directory
> semantic error: no match while resolving probe point module
> ("snd").function("*") [...]

I seem to recall that older linux-image-debug widgets from ubuntu
contained only vmlinux and not any module data.  If that's true for
your package too, this would explain the above error.


- FChE

Re: Location of modules with debug info

[Roberto Jung Drebes]

On 28/09/2009, at 23:43, Frank Ch. Eigler wrote:

> Roberto Jung Drebes <drebes@gmail.com> writes:
>
>> [...]
>> I'm trying to run a simple probe in Ununtu 9.04. I already installed
>> the linux-image-debug package with the debug information, so I do  
>> have
>> the vmlinux and modules with debug info on /usr/lib/debug/lib/ 
>> modules/
>> 2.6.28-15-generic. [...]
>
> Where did you get that one?  It doesn't appear to be in the standard
> 9.04 repositories.
> (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/289087)

I had to recompile my own packages changing skipdbg=true to  
skipdbg=false.

>>
>> probe module("snd").function("*") {
>> 	printf ("%s -> %s\n", thread_indent(1), probefunc())
>> }
>
> (You will want to add a ".call" at the end of that probe point.)

Will try that. I had just copied it from

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/html/SystemTap_Beginners_Guide/scripts.html

Maybe an old documentation.

>>
>> WARNING: cannot find module snd debuginfo: No such file or directory
>> semantic error: no match while resolving probe point module
>> ("snd").function("*") [...]
>
> I seem to recall that older linux-image-debug widgets from ubuntu
> contained only vmlinux and not any module data.  If that's true for
> your package too, this would explain the above error.

This is what I have:

[root@rerun ~]# find /lib/modules/2.6.28-15-generic/ -name snd.ko |  
xargs ls -l
-rw-r--r-- 1 root root 114496 2009-09-28 15:08 /lib/modules/2.6.28-15- 
generic/kernel/sound/core/snd.ko
[root@rerun ~]# find /usr/lib/debug/lib/modules/2.6.28-15-generic/ - 
name snd.ko | xargs ls -l
-rw-r--r-- 1 root root 1604291 2009-09-28 15:12 /usr/lib/debug/lib/ 
modules/2.6.28-15-generic/kernel/sound/core/snd.ko

So I guess stap is just trying to use the first one, without even  
looking for the second. Is there any way I can specify the location?

Thanks again,

Regards,

-- 
Roberto Jung Drebes - Tokyo, Japan
<drebes@gmail.com>
http://japao.drebes.org/

Re: Location of modules with debug info

[Frank Ch. Eigler]

Roberto Jung Drebes <drebes@gmail.com> writes:

> [...]
>> Where did you get that one?  It doesn't appear to be in the standard
>> 9.04 repositories.
>> (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/289087)
>
> I had to recompile my own packages changing skipdbg=true to
> skipdbg=false.

OK, I'm doing a similar run to reproduce the problem here. 
(Next time you rebuild, consider applying the utrace patches too:
http://people.redhat.com/roland/utrace/old/2.6.28/.)


> [...]
> [root@rerun ~]# find /lib/modules/2.6.28-15-generic/ -name snd.ko |
> xargs ls -l
> -rw-r--r-- 1 root root 114496 2009-09-28 15:08 /lib/modules/2.6.28-15-
> generic/kernel/sound/core/snd.ko
> [root@rerun ~]# find /usr/lib/debug/lib/modules/2.6.28-15-generic/ -
> name snd.ko | xargs ls -l
> -rw-r--r-- 1 root root 1604291 2009-09-28 15:12 /usr/lib/debug/lib/
> modules/2.6.28-15-generic/kernel/sound/core/snd.ko

> So I guess stap is just trying to use the first one, without even
> looking for the second. Is there any way I can specify the location?

Hm, it seems as though the /usr/lib/debug copy is the original
unstripped one, and snd.ko is simply stripped.  In Fedora land, we're
more used to separated debuginfo (so /usr/lib/debug/lib/modules/.../
ends up containing snd.ko.debug -- with only the .debug_* sections
included).

The stripped file probably doesn't have a ".gnu_debuglink" section
(confirm with readelf -S), so elfutils doesn't know that it should
bother looking in /usr/lib/debug for an alternative version.
(Roland?)

Regardless, if the /usr/lib/debug/lib copy of the module.ko is indeed
complete (it has all the .text etc. sections), you could just
overwrite the stripped copies in /lib/modules, and all should work.

- FChE

Re: Location of modules with debug info

[Roberto Jung Drebes]

On 29/09/2009, at 00:54, Frank Ch. Eigler wrote:

>>
>> So I guess stap is just trying to use the first one, without even
>> looking for the second. Is there any way I can specify the location?
>
> Hm, it seems as though the /usr/lib/debug copy is the original
> unstripped one, and snd.ko is simply stripped.  In Fedora land, we're
> more used to separated debuginfo (so /usr/lib/debug/lib/modules/.../
> ends up containing snd.ko.debug -- with only the .debug_* sections
> included).
>
> The stripped file probably doesn't have a ".gnu_debuglink" section
> (confirm with readelf -S), so elfutils doesn't know that it should
> bother looking in /usr/lib/debug for an alternative version.
> (Roland?)
>
> Regardless, if the /usr/lib/debug/lib copy of the module.ko is indeed
> complete (it has all the .text etc. sections), you could just
> overwrite the stripped copies in /lib/modules, and all should work.

I replaced the stripped module with the non-stripped one, and indeed  
my probe then works. If there's no way of specifying the alternate  
path, I think I'll just overwrite all modules with the debug ones.

Thanks for the help!

-- 
Roberto Jung Drebes - Tokyo, Japan
<drebes@gmail.com>
http://japao.drebes.org/

Re: Location of modules with debug info

[Frank Ch. Eigler]

Roberto Jung Drebes <drebes@gmail.com> writes:

> [...]
> I replaced the stripped module with the non-stripped one, and indeed
> my probe then works. 

Great.

> If there's no way of specifying the alternate path [...]

There is the DEBUGINFO_PATH environment variable, but the problem is
that elfutils is not even looking for debuginfo for these modules,
since once it found the plain stripped module, it didn't think there's
any hope to keep looking.

We could tweak this logic.

- FChE