socket-trace.stp: no match while resolving probe point kernel.function("*@net/socket.c")

socket-trace.stp: no match while resolving probe point kernel.function("*@net/socket.c")

[Dongtao Liu]

Hi:

I install systemtap in my new Ubuntu 2.6.24-19 system and try to run
the examples in tutorial. The hello word and strace-open work well,
however the third example socket-trace has some problems. Here is the
output:

dliu@dliu-desktop:~/Documents$ sudo stap -e 'probe
kernel.function("sys_open") {log("hello world") exit()}'
hello world

dliu@dliu-desktop:~/Documents$ cat trace-open.stp
probe syscall.open
{
printf ("%s(%d) open (%s)\n", execname(), pid(), argstr)
}
probe timer.ms(4000)
{
exit()
}

dliu@dliu-desktop:~/Documents$ sudo stap trace-open.stp
hald-addon-stor(5293) open ("/dev/scd1", O_RDONLY|O_EXCL|O_LARGEFILE|O_
NONBLOCK)
hald-addon-stor(5290) open ("/dev/scd0", O_RDONLY|O_EXCL|O_LARGEFILE|O_NONBLOCK)
hald-addon-stor(5290) open ("/dev/scd0", O_RDONLY|O_EXCL|O_LARGEFILE|O_NONBLOCK)
hald-addon-stor(5293) open ("/dev/scd1", O_RDONLY|O_EXCL|O_LARGEFILE|O_NONBLOCK)

dliu@dliu-desktop:~/Documents$ cat socket-trace.stp
#! /usr/bin/env stap

probe kernel.function("*@net/socket.c").call {
printf ("%s -> %s\n", thread_indent(1), probefunc())
}
probe kernel.function("*@net/socket.c").return {
printf ("%s <- %s\n", thread_indent(-1), probefunc())
}

dliu@dliu-desktop:~/Documents$ stap socket-trace.stp

semantic error: no match while resolving probe point
kernel.function("*@net/socket.c").call
semantic error: no match while resolving probe point
kernel.function("*@net/socket.c").return
Pass 2: analysis failed. Try again with more '-v' (verbose) options.

dliu@dliu-desktop:~/Documents$ stap -vvv socket-trace.stp
SystemTap translator/driver (version 0.6/0.127 built 2007-12-04)
Copyright (C) 2005-2007 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
Created temporary directory "/tmp/stap7Tz4nm"
Searched '/usr/share/systemtap/tapset/i686/*.stp', found 1
Searched '/usr/share/systemtap/tapset/*.stp', found 37
Pass 1: parsed user script and 38 library script(s) in 290usr/0sys/292real ms.
control symbols: kts: 0xc031c760 kte: 0xc031f910 stext: 0xc0101000
parsed '*@net/socket.c' -> func '*', file 'net/socket.c'
focused on module 'kernel = [0xc0100000-0xc04f4000, bias 0x0] file
/lib/modules/2.6.24-19-generic/vmlinux ELF machine i?86 (code 3)
semantic error: no match while resolving probe point
kernel.function("*@net/socket.c").call
control symbols: kts: 0xc031c760 kte: 0xc031f910 stext: 0xc0101000
parsed '*@net/socket.c' -> func '*', file 'net/socket.c'
focused on module 'kernel = [0xc0100000-0xc04f4000, bias 0x0] file
/lib/modules/2.6.24-19-generic/vmlinux ELF machine i?86 (code 3)
semantic error: no match while resolving probe point
kernel.function("*@net/socket.c").return
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
global(s) in 220usr/100sys/322real ms.
Pass 2: analysis failed. Try again with more '-v' (verbose) options.
Running rm -rf /tmp/stap7Tz4nm


Could anyone help me to solve this problem? I am very new for linux
kernel. If need more information of the system, I can provide it.

Here is the my installation output:

dliu@dliu-desktop:~$ uname -a
Linux dliu-desktop 2.6.24-19-generic #1 SMP Wed Jun 18 14:43:41 UTC
2008 i686 GNU/Linux

dliu@dliu-desktop:~/Documents$ sudo apt-get install systemtap
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libelf1
The following NEW packages will be installed:
libelf1 systemtap
0 upgraded, 2 newly installed, 0 to remove and 152 not upgraded.
Need to get 786kB of archives.
After this operation, 2867kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://us.archive.ubuntu.com hardy/universe libelf1 0.131-3 [45.8kB]
Get:2 http://us.archive.ubuntu.com hardy/universe systemtap
0.0.20071201-1 [740kB]
Fetched 786kB in 1s (688kB/s)
Selecting previously deselected package libelf1.
(Reading database ... 95844 files and directories currently installed.)
Unpacking libelf1 (from .../libelf1_0.131-3_i386.deb) ...
Selecting previously deselected package systemtap.
Unpacking systemtap (from .../systemtap_0.0.20071201-1_i386.deb) ...
Setting up libelf1 (0.131-3) ...
Setting up systemtap (0.0.20071201-1) ...
Adding stapdev group...
Adding stapusr group...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place

dliu@dliu-desktop:~/Documents$ sudo apt-get install
linux-image-debug-2.6.24-19-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
linux-image-debug-2.6.24-19-generic
0 upgraded, 1 newly installed, 0 to remove and 152 not upgraded.
Need to get 26.2MB of archives.
After this operation, 64.4MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com hardy-updates/main
linux-image-debug-2.6.24-19-generic 2.6.24-19.41 [26.2MB]
Fetched 26.2MB in 9s (2871kB/s)
Selecting previously deselected package linux-image-debug-2.6.24-19-generic.
(Reading database ... 96046 files and directories currently installed.)
Unpacking linux-image-debug-2.6.24-19-generic (from
.../linux-image-debug-2.6.24-19-generic_2.6.24-19.41_i386.deb) ...
Setting up linux-image-debug-2.6.24-19-generic (2.6.24-19.41) ...

dliu@dliu-desktop:~/Documents$ uname -r
2.6.24-19-generic
dliu@dliu-desktop:~/Documents$ sudo ln -s /boot/vmlinux-debug-$(uname
-r) /lib/modules/$(uname -r)/vmlinux

dliu@dliu-desktop:~/Documents$ sudo apt-get install
linux-headers-2.6.24-19-generic gcc libcap-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
gcc is already the newest version.
The following extra packages will be installed:
libc6 libc6-dev libc6-i686 linux-libc-dev
Suggested packages:
glibc-doc manpages-dev
The following NEW packages will be installed:
libc6-dev libcap-dev linux-libc-dev
The following packages will be upgraded:
libc6 libc6-i686 linux-headers-2.6.24-19-generic
3 upgraded, 3 newly installed, 0 to remove and 149 not upgraded.
Need to get 10.3MB of archives.
After this operation, 17.3MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://us.archive.ubuntu.com hardy-updates/main libc6
2.7-10ubuntu4 [4307kB]
Get:2 http://us.archive.ubuntu.com hardy-updates/main libc6-i686
2.7-10ubuntu4 [1243kB]
Get:3 http://us.archive.ubuntu.com hardy-updates/main linux-libc-dev
2.6.24-21.42 [698kB]
Get:4 http://us.archive.ubuntu.com hardy-updates/main libc6-dev
2.7-10ubuntu4 [3344kB]
Get:5 http://us.archive.ubuntu.com hardy/main libcap-dev
1:1.10-14build1 [20.6kB]
Get:6 http://us.archive.ubuntu.com hardy-updates/main
linux-headers-2.6.24-19-generic 2.6.24-19.41 [644kB]
Fetched 10.3MB in 4s (2339kB/s)
Preconfiguring packages ...
(Reading database ... 96050 files and directories currently installed.)
Preparing to replace libc6 2.7-10ubuntu3 (using
.../libc6_2.7-10ubuntu4_i386.deb) ...
Unpacking replacement libc6 ...
Setting up libc6 (2.7-10ubuntu4) ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
(Reading database ... 96050 files and directories currently installed.)
Preparing to replace libc6-i686 2.7-10ubuntu3 (using
.../libc6-i686_2.7-10ubuntu4_i386.deb) ...
Unpacking replacement libc6-i686 ...
Selecting previously deselected package linux-libc-dev.
Unpacking linux-libc-dev (from .../linux-libc-dev_2.6.24-21.42_i386.deb) ...
Selecting previously deselected package libc6-dev.
Unpacking libc6-dev (from .../libc6-dev_2.7-10ubuntu4_i386.deb) ...
Selecting previously deselected package libcap-dev.
Unpacking libcap-dev (from .../libcap-dev_1%3a1.10-14build1_i386.deb) ...
Preparing to replace linux-headers-2.6.24-19-generic 2.6.24-19.34
(using .../linux-headers-2.6.24-19-generic_2.6.24-19.41_i386.deb) ...
Unpacking replacement linux-headers-2.6.24-19-generic ...
Setting up libc6-i686 (2.7-10ubuntu4) ...
Setting up linux-libc-dev (2.6.24-21.42) ...
Setting up libc6-dev (2.7-10ubuntu4) ...
Setting up libcap-dev (1:1.10-14build1) ...
Setting up linux-headers-2.6.24-19-generic (2.6.24-19.41) ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place

Re: socket-trace.stp: no match while resolving probe point kernel.function("*@net/socket.c")

[Mark Wielaard]

Hi Dongtao,

On Thu, 2008-10-23 at 23:00 -0400, Dongtao Liu wrote:
> I install systemtap in my new Ubuntu 2.6.24-19 system and try to run
> the examples in tutorial. The hello word and strace-open work well,
> however the third example socket-trace has some problems.

I don't immediately see why this is failing, and I don't have an ubuntu
install around. It seems you followed
http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu correctly. And it
is strange that trace-open.stp does work, but socket-trace.stp doesn't.

Are there other examples (http://sourceware.org/systemtap/examples/)
that do/don't work?

Maybe you can try running the testsuite (make installcheck would be
ideal) and/or upgrading to a newer systemtap (either the last release
0.7.3 or current git http://sourceware.org/systemtap/getinvolved.html)
to see if that gives more hints of what is going wrong in this case.

Cheers,

Mark

Re: socket-trace.stp: no match while resolving probe point kernel.function("*@net/socket.c")

[Dongtao Liu]

Hi:

Thank you for reply. I found it pretty strange that if I do not use
"*@net/socket.c" but use function in socket.c directly such as
sock_create. SystemTap can work well and trace all the calls of
fuction sock_create. I don't know why...

Dongtao

2008/10/24 Mark Wielaard <mjw@redhat.com>:
> Hi Dongtao,
>
> On Thu, 2008-10-23 at 23:00 -0400, Dongtao Liu wrote:
>> I install systemtap in my new Ubuntu 2.6.24-19 system and try to run
>> the examples in tutorial. The hello word and strace-open work well,
>> however the third example socket-trace has some problems.
>
> I don't immediately see why this is failing, and I don't have an ubuntu
> install around. It seems you followed
> http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu correctly. And it
> is strange that trace-open.stp does work, but socket-trace.stp doesn't.
>
> Are there other examples (http://sourceware.org/systemtap/examples/)
> that do/don't work?
>
> Maybe you can try running the testsuite (make installcheck would be
> ideal) and/or upgrading to a newer systemtap (either the last release
> 0.7.3 or current git http://sourceware.org/systemtap/getinvolved.html)
> to see if that gives more hints of what is going wrong in this case.
>
> Cheers,
>
> Mark
>
>

Re: socket-trace.stp: no match while resolving probe point kernel.function("*@net/socket.c")

[Mark Wielaard]

On Wed, 2008-10-29 at 21:41 -0400, Dongtao Liu wrote:
> Thank you for reply. I found it pretty strange that if I do not use
> "*@net/socket.c" but use function in socket.c directly such as
> sock_create. SystemTap can work well and trace all the calls of
> fuction sock_create. I don't know why...

That is indeed strange. So this works for you:
$ stap -e 'probe kernel.function("sock_sendmsg"){log(pp())}'
But this doesn't:
$ stap -e 'probe kernel.function("sock_sendmsg@net/socket.c"){log(pp())}'

Does the output (pp()) of the running version, or if you run it with
-vv, give any hints about the why?

Cheers,

Mark

Re: socket-trace.stp: no match while resolving probe point kernel.function("*@net/socket.c")

[Mark Wielaard]

On Mon, 2008-11-03 at 12:50 +0100, Mark Wielaard wrote:
> On Wed, 2008-10-29 at 21:41 -0400, Dongtao Liu wrote:
> > Thank you for reply. I found it pretty strange that if I do not use
> > "*@net/socket.c" but use function in socket.c directly such as
> > sock_create. SystemTap can work well and trace all the calls of
> > fuction sock_create. I don't know why...
> 
> That is indeed strange. So this works for you:
> $ stap -e 'probe kernel.function("sock_sendmsg"){log(pp())}'
> But this doesn't:
> $ stap -e 'probe kernel.function("sock_sendmsg@net/socket.c"){log(pp())}'
> 
> Does the output (pp()) of the running version, or if you run it with
> -vv, give any hints about the why?

I got another report of this issue, from srikar, but this time on SLES.
stap is 0.7.1 and elfutils 0.137, kernel 2.6.27.4-2-pae

When its not working it says:
  semantic error: libdw failure (dwarf_getsrcfiles): invalid DWARF
  semantic error: no match while resolving probe point kernel.function("sock_sendmsg@net/socket.c").call

When working:
  kernel.function("sock_sendmsg@net/socket.c:569")

So it looks like the file name is actually there, but libdw doesn't like
how it is encoded. I filed a new bug report for this:
http://sourceware.org/bugzilla/show_bug.cgi?id=7010

Cheers,

Mark