[Bug runtime/28378] New: [Systemtap] runtime error: Kernel function symbol table missing [man warning::symbols]

[Bug runtime/28378] New: [Systemtap] runtime error: Kernel function symbol table missing [man warning::symbols]

[nylon7717 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

            Bug ID: 28378
           Summary: [Systemtap] runtime error: Kernel function symbol
                    table missing [man warning::symbols]
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: nylon7717 at gmail dot com
  Target Milestone: ---

I use systemtap in Ubuntu-18.04


# git clone git://sourceware.org/git/systemtap.git 
# cd systemtap
# ./configure && make
# sudo make install
# sudo apt install linux-headers-$(uname -r)
# sudo apt install kernel-package
# sudo apt-get install linux-image-$(uname -r)-generic-dbgsym

## test
# sudo stap -e 'probe begin { printf("Hello, World!\n"); exit() }'
Hello, World!
# stap -L 'process("/lib/x86_64-linux-gnu/libc.so.6").function("malloc")'
process("/lib/x86_64-linux-gnu/libc-2.27.so").function("__libc_malloc@/build/glibc-S9d2JN/glibc-2.27/malloc/malloc.c:3037")
$bytes:size_t

## run my stap script
# stap -p4 -r $(uname -r) tracer.stp -m tracer -v -g --suppress-time-limits
WARNING: Kernel function symbol table missing [man warning::symbols]
Pass 1: parsed user script and 481 library scripts using
129828virt/93224res/5884shr/87548data kb, in 190usr/20sys/212real ms.
semantic error: while resolving probe point: identifier 'process' at
tracer.stp:1194:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("getenv").return {
                      ^

semantic error: process return probes not available [man error::inode-uprobes]

semantic error: while resolving probe point: identifier 'process' at :1203:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("unsetenv").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1212:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("setenv").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1222:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("putenv").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1231:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("getaddrinfo").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1238:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("getnameinfo").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1245:7
        source: probe
process("/lib/x86_64-linux-gnu/libc.so.6").function("gethostbyname").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1251:7
        source: probe
process("/lib64/libc.so.6").function("__strstr_sse42").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1257:7
        source: probe
process("/lib64/libc.so.6").function("__strcasestr_sse42").return {
                      ^

semantic error: while resolving probe point: identifier 'process' at :1263:7
        source: probe
process("/usr/lib/x86_64-linux-gnu/libcurl.so").function("curl_easy_perform").return
{
                      ^

semantic error: while resolving probe point: identifier 'process' at :1293:7
        source: probe process("/bin/bash").function("*_builtin").return {
                      ^

Pass 2: analyzed script: 988 probes, 921 functions, 107 embeds, 45 globals
using 443952virt/408624res/7420shr/401672data kb, in 2750usr/360sys/3490real
ms.
Pass 2: analysis failed.  [man error::pass2]


I can't find out any useful information in internet

I also try the other systemtap example, but didn't encounter the same
situation.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28378] semantic error: process return probes not available [man error::inode-uprobes]

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org
            Summary|[Systemtap] runtime error:  |semantic error: process
                   |Kernel function symbol      |return probes not available
                   |table missing [man          |[man error::inode-uprobes]
                   |warning::symbols]           |

--- Comment #1 from Mark Wielaard <mark at klomp dot org> ---
The real issue isn't the warning, but the error:

semantic error: process return probes not available [man error::inode-uprobes]

Hopefully the information given is useful.
Try: man error::inode-uprobes
Which should give you:
https://manpages.ubuntu.com/manpages/impish/man7/error::inode-uprobes.7stap.html

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28378] semantic error: process return probes not available [man error::inode-uprobes]

[nylon7717 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

--- Comment #2 from Nylon Chen <nylon7717 at gmail dot com> ---
Hi, Mark Wielaard 

thanks for your response.

I try to change the suggest and remove *.return because Ubuntu 18.04 kernel
version is 5.4.0

```
-probe process("/usr/bin/bash").function("*_builtin").return {
+probe process("/bin/bash").function("*_builtin") {
    if (in_call_tree(pid(), DONLY) == 1) {
        if(@defined($list)){
            printf("%s %d = %s(%s)\n", logstamp(0), $return, ppfunc(),
bash_builtin_arg[ppfunc()])
                delete bash_builtin_arg[ppfunc()]
        }
    }
}
```

But I still encounter some error.

# stap -p4 -r $(uname -r) tracer.stp -m tracer -v -g --suppress-time-limits
WARNING: Probe 'process' has been elided: keyword at :1294:1
 source: probe process("/bin/bash").function("*_builtin") {
         ^
semantic error: while processing probe tp_syscall.open.return from:
nd_syscall.open.return from: syscall.open.return from: syscall.open.return

semantic error: internal error: @entry is only valid in .return probes:
operator '@entry' at :652:32
        source:         filename = user_string(@entry($filename), err_msg)
                                               ^

May I have misunderstood your tip?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28378] semantic error: process return probes not available [man error::inode-uprobes]

[nylon7717 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

--- Comment #3 from Nylon Chen <nylon7717 at gmail dot com> ---
Hi Mark Wielaard:

If I want to use "process.statement()" replace "process.function().return"
-probe process("/usr/bin/bash").function("*_builtin").return

Would you give me some feedback?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28378] semantic error: process return probes not available [man error::inode-uprobes]

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

--- Comment #4 from Mark Wielaard <mark at klomp dot org> ---
I must say I don't really understand some of the issues you are seeing. But it
looks like you are using a couple of years old Ubuntu install. Maybe you can
ask them if they are still supporting that and can help you with using these
more modern systemtap features.

> # stap -p4 -r $(uname -r) tracer.stp -m tracer -v -g --suppress-time-limits
> WARNING: Probe 'process' has been elided: keyword at :1294:1
>  source: probe process("/bin/bash").function("*_builtin") {
>          ^

This is weird, it means stap believes this probe is empty. But what you show
above looks like it isn't. Do all if statements evaluate to false?

> semantic error: while processing probe tp_syscall.open.return from: nd_syscall.open.return from: syscall.open.return from: syscall.open.return
> 
> semantic error: internal error: @entry is only valid in .return probes: operator '@entry' at :652:32
>         source:         filename = user_string(@entry($filename), err_msg)
>                                                ^

This is even more curious. This seems to have nothing to do with the script you
posted. I cannot explain why the above expands to something that uses an
@entry.

Might there be more to the script than what you posted in this bug?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28378] semantic error: process return probes not available [man error::inode-uprobes]

[nylon7717 at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

--- Comment #5 from Nylon Chen <nylon7717 at gmail dot com> ---
Hi Mark Wielaard,

thanks for your help

When I use process.statement() or process.function replace 
process.function().return 

I can fix "error::inode-uprobes" issue

but my stap has other error
"semantic error: while processing probe tp_syscall.read.return from:
nd_syscall.read.return from: syscall.read.return from: syscall.read.return"

I try to reduce this stap script

```
...

probe syscall.write { write_fds[tid()] = fd }

probe syscall.write.return {
  p = pid()
  // Get fd even if non-dwarf syscall return probe.
  fd = write_fds[tid()]
  delete write_fds[tid()]
  bytes = retval
  time = gettimeofday_us() - @entry(gettimeofday_us())
  if (bytes > 0)
    filewrite[p, fd] <<< bytes
  time_io[p, fd] <<< time
}

...
```

the error message point out the problem is "syscall.xxx.return"

I try to comment (/**/) it, and the compiler's result is ok

I just curious about this result.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28378] semantic error: process return probes not available [man error::inode-uprobes]

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28378

--- Comment #6 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Nylon Chen from comment #5)
> but my stap has other error
> "semantic error: while processing probe tp_syscall.read.return from:
> nd_syscall.read.return from: syscall.read.return from: syscall.read.return"

Right, but that seems a totally other issue than you originally reported.
If you were using syscall probes then you really should have added that to the
original report. I suspect other issues you are seeing might also come from not
providing the full script you are using.

Might I again suggest you upgrade your system to something more modern and/or
contact Ubuntu support about the kernel/systemtap combination they are shipping
to make sure they are compatible.

> I try to reduce this stap script
> 
> ```
> ...
> 
> probe syscall.write { write_fds[tid()] = fd }
> 
> probe syscall.write.return {
>   p = pid()
>   // Get fd even if non-dwarf syscall return probe.
>   fd = write_fds[tid()]
>   delete write_fds[tid()]
>   bytes = retval
>   time = gettimeofday_us() - @entry(gettimeofday_us())
>   if (bytes > 0)
>     filewrite[p, fd] <<< bytes
>   time_io[p, fd] <<< time
> }
> 
> ...
> ```
> 
> the error message point out the problem is "syscall.xxx.return"
> 
> I try to comment (/**/) it, and the compiler's result is ok
> 
> I just curious about this result.

Right, since your systemtap/kernel doesn't support return probes you cannot use
@entry in your script. Since you already have an syscall entry probe I suggest
to record the time in the same way you record the write_fds.

-- 
You are receiving this mail because:
You are the assignee for the bug.