"No match for probe point" for syscall.open with tutorial example strace-open.stp

"No match for probe point" for syscall.open with tutorial example strace-open.stp

[Fredy Neeser]

Hi,

I am trying to use systemtap (20070818 snapshot) on an IA32 Core Duo system with
Fedora Core 5 and a kernel which I built from vanilla 2.6.18 sources.

For systemtap installation, I followed the README on
http://sources.redhat.com/cgi-bin/cvsweb.cgi/~checkout~//src/README?cvsroot=systemtap
and also observed
http://sourceware.org/systemtap/wiki/SystemTapWithSelfBuiltKernel

Installation went fine after I had applied the fix for the patched elfutils
reported at
	http://blog.gmane.org/gmane.linux.systemtap/day=20070629

The
        stap -vv hello-world.stp

example from the tutorial

        http://sourceware.org/systemtap/tutorial/

works just fine, but the next tutorial example

        [root@mythen stap_tests]# stap -vvv strace-open.stp

gives "No match for probe point" for syscall.open:

SystemTap translator/driver (version 0.6/0.128 built 2007-08-23)
Copyright (C) 2005-2007 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
Created temporary directory "/tmp/stapyFwODp"
Searched '/usr/local/share/systemtap/tapset/i686/*.stp', found 1
Searched '/usr/local/share/systemtap/tapset/*.stp', found 36
Searched '/usr/local/share/systemtap/tapset/LKET/*.stp', found 19
Pass 1: parsed user script and 56 library script(s) in 320usr/10sys/356real ms.
control symbols: kts: 0x0 kte: 0x0 stext: 0x0
parsed 'sys_open' -> func 'sys_open'
focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] file
/boot/vmlinux-2.6.18-nfd1 ELF machine i?86 (code 3)
control symbols: kts: 0x0 kte: 0x0 stext: 0x0
parsed 'compat_sys_open' -> func 'compat_sys_open'
focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] file
/boot/vmlinux-2.6.18-nfd1 ELF machine i?86 (code 3)
control symbols: kts: 0x0 kte: 0x0 stext: 0x0
parsed 'sys32_open' -> func 'sys32_open'
focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] file
/boot/vmlinux-2.6.18-nfd1 ELF machine i?86 (code 3)
semantic error: no match for probe point while resolving probe point syscall.open
Pass 2: analyzed script: 1 probe(s), 6 function(s), 0 embed(s), 0 global(s) in
340usr/1990sys/64170real ms.
Pass 2: analysis failed.  Try again with more '-v' (verbose) options.
Running rm -rf /tmp/stapyFwODp


My /boot/vmlinux-2.6.18-nfd1 is apparently found.
The kernel was built with CONFIG_DEBUG_INFO, CONFIG_KPROBES, CONFIG_RELAY and
CONFIG_DEBUG_FS set to 'y'.

The same problem occurs also when I boot into the newer 2.6.22.1 vanilla kernel.
Where could the problem be?
Any idea what to check?

Many thanks,
Fredy.


PS: The example

        cat test.stp
        probe kernel.function("*@net/socket.c") { }
        probe kernel.function("*@net/socket.c").return { }
        stap -vvv test.stp

shows the same problem:

...
Searched '/usr/local/share/systemtap/tapset/i686/*.stp', found 1
Searched '/usr/local/share/systemtap/tapset/*.stp', found 36
Searched '/usr/local/share/systemtap/tapset/LKET/*.stp', found 19
Pass 1: parsed user script and 56 library script(s) in 320usr/0sys/335real ms.
control symbols: kts: 0x0 kte: 0x0 stext: 0x0
parsed '*@net/socket.c' -> func '*', file 'net/socket.c'
focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] file
/boot/vmlinux-2.6.18-nfd1 ELF machine i?86 (code 3)
semantic error: no match for probe point while resolving probe point
kernel.function("*@net/socket.c")
...

Re: "No match for probe point" for syscall.open with tutorial example strace-open.stp

[David Smith]

Fredy Neeser wrote:
> Hi,
> 
> I am trying to use systemtap (20070818 snapshot) on an IA32 Core Duo system with
> Fedora Core 5 and a kernel which I built from vanilla 2.6.18 sources.
> 

(Note that I've never used systemtap with a self-built kernel, but I'll 
take a stab at this anyway.)

... stuff deleted ...

> the next tutorial example
> 
>         [root@mythen stap_tests]# stap -vvv strace-open.stp
> 
> gives "No match for probe point" for syscall.open:
> 
> SystemTap translator/driver (version 0.6/0.128 built 2007-08-23)
> Copyright (C) 2005-2007 Red Hat, Inc. and others
> This is free software; see the source for copying conditions.
> Created temporary directory "/tmp/stapyFwODp"
> Searched '/usr/local/share/systemtap/tapset/i686/*.stp', found 1
> Searched '/usr/local/share/systemtap/tapset/*.stp', found 36
> Searched '/usr/local/share/systemtap/tapset/LKET/*.stp', found 19
> Pass 1: parsed user script and 56 library script(s) in 320usr/10sys/356real ms.
> control symbols: kts: 0x0 kte: 0x0 stext: 0x0

This is very curious.  those 3 control symbols shouldn't be 0: kts is 
'__kprobes_text_start', kte is '__kprobes_text_end', and stext is 
'_stext'.  Can you grep for those symbols in /proc/kallsyms and see if 
you find them?

If you can't find /proc/kallsyms, you need to build your kernel with 
CONFIG_KALLSYMS.  If you can find those symbols in /proc/kallsyms, I'd 
bet systemtap can't find your kernel's debuginfo.

Hope this helps.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: "No match for probe point" for syscall.open with tutorial example strace-open.stp

[Fredy Neeser]

David Smith schrieb:
> Fredy Neeser wrote:
>> Hi,
>>
>> I am trying to use systemtap (20070818 snapshot) on an IA32 Core Duo 
>> system with
>> Fedora Core 5 and a kernel which I built from vanilla 2.6.18 sources.
>>
>
> (Note that I've never used systemtap with a self-built kernel, but 
> I'll take a stab at this anyway.)
>
> ... stuff deleted ...
>
>> the next tutorial example
>>
>>         [root@mythen stap_tests]# stap -vvv strace-open.stp
>>
>> gives "No match for probe point" for syscall.open:
>>
>> SystemTap translator/driver (version 0.6/0.128 built 2007-08-23)
>> Copyright (C) 2005-2007 Red Hat, Inc. and others
>> This is free software; see the source for copying conditions.
>> Created temporary directory "/tmp/stapyFwODp"
>> Searched '/usr/local/share/systemtap/tapset/i686/*.stp', found 1
>> Searched '/usr/local/share/systemtap/tapset/*.stp', found 36
>> Searched '/usr/local/share/systemtap/tapset/LKET/*.stp', found 19
>> Pass 1: parsed user script and 56 library script(s) in 
>> 320usr/10sys/356real ms.
>> control symbols: kts: 0x0 kte: 0x0 stext: 0x0
>
> This is very curious.  those 3 control symbols shouldn't be 0: kts is 
> '__kprobes_text_start', kte is '__kprobes_text_end', and stext is 
> '_stext'.  Can you grep for those symbols in /proc/kallsyms and see if 
> you find them?
>
> If you can't find /proc/kallsyms, you need to build your kernel with 
> CONFIG_KALLSYMS.  If you can find those symbols in /proc/kallsyms, I'd 
> bet systemtap can't find your kernel's debuginfo.
>
> Hope this helps.
>
Thanks for your thoughts.  The kernel is configured with
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_EXTRA_PASS=y

The symbols are there:
[fn@mythen ~]$ cat /proc/kallsyms | grep __kprobes_text_start
c02edf20 T __kprobes_text_start
[fn@mythen ~]$ cat /proc/kallsyms | grep __kprobes_text_end
c02efb87 T __kprobes_text_end
[fn@mythen ~]$ cat /proc/kallsyms | grep _stext
c0100294 T _stext

Hmm ... I will also try on another machine.

Cheers,
Fredy.

Re: "No match for probe point" for syscall.open with tutorial example strace-open.stp

[Frank Ch. Eigler]

Fredy Neeser <Fredy.Neeser@solnet.ch> writes:

> I am trying to use systemtap (20070818 snapshot) on an IA32 Core Duo system with
> Fedora Core 5 and a kernel which I built from vanilla 2.6.18 sources.

Thank you for your high-quality problem report!

Other than dsmith's comment, there's another telltale problem:

> [...]
> focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] file /boot/vmlinux-2.6.18-nfd1 ELF machine i?86 (code 3)
> control symbols: kts: 0x0 kte: 0x0 stext: 0x0
> parsed 'sys_open' -> func 'sys_open'
> focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] [...]

Before that second "focused" line, you should be seeing something like:

pattern 'sys_open' matches function 'sys_open'
selected function sys_open
probe sys_open@fs/open.c:977 kernel section=.text pc=0xc0471088

Can you do a smell test of your vmlinux file:

% ls -l /boot/vmlinux*
% readelf -S /boot/vmlinux* | grep -i debug 

- FChE

Re:

[Fredy Neeser]

Frank Ch. Eigler <fche <at> redhat.com> writes:

> 
> Fredy Neeser <Fredy.Neeser <at> solnet.ch> writes:
> 
> > I am trying to use systemtap (20070818 snapshot) on an IA32 Core Duo
> > system with Fedora Core 5 and a kernel which I built from vanilla
> > 2.6.18 sources.
> 
> Thank you for your high-quality problem report!
> 
> Other than dsmith's comment, there's another telltale problem:
> 
> > [...]
> > focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] file
> > /boot/vmlinux-2.6.18-nfd1 ELF machine i?86 (code 3)
> > control symbols: kts: 0x0 kte: 0x0 stext: 0x0
> > parsed 'sys_open' -> func 'sys_open'
> > focused on module 'kernel = [0x100000-0x2bd504, bias 0x0] [...]
> 
> Before that second "focused" line, you should be seeing something like:
> 
> pattern 'sys_open' matches function 'sys_open'
> selected function sys_open
> probe sys_open <at> fs/open.c:977 kernel section=.text pc=0xc0471088
> 
> Can you do a smell test of your vmlinux file:
> 
> % ls -l /boot/vmlinux*
> % readelf -S /boot/vmlinux* | grep -i debug 
> 
> - FChE
> 
> 

Thanks!

Your reaction made me suspicious of having the wrong vmlinux file.
Bingo, we have two of them, one under directory
<my-kernel>/arch/i386/boot/compressed
and the other (the much larger one, i.e., the right one ...)
under just <my-kernel> ...

Great, SystemTap is working now for me :-)

The symptom mentioned by dsmith is gone and I see
"matches function 'sys_open'" as you said above.

Since it's quite easy to confuse the two vmlinux versions (unless
one looks at their sizes ...), an update in the README on how to
make systemtap aware of kernel debug info (for a hand-built kernel)
would probably be helpful.

Let me draft something that I can send you ...

Thanks again,
FN

PS: Just for the records, the wrong vmlinux gives

% ls -l arch/i386/boot/compressed/vmlinux
-rwxrwxr-x 1 fn fn 1798025 23. Jul 12:58 arch/i386/boot/compressed/vmlinux

% readelf -S arch/i386/boot/compressed/vmlinux | grep -i debug
  [ 7] .debug_aranges    PROGBITS    00000000 1b10bd 000020 00      0   0  1
  [ 8] .debug_pubnames   PROGBITS    00000000 1b10dd 000061 00      0   0  1
  [ 9] .debug_info       PROGBITS    00000000 1b113e 0019d2 00      0   0  1
  [10] .debug_abbrev     PROGBITS    00000000 1b2b10 000411 00      0   0  1
  [11] .debug_line       PROGBITS    00000000 1b2f21 0005d0 00      0   0  1
  [12] .debug_frame      PROGBITS    00000000 1b34f4 0001c4 00      0   0  4
  [13] .debug_str        PROGBITS    00000000 1b36b8 00041f 01  MS  0   0  1
  [14] .debug_loc        PROGBITS    00000000 1b3ad7 0022c0 00      0   0  1
  [15] .debug_ranges     PROGBITS    00000000 1b5d97 000680 00      0   0  1

while the right vmlinux gives
% ls -l vmlinux
-rwxrwxr-x 1 fn fn 42883188 23. Jul 12:23 vmlinux

% readelf -S vmlinux | grep -i debug
  [34] .debug_aranges    PROGBITS    00000000 37260d 008980 00      0   0  1
  [35] .debug_pubnames   PROGBITS    00000000 37af8d 02c222 00      0   0  1
  [36] .debug_info       PROGBITS    00000000 3a71af 1dc21c0 00      0   0  1
  [37] .debug_abbrev     PROGBITS    00000000 216936f 0f1d3c 00      0   0  1
  [38] .debug_line       PROGBITS    00000000 225b0ab 19a111 00      0   0  1
  [39] .debug_frame      PROGBITS    00000000 23f51bc 076c20 00      0   0  4
  [40] .debug_str        PROGBITS    00000000 246bddc 0c10d1 01  MS  0   0  1
  [41] .debug_loc        PROGBITS    00000000 252cead 244ea1 00      0   0  1
  [42] .debug_ranges     PROGBITS    00000000 2771d4e 062200 00      0   0  1

Re:

[David Smith]

On Thu, Jul 27, 2017 at 9:47 AM, Eyal Yehuda via systemtap
<systemtap@sourceware.org> wrote:

... stuff deleted ...

> Please advice
> 1.
>  what should i do in addition to solve this issue as I must create an
> interaction between systemtap script and another kernel module
> 2.
>  In case this is not feasible can u please offer another alternative to
> make systemtap script use another kernel module services ?

I'm not 100% sure, but I believe you need to copy your module with
foo() to the /lib/modules/UNAME/kernel directory before trying to
build the systemtap module.

I don't think this really is a systemtap problem, I believe you'd have
the same problem with a 2nd kernel module that tried to access foo().

Give that a shot and let us know if this fixes your problem.

-- 
David Smith
Principal Software Engineer
Red Hat

Re:

[Arkady]

STAP runtime flag "-d MODULE" can potentially expose the external
symbols in the module. I have never tried to use it.

On Sun, Jul 30, 2017 at 10:26 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> See 6.3 in https://www.kernel.org/doc/Documentation/kbuild/modules.txt
>  for intricate details. The bottom line is that STAP builds modules in
> a specific environment which can not be easily (or at least I do not
> know how to) controlled.
>
> On Sun, Jul 30, 2017 at 10:17 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> This link can be relevant
>> https://stackoverflow.com/questions/33718358/insmod-fails-with-unknown-symbol-in-module-for-a-symbol-defined-in-another-mod
>>
>> On Sun, Jul 30, 2017 at 10:09 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>>> Daniel,
>>>
>>> STAP attempts to compile the kernel module against the kernel source
>>> tree installed in the system. One way to resolve the issue is to
>>> update the kernel source tree installed in the system. Without
>>> updating the kernel sources the compilation will fail.
>>>
>>> On Sun, Jul 30, 2017 at 9:56 AM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> How is systemtap's kernel module different from other modules?
>>>> Given module A which exports a function and module B using that exported
>>>> function and A is insmodded before B, I have not seen any problem. Why is
>>>> Systemtap's KO different? Is it not the kernel's job to resolve the exported
>>>> symbol?
>>>>
>>>>
>>>> Daniel.
>>>>
>>>> On Thu, 27 Jul 2017 at 19:29 Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>>
>>>>> P.S.1 There is yet another way - very cool way - to to add a symbol to
>>>>> the stap script - via /sys/kernel/debug/systemtap/<module name>/.cmd
>>>>> (CTL_CHANNEL_NAME below)
>>>>>
>>>>> This is a Python sample
>>>>>
>>>>> def write_relocate_string(f, s, size):
>>>>>     f.write(s)
>>>>>     padding = size - len(s)
>>>>>     for _ in range(padding):
>>>>>         f.write('\x00')
>>>>>
>>>>> def write_relocate(module_name, symbol_name, address):
>>>>>     STP_MODULE_NAME_LEN = 128
>>>>>     STP_SYMBOL_NAME_LEN = 128
>>>>>
>>>>>     with open(CTL_CHANNEL_NAME, "wb") as f:
>>>>>     # STAP_RELOCATE (9)
>>>>>         f.write(struct.pack('<I', 9))
>>>>>         write_relocate_string(f, module_name, STP_MODULE_NAME_LEN)
>>>>>         write_relocate_string(f, symbol_name, STP_SYMBOL_NAME_LEN)
>>>>>         f.write(struct.pack('<Q', address))
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 27, 2017 at 7:20 PM, Arkady <arkady.miasnikov@gmail.com>
>>>>> wrote:
>>>>> > P.S. Another idea is to use kernel IPC instead of function calls.
>>>>> > Files, named pipes, sockets, memory mapped files - just about anything
>>>>> > can do the trick.
>>>>> >
>>>>> > On Thu, Jul 27, 2017 at 7:03 PM, Arkady <arkady.miasnikov@gmail.com>
>>>>> > wrote:
>>>>> >> Eyal,
>>>>> >>
>>>>> >> Utility stap employs the kernel debug information installed in the
>>>>> >> system. When you load a module command insmod does not update the
>>>>> >> kernel debug info. There are simple two ways to resolve the issue
>>>>> >> *  Load the module, get the address of the function foo using
>>>>> >> /proc/kallsyms, provide the address of the foo as an argument to the
>>>>> >> STAP script.
>>>>> >> *  Link the module statically with the kernel and update the
>>>>> >> kernel-debuginfo
>>>>> >>
>>>>> >> Arkady
>>>>> >>
>>>>> >> On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
>>>>> >> <systemtap@sourceware.org> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> hello
>>>>> >>>
>>>>> >>> The
>>>>> >>>  issue I am facing is of undefined symbols in system tap kernel module
>>>>> >>> when trying to link to another kernel module with exported symbol
>>>>> >>>
>>>>> >>> please note environment setup:
>>>>> >>> os:  debian 9 stretch 64bit
>>>>> >>> kernel version:  4.9.0-3-amd64
>>>>> >>>
>>>>> >>>
>>>>> >>>           I created the following simple systemtap script using guru
>>>>> >>> mode
>>>>> >>>
>>>>> >>>           system tap script:      temp.stp
>>>>> >>>
>>>>> >>>
>>>>> >>>          %{
>>>>> >>>                  extern int foo(void);
>>>>> >>>          %}
>>>>> >>>
>>>>> >>>           function call_foo:long() %{/* pure */
>>>>> >>>                   STAP_RETVALUE = foo();
>>>>> >>>          %}
>>>>> >>>
>>>>> >>>           probe syscall.open {
>>>>> >>>                  printf("foo() : %d\n",  call_foo())
>>>>> >>>           }
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>  In addition to that I created the following simple kernel module with
>>>>> >>> an exported symbol function to be used from inside system tap
>>>>> >>> script
>>>>> >>>
>>>>> >>>          kernel module:       xxx.c
>>>>> >>>
>>>>> >>>          #include <linux/kernel.h>
>>>>> >>>          #include <linux/module.h>
>>>>> >>>          #include <linux/printk.h>
>>>>> >>>
>>>>> >>>          static int foo(void){
>>>>> >>>                  pr_info("foo()\n"); return 0;
>>>>> >>>          }
>>>>> >>>          EXPORT_SYMBOL(foo);
>>>>> >>>
>>>>> >>>          static int __init xxx_init(void){
>>>>> >>>                 pr_info("xxx_init()\n");
>>>>> >>>            return 0;
>>>>> >>>          }
>>>>> >>>
>>>>> >>>          static void __exit xxx_exit(void){
>>>>> >>>                 pr_info("xxx_exit() \n");
>>>>> >>>          }
>>>>> >>>
>>>>> >>>          module_init(xxx_init);
>>>>> >>>          module_exit(xxx_exit);
>>>>> >>>          MODULE_LICENSE("GPL");
>>>>> >>>
>>>>> >>>
>>>>> >>>          After compiling the kernel module and insmoding it, I run the
>>>>> >>> command to compile systemtap script as follow:
>>>>> >>>          stap -gk -v temp.stp
>>>>> >>>
>>>>> >>>          the compilation command result with the following error:
>>>>> >>>
>>>>> >>>
>>>>> >>>  Pass 1: parsed user script and 465 library scripts using
>>>>> >>> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>>>>> >>>
>>>>> >>>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
>>>>> >>> using 165912virt/100212res/7716shr/92564data kb, in
>>>>> >>> 680usr/50sys/731real
>>>>> >>>  ms.
>>>>> >>>
>>>>> >>>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
>>>>> >>> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>>>>> >>>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>>>>> >>>          Pass 4: compiled C into "stap_1837.ko" in
>>>>> >>> 1410usr/120sys/1963real ms.
>>>>> >>>          Pass 5: starting run.
>>>>> >>>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko':
>>>>> >>> Invalid parameters
>>>>> >>>          WARNING: /usr/bin/staprun exited with status: 1
>>>>> >>>          Pass 5: run completed in 0usr/0sys/16real ms.
>>>>> >>>          Pass 5: run failed.  [man error::pass5]
>>>>> >>>          Tip: /usr/share/doc/systemtap/README.Debian should help you
>>>>> >>> get started.
>>>>> >>>          Keeping temporary directory "/tmp/stapzKUZ11"
>>>>> >>>
>>>>> >>>
>>>>> >>>          in dmesg I can see the following lines
>>>>> >>>
>>>>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no
>>>>> >>> symbol version for foo
>>>>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230:
>>>>> >>> Unknown symbol foo (err -22)
>>>>> >>>
>>>>> >>>
>>>>> >>>          I also checked /proc/kallsyms
>>>>> >>>
>>>>> >>>
>>>>> >>>          cat /proc/kallsyms | grep xxx
>>>>> >>>          ffffffffc0584000 T foo    [xxx]
>>>>> >>>          ffffffffc0584014 t xxx_exit    [xxx]
>>>>> >>>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>>>>> >>>          ffffffffc058506e r __kstrtab_foo    [xxx]
>>>>> >>>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>>>>> >>>          ffffffffc0586000 d __this_module    [xxx]
>>>>> >>>          ffffffffc0584014 t cleanup_module    [xxx]
>>>>> >>>
>>>>> >>>
>>>>> >>> Please advice
>>>>> >>> 1.
>>>>> >>>  what should i do in addition to solve this issue as I must create an
>>>>> >>> interaction between systemtap script and another kernel module
>>>>> >>> 2.
>>>>> >>>  In case this is not feasible can u please offer another alternative
>>>>> >>> to
>>>>> >>> make systemtap script use another kernel module services ?
>>>>> >>>
>>>>> >>> Thanks eyal yehuda
>>>>>
>>>>

Re:

[Arkady]

See 6.3 in https://www.kernel.org/doc/Documentation/kbuild/modules.txt
 for intricate details. The bottom line is that STAP builds modules in
a specific environment which can not be easily (or at least I do not
know how to) controlled.

On Sun, Jul 30, 2017 at 10:17 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> This link can be relevant
> https://stackoverflow.com/questions/33718358/insmod-fails-with-unknown-symbol-in-module-for-a-symbol-defined-in-another-mod
>
> On Sun, Jul 30, 2017 at 10:09 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Daniel,
>>
>> STAP attempts to compile the kernel module against the kernel source
>> tree installed in the system. One way to resolve the issue is to
>> update the kernel source tree installed in the system. Without
>> updating the kernel sources the compilation will fail.
>>
>> On Sun, Jul 30, 2017 at 9:56 AM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
>>> Hi,
>>>
>>> How is systemtap's kernel module different from other modules?
>>> Given module A which exports a function and module B using that exported
>>> function and A is insmodded before B, I have not seen any problem. Why is
>>> Systemtap's KO different? Is it not the kernel's job to resolve the exported
>>> symbol?
>>>
>>>
>>> Daniel.
>>>
>>> On Thu, 27 Jul 2017 at 19:29 Arkady <arkady.miasnikov@gmail.com> wrote:
>>>>
>>>> P.S.1 There is yet another way - very cool way - to to add a symbol to
>>>> the stap script - via /sys/kernel/debug/systemtap/<module name>/.cmd
>>>> (CTL_CHANNEL_NAME below)
>>>>
>>>> This is a Python sample
>>>>
>>>> def write_relocate_string(f, s, size):
>>>>     f.write(s)
>>>>     padding = size - len(s)
>>>>     for _ in range(padding):
>>>>         f.write('\x00')
>>>>
>>>> def write_relocate(module_name, symbol_name, address):
>>>>     STP_MODULE_NAME_LEN = 128
>>>>     STP_SYMBOL_NAME_LEN = 128
>>>>
>>>>     with open(CTL_CHANNEL_NAME, "wb") as f:
>>>>     # STAP_RELOCATE (9)
>>>>         f.write(struct.pack('<I', 9))
>>>>         write_relocate_string(f, module_name, STP_MODULE_NAME_LEN)
>>>>         write_relocate_string(f, symbol_name, STP_SYMBOL_NAME_LEN)
>>>>         f.write(struct.pack('<Q', address))
>>>>
>>>>
>>>>
>>>> On Thu, Jul 27, 2017 at 7:20 PM, Arkady <arkady.miasnikov@gmail.com>
>>>> wrote:
>>>> > P.S. Another idea is to use kernel IPC instead of function calls.
>>>> > Files, named pipes, sockets, memory mapped files - just about anything
>>>> > can do the trick.
>>>> >
>>>> > On Thu, Jul 27, 2017 at 7:03 PM, Arkady <arkady.miasnikov@gmail.com>
>>>> > wrote:
>>>> >> Eyal,
>>>> >>
>>>> >> Utility stap employs the kernel debug information installed in the
>>>> >> system. When you load a module command insmod does not update the
>>>> >> kernel debug info. There are simple two ways to resolve the issue
>>>> >> *  Load the module, get the address of the function foo using
>>>> >> /proc/kallsyms, provide the address of the foo as an argument to the
>>>> >> STAP script.
>>>> >> *  Link the module statically with the kernel and update the
>>>> >> kernel-debuginfo
>>>> >>
>>>> >> Arkady
>>>> >>
>>>> >> On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
>>>> >> <systemtap@sourceware.org> wrote:
>>>> >>>
>>>> >>>
>>>> >>> hello
>>>> >>>
>>>> >>> The
>>>> >>>  issue I am facing is of undefined symbols in system tap kernel module
>>>> >>> when trying to link to another kernel module with exported symbol
>>>> >>>
>>>> >>> please note environment setup:
>>>> >>> os:  debian 9 stretch 64bit
>>>> >>> kernel version:  4.9.0-3-amd64
>>>> >>>
>>>> >>>
>>>> >>>           I created the following simple systemtap script using guru
>>>> >>> mode
>>>> >>>
>>>> >>>           system tap script:      temp.stp
>>>> >>>
>>>> >>>
>>>> >>>          %{
>>>> >>>                  extern int foo(void);
>>>> >>>          %}
>>>> >>>
>>>> >>>           function call_foo:long() %{/* pure */
>>>> >>>                   STAP_RETVALUE = foo();
>>>> >>>          %}
>>>> >>>
>>>> >>>           probe syscall.open {
>>>> >>>                  printf("foo() : %d\n",  call_foo())
>>>> >>>           }
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>  In addition to that I created the following simple kernel module with
>>>> >>> an exported symbol function to be used from inside system tap
>>>> >>> script
>>>> >>>
>>>> >>>          kernel module:       xxx.c
>>>> >>>
>>>> >>>          #include <linux/kernel.h>
>>>> >>>          #include <linux/module.h>
>>>> >>>          #include <linux/printk.h>
>>>> >>>
>>>> >>>          static int foo(void){
>>>> >>>                  pr_info("foo()\n"); return 0;
>>>> >>>          }
>>>> >>>          EXPORT_SYMBOL(foo);
>>>> >>>
>>>> >>>          static int __init xxx_init(void){
>>>> >>>                 pr_info("xxx_init()\n");
>>>> >>>            return 0;
>>>> >>>          }
>>>> >>>
>>>> >>>          static void __exit xxx_exit(void){
>>>> >>>                 pr_info("xxx_exit() \n");
>>>> >>>          }
>>>> >>>
>>>> >>>          module_init(xxx_init);
>>>> >>>          module_exit(xxx_exit);
>>>> >>>          MODULE_LICENSE("GPL");
>>>> >>>
>>>> >>>
>>>> >>>          After compiling the kernel module and insmoding it, I run the
>>>> >>> command to compile systemtap script as follow:
>>>> >>>          stap -gk -v temp.stp
>>>> >>>
>>>> >>>          the compilation command result with the following error:
>>>> >>>
>>>> >>>
>>>> >>>  Pass 1: parsed user script and 465 library scripts using
>>>> >>> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>>>> >>>
>>>> >>>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
>>>> >>> using 165912virt/100212res/7716shr/92564data kb, in
>>>> >>> 680usr/50sys/731real
>>>> >>>  ms.
>>>> >>>
>>>> >>>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
>>>> >>> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>>>> >>>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>>>> >>>          Pass 4: compiled C into "stap_1837.ko" in
>>>> >>> 1410usr/120sys/1963real ms.
>>>> >>>          Pass 5: starting run.
>>>> >>>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko':
>>>> >>> Invalid parameters
>>>> >>>          WARNING: /usr/bin/staprun exited with status: 1
>>>> >>>          Pass 5: run completed in 0usr/0sys/16real ms.
>>>> >>>          Pass 5: run failed.  [man error::pass5]
>>>> >>>          Tip: /usr/share/doc/systemtap/README.Debian should help you
>>>> >>> get started.
>>>> >>>          Keeping temporary directory "/tmp/stapzKUZ11"
>>>> >>>
>>>> >>>
>>>> >>>          in dmesg I can see the following lines
>>>> >>>
>>>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no
>>>> >>> symbol version for foo
>>>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230:
>>>> >>> Unknown symbol foo (err -22)
>>>> >>>
>>>> >>>
>>>> >>>          I also checked /proc/kallsyms
>>>> >>>
>>>> >>>
>>>> >>>          cat /proc/kallsyms | grep xxx
>>>> >>>          ffffffffc0584000 T foo    [xxx]
>>>> >>>          ffffffffc0584014 t xxx_exit    [xxx]
>>>> >>>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>>>> >>>          ffffffffc058506e r __kstrtab_foo    [xxx]
>>>> >>>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>>>> >>>          ffffffffc0586000 d __this_module    [xxx]
>>>> >>>          ffffffffc0584014 t cleanup_module    [xxx]
>>>> >>>
>>>> >>>
>>>> >>> Please advice
>>>> >>> 1.
>>>> >>>  what should i do in addition to solve this issue as I must create an
>>>> >>> interaction between systemtap script and another kernel module
>>>> >>> 2.
>>>> >>>  In case this is not feasible can u please offer another alternative
>>>> >>> to
>>>> >>> make systemtap script use another kernel module services ?
>>>> >>>
>>>> >>> Thanks eyal yehuda
>>>>
>>>

Re:

[Arkady]

This link can be relevant
https://stackoverflow.com/questions/33718358/insmod-fails-with-unknown-symbol-in-module-for-a-symbol-defined-in-another-mod

On Sun, Jul 30, 2017 at 10:09 AM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Daniel,
>
> STAP attempts to compile the kernel module against the kernel source
> tree installed in the system. One way to resolve the issue is to
> update the kernel source tree installed in the system. Without
> updating the kernel sources the compilation will fail.
>
> On Sun, Jul 30, 2017 at 9:56 AM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
>> Hi,
>>
>> How is systemtap's kernel module different from other modules?
>> Given module A which exports a function and module B using that exported
>> function and A is insmodded before B, I have not seen any problem. Why is
>> Systemtap's KO different? Is it not the kernel's job to resolve the exported
>> symbol?
>>
>>
>> Daniel.
>>
>> On Thu, 27 Jul 2017 at 19:29 Arkady <arkady.miasnikov@gmail.com> wrote:
>>>
>>> P.S.1 There is yet another way - very cool way - to to add a symbol to
>>> the stap script - via /sys/kernel/debug/systemtap/<module name>/.cmd
>>> (CTL_CHANNEL_NAME below)
>>>
>>> This is a Python sample
>>>
>>> def write_relocate_string(f, s, size):
>>>     f.write(s)
>>>     padding = size - len(s)
>>>     for _ in range(padding):
>>>         f.write('\x00')
>>>
>>> def write_relocate(module_name, symbol_name, address):
>>>     STP_MODULE_NAME_LEN = 128
>>>     STP_SYMBOL_NAME_LEN = 128
>>>
>>>     with open(CTL_CHANNEL_NAME, "wb") as f:
>>>     # STAP_RELOCATE (9)
>>>         f.write(struct.pack('<I', 9))
>>>         write_relocate_string(f, module_name, STP_MODULE_NAME_LEN)
>>>         write_relocate_string(f, symbol_name, STP_SYMBOL_NAME_LEN)
>>>         f.write(struct.pack('<Q', address))
>>>
>>>
>>>
>>> On Thu, Jul 27, 2017 at 7:20 PM, Arkady <arkady.miasnikov@gmail.com>
>>> wrote:
>>> > P.S. Another idea is to use kernel IPC instead of function calls.
>>> > Files, named pipes, sockets, memory mapped files - just about anything
>>> > can do the trick.
>>> >
>>> > On Thu, Jul 27, 2017 at 7:03 PM, Arkady <arkady.miasnikov@gmail.com>
>>> > wrote:
>>> >> Eyal,
>>> >>
>>> >> Utility stap employs the kernel debug information installed in the
>>> >> system. When you load a module command insmod does not update the
>>> >> kernel debug info. There are simple two ways to resolve the issue
>>> >> *  Load the module, get the address of the function foo using
>>> >> /proc/kallsyms, provide the address of the foo as an argument to the
>>> >> STAP script.
>>> >> *  Link the module statically with the kernel and update the
>>> >> kernel-debuginfo
>>> >>
>>> >> Arkady
>>> >>
>>> >> On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
>>> >> <systemtap@sourceware.org> wrote:
>>> >>>
>>> >>>
>>> >>> hello
>>> >>>
>>> >>> The
>>> >>>  issue I am facing is of undefined symbols in system tap kernel module
>>> >>> when trying to link to another kernel module with exported symbol
>>> >>>
>>> >>> please note environment setup:
>>> >>> os:  debian 9 stretch 64bit
>>> >>> kernel version:  4.9.0-3-amd64
>>> >>>
>>> >>>
>>> >>>           I created the following simple systemtap script using guru
>>> >>> mode
>>> >>>
>>> >>>           system tap script:      temp.stp
>>> >>>
>>> >>>
>>> >>>          %{
>>> >>>                  extern int foo(void);
>>> >>>          %}
>>> >>>
>>> >>>           function call_foo:long() %{/* pure */
>>> >>>                   STAP_RETVALUE = foo();
>>> >>>          %}
>>> >>>
>>> >>>           probe syscall.open {
>>> >>>                  printf("foo() : %d\n",  call_foo())
>>> >>>           }
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>  In addition to that I created the following simple kernel module with
>>> >>> an exported symbol function to be used from inside system tap
>>> >>> script
>>> >>>
>>> >>>          kernel module:       xxx.c
>>> >>>
>>> >>>          #include <linux/kernel.h>
>>> >>>          #include <linux/module.h>
>>> >>>          #include <linux/printk.h>
>>> >>>
>>> >>>          static int foo(void){
>>> >>>                  pr_info("foo()\n"); return 0;
>>> >>>          }
>>> >>>          EXPORT_SYMBOL(foo);
>>> >>>
>>> >>>          static int __init xxx_init(void){
>>> >>>                 pr_info("xxx_init()\n");
>>> >>>            return 0;
>>> >>>          }
>>> >>>
>>> >>>          static void __exit xxx_exit(void){
>>> >>>                 pr_info("xxx_exit() \n");
>>> >>>          }
>>> >>>
>>> >>>          module_init(xxx_init);
>>> >>>          module_exit(xxx_exit);
>>> >>>          MODULE_LICENSE("GPL");
>>> >>>
>>> >>>
>>> >>>          After compiling the kernel module and insmoding it, I run the
>>> >>> command to compile systemtap script as follow:
>>> >>>          stap -gk -v temp.stp
>>> >>>
>>> >>>          the compilation command result with the following error:
>>> >>>
>>> >>>
>>> >>>  Pass 1: parsed user script and 465 library scripts using
>>> >>> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>>> >>>
>>> >>>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
>>> >>> using 165912virt/100212res/7716shr/92564data kb, in
>>> >>> 680usr/50sys/731real
>>> >>>  ms.
>>> >>>
>>> >>>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
>>> >>> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>>> >>>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>>> >>>          Pass 4: compiled C into "stap_1837.ko" in
>>> >>> 1410usr/120sys/1963real ms.
>>> >>>          Pass 5: starting run.
>>> >>>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko':
>>> >>> Invalid parameters
>>> >>>          WARNING: /usr/bin/staprun exited with status: 1
>>> >>>          Pass 5: run completed in 0usr/0sys/16real ms.
>>> >>>          Pass 5: run failed.  [man error::pass5]
>>> >>>          Tip: /usr/share/doc/systemtap/README.Debian should help you
>>> >>> get started.
>>> >>>          Keeping temporary directory "/tmp/stapzKUZ11"
>>> >>>
>>> >>>
>>> >>>          in dmesg I can see the following lines
>>> >>>
>>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no
>>> >>> symbol version for foo
>>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230:
>>> >>> Unknown symbol foo (err -22)
>>> >>>
>>> >>>
>>> >>>          I also checked /proc/kallsyms
>>> >>>
>>> >>>
>>> >>>          cat /proc/kallsyms | grep xxx
>>> >>>          ffffffffc0584000 T foo    [xxx]
>>> >>>          ffffffffc0584014 t xxx_exit    [xxx]
>>> >>>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>>> >>>          ffffffffc058506e r __kstrtab_foo    [xxx]
>>> >>>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>>> >>>          ffffffffc0586000 d __this_module    [xxx]
>>> >>>          ffffffffc0584014 t cleanup_module    [xxx]
>>> >>>
>>> >>>
>>> >>> Please advice
>>> >>> 1.
>>> >>>  what should i do in addition to solve this issue as I must create an
>>> >>> interaction between systemtap script and another kernel module
>>> >>> 2.
>>> >>>  In case this is not feasible can u please offer another alternative
>>> >>> to
>>> >>> make systemtap script use another kernel module services ?
>>> >>>
>>> >>> Thanks eyal yehuda
>>>
>>

Re:

[Arkady]

Daniel,

STAP attempts to compile the kernel module against the kernel source
tree installed in the system. One way to resolve the issue is to
update the kernel source tree installed in the system. Without
updating the kernel sources the compilation will fail.

On Sun, Jul 30, 2017 at 9:56 AM, Daniel Doron <danielmeirdoron@gmail.com> wrote:
> Hi,
>
> How is systemtap's kernel module different from other modules?
> Given module A which exports a function and module B using that exported
> function and A is insmodded before B, I have not seen any problem. Why is
> Systemtap's KO different? Is it not the kernel's job to resolve the exported
> symbol?
>
>
> Daniel.
>
> On Thu, 27 Jul 2017 at 19:29 Arkady <arkady.miasnikov@gmail.com> wrote:
>>
>> P.S.1 There is yet another way - very cool way - to to add a symbol to
>> the stap script - via /sys/kernel/debug/systemtap/<module name>/.cmd
>> (CTL_CHANNEL_NAME below)
>>
>> This is a Python sample
>>
>> def write_relocate_string(f, s, size):
>>     f.write(s)
>>     padding = size - len(s)
>>     for _ in range(padding):
>>         f.write('\x00')
>>
>> def write_relocate(module_name, symbol_name, address):
>>     STP_MODULE_NAME_LEN = 128
>>     STP_SYMBOL_NAME_LEN = 128
>>
>>     with open(CTL_CHANNEL_NAME, "wb") as f:
>>     # STAP_RELOCATE (9)
>>         f.write(struct.pack('<I', 9))
>>         write_relocate_string(f, module_name, STP_MODULE_NAME_LEN)
>>         write_relocate_string(f, symbol_name, STP_SYMBOL_NAME_LEN)
>>         f.write(struct.pack('<Q', address))
>>
>>
>>
>> On Thu, Jul 27, 2017 at 7:20 PM, Arkady <arkady.miasnikov@gmail.com>
>> wrote:
>> > P.S. Another idea is to use kernel IPC instead of function calls.
>> > Files, named pipes, sockets, memory mapped files - just about anything
>> > can do the trick.
>> >
>> > On Thu, Jul 27, 2017 at 7:03 PM, Arkady <arkady.miasnikov@gmail.com>
>> > wrote:
>> >> Eyal,
>> >>
>> >> Utility stap employs the kernel debug information installed in the
>> >> system. When you load a module command insmod does not update the
>> >> kernel debug info. There are simple two ways to resolve the issue
>> >> *  Load the module, get the address of the function foo using
>> >> /proc/kallsyms, provide the address of the foo as an argument to the
>> >> STAP script.
>> >> *  Link the module statically with the kernel and update the
>> >> kernel-debuginfo
>> >>
>> >> Arkady
>> >>
>> >> On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
>> >> <systemtap@sourceware.org> wrote:
>> >>>
>> >>>
>> >>> hello
>> >>>
>> >>> The
>> >>>  issue I am facing is of undefined symbols in system tap kernel module
>> >>> when trying to link to another kernel module with exported symbol
>> >>>
>> >>> please note environment setup:
>> >>> os:  debian 9 stretch 64bit
>> >>> kernel version:  4.9.0-3-amd64
>> >>>
>> >>>
>> >>>           I created the following simple systemtap script using guru
>> >>> mode
>> >>>
>> >>>           system tap script:      temp.stp
>> >>>
>> >>>
>> >>>          %{
>> >>>                  extern int foo(void);
>> >>>          %}
>> >>>
>> >>>           function call_foo:long() %{/* pure */
>> >>>                   STAP_RETVALUE = foo();
>> >>>          %}
>> >>>
>> >>>           probe syscall.open {
>> >>>                  printf("foo() : %d\n",  call_foo())
>> >>>           }
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>  In addition to that I created the following simple kernel module with
>> >>> an exported symbol function to be used from inside system tap
>> >>> script
>> >>>
>> >>>          kernel module:       xxx.c
>> >>>
>> >>>          #include <linux/kernel.h>
>> >>>          #include <linux/module.h>
>> >>>          #include <linux/printk.h>
>> >>>
>> >>>          static int foo(void){
>> >>>                  pr_info("foo()\n"); return 0;
>> >>>          }
>> >>>          EXPORT_SYMBOL(foo);
>> >>>
>> >>>          static int __init xxx_init(void){
>> >>>                 pr_info("xxx_init()\n");
>> >>>            return 0;
>> >>>          }
>> >>>
>> >>>          static void __exit xxx_exit(void){
>> >>>                 pr_info("xxx_exit() \n");
>> >>>          }
>> >>>
>> >>>          module_init(xxx_init);
>> >>>          module_exit(xxx_exit);
>> >>>          MODULE_LICENSE("GPL");
>> >>>
>> >>>
>> >>>          After compiling the kernel module and insmoding it, I run the
>> >>> command to compile systemtap script as follow:
>> >>>          stap -gk -v temp.stp
>> >>>
>> >>>          the compilation command result with the following error:
>> >>>
>> >>>
>> >>>  Pass 1: parsed user script and 465 library scripts using
>> >>> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>> >>>
>> >>>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
>> >>> using 165912virt/100212res/7716shr/92564data kb, in
>> >>> 680usr/50sys/731real
>> >>>  ms.
>> >>>
>> >>>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
>> >>> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>> >>>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>> >>>          Pass 4: compiled C into "stap_1837.ko" in
>> >>> 1410usr/120sys/1963real ms.
>> >>>          Pass 5: starting run.
>> >>>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko':
>> >>> Invalid parameters
>> >>>          WARNING: /usr/bin/staprun exited with status: 1
>> >>>          Pass 5: run completed in 0usr/0sys/16real ms.
>> >>>          Pass 5: run failed.  [man error::pass5]
>> >>>          Tip: /usr/share/doc/systemtap/README.Debian should help you
>> >>> get started.
>> >>>          Keeping temporary directory "/tmp/stapzKUZ11"
>> >>>
>> >>>
>> >>>          in dmesg I can see the following lines
>> >>>
>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no
>> >>> symbol version for foo
>> >>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230:
>> >>> Unknown symbol foo (err -22)
>> >>>
>> >>>
>> >>>          I also checked /proc/kallsyms
>> >>>
>> >>>
>> >>>          cat /proc/kallsyms | grep xxx
>> >>>          ffffffffc0584000 T foo    [xxx]
>> >>>          ffffffffc0584014 t xxx_exit    [xxx]
>> >>>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>> >>>          ffffffffc058506e r __kstrtab_foo    [xxx]
>> >>>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>> >>>          ffffffffc0586000 d __this_module    [xxx]
>> >>>          ffffffffc0584014 t cleanup_module    [xxx]
>> >>>
>> >>>
>> >>> Please advice
>> >>> 1.
>> >>>  what should i do in addition to solve this issue as I must create an
>> >>> interaction between systemtap script and another kernel module
>> >>> 2.
>> >>>  In case this is not feasible can u please offer another alternative
>> >>> to
>> >>> make systemtap script use another kernel module services ?
>> >>>
>> >>> Thanks eyal yehuda
>>
>

Re:

[Arkady]

P.S.1 There is yet another way - very cool way - to to add a symbol to
the stap script - via /sys/kernel/debug/systemtap/<module name>/.cmd
(CTL_CHANNEL_NAME below)

This is a Python sample

def write_relocate_string(f, s, size):
    f.write(s)
    padding = size - len(s)
    for _ in range(padding):
        f.write('\x00')

def write_relocate(module_name, symbol_name, address):
    STP_MODULE_NAME_LEN = 128
    STP_SYMBOL_NAME_LEN = 128

    with open(CTL_CHANNEL_NAME, "wb") as f:
    # STAP_RELOCATE (9)
        f.write(struct.pack('<I', 9))
        write_relocate_string(f, module_name, STP_MODULE_NAME_LEN)
        write_relocate_string(f, symbol_name, STP_SYMBOL_NAME_LEN)
        f.write(struct.pack('<Q', address))



On Thu, Jul 27, 2017 at 7:20 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> P.S. Another idea is to use kernel IPC instead of function calls.
> Files, named pipes, sockets, memory mapped files - just about anything
> can do the trick.
>
> On Thu, Jul 27, 2017 at 7:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
>> Eyal,
>>
>> Utility stap employs the kernel debug information installed in the
>> system. When you load a module command insmod does not update the
>> kernel debug info. There are simple two ways to resolve the issue
>> *  Load the module, get the address of the function foo using
>> /proc/kallsyms, provide the address of the foo as an argument to the
>> STAP script.
>> *  Link the module statically with the kernel and update the kernel-debuginfo
>>
>> Arkady
>>
>> On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
>> <systemtap@sourceware.org> wrote:
>>>
>>>
>>> hello
>>>
>>> The
>>>  issue I am facing is of undefined symbols in system tap kernel module
>>> when trying to link to another kernel module with exported symbol
>>>
>>> please note environment setup:
>>> os:  debian 9 stretch 64bit
>>> kernel version:  4.9.0-3-amd64
>>>
>>>
>>>           I created the following simple systemtap script using guru mode
>>>
>>>           system tap script:      temp.stp
>>>
>>>
>>>          %{
>>>                  extern int foo(void);
>>>          %}
>>>
>>>           function call_foo:long() %{/* pure */
>>>                   STAP_RETVALUE = foo();
>>>          %}
>>>
>>>           probe syscall.open {
>>>                  printf("foo() : %d\n",  call_foo())
>>>           }
>>>
>>>
>>>
>>>
>>>  In addition to that I created the following simple kernel module with
>>> an exported symbol function to be used from inside system tap
>>> script
>>>
>>>          kernel module:       xxx.c
>>>
>>>          #include <linux/kernel.h>
>>>          #include <linux/module.h>
>>>          #include <linux/printk.h>
>>>
>>>          static int foo(void){
>>>                  pr_info("foo()\n"); return 0;
>>>          }
>>>          EXPORT_SYMBOL(foo);
>>>
>>>          static int __init xxx_init(void){
>>>                 pr_info("xxx_init()\n");
>>>            return 0;
>>>          }
>>>
>>>          static void __exit xxx_exit(void){
>>>                 pr_info("xxx_exit() \n");
>>>          }
>>>
>>>          module_init(xxx_init);
>>>          module_exit(xxx_exit);
>>>          MODULE_LICENSE("GPL");
>>>
>>>
>>>          After compiling the kernel module and insmoding it, I run the command to compile systemtap script as follow:
>>>          stap -gk -v temp.stp
>>>
>>>          the compilation command result with the following error:
>>>
>>>
>>>  Pass 1: parsed user script and 465 library scripts using
>>> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>>>
>>>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
>>> using 165912virt/100212res/7716shr/92564data kb, in 680usr/50sys/731real
>>>  ms.
>>>
>>>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
>>> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>>>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>>>          Pass 4: compiled C into "stap_1837.ko" in 1410usr/120sys/1963real ms.
>>>          Pass 5: starting run.
>>>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko': Invalid parameters
>>>          WARNING: /usr/bin/staprun exited with status: 1
>>>          Pass 5: run completed in 0usr/0sys/16real ms.
>>>          Pass 5: run failed.  [man error::pass5]
>>>          Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
>>>          Keeping temporary directory "/tmp/stapzKUZ11"
>>>
>>>
>>>          in dmesg I can see the following lines
>>>
>>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no symbol version for foo
>>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230: Unknown symbol foo (err -22)
>>>
>>>
>>>          I also checked /proc/kallsyms
>>>
>>>
>>>          cat /proc/kallsyms | grep xxx
>>>          ffffffffc0584000 T foo    [xxx]
>>>          ffffffffc0584014 t xxx_exit    [xxx]
>>>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>>>          ffffffffc058506e r __kstrtab_foo    [xxx]
>>>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>>>          ffffffffc0586000 d __this_module    [xxx]
>>>          ffffffffc0584014 t cleanup_module    [xxx]
>>>
>>>
>>> Please advice
>>> 1.
>>>  what should i do in addition to solve this issue as I must create an
>>> interaction between systemtap script and another kernel module
>>> 2.
>>>  In case this is not feasible can u please offer another alternative to
>>> make systemtap script use another kernel module services ?
>>>
>>> Thanks eyal yehuda

Re:

[Arkady]

P.S. Another idea is to use kernel IPC instead of function calls.
Files, named pipes, sockets, memory mapped files - just about anything
can do the trick.

On Thu, Jul 27, 2017 at 7:03 PM, Arkady <arkady.miasnikov@gmail.com> wrote:
> Eyal,
>
> Utility stap employs the kernel debug information installed in the
> system. When you load a module command insmod does not update the
> kernel debug info. There are simple two ways to resolve the issue
> *  Load the module, get the address of the function foo using
> /proc/kallsyms, provide the address of the foo as an argument to the
> STAP script.
> *  Link the module statically with the kernel and update the kernel-debuginfo
>
> Arkady
>
> On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
> <systemtap@sourceware.org> wrote:
>>
>>
>> hello
>>
>> The
>>  issue I am facing is of undefined symbols in system tap kernel module
>> when trying to link to another kernel module with exported symbol
>>
>> please note environment setup:
>> os:  debian 9 stretch 64bit
>> kernel version:  4.9.0-3-amd64
>>
>>
>>           I created the following simple systemtap script using guru mode
>>
>>           system tap script:      temp.stp
>>
>>
>>          %{
>>                  extern int foo(void);
>>          %}
>>
>>           function call_foo:long() %{/* pure */
>>                   STAP_RETVALUE = foo();
>>          %}
>>
>>           probe syscall.open {
>>                  printf("foo() : %d\n",  call_foo())
>>           }
>>
>>
>>
>>
>>  In addition to that I created the following simple kernel module with
>> an exported symbol function to be used from inside system tap
>> script
>>
>>          kernel module:       xxx.c
>>
>>          #include <linux/kernel.h>
>>          #include <linux/module.h>
>>          #include <linux/printk.h>
>>
>>          static int foo(void){
>>                  pr_info("foo()\n"); return 0;
>>          }
>>          EXPORT_SYMBOL(foo);
>>
>>          static int __init xxx_init(void){
>>                 pr_info("xxx_init()\n");
>>            return 0;
>>          }
>>
>>          static void __exit xxx_exit(void){
>>                 pr_info("xxx_exit() \n");
>>          }
>>
>>          module_init(xxx_init);
>>          module_exit(xxx_exit);
>>          MODULE_LICENSE("GPL");
>>
>>
>>          After compiling the kernel module and insmoding it, I run the command to compile systemtap script as follow:
>>          stap -gk -v temp.stp
>>
>>          the compilation command result with the following error:
>>
>>
>>  Pass 1: parsed user script and 465 library scripts using
>> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>>
>>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
>> using 165912virt/100212res/7716shr/92564data kb, in 680usr/50sys/731real
>>  ms.
>>
>>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
>> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>>          Pass 4: compiled C into "stap_1837.ko" in 1410usr/120sys/1963real ms.
>>          Pass 5: starting run.
>>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko': Invalid parameters
>>          WARNING: /usr/bin/staprun exited with status: 1
>>          Pass 5: run completed in 0usr/0sys/16real ms.
>>          Pass 5: run failed.  [man error::pass5]
>>          Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
>>          Keeping temporary directory "/tmp/stapzKUZ11"
>>
>>
>>          in dmesg I can see the following lines
>>
>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no symbol version for foo
>>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230: Unknown symbol foo (err -22)
>>
>>
>>          I also checked /proc/kallsyms
>>
>>
>>          cat /proc/kallsyms | grep xxx
>>          ffffffffc0584000 T foo    [xxx]
>>          ffffffffc0584014 t xxx_exit    [xxx]
>>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>>          ffffffffc058506e r __kstrtab_foo    [xxx]
>>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>>          ffffffffc0586000 d __this_module    [xxx]
>>          ffffffffc0584014 t cleanup_module    [xxx]
>>
>>
>> Please advice
>> 1.
>>  what should i do in addition to solve this issue as I must create an
>> interaction between systemtap script and another kernel module
>> 2.
>>  In case this is not feasible can u please offer another alternative to
>> make systemtap script use another kernel module services ?
>>
>> Thanks eyal yehuda

Re:

[Arkady]

Eyal,

Utility stap employs the kernel debug information installed in the
system. When you load a module command insmod does not update the
kernel debug info. There are simple two ways to resolve the issue
*  Load the module, get the address of the function foo using
/proc/kallsyms, provide the address of the foo as an argument to the
STAP script.
*  Link the module statically with the kernel and update the kernel-debuginfo

Arkady

On Thu, Jul 27, 2017 at 5:47 PM, Eyal Yehuda via systemtap
<systemtap@sourceware.org> wrote:
>
>
> hello
>
> The
>  issue I am facing is of undefined symbols in system tap kernel module
> when trying to link to another kernel module with exported symbol
>
> please note environment setup:
> os:  debian 9 stretch 64bit
> kernel version:  4.9.0-3-amd64
>
>
>           I created the following simple systemtap script using guru mode
>
>           system tap script:      temp.stp
>
>
>          %{
>                  extern int foo(void);
>          %}
>
>           function call_foo:long() %{/* pure */
>                   STAP_RETVALUE = foo();
>          %}
>
>           probe syscall.open {
>                  printf("foo() : %d\n",  call_foo())
>           }
>
>
>
>
>  In addition to that I created the following simple kernel module with
> an exported symbol function to be used from inside system tap
> script
>
>          kernel module:       xxx.c
>
>          #include <linux/kernel.h>
>          #include <linux/module.h>
>          #include <linux/printk.h>
>
>          static int foo(void){
>                  pr_info("foo()\n"); return 0;
>          }
>          EXPORT_SYMBOL(foo);
>
>          static int __init xxx_init(void){
>                 pr_info("xxx_init()\n");
>            return 0;
>          }
>
>          static void __exit xxx_exit(void){
>                 pr_info("xxx_exit() \n");
>          }
>
>          module_init(xxx_init);
>          module_exit(xxx_exit);
>          MODULE_LICENSE("GPL");
>
>
>          After compiling the kernel module and insmoding it, I run the command to compile systemtap script as follow:
>          stap -gk -v temp.stp
>
>          the compilation command result with the following error:
>
>
>  Pass 1: parsed user script and 465 library scripts using
> 113912virt/46752res/6476shr/40564data kb, in 110usr/20sys/124real ms.
>
>  Pass 2: analyzed script: 2 probes, 2 functions, 99 embeds, 0 globals
> using 165912virt/100212res/7716shr/92564data kb, in 680usr/50sys/731real
>  ms.
>
>  Pass 3: translated to C into "/tmp/stapzKUZ11/stap_1837_src.c" using
> 165912virt/100404res/7908shr/92564data kb, in 10usr/10sys/9real ms.
>          WARNING: "foo" [/tmp/stapzKUZ11/stap_1837.ko] undefined!
>          Pass 4: compiled C into "stap_1837.ko" in 1410usr/120sys/1963real ms.
>          Pass 5: starting run.
>          ERROR: Couldn't insert module '/tmp/stapzKUZ11/stap_1837.ko': Invalid parameters
>          WARNING: /usr/bin/staprun exited with status: 1
>          Pass 5: run completed in 0usr/0sys/16real ms.
>          Pass 5: run failed.  [man error::pass5]
>          Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
>          Keeping temporary directory "/tmp/stapzKUZ11"
>
>
>          in dmesg I can see the following lines
>
>          Jul 27 03:56:14 debian9 kernel: [ 3938.166906] stap_2230: no symbol version for foo
>          Jul 27 03:56:14 debian9 kernel: [ 3938.166908] stap_2230: Unknown symbol foo (err -22)
>
>
>          I also checked /proc/kallsyms
>
>
>          cat /proc/kallsyms | grep xxx
>          ffffffffc0584000 T foo    [xxx]
>          ffffffffc0584014 t xxx_exit    [xxx]
>          ffffffffc0585030 r __ksymtab_foo    [xxx]
>          ffffffffc058506e r __kstrtab_foo    [xxx]
>          ffffffffc0585040 r __kcrctab_foo    [xxx]
>          ffffffffc0586000 d __this_module    [xxx]
>          ffffffffc0584014 t cleanup_module    [xxx]
>
>
> Please advice
> 1.
>  what should i do in addition to solve this issue as I must create an
> interaction between systemtap script and another kernel module
> 2.
>  In case this is not feasible can u please offer another alternative to
> make systemtap script use another kernel module services ?
>
> Thanks eyal yehuda