Using systemtap in an Environment with Multiple Versions of glibc

Using systemtap in an Environment with Multiple Versions of glibc

[wei gong]

[-- Attachment #1: Type: text/plain, Size: 6000 bytes --]

Dear SystemTap Community,

I am currently experiencing challenges using SystemTap in an environment
that includes multiple versions of glibc. I am hoping someone here may have
encountered a similar issue and might be able to provide guidance.


To provide some background, here is a brief account of what I’ve been
undertaking and the hurdles I’ve stumbled upon…


Here is my environment information:

CentOS Linux release 7.6 (Final)

Kernel: 4.19.0, and the corresponding debug packages have been installed.

Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
to use a newer version of GCC, I installed an additional version of GCC and
matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
directory. My software is compiled using this newly installed version of
gcc-8.2.


The information of my compiled software is:

1. /opt/compiler/gcc-8.2/bin/ldd myapp
linux-vdso.so.1 (0x00007ffe7a9e7000)
libpixman-1.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 (
0x00007fa93144a000)
libz.so.1 => /opt/compiler/gcc-8.2/usr/lib/libz.so.1 (0x00007fa93142d000)
libutil.so.1 => /opt/compiler/gcc-8.2/usr/lib/libutil.so.1 (
0x00007fa931428000)
libglib-2.0.so.0 => /opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 (
0x00007fa931112000)
libdl.so.2 => /opt/compiler/gcc-8.2/usr/lib/libdl.so.2 (0x00007fa93110c000)
libcurl.so.4 => /opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 (
0x00007fa9310ab000)
libaio.so.1 => /opt/compiler/gcc-8.2/usr/lib/libaio.so.1 (0x00007fa930ea9000
)
libstdc++.so.6 => /opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 (
0x00007fa930d1d000)
libgflags.so.2.1 => /opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 (
0x00007fa930afc000)
libprotobuf.so.8 => /opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 (
0x00007fa9307ea000)
libleveldb.so.1 => /opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 (
0x00007fa930595000)
libm.so.6 => /opt/compiler/gcc-8.2/usr/lib/libm.so.6 (0x00007fa93048f000)
libgcc_s.so.1 => /opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 (
0x00007fa930475000)
libpthread.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 (
0x00007fa930455000)
libc.so.6 => /opt/compiler/gcc-8.2/usr/lib/libc.so.6 (0x00007fa9302b0000)
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 (0x00007fa9316f3000)
libpcre.so.1 => /opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 (
0x00007fa93004e000)
libsnappy.so.1 => /opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 (
0x00007fa92fe47000)


2. /usr/bin/ldd myapp
linux-vdso.so.1 => (0x00007fffcff84000)
libpixman-1.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 (
0x00007f6691442000)
libz.so.1 => /opt/compiler/gcc-8.2/usr/lib/libz.so.1 (0x00007f66918ee000)
libutil.so.1 => /opt/compiler/gcc-8.2/usr/lib/libutil.so.1 (
0x00007f66918e9000)
libglib-2.0.so.0 => /opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 (
0x00007f669112c000)
libdl.so.2 => /opt/compiler/gcc-8.2/usr/lib/libdl.so.2 (0x00007f66918e3000)
libcurl.so.4 => /opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 (
0x00007f6691882000)
libaio.so.1 => /opt/compiler/gcc-8.2/usr/lib/libaio.so.1 (0x00007f6690f2a000
)
libstdc++.so.6 => /opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 (
0x00007f6690d9f000)
libgflags.so.2.1 => /opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 (
0x00007f6690b7e000)
libprotobuf.so.8 => /opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 (
0x00007f669086c000)
libleveldb.so.1 => /opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 (
0x00007f6690618000)
libm.so.6 => /opt/compiler/gcc-8.2/usr/lib/libm.so.6 (0x00007f669177a000)
libgcc_s.so.1 => /opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 (
0x00007f6691760000)
libpthread.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 (
0x00007f6691740000)
libc.so.6 => /opt/compiler/gcc-8.2/usr/lib/libc.so.6 (0x00007f6690473000)
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-
64.so.2 (0x00007f66916eb000)
libpcre.so.1 => /opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 (
0x00007f6690211000)
libsnappy.so.1 => /opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 (
0x00007f669000b000)



VERSION: systemtap 4.0

When I use systemtap (whether installed through yum or compiled from source
code) to probe myapp, it always gives an error:


./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
process("/path/myapp").function("funtion_name").call {print_ubacktrace()}'
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
_stp_stack_unwind_one_user:496: STARTING user unwind
0x9e527b
_stp_stack_unwind_one_user:512: CONTINUING user unwind to depth 1
unwind:1483: pc=9e527b, 9e527b
unwind:1522: No module found for pc=9e527b
_stp_stack_unwind_one_user:537: ret=-22 PC=9e527b SP=7fffcd857f50
_stp_stack_unwind_one_user:496: STARTING user unwind



I can retrieve it normally when using the program on my system：


./stap -d /usr/sbin/ls -e 'probe process("ls").function("xmalloc").call
{print_ubacktrace()}' -c "ls /"
0x411680 : xmalloc+0x0/0x20 [/usr/bin/ls]
0x411854 : xmemdup+0x14/0x30 [/usr/bin/ls]
0x40efea : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
0x402c10 : main+0x350/0x1a14 [/usr/bin/ls]
0x7f94b8081555 [/usr/lib64/libc-2.17.so+0x22555/0x3ce000]
0x411680 : xmalloc+0x0/0x20 [/usr/bin/ls]
0x411854 : xmemdup+0x14/0x30 [/usr/bin/ls]
0x40efea : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
0x402c74 : main+0x3b4/0x1a14 [/usr/bin/ls]
0x7f94b8081555 [/usr/lib64/libc-2.17.so+0x22555/0x3ce000]
...
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/
libc-2.17.so'
WARNING: too many pending (warning) messages



I’ve tried many methods but none of them have been effective. I don’t quite
understand the principles of systemtap, and I’m not sure why the program
compiled with a standalone gcc-8.2 cannot be used normally.

Any advice, tips, or guidance you could provide would be immensely
appreciated. Thank you in advance for your assistance!


Best Regards,

Wei Gong

Re: Using systemtap in an Environment with Multiple Versions of glibc

[Frank Ch. Eigler]

wei gong <gongwei833x@gmail.com> writes:

> [...]
> I am currently experiencing challenges using SystemTap in an environment
> that includes multiple versions of glibc. I am hoping someone here may have
> encountered a similar issue and might be able to provide guidance.
> [...]
> Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
> to use a newer version of GCC, I installed an additional version of GCC and
> matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
> directory. My software is compiled using this newly installed version of
> gcc-8.2.
> [...]

> ./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
> process("/path/myapp").function("funtion_name").call {print_ubacktrace()}'
> WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
> retry with -DDEBUG_UNWIND)'

One possibility is that this program, being compiled with your other
gcc/glibc toolchain, is not getting identified to systemtap for purpose
of extracting unwind/symbol data.

> WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/
> libc-2.17.so'
> WARNING: too many pending (warning) messages

Yeah, that's a hint.  The "-d FILE" and "--ldd" options should let you
give systemtap enough hints to find the alternate toolchain shared
libraries.  Try:

   # stap --ldd -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -e 'probe
     ("myapp").......'

If more missing library names come up as runtime warnings, rerun, and
add them as -d PATHs too.


- FChE

Re: Using systemtap in an Environment with Multiple Versions of glibc

[wei gong]

[-- Attachment #1: Type: text/plain, Size: 2368 bytes --]

Frank Ch. Eigler <fche@redhat.com> 于2023年7月29日周六 03:23写道：

>
> wei gong <gongwei833x@gmail.com> writes:
>
> > [...]
> > I am currently experiencing challenges using SystemTap in an environment
> > that includes multiple versions of glibc. I am hoping someone here may
> have
> > encountered a similar issue and might be able to provide guidance.
> > [...]
> > Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
> > to use a newer version of GCC, I installed an additional version of GCC
> and
> > matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
> > directory. My software is compiled using this newly installed version of
> > gcc-8.2.
> > [...]
>
> > ./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
> > process("/path/myapp").function("funtion_name").call
> {print_ubacktrace()}'
> > WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
> > retry with -DDEBUG_UNWIND)'
>
> One possibility is that this program, being compiled with your other
> gcc/glibc toolchain, is not getting identified to systemtap for purpose
> of extracting unwind/symbol data.
>
> > WARNING: Missing unwind data for a module, rerun with 'stap -d
> /usr/lib64/
> > libc-2.17.so'
> > WARNING: too many pending (warning) messages
>
> Yeah, that's a hint.  The "-d FILE" and "--ldd" options should let you
> give systemtap enough hints to find the alternate toolchain shared
> libraries.  Try:
>
>    # stap --ldd -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -e 'probe
>      ("myapp").......'
>
> If more missing library names come up as runtime warnings, rerun, and
> add them as -d PATHs too.
>
>
> stap  --ldd  -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -d
/opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2 -d
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 -e 'probe
process("myapp").function("func_name").call {print_ubacktrace()}'
WARNING: module /home/opt/compiler/gcc-8.2/lib/libc-2.21.so --ldd skipped:
unsupported interpreter: /opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2
WARNING: module myapp --ldd skipped: unsupported interpreter:
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2

Trigger probe point function

WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
 0x8789b6


> - FChE
>
>

Re: Using systemtap in an Environment with Multiple Versions of glibc

[Frank Ch. Eigler]

Hi -

> > stap  --ldd  -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -d
> /opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2 -d
> /opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 -e 'probe
> process("myapp").function("func_name").call {print_ubacktrace()}'
> WARNING: module /home/opt/compiler/gcc-8.2/lib/libc-2.21.so --ldd skipped:
> unsupported interpreter: /opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2
> WARNING: module myapp --ldd skipped: unsupported interpreter:
> /opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2

Ah yeah.  Having a custom ld.so like that is unusual.  Sorry I didn't
realize this aspect of your situation.  You could try running ldd
yourself on your binary, and manually add "-d
/opt/compiler/..../lib..." flags for each.


- FChE

Re: Using systemtap in an Environment with Multiple Versions of glibc

[wei gong]

[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]

Hi


> Ah yeah.  Having a custom ld.so like that is unusual.  Sorry I didn't
> realize this aspect of your situation.  You could try running ldd
> yourself on your binary, and manually add "-d
> /opt/compiler/..../lib..." flags for each.
>
> I tried running ldd on my binary file and manually adding ‘-d
/opt/compiler/…/lib…’ for each one.

 stap  -d /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 -d
/opt/compiler/gcc-8.2/usr/lib/libz.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libutil.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 -d
/opt/compiler/gcc-8.2/usr/lib/libdl.so.2 -d
/opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 -d
/opt/compiler/gcc-8.2/usr/lib/libaio.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 -d
/opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 -d
/opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 -d
/opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libm.so.6 -d
/opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 -d
/opt/compiler/gcc-8.2/usr/lib/libc.so.6 -d
/opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 -d
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2  -e 'probe
process("myapp").function("function").call {print_ubacktrace()}'

Trigger probe point function

WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
 0x8789b6

Best Regards,
WeiGong