Using systemtap in an Environment with Multiple Versions of glibc

[wei gong <gongwei833x@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6000 bytes --]

Dear SystemTap Community,

I am currently experiencing challenges using SystemTap in an environment
that includes multiple versions of glibc. I am hoping someone here may have
encountered a similar issue and might be able to provide guidance.


To provide some background, here is a brief account of what I’ve been
undertaking and the hurdles I’ve stumbled upon…


Here is my environment information:

CentOS Linux release 7.6 (Final)

Kernel: 4.19.0, and the corresponding debug packages have been installed.

Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
to use a newer version of GCC, I installed an additional version of GCC and
matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
directory. My software is compiled using this newly installed version of
gcc-8.2.


The information of my compiled software is:

1. /opt/compiler/gcc-8.2/bin/ldd myapp
linux-vdso.so.1 (0x00007ffe7a9e7000)
libpixman-1.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 (
0x00007fa93144a000)
libz.so.1 => /opt/compiler/gcc-8.2/usr/lib/libz.so.1 (0x00007fa93142d000)
libutil.so.1 => /opt/compiler/gcc-8.2/usr/lib/libutil.so.1 (
0x00007fa931428000)
libglib-2.0.so.0 => /opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 (
0x00007fa931112000)
libdl.so.2 => /opt/compiler/gcc-8.2/usr/lib/libdl.so.2 (0x00007fa93110c000)
libcurl.so.4 => /opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 (
0x00007fa9310ab000)
libaio.so.1 => /opt/compiler/gcc-8.2/usr/lib/libaio.so.1 (0x00007fa930ea9000
)
libstdc++.so.6 => /opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 (
0x00007fa930d1d000)
libgflags.so.2.1 => /opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 (
0x00007fa930afc000)
libprotobuf.so.8 => /opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 (
0x00007fa9307ea000)
libleveldb.so.1 => /opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 (
0x00007fa930595000)
libm.so.6 => /opt/compiler/gcc-8.2/usr/lib/libm.so.6 (0x00007fa93048f000)
libgcc_s.so.1 => /opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 (
0x00007fa930475000)
libpthread.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 (
0x00007fa930455000)
libc.so.6 => /opt/compiler/gcc-8.2/usr/lib/libc.so.6 (0x00007fa9302b0000)
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 (0x00007fa9316f3000)
libpcre.so.1 => /opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 (
0x00007fa93004e000)
libsnappy.so.1 => /opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 (
0x00007fa92fe47000)


2. /usr/bin/ldd myapp
linux-vdso.so.1 => (0x00007fffcff84000)
libpixman-1.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 (
0x00007f6691442000)
libz.so.1 => /opt/compiler/gcc-8.2/usr/lib/libz.so.1 (0x00007f66918ee000)
libutil.so.1 => /opt/compiler/gcc-8.2/usr/lib/libutil.so.1 (
0x00007f66918e9000)
libglib-2.0.so.0 => /opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 (
0x00007f669112c000)
libdl.so.2 => /opt/compiler/gcc-8.2/usr/lib/libdl.so.2 (0x00007f66918e3000)
libcurl.so.4 => /opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 (
0x00007f6691882000)
libaio.so.1 => /opt/compiler/gcc-8.2/usr/lib/libaio.so.1 (0x00007f6690f2a000
)
libstdc++.so.6 => /opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 (
0x00007f6690d9f000)
libgflags.so.2.1 => /opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 (
0x00007f6690b7e000)
libprotobuf.so.8 => /opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 (
0x00007f669086c000)
libleveldb.so.1 => /opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 (
0x00007f6690618000)
libm.so.6 => /opt/compiler/gcc-8.2/usr/lib/libm.so.6 (0x00007f669177a000)
libgcc_s.so.1 => /opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 (
0x00007f6691760000)
libpthread.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 (
0x00007f6691740000)
libc.so.6 => /opt/compiler/gcc-8.2/usr/lib/libc.so.6 (0x00007f6690473000)
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-
64.so.2 (0x00007f66916eb000)
libpcre.so.1 => /opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 (
0x00007f6690211000)
libsnappy.so.1 => /opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 (
0x00007f669000b000)



VERSION: systemtap 4.0

When I use systemtap (whether installed through yum or compiled from source
code) to probe myapp, it always gives an error:


./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
process("/path/myapp").function("funtion_name").call {print_ubacktrace()}'
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
_stp_stack_unwind_one_user:496: STARTING user unwind
0x9e527b
_stp_stack_unwind_one_user:512: CONTINUING user unwind to depth 1
unwind:1483: pc=9e527b, 9e527b
unwind:1522: No module found for pc=9e527b
_stp_stack_unwind_one_user:537: ret=-22 PC=9e527b SP=7fffcd857f50
_stp_stack_unwind_one_user:496: STARTING user unwind



I can retrieve it normally when using the program on my system：


./stap -d /usr/sbin/ls -e 'probe process("ls").function("xmalloc").call
{print_ubacktrace()}' -c "ls /"
0x411680 : xmalloc+0x0/0x20 [/usr/bin/ls]
0x411854 : xmemdup+0x14/0x30 [/usr/bin/ls]
0x40efea : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
0x402c10 : main+0x350/0x1a14 [/usr/bin/ls]
0x7f94b8081555 [/usr/lib64/libc-2.17.so+0x22555/0x3ce000]
0x411680 : xmalloc+0x0/0x20 [/usr/bin/ls]
0x411854 : xmemdup+0x14/0x30 [/usr/bin/ls]
0x40efea : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
0x402c74 : main+0x3b4/0x1a14 [/usr/bin/ls]
0x7f94b8081555 [/usr/lib64/libc-2.17.so+0x22555/0x3ce000]
...
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/
libc-2.17.so'
WARNING: too many pending (warning) messages



I’ve tried many methods but none of them have been effective. I don’t quite
understand the principles of systemtap, and I’m not sure why the program
compiled with a standalone gcc-8.2 cannot be used normally.

Any advice, tips, or guidance you could provide would be immensely
appreciated. Thank you in advance for your assistance!


Best Regards,

Wei Gong