* Using systemtap in an Environment with Multiple Versions of glibc
@ 2023-07-28 13:59 wei gong
2023-07-28 19:21 ` Frank Ch. Eigler
0 siblings, 1 reply; 5+ messages in thread
From: wei gong @ 2023-07-28 13:59 UTC (permalink / raw)
To: systemtap
[-- Attachment #1: Type: text/plain, Size: 6000 bytes --]
Dear SystemTap Community,
I am currently experiencing challenges using SystemTap in an environment
that includes multiple versions of glibc. I am hoping someone here may have
encountered a similar issue and might be able to provide guidance.
To provide some background, here is a brief account of what I’ve been
undertaking and the hurdles I’ve stumbled upon…
Here is my environment information:
CentOS Linux release 7.6 (Final)
Kernel: 4.19.0, and the corresponding debug packages have been installed.
Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
to use a newer version of GCC, I installed an additional version of GCC and
matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
directory. My software is compiled using this newly installed version of
gcc-8.2.
The information of my compiled software is:
1. /opt/compiler/gcc-8.2/bin/ldd myapp
linux-vdso.so.1 (0x00007ffe7a9e7000)
libpixman-1.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 (
0x00007fa93144a000)
libz.so.1 => /opt/compiler/gcc-8.2/usr/lib/libz.so.1 (0x00007fa93142d000)
libutil.so.1 => /opt/compiler/gcc-8.2/usr/lib/libutil.so.1 (
0x00007fa931428000)
libglib-2.0.so.0 => /opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 (
0x00007fa931112000)
libdl.so.2 => /opt/compiler/gcc-8.2/usr/lib/libdl.so.2 (0x00007fa93110c000)
libcurl.so.4 => /opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 (
0x00007fa9310ab000)
libaio.so.1 => /opt/compiler/gcc-8.2/usr/lib/libaio.so.1 (0x00007fa930ea9000
)
libstdc++.so.6 => /opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 (
0x00007fa930d1d000)
libgflags.so.2.1 => /opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 (
0x00007fa930afc000)
libprotobuf.so.8 => /opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 (
0x00007fa9307ea000)
libleveldb.so.1 => /opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 (
0x00007fa930595000)
libm.so.6 => /opt/compiler/gcc-8.2/usr/lib/libm.so.6 (0x00007fa93048f000)
libgcc_s.so.1 => /opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 (
0x00007fa930475000)
libpthread.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 (
0x00007fa930455000)
libc.so.6 => /opt/compiler/gcc-8.2/usr/lib/libc.so.6 (0x00007fa9302b0000)
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 (0x00007fa9316f3000)
libpcre.so.1 => /opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 (
0x00007fa93004e000)
libsnappy.so.1 => /opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 (
0x00007fa92fe47000)
2. /usr/bin/ldd myapp
linux-vdso.so.1 => (0x00007fffcff84000)
libpixman-1.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 (
0x00007f6691442000)
libz.so.1 => /opt/compiler/gcc-8.2/usr/lib/libz.so.1 (0x00007f66918ee000)
libutil.so.1 => /opt/compiler/gcc-8.2/usr/lib/libutil.so.1 (
0x00007f66918e9000)
libglib-2.0.so.0 => /opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 (
0x00007f669112c000)
libdl.so.2 => /opt/compiler/gcc-8.2/usr/lib/libdl.so.2 (0x00007f66918e3000)
libcurl.so.4 => /opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 (
0x00007f6691882000)
libaio.so.1 => /opt/compiler/gcc-8.2/usr/lib/libaio.so.1 (0x00007f6690f2a000
)
libstdc++.so.6 => /opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 (
0x00007f6690d9f000)
libgflags.so.2.1 => /opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 (
0x00007f6690b7e000)
libprotobuf.so.8 => /opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 (
0x00007f669086c000)
libleveldb.so.1 => /opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 (
0x00007f6690618000)
libm.so.6 => /opt/compiler/gcc-8.2/usr/lib/libm.so.6 (0x00007f669177a000)
libgcc_s.so.1 => /opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 (
0x00007f6691760000)
libpthread.so.0 => /opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 (
0x00007f6691740000)
libc.so.6 => /opt/compiler/gcc-8.2/usr/lib/libc.so.6 (0x00007f6690473000)
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-
64.so.2 (0x00007f66916eb000)
libpcre.so.1 => /opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 (
0x00007f6690211000)
libsnappy.so.1 => /opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 (
0x00007f669000b000)
VERSION: systemtap 4.0
When I use systemtap (whether installed through yum or compiled from source
code) to probe myapp, it always gives an error:
./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
process("/path/myapp").function("funtion_name").call {print_ubacktrace()}'
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
_stp_stack_unwind_one_user:496: STARTING user unwind
0x9e527b
_stp_stack_unwind_one_user:512: CONTINUING user unwind to depth 1
unwind:1483: pc=9e527b, 9e527b
unwind:1522: No module found for pc=9e527b
_stp_stack_unwind_one_user:537: ret=-22 PC=9e527b SP=7fffcd857f50
_stp_stack_unwind_one_user:496: STARTING user unwind
I can retrieve it normally when using the program on my system:
./stap -d /usr/sbin/ls -e 'probe process("ls").function("xmalloc").call
{print_ubacktrace()}' -c "ls /"
0x411680 : xmalloc+0x0/0x20 [/usr/bin/ls]
0x411854 : xmemdup+0x14/0x30 [/usr/bin/ls]
0x40efea : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
0x402c10 : main+0x350/0x1a14 [/usr/bin/ls]
0x7f94b8081555 [/usr/lib64/libc-2.17.so+0x22555/0x3ce000]
0x411680 : xmalloc+0x0/0x20 [/usr/bin/ls]
0x411854 : xmemdup+0x14/0x30 [/usr/bin/ls]
0x40efea : clone_quoting_options+0x2a/0x40 [/usr/bin/ls]
0x402c74 : main+0x3b4/0x1a14 [/usr/bin/ls]
0x7f94b8081555 [/usr/lib64/libc-2.17.so+0x22555/0x3ce000]
...
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/
libc-2.17.so'
WARNING: too many pending (warning) messages
I’ve tried many methods but none of them have been effective. I don’t quite
understand the principles of systemtap, and I’m not sure why the program
compiled with a standalone gcc-8.2 cannot be used normally.
Any advice, tips, or guidance you could provide would be immensely
appreciated. Thank you in advance for your assistance!
Best Regards,
Wei Gong
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Using systemtap in an Environment with Multiple Versions of glibc
2023-07-28 13:59 Using systemtap in an Environment with Multiple Versions of glibc wei gong
@ 2023-07-28 19:21 ` Frank Ch. Eigler
2023-07-29 12:46 ` wei gong
0 siblings, 1 reply; 5+ messages in thread
From: Frank Ch. Eigler @ 2023-07-28 19:21 UTC (permalink / raw)
To: wei gong; +Cc: systemtap
wei gong <gongwei833x@gmail.com> writes:
> [...]
> I am currently experiencing challenges using SystemTap in an environment
> that includes multiple versions of glibc. I am hoping someone here may have
> encountered a similar issue and might be able to provide guidance.
> [...]
> Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
> to use a newer version of GCC, I installed an additional version of GCC and
> matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
> directory. My software is compiled using this newly installed version of
> gcc-8.2.
> [...]
> ./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
> process("/path/myapp").function("funtion_name").call {print_ubacktrace()}'
> WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
> retry with -DDEBUG_UNWIND)'
One possibility is that this program, being compiled with your other
gcc/glibc toolchain, is not getting identified to systemtap for purpose
of extracting unwind/symbol data.
> WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/
> libc-2.17.so'
> WARNING: too many pending (warning) messages
Yeah, that's a hint. The "-d FILE" and "--ldd" options should let you
give systemtap enough hints to find the alternate toolchain shared
libraries. Try:
# stap --ldd -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -e 'probe
("myapp").......'
If more missing library names come up as runtime warnings, rerun, and
add them as -d PATHs too.
- FChE
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Using systemtap in an Environment with Multiple Versions of glibc
2023-07-28 19:21 ` Frank Ch. Eigler
@ 2023-07-29 12:46 ` wei gong
2023-07-29 13:16 ` Frank Ch. Eigler
0 siblings, 1 reply; 5+ messages in thread
From: wei gong @ 2023-07-29 12:46 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: systemtap
[-- Attachment #1: Type: text/plain, Size: 2368 bytes --]
Frank Ch. Eigler <fche@redhat.com> 于2023年7月29日周六 03:23写道:
>
> wei gong <gongwei833x@gmail.com> writes:
>
> > [...]
> > I am currently experiencing challenges using SystemTap in an environment
> > that includes multiple versions of glibc. I am hoping someone here may
> have
> > encountered a similar issue and might be able to provide guidance.
> > [...]
> > Currently, my environment has glibc-2.17-326.el7_9.x86_64. Because I want
> > to use a newer version of GCC, I installed an additional version of GCC
> and
> > matched it with glibc 2.21, which I placed in the /opt/compiler/gcc-8.2/
> > directory. My software is compiled using this newly installed version of
> > gcc-8.2.
> > [...]
>
> > ./stap -d /path/myapp-DDEBUG_UNWIND -e 'probe
> > process("/path/myapp").function("funtion_name").call
> {print_ubacktrace()}'
> > WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
> > retry with -DDEBUG_UNWIND)'
>
> One possibility is that this program, being compiled with your other
> gcc/glibc toolchain, is not getting identified to systemtap for purpose
> of extracting unwind/symbol data.
>
> > WARNING: Missing unwind data for a module, rerun with 'stap -d
> /usr/lib64/
> > libc-2.17.so'
> > WARNING: too many pending (warning) messages
>
> Yeah, that's a hint. The "-d FILE" and "--ldd" options should let you
> give systemtap enough hints to find the alternate toolchain shared
> libraries. Try:
>
> # stap --ldd -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -e 'probe
> ("myapp").......'
>
> If more missing library names come up as runtime warnings, rerun, and
> add them as -d PATHs too.
>
>
> stap --ldd -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -d
/opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2 -d
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 -e 'probe
process("myapp").function("func_name").call {print_ubacktrace()}'
WARNING: module /home/opt/compiler/gcc-8.2/lib/libc-2.21.so --ldd skipped:
unsupported interpreter: /opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2
WARNING: module myapp --ldd skipped: unsupported interpreter:
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2
Trigger probe point function
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
0x8789b6
> - FChE
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Using systemtap in an Environment with Multiple Versions of glibc
2023-07-29 12:46 ` wei gong
@ 2023-07-29 13:16 ` Frank Ch. Eigler
2023-07-30 4:32 ` wei gong
0 siblings, 1 reply; 5+ messages in thread
From: Frank Ch. Eigler @ 2023-07-29 13:16 UTC (permalink / raw)
To: wei gong; +Cc: systemtap
Hi -
> > stap --ldd -d /opt/compiler/gcc-8.2/usr/lib/libc.so.6 -d
> /opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2 -d
> /opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 -e 'probe
> process("myapp").function("func_name").call {print_ubacktrace()}'
> WARNING: module /home/opt/compiler/gcc-8.2/lib/libc-2.21.so --ldd skipped:
> unsupported interpreter: /opt/compiler/gcc-8.2/lib/ld-linux-x86-64.so.2
> WARNING: module myapp --ldd skipped: unsupported interpreter:
> /opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2
Ah yeah. Having a custom ld.so like that is unusual. Sorry I didn't
realize this aspect of your situation. You could try running ldd
yourself on your binary, and manually add "-d
/opt/compiler/..../lib..." flags for each.
- FChE
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Using systemtap in an Environment with Multiple Versions of glibc
2023-07-29 13:16 ` Frank Ch. Eigler
@ 2023-07-30 4:32 ` wei gong
0 siblings, 0 replies; 5+ messages in thread
From: wei gong @ 2023-07-30 4:32 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: systemtap
[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]
Hi
> Ah yeah. Having a custom ld.so like that is unusual. Sorry I didn't
> realize this aspect of your situation. You could try running ldd
> yourself on your binary, and manually add "-d
> /opt/compiler/..../lib..." flags for each.
>
> I tried running ldd on my binary file and manually adding ‘-d
/opt/compiler/…/lib…’ for each one.
stap -d /opt/compiler/gcc-8.2/usr/lib/libpixman-1.so.0 -d
/opt/compiler/gcc-8.2/usr/lib/libz.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libutil.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libglib-2.0.so.0 -d
/opt/compiler/gcc-8.2/usr/lib/libdl.so.2 -d
/opt/compiler/gcc-8.2/usr/lib/libcurl.so.4 -d
/opt/compiler/gcc-8.2/usr/lib/libaio.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libstdc++.so.6 -d
/opt/compiler/gcc-8.2/usr/lib/libgflags.so.2.1 -d
/opt/compiler/gcc-8.2/usr/lib/libprotobuf.so.8 -d
/opt/compiler/gcc-8.2/usr/lib/libleveldb.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libm.so.6 -d
/opt/compiler/gcc-8.2/usr/lib/libgcc_s.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libpthread.so.0 -d
/opt/compiler/gcc-8.2/usr/lib/libc.so.6 -d
/opt/compiler/gcc-8.2/usr/lib/libpcre.so.1 -d
/opt/compiler/gcc-8.2/usr/lib/libsnappy.so.1 -d
/opt/compiler/gcc-8.2/lib64/ld-linux-x86-64.so.2 -e 'probe
process("myapp").function("function").call {print_ubacktrace()}'
Trigger probe point function
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown;
retry with -DDEBUG_UNWIND)'
0x8789b6
Best Regards,
WeiGong
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-07-30 4:32 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-28 13:59 Using systemtap in an Environment with Multiple Versions of glibc wei gong
2023-07-28 19:21 ` Frank Ch. Eigler
2023-07-29 12:46 ` wei gong
2023-07-29 13:16 ` Frank Ch. Eigler
2023-07-30 4:32 ` wei gong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).