* stap server is not able to use @ 2023-12-03 18:03 Lee Eric 2023-12-04 9:08 ` Martin Cermak 0 siblings, 1 reply; 7+ messages in thread From: Lee Eric @ 2023-12-03 18:03 UTC (permalink / raw) To: systemtap Hi, I just noticed my stap scripts need to run via stap-server and I followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot to set up stap server. However, I feel like the error messages from the stap command is really odd: # stap --list-server=all ... host=thinkpad01.local address=127.0.0.1 port=44621 sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 certinfo="00:c1:73:c9:a1" host=thinkpad01.local address=127.0.0.1 port=44621 sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 certinfo="00:c1:73:c9:a1" host=thinkpad01.local address=127.0.0.1 port=44621 sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 certinfo="00:c1:73:c9:a1" host=thinkpad01.local address=127.0.0.1 port=44621 sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 certinfo="00:c1:73:c9:a1" host=thinkpad01.local address=127.0.0.1 port=44621 sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 certinfo="00:c1:73:c9:a1" host=thinkpad01.local address=127.0.0.1 port=44621 sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 certinfo="00:c1:73:c9:a1" ... And I'm using Fedora 39, so I would like to test if stap can connect to a server regardless the stap command ONLY accepting hostname/ip/cert serial which they are all the same. # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' ... Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" Using a compile server. Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr '/tmp/stapvTSXTA/client.zip' * Spawn waitpid result (0x0): 0 Servers matching 127.0.0.1:44621: host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" version=unknown certinfo="unknown" All specified servers: host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" version=unknown certinfo="unknown" Unable to connect to a server. Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, in 0usr/0sys/4real ms. Passes: via server failed. Try again with another '-v' option. The kernel on your system requires modules to be signed for loading. The module created by compiling your script must be signed by a systemtap compile-server. [man stap-server] ... What's the meaning of that error exactly? Why stap cannot match one server in this case? I also did wireshark and I'm sure stap didn't talk to the tcp port 44621 Is there any clue about this usage? Any help would be appreciated. Thanks. Eric ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: stap server is not able to use 2023-12-03 18:03 stap server is not able to use Lee Eric @ 2023-12-04 9:08 ` Martin Cermak 2023-12-04 18:57 ` Lee Eric 0 siblings, 1 reply; 7+ messages in thread From: Martin Cermak @ 2023-12-04 9:08 UTC (permalink / raw) To: Lee Eric; +Cc: systemtap Hi Eric, On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > Hi, > > I just noticed my stap scripts need to run via stap-server and I > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > to set up stap server. However, I feel like the error messages from > the stap command is really odd: > > # stap --list-server=all > ... > host=thinkpad01.local address=127.0.0.1 port=44621 > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > certinfo="00:c1:73:c9:a1" > host=thinkpad01.local address=127.0.0.1 port=44621 > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > certinfo="00:c1:73:c9:a1" > host=thinkpad01.local address=127.0.0.1 port=44621 > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > certinfo="00:c1:73:c9:a1" > host=thinkpad01.local address=127.0.0.1 port=44621 > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > certinfo="00:c1:73:c9:a1" > host=thinkpad01.local address=127.0.0.1 port=44621 > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > certinfo="00:c1:73:c9:a1" > host=thinkpad01.local address=127.0.0.1 port=44621 > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > certinfo="00:c1:73:c9:a1" > ... > > And I'm using Fedora 39, so I would like to test if stap can connect > to a server regardless the stap command ONLY accepting > hostname/ip/cert serial which they are all the same. > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > ... > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > Using a compile server. > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > '/tmp/stapvTSXTA/client.zip' * > Spawn waitpid result (0x0): 0 > Servers matching 127.0.0.1:44621: > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > version=unknown certinfo="unknown" > All specified servers: > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > version=unknown certinfo="unknown" > Unable to connect to a server. > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > in 0usr/0sys/4real ms. > Passes: via server failed. Try again with another '-v' option. > The kernel on your system requires modules to be signed for loading. > The module created by compiling your script must be signed by a > systemtap compile-server. [man stap-server] > ... > > What's the meaning of that error exactly? Why stap cannot match one > server in this case? I also did wireshark and I'm sure stap didn't > talk to the tcp port 44621 > > Is there any clue about this usage? Any help would be appreciated. I think you are missing a `stap --trust-servers ...` step. We have a simple testcase for stap server in Fedora CI: https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh One of relatively fresh logs showing how it worked on Fedora 39 is here: https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt Hope this helps, Martin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: stap server is not able to use 2023-12-04 9:08 ` Martin Cermak @ 2023-12-04 18:57 ` Lee Eric 2023-12-04 20:53 ` Martin Cermak 0 siblings, 1 reply; 7+ messages in thread From: Lee Eric @ 2023-12-04 18:57 UTC (permalink / raw) To: Martin Cermak; +Cc: systemtap Hi Martin, Thanks for your reply and it seems no connection error on the compile server. However, do we have any updated steps on how to install kernel debuginfo RPM packages? I searched a lot and seems old methods to use debuginfo-install command does not work. Hui On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote: > > Hi Eric, > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > Hi, > > > > I just noticed my stap scripts need to run via stap-server and I > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > > to set up stap server. However, I feel like the error messages from > > the stap command is really odd: > > > > # stap --list-server=all > > ... > > host=thinkpad01.local address=127.0.0.1 port=44621 > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > > certinfo="00:c1:73:c9:a1" > > host=thinkpad01.local address=127.0.0.1 port=44621 > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > > certinfo="00:c1:73:c9:a1" > > host=thinkpad01.local address=127.0.0.1 port=44621 > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > > certinfo="00:c1:73:c9:a1" > > host=thinkpad01.local address=127.0.0.1 port=44621 > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > > certinfo="00:c1:73:c9:a1" > > host=thinkpad01.local address=127.0.0.1 port=44621 > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > > certinfo="00:c1:73:c9:a1" > > host=thinkpad01.local address=127.0.0.1 port=44621 > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > > certinfo="00:c1:73:c9:a1" > > ... > > > > And I'm using Fedora 39, so I would like to test if stap can connect > > to a server regardless the stap command ONLY accepting > > hostname/ip/cert serial which they are all the same. > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > > ... > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > Using a compile server. > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > '/tmp/stapvTSXTA/client.zip' * > > Spawn waitpid result (0x0): 0 > > Servers matching 127.0.0.1:44621: > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > version=unknown certinfo="unknown" > > All specified servers: > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > version=unknown certinfo="unknown" > > Unable to connect to a server. > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > > in 0usr/0sys/4real ms. > > Passes: via server failed. Try again with another '-v' option. > > The kernel on your system requires modules to be signed for loading. > > The module created by compiling your script must be signed by a > > systemtap compile-server. [man stap-server] > > ... > > > > What's the meaning of that error exactly? Why stap cannot match one > > server in this case? I also did wireshark and I'm sure stap didn't > > talk to the tcp port 44621 > > > > Is there any clue about this usage? Any help would be appreciated. > > I think you are missing a `stap --trust-servers ...` step. We > have a simple testcase for stap server in Fedora CI: > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh > > One of relatively fresh logs showing how it worked on Fedora 39 > is here: > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt > > Hope this helps, > Martin > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: stap server is not able to use 2023-12-04 18:57 ` Lee Eric @ 2023-12-04 20:53 ` Martin Cermak 2023-12-05 1:08 ` Lee Eric 2023-12-06 15:03 ` Martin Cermak 0 siblings, 2 replies; 7+ messages in thread From: Martin Cermak @ 2023-12-04 20:53 UTC (permalink / raw) To: Lee Eric; +Cc: systemtap Hi Eric, systemtap packages come with stap-prep command that should do it for you: https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup Depending on your environment, modern stap-prep may use debuginfod for you. That way you might have needed debugging information available without actually installing the debuginfo RPMs. https://sourceware.org/elfutils/Debuginfod.html Hope this helps, Martin On Mon 2023-12-04 13:57 , Lee Eric wrote: > Hi Martin, > > Thanks for your reply and it seems no connection error on the compile > server. However, do we have any updated steps on how to install kernel > debuginfo RPM packages? I searched a lot and seems old methods to use > debuginfo-install command does not work. > > Hui > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote: > > > > Hi Eric, > > > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > > Hi, > > > > > > I just noticed my stap scripts need to run via stap-server and I > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > > > to set up stap server. However, I feel like the error messages from > > > the stap command is really odd: > > > > > > # stap --list-server=all > > > ... > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > > > certinfo="00:c1:73:c9:a1" > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > > > certinfo="00:c1:73:c9:a1" > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > > > certinfo="00:c1:73:c9:a1" > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > > > certinfo="00:c1:73:c9:a1" > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > > > certinfo="00:c1:73:c9:a1" > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > > > certinfo="00:c1:73:c9:a1" > > > ... > > > > > > And I'm using Fedora 39, so I would like to test if stap can connect > > > to a server regardless the stap command ONLY accepting > > > hostname/ip/cert serial which they are all the same. > > > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > > > ... > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > > Using a compile server. > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > > '/tmp/stapvTSXTA/client.zip' * > > > Spawn waitpid result (0x0): 0 > > > Servers matching 127.0.0.1:44621: > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > version=unknown certinfo="unknown" > > > All specified servers: > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > version=unknown certinfo="unknown" > > > Unable to connect to a server. > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > > > in 0usr/0sys/4real ms. > > > Passes: via server failed. Try again with another '-v' option. > > > The kernel on your system requires modules to be signed for loading. > > > The module created by compiling your script must be signed by a > > > systemtap compile-server. [man stap-server] > > > ... > > > > > > What's the meaning of that error exactly? Why stap cannot match one > > > server in this case? I also did wireshark and I'm sure stap didn't > > > talk to the tcp port 44621 > > > > > > Is there any clue about this usage? Any help would be appreciated. > > > > I think you are missing a `stap --trust-servers ...` step. We > > have a simple testcase for stap server in Fedora CI: > > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh > > > > One of relatively fresh logs showing how it worked on Fedora 39 > > is here: > > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt > > > > Hope this helps, > > Martin > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: stap server is not able to use 2023-12-04 20:53 ` Martin Cermak @ 2023-12-05 1:08 ` Lee Eric 2023-12-06 15:03 ` Martin Cermak 1 sibling, 0 replies; 7+ messages in thread From: Lee Eric @ 2023-12-05 1:08 UTC (permalink / raw) To: Martin Cermak; +Cc: systemtap Thanks Martin. However, it seems the latest Fedora 39 kernel-debuginfo package is not working: $ uname -r 6.6.3-200.fc39.x86_64 $ sudo stap-prep Configuring for kernel release 6.6.3-200.fc39.x86_64 Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS $ echo $? 0 $ sudo stap --use-server=127.0.0.1:36863 -v sock_traffic_mntr.stp 5 Using a compile server. Pass 1: parsed user script and 531 library scripts using 539356virt/293212res/15744shr/276808data kb, in 690usr/70sys/760real ms. error: cannot open Packages database in WARNING: cannot find module kernel debuginfo: invalid ELF file [man warning::debuginfo] Out of memory. Please check --rlimit-as and memory availability. std::bad_alloc Passes: via server host=unknown address=127.0.0.1 port=36863 sysinfo="unknown" version=unknown certinfo="unknown" using 269840virt/24840res/20616shr/3244data kb, in 20usr/0sys/1662real ms. Passes: via server failed. Try again with another '-v' option. The kernel on your system requires modules to be signed for loading. The module created by compiling your script must be signed by a systemtap compile-server. [man stap-server] --use-server was automatically selected in order to request compilation by a compile-server. Do we know if any method we can fix this issue? Thanks. Eric On Mon, Dec 4, 2023 at 3:53 PM Martin Cermak <mcermak@redhat.com> wrote: > > Hi Eric, > > systemtap packages come with stap-prep command that should do it for you: > > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup > > Depending on your environment, modern stap-prep may use debuginfod > for you. That way you might have needed debugging information > available without actually installing the debuginfo RPMs. > > https://sourceware.org/elfutils/Debuginfod.html > > Hope this helps, > > Martin > > > On Mon 2023-12-04 13:57 , Lee Eric wrote: > > Hi Martin, > > > > Thanks for your reply and it seems no connection error on the compile > > server. However, do we have any updated steps on how to install kernel > > debuginfo RPM packages? I searched a lot and seems old methods to use > > debuginfo-install command does not work. > > > > Hui > > > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote: > > > > > > Hi Eric, > > > > > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > > > Hi, > > > > > > > > I just noticed my stap scripts need to run via stap-server and I > > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > > > > to set up stap server. However, I feel like the error messages from > > > > the stap command is really odd: > > > > > > > > # stap --list-server=all > > > > ... > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > ... > > > > > > > > And I'm using Fedora 39, so I would like to test if stap can connect > > > > to a server regardless the stap command ONLY accepting > > > > hostname/ip/cert serial which they are all the same. > > > > > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > > > > ... > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > > > Using a compile server. > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > > > '/tmp/stapvTSXTA/client.zip' * > > > > Spawn waitpid result (0x0): 0 > > > > Servers matching 127.0.0.1:44621: > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > version=unknown certinfo="unknown" > > > > All specified servers: > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > version=unknown certinfo="unknown" > > > > Unable to connect to a server. > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > > > > in 0usr/0sys/4real ms. > > > > Passes: via server failed. Try again with another '-v' option. > > > > The kernel on your system requires modules to be signed for loading. > > > > The module created by compiling your script must be signed by a > > > > systemtap compile-server. [man stap-server] > > > > ... > > > > > > > > What's the meaning of that error exactly? Why stap cannot match one > > > > server in this case? I also did wireshark and I'm sure stap didn't > > > > talk to the tcp port 44621 > > > > > > > > Is there any clue about this usage? Any help would be appreciated. > > > > > > I think you are missing a `stap --trust-servers ...` step. We > > > have a simple testcase for stap server in Fedora CI: > > > > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh > > > > > > One of relatively fresh logs showing how it worked on Fedora 39 > > > is here: > > > > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt > > > > > > Hope this helps, > > > Martin > > > > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: stap server is not able to use 2023-12-04 20:53 ` Martin Cermak 2023-12-05 1:08 ` Lee Eric @ 2023-12-06 15:03 ` Martin Cermak 2023-12-06 16:00 ` Lee Eric 1 sibling, 1 reply; 7+ messages in thread From: Martin Cermak @ 2023-12-06 15:03 UTC (permalink / raw) To: Lee Eric; +Cc: systemtap Hi Eric, hmmm, I think the configuration of your test system isn't default, because on Fedora 39, the default is to use debuginfod, while your system apparently is trying to install debuginfo RPMs and then somehow fails to consume them. I've tested your scenario with a fresh & up2date copy of Fedora 39 and it did work for me. One important thing is that you apparently use SecureBoot. If you don't need that, disable it, and your life will become easier. If you need it though, here is how it did work for me: > root@fedora:~# rpm -qa | fgrep systemtap > systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64 > systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64 > systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64 > systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64 > root@fedora:~# yum install systemtap-server > ... > root@fedora:~# mokutil --sb-state > SecureBoot enabled > root@fedora:~# uname -r > 6.6.3-200.fc39.x86_64 > root@fedora:~# stap-prep > Configuring for kernel release 6.6.3-200.fc39.x86_64 > Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz debuginfo > Increasing DEBUGINFOD_TIMEOUT to 300 temporarily > Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593720 > -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo > Download successful. Assuming debuginfod server usage. > root@fedora:~# service stap-server start > Redirecting to /bin/systemctl start stap-server.service > root@fedora:~# netstat -tlp | grep stap > tcp6 0 0 [::]:38541 [::]:* LISTEN 21523/stap-serverd > root@fedora:~# SERVER_IP=127.0.0.1 > root@fedora:~# SERVER_PORT=38541 > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }' > Using a compile server. > Pass 1: parsed user script and 529 library scripts using 537264virt/292632res/15232shr/276680data kb, in 770usr/90sys/892real ms. > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305944res/15872shr/289352 > # ... > # Here systemtap instructs you how to enroll a MOK key, I've lost these messages somehow, but > # see below how to proceed: > # ... > root@fedora:~# mokutil --import signing_key.x509 > # > # Now reboot, finish enrolling the MOK key and boot > # > # Having your system configured now you can: > # > root@fedora:~# mokutil --sb-state > SecureBoot enabled > root@fedora:~# netstat -tlp | grep stap > root@fedora:~# service stap-server start start > Redirecting to /bin/systemctl start stap-server.service > root@fedora:~# netstat -tlp | grep stap > tcp6 0 0 [::]:36707 [::]:* LISTEN 1979/stap-serverd > root@fedora:~# SERVER_IP=127.0.0.1; SERVER_PORT=36707 > root@fedora:~# stap --trust-servers=ssl,signer,all-users,no-prompt --use-server=$SERVER_IP:$SERVER_PORT > Adding trust in the following servers as an SSL peer for all users and as a module signer for all users > host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }' > Using a compile server. > Pass 1: parsed user script and 529 library scripts using 537264virt/292504res/15104shr/276680data kb, in 760usr/100sys/929real ms. > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms. > Pass 3: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.c > Pass 4: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko > Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key <server>/.systemtap/ssl/server/moks > Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8c:d0:e4:50:f4:b1:ef:7f:4e" > Passes: via server host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" using 267740virt/23952res/19856shr/3108data kb, in 30usr/0sys/1481real ms. > The kernel on your system requires modules to be signed for loading. > The module created by compiling your script must be signed by a systemtap compile-server. [man stap-server] > --use-server was automatically selected in order to request compilation by a compile-server. > Pass 5: starting run. > hey > Pass 5: run completed in 10usr/50sys/948real ms. > root@fedora:~# So, as you can see above, it works for me. For more info about using systemtap with SecureBoot, see here: https://sourceware.org/systemtap/wiki/SecureBoot HTH; Cheers, Martin On Mon 2023-12-04 21:53 , Martin Cermak wrote: > Hi Eric, > > systemtap packages come with stap-prep command that should do it for you: > > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup > > Depending on your environment, modern stap-prep may use debuginfod > for you. That way you might have needed debugging information > available without actually installing the debuginfo RPMs. > > https://sourceware.org/elfutils/Debuginfod.html > > Hope this helps, > > Martin > > > On Mon 2023-12-04 13:57 , Lee Eric wrote: > > Hi Martin, > > > > Thanks for your reply and it seems no connection error on the compile > > server. However, do we have any updated steps on how to install kernel > > debuginfo RPM packages? I searched a lot and seems old methods to use > > debuginfo-install command does not work. > > > > Hui > > > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote: > > > > > > Hi Eric, > > > > > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > > > Hi, > > > > > > > > I just noticed my stap scripts need to run via stap-server and I > > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > > > > to set up stap server. However, I feel like the error messages from > > > > the stap command is really odd: > > > > > > > > # stap --list-server=all > > > > ... > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > ... > > > > > > > > And I'm using Fedora 39, so I would like to test if stap can connect > > > > to a server regardless the stap command ONLY accepting > > > > hostname/ip/cert serial which they are all the same. > > > > > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > > > > ... > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > > > Using a compile server. > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > > > '/tmp/stapvTSXTA/client.zip' * > > > > Spawn waitpid result (0x0): 0 > > > > Servers matching 127.0.0.1:44621: > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > version=unknown certinfo="unknown" > > > > All specified servers: > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > version=unknown certinfo="unknown" > > > > Unable to connect to a server. > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > > > > in 0usr/0sys/4real ms. > > > > Passes: via server failed. Try again with another '-v' option. > > > > The kernel on your system requires modules to be signed for loading. > > > > The module created by compiling your script must be signed by a > > > > systemtap compile-server. [man stap-server] > > > > ... > > > > > > > > What's the meaning of that error exactly? Why stap cannot match one > > > > server in this case? I also did wireshark and I'm sure stap didn't > > > > talk to the tcp port 44621 > > > > > > > > Is there any clue about this usage? Any help would be appreciated. > > > > > > I think you are missing a `stap --trust-servers ...` step. We > > > have a simple testcase for stap server in Fedora CI: > > > > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh > > > > > > One of relatively fresh logs showing how it worked on Fedora 39 > > > is here: > > > > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt > > > > > > Hope this helps, > > > Martin > > > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: stap server is not able to use 2023-12-06 15:03 ` Martin Cermak @ 2023-12-06 16:00 ` Lee Eric 0 siblings, 0 replies; 7+ messages in thread From: Lee Eric @ 2023-12-06 16:00 UTC (permalink / raw) To: Martin Cermak; +Cc: systemtap Thank you, Martin. After disabling SecureBoot everything works fine now. You are a life saver, much appreciated. Eric On Wed, Dec 6, 2023 at 10:03 AM Martin Cermak <mcermak@redhat.com> wrote: > > Hi Eric, > > hmmm, I think the configuration of your test system isn't > default, because on Fedora 39, the default is to use debuginfod, > while your system apparently is trying to install debuginfo RPMs > and then somehow fails to consume them. I've tested your > scenario with a fresh & up2date copy of Fedora 39 and it did work > for me. > > One important thing is that you apparently use SecureBoot. If > you don't need that, disable it, and your life will become easier. > If you need it though, here is how it did work for me: > > > root@fedora:~# rpm -qa | fgrep systemtap > > systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64 > > systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64 > > systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64 > > systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64 > > root@fedora:~# yum install systemtap-server > > ... > > root@fedora:~# mokutil --sb-state > > SecureBoot enabled > > root@fedora:~# uname -r > > 6.6.3-200.fc39.x86_64 > > root@fedora:~# stap-prep > > Configuring for kernel release 6.6.3-200.fc39.x86_64 > > Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz debuginfo > > Increasing DEBUGINFOD_TIMEOUT to 300 temporarily > > Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593720 > > -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo > > Download successful. Assuming debuginfod server usage. > > root@fedora:~# service stap-server start > > Redirecting to /bin/systemctl start stap-server.service > > root@fedora:~# netstat -tlp | grep stap > > tcp6 0 0 [::]:38541 [::]:* LISTEN 21523/stap-serverd > > root@fedora:~# SERVER_IP=127.0.0.1 > > root@fedora:~# SERVER_PORT=38541 > > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }' > > Using a compile server. > > Pass 1: parsed user script and 529 library scripts using 537264virt/292632res/15232shr/276680data kb, in 770usr/90sys/892real ms. > > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305944res/15872shr/289352 > > # ... > > # Here systemtap instructs you how to enroll a MOK key, I've lost these messages somehow, but > > # see below how to proceed: > > # ... > > root@fedora:~# mokutil --import signing_key.x509 > > # > > # Now reboot, finish enrolling the MOK key and boot > > # > > # Having your system configured now you can: > > # > > root@fedora:~# mokutil --sb-state > > SecureBoot enabled > > root@fedora:~# netstat -tlp | grep stap > > root@fedora:~# service stap-server start start > > Redirecting to /bin/systemctl start stap-server.service > > root@fedora:~# netstat -tlp | grep stap > > tcp6 0 0 [::]:36707 [::]:* LISTEN 1979/stap-serverd > > root@fedora:~# SERVER_IP=127.0.0.1; SERVER_PORT=36707 > > root@fedora:~# stap --trust-servers=ssl,signer,all-users,no-prompt --use-server=$SERVER_IP:$SERVER_PORT > > Adding trust in the following servers as an SSL peer for all users and as a module signer for all users > > host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" > > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }' > > Using a compile server. > > Pass 1: parsed user script and 529 library scripts using 537264virt/292504res/15104shr/276680data kb, in 760usr/100sys/929real ms. > > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms. > > Pass 3: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.c > > Pass 4: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko > > Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key <server>/.systemtap/ssl/server/moks > > Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8c:d0:e4:50:f4:b1:ef:7f:4e" > > Passes: via server host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" using 267740virt/23952res/19856shr/3108data kb, in 30usr/0sys/1481real ms. > > The kernel on your system requires modules to be signed for loading. > > The module created by compiling your script must be signed by a systemtap compile-server. [man stap-server] > > --use-server was automatically selected in order to request compilation by a compile-server. > > Pass 5: starting run. > > hey > > Pass 5: run completed in 10usr/50sys/948real ms. > > root@fedora:~# > > So, as you can see above, it works for me. For more info about > using systemtap with SecureBoot, see here: > > https://sourceware.org/systemtap/wiki/SecureBoot > > HTH; Cheers, > Martin > > > On Mon 2023-12-04 21:53 , Martin Cermak wrote: > > Hi Eric, > > > > systemtap packages come with stap-prep command that should do it for you: > > > > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup > > > > Depending on your environment, modern stap-prep may use debuginfod > > for you. That way you might have needed debugging information > > available without actually installing the debuginfo RPMs. > > > > https://sourceware.org/elfutils/Debuginfod.html > > > > Hope this helps, > > > > Martin > > > > > > On Mon 2023-12-04 13:57 , Lee Eric wrote: > > > Hi Martin, > > > > > > Thanks for your reply and it seems no connection error on the compile > > > server. However, do we have any updated steps on how to install kernel > > > debuginfo RPM packages? I searched a lot and seems old methods to use > > > debuginfo-install command does not work. > > > > > > Hui > > > > > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote: > > > > > > > > Hi Eric, > > > > > > > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > > > > Hi, > > > > > > > > > > I just noticed my stap scripts need to run via stap-server and I > > > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > > > > > to set up stap server. However, I feel like the error messages from > > > > > the stap command is really odd: > > > > > > > > > > # stap --list-server=all > > > > > ... > > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > > > > > certinfo="00:c1:73:c9:a1" > > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > > > > > certinfo="00:c1:73:c9:a1" > > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > > > > > certinfo="00:c1:73:c9:a1" > > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > > > > > certinfo="00:c1:73:c9:a1" > > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > > > > > certinfo="00:c1:73:c9:a1" > > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > > > > > certinfo="00:c1:73:c9:a1" > > > > > ... > > > > > > > > > > And I'm using Fedora 39, so I would like to test if stap can connect > > > > > to a server regardless the stap command ONLY accepting > > > > > hostname/ip/cert serial which they are all the same. > > > > > > > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > > > > > ... > > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > > > > Using a compile server. > > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > > > > '/tmp/stapvTSXTA/client.zip' * > > > > > Spawn waitpid result (0x0): 0 > > > > > Servers matching 127.0.0.1:44621: > > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > > version=unknown certinfo="unknown" > > > > > All specified servers: > > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > > version=unknown certinfo="unknown" > > > > > Unable to connect to a server. > > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > > > > > in 0usr/0sys/4real ms. > > > > > Passes: via server failed. Try again with another '-v' option. > > > > > The kernel on your system requires modules to be signed for loading. > > > > > The module created by compiling your script must be signed by a > > > > > systemtap compile-server. [man stap-server] > > > > > ... > > > > > > > > > > What's the meaning of that error exactly? Why stap cannot match one > > > > > server in this case? I also did wireshark and I'm sure stap didn't > > > > > talk to the tcp port 44621 > > > > > > > > > > Is there any clue about this usage? Any help would be appreciated. > > > > > > > > I think you are missing a `stap --trust-servers ...` step. We > > > > have a simple testcase for stap server in Fedora CI: > > > > > > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh > > > > > > > > One of relatively fresh logs showing how it worked on Fedora 39 > > > > is here: > > > > > > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt > > > > > > > > Hope this helps, > > > > Martin > > > > > > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-12-06 16:00 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-12-03 18:03 stap server is not able to use Lee Eric 2023-12-04 9:08 ` Martin Cermak 2023-12-04 18:57 ` Lee Eric 2023-12-04 20:53 ` Martin Cermak 2023-12-05 1:08 ` Lee Eric 2023-12-06 15:03 ` Martin Cermak 2023-12-06 16:00 ` Lee Eric
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).