Re: stap server is not able to use

[Lee Eric <openlinuxsource@gmail.com>]

Thank you, Martin. After disabling SecureBoot everything works fine
now. You are a life saver, much appreciated.

Eric

On Wed, Dec 6, 2023 at 10:03 AM Martin Cermak <mcermak@redhat.com> wrote:
>
> Hi Eric,
>
> hmmm, I think the configuration of your test system isn't
> default, because on Fedora 39, the default is to use debuginfod,
> while your system apparently is trying to install debuginfo RPMs
> and then somehow fails to consume them.  I've tested your
> scenario with a fresh & up2date copy of Fedora 39 and it did work
> for me.
>
> One important thing is that you apparently use SecureBoot.  If
> you don't need that, disable it, and your life will become easier.
> If you need it though, here is how it did work for me:
>
> > root@fedora:~# rpm -qa | fgrep systemtap
> > systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64
> > systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64
> > systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64
> > systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64
> > root@fedora:~# yum install systemtap-server
> > ...
> > root@fedora:~# mokutil --sb-state
> > SecureBoot enabled
> > root@fedora:~# uname -r
> > 6.6.3-200.fc39.x86_64
> > root@fedora:~# stap-prep
> > Configuring for kernel release 6.6.3-200.fc39.x86_64
> > Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz debuginfo
> > Increasing DEBUGINFOD_TIMEOUT to 300 temporarily
> > Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593720
> > -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo
> > Download successful.  Assuming debuginfod server usage.
> > root@fedora:~# service stap-server start
> > Redirecting to /bin/systemctl start stap-server.service
> > root@fedora:~# netstat -tlp | grep stap
> > tcp6       0      0 [::]:38541              [::]:*                  LISTEN      21523/stap-serverd
> > root@fedora:~# SERVER_IP=127.0.0.1
> > root@fedora:~# SERVER_PORT=38541
> > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }'
> > Using a compile server.
> > Pass 1: parsed user script and 529 library scripts using 537264virt/292632res/15232shr/276680data kb, in 770usr/90sys/892real ms.
> > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305944res/15872shr/289352
> > # ...
> > # Here systemtap instructs you how to enroll a MOK key, I've lost these messages somehow, but
> > # see below how to proceed:
> > # ...
> > root@fedora:~# mokutil --import signing_key.x509
> > #
> > #  Now reboot, finish enrolling the MOK key and boot
> > #
> > #  Having your system configured now you can:
> > #
> > root@fedora:~# mokutil --sb-state
> > SecureBoot enabled
> > root@fedora:~# netstat -tlp | grep stap
> > root@fedora:~# service stap-server start start
> > Redirecting to /bin/systemctl start stap-server.service
> > root@fedora:~# netstat -tlp | grep stap
> > tcp6       0      0 [::]:36707              [::]:*                  LISTEN      1979/stap-serverd
> > root@fedora:~# SERVER_IP=127.0.0.1; SERVER_PORT=36707
> > root@fedora:~# stap --trust-servers=ssl,signer,all-users,no-prompt --use-server=$SERVER_IP:$SERVER_PORT
> > Adding trust in the following servers as an SSL peer for all users and as a module signer for all users
> >    host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown"
> > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }'
> > Using a compile server.
> > Pass 1: parsed user script and 529 library scripts using 537264virt/292504res/15104shr/276680data kb, in 760usr/100sys/929real ms.
> > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms.
> > Pass 3: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.c
> > Pass 4: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko
> > Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key <server>/.systemtap/ssl/server/moks
> > Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8c:d0:e4:50:f4:b1:ef:7f:4e"
> > Passes: via server  host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" using 267740virt/23952res/19856shr/3108data kb, in 30usr/0sys/1481real ms.
> > The kernel on your system requires modules to be signed for loading.
> > The module created by compiling your script must be signed by a systemtap compile-server.  [man stap-server]
> > --use-server was automatically selected in order to request compilation by a compile-server.
> > Pass 5: starting run.
> > hey
> > Pass 5: run completed in 10usr/50sys/948real ms.
> > root@fedora:~#
>
> So, as you can see above, it works for me.  For more info about
> using systemtap with SecureBoot, see here:
>
> https://sourceware.org/systemtap/wiki/SecureBoot
>
> HTH; Cheers,
> Martin
>
>
> On  Mon  2023-12-04  21:53 , Martin Cermak wrote:
> > Hi Eric,
> >
> > systemtap packages come with stap-prep command that should do it for you:
> >
> > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup
> >
> > Depending on your environment, modern stap-prep may use debuginfod
> > for you.  That way you might have needed debugging information
> > available without actually installing the debuginfo RPMs.
> >
> > https://sourceware.org/elfutils/Debuginfod.html
> >
> > Hope this helps,
> >
> > Martin
> >
> >
> > On  Mon  2023-12-04  13:57 , Lee Eric wrote:
> > > Hi Martin,
> > >
> > > Thanks for your reply and it seems no connection error on the compile
> > > server. However, do we have any updated steps on how to install kernel
> > > debuginfo RPM packages? I searched a lot and seems old methods to use
> > > debuginfo-install command does not work.
> > >
> > > Hui
> > >
> > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote:
> > > >
> > > > Hi Eric,
> > > >
> > > > On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> > > > > Hi,
> > > > >
> > > > > I just noticed my stap scripts need to run via stap-server and I
> > > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> > > > > to set up stap server. However, I feel like the error messages from
> > > > > the stap command is really odd:
> > > > >
> > > > > # stap --list-server=all
> > > > > ...
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > > ...
> > > > >
> > > > > And I'm using Fedora 39, so I would like to test if stap can connect
> > > > > to a server regardless the stap command ONLY accepting
> > > > > hostname/ip/cert serial which they are all the same.
> > > > >
> > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> > > > > ...
> > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> > > > > Using a compile server.
> > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> > > > > '/tmp/stapvTSXTA/client.zip' *
> > > > > Spawn waitpid result (0x0): 0
> > > > > Servers matching 127.0.0.1:44621:
> > > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > > version=unknown certinfo="unknown"
> > > > > All specified servers:
> > > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > > version=unknown certinfo="unknown"
> > > > > Unable to connect to a server.
> > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> > > > > in 0usr/0sys/4real ms.
> > > > > Passes: via server failed.  Try again with another '-v' option.
> > > > > The kernel on your system requires modules to be signed for loading.
> > > > > The module created by compiling your script must be signed by a
> > > > > systemtap compile-server.  [man stap-server]
> > > > > ...
> > > > >
> > > > > What's the meaning of that error exactly? Why stap cannot match one
> > > > > server in this case? I also did wireshark and I'm sure stap didn't
> > > > > talk to the tcp port 44621
> > > > >
> > > > > Is there any clue about this usage? Any help would be appreciated.
> > > >
> > > > I think you are missing a `stap --trust-servers ...` step.  We
> > > > have a simple testcase for stap server in Fedora CI:
> > > >
> > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh
> > > >
> > > > One of relatively fresh logs showing how it worked on Fedora 39
> > > > is here:
> > > >
> > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt
> > > >
> > > > Hope this helps,
> > > > Martin
> > > >
> > >
>