stap server is not able to use

stap server is not able to use

[Lee Eric]

Hi,

I just noticed my stap scripts need to run via stap-server and I
followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
to set up stap server. However, I feel like the error messages from
the stap command is really odd:

# stap --list-server=all
...
 host=thinkpad01.local address=127.0.0.1 port=44621
sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
certinfo="00:c1:73:c9:a1"
 host=thinkpad01.local address=127.0.0.1 port=44621
sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
certinfo="00:c1:73:c9:a1"
 host=thinkpad01.local address=127.0.0.1 port=44621
sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
certinfo="00:c1:73:c9:a1"
 host=thinkpad01.local address=127.0.0.1 port=44621
sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
certinfo="00:c1:73:c9:a1"
 host=thinkpad01.local address=127.0.0.1 port=44621
sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
certinfo="00:c1:73:c9:a1"
 host=thinkpad01.local address=127.0.0.1 port=44621
sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
certinfo="00:c1:73:c9:a1"
...

And I'm using Fedora 39, so I would like to test if stap can connect
to a server regardless the stap command ONLY accepting
hostname/ip/cert serial which they are all the same.

# stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
...
Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
Using a compile server.
Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
'/tmp/stapvTSXTA/client.zip' *
Spawn waitpid result (0x0): 0
Servers matching 127.0.0.1:44621:
 host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
version=unknown certinfo="unknown"
All specified servers:
 host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
version=unknown certinfo="unknown"
Unable to connect to a server.
Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
in 0usr/0sys/4real ms.
Passes: via server failed.  Try again with another '-v' option.
The kernel on your system requires modules to be signed for loading.
The module created by compiling your script must be signed by a
systemtap compile-server.  [man stap-server]
...

What's the meaning of that error exactly? Why stap cannot match one
server in this case? I also did wireshark and I'm sure stap didn't
talk to the tcp port 44621

Is there any clue about this usage? Any help would be appreciated.

Thanks.

Eric

Re: stap server is not able to use

[Martin Cermak]

Hi Eric,

On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> Hi,
> 
> I just noticed my stap scripts need to run via stap-server and I
> followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> to set up stap server. However, I feel like the error messages from
> the stap command is really odd:
> 
> # stap --list-server=all
> ...
>  host=thinkpad01.local address=127.0.0.1 port=44621
> sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> certinfo="00:c1:73:c9:a1"
>  host=thinkpad01.local address=127.0.0.1 port=44621
> sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> certinfo="00:c1:73:c9:a1"
>  host=thinkpad01.local address=127.0.0.1 port=44621
> sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> certinfo="00:c1:73:c9:a1"
>  host=thinkpad01.local address=127.0.0.1 port=44621
> sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> certinfo="00:c1:73:c9:a1"
>  host=thinkpad01.local address=127.0.0.1 port=44621
> sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> certinfo="00:c1:73:c9:a1"
>  host=thinkpad01.local address=127.0.0.1 port=44621
> sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> certinfo="00:c1:73:c9:a1"
> ...
> 
> And I'm using Fedora 39, so I would like to test if stap can connect
> to a server regardless the stap command ONLY accepting
> hostname/ip/cert serial which they are all the same.
> 
> # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> ...
> Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> Using a compile server.
> Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> '/tmp/stapvTSXTA/client.zip' *
> Spawn waitpid result (0x0): 0
> Servers matching 127.0.0.1:44621:
>  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> version=unknown certinfo="unknown"
> All specified servers:
>  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> version=unknown certinfo="unknown"
> Unable to connect to a server.
> Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> in 0usr/0sys/4real ms.
> Passes: via server failed.  Try again with another '-v' option.
> The kernel on your system requires modules to be signed for loading.
> The module created by compiling your script must be signed by a
> systemtap compile-server.  [man stap-server]
> ...
> 
> What's the meaning of that error exactly? Why stap cannot match one
> server in this case? I also did wireshark and I'm sure stap didn't
> talk to the tcp port 44621
> 
> Is there any clue about this usage? Any help would be appreciated.

I think you are missing a `stap --trust-servers ...` step.  We
have a simple testcase for stap server in Fedora CI:

https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh

One of relatively fresh logs showing how it worked on Fedora 39
is here:

https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt

Hope this helps,
Martin

Re: stap server is not able to use

[Lee Eric]

Hi Martin,

Thanks for your reply and it seems no connection error on the compile
server. However, do we have any updated steps on how to install kernel
debuginfo RPM packages? I searched a lot and seems old methods to use
debuginfo-install command does not work.

Hui

On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote:
>
> Hi Eric,
>
> On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> > Hi,
> >
> > I just noticed my stap scripts need to run via stap-server and I
> > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> > to set up stap server. However, I feel like the error messages from
> > the stap command is really odd:
> >
> > # stap --list-server=all
> > ...
> >  host=thinkpad01.local address=127.0.0.1 port=44621
> > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> > certinfo="00:c1:73:c9:a1"
> >  host=thinkpad01.local address=127.0.0.1 port=44621
> > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> > certinfo="00:c1:73:c9:a1"
> >  host=thinkpad01.local address=127.0.0.1 port=44621
> > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> > certinfo="00:c1:73:c9:a1"
> >  host=thinkpad01.local address=127.0.0.1 port=44621
> > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> > certinfo="00:c1:73:c9:a1"
> >  host=thinkpad01.local address=127.0.0.1 port=44621
> > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> > certinfo="00:c1:73:c9:a1"
> >  host=thinkpad01.local address=127.0.0.1 port=44621
> > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> > certinfo="00:c1:73:c9:a1"
> > ...
> >
> > And I'm using Fedora 39, so I would like to test if stap can connect
> > to a server regardless the stap command ONLY accepting
> > hostname/ip/cert serial which they are all the same.
> >
> > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> > ...
> > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> > Using a compile server.
> > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> > '/tmp/stapvTSXTA/client.zip' *
> > Spawn waitpid result (0x0): 0
> > Servers matching 127.0.0.1:44621:
> >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > version=unknown certinfo="unknown"
> > All specified servers:
> >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > version=unknown certinfo="unknown"
> > Unable to connect to a server.
> > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> > in 0usr/0sys/4real ms.
> > Passes: via server failed.  Try again with another '-v' option.
> > The kernel on your system requires modules to be signed for loading.
> > The module created by compiling your script must be signed by a
> > systemtap compile-server.  [man stap-server]
> > ...
> >
> > What's the meaning of that error exactly? Why stap cannot match one
> > server in this case? I also did wireshark and I'm sure stap didn't
> > talk to the tcp port 44621
> >
> > Is there any clue about this usage? Any help would be appreciated.
>
> I think you are missing a `stap --trust-servers ...` step.  We
> have a simple testcase for stap server in Fedora CI:
>
> https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh
>
> One of relatively fresh logs showing how it worked on Fedora 39
> is here:
>
> https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt
>
> Hope this helps,
> Martin
>

Re: stap server is not able to use

[Martin Cermak]

Hi Eric,

systemtap packages come with stap-prep command that should do it for you:

https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup

Depending on your environment, modern stap-prep may use debuginfod
for you.  That way you might have needed debugging information
available without actually installing the debuginfo RPMs.

https://sourceware.org/elfutils/Debuginfod.html

Hope this helps,

Martin


On  Mon  2023-12-04  13:57 , Lee Eric wrote:
> Hi Martin,
> 
> Thanks for your reply and it seems no connection error on the compile
> server. However, do we have any updated steps on how to install kernel
> debuginfo RPM packages? I searched a lot and seems old methods to use
> debuginfo-install command does not work.
> 
> Hui
> 
> On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote:
> >
> > Hi Eric,
> >
> > On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> > > Hi,
> > >
> > > I just noticed my stap scripts need to run via stap-server and I
> > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> > > to set up stap server. However, I feel like the error messages from
> > > the stap command is really odd:
> > >
> > > # stap --list-server=all
> > > ...
> > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> > > certinfo="00:c1:73:c9:a1"
> > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> > > certinfo="00:c1:73:c9:a1"
> > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> > > certinfo="00:c1:73:c9:a1"
> > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> > > certinfo="00:c1:73:c9:a1"
> > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> > > certinfo="00:c1:73:c9:a1"
> > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> > > certinfo="00:c1:73:c9:a1"
> > > ...
> > >
> > > And I'm using Fedora 39, so I would like to test if stap can connect
> > > to a server regardless the stap command ONLY accepting
> > > hostname/ip/cert serial which they are all the same.
> > >
> > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> > > ...
> > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> > > Using a compile server.
> > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> > > '/tmp/stapvTSXTA/client.zip' *
> > > Spawn waitpid result (0x0): 0
> > > Servers matching 127.0.0.1:44621:
> > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > version=unknown certinfo="unknown"
> > > All specified servers:
> > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > version=unknown certinfo="unknown"
> > > Unable to connect to a server.
> > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> > > in 0usr/0sys/4real ms.
> > > Passes: via server failed.  Try again with another '-v' option.
> > > The kernel on your system requires modules to be signed for loading.
> > > The module created by compiling your script must be signed by a
> > > systemtap compile-server.  [man stap-server]
> > > ...
> > >
> > > What's the meaning of that error exactly? Why stap cannot match one
> > > server in this case? I also did wireshark and I'm sure stap didn't
> > > talk to the tcp port 44621
> > >
> > > Is there any clue about this usage? Any help would be appreciated.
> >
> > I think you are missing a `stap --trust-servers ...` step.  We
> > have a simple testcase for stap server in Fedora CI:
> >
> > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh
> >
> > One of relatively fresh logs showing how it worked on Fedora 39
> > is here:
> >
> > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt
> >
> > Hope this helps,
> > Martin
> >
> 

Re: stap server is not able to use

[Lee Eric]

Thanks Martin.

However, it seems the latest Fedora 39 kernel-debuginfo package is not working:

$ uname -r
6.6.3-200.fc39.x86_64
$ sudo stap-prep
Configuring for kernel release 6.6.3-200.fc39.x86_64
Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS
$ echo $?
0
$ sudo stap --use-server=127.0.0.1:36863 -v sock_traffic_mntr.stp 5
Using a compile server.
Pass 1: parsed user script and 531 library scripts using
539356virt/293212res/15744shr/276808data kb, in 690usr/70sys/760real
ms.
error: cannot open Packages database in
WARNING: cannot find module kernel debuginfo: invalid ELF file [man
warning::debuginfo]
Out of memory.   Please check --rlimit-as and memory availability.
std::bad_alloc
Passes: via server  host=unknown address=127.0.0.1 port=36863
sysinfo="unknown" version=unknown certinfo="unknown" using
269840virt/24840res/20616shr/3244data kb, in 20usr/0sys/1662real ms.
Passes: via server failed.  Try again with another '-v' option.
The kernel on your system requires modules to be signed for loading.
The module created by compiling your script must be signed by a
systemtap compile-server.  [man stap-server]
--use-server was automatically selected in order to request
compilation by a compile-server.

Do we know if any method we can fix this issue?

Thanks.

Eric

On Mon, Dec 4, 2023 at 3:53 PM Martin Cermak <mcermak@redhat.com> wrote:
>
> Hi Eric,
>
> systemtap packages come with stap-prep command that should do it for you:
>
> https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup
>
> Depending on your environment, modern stap-prep may use debuginfod
> for you.  That way you might have needed debugging information
> available without actually installing the debuginfo RPMs.
>
> https://sourceware.org/elfutils/Debuginfod.html
>
> Hope this helps,
>
> Martin
>
>
> On  Mon  2023-12-04  13:57 , Lee Eric wrote:
> > Hi Martin,
> >
> > Thanks for your reply and it seems no connection error on the compile
> > server. However, do we have any updated steps on how to install kernel
> > debuginfo RPM packages? I searched a lot and seems old methods to use
> > debuginfo-install command does not work.
> >
> > Hui
> >
> > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote:
> > >
> > > Hi Eric,
> > >
> > > On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> > > > Hi,
> > > >
> > > > I just noticed my stap scripts need to run via stap-server and I
> > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> > > > to set up stap server. However, I feel like the error messages from
> > > > the stap command is really odd:
> > > >
> > > > # stap --list-server=all
> > > > ...
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > > ...
> > > >
> > > > And I'm using Fedora 39, so I would like to test if stap can connect
> > > > to a server regardless the stap command ONLY accepting
> > > > hostname/ip/cert serial which they are all the same.
> > > >
> > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> > > > ...
> > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> > > > Using a compile server.
> > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> > > > '/tmp/stapvTSXTA/client.zip' *
> > > > Spawn waitpid result (0x0): 0
> > > > Servers matching 127.0.0.1:44621:
> > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > version=unknown certinfo="unknown"
> > > > All specified servers:
> > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > version=unknown certinfo="unknown"
> > > > Unable to connect to a server.
> > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> > > > in 0usr/0sys/4real ms.
> > > > Passes: via server failed.  Try again with another '-v' option.
> > > > The kernel on your system requires modules to be signed for loading.
> > > > The module created by compiling your script must be signed by a
> > > > systemtap compile-server.  [man stap-server]
> > > > ...
> > > >
> > > > What's the meaning of that error exactly? Why stap cannot match one
> > > > server in this case? I also did wireshark and I'm sure stap didn't
> > > > talk to the tcp port 44621
> > > >
> > > > Is there any clue about this usage? Any help would be appreciated.
> > >
> > > I think you are missing a `stap --trust-servers ...` step.  We
> > > have a simple testcase for stap server in Fedora CI:
> > >
> > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh
> > >
> > > One of relatively fresh logs showing how it worked on Fedora 39
> > > is here:
> > >
> > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt
> > >
> > > Hope this helps,
> > > Martin
> > >
> >
>

Re: stap server is not able to use

[Martin Cermak]

Hi Eric,

hmmm, I think the configuration of your test system isn't
default, because on Fedora 39, the default is to use debuginfod,
while your system apparently is trying to install debuginfo RPMs
and then somehow fails to consume them.  I've tested your
scenario with a fresh & up2date copy of Fedora 39 and it did work
for me.

One important thing is that you apparently use SecureBoot.  If
you don't need that, disable it, and your life will become easier.
If you need it though, here is how it did work for me:

> root@fedora:~# rpm -qa | fgrep systemtap
> systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64
> systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64
> systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64
> systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64
> root@fedora:~# yum install systemtap-server
> ...
> root@fedora:~# mokutil --sb-state
> SecureBoot enabled
> root@fedora:~# uname -r
> 6.6.3-200.fc39.x86_64
> root@fedora:~# stap-prep
> Configuring for kernel release 6.6.3-200.fc39.x86_64
> Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz debuginfo
> Increasing DEBUGINFOD_TIMEOUT to 300 temporarily
> Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593720
> -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo
> Download successful.  Assuming debuginfod server usage.
> root@fedora:~# service stap-server start
> Redirecting to /bin/systemctl start stap-server.service
> root@fedora:~# netstat -tlp | grep stap
> tcp6       0      0 [::]:38541              [::]:*                  LISTEN      21523/stap-serverd
> root@fedora:~# SERVER_IP=127.0.0.1
> root@fedora:~# SERVER_PORT=38541
> root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }'
> Using a compile server.
> Pass 1: parsed user script and 529 library scripts using 537264virt/292632res/15232shr/276680data kb, in 770usr/90sys/892real ms.
> Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305944res/15872shr/289352
> # ...
> # Here systemtap instructs you how to enroll a MOK key, I've lost these messages somehow, but
> # see below how to proceed:
> # ...
> root@fedora:~# mokutil --import signing_key.x509
> #
> #  Now reboot, finish enrolling the MOK key and boot
> #
> #  Having your system configured now you can:
> #
> root@fedora:~# mokutil --sb-state
> SecureBoot enabled
> root@fedora:~# netstat -tlp | grep stap
> root@fedora:~# service stap-server start start
> Redirecting to /bin/systemctl start stap-server.service
> root@fedora:~# netstat -tlp | grep stap
> tcp6       0      0 [::]:36707              [::]:*                  LISTEN      1979/stap-serverd   
> root@fedora:~# SERVER_IP=127.0.0.1; SERVER_PORT=36707
> root@fedora:~# stap --trust-servers=ssl,signer,all-users,no-prompt --use-server=$SERVER_IP:$SERVER_PORT
> Adding trust in the following servers as an SSL peer for all users and as a module signer for all users
>    host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown"
> root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }'
> Using a compile server.
> Pass 1: parsed user script and 529 library scripts using 537264virt/292504res/15104shr/276680data kb, in 760usr/100sys/929real ms.
> Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms.
> Pass 3: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.c
> Pass 4: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko
> Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key <server>/.systemtap/ssl/server/moks
> Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8c:d0:e4:50:f4:b1:ef:7f:4e"
> Passes: via server  host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" using 267740virt/23952res/19856shr/3108data kb, in 30usr/0sys/1481real ms.
> The kernel on your system requires modules to be signed for loading.
> The module created by compiling your script must be signed by a systemtap compile-server.  [man stap-server]
> --use-server was automatically selected in order to request compilation by a compile-server.
> Pass 5: starting run.
> hey
> Pass 5: run completed in 10usr/50sys/948real ms.
> root@fedora:~#

So, as you can see above, it works for me.  For more info about
using systemtap with SecureBoot, see here:

https://sourceware.org/systemtap/wiki/SecureBoot

HTH; Cheers,
Martin


On  Mon  2023-12-04  21:53 , Martin Cermak wrote:
> Hi Eric,
> 
> systemtap packages come with stap-prep command that should do it for you:
> 
> https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup
> 
> Depending on your environment, modern stap-prep may use debuginfod
> for you.  That way you might have needed debugging information
> available without actually installing the debuginfo RPMs.
> 
> https://sourceware.org/elfutils/Debuginfod.html
> 
> Hope this helps,
> 
> Martin
> 
> 
> On  Mon  2023-12-04  13:57 , Lee Eric wrote:
> > Hi Martin,
> > 
> > Thanks for your reply and it seems no connection error on the compile
> > server. However, do we have any updated steps on how to install kernel
> > debuginfo RPM packages? I searched a lot and seems old methods to use
> > debuginfo-install command does not work.
> > 
> > Hui
> > 
> > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote:
> > >
> > > Hi Eric,
> > >
> > > On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> > > > Hi,
> > > >
> > > > I just noticed my stap scripts need to run via stap-server and I
> > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> > > > to set up stap server. However, I feel like the error messages from
> > > > the stap command is really odd:
> > > >
> > > > # stap --list-server=all
> > > > ...
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> > > > certinfo="00:c1:73:c9:a1"
> > > > ...
> > > >
> > > > And I'm using Fedora 39, so I would like to test if stap can connect
> > > > to a server regardless the stap command ONLY accepting
> > > > hostname/ip/cert serial which they are all the same.
> > > >
> > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> > > > ...
> > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> > > > Using a compile server.
> > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> > > > '/tmp/stapvTSXTA/client.zip' *
> > > > Spawn waitpid result (0x0): 0
> > > > Servers matching 127.0.0.1:44621:
> > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > version=unknown certinfo="unknown"
> > > > All specified servers:
> > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > version=unknown certinfo="unknown"
> > > > Unable to connect to a server.
> > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> > > > in 0usr/0sys/4real ms.
> > > > Passes: via server failed.  Try again with another '-v' option.
> > > > The kernel on your system requires modules to be signed for loading.
> > > > The module created by compiling your script must be signed by a
> > > > systemtap compile-server.  [man stap-server]
> > > > ...
> > > >
> > > > What's the meaning of that error exactly? Why stap cannot match one
> > > > server in this case? I also did wireshark and I'm sure stap didn't
> > > > talk to the tcp port 44621
> > > >
> > > > Is there any clue about this usage? Any help would be appreciated.
> > >
> > > I think you are missing a `stap --trust-servers ...` step.  We
> > > have a simple testcase for stap server in Fedora CI:
> > >
> > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh
> > >
> > > One of relatively fresh logs showing how it worked on Fedora 39
> > > is here:
> > >
> > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt
> > >
> > > Hope this helps,
> > > Martin
> > >
> > 

Re: stap server is not able to use

[Lee Eric]

Thank you, Martin. After disabling SecureBoot everything works fine
now. You are a life saver, much appreciated.

Eric

On Wed, Dec 6, 2023 at 10:03 AM Martin Cermak <mcermak@redhat.com> wrote:
>
> Hi Eric,
>
> hmmm, I think the configuration of your test system isn't
> default, because on Fedora 39, the default is to use debuginfod,
> while your system apparently is trying to install debuginfo RPMs
> and then somehow fails to consume them.  I've tested your
> scenario with a fresh & up2date copy of Fedora 39 and it did work
> for me.
>
> One important thing is that you apparently use SecureBoot.  If
> you don't need that, disable it, and your life will become easier.
> If you need it though, here is how it did work for me:
>
> > root@fedora:~# rpm -qa | fgrep systemtap
> > systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64
> > systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64
> > systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64
> > systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64
> > root@fedora:~# yum install systemtap-server
> > ...
> > root@fedora:~# mokutil --sb-state
> > SecureBoot enabled
> > root@fedora:~# uname -r
> > 6.6.3-200.fc39.x86_64
> > root@fedora:~# stap-prep
> > Configuring for kernel release 6.6.3-200.fc39.x86_64
> > Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz debuginfo
> > Increasing DEBUGINFOD_TIMEOUT to 300 temporarily
> > Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593720
> > -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo
> > Download successful.  Assuming debuginfod server usage.
> > root@fedora:~# service stap-server start
> > Redirecting to /bin/systemctl start stap-server.service
> > root@fedora:~# netstat -tlp | grep stap
> > tcp6       0      0 [::]:38541              [::]:*                  LISTEN      21523/stap-serverd
> > root@fedora:~# SERVER_IP=127.0.0.1
> > root@fedora:~# SERVER_PORT=38541
> > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }'
> > Using a compile server.
> > Pass 1: parsed user script and 529 library scripts using 537264virt/292632res/15232shr/276680data kb, in 770usr/90sys/892real ms.
> > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305944res/15872shr/289352
> > # ...
> > # Here systemtap instructs you how to enroll a MOK key, I've lost these messages somehow, but
> > # see below how to proceed:
> > # ...
> > root@fedora:~# mokutil --import signing_key.x509
> > #
> > #  Now reboot, finish enrolling the MOK key and boot
> > #
> > #  Having your system configured now you can:
> > #
> > root@fedora:~# mokutil --sb-state
> > SecureBoot enabled
> > root@fedora:~# netstat -tlp | grep stap
> > root@fedora:~# service stap-server start start
> > Redirecting to /bin/systemctl start stap-server.service
> > root@fedora:~# netstat -tlp | grep stap
> > tcp6       0      0 [::]:36707              [::]:*                  LISTEN      1979/stap-serverd
> > root@fedora:~# SERVER_IP=127.0.0.1; SERVER_PORT=36707
> > root@fedora:~# stap --trust-servers=ssl,signer,all-users,no-prompt --use-server=$SERVER_IP:$SERVER_PORT
> > Adding trust in the following servers as an SSL peer for all users and as a module signer for all users
> >    host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown"
> > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }'
> > Using a compile server.
> > Pass 1: parsed user script and 529 library scripts using 537264virt/292504res/15104shr/276680data kb, in 760usr/100sys/929real ms.
> > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms.
> > Pass 3: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.c
> > Pass 4: using cached <server>/.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko
> > Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key <server>/.systemtap/ssl/server/moks
> > Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8c:d0:e4:50:f4:b1:ef:7f:4e"
> > Passes: via server  host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" using 267740virt/23952res/19856shr/3108data kb, in 30usr/0sys/1481real ms.
> > The kernel on your system requires modules to be signed for loading.
> > The module created by compiling your script must be signed by a systemtap compile-server.  [man stap-server]
> > --use-server was automatically selected in order to request compilation by a compile-server.
> > Pass 5: starting run.
> > hey
> > Pass 5: run completed in 10usr/50sys/948real ms.
> > root@fedora:~#
>
> So, as you can see above, it works for me.  For more info about
> using systemtap with SecureBoot, see here:
>
> https://sourceware.org/systemtap/wiki/SecureBoot
>
> HTH; Cheers,
> Martin
>
>
> On  Mon  2023-12-04  21:53 , Martin Cermak wrote:
> > Hi Eric,
> >
> > systemtap packages come with stap-prep command that should do it for you:
> >
> > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup
> >
> > Depending on your environment, modern stap-prep may use debuginfod
> > for you.  That way you might have needed debugging information
> > available without actually installing the debuginfo RPMs.
> >
> > https://sourceware.org/elfutils/Debuginfod.html
> >
> > Hope this helps,
> >
> > Martin
> >
> >
> > On  Mon  2023-12-04  13:57 , Lee Eric wrote:
> > > Hi Martin,
> > >
> > > Thanks for your reply and it seems no connection error on the compile
> > > server. However, do we have any updated steps on how to install kernel
> > > debuginfo RPM packages? I searched a lot and seems old methods to use
> > > debuginfo-install command does not work.
> > >
> > > Hui
> > >
> > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak <mcermak@redhat.com> wrote:
> > > >
> > > > Hi Eric,
> > > >
> > > > On  Sun  2023-12-03  13:03 , Lee Eric via Systemtap wrote:
> > > > > Hi,
> > > > >
> > > > > I just noticed my stap scripts need to run via stap-server and I
> > > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot
> > > > > to set up stap server. However, I feel like the error messages from
> > > > > the stap command is really odd:
> > > > >
> > > > > # stap --list-server=all
> > > > > ...
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > >  host=thinkpad01.local address=127.0.0.1 port=44621
> > > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0
> > > > > certinfo="00:c1:73:c9:a1"
> > > > > ...
> > > > >
> > > > > And I'm using Fedora 39, so I would like to test if stap can connect
> > > > > to a server regardless the stap command ONLY accepting
> > > > > hostname/ip/cert serial which they are all the same.
> > > > >
> > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }'
> > > > > ...
> > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64
> > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build"
> > > > > Using a compile server.
> > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr
> > > > > '/tmp/stapvTSXTA/client.zip' *
> > > > > Spawn waitpid result (0x0): 0
> > > > > Servers matching 127.0.0.1:44621:
> > > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > > version=unknown certinfo="unknown"
> > > > > All specified servers:
> > > > >  host=unknown address=127.0.0.1 port=44621 sysinfo="unknown"
> > > > > version=unknown certinfo="unknown"
> > > > > Unable to connect to a server.
> > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb,
> > > > > in 0usr/0sys/4real ms.
> > > > > Passes: via server failed.  Try again with another '-v' option.
> > > > > The kernel on your system requires modules to be signed for loading.
> > > > > The module created by compiling your script must be signed by a
> > > > > systemtap compile-server.  [man stap-server]
> > > > > ...
> > > > >
> > > > > What's the meaning of that error exactly? Why stap cannot match one
> > > > > server in this case? I also did wireshark and I'm sure stap didn't
> > > > > talk to the tcp port 44621
> > > > >
> > > > > Is there any clue about this usage? Any help would be appreciated.
> > > >
> > > > I think you are missing a `stap --trust-servers ...` step.  We
> > > > have a simple testcase for stap server in Fedora CI:
> > > >
> > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh
> > > >
> > > > One of relatively fresh logs showing how it worked on Fedora 39
> > > > is here:
> > > >
> > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt
> > > >
> > > > Hope this helps,
> > > > Martin
> > > >
> > >
>