From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ua1-x92c.google.com (mail-ua1-x92c.google.com [IPv6:2607:f8b0:4864:20::92c]) by sourceware.org (Postfix) with ESMTPS id 5A4CE3858D20 for ; Wed, 6 Dec 2023 16:00:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5A4CE3858D20 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 5A4CE3858D20 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::92c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701878447; cv=none; b=JWhmEHbtxYbm3u7a0WN7ygk4vZ17xL7vGxWMm/ffqL7WkgY8jr61Cl4gVq6ynxeV3T3gg3bBjaaENL4ePU6X1FtnQr2W6bOk67w8GbNSOZiuSevCZ/poxQanRbem+JKJ6BW/mN4K0p0CEs6JF0bIDCc1vtqIovE20d/o9CUZETA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701878447; c=relaxed/simple; bh=htpQwr84/qnlz1tF6Nq60tPrC7R9HEv/9wQZ9VDYVJM=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=lULNWf2HhnD+4Sgpe7gZxAzIOA7vkE8rJoRvweNk3GqNwt1wo2TXMwiYglFhznSBmk0dinZSCfM05Z/1bNXbkb5Wwn9xnbZ2py0guPVGFvg4Ml0qUp9WX6rRInEkoLb013Nj8vWrQq1YzfDrqftdbXvo8Di6S1mcxHbIhXEMooo= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-ua1-x92c.google.com with SMTP id a1e0cc1a2514c-7c44f5f3ca2so1690312241.0 for ; Wed, 06 Dec 2023 08:00:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701878442; x=1702483242; darn=sourceware.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/jivElVQVDQ48K3XgSXkOSkaViotj0qd7D0oEffsaUU=; b=XQBzYE5G7ysAVZmDIqu6bIuCcaxj+6izsEtKNckQ3GHpNw7xHoq4cmfhuIWq73iZCQ p68hkm3+HEc9sbm4ktAieSfUjB+XxTAHCOpo2Q/qz9pxk2G9WITJ/xbGiE5EZ0pI3ZnP gLEifHTxLP3esI4eLaPFK/RQpVBRtvIUsJ6THvREJT7HqZF0hrLQnC62hDzK61bXqS// o9yuqkz4sHBkrflcQtV23qQefeD24HyAYDaokbnm/8W2pqHzMYVuL4nBRZx+296Mfnet mfeZabNSuySKPEuH5GOl5vbgXU5JmPorxCvBPoUXNF91kNKevaC0HpGfRrWukJEIVMJU UEEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701878442; x=1702483242; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/jivElVQVDQ48K3XgSXkOSkaViotj0qd7D0oEffsaUU=; b=F53VrW9c5SCqcrwWUxbkQrZJuRKSBTNbk0H43Ov+f/YtW/iKOErDmNZ+zzU3A2OgX4 zxFNYKWNRtTa7wjAfuBEJxzcVHmAM4xxbrT/GUJ+IqvzFoAm8NDGk7iWvPcrmmi9G2vW Jfk7tACNPx0ejtXwc4KERB3kJ93v+NjZsbceazrAEJyXm8dDVNDXnn74V5SKM4dvRAAJ WDp2bOaPr5kaAFMcUTqtriRF4R1rw0cTee75nRYhe6ZJ013hTfL8RSqWRHmR8nYfZ1VD c4+O3M6Wl4PxTTyCjEvvwksYj0XVrvmBnU/XvwBUu9jmKc1vSzldi732Pt+gQTM3EPy9 AabA== X-Gm-Message-State: AOJu0YzCLtvM2QN09LxX1jphKsH/hMocejiRGY55ZHPhbrZK00pyUQVo 2OFpepa778mSldafT9+s+Ft7u3C1WRmYGHqW3SEuQ7bUWC4= X-Google-Smtp-Source: AGHT+IH4y1FqnT7CbfbQ0IYxNal76f6Ba2vowd3/RE69XD/Gn+/frT6paep353E6RV+nms5J57pLYkM+3zd7mjgC9H4= X-Received: by 2002:a05:6122:7c8:b0:4b2:c554:eeff with SMTP id l8-20020a05612207c800b004b2c554eeffmr1352705vkr.17.1701878442389; Wed, 06 Dec 2023 08:00:42 -0800 (PST) MIME-Version: 1.0 References: <20231204090848.aue4z3iydlyl43id@lida.tpb.lab.eng.brq.redhat.com> <20231204205323.3jbqqvkpsftlrexq@lida.tpb.lab.eng.brq.redhat.com> <20231206150341.tkvm4drn4flp6bz5@lida.tpb.lab.eng.brq.redhat.com> In-Reply-To: <20231206150341.tkvm4drn4flp6bz5@lida.tpb.lab.eng.brq.redhat.com> From: Lee Eric Date: Wed, 6 Dec 2023 11:00:30 -0500 Message-ID: Subject: Re: stap server is not able to use To: Martin Cermak Cc: systemtap@sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Thank you, Martin. After disabling SecureBoot everything works fine now. You are a life saver, much appreciated. Eric On Wed, Dec 6, 2023 at 10:03=E2=80=AFAM Martin Cermak = wrote: > > Hi Eric, > > hmmm, I think the configuration of your test system isn't > default, because on Fedora 39, the default is to use debuginfod, > while your system apparently is trying to install debuginfo RPMs > and then somehow fails to consume them. I've tested your > scenario with a fresh & up2date copy of Fedora 39 and it did work > for me. > > One important thing is that you apparently use SecureBoot. If > you don't need that, disable it, and your life will become easier. > If you need it though, here is how it did work for me: > > > root@fedora:~# rpm -qa | fgrep systemtap > > systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64 > > systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64 > > systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64 > > systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64 > > root@fedora:~# yum install systemtap-server > > ... > > root@fedora:~# mokutil --sb-state > > SecureBoot enabled > > root@fedora:~# uname -r > > 6.6.3-200.fc39.x86_64 > > root@fedora:~# stap-prep > > Configuring for kernel release 6.6.3-200.fc39.x86_64 > > Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/= vmlinuz debuginfo > > Increasing DEBUGINFOD_TIMEOUT to 300 temporarily > > Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593= 720 > > -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_= client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo > > Download successful. Assuming debuginfod server usage. > > root@fedora:~# service stap-server start > > Redirecting to /bin/systemctl start stap-server.service > > root@fedora:~# netstat -tlp | grep stap > > tcp6 0 0 [::]:38541 [::]:* LIS= TEN 21523/stap-serverd > > root@fedora:~# SERVER_IP=3D127.0.0.1 > > root@fedora:~# SERVER_PORT=3D38541 > > root@fedora:~# stap --use-server=3D$SERVER_IP:$SERVER_PORT -v -e 'probe= oneshot { log("hey") }' > > Using a compile server. > > Pass 1: parsed user script and 529 library scripts using 537264virt/292= 632res/15232shr/276680data kb, in 770usr/90sys/892real ms. > > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals usin= g 549936virt/305944res/15872shr/289352 > > # ... > > # Here systemtap instructs you how to enroll a MOK key, I've lost these= messages somehow, but > > # see below how to proceed: > > # ... > > root@fedora:~# mokutil --import signing_key.x509 > > # > > # Now reboot, finish enrolling the MOK key and boot > > # > > # Having your system configured now you can: > > # > > root@fedora:~# mokutil --sb-state > > SecureBoot enabled > > root@fedora:~# netstat -tlp | grep stap > > root@fedora:~# service stap-server start start > > Redirecting to /bin/systemctl start stap-server.service > > root@fedora:~# netstat -tlp | grep stap > > tcp6 0 0 [::]:36707 [::]:* LIS= TEN 1979/stap-serverd > > root@fedora:~# SERVER_IP=3D127.0.0.1; SERVER_PORT=3D36707 > > root@fedora:~# stap --trust-servers=3Dssl,signer,all-users,no-prompt --= use-server=3D$SERVER_IP:$SERVER_PORT > > Adding trust in the following servers as an SSL peer for all users and = as a module signer for all users > > host=3Dunknown address=3D127.0.0.1 port=3D36707 sysinfo=3D"unknown" = version=3Dunknown certinfo=3D"unknown" > > root@fedora:~# stap --use-server=3D$SERVER_IP:$SERVER_PORT -v -e 'probe= oneshot { log("hey") }' > > Using a compile server. > > Pass 1: parsed user script and 529 library scripts using 537264virt/292= 504res/15104shr/276680data kb, in 760usr/100sys/929real ms. > > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals usin= g 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms. > > Pass 3: using cached /.systemtap/cache/f7/stap_f74bee21f2c4f35f= cace0072c2cd100d_1155.c > > Pass 4: using cached /.systemtap/cache/f7/stap_f74bee21f2c4f35f= cace0072c2cd100d_1155.ko > > Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key /.systemtap/ssl/server/moks > > Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8= c:d0:e4:50:f4:b1:ef:7f:4e" > > Passes: via server host=3Dunknown address=3D127.0.0.1 port=3D36707 sys= info=3D"unknown" version=3Dunknown certinfo=3D"unknown" using 267740virt/23= 952res/19856shr/3108data kb, in 30usr/0sys/1481real ms. > > The kernel on your system requires modules to be signed for loading. > > The module created by compiling your script must be signed by a systemt= ap compile-server. [man stap-server] > > --use-server was automatically selected in order to request compilation= by a compile-server. > > Pass 5: starting run. > > hey > > Pass 5: run completed in 10usr/50sys/948real ms. > > root@fedora:~# > > So, as you can see above, it works for me. For more info about > using systemtap with SecureBoot, see here: > > https://sourceware.org/systemtap/wiki/SecureBoot > > HTH; Cheers, > Martin > > > On Mon 2023-12-04 21:53 , Martin Cermak wrote: > > Hi Eric, > > > > systemtap packages come with stap-prep command that should do it for yo= u: > > > > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-system= tap.html#using-setup > > > > Depending on your environment, modern stap-prep may use debuginfod > > for you. That way you might have needed debugging information > > available without actually installing the debuginfo RPMs. > > > > https://sourceware.org/elfutils/Debuginfod.html > > > > Hope this helps, > > > > Martin > > > > > > On Mon 2023-12-04 13:57 , Lee Eric wrote: > > > Hi Martin, > > > > > > Thanks for your reply and it seems no connection error on the compile > > > server. However, do we have any updated steps on how to install kerne= l > > > debuginfo RPM packages? I searched a lot and seems old methods to use > > > debuginfo-install command does not work. > > > > > > Hui > > > > > > On Mon, Dec 4, 2023 at 4:08=E2=80=AFAM Martin Cermak wrote: > > > > > > > > Hi Eric, > > > > > > > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > > > > Hi, > > > > > > > > > > I just noticed my stap scripts need to run via stap-server and I > > > > > followed the doc link https://sourceware.org/systemtap/wiki/Secur= eBoot > > > > > to set up stap server. However, I feel like the error messages fr= om > > > > > the stap command is really odd: > > > > > > > > > > # stap --list-server=3Dall > > > > > ... > > > > > host=3Dthinkpad01.local address=3D127.0.0.1 port=3D44621 > > > > > sysinfo=3D"6.5.10-300.fc39.x86_64 x86_64" version=3D5.0 > > > > > certinfo=3D"00:c1:73:c9:a1" > > > > > host=3Dthinkpad01.local address=3D127.0.0.1 port=3D44621 > > > > > sysinfo=3D"6.5.10-200.fc38.x86_64 x86_64" version=3D5.0 > > > > > certinfo=3D"00:c1:73:c9:a1" > > > > > host=3Dthinkpad01.local address=3D127.0.0.1 port=3D44621 > > > > > sysinfo=3D"6.3.8-200.fc38.x86_64 x86_64" version=3D5.0 > > > > > certinfo=3D"00:c1:73:c9:a1" > > > > > host=3Dthinkpad01.local address=3D127.0.0.1 port=3D44621 > > > > > sysinfo=3D"6.3.8-100.fc37.x86_64 x86_64" version=3D5.0 > > > > > certinfo=3D"00:c1:73:c9:a1" > > > > > host=3Dthinkpad01.local address=3D127.0.0.1 port=3D44621 > > > > > sysinfo=3D"6.3.12-200.fc38.x86_64 x86_64" version=3D5.0 > > > > > certinfo=3D"00:c1:73:c9:a1" > > > > > host=3Dthinkpad01.local address=3D127.0.0.1 port=3D44621 > > > > > sysinfo=3D"6.5.9-200.fc38.x86_64 x86_64" version=3D5.0 > > > > > certinfo=3D"00:c1:73:c9:a1" > > > > > ... > > > > > > > > > > And I'm using Fedora 39, so I would like to test if stap can conn= ect > > > > > to a server regardless the stap command ONLY accepting > > > > > hostname/ip/cert serial which they are all the same. > > > > > > > > > > # stap -vvv --use-server=3D127.0.0.1:44621 -e 'probe begin { exit= () }' > > > > > ... > > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > > > > Using a compile server. > > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > > > > '/tmp/stapvTSXTA/client.zip' * > > > > > Spawn waitpid result (0x0): 0 > > > > > Servers matching 127.0.0.1:44621: > > > > > host=3Dunknown address=3D127.0.0.1 port=3D44621 sysinfo=3D"unkno= wn" > > > > > version=3Dunknown certinfo=3D"unknown" > > > > > All specified servers: > > > > > host=3Dunknown address=3D127.0.0.1 port=3D44621 sysinfo=3D"unkno= wn" > > > > > version=3Dunknown certinfo=3D"unknown" > > > > > Unable to connect to a server. > > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data = kb, > > > > > in 0usr/0sys/4real ms. > > > > > Passes: via server failed. Try again with another '-v' option. > > > > > The kernel on your system requires modules to be signed for loadi= ng. > > > > > The module created by compiling your script must be signed by a > > > > > systemtap compile-server. [man stap-server] > > > > > ... > > > > > > > > > > What's the meaning of that error exactly? Why stap cannot match o= ne > > > > > server in this case? I also did wireshark and I'm sure stap didn'= t > > > > > talk to the tcp port 44621 > > > > > > > > > > Is there any clue about this usage? Any help would be appreciated= . > > > > > > > > I think you are missing a `stap --trust-servers ...` step. We > > > > have a simple testcase for stap server in Fedora CI: > > > > > > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/S= anity/stap-server-basic-sanity/runtest.sh > > > > > > > > One of relatively fresh logs showing how it worked on Fedora 39 > > > > is here: > > > > > > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1= f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/= stap-server-basic-sanity-32/output.txt > > > > > > > > Hope this helps, > > > > Martin > > > > > > > >