From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 287AB3858D39 for ; Wed, 6 Dec 2023 15:03:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 287AB3858D39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 287AB3858D39 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701875028; cv=none; b=sgdrwbiDY8/+mLOhEq+2EkgjZohbFPb5bxPF+MYGHf1yzJn4NwECQw77tsh7yDfB2jqJt9ERn+ijp3xDzd3hRXZUoSgewRzlU9Ro/UWeE3mhD2HeyFbc2igwWfXhgNFAPHUt9scKkKvHMpOxoKwBE6A+Xjf1tzfU8RJ1nVsC/XE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701875028; c=relaxed/simple; bh=Cl9WShq+KptRR5ozuVbAgTYj40PSWj4NQUSeyQL2Fi4=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=s8R5Vz3GS+hCUz1NH8/1XW+Ir1YqmheVyVwET5UTQvKxDECLjhBG00TiQl+z3oMyjqdfDbS3igtaM8CkpBn87moSS8iI5O/7Sy/l9UpOQaS9fZIY8Fwm24htIjPJSVhGBJWXVgS+EfpfGwA2eKRq0RupKbn2H1JFvF8LGOyPleY= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701875025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xnKR6WVvshNVX88y+GngvCZo2RHCzlit9dUoDXu+kAI=; b=WvvKjleU6bPu+vrmEeFf99aCCmKXRsFAbZnU6ZOHdU+wME7aqJc5jwq5Zv3xAfR8SPVkst zeqTLPPk/6ynad4/astwi5sPXVRXzQn9XeDtSJPNBx+Gm4UOSXM5sP4w87uU79HTXcD7/8 hcP5Yyg1UOiWeS+fl8++Tahy/mWV7uQ= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-652-39LLLvkOPiG6G4aeYyK-Jw-1; Wed, 06 Dec 2023 10:03:44 -0500 X-MC-Unique: 39LLLvkOPiG6G4aeYyK-Jw-1 Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-a04b426b3c0so103593066b.0 for ; Wed, 06 Dec 2023 07:03:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701875023; x=1702479823; h=user-agent:in-reply-to:content-transfer-encoding :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xnKR6WVvshNVX88y+GngvCZo2RHCzlit9dUoDXu+kAI=; b=UV6b5/Mf2qFnyLVmE3dwRIIcDfhuKu4owOaI2ffvaTYGyIaplWcr/a2KlUe5XaoI4x N9f3nNTT8mtr3D83lKqEqH+Yvf92r1prlQ11ilWVI3huwEP5Zyc6uHYuBnj0BxLIC0W+ +h4Y6FmYhE5k6lSN+mdr1Kzs9YwXYsxVODOWtJgENtbYOrH/jeIWLMU6QL4Be5c66QBp Qqu3dtIcic89/eMNxT1dTtO50WTyaMaWqMFtyRVmUzPPW5mtg0bcFbvIRmzStKnjXP46 bWCxQyONevu221vEkeT9bzaXLr75TBQ2oSmSBa1tSYMyWMTPLynzwkxPCigRrkFnvnkO NqEw== X-Gm-Message-State: AOJu0YzuvGRAASLhHH3FWcnYq0YdQW0Nwko5vC4zRbLN1y+7x0JcIuu2 5ewXM7XwEJNMnAsVbZDzOaYFg3M0rjfZHYW/fYFLqbzlXulirzJTN7VExTWw75RUzP20XhJsLK7 F7XBVQrmDO3VXGHl1mhY= X-Received: by 2002:a17:906:2246:b0:a1a:81aa:56cc with SMTP id 6-20020a170906224600b00a1a81aa56ccmr3715249ejr.26.1701875023341; Wed, 06 Dec 2023 07:03:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IE9ZqX5SlL07m5xYD3Nuf8VpY6HDs1BPu7SDb63q50TuVJsydrve3zQPfzNCqeJBlISOXHoxQ== X-Received: by 2002:a17:906:2246:b0:a1a:81aa:56cc with SMTP id 6-20020a170906224600b00a1a81aa56ccmr3715235ejr.26.1701875022918; Wed, 06 Dec 2023 07:03:42 -0800 (PST) Received: from lida.tpb.lab.eng.brq.redhat.com (nat-pool-brq-t.redhat.com. [213.175.37.10]) by smtp.gmail.com with ESMTPSA id p27-20020a1709060ddb00b00a1ddf143020sm33613eji.54.2023.12.06.07.03.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Dec 2023 07:03:42 -0800 (PST) Date: Wed, 6 Dec 2023 16:03:41 +0100 From: Martin Cermak To: Lee Eric Cc: systemtap@sourceware.org Subject: Re: stap server is not able to use Message-ID: <20231206150341.tkvm4drn4flp6bz5@lida.tpb.lab.eng.brq.redhat.com> References: <20231204090848.aue4z3iydlyl43id@lida.tpb.lab.eng.brq.redhat.com> <20231204205323.3jbqqvkpsftlrexq@lida.tpb.lab.eng.brq.redhat.com> MIME-Version: 1.0 In-Reply-To: <20231204205323.3jbqqvkpsftlrexq@lida.tpb.lab.eng.brq.redhat.com> User-Agent: NeoMutt/20180716 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi Eric, hmmm, I think the configuration of your test system isn't default, because on Fedora 39, the default is to use debuginfod, while your system apparently is trying to install debuginfo RPMs and then somehow fails to consume them. I've tested your scenario with a fresh & up2date copy of Fedora 39 and it did work for me. One important thing is that you apparently use SecureBoot. If you don't need that, disable it, and your life will become easier. If you need it though, here is how it did work for me: > root@fedora:~# rpm -qa | fgrep systemtap > systemtap-runtime-5.0~pre16958465gca71442b-1.fc39.x86_64 > systemtap-client-5.0~pre16958465gca71442b-1.fc39.x86_64 > systemtap-devel-5.0~pre16958465gca71442b-1.fc39.x86_64 > systemtap-5.0~pre16958465gca71442b-1.fc39.x86_64 > root@fedora:~# yum install systemtap-server > ... > root@fedora:~# mokutil --sb-state > SecureBoot enabled > root@fedora:~# uname -r > 6.6.3-200.fc39.x86_64 > root@fedora:~# stap-prep > Configuring for kernel release 6.6.3-200.fc39.x86_64 > Please wait, attempting to download /lib/modules/6.6.3-200.fc39.x86_64/vmlinuz debuginfo > Increasing DEBUGINFOD_TIMEOUT to 300 temporarily > Downloading from https://debuginfod.fedoraproject.org/ 425593720/425593720 > -r--------. 1 root root 425593720 Nov 28 01:00 /root/.cache/debuginfod_client/7a67318d488fcc40764a3a4edf4af4ab8d7d5219/debuginfo > Download successful. Assuming debuginfod server usage. > root@fedora:~# service stap-server start > Redirecting to /bin/systemctl start stap-server.service > root@fedora:~# netstat -tlp | grep stap > tcp6 0 0 [::]:38541 [::]:* LISTEN 21523/stap-serverd > root@fedora:~# SERVER_IP=127.0.0.1 > root@fedora:~# SERVER_PORT=38541 > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }' > Using a compile server. > Pass 1: parsed user script and 529 library scripts using 537264virt/292632res/15232shr/276680data kb, in 770usr/90sys/892real ms. > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305944res/15872shr/289352 > # ... > # Here systemtap instructs you how to enroll a MOK key, I've lost these messages somehow, but > # see below how to proceed: > # ... > root@fedora:~# mokutil --import signing_key.x509 > # > # Now reboot, finish enrolling the MOK key and boot > # > # Having your system configured now you can: > # > root@fedora:~# mokutil --sb-state > SecureBoot enabled > root@fedora:~# netstat -tlp | grep stap > root@fedora:~# service stap-server start start > Redirecting to /bin/systemctl start stap-server.service > root@fedora:~# netstat -tlp | grep stap > tcp6 0 0 [::]:36707 [::]:* LISTEN 1979/stap-serverd > root@fedora:~# SERVER_IP=127.0.0.1; SERVER_PORT=36707 > root@fedora:~# stap --trust-servers=ssl,signer,all-users,no-prompt --use-server=$SERVER_IP:$SERVER_PORT > Adding trust in the following servers as an SSL peer for all users and as a module signer for all users > host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" > root@fedora:~# stap --use-server=$SERVER_IP:$SERVER_PORT -v -e 'probe oneshot { log("hey") }' > Using a compile server. > Pass 1: parsed user script and 529 library scripts using 537264virt/292504res/15104shr/276680data kb, in 760usr/100sys/929real ms. > Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 549936virt/305688res/15616shr/289352data kb, in 70usr/0sys/79real ms. > Pass 3: using cached /.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.c > Pass 4: using cached /.systemtap/cache/f7/stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko > Signing stap_f74bee21f2c4f35fcace0072c2cd100d_1155.ko with mok key /.systemtap/ssl/server/moks > Module signed with MOK, fingerprint "e7:4e:06:4c:e4:5a:c3:a5:8f:d4:08:8c:d0:e4:50:f4:b1:ef:7f:4e" > Passes: via server host=unknown address=127.0.0.1 port=36707 sysinfo="unknown" version=unknown certinfo="unknown" using 267740virt/23952res/19856shr/3108data kb, in 30usr/0sys/1481real ms. > The kernel on your system requires modules to be signed for loading. > The module created by compiling your script must be signed by a systemtap compile-server. [man stap-server] > --use-server was automatically selected in order to request compilation by a compile-server. > Pass 5: starting run. > hey > Pass 5: run completed in 10usr/50sys/948real ms. > root@fedora:~# So, as you can see above, it works for me. For more info about using systemtap with SecureBoot, see here: https://sourceware.org/systemtap/wiki/SecureBoot HTH; Cheers, Martin On Mon 2023-12-04 21:53 , Martin Cermak wrote: > Hi Eric, > > systemtap packages come with stap-prep command that should do it for you: > > https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup > > Depending on your environment, modern stap-prep may use debuginfod > for you. That way you might have needed debugging information > available without actually installing the debuginfo RPMs. > > https://sourceware.org/elfutils/Debuginfod.html > > Hope this helps, > > Martin > > > On Mon 2023-12-04 13:57 , Lee Eric wrote: > > Hi Martin, > > > > Thanks for your reply and it seems no connection error on the compile > > server. However, do we have any updated steps on how to install kernel > > debuginfo RPM packages? I searched a lot and seems old methods to use > > debuginfo-install command does not work. > > > > Hui > > > > On Mon, Dec 4, 2023 at 4:08 AM Martin Cermak wrote: > > > > > > Hi Eric, > > > > > > On Sun 2023-12-03 13:03 , Lee Eric via Systemtap wrote: > > > > Hi, > > > > > > > > I just noticed my stap scripts need to run via stap-server and I > > > > followed the doc link https://sourceware.org/systemtap/wiki/SecureBoot > > > > to set up stap server. However, I feel like the error messages from > > > > the stap command is really odd: > > > > > > > > # stap --list-server=all > > > > ... > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.10-300.fc39.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.10-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.8-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.8-100.fc37.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.3.12-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > host=thinkpad01.local address=127.0.0.1 port=44621 > > > > sysinfo="6.5.9-200.fc38.x86_64 x86_64" version=5.0 > > > > certinfo="00:c1:73:c9:a1" > > > > ... > > > > > > > > And I'm using Fedora 39, so I would like to test if stap can connect > > > > to a server regardless the stap command ONLY accepting > > > > hostname/ip/cert serial which they are all the same. > > > > > > > > # stap -vvv --use-server=127.0.0.1:44621 -e 'probe begin { exit() }' > > > > ... > > > > Session arch: x86_64 release: 6.5.10-300.fc39.x86_64 > > > > Build tree: "/lib/modules/6.5.10-300.fc39.x86_64/build" > > > > Using a compile server. > > > > Running sh -c cd '/tmp/stapvTSXTA/client' && zip -qr > > > > '/tmp/stapvTSXTA/client.zip' * > > > > Spawn waitpid result (0x0): 0 > > > > Servers matching 127.0.0.1:44621: > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > version=unknown certinfo="unknown" > > > > All specified servers: > > > > host=unknown address=127.0.0.1 port=44621 sysinfo="unknown" > > > > version=unknown certinfo="unknown" > > > > Unable to connect to a server. > > > > Passes: via server ? using 264956virt/19200res/16128shr/2424data kb, > > > > in 0usr/0sys/4real ms. > > > > Passes: via server failed. Try again with another '-v' option. > > > > The kernel on your system requires modules to be signed for loading. > > > > The module created by compiling your script must be signed by a > > > > systemtap compile-server. [man stap-server] > > > > ... > > > > > > > > What's the meaning of that error exactly? Why stap cannot match one > > > > server in this case? I also did wireshark and I'm sure stap didn't > > > > talk to the tcp port 44621 > > > > > > > > Is there any clue about this usage? Any help would be appreciated. > > > > > > I think you are missing a `stap --trust-servers ...` step. We > > > have a simple testcase for stap server in Fedora CI: > > > > > > https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/tests/Sanity/stap-server-basic-sanity/runtest.sh > > > > > > One of relatively fresh logs showing how it worked on Fedora 39 > > > is here: > > > > > > https://artifacts.dev.testing-farm.io/9d3c8552-145d-424f-a4fb-ddda1f5ef58e/work-ci1wn81l3u/plans/ci/execute/data/guest/default-0/tests/Sanity/stap-server-basic-sanity-32/output.txt > > > > > > Hope this helps, > > > Martin > > > > >