From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) by sourceware.org (Postfix) with ESMTPS id 247D83858D1E for ; Wed, 14 Jun 2023 12:12:35 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 247D83858D1E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=dneg.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=dneg.com Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-56d378b75f0so6972347b3.1 for ; Wed, 14 Jun 2023 05:12:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dneg.com; s=google; t=1686744754; x=1689336754; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=saP+VprK0UcvzwFAOnI9hMKl5/nn5RoIC06Z/lzJ3WQ=; b=sAaCaSF9G/m2wmo9cYWV7dnxutotSuPeS2QB9gyeNfhyAxwOtZvlE82d85L1NE0PZb 6b9FPqfKmgYDZ8pHFNMRcOOYYPaeXTvK8qZ515mvwykbxOFiSwEhEygZ3DPlUpN/1nmc 5S8f7s+SiPCkOyADRk+M3JEOyw1CBqiALVp2gjV5qhSZBWYvs1j3Pc14uJF4oSsnwRda HWVDY27B4HUowN7Q6SY+KjvKoraaOCEwKs2MYD/14Vw8DQyoZt3QwpY1caaRa2rYstIm Vp3kwJRlmPLxmONcqO4YBKHq7P1bNY1JTRnPDcok/f+dcRbXuIgRv6y5kwoTKC/PSRR+ M/Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686744754; x=1689336754; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=saP+VprK0UcvzwFAOnI9hMKl5/nn5RoIC06Z/lzJ3WQ=; b=ZoHixyqwB/oc5cx2D4nhynAQESUF2O8sh1SKPDfoscDfoQ0TsMTfx1zjaGTN8IBqUy qLgoWqLvLhOTN2ryzafjT5GuZjLn4fozJGMrfY4VZKBwqZBJQqLb3ZiTsWZOb3OC+00q 6Fffo4DRleTEd/hla9HsiUefpHkLPuOoEQob40e7p7SXdSIPRik/Ns1uGeE4S7M6iqyX FdrJPKC6B2anPLJvliXJI6VtHMksXvQxTw5jRllsHVOa8r8/HIeAiX+qynO/xSZtBjZw lE4rxi02hMrVXTUzC3D7eNfW/Kg1ZFgY1QOiG26YDQSXya3t8qDG79ODNn5hTu5pplCU 2Miw== X-Gm-Message-State: AC+VfDzGnrhqSIKDNRa0S1SizYdjANHm+MAU5Al0AtmYgBQek4/b40EP qlHJL8hDPwZUSaSJUYE2sSry0f+R9cfgfRF4W7t9Ue27HunEmZk5GOw= X-Google-Smtp-Source: ACHHUZ62ZgdXoVm8T6WSKeZ0pakuSpXZCb1pX0DaTpWdzRzLBOOE5/+8LQxLKAUDVSJNYplXbHOSUfRLonfqWHwm++s= X-Received: by 2002:a81:7407:0:b0:565:d401:7384 with SMTP id p7-20020a817407000000b00565d4017384mr1721882ywc.4.1686744754374; Wed, 14 Jun 2023 05:12:34 -0700 (PDT) MIME-Version: 1.0 References: <4893d129-6569-4318-cf3b-7821bef98441@redhat.com> In-Reply-To: From: Daire Byrne Date: Wed, 14 Jun 2023 13:11:57 +0100 Message-ID: Subject: Re: vfs.add_to_page_cache not working anymore? To: William Cohen Cc: "systemtap@sourceware.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-0.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Tue, 13 Jun 2023 at 18:34, William Cohen wrote: > > On 6/13/23 12:39, Daire Byrne wrote: > > On Tue, 13 Jun 2023 at 16:22, William Cohen wrote:> > >> Switching to systemtap-4.9 is probably not going to change the results > >> in this case as there are no changes in tapset/linux/vfs.stp between > >> 4.8 and 4.9. > > > > Good to know, I can skip trying to compile that then... > > Yes, running a newer version of software is often the first approach to s= ee if the problem has been fixed upstream. However, in this case the newer= version of systemtap is going to give the same results as the tapset in th= at area are the same. So the focus is find what is different between the w= orking older kernels and the current non-working kernel. > > > > >> Unfortunately, the kernels changes over time and some functions probed > >> by the tapset change over time or the way they are used by other parts > >> of the kernel changes. The vfs.add_to_page cache in the vfs.stp has > >> three possible functions it probes: add_to_page_cache_locked, > >> add_to_page_cache_lru, and add_to_page_cache. The first two functions > >> were added due to kernel commit f00654007fe1c15. Did some git commit > >> archeology and only add_to_page_cache_lru is in the kernel due to > >> kernel git commit 2bb876b58d593d7f2522ec0f41f20a74fde76822. > >> > >> The following URL show where add_to_page_cache_lru is used in 6.2.16 > >> kernels nfs and can provide some method of seeing how the nfs related > >> functions get called: > >> > >> https://elixir.bootlin.com/linux/v6.2.16/A/ident/add_to_page_cache_lru > > > > Thanks for the feedback and pointers, that helps me understand where > > the changes came from at least. It was still working on my last > > production kernel - v5.16. > > There are times were that is not possible when some function has been inl= ined and the return probe point isn't available or some argument is not ava= ilable at the probe point, but we do try to adapt the tapsets and examples = to work on newer kernels. > > > > > So if I recall, I used vfs.add_to_page cache because at the time it > > was the only (or easiest) way to work out total reads for mmap files > > from an NFS filesystem. > > > > I also would have thought it should work for any filesystem not just > > NFS - but I don't get any hits at all for an entire busy system. > > > >> As far as specifically what has changed to cause vfs.add_to_page_cache > >> not to trigger for NFS operations I am not sure. For the 6.2 kernel > >> it might be good to get a backtrace of the triggering of it and then > >> use that information to see what has changed in the functions on the > >> backtrace. > >> > >> stap -ldd -e 'probe vfs.add_to_page_cache { print_backtrace(); printf(= "Works.\n"); exit() }' > > > > I just get the error "Cannot specify a script with -l/-L/--dump-* > > switches" using systemtap v4.8. > > Sorry, missing a second '-' before ldd. The command below should work: > > stap --ldd -e 'probe vfs.add_to_page_cache { print_backtrace(); printf("W= orks.\n"); exit() }' > > It would be useful to know if the backtraces are. That would provide som= e information on how to adapt the script for newer kernels. Right, so I got set it up on the last known "working" kernel I had, v5.16, and this is a typical trace for a read: root@lonc400b1 daire]# stap --ldd -e 'probe vfs.add_to_page_cache { print_backtrace(); printf("Works.\n"); exit() }' WARNING: Missing unwind data for a module, rerun with 'stap -d kernel' 0xffffffff91258300 : add_to_page_cache_lru+0x0/0x30 [kernel] 0xffffffff912585b8 : read_cache_pages+0xd8/0x1a0 [kernel] 0xffffffffc0bbaccf 0xffffffffc0bbaccf 0xffffffff912589e5 : read_pages+0x155/0x250 [kernel] 0xffffffff91258cae : page_cache_ra_unbounded+0x1ce/0x250 [kernel] 0xffffffff91258ed0 : ondemand_readahead+0x1a0/0x300 [kernel] 0xffffffff912592ed : page_cache_sync_ra+0xbd/0xd0 [kernel] 0xffffffff9124cf13 : filemap_get_pages+0xe3/0x420 [kernel] 0xffffffff9124d31e : filemap_read+0xce/0x3c0 [kernel] 0xffffffff9124d700 : generic_file_read_iter+0xf0/0x160 [kernel] 0xffffffffc0baea64 0xffffffff91312c70 : new_sync_read+0x110/0x190 [kernel] 0xffffffff9131546f : vfs_read+0xff/0x1a0 [kernel] 0xffffffff91315b07 : ksys_read+0x67/0xe0 [kernel] 0xffffffff91315b99 : __x64_sys_read+0x19/0x20 [kernel] 0xffffffff91a6312b : do_syscall_64+0x3b/0x90 [kernel] 0xffffffff91c0007c : entry_SYSCALL_64_after_hwframe+0x44/0xae [kernel] Works. As you said earlier, it's hitting "add_to_page_cache_lru". I also tested with the v5.19 kernel and it no longer triggers anything with that. I'm going to stick my head out and say this stopped working due to all the folio conversion patches that were added between v5.17 & v6.0? Looking at the changelogs between v5.16 and v5.19 that's what jumps out to me anyway. Cheers, Daire > -Will > > > > > Thanks for the response. It sounds like I need to find a different way > > to work out total NFS reads for each filename path in modern kernels. > > > > Daire > > > > BTW, this is the code I had for tracking per process and file path read= IO: > > > > probe nfs.fop.open { > > pid =3D pid() > > filename =3D sprintf("%s", d_path(&$filp->f_path)) > > if (filename =3D~ "/hosts/.*/user_data") { > > files[pid, ino] =3D filename > > if ( !([pid, ino] in procinfo)) > > procinfo[pid, ino] =3D sprintf("%s", proc()) > > } > > } > > > > probe vfs.add_to_page_cache { > > pid =3D pid() > > if ([pid, ino] in files ) { > > readpage[pid, ino] +=3D 4096 > > files_store[pid, ino] =3D sprintf("%s", files[pid, ino]) > > } > > } > > >