From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id D2FB63858CDA for ; Tue, 13 Jun 2023 17:34:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D2FB63858CDA Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686677662; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AQbv7wZNTx3+IraYaMVuPlW5wcbqUXjrQODWID7BYIQ=; b=J/XeXNeUZcAUZB6VtI4NAKWeKo7tsd403TetkjamOprSj9H+0oqHhQj2tvfXOVa10uIXjy Jc0q55OqBr/g/q0h73iYGF97YRVnmSm4AlHSk8A/c+etUC6OX4HgUjzMh4c/c2nDskEHfW WZZOv5J57Y9fAHosBGbt5sIPqGNY8QY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-311-obHKxQ2ZM2yGwHx2-1-rRQ-1; Tue, 13 Jun 2023 13:34:18 -0400 X-MC-Unique: obHKxQ2ZM2yGwHx2-1-rRQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BA93E3C0F182; Tue, 13 Jun 2023 17:34:17 +0000 (UTC) Received: from [10.22.32.98] (unknown [10.22.32.98]) by smtp.corp.redhat.com (Postfix) with ESMTP id 908AF40CFD01; Tue, 13 Jun 2023 17:34:17 +0000 (UTC) Message-ID: Date: Tue, 13 Jun 2023 13:34:17 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Cc: wcohen@redhat.com, "systemtap@sourceware.org" Subject: Re: vfs.add_to_page_cache not working anymore? To: Daire Byrne References: <4893d129-6569-4318-cf3b-7821bef98441@redhat.com> From: William Cohen In-Reply-To: X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 6/13/23 12:39, Daire Byrne wrote: > On Tue, 13 Jun 2023 at 16:22, William Cohen wrote:> >> Switching to systemtap-4.9 is probably not going to change the results >> in this case as there are no changes in tapset/linux/vfs.stp between >> 4.8 and 4.9. > > Good to know, I can skip trying to compile that then... Yes, running a newer version of software is often the first approach to see if the problem has been fixed upstream. However, in this case the newer version of systemtap is going to give the same results as the tapset in that area are the same. So the focus is find what is different between the working older kernels and the current non-working kernel. > >> Unfortunately, the kernels changes over time and some functions probed >> by the tapset change over time or the way they are used by other parts >> of the kernel changes. The vfs.add_to_page cache in the vfs.stp has >> three possible functions it probes: add_to_page_cache_locked, >> add_to_page_cache_lru, and add_to_page_cache. The first two functions >> were added due to kernel commit f00654007fe1c15. Did some git commit >> archeology and only add_to_page_cache_lru is in the kernel due to >> kernel git commit 2bb876b58d593d7f2522ec0f41f20a74fde76822. >> >> The following URL show where add_to_page_cache_lru is used in 6.2.16 >> kernels nfs and can provide some method of seeing how the nfs related >> functions get called: >> >> https://elixir.bootlin.com/linux/v6.2.16/A/ident/add_to_page_cache_lru > > Thanks for the feedback and pointers, that helps me understand where > the changes came from at least. It was still working on my last > production kernel - v5.16. There are times were that is not possible when some function has been inlined and the return probe point isn't available or some argument is not available at the probe point, but we do try to adapt the tapsets and examples to work on newer kernels. > > So if I recall, I used vfs.add_to_page cache because at the time it > was the only (or easiest) way to work out total reads for mmap files > from an NFS filesystem. > > I also would have thought it should work for any filesystem not just > NFS - but I don't get any hits at all for an entire busy system. > >> As far as specifically what has changed to cause vfs.add_to_page_cache >> not to trigger for NFS operations I am not sure. For the 6.2 kernel >> it might be good to get a backtrace of the triggering of it and then >> use that information to see what has changed in the functions on the >> backtrace. >> >> stap -ldd -e 'probe vfs.add_to_page_cache { print_backtrace(); printf("Works.\n"); exit() }' > > I just get the error "Cannot specify a script with -l/-L/--dump-* > switches" using systemtap v4.8. Sorry, missing a second '-' before ldd. The command below should work: stap --ldd -e 'probe vfs.add_to_page_cache { print_backtrace(); printf("Works.\n"); exit() }' It would be useful to know if the backtraces are. That would provide some information on how to adapt the script for newer kernels. -Will > > Thanks for the response. It sounds like I need to find a different way > to work out total NFS reads for each filename path in modern kernels. > > Daire > > BTW, this is the code I had for tracking per process and file path read IO: > > probe nfs.fop.open { > pid = pid() > filename = sprintf("%s", d_path(&$filp->f_path)) > if (filename =~ "/hosts/.*/user_data") { > files[pid, ino] = filename > if ( !([pid, ino] in procinfo)) > procinfo[pid, ino] = sprintf("%s", proc()) > } > } > > probe vfs.add_to_page_cache { > pid = pid() > if ([pid, ino] in files ) { > readpage[pid, ino] += 4096 > files_store[pid, ino] = sprintf("%s", files[pid, ino]) > } > } >