[Bug runtime/28958] New: Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[Bug runtime/28958] New: Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[bill at broadley dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

            Bug ID: 28958
           Summary: Current git + RHEL (4.18.0) has 3 working NFSd
                    examples, and one broken.
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: bill at broadley dot org
  Target Milestone: ---

I did a git clone today (Mar 11th, 2022) and nfsd-recent.stp, nfsdtop.std, and
nfsd_unlik.stp work (in testsuite/systemtap.examples/network).

However nfsd-trace.stp results in:
# /nopt/apps/systemtap-git-2022-03-11/bin/stap nfsd-trace.stp
Fri Mar 11 12:03:39 2022 MST 192.168.12.20:34051 nfsd.proc4.write 932,0 ERROR:
read fault [man error::fault] at 0x10 near operator '->' at
/nopt/nrel/apps/systemtap-2022-03-11/share/systemtap/tapset/linux/dentry.stp:267:19
WARNING: Number of errors: 1, skipped probes: 1
WARNING: /nopt/nrel/apps/systemtap-2022-03-11/bin/staprun exited with status: 1
Pass 5: run failed.  [man error::pass5]

Here's the version:
[root@nas-1-0 network]# /nopt/nrel/apps/systemtap-2022-03-11/bin/stap -V
Systemtap translator/driver (version 4.7/0.185, commit
release-4.6-58-gaa27023c941f + changes)
Copyright (C) 2005-2021 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.32 ... 5.17.0-rc4
enabled features: BPF NLS

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[bill at broadley dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

--- Comment #1 from BillB <bill at broadley dot org> ---
Forgot the kernel:

# uname -a
Linux nas-1-0.swift.hpc.foo.gov 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Wed Jan 19
17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[bill at broadley dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

BillB <bill at broadley dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|runtime                     |testsuite
           Keywords|                            |testsuite

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wcohen at redhat dot com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[bill at broadley dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

BillB <bill at broadley dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bill at broadley dot org

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2022-03-18
     Ever confirmed|0                           |1

--- Comment #2 from William Cohen <wcohen at redhat dot com> ---
I am able to replicate error on RHEL8 vm exporting nfs file system:

$ sudo ../install/bin/stap -v
testsuite/systemtap.examples/network/nfsd-trace.stp 
Pass 1: parsed user script and 483 library scripts using
294924virt/95612res/17928shr/79820data kb, in 180usr/160sys/485real ms.
Pass 2: analyzed script: 22 probes, 64 functions, 7 embeds, 1 global using
553136virt/351840res/19852shr/338032data kb, in 5550usr/1570sys/7502real ms.
Pass 3: translated to C into
"/tmp/stap13rEn4/stap_e2015f4beef9bcfacf31be2a7eccb277_61031_src.c" using
553136virt/351968res/19980shr/338032data kb, in 660usr/100sys/773real ms.
Pass 4: compiled C into "stap_e2015f4beef9bcfacf31be2a7eccb277_61031.ko" in
19610usr/3730sys/24872real ms.
Pass 5: starting run.
Fri Mar 18 10:30:25 2022 EDT 192.168.122.253:32771 nfsd.proc4.lookup testing
ERROR: read fault [man error::fault] at 0x10 near operator '->' at
/home/wcohen/systemtap_write/install/share/systemtap/tapset/linux/dentry.stp:267:19
WARNING: Number of errors: 1, skipped probes: 1
WARNING: /home/wcohen/systemtap_write/install/bin/staprun exited with status: 1
Pass 5: run completed in 20usr/250sys/52136real ms.
Pass 5: run failed.  [man error::pass5]

The nfsd-trace.stp is the only using task_dentry_path
(https://sourceware.org/systemtap/tapsets/API-task-dentry-path.html).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

--- Comment #3 from William Cohen <wcohen at redhat dot com> ---
Found other testsuite scripts that end up using task_dentry_path directly and
indirectly:

systemtap.examples/network/autofs4.stp
systemtap.examples/process/pfiles.stp
systemtap.base/task_paths.stp
systemtap.base/task_fd_lookup.stp

Of these the task_paths.stp is the simplest and looks to be exhibiting the same
behavior.  Note that the last two ERROR messages are expected and correct, but
the first two should print out strings:

$ sudo ../../install/bin/stap  systemtap.base/task_paths.stp -T 1
ERROR: read fault [man error::fault] at 0x10
ERROR: read fault [man error::fault] at 0x10
ERROR: read fault [man error::fault] at 0x0
ERROR: read fault [man error::fault] at 0xffffffffffffffff

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #4 from William Cohen <wcohen at redhat dot com> ---
This has been fixed by commit 3739d47c4cc427ce4818d884f429a3efa85c38ae

An example of nfsd-tracing.stp running correctly on a rhel8 vm:

$ sudo ../install/bin/stap  testsuite/systemtap.examples/network/nfsd-trace.stp 
Tue Mar 22 13:51:51 2022 EDT 192.168.122.253:64003 nfsd.proc4.read 13,0
/mnt/myshareddir/testing
Tue Mar 22 13:51:51 2022 EDT 192.168.122.253:64003 nfsd.proc4.lookup .#testing
/mnt/myshareddir/.#testing
Tue Mar 22 13:51:51 2022 EDT 192.168.122.253:64003 nfsd.proc4.lookup #testing#
/mnt/myshareddir/#testing#
...


Also get the expected result for task_paths.stp now:

$ sudo ../install/bin/stap testsuite/systemtap.base/task_paths.stp -T 1
current cwd: /home/wcohen/systemtap_write/systemtap
current exe: /home/wcohen/systemtap_write/install/libexec/systemtap/stapio
ERROR: read fault [man error::fault] at 0x0
ERROR: read fault [man error::fault] at 0xffffffffffffffff

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[bill at broadley dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

--- Comment #5 from BillB <bill at broadley dot org> ---
Much appreciated, felt helpless to debug what a busy NFS file server was doing
without this functionality.  

I confirmed it worked for me with kernel 4.18.0-348.12.2.el8_5.x86_64.

I have some scripts that can process the logs into useful reports, I'll mention
this on the irc channel with an example and see if it's of use to anyone and a
link should be added to a nfsd-trace.README or maybe a contrib/nfsd directory
or similar.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

--- Comment #6 from William Cohen <wcohen at redhat dot com> ---
Glad that the nfsd-trace.stp is working for you.

Do you have some repository for the scripts processing the output of the
systemtap scripts?


Not sure what would be the best way to make the supplemental scripts
functionality available to others. Some possibilities:
-Add a directory in testsuite/systemtap.examples to hold the processing
scripts, maybe something like the the lwtools directoy
-Add an entry to the wiki at https://sourceware.org/systemtap/wiki describing
the scripts, where to get them, and how to use them.
-Update existing systemtap script to include the functionality.  The examples
allow using shell scripts.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[bill at broadley dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

--- Comment #7 from BillB <bill at broadley dot org> ---
No, but I could create one.

One of my summaries scripts produces this:

begin=1648245843
end=1648246023

Total ops for 180 seconds

base dir                                          Operations IOPs  IOPS%
Bandwidth bandwidth%
/nas-1-0/abcdefg/home/user99/.schrodinger  7358       40.88 2.30%         0    
 0.00%
/scratch/carsXY.Z9_rsm#z1-14-0-3e/fort.50  4312       23.96 1.35%   1832394    
 1.68%
/scratch/carsXY.Z9_rsm#z8-77-0-99/fort.50  4103       22.79 1.28%   2376893    
 2.18%

hosts:               Operations
   192.168.6.55:31235     37529     11.74%
   192.168.5.51:17155     10623      3.32%
   192.168.13.28:9219      6981      2.18%
   192.168.6.15:57603      6973      2.18%

Operations
  nfsd.proc4.write    146374     45.77%
   nfsd.proc4.read    123553     38.64%
 nfsd.proc4.lookup     20517      6.42%
 nfsd.proc4.commit     19071      5.96%
 nfsd.proc4.remove      7566      2.37%
 nfsd.proc4.rename      2486      0.78%
 nfsd.proc4.create       223      0.07%

I find it great for finding weird behavior, like say the above mentioned file
that's using zero bandwidth, but 40% of the OPS, turns out nearly all are
nfsd.proc4.lookup.  Another common problem is this weird issue where c++
programs seek to X, write 4 byte, seek to X write 8 bytes, seek to X and write
12 bytes .... 4096 bytes, then seek to X+4096.  So only 1 in 1024 writes are
actually kept.

Do scripts like this seem generally useful?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug testsuite/28958] Current git + RHEL (4.18.0) has 3 working NFSd examples, and one broken.

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=28958

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #8 from Frank Ch. Eigler <fche at redhat dot com> ---
Definitely useful, whether for use as is, or as an educational example to
study.
We'd be glad to include it in the samples library.

-- 
You are receiving this mail because:
You are the assignee for the bug.