Read fault when running example network/nfsd-trace.stp stap

Read fault when running example network/nfsd-trace.stp stap

[Yun-Chih Chen]

I am running the example NFS stap file network/nfsd-trace.stp[1] on a
NFS server.
The stap file fails with the following message when a new NFS read
request comes in:

# sudo stap -v nfsd-trace.stp
Pass 1: parsed user script and 467 library scripts using
138276virt/44988res/7568shr/37488data kb, in 130usr/20sys/154real ms.
Pass 2: analyzed script: 22 probes, 66 functions, 9 embeds, 1 global
using 342304virt/246608res/9036shr/241516data kb, in
6020usr/730sys/6779real ms.
Pass 3: translated to C into
"/tmp/stapoyPncz/stap_b12e64c3fc1fc31297eb3b8c93c2dacf_107403_src.c"
using 342304virt/246856res/9228shr/241516data kb, in
200usr/70sys/265real ms.
Pass 4: compiled C into
"stap_b12e64c3fc1fc31297eb3b8c93c2dacf_107403.ko" in
4230usr/880sys/5113real ms.
Pass 5: starting run.
ERROR: read fault [man error::fault] at 0x0000000000000040 (addr) near
operator '->' at nfsd-trace.stp:18:31

It says "fh->fh_export->ex_path->mnt" is probably a reference on null
pointer ....

The test is conducted on both Linux 4.10 on Fedora 25 and Linux 3.10
on CentOS 7.  Both fails at the same place.
Does anyone know what's going on here?

Thanks,


Yunchih

[1] https://sourceware.org/systemtap/examples/network/nfsd-trace.stp

Re: Read fault when running example network/nfsd-trace.stp stap

[Daniel Doron]

ex_path is a struct path not a pointer to a struct path so maybe:

fh->fh_export->ex_path.mnt ?

On Fri, May 26, 2017 at 5:59 PM, Yun-Chih Chen <b03902074@ntu.edu.tw> wrote:
> I am running the example NFS stap file network/nfsd-trace.stp[1] on a
> NFS server.
> The stap file fails with the following message when a new NFS read
> request comes in:
>
> # sudo stap -v nfsd-trace.stp
> Pass 1: parsed user script and 467 library scripts using
> 138276virt/44988res/7568shr/37488data kb, in 130usr/20sys/154real ms.
> Pass 2: analyzed script: 22 probes, 66 functions, 9 embeds, 1 global
> using 342304virt/246608res/9036shr/241516data kb, in
> 6020usr/730sys/6779real ms.
> Pass 3: translated to C into
> "/tmp/stapoyPncz/stap_b12e64c3fc1fc31297eb3b8c93c2dacf_107403_src.c"
> using 342304virt/246856res/9228shr/241516data kb, in
> 200usr/70sys/265real ms.
> Pass 4: compiled C into
> "stap_b12e64c3fc1fc31297eb3b8c93c2dacf_107403.ko" in
> 4230usr/880sys/5113real ms.
> Pass 5: starting run.
> ERROR: read fault [man error::fault] at 0x0000000000000040 (addr) near
> operator '->' at nfsd-trace.stp:18:31
>
> It says "fh->fh_export->ex_path->mnt" is probably a reference on null
> pointer ....
>
> The test is conducted on both Linux 4.10 on Fedora 25 and Linux 3.10
> on CentOS 7.  Both fails at the same place.
> Does anyone know what's going on here?
>
> Thanks,
>
>
> Yunchih
>
> [1] https://sourceware.org/systemtap/examples/network/nfsd-trace.stp

Re: Read fault when running example network/nfsd-trace.stp stap

[David Smith]

Hmm. The good news here is that systemtap is properly exiting here
instead of accessing that NULL pointer and crashing your system.

Does this work for a while then fail or does it always fail on the
first read request?

I'm not sure what is going on here, the syntax looks reasonable.

(BTW, it doesn't matter that ex_path is a struct path and not a
pointer, in systemtap you always use '->' in script code.)

On Fri, May 26, 2017 at 9:59 AM, Yun-Chih Chen <b03902074@ntu.edu.tw> wrote:
> I am running the example NFS stap file network/nfsd-trace.stp[1] on a
> NFS server.
> The stap file fails with the following message when a new NFS read
> request comes in:
>
> # sudo stap -v nfsd-trace.stp
> Pass 1: parsed user script and 467 library scripts using
> 138276virt/44988res/7568shr/37488data kb, in 130usr/20sys/154real ms.
> Pass 2: analyzed script: 22 probes, 66 functions, 9 embeds, 1 global
> using 342304virt/246608res/9036shr/241516data kb, in
> 6020usr/730sys/6779real ms.
> Pass 3: translated to C into
> "/tmp/stapoyPncz/stap_b12e64c3fc1fc31297eb3b8c93c2dacf_107403_src.c"
> using 342304virt/246856res/9228shr/241516data kb, in
> 200usr/70sys/265real ms.
> Pass 4: compiled C into
> "stap_b12e64c3fc1fc31297eb3b8c93c2dacf_107403.ko" in
> 4230usr/880sys/5113real ms.
> Pass 5: starting run.
> ERROR: read fault [man error::fault] at 0x0000000000000040 (addr) near
> operator '->' at nfsd-trace.stp:18:31
>
> It says "fh->fh_export->ex_path->mnt" is probably a reference on null
> pointer ....
>
> The test is conducted on both Linux 4.10 on Fedora 25 and Linux 3.10
> on CentOS 7.  Both fails at the same place.
> Does anyone know what's going on here?
>
> Thanks,
>
>
> Yunchih
>
> [1] https://sourceware.org/systemtap/examples/network/nfsd-trace.stp



-- 
David Smith
Principal Software Engineer
Red Hat

Re: Read fault when running example network/nfsd-trace.stp stap

[Yun-Chih Chen]

On Wed, May 31, 2017 at 2:44 AM, David Smith <dsmith@redhat.com> wrote:
> Hmm. The good news here is that systemtap is properly exiting here
> instead of accessing that NULL pointer and crashing your system.
>
> Does this work for a while then fail or does it always fail on the
> first read request?

It always fail at the first NFSv3 read request (NFSv4 is okay).

To speed up the tracing, I ended up printing inode number only and get
filename from inode number at post-processing stage:

probe kernel.function("nfsd_vfs_read@fs/nfsd/vfs.c") !,
          module("nfsd").function("nfsd_vfs_read@fs/nfsd/vfs.c") ? {
    i = $file->f_inode;
    inode = @cast(i, "struct inode")->i_ino;
    printf("N %d %d %d\n", gettimeofday_ms(), inode);
}

Re: Read fault when running example network/nfsd-trace.stp stap

[Frank Ch. Eigler]

Yun-Chih Chen <b03902074@ntu.edu.tw> writes:

> [...]
> To speed up the tracing, I ended up printing inode number only and get
> filename from inode number at post-processing stage:
>
> probe kernel.function("nfsd_vfs_read@fs/nfsd/vfs.c") !,
>           module("nfsd").function("nfsd_vfs_read@fs/nfsd/vfs.c") ? {
>     i = $file->f_inode;
>     inode = @cast(i, "struct inode")->i_ino;
>     printf("N %d %d %d\n", gettimeofday_ms(), inode);
> }

You can proceed a couple of ways:
- add a   try { } catch { }
  block around your context-variable expressions
- use stap --suppress-handler-errors
- use stap --skip-badvars
- add explicit checks to the variables:  if (i != 0) { ... }


- FChE