Ways to Monitor NFS Calls from Client

Ways to Monitor NFS Calls from Client

[Lee Eric]

Hi,

I'm not sure if this topic had been raised previously. I am exploring
some ways to monitor NFS calls from client side. So I checked
https://sourceware.org/systemtap/examples/io/nfs_func_users.stp and
https://sourceware.org/systemtap/tapsets/nfsd.stp.html. However, that
makes me confused that because both does not work(or maybe the way I'm
using is wrong).

For the first one there's no output when client access files on NFS
server. For the second one I wrote a very small and simple stp:

probe nfs.proc.open {
    printf("Opening file %s from server %x\n", filename, server_ip)
}

Still, if I access any files there's no output either.

So I'm curious is there any actual way to capture NFS calls from
client side via SystemTap? Also, for the second method, any way to get
other NFS calls like GETATTR, READDIRPLUS etc.?

Thanks.

Hui

Re: Ways to Monitor NFS Calls from Client

[David Smith]

On 08/22/2016 02:40 AM, Lee Eric wrote:
> Hi,
> 
> I'm not sure if this topic had been raised previously. I am exploring
> some ways to monitor NFS calls from client side. So I checked
> https://sourceware.org/systemtap/examples/io/nfs_func_users.stp and
> https://sourceware.org/systemtap/tapsets/nfsd.stp.html. However, that
> makes me confused that because both does not work(or maybe the way I'm
> using is wrong).
>
> For the first one there's no output when client access files on NFS
> server. For the second one I wrote a very small and simple stp:
> 
> probe nfs.proc.open {
>     printf("Opening file %s from server %x\n", filename, server_ip)
> }
> 
> Still, if I access any files there's no output either.
> 
> So I'm curious is there any actual way to capture NFS calls from
> client side via SystemTap? Also, for the second method, any way to get
> other NFS calls like GETATTR, READDIRPLUS etc.?

Systemtap can monitor nfs client and nfs server activity.

The nfsd.stp tapset is for NFS server monitoring. Any script using the
nfsd.stp tapset must be run on the NFS server system. A script using the
nfsd.stp tapset will see NFS activity from all NFS client systems.

The nfs_proc.stp tapset is for NFS client monitoring. Any script using
the nfs_proc.stp tapset must be run on the particular NFS client system.
Note that a script using nfs_proc.stp to do client side NFS monitoring
will only be able to monitor NFS activity on that particular client
system. It won't see any NFS activity from different client systems.

Since you'd like to monitor NFS calls from a NFS client, you'd want to
use the nfs_proc.stp tapset. Here's an example run from one of my
systems that is an NFS client. In the following example, the file
'ChangeLog' is stored on an NFS partition:

====
# stap -e 'probe nfs.proc.open { printf("%s(%s)\n", name, argstr) }' -c
'cat ChangeLog > /dev/null'
nfs.proc.open(ChangeLog,32768,32797)
====

If something similar doesn't work for you, we'll have to dig deeper into
your setup.

As far as GETATTR and READDIRPLUS goes, it appears that the nfs_proc.stp
tapset doesn't have probes for those calls. However, it would be fairly
easy to add your own.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Ways to Monitor NFS Calls from Client

[Lee Eric]

Hi David,

Thanks for your information and I just gave a try on nfs.proc.open
which is working fine. However, I have some questions about the return
values.

from the tapset manual nfs.proc.open has 6 return values so how do I
know the types of those return values? For instance, how do I look up
the return value type of prot?

Thanks.

Hui

On Mon, Aug 22, 2016 at 7:43 AM, David Smith <dsmith@redhat.com> wrote:
> On 08/22/2016 02:40 AM, Lee Eric wrote:
>> Hi,
>>
>> I'm not sure if this topic had been raised previously. I am exploring
>> some ways to monitor NFS calls from client side. So I checked
>> https://sourceware.org/systemtap/examples/io/nfs_func_users.stp and
>> https://sourceware.org/systemtap/tapsets/nfsd.stp.html. However, that
>> makes me confused that because both does not work(or maybe the way I'm
>> using is wrong).
>>
>> For the first one there's no output when client access files on NFS
>> server. For the second one I wrote a very small and simple stp:
>>
>> probe nfs.proc.open {
>>     printf("Opening file %s from server %x\n", filename, server_ip)
>> }
>>
>> Still, if I access any files there's no output either.
>>
>> So I'm curious is there any actual way to capture NFS calls from
>> client side via SystemTap? Also, for the second method, any way to get
>> other NFS calls like GETATTR, READDIRPLUS etc.?
>
> Systemtap can monitor nfs client and nfs server activity.
>
> The nfsd.stp tapset is for NFS server monitoring. Any script using the
> nfsd.stp tapset must be run on the NFS server system. A script using the
> nfsd.stp tapset will see NFS activity from all NFS client systems.
>
> The nfs_proc.stp tapset is for NFS client monitoring. Any script using
> the nfs_proc.stp tapset must be run on the particular NFS client system.
> Note that a script using nfs_proc.stp to do client side NFS monitoring
> will only be able to monitor NFS activity on that particular client
> system. It won't see any NFS activity from different client systems.
>
> Since you'd like to monitor NFS calls from a NFS client, you'd want to
> use the nfs_proc.stp tapset. Here's an example run from one of my
> systems that is an NFS client. In the following example, the file
> 'ChangeLog' is stored on an NFS partition:
>
> ====
> # stap -e 'probe nfs.proc.open { printf("%s(%s)\n", name, argstr) }' -c
> 'cat ChangeLog > /dev/null'
> nfs.proc.open(ChangeLog,32768,32797)
> ====
>
> If something similar doesn't work for you, we'll have to dig deeper into
> your setup.
>
> As far as GETATTR and READDIRPLUS goes, it appears that the nfs_proc.stp
> tapset doesn't have probes for those calls. However, it would be fairly
> easy to add your own.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)

Re: Ways to Monitor NFS Calls from Client

[David Smith]

On 08/22/2016 10:49 PM, Lee Eric wrote:
> Hi David,
> 
> Thanks for your information and I just gave a try on nfs.proc.open
> which is working fine. However, I have some questions about the return
> values.
> 
> from the tapset manual nfs.proc.open has 6 return values so how do I
> know the types of those return values? For instance, how do I look up
> the return value type of prot?

Those aren't really return values, we call those values 'convenience
variables'. A 'nfs.proc.open' probe hits when the corresponding kernel
function ("nfs_open") is called. Systemtap tries to provide convenience
variables - variables/values that someone might need when that probe hits.

As far as types go, the easiest thing to do might be to look at the
probe source. For nfs.proc.open, it appears that those values are all
binary values, except for 'filename'. 'prot' is the protocol number. You
can use the sock_prot_num2str() function to convert that protocol number
to a string value if you'd like.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Ways to Monitor NFS Calls from Client

[Lee Eric]

Thanks David. That explains my question.

On Tue, Aug 23, 2016 at 6:44 AM, David Smith <dsmith@redhat.com> wrote:
> On 08/22/2016 10:49 PM, Lee Eric wrote:
>> Hi David,
>>
>> Thanks for your information and I just gave a try on nfs.proc.open
>> which is working fine. However, I have some questions about the return
>> values.
>>
>> from the tapset manual nfs.proc.open has 6 return values so how do I
>> know the types of those return values? For instance, how do I look up
>> the return value type of prot?
>
> Those aren't really return values, we call those values 'convenience
> variables'. A 'nfs.proc.open' probe hits when the corresponding kernel
> function ("nfs_open") is called. Systemtap tries to provide convenience
> variables - variables/values that someone might need when that probe hits.
>
> As far as types go, the easiest thing to do might be to look at the
> probe source. For nfs.proc.open, it appears that those values are all
> binary values, except for 'filename'. 'prot' is the protocol number. You
> can use the sock_prot_num2str() function to convert that protocol number
> to a string value if you'd like.
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)