NFSD tapset: how can I get filename from filehandles ?

NFSD tapset: how can I get filename from filehandles ?

[sbocahu]

Hi all,

I hope I'm not disturbing too much... I'm a new systemtap user without any
kernel knowledge, trying to write a few scripts.
Since I fail to write the second one, I would like to ask you for
pointers:

I would like to gather some stats from nfsd probes to know which files are
most accessed/written/read/etc...
This is pretty easy with the create(), remove() and lookup() nfsops as the
nfsd tapset exposes the filename variable (wich seems to be easy as well,
because the filename is given as argument to the call).

The problem is: I can't get the filename for other nfsops. The filename is
not given as argument.


I guess it is possible to get it from the filehandle, though:
(warning: I might be _totally_ wrong, I just tried to read the source and
understand things by myself...)


from the nfsd tapset, nfsd.proc3.write:

    fh = & @cast($argp, "nfsd3_writeargs", "kernel:nfsd")->fh

* fh seems to be of type of svc_fh:

linux/include/linux/nfsd/xdr3.h:

    struct nfsd3_writeargs {
            svc_fh                  fh;
            __u64                   offset;
            __u32                   count;
            int                     stable;
            __u32                   len;
            int                     vlen;
    };

* svc_fh seems to contain a dentry* member

linux/nfsd/nfsfh.h:

    typedef struct svc_fh {
            struct knfsd_fh         fh_handle;      /* FH data */
            struct dentry *         fh_dentry;      /* validated dentry */
            struct svc_export *     fh_export;      /* export pointer */

            ...

* there should be a qstr member in dentry...

linux/dcache.h:

    struct dentry {
            atomic_t d_count;
            ...
            struct qstr d_name;
            ...

* ... which contains the filename

linux/dcache.h:

    struct qstr {
            unsigned int hash;
            unsigned int len;
            const unsigned char *name;
    };


I've tried a few things without success... such as:


    filename = @cast($argp, "nfsd3_writeargs",
"kernel:nfsd")->fh->fh_dentry->d_name->name


which ended with a segfault.


Well, would you correct me and give me pointers on how achieving this,
please ?

Thanks a lot !
Best regards,

Sébastien

Re: NFSD tapset: how can I get filename from filehandles ?

[David Smith]

On 08/24/2010 09:23 AM, sbocahu@bearstech.com wrote:
> Hi all,
> 
> I hope I'm not disturbing too much... I'm a new systemtap user without any
> kernel knowledge, trying to write a few scripts.
> Since I fail to write the second one, I would like to ask you for
> pointers:

No problem - ask away.

> I would like to gather some stats from nfsd probes to know which files are
> most accessed/written/read/etc...
> This is pretty easy with the create(), remove() and lookup() nfsops as the
> nfsd tapset exposes the filename variable (wich seems to be easy as well,
> because the filename is given as argument to the call).
> 
> The problem is: I can't get the filename for other nfsops. The filename is
> not given as argument.

Right.

> I guess it is possible to get it from the filehandle, though:
> (warning: I might be _totally_ wrong, I just tried to read the source and
> understand things by myself...)

... stuff deleted ...

> I've tried a few things without success... such as:
> 
> 
>     filename = @cast($argp, "nfsd3_writeargs",
> "kernel:nfsd")->fh->fh_dentry->d_name->name
> 
> 
> which ended with a segfault.

OK, I'm going to need some details here.  First, what kernel are you using?

When you say a 'segfault', do you mean the stap executable segfaulted,
or you got a kernel read fault, something like this:

---
ERROR: kernel read fault at 0x0000000000000034 (addr) near identifier
'@cast' at /usr/local/share/systemtap/tapset/dentry.stp:41:15
---

I'll include the script in a sec, but that is systemtap protecting me
from crashing my kernel.  Instead of letting me read from that bogus
address, systemtap errors out.  This is a good thing.

If the stap executable segfaulted, could you show us your script and
exact output you are seeing?

> Well, would you correct me and give me pointers on how achieving this,
> please ?

I need to do something like this myself, so I've looked into it.  Here's
what I started with:

==========
probe nfsd.proc3.write
{
	printf("fh = %p\n", fh)
	dentry = @cast(fh, "svc_fh", "kernel:nfsd")->fh_dentry
	printf("dentry = %p\n", dentry)
	printf("writing %s\n", d_name(dentry))
}
==========

That's what gives the kernel read fault I showed above.  Here's a
"fixed" version - it doesn't error out, but it doesn't work either.

==========
probe nfsd.proc3.write
{
	printf("fh = %p\n", fh)
	dentry = @cast(fh, "svc_fh", "kernel:nfsd")->fh_dentry
	printf("dentry = %p\n", dentry)
	if (dentry)
		printf("writing %s\n", d_name(dentry))
	else
		printf("no dentry!\n")
}
==========

At that point dentry is NULL.

When(/if) I figure out how to get the filename, I'll let you know.  If
you figure it out, let me know.

Feel free to email the list with more questions.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: NFSD tapset: how can I get filename from filehandles ?

[David Smith]

On 08/24/2010 11:13 AM, David Smith wrote:
> When(/if) I figure out how to get the filename, I'll let you know.  If
> you figure it out, let me know.

OK, here's what I found after some more experimentation.  Kernel
function  nfsd3_proc_write() (which is what 'nfsd.proc3.write' probes)
sets up a few things, then calls nfsd_write() (which is what
'nfsd.write' probes).  nfsd_write() makes sure the file pointer is set
up and then calls nfsd_vfs_write().  So, that's the first place the file
pointer is valid.

Here's that previous script updated to probe nfsd_vfs_write().  Let me
know if this doesn't work for you.

====
probe nfsd.proc3.write
{
	printf("fh = %p\n", fh)
	dentry = @cast(fh, "svc_fh", "kernel:nfsd")->fh_dentry
	printf("dentry = %p\n", dentry)
	if (dentry)
		printf("writing file %s\n", d_name(dentry))
	else
		printf("no dentry!\n")
}

probe nfsd.write
{
	printf("nfsd.write: fh = %s, file = %p\n", fh, file)
	dentry = @cast($fhp, "svc_fh", "kernel:nfsd")->fh_dentry
	printf("nfsd.write: dentry = %p\n", dentry)
}

probe kernel.function("nfsd_vfs_write") !,
      module("nfsd").function("nfsd_vfs_write")
{
	if ($file)
		printf("nfsd_vfs_write: file = %p (%s)\n", $file,
			d_name($file->f_dentry))
	else
		printf("nfsd_vfs_write: file = %p (NULL)\n", $file)
}
====


-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: NFSD tapset: how can I get filename from filehandles ?

[sbocahu]

>> When(/if) I figure out how to get the filename, I'll let you know.  If
>> you figure it out, let me know.
> 
> OK, here's what I found after some more experimentation.  Kernel
> function  nfsd3_proc_write() (which is what 'nfsd.proc3.write' probes)
> sets up a few things, then calls nfsd_write() (which is what
> 'nfsd.write' probes).  nfsd_write() makes sure the file pointer is set
> up and then calls nfsd_vfs_write().  So, that's the first place the file
> pointer is valid.

aaaaaah 8) ! It explains my read faults, I was looking at the wrong
place...

> Here's that previous script updated to probe nfsd_vfs_write().  Let me
> know if this doesn't work for you.

...script deleted...

It works ! I only changed 

 		printf("nfsd_vfs_write: file = %p (%s)\n", $file,
 			d_name($file->f_dentry))

to

 		printf("nfsd_vfs_write: file = %p (%s)\n", $file,
 			d_name($file->f_path->dentry))


Although my kernel's fs.h has 

 #define f_dentry        f_path.dentry

in the 'struct file' definition, systemtap shows this error when accessing
its f_dentry member:

semantic error: unable to find member 'f_dentry' for struct file
(alternatives: f_u f_path f_op f_lock f_count f_flags f_mode f_pos f_owner
f_cred f_ra f_version f_security private_data f_ep_links f_mapping):
operator '->' at ./nfsd-filestats.stp:53:16
        source:                         d_name($file->f_dentry)


I guess I'm now able to achieve my script :)
Thank you very much for your very good and quick help !

Cheers,

Sebastien

Re: NFSD tapset: how can I get filename from filehandles ?

[David Smith]

On 08/24/2010 12:49 PM, sbocahu@bearstech.com wrote:
>>> When(/if) I figure out how to get the filename, I'll let you know.  If
>>> you figure it out, let me know.
>>
>> Here's that previous script updated to probe nfsd_vfs_write().  Let me
>> know if this doesn't work for you.
> 
> ...script deleted...
> 
> It works ! I only changed 
> 
>  		printf("nfsd_vfs_write: file = %p (%s)\n", $file,
>  			d_name($file->f_dentry))
> 
> to
> 
>  		printf("nfsd_vfs_write: file = %p (%s)\n", $file,
>  			d_name($file->f_path->dentry))
> 
> 
> Although my kernel's fs.h has 

When I started playing around with this I figured you were on an older
kernel since you were using NFSv3 - but I was wrong.  Here's that
portion of the script modified to handle older or newer kernels.  It
also prints the export name.

====
probe kernel.function("nfsd_vfs_write") !,
      module("nfsd").function("nfsd_vfs_write")
{
	if ($file)
		printf("nfsd_vfs_write: file = %p (%s)\n", $file,
		       d_name(@defined($file->f_path->dentry)
			      ? $file->f_path->dentry
			      : $file->f_dentry))
	else
		printf("nfsd_vfs_write: file = %p (NULL)\n", $file)
	if ($fhp->fh_export)
		printf("export: %s\n",
		       @defined($fhp->fh_export->ex_pathname)
				? kernel_string($fhp->fh_export->ex_pathname)
				: kernel_string($fhp->fh_export->ex_path))
}
====

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: NFSD tapset: how can I get filename from filehandles ?

[sbocahu]

>> I've tried a few things without success... such as:
>> 
>> 
>>     filename = @cast($argp, "nfsd3_writeargs",
>> "kernel:nfsd")->fh->fh_dentry->d_name->name
>> 
>> 
>> which ended with a segfault.

> OK, I'm going to need some details here.  First, what kernel are you
using?

I'm using Debian's 2.6.32.

 
> When you say a 'segfault', do you mean the stap executable segfaulted,
> or you got a kernel read fault, something like this:
> 
> ---
> ERROR: kernel read fault at 0x0000000000000034 (addr) near identifier
> '@cast' at /usr/local/share/systemtap/tapset/dentry.stp:41:15
> ---

You're right ! They were read faults.

 
> I'll include the script in a sec, but that is systemtap protecting me
> from crashing my kernel.  Instead of letting me read from that bogus
> address, systemtap errors out.  This is a good thing.

Ok, It is not obvious to me, but right, that's far better than a kernel
oops :) 
(I got one, experiencing the guru mode on that filename problem :p)