New example script: nfstop

New example script: nfstop

[David Smith]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

Back at FUDCon, during the systemtap info session I helped an attendee
with a script.  He had been using a dtrace script, called nfstop, from
the following link:

<http://blogs.oracle.com/erickustarz/entry/dscript_to_retrieve_active_nfs>

I made a quick approximation of the script at the conference, but hadn't
had time to get back to it and spiffy it up a bit.

I took some time yesterday and finally made some changes.  The script
acts a bit like 'top', listing the NFS clients who made the most NFS
operations every 5 seconds.  Current versions of systemtap should have
IPv4 and IPv6 client support.

I'll be incorporating this into the systemtap examples.

Any feedback on the script would be appreciated.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

[-- Attachment #2: nfstop.stp --]
[-- Type: text/plain, Size: 868 bytes --]

#!/usr/bin/stap -v

global nfsops

# Note that this script counts nfsops/client, ignoring the fact
# whether the nfs operation succeeded or not.

probe nfsd.dispatch
{
	nfsops[client_ip] <<< 1
}

probe timer.s(5)
{
	# Clear the screen and print out the header.  Because the
	# 'HOST:PORT' can get quite long for IPv6 (the longest the
	# host:port combination could be is:
	#  [0000:0000:0000:0000:0000:0000:0000:0001]:XXXXXX
	# or 48 chars), we'll print the 'NFS CALLS' first.
	ansi_clear_screen()
	ansi_set_color2(37, 40)
	printf("NFS CALLS   %-48s\n", "HOST:PORT")
	ansi_reset_color()

	# Print out the sorted stats for the top 20 clients.
	foreach ([client_ip] in nfsops- limit 20) {
		printf("%9d   %s\n", @count(nfsops[client_ip]), client_ip)
	}

	# Clear out the array for the next iteration.
	delete nfsops
}

probe begin
{
	printf("gathering NFS data...\n")
}

Re: New example script: nfstop

[Bryn M. Reeves]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/23/2012 02:56 PM, David Smith wrote:
> Back at FUDCon, during the systemtap info session I helped an
> attendee with a script.  He had been using a dtrace script, called
> nfstop, from the following link:
> 
> <http://blogs.oracle.com/erickustarz/entry/dscript_to_retrieve_active_nfs>
>
>  I made a quick approximation of the script at the conference, but
> hadn't had time to get back to it and spiffy it up a bit.
> 
> I took some time yesterday and finally made some changes.  The
> script acts a bit like 'top', listing the NFS clients who made the
> most NFS operations every 5 seconds.  Current versions of systemtap
> should have IPv4 and IPv6 client support.
> 
> I'll be incorporating this into the systemtap examples.
> 
> Any feedback on the script would be appreciated.
> 

Funnily enough I hacked something similar together last year. It's
been on my TODO since then to clean it up a bit and use it for a Red
Hat article.

I started working on a version using aggregates but that got ugly fast
(as it needed to handle empty and non-empty aggregates explicitly at
the time).

I think I talked to Frank at the time about ways to make this cleaner
(iirc all the @ops were wrapped in if(have_xxx_stats) constructs) but
I haven't been back to look again since.

Here's my original version. It just tracks lookups, reads and writes
and maintains global and per-client byte counters. It's been
successfully used in production at a few sites (iirc on RHEL5) for
tracking down badly behaved NFS clients and observing general server
performance.

I should probably dust off the improved version and see if I can
de-ugly it a bit.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9skykACgkQ6YSQoMYUY96p0wCg26CFCTh6slLkFnlFRDG1ZP3X
gSoAnjvcmf3G9mfcTGwkiy9W5OPcRNXM
=NWpF
-----END PGP SIGNATURE-----

[-- Attachment #2: nfsdtop.stp --]
[-- Type: text/plain, Size: 3114 bytes --]

# nfsd global counters
global nfsd_lookups
global nfsd_reads
global nfsd_read_bytes
global nfsd_writes
global nfsd_write_bytes
global nfsd_creates
global nfsd_commits
global nfsd_removes

# nfsd client tables
global nfsd_lookup_clients
global nfsd_lookup_clients_last_file
global nfsd_read_clients
global nfsd_read_clients_bytes
global nfsd_write_clients
global nfsd_write_clients_bytes


# Accumulate lookup stats
# Keep a count of lookups globally and by client_ip
# also keep track of the last file looked up by each
# client_ip
probe nfsd.proc.lookup {
	nfsd_lookups++
	nfsd_lookup_clients[client_ip]++
	nfsd_lookup_clients_last_file[client_ip] = filename
}

# Accumulate read stats
# Keep a count of reads globally and by client_ip
# also keep track of the number of bytes read globally
# and per-client_ip
probe nfsd.proc.read {
	nfsd_reads++
	nfsd_read_bytes+=size
	nfsd_read_clients[client_ip]++
	nfsd_read_clients_bytes[client_ip] += size
}

# Accumulate write stats
# Keep a count of writes globally and by client_ip
# also keep track of the number of bytes writen globally
# and per-client_ip
probe nfsd.proc.write {
	nfsd_writes++
	nfsd_write_bytes+=size
	nfsd_write_clients[client_ip]++
	nfsd_write_clients_bytes[client_ip] += size
}

# Just count creates for now
probe nfsd.proc.create {
	nfsd_creates++
}

# Just count commits for now
probe nfsd.proc.commit {
	nfsd_commits++
}

# Just count removes for now
probe nfsd.proc.remove {
	nfsd_removes++
}

# This is our "main loop" executed once every $interval
# We clear the terminal (top-style screen updates) and then
# to write out all our stats areas as fast as possible.
# Currently there are three sections:
#
# Global stats
# Top 10 lookup clients
# Top 10 reading clients
# Top 10 writing clients
#
probe timer.ms(1000)
{
	ansi_clear_screen()
	print("\n")
	printf("lookups  : %8d\n", nfsd_lookups)
	printf("reads    : %8d\n", nfsd_reads)
	printf("r/bytes  : %8d KiB\n", nfsd_read_bytes >> 10)
	printf("writes   : %8d\n", nfsd_writes)
	printf("w/bytes  : %8d KiB\n", nfsd_write_bytes >> 10)
	printf("creates  : %8d\n", nfsd_creates)
	printf("commits  : %8d\n", nfsd_commits)
	printf("removes  : %8d\n", nfsd_removes)
	printf("\n")
	print("lookups\n")
	printf("client\tlast file\n")
	foreach (ip in nfsd_lookup_clients- limit 10)
		printf("%s\t%s\n", ip, nfsd_lookup_clients_last_file[ip])
	print("\n")
	print("reads\n")
	printf("client\t\t\treads\tbytes\n")
	foreach (ip in nfsd_read_clients_bytes- limit 10)
		printf("%s\t%d\t%d\n", ip, nfsd_read_clients[ip],
			nfsd_read_clients_bytes[ip])
	print("\n")
	printf("writes\t\t\twrites\tbytes\n")
	foreach (ip in nfsd_write_clients_bytes- limit 10)
		printf("%s\t%d\t%d\n", ip, nfsd_write_clients[ip],
			nfsd_write_clients_bytes[ip])

	delete nfsd_lookups
	delete nfsd_reads
	delete nfsd_read_bytes
	delete nfsd_writes
	delete nfsd_write_bytes
	delete nfsd_creates
	delete nfsd_commits
	delete nfsd_removes

	delete nfsd_lookup_clients
	delete nfsd_lookup_clients_last_file
	delete nfsd_read_clients
	delete nfsd_read_clients_bytes
	delete nfsd_write_clients
	delete nfsd_write_clients_bytes

}

Re: New example script: nfstop

[David Smith]

On 03/23/2012 10:13 AM, Bryn M. Reeves wrote:

> Funnily enough I hacked something similar together last year. It's
> been on my TODO since then to clean it up a bit and use it for a Red
> Hat article.
> 
> I started working on a version using aggregates but that got ugly fast
> (as it needed to handle empty and non-empty aggregates explicitly at
> the time).
> 
> I think I talked to Frank at the time about ways to make this cleaner
> (iirc all the @ops were wrapped in if(have_xxx_stats) constructs) but
> I haven't been back to look again since.
> 
> Here's my original version. It just tracks lookups, reads and writes
> and maintains global and per-client byte counters. It's been
> successfully used in production at a few sites (iirc on RHEL5) for
> tracking down badly behaved NFS clients and observing general server
> performance.
> 
> I should probably dust off the improved version and see if I can
> de-ugly it a bit.

I went ahead and checked in this version.  I then went through and
converted it to use aggregates, which simplified things a bit.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: New example script: nfstop

[Bryn M. Reeves]

On 04/05/2012 08:39 PM, David Smith wrote:
>> I should probably dust off the improved version and see if I can
>> de-ugly it a bit.
> 
> I went ahead and checked in this version.  I then went through and
> converted it to use aggregates, which simplified things a bit.
> 

Great! Thanks very much - I'd wanted to do this previously but hit a few
snags and then other tasks took priority. I'll have a look at your
version and run it past a few of the folks who were using the previous
version for their feedback.

Regards,
Bryn.