master - askant: Import askant into tree

master - askant: Import askant into tree

[Andrew Price]

Gitweb:        http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=d5d8e39eef7802a7d314cafe112deee0133a8227
Commit:        d5d8e39eef7802a7d314cafe112deee0133a8227
Parent:        f1cd52dd57737623008f2e32b460cd7b426c51e0
Author:        Andrew Price <andy@andrewprice.me.uk>
AuthorDate:    Fri Aug 15 08:54:34 2008 +0100
Committer:     Fabio M. Di Nitto <fdinitto@redhat.com>
CommitterDate: Wed Aug 20 12:28:31 2008 +0200

askant: Import askant into tree

Askant is the beginnings of a file system performance analysis tool. This commit
imports it into the cluster tree.

Ref: https://bugzilla.redhat.com/show_bug.cgi?id=239656

Signed-off-by: Andrew Price <andy@andrewprice.me.uk>
---
 askant/INSTALL                     |   42 ++++
 askant/PLUGINAPI                   |   65 ++++++
 askant/README                      |   74 +++++++
 askant/askant/about.py             |    5 +
 askant/askant/askant.py            |   24 ++
 askant/askant/blktrace.py          |   93 ++++++++
 askant/askant/commands.py          |  333 +++++++++++++++++++++++++++++
 askant/askant/sysfs.py             |   86 ++++++++
 askant/fsplugins/gfs2/gfs2.c       |  405 ++++++++++++++++++++++++++++++++++++
 askant/fsplugins/gfs2/gfs2.h       |    3 +
 askant/fsplugins/gfs2/gfs2module.c |  104 +++++++++
 askant/scripts/askant              |    6 +
 askant/setup.py                    |   18 ++
 13 files changed, 1258 insertions(+), 0 deletions(-)

diff --git a/askant/INSTALL b/askant/INSTALL
new file mode 100644
index 0000000..ce26b82
--- /dev/null
+++ b/askant/INSTALL
@@ -0,0 +1,42 @@
+
+  How To Install Askant
+  ---------------------
+
+o Build Dependencies
+  - Python
+  - libgfs2 (part of the Red Hat cluster suite as a static lib)
+
+o Dependencies
+  - Python
+  - blktrace and blkparse with appropriate kernel config options enabled and
+    debugfs mounted
+
+o Intro
+  Askant uses Python's distutils for its standard installation procedure so the
+  setup.py script provides installation routines. Note that it doesn't provide
+  uninstallation routines so the best way to install it is to obtain a package
+  for your particular distribution and install that.
+
+o Installation
+  To install askant to the default location on your system, run as root:
+
+    ./setup.py install
+
+  If you wish to install it to a different root directory (e.g. /var/mychroot)
+  or prefix directory (e.g. /usr/local instead of /usr) see:
+  
+    ./setup.py install --help
+
+  To find out everything else setup.py can do:
+
+    ./setup.py --help-commands
+
+o Use Without Installing
+  To use askant without installing it (e.g. if you want to test patches
+  quickly) it is necessary to carry out the build stage to compile the required
+  plugins. Do:
+
+    ./setup.py build
+    cd build/lib*
+    python askant/askant.py
+
diff --git a/askant/PLUGINAPI b/askant/PLUGINAPI
new file mode 100644
index 0000000..4c2eceb
--- /dev/null
+++ b/askant/PLUGINAPI
@@ -0,0 +1,65 @@
+
+  Writing File System Plugins for Askant
+  --------------------------------------
+
+0. Contents
+
+  1. Intro
+  2. API
+  3. Reference
+
+
+1. Intro
+
+In order to gather file system data with which to enhance the data provided by
+blktrace, there must be specific code written for each file system type we wish
+to support. This is where file system plugins come along. The plugins must
+basically traverse an on-disk file system structure on a block device and report
+back their findings through a callback handed to them by askant. They must also
+expose a function which stops their traversal loop in case askant wishes to stop
+their execution prematurely.
+
+
+2. API
+
+A file system plugin must be written as a Python module. This means that C, C++
+and Python are suitable languages to write them in. They must expose a number of
+Python functions to askant:
+
+parsefs(dev)
+    
+    This function is called when askant wants the plugin to parse the file
+    system on the device specified by the device, dev.
+
+get_block_size()
+
+    This function is called to ascertain the file system block size which is
+    required in order to map the disk sector numbers provided by blktrace to
+    file system block numbers. It takes no arguments and should return a Python
+    integer value.
+
+set_report_hook(func)
+
+    This function is called to pass in a report function which the plugin should
+    call to report the details of a file system block. func is a function which
+    accepts four arguments in this order:
+    
+    Block number: a Python long integer
+    Block type: a Python string
+    Block number of parent: a Python long integer
+    Block file name: a Python string (should be "" for non-inode blocks)
+
+handle_sigint()
+
+    This function should make the plugin stop its parsing loop. This allows it
+    to be interrupted if askant catches a signal, for example. It takes no
+    arguments and its return value is not used.
+
+
+3. Reference
+
+o See fsplugins/gfs2/gfs2module.c for an example of writing a Python fs plugin
+  module in C.
+
+o http://www.python.org/doc/ext/intro.html shows how to extend Python with C or
+  C++.
diff --git a/askant/README b/askant/README
new file mode 100644
index 0000000..74e3219
--- /dev/null
+++ b/askant/README
@@ -0,0 +1,74 @@
+
+ Askant - A Block I/O Analysis Tool
+ ----------------------------------
+
+0. Contents
+
+  1. Intro
+  2. Commands and their options
+  3. Examples
+
+
+1. Intro
+
+Askant allows you to gather file system block I/O data for performance analysis.
+It uses blktrace to trace block operations and adds file system specific data by
+parsing fs block information straight from the storage device. To do this it
+invokes functionality from file system parser plugins which traverse the on-disk
+structures and report details of the blocks they encounter through callbacks.
+
+Usage: askant <command> [options...]
+
+o See 'askant --commands' for a full list of commands.
+
+
+2. Commands and their options
+
+  dumpfs
+
+    Dumps the block data provided by an fs plugin from the given file system.
+
+    -d  The block device which contains the file system under scrutiny
+    -t  The type of the file system, i.e. the name of the fs plugin to load
+    -o  Output file (optional). Stdout is used if '-' or omitted.
+        (Note that the output file should be on a different device to the one
+	specified with -d.)
+  
+  unlinks
+
+    Does a dumpfs and runs blktrace until interrupted. It then blkparses the
+    blktrace data and adds the block data. This is suitable for tracing I/O
+    operations during an unlink test run.
+
+    -d  The block device which contains the file system under scrutiny
+    -t  The type of the file system, i.e. the name of the fs plugin to load
+    -g  The path to the debugfs mount point, defaults to /sys/kernel/debug
+    -D  The directory in which to store the block dumps and blktrace files.
+
+  creates
+
+    Runs blktrace until interrupted. It then does a dumpfs and blkparses the
+    blktrace data and adds the block data. This is suitable for tracing I/O
+    operations during a file creation test run.
+
+    -d  The block device which contains the file system under scrutiny
+    -t  The type of the file system, i.e. the name of the fs plugin to load
+    -g  The path to the debugfs mount point, defaults to /sys/kernel/debug
+    -D  The directory in which to store the block dumps and blktrace files.
+
+3. Examples
+
+o Parse the file system and dump information about each block to stdout e.g.:
+
+    # askant dumpfs -t gfs2 -d /dev/sda3
+
+o Trace 'unlinks' by dumping block information before the blktrace and matching
+  block data to provide a more detailed blkparse output e.g.:
+
+    # askant unlinks -t gfs2 -d /dev/sda3 -D /tmp/dir -g /mnt/debugfs
+
+o Trace 'creates' by dumping block information after the blktrace and matching
+  block data to provide a more detailed blkparse output e.g.:
+
+    # askant creates -t gfs2 -d /dev/sda3 -D /tmp/dir -g /mnt/debugfs
+
diff --git a/askant/askant/__init__.py b/askant/askant/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/askant/askant/about.py b/askant/askant/about.py
new file mode 100644
index 0000000..3d40a74
--- /dev/null
+++ b/askant/askant/about.py
@@ -0,0 +1,5 @@
+"""
+Version information for askant.
+"""
+
+version = "0.0.1"
diff --git a/askant/askant/askant.py b/askant/askant/askant.py
new file mode 100644
index 0000000..a4376b6
--- /dev/null
+++ b/askant/askant/askant.py
@@ -0,0 +1,24 @@
+"""
+The main entry point for askant.
+"""
+
+import sys
+import commands
+
+def main():
+
+	"""Run askant"""
+
+	cmds = commands.Commands()
+	cmds.register_command(commands.DumpFSCommand())
+	cmds.register_command(commands.UnlinksCommand())
+	cmds.register_command(commands.CreatesCommand())
+
+	try:
+		cmds.process_argv(sys.argv)
+	except commands.NoFSPluginException, p:
+		print >>sys.stderr, "Plugin not found: %s" %p
+
+
+if __name__ == '__main__':
+	main()
diff --git a/askant/askant/blktrace.py b/askant/askant/blktrace.py
new file mode 100644
index 0000000..fdf0219
--- /dev/null
+++ b/askant/askant/blktrace.py
@@ -0,0 +1,93 @@
+"""
+Wrapper classes for running blktrace and blkparse and using their output
+"""
+
+import os
+import sys
+import time
+import signal
+from subprocess import Popen
+from subprocess import PIPE
+from sysfs import Sysfs
+
+class BlktraceException(Exception):
+
+	def __init__(self, val, msg):
+		Exception.__init__(self)
+		self.val = val
+		self.msg = msg
+	
+	def __str__(self):
+		return self.msg
+
+class Blktrace:
+
+	def __init__(self, dev, debugfs='/sys/kernel/debug'):
+		self.dev = dev
+		self.debugfs = debugfs
+		self.sysfs = Sysfs(dev)
+		self.btpid = -1
+
+	def handle_sigint(self, sig, frame):
+		if self.btpid >= 0:
+			os.kill(self.btpid, signal.SIGTERM)
+
+	def trace(self, tracefile):
+
+		if not self.dev:
+			raise Exception("No device specified.")
+
+		btargs = ['blktrace',
+			'-d', self.dev,
+			'-r', self.debugfs,
+			'-o', tracefile]
+
+		blktrace = Popen(btargs, bufsize=1, stdout=PIPE,
+						stderr=open('/dev/null','w'))
+		self.btpid = blktrace.pid
+		btres = None
+		while btres is None:
+			time.sleep(1)
+			btres = blktrace.poll()
+
+		self.btpid = -1
+		if btres:
+			raise BlktraceException(btres,
+				'blktrace exited with code ' + str(btres))
+
+
+
+	def parse(self, tracefile, getblk):
+
+		if not self.dev:
+			raise Exception("No device specified.")
+
+		offset = self.sysfs.get_partition_start_sector()
+
+		bpargs = ['blkparse', '-i', tracefile]
+		blkparse = Popen(bpargs, bufsize=1, stdout=PIPE)
+
+		bpres = None
+		while bpres is None:
+			output = blkparse.stdout.readline().strip()
+			if output:
+				chunks = output.split()
+				try:
+					# chunks[7] is the sector number
+					blk = list(getblk(int(chunks[7])))
+					print "%s %s %s" %\
+						(' '.join(blk[0:3]),\
+						 output.strip(),
+						 blk[3].strip())
+				except KeyError:
+					pass
+				except ValueError:
+					pass
+
+			bpres = blkparse.poll()
+
+		if bpres:
+			raise BlktraceException(bpres,
+				'blkparse exited with code ' + str(bpres))
+
+
diff --git a/askant/askant/commands.py b/askant/askant/commands.py
new file mode 100644
index 0000000..ad4639f
--- /dev/null
+++ b/askant/askant/commands.py
@@ -0,0 +1,333 @@
+"""Command line parsing and processing for askant"""
+
+import fs
+import os
+import sys
+import time
+import about
+import signal
+import tempfile
+from sysfs import Sysfs, SysfsException
+from blktrace import Blktrace, BlktraceException
+from optparse import OptionParser, make_option
+
+class NoFSPluginException(Exception):
+
+	"""A generic Exception class for when file system plugins are missing"""
+
+	pass # Nothing special about this exception
+
+class Commands:
+
+	"""Provides a convenient interface to the askant commands"""
+
+	def __init__(self):
+
+		self.commands = {}
+		self.parser = OptionParser(usage="%prog COMMAND [options]",
+				version=about.version)
+
+
+	def register_command(self, command):
+		self.commands[str(command)] = command
+	
+	def __lookup_command(self, command):
+		try:
+			return self.commands[command]
+		except KeyError, k:
+			return DefaultCommand()
+	
+	def __parse_command(self, command, argv):
+		self.parser.prog = "%s %s" %\
+			(self.parser.get_prog_name(), str(command))
+		self.parser.set_usage(command.get_usage())
+		optlst = command.get_options()
+		for o in optlst:
+			self.parser.add_option(o)
+		self.parser.set_description(command.get_help())
+
+		if str(command) != "":
+			argv = argv[1:]
+
+		return self.parser.parse_args(argv)
+
+	def process_argv(self, argv):
+		try:
+			command = self.__lookup_command(argv[1])
+		except IndexError, i:
+			# This exits
+			self.parser.error("No command specified. Try --help")
+
+		options, args = self.__parse_command(command, argv[1:])
+
+		command.do_command(options, args, self)
+
+class Command:
+	def __init__(self):
+		self.options = []
+
+class DefaultCommand(Command):
+	def __str__(self):
+		return ""
+
+	def get_options(self):
+		self.options.append(make_option("-c","--commands",
+			action="store_true",
+			help="Lists available commands"))
+		return self.options
+
+	def get_usage(self):
+		return "%prog COMMAND [options] [args]"
+
+	def get_help(self):
+		return "Askant is a tool for tracing I/O activity in Linux "\
+			"and adding extra file system context by "\
+			"parsing file system data directly."
+	
+	def do_command(self, options, args, base):
+		if options.commands:
+			base.parser.print_usage()
+			print "Available commands:"
+			keys = base.commands.keys()
+			keys.sort()
+			for k in keys:
+				print " %10s  %s" %\
+					(k, base.commands[k].get_help())
+			print ""
+			print "For command-specific help, use "\
+				"%sCOMMAND --help" % base.parser.get_prog_name()
+		else:
+			base.parser.error("Command not found. Try --commands")
+
+
+class FSCommandTemplate(Command):
+	def __init__(self):
+		self.fsmod = None
+		self.block_table = {}
+		self.block_func = self.__report_hook
+		self.sector_size = None
+		self.offset = None
+		self.options = [
+			make_option("-d","--device", metavar="DEVICE",
+				help="The device to analyse."),
+		]
+
+	def load_fs_plugin(self, fsname):
+		try:
+			self.fsmod = __import__("fs." + fsname,
+					globals(), locals(), ["fs"])
+		except ImportError, i:
+			raise NoFSPluginException(fsname)
+
+	def parse_fs(self, dev, hook=None):
+		if hook is None:
+			hook = self.block_func
+
+		sfs = Sysfs(dev)
+		offset = sfs.get_partition_start_sector()
+		sector_size = sfs.get_dev_sector_size()
+
+		self.fsmod.set_report_hook(hook)
+
+		try:
+	                sfs = Sysfs(dev)
+        	        self.offset = sfs.get_partition_start_sector()
+                	self.sector_size = sfs.get_dev_sector_size()
+			signal.signal(signal.SIGINT, self.fsmod.handle_sigint)
+			self.blk_size = self.fsmod.get_block_size(dev)
+			self.fsmod.parsefs(dev)
+			signal.signal(signal.SIGINT, signal.SIG_DFL)
+		except IOError, i:
+			print >>sys.stderr, str(i)
+			sys.exit(1)
+
+	def set_outfile(self, outfile):
+		self.outfile = outfile
+
+	def trace(self, options, dump_before):
+
+		sfs = Sysfs(options.device)
+		self.offset = sfs.get_partition_start_sector()
+		self.sector_size = sfs.get_dev_sector_size()
+
+		tmstamp = time.strftime('%Y%m%d%H%M%S')
+		self.block_func = self.__report_hook
+		try:
+			self.outfile = open(os.path.join(options.outdir, "blocks-%s" %
+				tmstamp), 'w')
+		except IOError, e:
+			print >>sys.stderr, str(e)
+			sys.exit(1)
+
+		if dump_before:
+			print >>sys.stderr, "Gathering block data..."
+			self.parse_fs(options.device)
+			self.outfile.close()
+
+		if options.debugfs:
+			bt = Blktrace(options.device, options.debugfs)
+		else:
+			bt = Blktrace(options.device)
+
+		os.chdir(options.outdir) # Blktrace doesn't use absolute paths
+		print >>sys.stderr, "Tracing. Hit Ctrl-C to end..."
+		try:
+			signal.signal(signal.SIGINT, bt.handle_sigint)
+			bt.trace(tmstamp)
+			signal.signal(signal.SIGINT, signal.SIG_DFL)
+		except BlktraceException, b:
+			signal.signal(signal.SIGINT, signal.SIG_DFL)
+			print >>sys.stderr, str(b)
+			sys.exit(1)
+
+		if not dump_before:
+			print >>sys.stderr, "Gathering block data..."
+			self.parse_fs(options.device)
+			self.outfile.close()
+
+		print >>sys.stderr, "Matching blocks..."
+		blockdb = open(self.outfile.name, 'r')
+		blocks = {}
+		for l in blockdb.readlines():
+			s = l.split('\t')
+			blocks[int(s[0])] = tuple(s[1:])
+
+		bt.parse(os.path.join(options.outdir, tmstamp), blocks.__getitem__)
+		blockdb.close()
+
+	def __report_hook(self, blk, type, parent, fn):
+		if not fn:
+			fn = ""
+		try:
+			self.outfile.write("%d\t%d\t%s\t%d\t\"%s\"\n" % (
+			((blk * self.blk_size)/self.sector_size) + self.offset,
+			blk, type, parent, fn))
+		except Exception, e:
+			print str(e)
+
+	def __dict_hook(self, blk, type, parent, fn):
+		self.block_table[
+		    ((blk * self.blk_size)/self.sector_size)+self.offset] =\
+		    (blk, type, parent, fn)
+
+
+class DumpFSCommand(FSCommandTemplate):
+	def __str__(self):
+		return "dumpfs"
+
+	def get_options(self):
+		self.options.append(make_option("-t","--type", metavar="FSTYPE",
+			help="The type of file system on the device."))
+		self.options.append(make_option("-o","--output", metavar="FILE",
+			default="-",
+			help="File to write output to or '-' for stdout "
+			"(default)"))
+		return self.options
+	
+	def get_usage(self):
+		return "%prog [options]"
+	
+	def get_help(self):
+		return "Dump fs block information in TSV format."
+	
+	def do_command(self, options, args, base):
+
+		if not options.device:
+			base.parser.error("No device specified. Use -d.")
+
+		if not options.type:
+			base.parser.error("No file system type specified. "
+								"Use -t.")
+		try:
+			if options.output != "-":
+				self.set_outfile(open(options.output, 'w'))
+			else:
+				self.set_outfile(sys.stdout)
+		except IOError, e:
+			print >>sys.stderr,\
+				"Unable to open file for writing: %s" %\
+				options.output
+			return
+		except KeyError:
+			pass
+
+		self.load_fs_plugin(options.type)
+		try:
+			self.parse_fs(options.device)
+		except SysfsException, s:
+			base.parser.error(s)
+		
+class UnlinksCommand(FSCommandTemplate):
+	def __str__(self):
+		return "unlinks"
+
+	def get_options(self):
+		self.options.append(make_option("-t","--type", metavar="FSTYPE",
+			help="The type of file system on the device."))
+		self.options.append(make_option("-g","--debugfs", metavar="PATH",
+			default="/sys/kernel/debug",
+			help="The path to the debugfs mountpoint. Default "
+			"/sys/kernel/debug"))
+		self.options.append(make_option("-D","--outdir", metavar="DIR",
+			help="Directory to write output to. Required."))
+		return self.options
+	
+	def get_usage(self):
+		return "%prog [options]"
+	
+	def get_help(self):
+		return "Trace block I/O activity during unlink tests."
+	
+	def do_command(self, options, args, base):
+		if not options.device:
+			base.parser.error("No device specified. Use -d.")
+
+		self.device = options.device
+
+		if not options.type:
+			base.parser.error("No file system type specified. "
+								"Use -t.")
+		if not options.outdir:
+			base.parser.error("No output directory specified. "
+								"Use -D.")
+
+		self.load_fs_plugin(options.type)
+		self.trace(options, dump_before=True)
+
+class CreatesCommand(FSCommandTemplate):
+	def __str__(self):
+		return "creates"
+
+	def get_options(self):
+		self.options.append(make_option("-t","--type", metavar="FSTYPE",
+			help="The type of file system on the device."))
+		self.options.append(make_option("-g","--debugfs", metavar="PATH",
+			default="/sys/kernel/debug",
+			help="The path to the debugfs mountpoint. Default "
+			"/sys/kernel/debug"))
+		self.options.append(make_option("-D","--outdir", metavar="DIR",
+			help="Directory to write output to. Required."))
+		return self.options
+	
+	def get_usage(self):
+		return "%prog [options]"
+	
+	def get_help(self):
+		return "Trace block I/O activity during create tests."
+	
+	def do_command(self, options, args, base):
+		if not options.device:
+			base.parser.error("No device specified. Use -d.")
+
+		self.device = options.device
+
+		if not options.type:
+			base.parser.error("No file system type specified. "
+								"Use -t.")
+		if not options.outdir:
+			base.parser.error("No output directory specified. "
+								"Use -D.")
+
+		self.load_fs_plugin(options.type)
+		self.trace(options, dump_before=False)
+
diff --git a/askant/askant/fs/__init__.py b/askant/askant/fs/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/askant/askant/sysfs.py b/askant/askant/sysfs.py
new file mode 100644
index 0000000..12b7a3b
--- /dev/null
+++ b/askant/askant/sysfs.py
@@ -0,0 +1,86 @@
+"""
+Provides access to sysfs data pertaining to storage partitions.
+"""
+
+import os
+import os.path
+
+class SysfsException(Exception):
+	"""
+	An exception which is raised when things go wrong with Sysfs.
+	"""
+	pass # No functionality to add over Exception class
+
+class Sysfs:
+	"""
+	Provides access to sysfs data pertaining to storage partitions.
+	"""
+	def __init__(self, partition, sysfs_path='/sys'):
+		"""
+		Instantiate a Sysfs object. partition should be a path to a
+		disc partition device e.g. /dev/sda3.
+		"""
+		self.partition = partition
+		self.device = self.__partition2parent()
+		self.sysfs_path = sysfs_path
+		line = self.__firstline(os.path.join(
+				sysfs_path,'block',
+				os.path.split(self.device)[1],
+				os.path.split(self.partition)[1],
+				'start'))
+		self.partition_start = int(line)
+		line = self.__firstline(os.path.join(
+				sysfs_path,'block',
+				os.path.split(self.device)[1],
+				'queue',
+				'hw_sector_size'))
+		self.dev_sector_size = int(line)
+		line = self.__firstline(os.path.join(
+				sysfs_path,'block',
+				os.path.split(self.device)[1],
+				os.path.split(self.partition)[1],
+				'size'))
+		self.partition_size = int(line)
+
+	def __firstline(self, fname):
+		"""
+		Read the first line of a file
+		"""
+		try:
+			fobj = open(fname, 'r')
+		except IOError:
+			raise SysfsException(
+				'Could not open file "%s" for reading. Please '
+				'check your kernel is 2.6.25 or later and '
+				'sysfs is mounted.' % fname)
+		line = fobj.readline()
+		fobj.close()
+		return line
+
+	def __partition2parent(self):
+		"""
+		Return the name of a partition device's parent device
+		e.g. /dev/sda3 -> /dev/sda
+		"""
+		# This may need thinking about a bit more -
+		# nothing in life can be this simple.
+		return self.partition.rstrip('0123456789')
+
+
+	def get_partition_start_sector(self):
+		"""
+		Look up the start sector of the partition.
+		"""
+		return self.partition_start
+	
+	def get_dev_sector_size(self):
+		"""
+		Look up the size of the device's sectors.
+		"""
+		return self.dev_sector_size
+	
+	def get_partition_size(self):
+		"""
+		Look up the size of the partition.
+		"""
+		return self.partition_size
diff --git a/askant/fsplugins/__init__.py b/askant/fsplugins/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/askant/fsplugins/gfs2/gfs2.c b/askant/fsplugins/gfs2/gfs2.c
new file mode 100644
index 0000000..bac9c35
--- /dev/null
+++ b/askant/fsplugins/gfs2/gfs2.c
@@ -0,0 +1,405 @@
+/**
+ * main.c - Functions for parsing the on-disk structure of a GFS2 fs.
+ *          Written by Andrew Price
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <string.h>
+#include <fcntl.h>
+#include <libgfs2.h>
+#include <errno.h>
+
+void (*report_func)(long int blk, char *type, long int parent, char *fn);
+
+/* Define how the block information is reported */
+#define report(b,p,t,n) \
+	((*report_func)((long int)b, t, (long int)p,n))
+/*	(printf("%lu\t%s\t%lu\t%s\n", (long int)b, t, (long int)p, f)) */
+#define report_data(b,p) report(b,p,"D","")
+#define report_leaf(b,p) report(b,p,"L","")
+#define report_indir(b,p) report(b,p,"i","")
+#define report_inode(b,p,n) report(b,p,"I",n)
+
+struct blk_extended {
+	uint64_t parent_blk;
+	char *fname;
+	char *blk;
+};
+
+/* FIXME: A static length block stack is most likely a stupid idea */
+struct blkstack {
+	int top;
+	struct blk_extended blocks[1024];
+};
+
+struct blkstack blk_stack;
+struct gfs2_sb sb;
+uint32_t blk_size;
+off_t max_seek;
+int fd;
+int flag_stop;
+
+/* prog_name and print_it() are needed to satisfy externs in libgfs2 */
+char *prog_name = "askant";
+
+void print_it(const char *label, const char *fmt, const char *fmt2, ...)
+{
+	va_list args;
+
+	va_start(args, fmt2);
+	printf("%s = ", label);
+	vprintf(fmt, args);
+	printf("\n");
+	va_end(args);
+}
+
+/**
+ * Push a block onto the stack
+ */
+static void push_blk(char *blk, uint64_t parent, char *fn)
+{
+	struct blk_extended eblk;
+
+	eblk.blk = blk;
+	eblk.parent_blk = parent;
+	eblk.fname = fn;
+
+	blk_stack.top++;
+	blk_stack.blocks[blk_stack.top] = eblk;
+}
+
+/**
+ * Initialise the block stack
+ */
+static int blk_stack_init(void)
+{
+	blk_stack.top = -1;
+	return 1;
+}
+
+/**
+ * Pop a block from the stack
+ */
+static struct blk_extended pop_blk(void)
+{
+	return blk_stack.blocks[blk_stack.top--];
+}
+
+/**
+ * Read the GFS2 superblock into the global sb variable.
+ * Returns 1 on error, 0 on success.
+ */
+static int read_gfs2_sb(void)
+{
+	off_t offsetsb;
+	off_t offsetres;
+	unsigned char buffer[GFS2_BASIC_BLOCK];
+	ssize_t readsz;
+
+	offsetsb = GFS2_SB_ADDR * GFS2_BASIC_BLOCK;
+	offsetres = lseek(fd, offsetsb, SEEK_SET);
+
+	if (offsetres != offsetsb) {
+		fprintf(stderr, "Could not seek to sb location on device.\n");
+		return 0;
+	}
+
+	readsz = read(fd, buffer, GFS2_BASIC_BLOCK);
+	if (readsz != GFS2_BASIC_BLOCK) {
+		fprintf(stderr, "Could not read superblock.\n");
+		return 0;
+	}
+
+	gfs2_sb_in(&sb, (char *)buffer);
+	if (check_sb(&sb)) {
+		fprintf(stderr, "Not a GFS2 filesystem.\n");
+		return 0;
+	}
+
+	return 1;
+}
+
+/**
+ * Read a block.
+ * blk_offset must be a block number.
+ * The returned pointer must be free'd.
+ */
+static char *read_gfs2_blk(off_t blk_offset)
+{
+	off_t offset;
+	off_t offsetres;
+	ssize_t readsz;
+	char *buffer;
+
+	buffer = (char *)malloc(blk_size);
+	if (!buffer) {
+		fprintf(stderr, "Could not allocate memory for block.\n");
+		return NULL;
+	}
+
+	offset = blk_offset * blk_size;
+
+	offsetres = lseek(fd, offset, SEEK_SET);
+	if (offsetres != offset) {
+		fprintf(stderr,
+			"Could not seek to block location: %lu error: %s\n",
+			(long int)blk_offset, strerror(errno));
+		return NULL;
+	}
+
+	readsz = read(fd, buffer, blk_size);
+	if (readsz != blk_size) {
+		fprintf(stderr, "Could not read block: %lu\n",
+					(long int)blk_offset);
+		return NULL;
+	}
+
+	return buffer;
+}
+
+/**
+ * Look at indirect pointers from a starting point in a block.
+ */
+static void do_indirect(char *start, uint16_t height, uint64_t parent)
+{
+	uint64_t ptr;
+	unsigned int i;
+	char *blk;
+
+	for (i = 0; i < blk_size; i += sizeof(uint64_t)) {
+		ptr = be64_to_cpu(*(uint64_t *)(start + i));
+		if (ptr > 0 && ptr < (max_seek / blk_size)) {
+			if (height == 1) {
+				report_data(ptr, parent);
+			} else if (height > 1) {
+				blk = read_gfs2_blk(ptr);
+				if (blk) {
+					report_indir(ptr, parent);
+					do_indirect(blk, height - 1, ptr);
+					free(blk);
+				}
+			}
+		} else {
+			break;
+		}
+	}
+}
+
+/**
+ * Parse count number of dirents from a starting point in a block.
+ */
+static void do_dirents(char *dirents, char *end, uint64_t parent, uint64_t gparent)
+{
+	struct gfs2_dirent dirent;
+	char *di_blk;
+	char *fname;
+
+	while (dirents < end) {
+		gfs2_dirent_in(&dirent, dirents);
+		if (dirent.de_inum.no_addr &&
+			dirent.de_inum.no_addr != parent &&
+			dirent.de_inum.no_addr != gparent &&
+			dirent.de_name_len > 0 &&
+			dirent.de_name_len <= GFS2_FNAMESIZE) {
+
+			fname = (char *)malloc(dirent.de_name_len + 1);
+			if (!fname) {
+				break;
+			}
+
+			memcpy(fname, dirents + sizeof(struct gfs2_dirent),
+				dirent.de_name_len);
+			fname[dirent.de_name_len] = '\0';
+
+			di_blk = read_gfs2_blk(dirent.de_inum.no_addr);
+			if (di_blk) {
+				push_blk(di_blk, parent, fname);
+			}
+		}
+		dirents += dirent.de_rec_len;
+	}
+}
+
+/**
+ * Examine the dirents in a leaf block.
+ * If the leaf is chained, do the chained leaves too.
+ */
+static void do_leaf(char *blk, uint64_t parent, uint64_t gparent)
+{
+	struct gfs2_leaf leaf;
+
+	while (blk) {
+		gfs2_leaf_in(&leaf, blk);
+		do_dirents(blk + sizeof(struct gfs2_leaf), blk + blk_size,
+							parent, gparent);
+		free(blk);
+		if (!leaf.lf_next) {
+			break;
+		}
+
+		blk = read_gfs2_blk(leaf.lf_next);
+	}
+}
+
+/**
+ * Parse leaf pointer data and examine the
+ * dirents in the destination leaf blocks.
+ */
+static void do_leaves(char *start, uint64_t parent, uint64_t gparent)
+{
+	uint64_t ptr;
+	uint64_t prev;
+	unsigned int i;
+	char *blk;
+
+	prev = 0;
+	for (i = 0; i < blk_size - sizeof(struct gfs2_dinode);
+					i += sizeof(uint64_t)) {
+		ptr = be64_to_cpu(*(uint64_t *)(start + i));
+
+		if (ptr >= (max_seek / blk_size)) {
+			break;
+		}
+
+		if (ptr && ptr != prev) {
+			blk = read_gfs2_blk(ptr);
+			if (blk) {
+				report_leaf(ptr, parent);
+				do_leaf(blk, parent, gparent);
+			}
+			prev = ptr;
+		}
+	}
+}
+
+/**
+ * Parse inode data from a block
+ */
+static void do_inode_blk(char *blk, uint64_t parent, char *fname)
+{
+	struct gfs2_dinode di;
+	char *data;
+
+	gfs2_dinode_in(&di, blk);
+	report_inode(di.di_num.no_addr, parent, fname);
+
+	data = (char *)((struct gfs2_dinode *)blk + 1);
+
+	if (di.di_height > 0) {
+		/* Indirect pointers */
+		do_indirect(data, di.di_height, di.di_num.no_addr);
+	} else if (S_ISDIR(di.di_mode) && !(di.di_flags & GFS2_DIF_EXHASH)) {
+		/* Stuffed directory */
+		do_dirents(data, blk + blk_size, di.di_num.no_addr, parent);
+	} else if (S_ISDIR(di.di_mode) && 
+				(di.di_flags & GFS2_DIF_EXHASH) && 
+				!(di.di_height)) {
+		/* Directory, has hashtable, height == 0 */
+		do_leaves(data, di.di_num.no_addr, parent);
+	}
+
+	/* free previously stacked block */
+	free(fname);
+	free(blk);
+}
+
+/**
+ * Get the root dir block and parse the fs
+ * using a stack to keep track of the unvisited
+ * inode blocks.
+ */
+static void parse_fs(void)
+{
+	struct gfs2_inum *root_dir_inum;
+	struct gfs2_inum *master_dir_inum;
+	struct blk_extended blk;
+	char *root_blk;
+	char *master_blk;
+
+	flag_stop = 0;
+
+	root_dir_inum = &(sb.sb_root_dir);
+	master_dir_inum = &(sb.sb_master_dir);
+
+	root_blk = read_gfs2_blk(root_dir_inum->no_addr);
+	master_blk = read_gfs2_blk(master_dir_inum->no_addr);
+	if (!root_blk || !master_blk) {
+		return;
+	}
+
+	push_blk(root_blk, root_dir_inum->no_addr, NULL);
+	while (blk_stack.top >= 0 && !flag_stop) {
+		blk = pop_blk();
+		do_inode_blk(blk.blk, blk.parent_blk, blk.fname);
+	}
+
+	push_blk(master_blk, master_dir_inum->no_addr, NULL);
+	while (blk_stack.top >= 0 && !flag_stop) {
+		blk = pop_blk();
+		/* TODO: Examine each block's magic number instead of assuming
+		 * they're inodes. Omitted for now due to time constraints and
+		 * the number of GFS2_METATYPE_*s which need catering for.
+		 */
+		do_inode_blk(blk.blk, blk.parent_blk, blk.fname);
+	}
+}
+
+/**
+ * Raise a flag to stop the parse loop cleanly
+ */
+void gfs2_stop(void)
+{
+	flag_stop = 1;
+}
+
+/**
+ * Parse a gfs2 file system on a given device
+ */
+int gfs2_parse(char *dev, void (*func)(long int b, char *t, long int p, char *f))
+{
+	report_func = func;
+
+	if (!blk_stack_init()) {
+		return 0;
+	}
+
+	if ((fd = open(dev, O_RDONLY)) < 0) {
+		return 0;
+	}
+
+	if (!read_gfs2_sb()) {
+		close(fd);
+		return 0;
+	}
+
+	blk_size = sb.sb_bsize;
+	max_seek = lseek(fd, 0, SEEK_END);
+
+	parse_fs();
+
+	close(fd);
+
+	return 1;
+}
+
+/**
+ * Return the block size of the gfs2 file system on a given device.
+ */
+uint32_t gfs2_block_size(char *dev)
+{
+	if ((fd = open(dev, O_RDONLY)) < 0) {
+		return 0;
+	}
+
+	if (!read_gfs2_sb()) {
+		close(fd);
+		return 0;
+	}
+	close(fd);
+	return sb.sb_bsize;
+}
diff --git a/askant/fsplugins/gfs2/gfs2.h b/askant/fsplugins/gfs2/gfs2.h
new file mode 100644
index 0000000..6c72eb0
--- /dev/null
+++ b/askant/fsplugins/gfs2/gfs2.h
@@ -0,0 +1,3 @@
+uint32_t gfs2_block_size(char *dev);
+int gfs2_parse(char *dev, void (*func)(long int b, char *t, long int p, char *f));
+void gfs2_stop(void);
diff --git a/askant/fsplugins/gfs2/gfs2module.c b/askant/fsplugins/gfs2/gfs2module.c
new file mode 100644
index 0000000..c6a0584
--- /dev/null
+++ b/askant/fsplugins/gfs2/gfs2module.c
@@ -0,0 +1,104 @@
+#include <Python.h>
+
+#include "gfs2.h"
+
+static PyObject *report_func = NULL;
+
+void report_func_wrapper(long int blk, char *type, long int parent, char *fn)
+{
+	PyObject *arglist;
+	PyObject *result;
+
+	arglist = Py_BuildValue("lsls", blk, type, parent, fn);
+	if (!arglist) {
+		return;
+	}
+	result = PyEval_CallObject(report_func, arglist);
+	Py_DECREF(arglist);
+	if (!result) {
+		return;
+	}
+	Py_DECREF(result);
+}
+
+static PyObject *gfs2_set_report_hook(PyObject *self, PyObject *args)
+{
+	PyObject *result = NULL;
+	PyObject *temp;
+
+	if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
+		if (!PyCallable_Check(temp)) {
+			PyErr_SetString(PyExc_TypeError, "parameter must be callable");
+			return NULL;
+		}
+		Py_XINCREF(temp);
+		Py_XDECREF(report_func);
+		report_func = temp;
+
+		Py_INCREF(Py_None);
+		result = Py_None;
+	}
+	return result;
+}
+
+static PyObject *gfs2_parsefs(PyObject *self, PyObject *args)
+{
+	char *dev;
+
+	if (!PyArg_ParseTuple(args, "s:parsefs", &dev)) {
+		return NULL;
+	}
+
+	if (!gfs2_parse(dev, &report_func_wrapper)) {
+		return PyErr_SetFromErrno(PyExc_IOError);
+	}
+	Py_INCREF(Py_None);
+
+	return Py_None;
+}
+
+static PyObject *gfs2_get_block_size(PyObject *self, PyObject *args)
+{
+	char *dev;
+	uint32_t blksize;
+
+	if (!PyArg_ParseTuple(args, "s:get_block_size", &dev)) {
+		return NULL;
+	}
+
+	blksize = gfs2_block_size(dev);
+	if (!blksize) {
+		return PyErr_SetFromErrno(PyExc_IOError);
+	}
+
+	PyObject *size = Py_BuildValue("I", blksize);
+	if (!size) {
+		return NULL;
+	}
+	Py_INCREF(size);
+	return size;
+}
+
+static PyObject *gfs2_handle_sigint(PyObject *signum, PyObject *frame)
+{
+	gfs2_stop();
+	Py_INCREF(Py_None);
+	return Py_None;
+}
+
+static PyMethodDef GFS2Methods[] = {
+	{"set_report_hook", gfs2_set_report_hook, METH_VARARGS,
+		"Specify a hook function through which to report blocks."},
+	{"parsefs", gfs2_parsefs, METH_VARARGS,
+		"Parses the given block device as a GFS2 file system."},
+	{"get_block_size", gfs2_get_block_size, METH_VARARGS,
+		"Returns the file system block size."},
+	{"handle_sigint", gfs2_handle_sigint, METH_VARARGS,
+		"Handles signal SIGINT."},
+	{NULL, NULL, 0, NULL}
+};
+
+PyMODINIT_FUNC initgfs2(void)
+{
+    (void)Py_InitModule("gfs2", GFS2Methods);
+}
diff --git a/askant/scripts/askant b/askant/scripts/askant
new file mode 100755
index 0000000..3375b0c
--- /dev/null
+++ b/askant/scripts/askant
@@ -0,0 +1,6 @@
+#! /usr/bin/env python
+
+from askant import askant
+
+if __name__ == '__main__':
+	askant.main()
diff --git a/askant/setup.py b/askant/setup.py
new file mode 100755
index 0000000..78daff2
--- /dev/null
+++ b/askant/setup.py
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+from distutils.core import setup, Extension
+from askant import about
+
+setup(name="askant",
+	  version=about.version,
+	  description="File system performance analysis tool",
+	  author="Andrew Price",
+	  author_email="andy@andrewprice.me.uk",
+	  url="http://andrewprice.me.uk/projects/askant",
+	  packages = ['askant','askant.fs'],
+	  ext_modules = [Extension("askant.fs.gfs2",
+		  sources = ["fsplugins/gfs2/gfs2module.c","fsplugins/gfs2/gfs2.c"],
+		  libraries = ["gfs2"],
+		  include_dirs=['../gfs2/libgfs2', '../gfs2/include', '../make'],
+		  library_dirs=['../gfs2/libgfs2'])],
+	  scripts = ['scripts/askant'])