[RFC 1/5] Kernel Fault injection framework using SystemTap

[RFC 1/5] Kernel Fault injection framework using SystemTap

[Anup C Shan]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

Hi.

We have designed a tapset for fault injection. It is meant to ease the 
process of injecting faults into the kernel. As use cases, we have 
ported in-kernel fault injection for slab and page_alloc using this 
framework. Refer Documentation/fault-injection/

We have also modified the existing SCSI fault-injection systemtap script 
(http://sourceforge.net/projects/scsifaultinjtst/) to use this framework.

Please find the tapset file and readme attached. The usecase scripts
are in the follow-up mails.

Comments and suggestions are welcome.

Please suggest a right location to place these tapset scripts in 
SystemTap source tree.

Thanks,
Kushal & Anup




[-- Attachment #2: README.faultinject --]
[-- Type: text/plain, Size: 4500 bytes --]

Introduction
------------

This tapset provides a framework to facilitate fault injections for
testing the kernel. The framework can be used by systemtap scripts to
actually inject faults. The framework processes the command line 
arguments and controls the fault injection process.

Following are the generic parameters used to set up the fault injection.
	a) failtimes - maximum number of times the process can be failed
	b) interval - number of successful hits between potential failures
	c) probability - probability of potential failure
	d) taskfilter - fail all processes or filter processes on pid
	e) space - number of successful hits before the first failure
	f) verbosity - control amount of output generated by the script
	g) totaltime - duration of fault injection session
	h) debug - print debug information for the script
	i) pid - process IDs of processes to inject failures into. This can 
		 also be specified using the -x option.

These parameters are registered in the tapset using the fij_add_option()
function which also sets the script specific default values and provides help 
text. The generic parameters are appended to the params[] array and can be 
accessed using params["variable_name"]. If you doesn't specify any of the 
parameters in command line, its default value is used. 
Using fij_load_param(), your script can also assign script-specific default
values to generic parameters.

You can define mandatory parameters, which are specific to the script depending
upon the kernel subsystem under test. These variables must necessarily be
specified on the command line during command execution. 
E.g: device numbers, inode numbers etc which cannot be given default values.

Such parameters can be registered using the fij_add_necessary_option() function.
On calling this function, the variable is appended to a mandatoryparams[] array.
If these parameters are not specified on the command line, an error is reported
and script is aborted. The variable can be accessed at params["variable_name"].

The framework controls the fault injection using fij_should_fail() and 
fij_done_fail() functions. Your script should probe the relevant kernel 
routine subjected to fault injection. The user-defined probe handler invokes 
fij_should_fail(), which returns 1 if it's time to inject a failure, or 0 
otherwise. Faults can be injected by your script in various ways like faking the
error return by changing the return value, by modifying data structures etc.
fij_done_fail() must be called immediately after fault injection to alert the
tapset of this. fij_done_fail() must not be called in case no fault was
injected.

fij_logger() - This is a wrapper for the SystemTap log() function with an added 
verbosity parameter. The message will be displayed only if the value of global
fij_verbosity is equal to or more than the parameter provided to the 
function.


How to use the tapset
---------------------

1) begin probe that adds user defined parameters and default values.
2) Probes for fault injection. Call fij_should_fail() before injecting the
   fault and fij_done_fail() after fault is injected.


Description of code flow
------------------------

1) begin(less than -1000) in the user script [OPTIONAL] - Preinitialization. 
   As of now, this is not necessary.

2) begin(-1000) in the tapset - This function initialises counters and
   registers all generic parameters with global defaults.

3) begin in the user script - User defined default parameters are supplied 
   here. Also any script specific parameters are registered at this stage.

4) begin(1000) in the tapset - Command line arguments are parsed and
   parameters assigned appropriate values.

5) begin(more than 1000) in the user script [OPTIONAL] - This can be used
   to copy values of arguments from params[] array to local/global variables 
   for easy referencing.

6) Script starts executing. It is interrupted every 10 milliseconds to
   check if script has run for the stipulated length of time.

7) When function/statement probes are hit, the script must invoke
   fij_should_fail() function to check if the conditions for failure have been
   satisfied.

8) Fail the function using suitable methods (changing return values,
   setting fake values to variables...)

9) Call fij_done_fail() function to inform tapset that fault has been injected.

10) Script will exit either when script calls exit() function or when a
    timeout is hit. At this point, stats of the experiment are printed.

[-- Attachment #3: faultinject.stp --]
[-- Type: text/plain, Size: 8789 bytes --]

%{
#include<linux/random.h>
%}

global fij_params	//Array of all parameters. (except fij_pids_to_fail)
global fij_paramshelp	//Array of help information for all parameters
global fij_mandatoryparams	
			//Array of mandatory parameters 

global fij_pids_to_fail	//Array of pids subject to fault injection
global fij_failcount	//Number of times failed so far
global fij_probehits	//Number of times the probe has been hit
global fij_intervalcount
			//Number of successful probe hits
global fij_aborted	//Boolean value to check whether the fault injection
			//procedure needs to continue or not
			//Needed for help option

global fij_failtimes
global fij_verbosity
global fij_debug
global fij_taskfilter
global fij_interval
global fij_probability
global fij_space
global fij_totaltime

function fij_random:long()
%{
	THIS->__retvalue = random32();
%}

function fij_add_process_to_fail(procid:long)
{
	fij_logger(1, sprintf("Adding process %d to the fail list", procid))
	fij_pids_to_fail[procid] = 1
}

/*
 * Add an option to the parameters list
 * This option can be provided on the command line as opt = value
 */
function fij_add_option(opt:string, defval, help:string)
{
	fij_params[opt] = defval
	fij_paramshelp[opt] = help
}

/*
 * Add an option to the necessary parameters list
 * This option MUST be provided on the command line
 */
function fij_add_necessary_option(opt:string, help:string)
{
	fij_mandatoryparams[opt] = 1
	fij_paramshelp[opt] = help
}

function fij_print_help()
{
	fij_logger(0, "Usage : stap script.stp [ option1=value1 [ option2=value2 [ ...]]]")
	fij_logger(0, "Options : ")
	fij_logger(0, "\tpid\r\t\t\t\t : PID of a process to be failed. Use this option repeatedly to add multiple processes to fail")
	
	foreach (option in fij_params) {
		fij_logger(0, sprintf("\t%s\r\t\t\t\t : %s", option,
							fij_paramshelp[option]))
	}

	needed_options_counter = 0
	foreach (option in fij_mandatoryparams) {
		if (needed_options_counter == 0) {
			fij_logger(0, "Necessary options : ")
			needed_options_counter++
		}
		fij_logger(0, sprintf("\t%s\r\t\t\t\t : %s", option,
							fij_paramshelp[option]))
	}

	fij_logger(0, "For help : stap script.stp help")
	fij_aborted = 1
}

function fij_process_argument(arg:string)
{
	if (isinstr(arg, "=") == 1) {
		parameter=tokenize(arg, "=")
		value_in_str = tokenize("", "=")
		value = strtol(value_in_str, 10)
		if (parameter in fij_params) {
			fij_params[parameter] = value
			fij_logger(1, sprintf("Parameter %s is assigned value %d",
					parameter, fij_params[parameter]))
		} else if (parameter in fij_mandatoryparams) {
			fij_add_option(parameter, value,
						fij_paramshelp[parameter])
			delete fij_mandatoryparams[parameter]
		} else if (parameter == "pid") {
			fij_add_process_to_fail(value)
		} else
			fij_logger(0, sprintf("WARNING : Argument %s is not found in parameter list. Ignoring..", parameter))
	}
	else
		fij_logger(0, sprintf("WARNING : Invalid command line argument : %s",
									arg))
}

function fij_show_params()
{
	fij_logger(1, "Status of parameters :")
	foreach (option in fij_params)
		fij_logger(1, sprintf("Option %s has value %d", option,
							fij_params[option]))
}

/*
 * Parse command line arguments
 */
function fij_parse_command_line_args()
{
	for (i = 1; i <= argc ; i++) {
		if (argv[i] == "help") {
			fij_print_help()
			return 0
		} else
			fij_process_argument(argv[i])
	}
	
	foreach (parameter in fij_mandatoryparams) {
		fij_logger(0, sprintf("ERROR: Necessary command line parameter %s not specified", parameter))
		fij_aborted = 1
	}
}

/*
 * Load script specific default parameters
 * This function is called by the script using this tapset to set custom
 * default values
 */
function fij_load_param(arg_times:long, arg_interval:long, arg_probability:long,
			arg_taskfilter:long, arg_space:long, arg_verbose:long,
							arg_totaltime:long)
{
	fij_add_option("failtimes", arg_times,
			"Number of times to fail (0 = no limit)")
	fij_add_option("interval", arg_interval, "Number of successful hits between potential failures (0 to fail everytime)")
	fij_add_option("probability", arg_probability,
		"Probability of failure (1<=probability<=100) (0 to disable)")
	fij_add_option("taskfilter", arg_taskfilter, "0=>Fail all processes, 1=>Fail processes based on pid command line argument or -x option.")
	fij_add_option("space", arg_space, "Number of successful hits before the first failure (0 to disable)")
	fij_add_option("verbosity", arg_verbose, "0=>Success or Failure messages, 1=>Print parameter status, 2=>All probe hits, backtrace and register states")
	fij_add_option("totaltime", arg_totaltime, "Duration of fault injection session in milliseconds (Default : 1000 milliseconds)")
}

/*
 * Modified log function with an additional verbosity parameter
 * The message is printed only if the current fij_verbosity parameter
 * is greater than the minimum verbosity specified. Minverbosity value of 100
 * is reserved only for debugging the script.
 */
function fij_logger(minverbosity:long, msg:string)
{
	if (fij_verbosity >= minverbosity)
		log(msg)
	else if (minverbosity == 100 && fij_debug == 1)
		log(msg)
}

/*
 * Checks whether the specified constraints for failure have been met
 * Returns 1 if process must be failed,  else returns 0
 */
function fij_should_fail:long()
{
	fij_probehits++

	fij_logger(2, "Probe hit")

	if (fij_taskfilter != 0) {
		if(!(pid() in fij_pids_to_fail)) {
			fij_logger(100, sprintf("Skipping because wrong process %d - %s probed",
						pid(), execname()))
			return 0
		} else	
			fij_logger(100, sprintf("Continuing with probing process %d - %s %d",
						pid(), execname(), target()))
	}

	if (fij_failcount == 0) {
		if (fij_space != 0)	{
			if (fij_intervalcount < fij_space) {
				fij_logger(100, sprintf("Skipping on space : %d",
								fij_intervalcount))
				fij_intervalcount++
				return 0
			} else {
				fij_intervalcount = 0
				fij_logger(100, sprintf("Done skipping on space"))
				fij_space = 0
			}
		}
	}

	if (fij_failtimes != 0 && fij_failcount >= fij_failtimes) {
			fij_logger(100, sprintf("Failed %d times already. Skipping..",
						fij_failcount))
			return 0
	}

	if (fij_interval != 0) {
		if (fij_intervalcount != 0) {
			fij_logger(100, sprintf("Skipping on interval : %d",
								fij_intervalcount))
			fij_intervalcount++
			fij_intervalcount %= fij_interval
			return 0
		} else
			fij_intervalcount++
	}

	if (fij_probability != 0) {
		if (fij_random() % 100 > fij_probability) {
			fij_logger(100, sprintf("Skipping on probability"))
			return 0
		} else
			fij_logger(100, sprintf("Continuing on probability"))
	}

	return 1
}

/*
 * Post injection cleanup
 * This function MUST be called after the process has been failed
 */
function fij_done_fail()
{
	fij_failcount++
	fij_logger(0, sprintf("Failed process %d - %s", pid(), execname()))
	
	if (fij_params["verbosity"] >= 2 && fij_verbosity != 100) {
		print_backtrace()
		print_regs()
	}
}

function fij_display_stats()
{
	fij_logger(0, sprintf("Probe was hit %d times.", fij_probehits))
	fij_logger(0, sprintf("Function was failed %d times.", fij_failcount))
}

/*
 * The first begin function
 * Initialises counters and adds generic parameters to the parameters list
 * In case the script requires a begin function to be executed prior to this,
 * parameter of less than -1000 must be specified to begin()
 */
probe begin(-1000)
{
	fij_failcount = 0
	fij_probehits = 0
	fij_intervalcount = 0
	fij_aborted = 0

	fij_load_param(0, 0, 0, 0, 0, 0, 1000)	//Loading default values
	fij_add_option("debug", 0, "Display debug information. Requires verbosity=100")

	fij_mandatoryparams["initialize"] = 1;	//Needed to register mandatory
						//params as an array
	delete fij_mandatoryparams["initialize"]
}

/*
 * The last begin function
 * Does parsing of command line arguments
 * In case the script requires a begin function to be executed after all
 * initialization,  parameter of greater than 1000 must be specified to begin()
 * Eg: when you need to copy one of the fij_params[] options into a local variable 
 * after parsing command line args
 */
probe begin(1000)
{
	if (target()!=0)
		fij_add_process_to_fail(target())

	fij_parse_command_line_args()

	fij_failtimes = fij_params["failtimes"]
	fij_interval = fij_params["interval"]
	fij_probability = fij_params["probability"]
	fij_taskfilter = fij_params["taskfilter"]
	fij_space = fij_params["space"]
	fij_verbosity = fij_params["verbosity"]
	fij_totaltime = fij_params["totaltime"]
	fij_debug = fij_params["debug"]

	if (fij_aborted)
		exit()
	else
		fij_show_params()
}

probe end
{
	if (!fij_aborted)
		fij_display_stats()
}

//Check every 10 ms if the stipulated execution time has expired
probe timer.ms(10)
{
	fij_totaltime -= 10
	if (fij_totaltime <= 0)
		exit()
}

[RFC 2/5] Memory subsystem fault injection tapset

[Anup C Shan]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: mm-faultinject.stp --]
[-- Type: text/plain, Size: 1270 bytes --]

/*
 * This tapset contains functions and parameters specific to the 
 * memory management subsystem of the kernel.
 */

function fij_add_gfp_wait_param()
{
	fij_add_option("ignore_gfp_wait",1,
			"inject failures only into non-sleep allocations")
}

function fij_get_gfp_wait()
%{
	THIS->__retvalue = __GFP_WAIT;
%}

function fij_should_fail_gfp_wait(flags:long)
{
	GFPWAIT = fij_get_gfp_wait()
	if (fij_params["ignore_gfp_wait"] == 1) {
		if (flags&GFPWAIT) {
			fij_logger(100,
				sprintf("Skipping on ignore_gfp_wait %d",
								flags&GFPWAIT))
			return 0
		} else
			fij_logger(100,
				sprintf("Continuing on ignore_gfp_wait %d",
								flags&GFPWAIT))
	}
	return 1
}

function fij_add_gfp_highmem_param()
{
	fij_add_option("ignore_gfp_highmem",1,
			"1 => inject failures highmem/user allocations")
}

function fij_get_gfp_highmem()
%{
	THIS->__retvalue = __GFP_HIGHMEM;
%}

function fij_should_fail_gfp_highmem(flags:long)
{
	GFPHIGHMEM = fij_get_gfp_highmem()
	if (fij_params["ignore_gfp_highmem"] == 1) {
		if (flags&GFPHIGHMEM) {
			fij_logger(100,
				sprintf("Skipping on ignore_gfp_highmem %d",
							flags&GFPHIGHMEM))
			return 0
		} else
			fij_logger(100,
				sprintf("Continuing on ignore_gfp_highmem %d"
							,flags&GFPHIGHMEM))
	}
	return 1
}

[RFC 4/5] Page allocation fault injection script

[Anup C Shan]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: alloc_page-faultinject.stp --]
[-- Type: text/plain, Size: 2063 bytes --]

#!/usr/local/bin/stap

/*
 * This script is used to fail __alloc_pages() function in mm/page_alloc.c
 */

probe begin(0)
{
	fij_load_param(20000,0,100,0,0,1,1000);
	fij_add_option("min_order",0,"Minimum order(base 2) of the allocation")
	fij_add_gfp_wait_param()
	fij_add_gfp_highmem_param()
}

global min_order

probe begin(2000)
{
	min_order = fij_params["min_order"]
}

function should_fail_alloc(order:long,flags:long)
{
	if (order < min_order) {
		fij_logger(100,sprintf("Skipping on order %d",order))
		return 0
	} else
		fij_logger(100,sprintf("Continuing on order %d",order))

	if (fij_should_fail_gfp_wait(flags) == 0)
		return 0

	if (fij_should_fail_gfp_highmem(flags) == 0)
		return 0

	if (fij_should_fail() == 0)
		return 0

	return 1
}

global zones_save,under_fail

/*
 * zones_save : Temporary variable to save zone information that will be 
 * assigned fake value to induce fault.
 * under_fail : Flag to denote whether system is under fault injected condition.
 * Both these variables are 'per-cpu' variables to prevent race conditions.
 */

function nullify(zptr:long,cpu:long)
%{
	struct zones **z = (struct zones **)(THIS->zptr);

	THIS->__retvalue = (long)*z;
	*z = NULL;
%}

/*
 * Method of fault injection : 
 * The function __alloc_pages() is failed by using an if condition on
 * variable z (=zonelist->zones).
 * To fail, the value of *(zonelist->zones) is saved in zones_save and
 * set to NULL to cause the function to return NULL value.
 * Once the fault is injected, the value of zonelist->zones is restored.
 */

probe kernel.function("__alloc_pages@mm/page_alloc.c")
{
	if (should_fail_alloc($order,$gfp_mask) == 0)
		next

	zones_save[cpu()] = nullify($zonelist->zones,cpu())
	under_fail[cpu()] = 1
	fij_done_fail()
}

function restore(zones:long,zones_save:long)
%{
	struct zones **z = (struct zones **)(THIS->zones);
	*z = (struct zones *)THIS->zones_save;
%}

probe kernel.function("__alloc_pages@mm/page_alloc.c").return
{
	if (under_fail[cpu()]) {
		restore($zonelist->zones,zones_save[cpu()])
		under_fail[cpu()] = 0
	}
}

[RFC 3/5] Slab allocation fault injection script

[Anup C Shan]

[-- Attachment #1: Type: text/plain, Size: 109 bytes --]

The following script implements fault injection for kmem_cache_alloc() 
using the fault injection framework.

[-- Attachment #2: slab-faultinject.stp --]
[-- Type: text/plain, Size: 981 bytes --]

#!/usr/local/bin/stap
%{
#include<linux/slab.h>
#include<linux/gfp.h>
%}

probe begin
{
	fij_load_param(0,0,100,0,0,1,2000)
	fij_add_gfp_wait_param()
}

/*
 * Free the node if it was allocated. Effectively rolling back actions of
 * kmem_cache_alloc() function.
 */
function cleanup_alloc(cacheptr:long,objptr:long)
%{
	void *objp = (void *)THIS->objptr;
	struct kmem_cache *cachep = (struct kmem_cache *)THIS->cacheptr;
	if (objp != NULL)	{
		kmem_cache_free(cachep,objp);
		objp = NULL;
	}
%}

/*
 * Check additional parameter of ignore_gfp_wait
 */
function should_fail_slab:long (flags:long)
{
	if (fij_should_fail_gfp_wait(flags) == 0)
		return 0

	return fij_should_fail()
}

/*
 * Method of fault injection:
 * Free the cache page created using kmem_cache_free() and send
 * fake return value NULL.
 */

probe kernel.function("kmem_cache_alloc@mm/slab.c").return
{

	if (should_fail_slab($flags) == 0)
		next

	cleanup_alloc($cachep,$return)
	$return = 0
	fij_done_fail()
}

Re: [RFC 1/5] Kernel Fault injection framework using SystemTap

[Frank Ch. Eigler]

Anup C Shan <anupcshan@gmail.com> writes:

> [...]  We have designed a tapset for fault injection. It is meant to
> ease the process of injecting faults into the kernel. As use cases,
> we have ported in-kernel fault injection for slab and page_alloc
> using this framework. Refer Documentation/fault-injection/

Neat.

> Comments and suggestions are welcome.  Please suggest a right
> location to place these tapset scripts in SystemTap source tree.

It seems like it's special-purpose enough to warrant a place in among
the examples, which are currently checked in at
.../testsuite/systemtap.examples/, as opposed to the tapset proper.
The scripts and a README could go into a new subdirectory.  For
cataloging the scripts, a brief ".meta" file should accompany each
end-user (as opposed to tapset) script.  They could each have a
"test_check" field to compile-test the scripts.  (A test_installcheck
field to actually inject faults into a running developer system is
probably not appropriate. :-)

Please place proper copyright notes and GPLv2+ license terms on the
files; then we can package them.

- FChE