* [RFC 5/5] SCSI Fault Injection using SystemTap
@ 2008-07-11 10:28 kushal ghosh
0 siblings, 0 replies; only message in thread
From: kushal ghosh @ 2008-07-11 10:28 UTC (permalink / raw)
To: systemtap; +Cc: akinobu.mita, k-tanaka, Anup C Shan
[-- Attachment #1: Type: text/plain, Size: 3777 bytes --]
On 7/11/08, Anup C Shan <anupcshan@gmail.com> wrote:
> Hi.
>
> We have designed a tapset for fault injection. It is meant to ease the
> process of injecting faults into the kernel. As use cases, we have
> ported in-kernel fault injection for slab and page_alloc using this
> framework. Refer Documentation/fault-injection/
>
> We have also modified the existing SCSI fault-injection systemtap script
> (http://sourceforge.net/projects/scsifaultinjtst/) to use this framework.
>
> Please find the tapset file and readme attached. The usecase scripts
> are in the follow-up mails.
>
> Comments and suggestions are welcome.
>
> Please suggest a right location to place these tapset scripts in
> SystemTap source tree.
>
> Thanks,
> Kushal & Anup
>
Attached is the patch for SCSI Fault Injection SystemTap scripts found at
http://sourceforge.net/projects/scsifaultinjtst/
A few changes have been made like
1. A new tapset called faultinject.stp have been developed for
generic fault injection
framework which can be used to inject fault in a
simplified manner. It enables the user
to inject faults by choosing number of times to inject,
probability of failure,
which process to fail etc.
2. A tapset for SCSI called scsi_tapset.stp have been developed over the above
mentioned tapset. It makes giving the command line arguments
easier in any order
of the form 'parameter=value'.
3. There is a set of default parameters in faultinject.stp which are
fij_failtimes,
fij_totaltime, fij_space, fij_probability, fij_pids to fail,
fij_verbosity, fij_taskfilter, fij_interval,
fij_debug. All these parameters are optional. They will be loaded
with their default
values if not specified. But the parameters for the scsi fault
injection are mandatory.
They are dev_major, dev_minor_min, dev_minor_max, inode_lba_flag,
inode_lba_val.
The function fij_scsi_add_option() adds all these mandatory
parameters to the
parameters' list by calling the function
fij_add_necessary_option(option_name,
help_string) defined in faultinject.stp. They have to be provided
in the form as
mentioned in the above point. Not specifying any of these will
result in an error.
4. fij_scsi_assign_params() in scsi_tapset.stp assigns all the scsi
command line
arguments' values to the global variables dev_major, dev_minor_min,
dev_minor_max, inode_lba_flag, inode_lba_val.
5. So in all the 8 scsi scripts the assignments statements have been
replaced with
fij_scsi_add_option() and fij_scsi_assign_params(), giving the
user the flexibility of
providing the command line arguments in any order.
6. The field request_bufflen in struct scsi_cmnd is not found in the
current kernel versions
so all references to it have been deleted.
7. A global variable fij_flag has been introduced in
scsi_timeout_injection.stp in all the 4
tapsets to control the execution of the probe handlers depending on
when the fault
injection took place.
8. The typecasting of THIS->__retvalue has been changed from unsigned
int to long in
functions save_start_site() and save_entry_next() to avoid
dependencies on architecture
in scsi_timeout_injection.stp
9. There is a shell script scsi_exe which is completely interactive
and asks the user script
name and taspset for execution along with scsi command
line arguments and gives a menu for optional parameters.
10. However scripts can be executed without this also like
#stap -v -g disk_rerr.stp -I fault_injection_common_scsi -I
tapset dev_major=8 dev_minor_min=0 dev_minor_max=49 inode_lba_flag=1
inode_lba_val=0 failtimes=10 probability=100 verbosity=2
11. A documentation README.new has been given.
Thanks,
Kushal & Anup
[-- Attachment #2: scsi_fault_injection_test_tool-1.0.1-using-faultinject-framework.patch --]
[-- Type: application/octet-stream, Size: 35985 bytes --]
README.new | 29 ++
disk_rerr.stp | 11
disk_rwerr.stp | 11
fault_injection_common_sata/scsi_fault_injection.stp | 7
fault_injection_common_sata/scsi_timeout_injection.stp | 12
fault_injection_common_sata_raid56/scsi_fault_injection.stp | 7
fault_injection_common_sata_raid56/scsi_timeout_injection.stp | 12
fault_injection_common_scsi/scsi_fault_injection.stp | 7
fault_injection_common_scsi/scsi_timeout_injection.stp | 12
fault_injection_common_scsi_raid56/scsi_fault_injection.stp | 7
fault_injection_common_scsi_raid56/scsi_timeout_injection.stp | 12
r_timeout.stp | 11
rw_timeout.stp | 11
sample_scripts/md_rerr_test_sample.sh | 12
sample_scripts/md_werr_test_sample.sh | 12
scsi_exe | 143 ++++++++++
sector_rerr.stp | 11
tapset/scsi_tapset.stp | 17 +
temporary_rerr.stp | 12
temporary_werr.stp | 11
w_timeout.stp | 11
21 files changed, 304 insertions(+), 74 deletions(-)
diff -urNp scsi_fault_injection_test_tool-1.0.1/disk_rerr.stp scsi_fault_injection_test_tool-1.0.1-new/disk_rerr.stp
--- scsi_fault_injection_test_tool-1.0.1/disk_rerr.stp 2008-02-20 13:25:01.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/disk_rerr.stp 2008-07-11 11:56:30.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 1
access_type = 2
}
@@ -37,3 +33,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/disk_rwerr.stp scsi_fault_injection_test_tool-1.0.1-new/disk_rwerr.stp
--- scsi_fault_injection_test_tool-1.0.1/disk_rwerr.stp 2008-02-20 13:25:01.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/disk_rwerr.stp 2008-07-11 11:57:09.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 1
access_type = 3
}
@@ -37,3 +33,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata/scsi_fault_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata/scsi_fault_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata/scsi_fault_injection.stp 2008-02-20 13:25:01.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata/scsi_fault_injection.stp 2008-07-11 11:51:09.000000000 +0530
@@ -107,7 +107,6 @@ probe module("*").function("scsi_decide_
major = $scmd->request->rq_disk->major
minor = $scmd->request->rq_disk->first_minor
block = $scmd->request->sector
- req_len = $scmd->request_bufflen
if(major == dev_major && minor == target_minor && ((block == target_block) || ((block <= target_block) && (target_block < block + (req_len >> 9)))))
{
@@ -119,7 +118,11 @@ probe module("*").function("scsi_decide_
printf("scsi_decide_disposition : major=%d minor=%d scmd=%d \n",major, minor, $scmd)
/* create fake status and sense data */
temp_failure++
+ if(fij_should_fail())
+ {
set_sense_buf($scmd, 0x02, 0x03, 0x11, 0x04)
+ fij_done_fail()
+ }
}
}
}
@@ -141,7 +144,6 @@ probe module("*").function("scsi_dispatc
{
struct_bio= $cmd->request->bio
block = $cmd->request->sector
- req_len = $cmd->request_bufflen
if($cmd->request->rq_disk != 0)
{
@@ -167,7 +169,6 @@ probe module("*").function("scsi_dispatc
printf("SCSI_DISPATCH_CMD: major= %d minor= %d \n", major, minor)
printf("SCSI_DISPATCH_CMD: flag(0:LBA, 1:inode)= %d \n", inode_lba_flag)
printf("SCSI_DISPATCH_CMD: start sector= %d \n", $cmd->request->sector)
- printf("SCSI_DISPATCH_CMD: req bufflen= %d \n", $cmd->request_bufflen)
printf("SCSI_DISPATCH_CMD: inode= %d \n", inode)
printf("SCSI_DISPATCH_CMD: scmd = %d \n", $cmd)
printf("SCSI_DISPATCH_CMD: [7]=%d [8]=%d \n", $cmd->cmnd[7],$cmd->cmnd[8])
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata/scsi_timeout_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata/scsi_timeout_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata/scsi_timeout_injection.stp 2008-02-20 13:25:01.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata/scsi_timeout_injection.stp 2008-07-11 11:53:37.000000000 +0530
@@ -41,6 +41,7 @@ global restore_state
global target_access_t
global entire_retries
global timeout_period
+global fij_flag //to keep track of fault injection
function restore_val:long (cmd:long, entrynext:long, startsite:long, func:long, hoststs:long)
%{
@@ -59,14 +60,14 @@ function restore_val:long (cmd:long, ent
function save_start_site:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
- THIS->__retvalue = (unsigned int)scmd->eh_timeout.start_site;
+ THIS->__retvalue = (long)scmd->eh_timeout.start_site;
%}
function save_entry_next:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
unsigned int tempval;
- tempval = (unsigned int)scmd->eh_timeout.entry.next;
+ tempval = (long)scmd->eh_timeout.entry.next;
scmd->eh_timeout.entry.next = NULL;
THIS->__retvalue = tempval;
%}
@@ -74,7 +75,7 @@ function save_entry_next:long (cmd:long)
probe module("*").function("scsi_delete_timer@drivers/scsi/scsi_error.c") {
- if(global_scmd == $scmd)
+ if((global_scmd == $scmd) && (fij_flag==1))
{
startsite = save_start_site($scmd)
entrynext = save_entry_next($scmd)
@@ -96,8 +97,11 @@ probe module("*").function("scsi_delete_
probe module("*").statement("scsi_add_timer@drivers/scsi/scsi_error.c")
{
- if((target_access_t != 0) && (global_scmd == $scmd))
+ fij_flag=0
+ if((target_access_t != 0) && (global_scmd == $scmd) && (fij_should_fail()))
{
+ fij_flag=1
+ fij_done_fail()
timeoutfunction = $complete
$timeout = timeout_period
printf("\n scsi_add_timer: inode= %d, scmd= %d, timeout= %d, delete_timer_block= %d, timeoutfinction= %d \n", inode, $scmd, $timeout, delete_timer_block, timeoutfunction)
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata_raid56/scsi_fault_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata_raid56/scsi_fault_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata_raid56/scsi_fault_injection.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata_raid56/scsi_fault_injection.stp 2008-07-11 11:54:21.000000000 +0530
@@ -155,7 +155,6 @@ probe module("*").function("scsi_decide_
major = $scmd->request->rq_disk->major
minor = $scmd->request->rq_disk->first_minor
block = $scmd->request->sector
- req_len = $scmd->request_bufflen
if(major == dev_major && minor == target_minor && ((block == target_block) || ((block <= target_block) && (target_block < block + (req_len >> 9)))))
{
@@ -167,7 +166,11 @@ probe module("*").function("scsi_decide_
printf("scsi_decide_disposition : major=%d minor=%d scmd=%d \n",major, minor, $scmd)
/* create fake status and sense data */
temp_failure++
+ if(fij_should_fail())
+ {
set_sense_buf($scmd, 0x02, 0x03, 0x11, 0x04)
+ fij_done_fail()
+ }
}
}
}
@@ -189,7 +192,6 @@ probe module("*").function("scsi_dispatc
{
struct_bio= $cmd->request->bio
block = $cmd->request->sector
- req_len = $cmd->request_bufflen
if($cmd->request->rq_disk != 0)
{
@@ -220,7 +222,6 @@ probe module("*").function("scsi_dispatc
printf("SCSI_DISPATCH_CMD: major= %d minor= %d \n", major, minor)
printf("SCSI_DISPATCH_CMD: flag(0:LBA, 1:inode)= %d \n", inode_lba_flag)
printf("SCSI_DISPATCH_CMD: start sector= %d \n", $cmd->request->sector)
- printf("SCSI_DISPATCH_CMD: req bufflen= %d \n", $cmd->request_bufflen)
printf("SCSI_DISPATCH_CMD: inode= %d \n", inode)
printf("SCSI_DISPATCH_CMD: scmd = %d \n", $cmd)
printf("SCSI_DISPATCH_CMD: [7]=%d [8]=%d \n", $cmd->cmnd[7],$cmd->cmnd[8])
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata_raid56/scsi_timeout_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata_raid56/scsi_timeout_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_sata_raid56/scsi_timeout_injection.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_sata_raid56/scsi_timeout_injection.stp 2008-07-11 11:55:16.000000000 +0530
@@ -41,6 +41,7 @@ global restore_state
global target_access_t
global entire_retries
global timeout_period
+global fij_flag //to keep track of fault injection
function restore_val:long (cmd:long, entrynext:long, startsite:long, func:long, hoststs:long)
%{
@@ -59,14 +60,14 @@ function restore_val:long (cmd:long, ent
function save_start_site:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
- THIS->__retvalue = (unsigned int)scmd->eh_timeout.start_site;
+ THIS->__retvalue = (long)scmd->eh_timeout.start_site;
%}
function save_entry_next:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
unsigned int tempval;
- tempval = (unsigned int)scmd->eh_timeout.entry.next;
+ tempval = (long)scmd->eh_timeout.entry.next;
scmd->eh_timeout.entry.next = NULL;
THIS->__retvalue = tempval;
%}
@@ -74,7 +75,7 @@ function save_entry_next:long (cmd:long)
probe module("*").function("scsi_delete_timer@drivers/scsi/scsi_error.c") {
- if(global_scmd == $scmd)
+ if((global_scmd == $scmd) && (fij_flag==1))
{
startsite = save_start_site($scmd)
entrynext = save_entry_next($scmd)
@@ -96,8 +97,11 @@ probe module("*").function("scsi_delete_
probe module("*").statement("scsi_add_timer@drivers/scsi/scsi_error.c")
{
- if((target_access_t != 0) && (global_scmd == $scmd))
+ fij_flag=0
+ if((target_access_t != 0) && (global_scmd == $scmd) && (fij_should_fail()))
{
+ fij_flag=1
+ fij_done_fail()
timeoutfunction = $complete
$timeout = timeout_period
printf("\n scsi_add_timer: inode= %d, scmd= %d, timeout= %d, delete_timer_block= %d, timeoutfinction= %d \n", inode, $scmd, $timeout, delete_timer_block, timeoutfunction)
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi/scsi_fault_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi/scsi_fault_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi/scsi_fault_injection.stp 2008-07-04 12:23:45.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi/scsi_fault_injection.stp 2008-07-11 11:47:10.000000000 +0530
@@ -107,7 +107,6 @@ probe module("*").function("scsi_decide_
major = $scmd->request->rq_disk->major
minor = $scmd->request->rq_disk->first_minor
block = $scmd->request->sector
- req_len = $scmd->request_bufflen
if(major == dev_major && minor == target_minor && ((block == target_block) || ((block <= target_block) && (target_block < block + (req_len >> 9)))))
{
@@ -119,7 +118,11 @@ probe module("*").function("scsi_decide_
printf("scsi_decide_disposition : major=%d minor=%d scmd=%d \n",major, minor, $scmd)
/* create fake status and sense data */
temp_failure++
+ if(fij_should_fail())
+ {
set_sense_buf($scmd, 0x02, 0x03, 0x11, 0x04)
+ fij_done_fail()
+ }
}
}
}
@@ -141,7 +144,6 @@ probe module("*").function("scsi_dispatc
{
struct_bio= $cmd->request->bio
block = $cmd->request->sector
- req_len = $cmd->request_bufflen
if($cmd->request->rq_disk != 0)
{
@@ -167,7 +169,6 @@ probe module("*").function("scsi_dispatc
printf("SCSI_DISPATCH_CMD: major= %d minor= %d \n", major, minor)
printf("SCSI_DISPATCH_CMD: flag(0:LBA, 1:inode)= %d \n", inode_lba_flag)
printf("SCSI_DISPATCH_CMD: start sector= %d \n", $cmd->request->sector)
- printf("SCSI_DISPATCH_CMD: req bufflen= %d \n", $cmd->request_bufflen)
printf("SCSI_DISPATCH_CMD: inode= %d \n", inode)
printf("SCSI_DISPATCH_CMD: scmd = %d \n", $cmd)
printf("SCSI_DISPATCH_CMD: [7]=%d [8]=%d \n", $cmd->cmnd[7],$cmd->cmnd[8])
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi/scsi_timeout_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi/scsi_timeout_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi/scsi_timeout_injection.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi/scsi_timeout_injection.stp 2008-07-11 11:48:25.000000000 +0530
@@ -41,6 +41,7 @@ global restore_state
global target_access_t
global entire_retries
global timeout_period
+global fij_flag //to keep track of fault injection
function restore_val:long (cmd:long, entrynext:long, startsite:long, func:long, hoststs:long)
%{
@@ -59,14 +60,14 @@ function restore_val:long (cmd:long, ent
function save_start_site:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
- THIS->__retvalue = (unsigned int)scmd->eh_timeout.start_site;
+ THIS->__retvalue = (long)scmd->eh_timeout.start_site;
%}
function save_entry_next:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
unsigned int tempval;
- tempval = (unsigned int)scmd->eh_timeout.entry.next;
+ tempval = (long)scmd->eh_timeout.entry.next;
scmd->eh_timeout.entry.next = NULL;
THIS->__retvalue = tempval;
%}
@@ -74,7 +75,7 @@ function save_entry_next:long (cmd:long)
probe module("*").function("scsi_delete_timer@drivers/scsi/scsi_error.c") {
- if(global_scmd == $scmd)
+ if((global_scmd == $scmd) && (fij_flag==1))
{
startsite = save_start_site($scmd)
entrynext = save_entry_next($scmd)
@@ -96,8 +97,11 @@ probe module("*").function("scsi_delete_
probe module("*").statement("scsi_add_timer@drivers/scsi/scsi_error.c")
{
- if((target_access_t != 0) && (global_scmd == $scmd))
+ fij_flag=0
+ if((target_access_t != 0) && (global_scmd == $scmd) && (fij_should_fail()))
{
+ fij_flag=1
+ fij_done_fail()
timeoutfunction = $complete
$timeout = timeout_period
printf("\n scsi_add_timer: inode= %d, scmd= %d, timeout= %d, delete_timer_block= %d, timeoutfinction= %d \n", inode, $scmd, $timeout, delete_timer_block, timeoutfunction)
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi_raid56/scsi_fault_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi_raid56/scsi_fault_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi_raid56/scsi_fault_injection.stp 2008-07-04 12:24:06.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi_raid56/scsi_fault_injection.stp 2008-07-11 11:49:15.000000000 +0530
@@ -155,7 +155,6 @@ probe module("*").function("scsi_decide_
major = $scmd->request->rq_disk->major
minor = $scmd->request->rq_disk->first_minor
block = $scmd->request->sector
- req_len = $scmd->request_bufflen
if(major == dev_major && minor == target_minor && ((block == target_block) || ((block <= target_block) && (target_block < block + (req_len >> 9)))))
{
@@ -167,7 +166,11 @@ probe module("*").function("scsi_decide_
printf("scsi_decide_disposition : major=%d minor=%d scmd=%d \n",major, minor, $scmd)
/* create fake status and sense data */
temp_failure++
+ if(fij_should_fail())
+ {
set_sense_buf($scmd, 0x02, 0x03, 0x11, 0x04)
+ fij_done_fail()
+ }
}
}
}
@@ -189,7 +192,6 @@ probe module("*").function("scsi_dispatc
{
struct_bio= $cmd->request->bio
block = $cmd->request->sector
- req_len = $cmd->request_bufflen
if($cmd->request->rq_disk != 0)
{
@@ -220,7 +222,6 @@ probe module("*").function("scsi_dispatc
printf("SCSI_DISPATCH_CMD: major= %d minor= %d \n", major, minor)
printf("SCSI_DISPATCH_CMD: flag(0:LBA, 1:inode)= %d \n", inode_lba_flag)
printf("SCSI_DISPATCH_CMD: start sector= %d \n", $cmd->request->sector)
- printf("SCSI_DISPATCH_CMD: req bufflen= %d \n", $cmd->request_bufflen)
printf("SCSI_DISPATCH_CMD: inode= %d \n", inode)
printf("SCSI_DISPATCH_CMD: scmd = %d \n", $cmd)
printf("SCSI_DISPATCH_CMD: [7]=%d [8]=%d \n", $cmd->cmnd[7],$cmd->cmnd[8])
diff -urNp scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi_raid56/scsi_timeout_injection.stp scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi_raid56/scsi_timeout_injection.stp
--- scsi_fault_injection_test_tool-1.0.1/fault_injection_common_scsi_raid56/scsi_timeout_injection.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/fault_injection_common_scsi_raid56/scsi_timeout_injection.stp 2008-07-11 11:50:04.000000000 +0530
@@ -41,6 +41,7 @@ global restore_state
global target_access_t
global entire_retries
global timeout_period
+global fij_flag //to keep track of fault injection
function restore_val:long (cmd:long, entrynext:long, startsite:long, func:long, hoststs:long)
%{
@@ -59,14 +60,14 @@ function restore_val:long (cmd:long, ent
function save_start_site:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
- THIS->__retvalue = (unsigned int)scmd->eh_timeout.start_site;
+ THIS->__retvalue = (long)scmd->eh_timeout.start_site;
%}
function save_entry_next:long (cmd:long)
%{
struct scsi_cmnd * scmd = (struct scsi_cmnd *)(long)THIS->cmd;
unsigned int tempval;
- tempval = (unsigned int)scmd->eh_timeout.entry.next;
+ tempval = (long)scmd->eh_timeout.entry.next;
scmd->eh_timeout.entry.next = NULL;
THIS->__retvalue = tempval;
%}
@@ -74,7 +75,7 @@ function save_entry_next:long (cmd:long)
probe module("*").function("scsi_delete_timer@drivers/scsi/scsi_error.c") {
- if(global_scmd == $scmd)
+ if((global_scmd == $scmd) && (fij_flag==1))
{
startsite = save_start_site($scmd)
entrynext = save_entry_next($scmd)
@@ -96,8 +97,11 @@ probe module("*").function("scsi_delete_
probe module("*").statement("scsi_add_timer@drivers/scsi/scsi_error.c")
{
- if((target_access_t != 0) && (global_scmd == $scmd))
+ fij_flag=0
+ if((target_access_t != 0) && (global_scmd == $scmd) && (fij_should_fail()))
{
+ fij_flag=1
+ fij_done_fail()
timeoutfunction = $complete
$timeout = timeout_period
printf("\n scsi_add_timer: inode= %d, scmd= %d, timeout= %d, delete_timer_block= %d, timeoutfinction= %d \n", inode, $scmd, $timeout, delete_timer_block, timeoutfunction)
diff -urNp scsi_fault_injection_test_tool-1.0.1/README.new scsi_fault_injection_test_tool-1.0.1-new/README.new
--- scsi_fault_injection_test_tool-1.0.1/README.new 1970-01-01 05:30:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/README.new 2008-07-11 12:28:32.000000000 +0530
@@ -0,0 +1,29 @@
+This scsi fault injection mechanism has been integrated with the generic tapset developed which gives the user more flexibility to control the fault injection procedure.
+The new tapset "faultinject-tapset.stp" has been designed to provide more features like the number of times to inject fault, probability of failure, more verbosity etc.
+The "scsi_tapset.stp" makes the scsi implementation simpler. Instead of giving all necessary parameters in the command line in fixed order the user can give them in any order in the format as "parametar=value".
+The necessary paramaters for scsi are device major, device minor(min and max), inode/lba flag, inode/lba value etc has to be provided in the command line in the format as specified.
+
+NOTE:Not specifying any of the needed parameters will result in an error.
+
+A sample implementation could be:
+
+ #stap -v disk_rerr.stp -g -I fault_injection_common_scsi dev_major=8 dev_minor_min=0 dev_minor_max=49 inode_lba_flag=1 inode_lba_val=0
+
+injects a disk read error for scsi disks with major 8(for HDD) within the range of 0 to 49 for the file with the inode value 0.
+
+For help give the "help" in command line
+
+ #stap -v disk_rerr.stp -g -I fault_injection_common_scsi help
+
+displays all the necessary parameters and the optional parameters(failtimes, probability, verbosity, taskfilter, space, totaltime) and their meanings.
+
+A shell script has been added called "scsi_exe".
+
+Just run ./scsi_exe.
+It is a completely interactive script which takes all necessary parameters from user along with script name and tapset path.
+Also gives a menu allowing user to specify whichever optional parameters he needs to change.
+
+The number of processes to fail could be specified by setting the taskfilter to 1 and giving the pids in the command line.
+
+ #stap -v disk_rerr.stp -g -I fault_injection_common_scsi dev_major=8 dev_minor_min=0 dev_minor_max=49 inode_lba_flag=1 inode_lba_val=0 fij_taskfilter=1 fij_pid=0 fij_pid=19999
+
diff -urNp scsi_fault_injection_test_tool-1.0.1/r_timeout.stp scsi_fault_injection_test_tool-1.0.1-new/r_timeout.stp
--- scsi_fault_injection_test_tool-1.0.1/r_timeout.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/r_timeout.stp 2008-07-11 11:57:45.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 0
access_type = 2
@@ -42,3 +38,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/rw_timeout.stp scsi_fault_injection_test_tool-1.0.1-new/rw_timeout.stp
--- scsi_fault_injection_test_tool-1.0.1/rw_timeout.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/rw_timeout.stp 2008-07-11 11:58:25.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 1
access_type = 3
@@ -40,3 +36,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/sample_scripts/md_rerr_test_sample.sh scsi_fault_injection_test_tool-1.0.1-new/sample_scripts/md_rerr_test_sample.sh
--- scsi_fault_injection_test_tool-1.0.1/sample_scripts/md_rerr_test_sample.sh 2008-02-21 11:31:04.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/sample_scripts/md_rerr_test_sample.sh 2008-07-04 15:54:05.000000000 +0530
@@ -11,10 +11,15 @@ RAIDENV=$3
SCRIPTNAME=$4
RAIDDEV=/dev/md0
TESTFILENAME='targetfile_'$(date --rfc-3339=date)
-TARGETDEVS="8 16 81"
+#the parameters TARGETDEVS is modified to match with current framework by separating the major, minor_min, minor_max
+TARGETDEVS_major="8"
+TARGETDEVS_minor_min="16"
+TARGETDEVS_minor_max="81"
if [ $RAIDENV = "recov" ]; then
- TARGETDEVS="8 33 81"
+ TARGETDEVS_major="8"
+ TARGETDEVS_minor_min="33"
+ TARGETDEVS_minor_max="81"
fi
@@ -72,7 +77,8 @@ echo "TARGETINODE= $TARGETINODE"
echo "# Runing $SCRIPTNAME "
echo "# "
echo "#############################"
- stap $TOOLDIR/$SCRIPTNAME $TARGETDEVS 1 $TARGETINODE -g -I $TOOLDIR/fault_injection_common_$DEVTYPE/ -v | tee stapresult.txt &
+#change in the format of stap command
+ stap $TOOLDIR/$SCRIPTNAME dev_major=$TARGETDEVS_major dev_minor_min=$TARGETDEVS_minor_min dev_minor_max=$TARGETDEVS_minor_max inode_lba_flag=1 inode_lba_val=$TARGETINODE -g -I $TOOLDIR/fault_injection_common_$DEVTYPE/ -I $TOOLDIR/tapset -v | tee stapresult.txt &
echo "waiting stap startup"
sleep 1
diff -urNp scsi_fault_injection_test_tool-1.0.1/sample_scripts/md_werr_test_sample.sh scsi_fault_injection_test_tool-1.0.1-new/sample_scripts/md_werr_test_sample.sh
--- scsi_fault_injection_test_tool-1.0.1/sample_scripts/md_werr_test_sample.sh 2008-02-21 11:31:52.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/sample_scripts/md_werr_test_sample.sh 2008-07-04 15:54:47.000000000 +0530
@@ -11,10 +11,15 @@ RAIDENV=$3
SCRIPTNAME=$4
RAIDDEV=/dev/md0
TESTFILENAME='targetfile_'$(date --rfc-3339=date)
-TARGETDEVS="8 16 81"
+#the parameters TARGETDEVS is modified to match with current framework by separating the major, minor_min, minor_max
+TARGETDEVS_major="8"
+TARGETDEVS_minor_min="16"
+TARGETDEVS_minor_max="81"
if [ $RAIDENV = "recov" ]; then
- TARGETDEVS="8 33 81"
+ TARGETDEVS_major="8"
+ TARGETDEVS_minor_min="33"
+ TARGETDEVS_minor_max="81"
fi
@@ -76,7 +81,8 @@ echo "TARGETINODE= $TARGETINODE"
echo "# Runing $SCRIPTNAME "
echo "# "
echo "#############################"
- stap $TOOLDIR/$SCRIPTNAME $TARGETDEVS 1 $TARGETINODE -g -I $TOOLDIR/fault_injection_common_$DEVTYPE/ -v | tee stapresult.txt &
+#change in the format of stap command
+ stap $TOOLDIR/$SCRIPTNAME dev_major=$TARGETDEVS_major dev_minor_min=$TARGETDEVS_minor_min dev_minor_max=$TARGETDEVS_minor_max inode_lba_flag=1 inode_lba_val=$TARGETINODE -g -I $TOOLDIR/fault_injection_common_$DEVTYPE/ -I $TOOLDIR/tapset-v | tee stapresult.txt &
echo "waiting stap startup"
sleep 1
diff -urNp scsi_fault_injection_test_tool-1.0.1/scsi_exe scsi_fault_injection_test_tool-1.0.1-new/scsi_exe
--- scsi_fault_injection_test_tool-1.0.1/scsi_exe 1970-01-01 05:30:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/scsi_exe 2008-07-11 14:08:14.000000000 +0530
@@ -0,0 +1,143 @@
+#a script to read the command line parameters from the user
+#for executing the scsi fault injection scripts
+
+val_time=1000
+val_fail=0
+val_prob=0
+val_verbo=0
+val_task=0
+val_int=0
+val_space=0
+val_debug=0
+
+echo Do you need help:y for yes n for no
+read help
+
+echo
+
+echo Enter the script name:
+read script
+
+echo Enter the tapset path:
+read path
+
+if [ $help == "y" ]
+then
+ stap -v -g $script -I tapset -I $path help
+fi
+
+if [ $help == "n" ]
+then
+
+ echo
+
+ echo SPECIFY THE NEEDED PARAMETERS
+ echo
+
+ echo Enter the device major number:
+ read major
+
+ echo Enter the device minor number minimum:
+ read minor_min
+
+ echo Enter the device minor number maximum:
+ read minor_max
+
+ echo Enter the flag for inode or lba specification:0 for lba and 1 for inode
+ read flag
+
+ echo Enter the inode or lba value as per flag specification:
+ read value
+
+ i=1
+
+ echo
+ echo MENU FOR OPTIONAL PARAMETERS
+ echo
+
+ while [ $i -eq 1 ]
+ do
+ echo
+ echo 1 Failtimes
+ echo 2 Probability
+ echo 3 Totaltime
+ echo 4 Verbosity
+ echo 5 Taskfilter
+ echo 6 Interval
+ echo 7 Space
+ echo 8 Debug
+ echo 9 Exit
+ echo Enter choice:
+ read choice
+
+ if [ $choice -lt 9 ]
+ then
+ echo Enter the required value:
+ read val
+ fi
+ if [ $choice -eq 1 ]
+ then
+ val_fail=$val
+ fi
+ if [ $choice -eq 2 ]
+ then
+ val_prob=$val
+ fi
+ if [ $choice -eq 3 ]
+ then
+ val_time=$val
+ fi
+ if [ $choice -eq 4 ]
+ then
+ val_verbo=$val
+ fi
+ if [ $choice -eq 5 ]
+ then
+ val_task=$val
+ fi
+
+ if [ $choice -eq 6 ]
+ then
+ val_int=$val
+ fi
+ if [ $choice -eq 7 ]
+ then
+ val_space=$val
+ fi
+ if [ $choice -eq 8 ]
+ then
+ val_debug=$val
+ fi
+ if [ $choice -eq 9 ]
+ then
+ i=2
+ fi
+
+ done
+
+ #read pid list only if taskfilter is set to 1
+
+ if [ $val_task -eq 1 ]
+ then
+ echo Enter the total number of pids:
+ read num
+ i=0
+
+ while [ $i -lt $num ]
+ do
+ echo Enter pid:
+ read pid
+ str=$str" pid="$pid
+ i=$[$i+1]
+ done
+ fi
+
+ #clearing cache entries
+
+ echo 1 > /proc/sys/vm/drop_caches
+
+ #running the script with all parameters with their values if specified by user else the default values
+
+ stap -v -g $script -I tapset -I $path dev_major=$major dev_minor_min=$minor_min dev_minor_max=$minor_max inode_lba_flag=$flag inode_lba_val=$value failtimes=$val_fail probability=$val_prob totaltime=$val_time taskfilter=$val_task verbosity=$val_verbo interval=$val_int debug=$val_debug space=$val_space$str
+
+fi
diff -urNp scsi_fault_injection_test_tool-1.0.1/sector_rerr.stp scsi_fault_injection_test_tool-1.0.1-new/sector_rerr.stp
--- scsi_fault_injection_test_tool-1.0.1/sector_rerr.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/sector_rerr.stp 2008-07-11 11:59:34.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 1
access_type = 4
}
@@ -37,3 +33,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/tapset/scsi_tapset.stp scsi_fault_injection_test_tool-1.0.1-new/tapset/scsi_tapset.stp
--- scsi_fault_injection_test_tool-1.0.1/tapset/scsi_tapset.stp 1970-01-01 05:30:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/tapset/scsi_tapset.stp 2008-07-11 12:13:47.000000000 +0530
@@ -0,0 +1,17 @@
+function fij_scsi_add_option()
+{
+ fij_add_necessary_option("dev_major","device major number:8 for HDD")
+ fij_add_necessary_option("dev_minor_min","device minor starting value")
+ fij_add_necessary_option("dev_minor_max","device minor ending value")
+ fij_add_necessary_option("inode_lba_flag","flag for providing the lba or inode value:0 for lba and 1 for inode")
+ fij_add_necessary_option("inode_lba_val","lba value or the inode number depending on the flag")
+}
+
+function fij_scsi_assign_params()
+{
+ dev_major=fij_params["dev_major"]
+ dev_minor_min=fij_params["dev_minor_min"]
+ dev_minor_max=fij_params["dev_minor_max"]
+ inode_lba_flag=fij_params["inode_lba_flag"]
+ inode_lba_val=fij_params["inode_lba_val"]
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/temporary_rerr.stp scsi_fault_injection_test_tool-1.0.1-new/temporary_rerr.stp
--- scsi_fault_injection_test_tool-1.0.1/temporary_rerr.stp 2008-02-20 13:25:01.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/temporary_rerr.stp 2008-07-11 12:01:36.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 0
access_type = 2
}
@@ -36,3 +32,9 @@ probe end
{
printf("\nDONE\n")
}
+
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/temporary_werr.stp scsi_fault_injection_test_tool-1.0.1-new/temporary_werr.stp
--- scsi_fault_injection_test_tool-1.0.1/temporary_werr.stp 2008-02-20 13:25:01.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/temporary_werr.stp 2008-07-11 12:04:02.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 0
access_type = 1
}
@@ -37,3 +33,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
diff -urNp scsi_fault_injection_test_tool-1.0.1/w_timeout.stp scsi_fault_injection_test_tool-1.0.1-new/w_timeout.stp
--- scsi_fault_injection_test_tool-1.0.1/w_timeout.stp 2008-02-20 13:25:00.000000000 +0530
+++ scsi_fault_injection_test_tool-1.0.1-new/w_timeout.stp 2008-07-11 11:58:56.000000000 +0530
@@ -23,11 +23,7 @@
probe begin
{
- dev_major = $1
- dev_minor_min = $2
- dev_minor_max = $3
- inode_lba_flag = $4
- inode_lba_val = $5
+ fij_scsi_add_option()
error_type = 0
access_type = 1
@@ -40,3 +36,8 @@ probe end
printf("\nDONE\n")
}
+//scsi arguments are added at probe begin and parsed from the command line in probe begin(1000)
+probe begin(2000)
+{
+ fij_scsi_assign_params()
+}
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2008-07-11 10:28 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-07-11 10:28 [RFC 5/5] SCSI Fault Injection using SystemTap kushal ghosh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).