[Fwd: Re: Accuracy of disk statistics IO counter]

[Fwd: Re: Accuracy of disk statistics IO counter]

[Alan D. Brunelle]

[-- Attachment #1: Type: text/plain, Size: 941 bytes --]

Here's a "success story" with SystemTAP: a developer here was having 
trouble with getting iostat results to match internal tools he was 
working on, and with the help of a _very_ simple SystemTAP script, he 
was able to determine the nature of the problem, and get going very 
quickly...

Alan
PS. Here's the script I started him with...

global rqs
global lun, id, channel, host_no

probe begin
{
        host_no = 0
        channel = 0
        id = 7
        lun = 12
}

probe module("*scsi_mod*").function("scsi_dispatch_cmd")
{
        if (1       != $cmd->sc_data_direction) next
        if (lun     != $cmd->device->lun) next
        if (id      != $cmd->device->id) next
        if (channel != $cmd->device->channel) next
        if (host_no != $cmd->device->host->host_no) next

        rqs[$cmd->request_bufflen / 1024]++
}

probe end
{
        foreach (rec+ in rqs)
              printf("%d %d\n", rec, rqs[rec])
        exit()
}

[-- Attachment #2: Re: Accuracy of disk statistics IO counter --]
[-- Type: message/rfc822, Size: 3031 bytes --]

From: Mark Seger <Mark.Seger@hp.com>
To: Jens Axboe <axboe@suse.de>
Cc: linux-kernel@vger.kernel.org, Alan Brunelle <Alan.Brunelle@hp.com>
Subject: Re: Accuracy of disk statistics IO counter
Date: Tue, 01 Aug 2006 15:30:40 -0400
Message-ID: <44CFABE0.8020501@hp.com>


>>Specifically, I wrote a 1GB file with a blocksize of 1MB, which would 
>>result in 1000 writes at the application level.  What I believe then 
>>happens is that each write turns into 8 128KB requests to the driver, 
>>which should result in 8000 actual writes.  Toss in metadata operations 
>>and who knows what else and the actual number should be a little 
>>higher.  What I've see after repeating the tests a number of times on 
>>2.6.16-27 is numbers ranging from 6800-7000 writes which feels like a 
>>big enough difference to at least point out.
>>    
>>
>
>Install http://brick.kernel.dk/snaps/blktrace-git-20060723022503.tar.gz
>and blktrace your disk for the duration of the test and compare the io
>numbers. Requires 2.6.17 or later, though.
>  
>
I had problems getting blktrace going and posted a note to 
linux-btrace@vger.kernel.org at Alan Brunelle's suggestion and he also 
said he'd take a closer look at it himself.  On the other hand he 
pointed me to 'stap' and I was able to use it to get the details I was 
looking for - SystemTAP really rocks for this type of analysis!  As it 
turns out my assumption about driver blocksize was wrong (sorry about 
that).  It turns out that the size of requests from the driver to the 
disk is 160mb and so the I/O count was smaller than I had anticipated.
-mark

Re: [Fwd: Re: Accuracy of disk statistics IO counter]

[Li Guanglei]

Alan D. Brunelle ??:
> Here's a "success story" with SystemTAP: a developer here was having 
> trouble with getting iostat results to match internal tools he was 
> working on, and with the help of a _very_ simple SystemTAP script, he 
> was able to determine the nature of the problem, and get going very 
> quickly...
> 
> Alan
> PS. Here's the script I started him with...
> 
> global rqs
> global lun, id, channel, host_no
> 
> probe begin
> {
>        host_no = 0
>        channel = 0
>        id = 7
>        lun = 12
> }
> 
> probe module("*scsi_mod*").function("scsi_dispatch_cmd")
> {
>        if (1       != $cmd->sc_data_direction) next
>        if (lun     != $cmd->device->lun) next
>        if (id      != $cmd->device->id) next
>        if (channel != $cmd->device->channel) next
>        if (host_no != $cmd->device->host->host_no) next
> 
>        rqs[$cmd->request_bufflen / 1024]++
> }
> 
> probe end
> {
>        foreach (rec+ in rqs)
>              printf("%d %d\n", rec, rqs[rec])
>        exit()
> }
> 

An excellent example. But actually systemtap already has the same 
probe point of scsi_dispatch_cmd defined in 
/usr/share/systemtap/tapset/scsi.stp. With that tapsets, you can just:

global rqs
global _lun, _id, _channel, _host_no

probe begin
{
        _host_no = 0
        _channel = 0
        _id = 7
        _lun = 12
}

probe scsi.iodispatching
{
        if (1       != data_direction) next
        if (_lun     != lun) next
        if (_id      != dev_id) next
        if (_channel != channel) next
        if (_host_no != host_no) next

        rqs[req_bufflen / 1024]++
}

probe end
{
        foreach (rec+ in rqs)
              printf("%d %d\n", rec, rqs[rec])
        exit()
}

  And this is what BZ 2949 "need 'probe listing' command line option" 
aims for. We should make the stap users aware of current implemented 
stap probes instead of letting them start from scratch. I will add a 
section of "PROBE LIST" in stapprobes(5) man page and start to 
document the tapsets checked in. How about it?

- Guanglei

Re: [Fwd: Re: Accuracy of disk statistics IO counter]

[Frank Ch. Eigler]

Li Guanglei <guanglei@cn.ibm.com> writes:

> [...]
> probe scsi.iodispatching
> [...]
> I will add a section of "PROBE LIST" in stapprobes(5) man page and
> start to document the tapsets checked in. How about it?

While the stap "list" command may help, the newfangled tapsets like
scsi.stp independently need documentation of the same nature as other
tapsets already include in stapprobes(5)/stapfuncs(5).  There is no
"PROBE LIST" section needed - the whole man page is a probe list.

- FChE