RE: [UPDATE] Linux Kernel Event Trace tool(LKET)

RE: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Stone, Joshua I]

On Sunday, May 28, 2006 11:27 PM, Li Guanglei wrote:
> Li Guanglei ??:
>> 
>> 
>> I don't have a x86_64 at hand. I tried
>> __switch_to(scheduler.ctxswitch in src/tapset/schedule.stp) and it
>> will cause kernel panic. I will take a further look at this and will
>> open a bug for this if needed. 
> 
> I tested again with 2.6.16.16 & 2.6.16-1.2211_FC6 on x86, _switch_to
> won't cause kernel panic any more. So I think this bug has been fixed
> in latest kernel.

Probes on __switch_to are disallowed on x86_64 -- see bug #2068.
http://sources.redhat.com/bugzilla/show_bug.cgi?id=2068


Josh

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

Stone, Joshua I ??:
> 
> Please note that b2a is generating several warnings on x86_64:
> [...]
> Josh

Thanks. Changes have been submitted. I tested on ppc64 with/without 
-m64 flag and it doesn't give any warning again.

- Guanglei

RE: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Stone, Joshua I]

On Friday, May 26, 2006 7:35 PM, Guang Lei Li wrote:
>> BTW, I checked in binary2ASCII converter yesterday. I once let
>> SystemTap build b2a converter by default, but after some considers I
>> deleted it from SystemTap's makefile system. The reason is that I
>> used a balanced binary search in b2a which comes from GLIB2. So now
>> b2a depends on glib2 & glib2-devel. I think maybe it's not good to
>> let SystemTap depends on a new package but I am not quite sure since
>> the functionalities provided by glib2 is really useful. Any ideas?
>> 
> 
> Added conditional build of lket-b2a. If glib2-devel is not installed,
> just skips and gives a warning.
> I will add man page of lket-b2a

Please note that b2a is generating several warnings on x86_64:

Making all in runtime/lket/b2a
make[2]: Entering directory
`/home/jistone/systemtap/src/runtime/lket/b2a'
if gcc -DHAVE_CONFIG_H -I. -I. -I../../..    -O2 -D_GNU_SOURCE -Wall
`pkg-config --cflags glib-2.0` -g -O2 -MT lket_b2a-lket_b2a.o -MD -MP
-MF ".deps/lket_b2a-lket_b2a.Tpo" -c -o lket_b2a-lket_b2a.o `test -f
'lket_b2a.c' || echo './'`lket_b2a.c; \
then mv -f ".deps/lket_b2a-lket_b2a.Tpo" ".deps/lket_b2a-lket_b2a.Po";
else rm -f ".deps/lket_b2a-lket_b2a.Tpo"; exit 1; fi
lket_b2a.c: In function 'register_appname':
lket_b2a.c:229: warning: cast to pointer from integer of different size
lket_b2a.c: In function 'compareFunc':
lket_b2a.c:235: warning: cast from pointer to integer of different size
lket_b2a.c:235: warning: cast from pointer to integer of different size
lket_b2a.c:236: warning: cast from pointer to integer of different size
lket_b2a.c:236: warning: cast from pointer to integer of different size
lket_b2a.c: In function 'print_pkt_header':
lket_b2a.c:405: warning: cast to pointer from integer of different size
lket_b2a.c: In function 'b2a_vsnprintf':
lket_b2a.c:526: warning: format '%lld' expects type 'long long int', but
argument 3 has type 'long int'
gcc  -g -O2   -o lket-b2a `pkg-config --libs glib-2.0`
lket_b2a-lket_b2a.o
make[2]: Leaving directory
`/home/jistone/systemtap/src/runtime/lket/b2a'


Josh

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

Li Guanglei ??:
> 
> 
> I don't have a x86_64 at hand. I tried __switch_to(scheduler.ctxswitch 
> in src/tapset/schedule.stp) and it will cause kernel panic. I will take 
> a further look at this and will open a bug for this if needed.

I tested again with 2.6.16.16 & 2.6.16-1.2211_FC6 on x86, _switch_to 
won't cause kernel panic any more. So I think this bug has been fixed 
in latest kernel.

- Li Guanglei

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Guang Lei Li]

> 
> Hi folks,
> 
> BTW, I checked in binary2ASCII converter yesterday. I once let 
> SystemTap build b2a converter by default, but after some considers I 
> deleted it from SystemTap's makefile system. The reason is that I used 
> a balanced binary search in b2a which comes from GLIB2. So now b2a 
> depends on glib2 & glib2-devel. I think maybe it's not good to let 
> SystemTap depends on a new package but I am not quite sure since the 
> functionalities provided by glib2 is really useful. Any ideas?
> 

Added conditional build of lket-b2a. If glib2-devel is not installed, just 
skips and gives a warning. 
I will add man page of lket-b2a

Best Regards,

Li Guanglei

- Linux Performance, China Systems & Technology Lab
Email: liguangl@cn.ibm.com, IMAP: guanglei@cn.ibm.com

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

Frank Ch. Eigler ??:
> Li Guanglei <guanglei@cn.ibm.com> writes:
> 
>> [...]  I've checked into cvs the draft man page for LKET. Now you
>> can type "man lket" to get an outline about LKET, including event
>> hooks intro, examples, trace data format etc.
> 
> OK.  It would be good to add some cross-references from stapprobes(5),
> and of course some test cases.

I checked in some testcases into testsuite/buildok/. These testcases 
will test the generic tapsets of LKET including the probe alias 
definitions and it will references all the variables defined in each 
probe alias.

I will try to check in more testcases.

> 
>> But I think it will be better if I can add LKET's man page onto
>> SystemTap's website, i.e the "Man Pages" section in:
>> http://sourceware.org/systemtap/documentation.html
>> [...]
> 
> The web pages are also in CVS, in the "/htdocs" top level directory
> right beside the familiar "/src" and "/tests".
 >
 > - FChE

checked in. Add a link "Linux Kernel Event Trace" in the "Man Pages" 
section of documentation link. I am not quite sure if the name of the 
link is appropriate.

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

William Cohen ??:
> 
> I read through the LKET manpage this morning and tried out LKET with 
> providing ASCII output on a 32-bit i686 FC5 machine and a 64-bit x86_64 
> FC5 machine. 
 > [...]
> 
> It looks like there are some portability issues that need to be worked 
> out in the code. Things seemed to work better on the 64-bit platform 
> than on the 32-bit platform.
Today I used my thinkpad install with FC5/2.6.16.13, and found the 
same problems as you mentioned.

This is because that _stp_vsnprintf use va_arg() to get arguments and 
some arguments have wrong type in 32-bit applications. I fixed this in 
all current LKET tapsets and checked them into CVS.

I did a testing for all event hooks in ppc64/2.6.16.5 & i386/2.6.16.13 
for both ascii tracing(-D ASCII_TRACE) and binary tracing(with -bM and 
later use runtime/lket/b2a/lket-b2a to convert into ascii format), and 
most of the problems have gone, except the _switch_to & idle_balance 
which will be explained below.

> 
> The tskdispatch didn't work on either machine. Couldn't find 
> kernel.function("__switch_to") on 64-bit and couldn't find 
> kernel.inline("idle_balance") on 32-bit

I don't have a x86_64 at hand. I tried __switch_to(scheduler.ctxswitch 
in src/tapset/schedule.stp) and it will cause kernel panic. I will 
take a further look at this and will open a bug for this if needed.

idle_balance is only available on SMP boxes. On single-cpu box, it 
will be commented out. I will try to find other replacements of 
idle_balance.

> 
> Neither machine has scsi interface. When trying out the flagged errors 
> because no scsi modules.

I tested on my thinkpad since it has scsi device(SATA) and it works well.

> 
> $ stap -D ASCII_TRACE -e "probe addevent.pagefault {}" -I 
> /usr/local/share/systemtap/tapset/LKET/
> 
> 
> The data on the 32-bit machine looked bogus (see below). lots of 
> negative references. Also would it make more sens to have the address 
> printed in hexidecimal for the ascii trace?
> 
> 6|1|0|1148483972|626999|1848|1|1848|35705849838043136|1
> 6|1|0|1148483972|627050|1848|1|1848|-4615772671219597312|1
> 6|1|0|1148483974|465231|1324|1|1324|-5191788353242005504|1
> 

For ascii tracing, I printed the pointer in %d, so this is why you see 
negtive values.

> 
> $ stap -D ASCII_TRACE  -e "probe addevent.netdev.* {}" -I 
> /usr/local/share/systemtap/tapset/LKET/
> 
> The netdev worked on 64-bit, but it failed compilation on 32-bit:
> 
> cc1: warnings being treated as errors
> /tmp/stapBs84Hc/stap_15261.c: In function 'function_log_netdev_extra':
> /tmp/stapBs84Hc/stap_15261.c:434: warning: cast to pointer from integer 
> of different size

I've fixed this in CVS

> 
> -Will
> 

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

William Cohen ??:
> Li Guanglei wrote:
>> Li Guanglei ??:
>>
> 
> I read through the LKET  [ ... ]
> 
> -Will
> 
Thanks for your tryout of LKET.

In fact, we usually use ppc64/RHEL4 u3 for development & testing. And 
  although I did some tests in i686/RHEL4 U3 too, I am afraid it won't 
work as well as in pp64, especially in Fedora Core.

But I think this is one reason why we want LKET merged into SystemTap 
so that more people can improve it together.

Another point is that during our internal testing of LKET, we seldom 
use the -D ASCII_TRACE. It exists there only for convenience, but it 
is not so stable. So I suggest you should use "stap -bM ..." to 
generate per-cpu files and then use the tool in 
src/runtime/lket/b2a/b2a to convert into ascii format.


Hi folks,

BTW, I checked in binary2ASCII converter yesterday. I once let 
SystemTap build b2a converter by default, but after some considers I 
deleted it from SystemTap's makefile system. The reason is that I used 
a balanced binary search in b2a which comes from GLIB2. So now b2a 
depends on glib2 & glib2-devel. I think maybe it's not good to let 
SystemTap depends on a new package but I am not quite sure since the 
functionalities provided by glib2 is really useful. Any ideas?

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Frank Ch. Eigler]

Li Guanglei <guanglei@cn.ibm.com> writes:

> [...]  I've checked into cvs the draft man page for LKET. Now you
> can type "man lket" to get an outline about LKET, including event
> hooks intro, examples, trace data format etc.

OK.  It would be good to add some cross-references from stapprobes(5),
and of course some test cases.

> But I think it will be better if I can add LKET's man page onto
> SystemTap's website, i.e the "Man Pages" section in:
> http://sourceware.org/systemtap/documentation.html
> [...]

The web pages are also in CVS, in the "/htdocs" top level directory
right beside the familiar "/src" and "/tests".

- FChE

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[William Cohen]

Li Guanglei wrote:
> Li Guanglei ??:
> 
>> Li Guanglei ??:
>>
>>> [...]
>>> So I'd like to break LKET's tapsets into the following two layers:
>>>
>>> <1> Generic Tapsets
>>> [...]
>>
>> I've checked them into tapset directory(together with some testcases),
>> They are:
>> ioscheduler.stp, memory.stp, networking.stp, scsi.stp,
>>
>> tskschedule.stp & process.stp from LKET were merged into scheduler.stp 
>> and process.stp respectively.
>>
>> In a summary, the following probe alias are defined:
>> [...]
>>
>>>
>>> <2> Tapsets Specific to Tracing
>>> [...]
>>> --Guanglei
>>>
>> All these traceing specific tapsets are checked into tapset/LKET 
>> direcotry
>>
>> I am writing the man pages for lket to make it easier to use.
>>
> 
> I've checked into cvs the draft man page for LKET. Now you can type "man 
> lket" to get an outline about LKET, including event hooks intro, 
> examples, trace data format etc.
> 
> But I think it will be better if I can add LKET's man page onto 
> SystemTap's website, i.e the "Man Pages" section in:
> http://sourceware.org/systemtap/documentation.html
> Because SystemTap's website is the first place people will go.
> 
> But I don't know how to add entries into the webpage. I am afraid I 
> don't have the permission. Could somebody help me on this?

I read through the LKET manpage this morning and tried out LKET with 
providing ASCII output on a 32-bit i686 FC5 machine and a 64-bit x86_64 
FC5 machine. The kernels are listed below.

Linux montague.devel.redhat.com 2.6.16-1.2122_FC5 #1 Sun May 21 15:01:01 
EDT 2006 i686 i686 i386 GNU/Linux

Linux dhcp59-221.rdu.redhat.com 2.6.16-1.2208_FC6 #1 SMP Sat May 20 
21:48:05 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux

It looks like there are some portability issues that need to be worked 
out in the code. Things seemed to work better on the 64-bit platform 
than on the 32-bit platform.



$ stap -D ASCII_TRACE -e "probe addevent.syscall.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

syscall seemed to generate reasonable data on 64-bit machine. However, 
on the 32-bit machine I got traces that looked like the following:

1|1|0|1148483176|803233|3311|3308|3311|<NULL>
1|2|0|1148483176|803245|3311|3308|3311|<NULL>
1|1|0|1148483176|803259|3311|3308|3311|<NULL>
1|2|0|1148483176|803303|3311|3308|3311|<NULL>


$ stap -D ASCII_TRACE -e "probe addevent.process.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

process trace seemed to generate reasonable information on both 32- and 
64-bit machines initially. Then later the 32-bit seemed to miss 
information like the following:

2|1|1148483294|472220|14862|14772|14862|0|13853|emacs-x
2|1|1148483294|472225|14862|14772|14862|0|14772|stap
2|3|0|1148483297|562834|1848|1|1848|0
2|3|0|1148483312|563887|1848|1|1848|0

$ stap -D ASCII_TRACE -e "probe addevent.ioscheduler.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

On 32-bit patform got bogus information:
3|1|0|1148483567|535253|0|0|0|<NULL>|-1070280432|22
3|3|0|1148483567|536381|3311|3308|3311|<NULL>|-1070280432|22
3|3|0|1148483567|537053|2091|2068|2091|<NULL>|-1070280432|22

The 64-bit information looked better however, sometimes saw traces like 
the following:

3|1|1148483662|28320|11716|11530|11716|0|cfq|-1|-1


stap -D ASCII_TRACE -e "probe addevent.tskdispatch.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

The tskdispatch didn't work on either machine. Couldn't find 
kernel.function("__switch_to") on 64-bit and couldn't find 
kernel.inline("idle_balance") on 32-bit


Neither machine has scsi interface. When trying out the flagged errors 
because no scsi modules.

$ stap -D ASCII_TRACE -e "probe addevent.pagefault {}" -I 
/usr/local/share/systemtap/tapset/LKET/


The data on the 32-bit machine looked bogus (see below). lots of 
negative references. Also would it make more sens to have the address 
printed in hexidecimal for the ascii trace?

6|1|0|1148483972|626999|1848|1|1848|35705849838043136|1
6|1|0|1148483972|627050|1848|1|1848|-4615772671219597312|1
6|1|0|1148483974|465231|1324|1|1324|-5191788353242005504|1


$ stap -D ASCII_TRACE  -e "probe addevent.netdev.* {}" -I 
/usr/local/share/systemtap/tapset/LKET/

The netdev worked on 64-bit, but it failed compilation on 32-bit:

cc1: warnings being treated as errors
/tmp/stapBs84Hc/stap_15261.c: In function 'function_log_netdev_extra':
/tmp/stapBs84Hc/stap_15261.c:434: warning: cast to pointer from integer 
of different size



-Will

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

Li Guanglei ??:
> Li Guanglei ??:
> 
>> [...]
>> So I'd like to break LKET's tapsets into the following two layers:
>>
>> <1> Generic Tapsets
>> [...]
> I've checked them into tapset directory(together with some testcases),
> They are:
> ioscheduler.stp, memory.stp, networking.stp, scsi.stp,
> 
> tskschedule.stp & process.stp from LKET were merged into scheduler.stp 
> and process.stp respectively.
> 
> In a summary, the following probe alias are defined:
> [...]
> 
>>
>> <2> Tapsets Specific to Tracing
>> [...]
>> --Guanglei
>>
> All these traceing specific tapsets are checked into tapset/LKET direcotry
> 
> I am writing the man pages for lket to make it easier to use.
>

I've checked into cvs the draft man page for LKET. Now you can type 
"man lket" to get an outline about LKET, including event hooks intro, 
examples, trace data format etc.

But I think it will be better if I can add LKET's man page onto 
SystemTap's website, i.e the "Man Pages" section in:
http://sourceware.org/systemtap/documentation.html
Because SystemTap's website is the first place people will go.

But I don't know how to add entries into the webpage. I am afraid I 
don't have the permission. Could somebody help me on this?

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

Li Guanglei ??:

> [...]
> So I'd like to break LKET's tapsets into the following two layers:
> 
> <1> Generic Tapsets
> The first layer are generic tapsets which could not only be used for 
> tracing, but also for general purpose diagnosis. It will include the 
> probe points defined in current LKET's tapsets and some stap script 
> statements of getting the interesting data related to that event.
> 
> So this will need some .stp files to be added into 
> /usr/local/share/SystemTap/tapsets. These files are scsi.stp, 
> ioscheduler.stp, tskdispatching.stp, networking.stp etc. Take an example 
> of scsi.stp, it will look like:
> 
> ...
> probe scsi.ioentry
>         = module("*").function("scsi_prep_fn@drivers/scsi/scsi_lib.c")
> {
>     disk_major = $rq->rq_disk->major
>     disk_minor = $rq->rq_disk->>first_minor
>     dev_state =  $q->queuedata->sdev_state
>     // maybe a argstr?
> }
> ...
> 
> These stp scripts not only address where inside the kernel to be 
> instrumented, but also get out the interesting data related to that 
> probed points. They look in the same style as current syscall.stp 
> shipped with SystemTap.
I've checked them into tapset directory(together with some testcases),
They are:
ioscheduler.stp, memory.stp, networking.stp, scsi.stp,

tskschedule.stp & process.stp from LKET were merged into scheduler.stp 
and process.stp respectively.

In a summary, the following probe alias are defined:

   ioscheduler.stp:
      ioscheduler.elv_next_request
      ioscheduler.elv_add_request
      ioscheduler.elv_completed_request

   memory.stp
      pagefault

   networking.stp
      netdev.receive
      netdev.transmit

   scsi.stp
      scsi.ioentry
      scsi.iodispatching
      scsi.iodone
      scsi.iocompleted

   scheduler.stp
      scheduler.ctxswitch

  You can refer to the .stp files for more info

> 
> <2> Tapsets Specific to Tracing
> The second part will be tracing specific, which may be located in 
> /usr/local/share/systemtap/tapsets/LKET/. They include those embedded c 
> functions in current LKET. Take an example of scsi.stp:
> 
> probe addevent.scsi.ioentry
>     = probe scsi.ioentry
> {
>     log_scsi_ioentry(HOOKID_SCSI_IOENTRY, $q, $req, backtrace)
> }
> 
> Thanks to the optimizations of translator, the statements of "disk_major 
> = $rq->rq_disk->major" will be compiled away and the generated c file 
> will look exactly like the original LKET.
> 
> And "stap -bM -I /usr/local/share/SystemTap/tapsets/LKET sample.stp" 
> will work the same as current LKET.
> 
> --Guanglei
> 
All these traceing specific tapsets are checked into tapset/LKET direcotry

I am writing the man pages for lket to make it easier to use.

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=GB2312, Size: 1753 bytes --]

Nathan DeBardeleben дµÀ:
> I was trying to take a look at this. Now, I don't know much about
> SystemTap (yet) but I cannot seem to run this - while I can run the
> scripts from the tutorial, demo, and others that I've written myself.
> 
> Using your README there's an example of how to run your scripts and when
> I do that, I get:
>> [ndebard@fedora64 LKET-2006-05-11]$ stap -e "probe addevent.* {}" -I
>> tapsets -b -M
>> semantic error: probe point mismatch at position 2 (alternatives:)
>> while: resolving probe point addevent.syscall.entry
>> semantic error: probe point mismatch at position 2 (alternatives:)
>> while: resolving probe point addevent.syscall.return
>> semantic error: no match for probe point
>> while: resolving probe point addevent.syscall
>> semantic error: probe point mismatch at position 2 (alternatives: cpuidle)
>> while: resolving probe point addevent.tskdispatch.ctxswitch
>> Pass 2: analysis failed. Try again with more '-v' (verbose) options.
>> [ndebard@fedora64 LKET-2006-05-11]$ 
> I get the same message on the other examples you gave in there.
> 
> Can you tell me what I'm doing wrong, please?

I guess you are running LKET on the platform other than ppc64/i686,
right? So if you are using other platform, say, x86_64, LKET will
complain of not being able to find the alias definitions.

A quick fix to work for x86_64 is:
cd LKET-2006-05-11/tapsets
cp -rf i686 x86_64

Then it should be ok.

In fact, most of our development & testing is doing on ppc64 platform
and RHEL4. So LKET needs slight changes in order to work on all
platforms. This is one of the reason why I want to merge LKET into
SystemTap's tapsets library so that more people could contribute to it.

Let me know if you still have problems

RE: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Stone, Joshua I]

On Sunday, May 14, 2006 11:35 PM, Li Guanglei wrote:
> So I'd like to break LKET's tapsets into the following two layers:
> 
> <1> Generic Tapsets
> [...]
> <2> Tapsets Specific to Tracing

Great, this is exactly the model I had in mind.


Josh

Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

Stone, Joshua I ??:
> Hi Li,
> 
> Currently I am working on a few generic tapsets, and there appears to be
> some overlap with what you've done.  For example, earlier this week I
> checked in a process tapset.  This should be able to give you the fork
> and exec events you're using in LKET.
> 
> Thang and I are also working on IO, scheduler, networking, and VM
> tapsets, but we don't have anything ready to check in yet.  I will
> definitely take a look at what you've provided though, and leverage that
> which overlaps.
> 
> I think it would be great to have LKET as a tapset, but I would like to
> see a more layered approach.  What you've given is very specific to
> tracing, but the events that you probe have a more general audience as
> well.  A broader approach would be to implement the events as general
> tapsets, and then build LKET as a layer on top of the base tapsets.
> 
> The syscalls are a good example of this -- in LKET you've captured
> syscalls using function wildcards.  However, we have the syscall tapset
> that has already gone through pains to define all of the actual system
> calls available.  If we run into syscall issues in the future, it would
> be much nicer to need only fix the syscall tapset, and let LKET benefit
> from that.  So instead of probing 'kernel.function("sys_*")', you should
> use 'syscall.*'.  And if there are reasons why this doesn't work well
> for you, let's address that rather than duplicating efforts.
> 
> 
> Josh

So I'd like to break LKET's tapsets into the following two layers:

<1> Generic Tapsets
The first layer are generic tapsets which could not only be used for 
tracing, but also for general purpose diagnosis. It will include the 
probe points defined in current LKET's tapsets and some stap script 
statements of getting the interesting data related to that event.

So this will need some .stp files to be added into 
/usr/local/share/SystemTap/tapsets. These files are scsi.stp, 
ioscheduler.stp, tskdispatching.stp, networking.stp etc. Take an 
example of scsi.stp, it will look like:

...
probe scsi.ioentry
         = module("*").function("scsi_prep_fn@drivers/scsi/scsi_lib.c")
{
     disk_major = $rq->rq_disk->major
     disk_minor = $rq->rq_disk->>first_minor
     dev_state =  $q->queuedata->sdev_state
     // maybe a argstr?
}
...

These stp scripts not only address where inside the kernel to be 
instrumented, but also get out the interesting data related to that 
probed points. They look in the same style as current syscall.stp 
shipped with SystemTap.

<2> Tapsets Specific to Tracing
The second part will be tracing specific, which may be located in 
/usr/local/share/systemtap/tapsets/LKET/. They include those embedded 
c functions in current LKET. Take an example of scsi.stp:

probe addevent.scsi.ioentry
	= probe scsi.ioentry
{
	log_scsi_ioentry(HOOKID_SCSI_IOENTRY, $q, $req, backtrace)
}

Thanks to the optimizations of translator, the statements of 
"disk_major = $rq->rq_disk->major" will be compiled away and the 
generated c file will look exactly like the original LKET.

And "stap -bM -I /usr/local/share/SystemTap/tapsets/LKET sample.stp" 
will work the same as current LKET.

I'd like to check into CVS the first part of generic tapsets as a 
start point of getting more event hooks into SystemTap. And then 
checking the tracing specific tapsets into SystemTap's tapset library.

Any comments about this?

--Guanglei

RE: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Stone, Joshua I]

Hi Li,

Currently I am working on a few generic tapsets, and there appears to be
some overlap with what you've done.  For example, earlier this week I
checked in a process tapset.  This should be able to give you the fork
and exec events you're using in LKET.

Thang and I are also working on IO, scheduler, networking, and VM
tapsets, but we don't have anything ready to check in yet.  I will
definitely take a look at what you've provided though, and leverage that
which overlaps.

I think it would be great to have LKET as a tapset, but I would like to
see a more layered approach.  What you've given is very specific to
tracing, but the events that you probe have a more general audience as
well.  A broader approach would be to implement the events as general
tapsets, and then build LKET as a layer on top of the base tapsets.

The syscalls are a good example of this -- in LKET you've captured
syscalls using function wildcards.  However, we have the syscall tapset
that has already gone through pains to define all of the actual system
calls available.  If we run into syscall issues in the future, it would
be much nicer to need only fix the syscall tapset, and let LKET benefit
from that.  So instead of probing 'kernel.function("sys_*")', you should
use 'syscall.*'.  And if there are reasons why this doesn't work well
for you, let's address that rather than duplicating efforts.


Josh


On Thursday, May 11, 2006 1:25 AM, Li Guanglei wrote:
> Hi folks,
> 
>   This is the updated version of LKET. Compared with the previous
> version, it includes the following enhancements:
>   1. more event hooks
>   2. binary tracing for better performance
>   3. other performance enhancements & bugfixes
>   4. more backtrace support
>   5. a b2a converter to merge per-cpu files and convert binary format
> into ASCII format
> 
>   Currently LKET is actually a collection of SystemTap tapsets which
> are used to trace various events inside the kernel and generate the
> tracing data. This tool aims to provide an easy way to trace various
> events inside the kernel for the purpose of performance analysis.
> 
>   Since LKET is an extension to the tapsets library available on
> SystemTap, I hope LKET could go into SystemTap's tapsets library. So
> it will make SystemTap more easier to use even for those who don't
> know much about Linux Kernel internal, and thus benefit more people.
> Any comments about this proposal?
> 
>  Here's the events being implemented by now:
> 
>  System Call:
>     entry of syscall
>     return of syscall
>  Process Creation:
>     fork
>     execve
>  IO Scheduler:
>     a request is retrieved from request queue
>     a request is added to the request queue
>     a request is completed
>  Task Dispatching(scheduler):
>     context switch
>     cpu goes idle
>  Scsi Activities
>     mid-layer prepared a IO request
>     Dispatch a command to the LLD
>     I/O is done by LLD
>     mid-layer processes the completed IO
>  Pagefault
>     a pagefault is taking place
>  Networking Device
>     network device receives a packet
>     a packet will be sent out by network device
> 
>  During the LKET work, we also did some testing for overhead &
> accuracy. I planned to disclose the overhead testing result of LKET in
> the next couple of days.
> 
>  I attached the tgz package of LKET since the # of files is a little
> big. 
> 
>  Suggestions, Comments, Flames are welcome. And thanks in advance
> for your help.

[UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei]

[-- Attachment #1: Type: text/plain, Size: 1890 bytes --]

Hi folks,

  This is the updated version of LKET. Compared with the previous
version, it includes the following enhancements:
  1. more event hooks
  2. binary tracing for better performance
  3. other performance enhancements & bugfixes
  4. more backtrace support
  5. a b2a converter to merge per-cpu files and convert binary format
into ASCII format

  Currently LKET is actually a collection of SystemTap tapsets which
are used to trace various events inside the kernel and generate the
tracing data. This tool aims to provide an easy way to trace various
events inside the kernel for the purpose of performance analysis.

  Since LKET is an extension to the tapsets library available on
SystemTap, I hope LKET could go into SystemTap's tapsets library. So
it will make SystemTap more easier to use even for those who don't
know much about Linux Kernel internal, and thus benefit more people.
Any comments about this proposal?

 Here's the events being implemented by now:

 System Call:
    entry of syscall
    return of syscall
 Process Creation:
    fork
    execve
 IO Scheduler:
    a request is retrieved from request queue
    a request is added to the request queue
    a request is completed
 Task Dispatching(scheduler):
    context switch
    cpu goes idle
 Scsi Activities
    mid-layer prepared a IO request
    Dispatch a command to the LLD
    I/O is done by LLD
    mid-layer processes the completed IO
 Pagefault
    a pagefault is taking place
 Networking Device
    network device receives a packet
    a packet will be sent out by network device

 During the LKET work, we also did some testing for overhead &
accuracy. I planned to disclose the overhead testing result of LKET in
the next couple of days.

 I attached the tgz package of LKET since the # of files is a little big.

 Suggestions, Comments, Flames are welcome. And thanks in advance
for your help.


[-- Attachment #2: LKET-2006-05-11.tgz --]
[-- Type: application/x-compressed, Size: 14384 bytes --]