Re: [Qemu-devel] Using the qemu tracepoints with SystemTap

[William Cohen <wcohen@redhat.com>]

On 09/14/2011 10:51 AM, Stefan Hajnoczi wrote:
> On Tue, Sep 13, 2011 at 8:38 PM, William Cohen <wcohen@redhat.com> wrote:
>> On 09/13/2011 12:10 PM, William Cohen wrote:
>>
>>> Should the qemu.kvm.cpu_in and qemu.kvm.cpu_out match up?  There are a lot more qemu.kvm.cpu_out than qemu.kvm.cpu_in count.
>>
>> I found that cpu_in and cpu_out refer to input and output instructions.  I wrote a little script tally up the input and output operations on each port to run on a qemu on fc15 machine.
>>
>> It generates output like the following:
>>
>>
>> cpu_in
>>  port    count
>> 0x01f7     3000
>> 0x03d5      120
>> 0xc000     2000
>> 0xc002     3000
>>
>> cpu_out
>>  port    count
>> 0x0080      480
>> 0x01f1     2000
>> 0x01f2     2000
>> 0x01f3     2000
>> 0x01f4     2000
>> 0x01f5     2000
>> 0x01f6     2000
>> 0x01f7     1000
>> 0x03d4      480
>> 0x03d5      120
>> 0x03f6     1000
>> 0xc000     3000
>> 0xc002     2000
>> 0xc004     1000
>> 0xc090        4
>>
>> Looks like lots of touching the ide device ports (0x01f0-0x01ff) and some vga controller (0x03d0-0x3df). This is kind of what would be expected when the machine is doing a fsck and selinux relabel on the guest virtual machines. Look like some pci device access (http://www.tech-pro.net/intro_pci.html) also.
> 
> I think the cpu_in/cpu_out tracing script can be useful in identifying
> performance problems, especially when obscure guest OSes experience
> poor performance due to weird ways of interacting with hardware.

Glad this example looks useful.

> Where are you putting your SystemTap scripts?  I suggest creating a
> public git repo and adding a link from the QEMU wiki tracing page:
> http://wiki.qemu.org/Features/Tracing

Definitely need a more lasting place to put the QEMU examples. SystemTap has an examples directory which try to put things like this into (http://sourceware.org/systemtap/examples/). These are run as part of the testsuite. These qemu examples are not ready for inclusion in the examples. I have set up https://github.com/wcohen/qemu_systemtap for the qemu examples.

> Perhaps we could even include them in a contrib/ or similar directory
> in qemu.git so that distros can ship them.  But before we worry about
> that we need a useful set of things that can be observed.

We definitely want to develop scripts that give people a better understanding what is going on. I am just now learning how kvm and qemu work, so the early scripts are just counting events to give people an idea events are frequent. In some cases these could point to issues where some assumptions about event frequency are incorrect. I would like to have scripts that examine the sequences of events and indicate when long latencies occur, but I don't yet have enough understanding of the how kvm and qemu work to implement those.

There are other userspace applications like python and java that have similar probes (http://sourceware.org/systemtap/wiki "Applications with built-in User-Space Markers"). Certainly making the systemtap scripts easily available would be good. Having the scripts with qemu is one way to do it. The other way would be to include in the systemtap examples. Advantage with including with qemu would be scripts would match the version of the qemu running and avoid problems with changes in the qemu probe points. Having the scripts in systemtap would allow regular testing of the scripts.

-Will