Systemtap FAQ

Systemtap FAQ

[joaoandreferro]

Hello all,
   
  I've just installed Systemtap, and to start I have a couple of doubts that I
would like to share with you:
   
  1 - Firstly, what I'm trying to achieve is some kind of interrupt handler:
I would like to momentarily interrupt the execution of the OS, and then be
able to run some code of mine (to start, inject a single bit-flip into a
CPU register), and then resume the execution of the OS, with the
possibility of having information on the pre-interruption context. Is this
somehow possible with Systemtap?
   
  2 - I'm using CentOs 6.5 32 bit (kernel version 2.6.32-431.23.3.el6.i686).
Will Systemtap work here? I think this information may be relevant, since
this isn't the last version of the distro, and some other tools I've
already tried to use in the past didn't worked because of my (old) kernel
version.
   
  Thanks and all the best,
  João

Re: Systemtap FAQ

[William Cohen]

On 08/22/2014 04:51 PM, joaoandreferro@sapo.pt wrote:
> Hello all,
>   
>  I've just installed Systemtap, and to start I have a couple of doubts that I
> would like to share with you:
>   
>  1 - Firstly, what I'm trying to achieve is some kind of interrupt handler:
> I would like to momentarily interrupt the execution of the OS, and then be
> able to run some code of mine (to start, inject a single bit-flip into a
> CPU register), and then resume the execution of the OS, with the
> possibility of having information on the pre-interruption context. Is this
> somehow possible with Systemtap?

Hi,

It sound like you are trying to do fault injection with systemtap.  SystemTap has been used successfully for fault injection on the Linux kernel's scsi layer:

http://lwn.net/Articles/289932/

SystemTap's guru mode (-g) has the ability to modify target variable value. For the kprobes, uprobe, and timer based probes you probably could change the values in the pt_regs struct and effect a bit-flip.

You could use the associative arrays in systemtap to store information about where (backtrace) and when these bit-flips are done.  Look at the the tapset library for information on various functions to get timeofday and backtraces https://sourceware.org/systemtap/tapsets/ .


>   
>  2 - I'm using CentOs 6.5 32 bit (kernel version 2.6.32-431.23.3.el6.i686).
> Will Systemtap work here? I think this information may be relevant, since
> this isn't the last version of the distro, and some other tools I've
> already tried to use in the past didn't worked because of my (old) kernel
> version.

SystemTap should work on CentOs.  SystemTap does need other rpms such as matching kernel-devel and kernel-debuginfo.  You might need to manually download and install those because yum typically tries to get the most recent versions of packages.  The stap-prep command should list out what packages you need to get sysetmtap working on the system.  The following wiki page provides some additional suggestions on setting up systemtap on CentOs:

https://sourceware.org/systemtap/wiki/SystemTapOnCentOS

-Will

Re: Systemtap FAQ

[joaoandreferro]

Hello William,

Thanks for your reply. In fact, what I'm trying to do is indeed fault  
injection. I already discovered that link and heard about guru mode as  
well, but I haven't got the time to study both subjects on detail.  
Your last paragraph sounds very promising! Till now I only was able to  
do a small kernel module to bit-flip a control register, but I'll  
start looking into SystemTap, so I can enable context handling.

Concerning my second doubt, I knew it should work with CentOS, I only  
feared that it would cause me problems because of being an version of  
the OS, but it seems everything is working fine.

Regards,
João

Citando William Cohen <wcohen@redhat.com>:

> On 08/22/2014 04:51 PM, joaoandreferro@sapo.pt wrote:
>> Hello all,
>>
>>  I've just installed Systemtap, and to start I have a couple of  
>> doubts that I
>> would like to share with you:
>>
>>  1 - Firstly, what I'm trying to achieve is some kind of interrupt handler:
>> I would like to momentarily interrupt the execution of the OS, and then be
>> able to run some code of mine (to start, inject a single bit-flip into a
>> CPU register), and then resume the execution of the OS, with the
>> possibility of having information on the pre-interruption context. Is this
>> somehow possible with Systemtap?
>
> Hi,
>
> It sound like you are trying to do fault injection with systemtap.   
> SystemTap has been used successfully for fault injection on the  
> Linux kernel's scsi layer:
>
> http://lwn.net/Articles/289932/
>
> SystemTap's guru mode (-g) has the ability to modify target variable  
> value. For the kprobes, uprobe, and timer based probes you probably  
> could change the values in the pt_regs struct and effect a bit-flip.
>
> You could use the associative arrays in systemtap to store  
> information about where (backtrace) and when these bit-flips are  
> done.  Look at the the tapset library for information on various  
> functions to get timeofday and backtraces  
> https://sourceware.org/systemtap/tapsets/ .
>
>
>>
>>  2 - I'm using CentOs 6.5 32 bit (kernel version 2.6.32-431.23.3.el6.i686).
>> Will Systemtap work here? I think this information may be relevant, since
>> this isn't the last version of the distro, and some other tools I've
>> already tried to use in the past didn't worked because of my (old) kernel
>> version.
>
> SystemTap should work on CentOs.  SystemTap does need other rpms  
> such as matching kernel-devel and kernel-debuginfo.  You might need  
> to manually download and install those because yum typically tries  
> to get the most recent versions of packages.  The stap-prep command  
> should list out what packages you need to get sysetmtap working on  
> the system.  The following wiki page provides some additional  
> suggestions on setting up systemtap on CentOs:
>
> https://sourceware.org/systemtap/wiki/SystemTapOnCentOS
>
> -Will

Re: Systemtap FAQ

[William Cohen]

On 08/28/2014 11:58 AM, joaoandreferro@sapo.pt wrote:
> Hello William,
> 
> Thanks for your reply. In fact, what I'm trying to do is indeed fault injection. I already discovered that link and heard about guru mode as well, but I haven't got the time to study both subjects on detail. Your last paragraph sounds very promising! Till now I only was able to do a small kernel module to bit-flip a control register, but I'll start looking into SystemTap, so I can enable context handling.
> 
> Concerning my second doubt, I knew it should work with CentOS, I only feared that it would cause me problems because of being an version of the OS, but it seems everything is working fine.
> 
> Regards,
> João

The common issue with using systemtap is that it is very particular about the kernel debug info and kernel devel files matching the instrumented kernel exactly.  That means that rpms need to come from the same build; being built at different times from the same source rpm is not sufficient.

-Will


> 
> Citando William Cohen <wcohen@redhat.com>:
> 
>> On 08/22/2014 04:51 PM, joaoandreferro@sapo.pt wrote:
>>> Hello all,
>>>
>>>  I've just installed Systemtap, and to start I have a couple of doubts that I
>>> would like to share with you:
>>>
>>>  1 - Firstly, what I'm trying to achieve is some kind of interrupt handler:
>>> I would like to momentarily interrupt the execution of the OS, and then be
>>> able to run some code of mine (to start, inject a single bit-flip into a
>>> CPU register), and then resume the execution of the OS, with the
>>> possibility of having information on the pre-interruption context. Is this
>>> somehow possible with Systemtap?
>>
>> Hi,
>>
>> It sound like you are trying to do fault injection with systemtap.  SystemTap has been used successfully for fault injection on the Linux kernel's scsi layer:
>>
>> http://lwn.net/Articles/289932/
>>
>> SystemTap's guru mode (-g) has the ability to modify target variable value. For the kprobes, uprobe, and timer based probes you probably could change the values in the pt_regs struct and effect a bit-flip.
>>
>> You could use the associative arrays in systemtap to store information about where (backtrace) and when these bit-flips are done.  Look at the the tapset library for information on various functions to get timeofday and backtraces https://sourceware.org/systemtap/tapsets/ .
>>
>>
>>>
>>>  2 - I'm using CentOs 6.5 32 bit (kernel version 2.6.32-431.23.3.el6.i686).
>>> Will Systemtap work here? I think this information may be relevant, since
>>> this isn't the last version of the distro, and some other tools I've
>>> already tried to use in the past didn't worked because of my (old) kernel
>>> version.
>>
>> SystemTap should work on CentOs.  SystemTap does need other rpms such as matching kernel-devel and kernel-debuginfo.  You might need to manually download and install those because yum typically tries to get the most recent versions of packages.  The stap-prep command should list out what packages you need to get sysetmtap working on the system.  The following wiki page provides some additional suggestions on setting up systemtap on CentOs:
>>
>> https://sourceware.org/systemtap/wiki/SystemTapOnCentOS
>>
>> -Will
> 
> 

Re: Systemtap FAQ

[joaoandreferro]

Hello again, William,

Humm, I got both .rpm packages from  
http://debuginfo.centos.org/6/i386/ (matching exactly the same kernel  
version as the target kernel, of course, in this case  
2.6.32-431.23.3.el6.i686)

Regards
João

Citando William Cohen <wcohen@redhat.com>:

> On 08/28/2014 11:58 AM, joaoandreferro@sapo.pt wrote:
>> Hello William,
>>
>> Thanks for your reply. In fact, what I'm trying to do is indeed  
>> fault injection. I already discovered that link and heard about  
>> guru mode as well, but I haven't got the time to study both  
>> subjects on detail. Your last paragraph sounds very promising! Till  
>> now I only was able to do a small kernel module to bit-flip a  
>> control register, but I'll start looking into SystemTap, so I can  
>> enable context handling.
>>
>> Concerning my second doubt, I knew it should work with CentOS, I  
>> only feared that it would cause me problems because of being an  
>> version of the OS, but it seems everything is working fine.
>>
>> Regards,
>> João
>
> The common issue with using systemtap is that it is very particular  
> about the kernel debug info and kernel devel files matching the  
> instrumented kernel exactly.  That means that rpms need to come from  
> the same build; being built at different times from the same source  
> rpm is not sufficient.
>
> -Will
>
>
>>
>> Citando William Cohen <wcohen@redhat.com>:
>>
>>> On 08/22/2014 04:51 PM, joaoandreferro@sapo.pt wrote:
>>>> Hello all,
>>>>
>>>>  I've just installed Systemtap, and to start I have a couple of  
>>>> doubts that I
>>>> would like to share with you:
>>>>
>>>>  1 - Firstly, what I'm trying to achieve is some kind of  
>>>> interrupt handler:
>>>> I would like to momentarily interrupt the execution of the OS, and then be
>>>> able to run some code of mine (to start, inject a single bit-flip into a
>>>> CPU register), and then resume the execution of the OS, with the
>>>> possibility of having information on the pre-interruption context. Is this
>>>> somehow possible with Systemtap?
>>>
>>> Hi,
>>>
>>> It sound like you are trying to do fault injection with systemtap.  
>>>  SystemTap has been used successfully for fault injection on the  
>>> Linux kernel's scsi layer:
>>>
>>> http://lwn.net/Articles/289932/
>>>
>>> SystemTap's guru mode (-g) has the ability to modify target  
>>> variable value. For the kprobes, uprobe, and timer based probes  
>>> you probably could change the values in the pt_regs struct and  
>>> effect a bit-flip.
>>>
>>> You could use the associative arrays in systemtap to store  
>>> information about where (backtrace) and when these bit-flips are  
>>> done.  Look at the the tapset library for information on various  
>>> functions to get timeofday and backtraces  
>>> https://sourceware.org/systemtap/tapsets/ .
>>>
>>>
>>>>
>>>>  2 - I'm using CentOs 6.5 32 bit (kernel version  
>>>> 2.6.32-431.23.3.el6.i686).
>>>> Will Systemtap work here? I think this information may be relevant, since
>>>> this isn't the last version of the distro, and some other tools I've
>>>> already tried to use in the past didn't worked because of my (old) kernel
>>>> version.
>>>
>>> SystemTap should work on CentOs.  SystemTap does need other rpms  
>>> such as matching kernel-devel and kernel-debuginfo.  You might  
>>> need to manually download and install those because yum typically  
>>> tries to get the most recent versions of packages.  The stap-prep  
>>> command should list out what packages you need to get sysetmtap  
>>> working on the system.  The following wiki page provides some  
>>> additional suggestions on setting up systemtap on CentOs:
>>>
>>> https://sourceware.org/systemtap/wiki/SystemTapOnCentOS
>>>
>>> -Will
>>
>>