Integrate delay accounting with systemtap?

Integrate delay accounting with systemtap?

[Ananth N Mavinakayanahalli]

Hi,

The delay accounting infrastructure has been part of the mainline kernel
for sometime now. From what I've heard from customer-facing people, it
is a great tool to determine where the performance bottlenecks are.

See Documentation/accounting/* for more information about this
infrastructure.

Combining this feature with the features SystemTap currently provides
would be a good thing. Would it therefore be possible to add support for
this infrastructure in SystemTap?

Ananth

Re: Integrate delay accounting with systemtap?

[Mike Mason]

We can easily add functions to access the task_delay_info values in task_struct.  Would that be sufficient or are you thinking of something more than that?

Mike

Ananth N Mavinakayanahalli wrote:
> Hi,
> 
> The delay accounting infrastructure has been part of the mainline kernel
> for sometime now. From what I've heard from customer-facing people, it
> is a great tool to determine where the performance bottlenecks are.
> 
> See Documentation/accounting/* for more information about this
> infrastructure.
> 
> Combining this feature with the features SystemTap currently provides
> would be a good thing. Would it therefore be possible to add support for
> this infrastructure in SystemTap?
> 
> Ananth

Re: Integrate delay accounting with systemtap?

[Ananth N Mavinakayanahalli]

On Wed, Jul 11, 2007 at 11:03:25AM -0700, Mike Mason wrote:
> We can easily add functions to access the task_delay_info values in 
> task_struct.  Would that be sufficient or are you thinking of something 
> more than that?

We shouldn't be using any probes or functions in kernel for this
feature. Rather, SystemTap should be taught to listen to the netlink
socket where this information is available.

Ananth

> 
> Mike
> 
> Ananth N Mavinakayanahalli wrote:
> >Hi,
> >
> >The delay accounting infrastructure has been part of the mainline kernel
> >for sometime now. From what I've heard from customer-facing people, it
> >is a great tool to determine where the performance bottlenecks are.
> >
> >See Documentation/accounting/* for more information about this
> >infrastructure.
> >
> >Combining this feature with the features SystemTap currently provides
> >would be a good thing. Would it therefore be possible to add support for
> >this infrastructure in SystemTap?
> >
> >Ananth

Re: Integrate delay accounting with systemtap?

[Frank Ch. Eigler]

Ananth N Mavinakayanahalli <ananth@in.ibm.com> writes:

> [...]  We shouldn't be using any probes or functions in kernel for
> this feature. Rather, SystemTap should be taught to listen to the
> netlink socket where this information is available.

What do you mean by "listen to the netlink socket"?  The netlink
widget is (roughly speaking) a way for user-space programs to run
kernel queries outside the many other available methods.  But
systemtap scripts already run in kernel space.  How/why would you want
them to talk across netlink sockets?

- FChE

Re: Integrate delay accounting with systemtap?

[Ananth N Mavinakayanahalli]

On Thu, Jul 12, 2007 at 10:13:16AM -0400, Frank Ch. Eigler wrote:
> Ananth N Mavinakayanahalli <ananth@in.ibm.com> writes:
> 
> > [...]  We shouldn't be using any probes or functions in kernel for
> > this feature. Rather, SystemTap should be taught to listen to the
> > netlink socket where this information is available.
> 
> What do you mean by "listen to the netlink socket"?  The netlink
> widget is (roughly speaking) a way for user-space programs to run
> kernel queries outside the many other available methods.  But
> systemtap scripts already run in kernel space.  How/why would you want
> them to talk across netlink sockets?

See Documentation/accounting/getdelays.c

Given the data is available without any additional kernel probe/hook, it
would be good for SystemTap to do something similar to getdelays.c

Ananth

Re: Integrate delay accounting with systemtap?

[Frank Ch. Eigler]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

Hi -

On Thu, Jul 12, 2007 at 07:59:23PM +0530, Ananth N Mavinakayanahalli wrote:

> > [...]  But systemtap scripts already run in kernel space.  How/why
> > would you want them to talk across netlink sockets?

> See Documentation/accounting/getdelays.c

Yes, I see that, but this is exactly a user-space program using the
netlink api to query the kernel.  This scenario does not apply to
systemtap probes that run in kernel space.

> Given the data is available without any additional kernel
> probe/hook, it would be good for SystemTap to do something similar
> to getdelays.c

But the netlink widget is necessary primarly because user space has no
access to the task struct with all the statistics fields in it.  A
probe running in kernel space can access the task struct (more or
less) directly.

- FChE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Integrate delay accounting with systemtap?

[Ananth N Mavinakayanahalli]

On Thu, Jul 12, 2007 at 10:34:31AM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Jul 12, 2007 at 07:59:23PM +0530, Ananth N Mavinakayanahalli wrote:
> 
> > > [...]  But systemtap scripts already run in kernel space.  How/why
> > > would you want them to talk across netlink sockets?
> 
> > See Documentation/accounting/getdelays.c
> 
> Yes, I see that, but this is exactly a user-space program using the
> netlink api to query the kernel.  This scenario does not apply to
> systemtap probes that run in kernel space.
> 
> > Given the data is available without any additional kernel
> > probe/hook, it would be good for SystemTap to do something similar
> > to getdelays.c
> 
> But the netlink widget is necessary primarly because user space has no
> access to the task struct with all the statistics fields in it.  A
> probe running in kernel space can access the task struct (more or
> less) directly.

Agreed. I was just thinking we should possibly not take a probe hit if
its not needed. I am fine as long as the information is available either
way.

Ananth