systemtap application to find applications doing polling

systemtap application to find applications doing polling

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

Hi All,

Uli Drepper mentions in a blog entry need "avoid unnecessary wakeups" and that a
systemtap script to monitor this would be useful:

http://udrepper.livejournal.com/19041.html

I talked with Uli about developing the script that identify the processes that
are doing a lot of polling.  The attached script, timeout.stp, monitors the
poll, epoll_wait,  select, futex, nanosleep, timer (it_real_fn). The poll and
epoll are only recorded if the timeout value is greater than zero. The resulting
output is displayed in a top-like format for the top twenty processes with the
entries ordered from most problem calls to fewest. The columns indicate the
count of each type. The output ends up like the following:

  uid |   poll  select   epoll  itimer   futex nanosle  signal| process
 2628 |      0     364       0       0       0       0       0| Xorg
 3586 |     21       0       0       0     179       0       0| thunderbird-bin
 3575 |     41       0       0       0       0      20       0| xchat
 3454 |      0      60       0       0       0       0       0| emacs
 3325 |     43       0       0       0       0       0       0| gnome-terminal
 3082 |     11       0       0       0       0       0       0| gnome-panel
 3068 |      7       0       0       0       0       0       0| metacity
 3181 |      6       0       0       0       0       0       0| wnck-applet
 3119 |      0       5       0       0       0       0       0| httpd
 2135 |      4       0       0       0       0       0       0| hald
 2307 |      4       0       0       0       0       0       0| NetworkManager
 2362 |      4       0       0       0       0       0       0| setroubleshootd
 2530 |      0       0       0       0       0       4       0| cups-polld
 3084 |      3       0       0       0       0       0       0| nautilus
 3616 |      0       0       0       0       3       0       0| firefox
 3060 |      2       0       0       0       0       0       0| gnome-settings-
 2304 |      2       0       0       0       0       0       0| hald-addon-stor
    0 |      0       0       0       1       0       0       0| swapper

I plan to check this into systemtap.examples directory in next day or so. Just
looking to see if people have additional suggestions.

-Will

[-- Attachment #2: timeout.stp --]
[-- Type: text/plain, Size: 2268 bytes --]

# Copyright (C) 2009 Red Hat, Inc.
# Written by Ulrich Drepper <drepper@redhat.com>
# Modified by William Cohen <wcohen@redhat.com>

global process, timeout_count, to
global poll_timeout, epoll_timeout, select_timeout, itimer_timeout
global nanosleep_timeout, futex_timeout, signal_timeout

probe syscall.poll, syscall.epoll_wait {
  if (timeout) to[pid()]=timeout
}

probe syscall.poll.return {
  p = pid()
  if ($return == 0 && to[p] > 0 ) {
    poll_timeout[p]++
    timeout_count[p]++
    process[p] = execname()
    delete to[p]
  }
}

probe syscall.epoll_wait.return {
  if ($return == 0 && to[p] > 0 ) {
    p = pid()
    epoll_timeout[p]++
    timeout_count[p]++
    process[p] = execname()
    delete to[p]
  }
}

probe syscall.select.return {
  if ($return == 0) {
    p = pid()
    select_timeout[p]++
    timeout_count[p]++
    process[p] = execname()
  }
}

probe syscall.futex.return {
  if (errno_str($return) == "ETIMEDOUT") {
    p = pid()
    futex_timeout[p]++
    timeout_count[p]++
    process[p] = execname()
  }
}

probe syscall.nanosleep.return {
  if ($return == 0) {
    p = pid()
    nanosleep_timeout[p]++
    timeout_count[p]++
    process[p] = execname()
  }
}

probe kernel.function("it_real_fn") {
  p = pid()
  itimer_timeout[p]++
  timeout_count[p]++
  process[p] = execname()
}

probe syscall.rt_sigtimedwait.return {
  if (errno_str($return) == "EAGAIN") {
    p = pid()
    signal_timeout[p]++
    timeout_count[p]++
    process[p] = execname()
  }
}

probe syscall.exit {
  p = pid()
  if (p in process) {
    delete process[p]
    delete timeout_count[p]
    delete poll_timeout[p]
    delete epoll_timeout[p]
    delete select_timeout[p]
    delete itimer_timeout[p]
    delete futex_timeout[p]
    delete nanosleep_timeout[p]
    delete signal_timeout[p]
  }
}

probe timer.s(1) {
  printf("\033[2J\033[1;1H") /* clear screen */
  printf ("  uid |   poll  select   epoll  itimer   futex nanosle  signal| process\n")
  foreach (p in timeout_count- limit 20) {
     printf ("%5d |%7d %7d %7d %7d %7d %7d %7d| %-.38s\n", p,
              poll_timeout[p], select_timeout[p],
              epoll_timeout[p], itimer_timeout[p],
              futex_timeout[p], nanosleep_timeout[p],
              signal_timeout[p], process[p])
  }
}

Re: systemtap application to find applications doing polling

[Jim Keniston]

On Wed, 2009-01-28 at 11:52 -0500, William Cohen wrote:
> Hi All,
> 
> Uli Drepper mentions in a blog entry need "avoid unnecessary wakeups" and that a
> systemtap script to monitor this would be useful:
...
> 
> I plan to check this into systemtap.examples directory in next day or so. Just
> looking to see if people have additional suggestions.
> 
> -Will
> # Copyright (C) 2009 Red Hat, Inc.
> # Written by Ulrich Drepper <drepper@redhat.com>
> # Modified by William Cohen <wcohen@redhat.com>
> 
> global process, timeout_count, to
> global poll_timeout, epoll_timeout, select_timeout, itimer_timeout
> global nanosleep_timeout, futex_timeout, signal_timeout
> 
> probe syscall.poll, syscall.epoll_wait {
>   if (timeout) to[pid()]=timeout
> }
> 
> probe syscall.poll.return {
>   p = pid()
>   if ($return == 0 && to[p] > 0 ) {
>     poll_timeout[p]++
>     timeout_count[p]++
>     process[p] = execname()
>     delete to[p]
>   }
> }

I'm wondering if this script could be shorter (though perhaps less
efficient) if you used a function like this:

global timeouts
function tally_timeout(name:string) {
	p = pid()
	timeouts[name, p]++
	timeout_count[p]++
	process[p] = execname()
}
> 
> probe syscall.epoll_wait.return {
>   if ($return == 0 && to[p] > 0 ) {

Here p is used before it is set.

>     p = pid()
>     epoll_timeout[p]++
>     timeout_count[p]++
>     process[p] = execname()
>     delete to[p]
>   }
> }

The above could turn into

	p = pid()
	if ($return == 0 && to[p] > 0) {
		tally_timeout(name)
		delete to[p]
	}
> 
> probe syscall.select.return {
>   if ($return == 0) {
>     p = pid()
>     select_timeout[p]++
>     timeout_count[p]++
>     process[p] = execname()
>   }
> }
> 
> probe syscall.futex.return {
>   if (errno_str($return) == "ETIMEDOUT") {
>     p = pid()
>     futex_timeout[p]++
>     timeout_count[p]++
>     process[p] = execname()
>   }
> }
> 
> probe syscall.nanosleep.return {
>   if ($return == 0) {
>     p = pid()
>     nanosleep_timeout[p]++
>     timeout_count[p]++
>     process[p] = execname()
>   }
> }
> 
> probe kernel.function("it_real_fn") {
>   p = pid()
>   itimer_timeout[p]++
>   timeout_count[p]++
>   process[p] = execname()
> }
> 
> probe syscall.rt_sigtimedwait.return {
>   if (errno_str($return) == "EAGAIN") {
>     p = pid()
>     signal_timeout[p]++
>     timeout_count[p]++
>     process[p] = execname()
>   }
> }
> 
> probe syscall.exit {
>   p = pid()
>   if (p in process) {
>     delete process[p]
>     delete timeout_count[p]
>     delete poll_timeout[p]
>     delete epoll_timeout[p]
>     delete select_timeout[p]
>     delete itimer_timeout[p]
>     delete futex_timeout[p]
>     delete nanosleep_timeout[p]
>     delete signal_timeout[p]
>   }
> }
> 
> probe timer.s(1) {
>   printf("\033[2J\033[1;1H") /* clear screen */
>   printf ("  uid |   poll  select   epoll  itimer   futex nanosle  signal| process\n")
>   foreach (p in timeout_count- limit 20) {
>      printf ("%5d |%7d %7d %7d %7d %7d %7d %7d| %-.38s\n", p,
>               poll_timeout[p], select_timeout[p],
>               epoll_timeout[p], itimer_timeout[p],
>               futex_timeout[p], nanosleep_timeout[p],
>               signal_timeout[p], process[p])
>   }
> }

Jim