Proposed systemtap access to perfmon hardware

[William Cohen <wcohen@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

I have written up material describing how I would think that systemtap 
could use the performance monitoring hardware. It is a work in progress, 
but I would appreciate people's comments on it.

-Will

[-- Attachment #2: stapperfmon.txt --]
[-- Type: text/plain, Size: 5928 bytes --]

Systemtap Performance Monitoring Hardware Support Proposal

March 15, 2006

Most modern processors have performance monitoring hardware that can
count event such as processor clock cycles, memory references, cache
misses, branches, and branch mispredictions.  The hardware counts can
be used directly to guage the cost of operations or the counts can be
used to trigger sampling to find out where these operations occur in
code.  SystemTap should have the ability to uses this performance
monitoring hardware to indicate what the underlying causes of the
performance problems are.



SYSTEMTAP PERFORMANCE MONITORING API


perfmon_allocate_counter:long (event_spec:string)

All the perfmon_allocate_counter() calls must be in the probe begin
(removing this restrictions will be considered later). A string as
specified in the EVENT SPECIFICATION section describes the event
performance counter configuration. If the configuration is sucessful a
even_handle in the form of a non-zero 64-bit value will be returned. A
zero value indicates that there was a problem with the counter
allocation. This event_handle will be used by other functions to
uniquely identify the counter being used. The counters are not set up
or running until the perfmon_create_context is performed.


perfmon_free_counter:long (event_handle:long)

All perfmon_free_counter() calls must be in the probe end (removing
this restrictions will be considered later). The function returns the
event_handle for a successful free operation and zero for an
unsuccessful operation.


perfmon_create_context:long ()

The perfmon_create_context command sets up the performance monitoring
hardware for the allocated contexts and starts the counters running.
If successful, the function will return zero. If the operation is
unsuccessful because an error code will be returned. This function
should only be used in probe begin. (FIXME list error code returned.)


perfmon_get_counter:long (event_handle:long)

The event_handle passed in indicates which counter to read. The value
is returned as a 64-bit long of the current counter value; the counter
could be either running or stopped.  The return value is undefined for
an invalid event_handle.


perfmon_start_counter:long (event_handle:long)

The event_handle passed in indicates which counter to start. The value
is returned as a 64-bit long of the current counter value.  The return
value is undefined for an invalid event_handle.


perfmon_stop_counter:long (event_handle:long)

The event_handle passed in indicates which counter to stop. The value
is returned as a 64-bit long of the current counter value.  The return
value is undefined for an invalid event_handle.


perfmon_handle_to_string:string (event_handle:long)

The perfmon_handle_to_string operation returns the string used by the
perfmon_allocate_counter to generate the handle.


probe kernel.perfmon.sample(event_handle:long) {/*body*/}

The kernel.perfmon.sample probe indicates the action to implement when
the counter specified by event_handle overflows. This could be
triggered at anytime, so the context information is limited to the
same data available for an asynchronous timer probe.

The event_handle is a global variable in the instrumentation
script. Multiple probes for a particular global variable is allowed.


EVENT SPECIFICATION

The performance monitoring events are specified in strings. The
information at the very least include the event name being monitored
by the counter.  Additional information would include a event mask to
specify subevents, whether to count in kernel or user space, whether
to keep track of counts on a per thread or per CPU basis, and the
interval for the sampling.

(FIXME more detail on the string compositions)


SYSTEMTAP PERFORMANCE HARDWARE ACCESS IMPLEMENTATION

The SystemTap access performance monitoring hardware is planned to be
built on the perfmon2 kernel support. The perfom2 provides reservation
and access to the performance monitoring hardware on ia64, i386, and
PowerPC processors. The perfmon2 support is not yet in the upstream
kernels, but patches are available.

Outline where things are done.

	In Translator:
	   group all probe kernel.perfmon.sample() together

	In perfmon tapset:
	   perfmon_allocate_counter()
	   perfmon_free_counter()
	   perfmon_create_context()
	   perfmon_get_counter()
	   perfmon_start_counter()
	   perfmon_stop_counter()
	   perfmon_handle_to_string()


	On startup (probe begin):
	   if perfmon.sample used, register perfmon custom buffer mechanism
	   The following steps will need some work done in userspace (libpfm):
	   -translate each of the perfmon_allocate_counter into perfmon config
	   -set up the perfmon contexts (either per processor or per pid)
	   -activate the perfmon contexts

	On shutdown (probe end):
	   The following steps will need some work done in userspace (libpfm):
	   -destroy the perfmon contexts
	   -if perfmon.sample used, unregister perfmon custom buffer mechanism

FIXME more details on the proposed implementation.



SYSTEMTAP PERFMON ISSUES

-There are numerous constraints on event setup. It is possible to
 request a configuration that cannot be set up in the performance
 monitoring hardware.

-This mechanism does not provide access to other related information
 provided by the performance monitoring hardware, e.g. the performance
 monitoring registers storing the data address tha caused a cache miss
 on ia64.

-The perfmon clones the context for new threads that have the perfmon
 context set up, but we probably do not want to attach to each
 existing thread and set up the context on it. That is going to be
 relatively expensive.

-Perfmon can either do global or per thread monitoring, but they
 cannot be mixed.


REFERENCES


Stephane Eranian, The perfmon2 interface specification
HP Laboratories, HPL-2004-200(R.1), February 7, 2005.
http://www.hpl.hp.com/techreports/2004/HPL-2004-200R1.html