RE: Proposed systemtap access to perfmon hardware

RE: Proposed systemtap access to perfmon hardware

[Stone, Joshua I]

William Cohen wrote:
> It is and open question what the counters default are; do they start
> running by default or have to be explicitly started.

This could be solved with an extension on the probe declaration.  For
example, the sensible default is probably to start the counters, but we
could allow a ".paused" to override that default.

> If they are started by default, where exactly are they running?
> Beginning of begin probe? End of begin probe?

I think the perfmon setup needs to happen before all begin probes, so
that the handle can be accessed within begin probes.  I don't know that
it really matters when you actually start the counters, but I would lean
towards putting that after.

A side note about handles - it might be useful to add a language
semantic to make it easier to capture handles.  There are actions that
would make sense for handles of other probe types besides perfmon,
especially being able to dynamically enable/disable kprobes and timers.


Josh

Re: Proposed systemtap access to perfmon hardware

[Frank Ch. Eigler]

joshua wrote:

>  [...]
> > If they are started by default, where exactly are they running?
> > Beginning of begin probe? End of begin probe?
> 
> I think the perfmon setup needs to happen before all begin probes, so
> that the handle can be accessed within begin probes.  [...]

How important would that be?  At this time, we don't provide any
ordering guarantees amongst begin/end probes.  If a begin probe would
have to manipulate the handle, why not put that right into the
perfctr.*.setup probe directly?

> A side note about handles [...]  especially being able to
> dynamically enable/disable kprobes and timers.

Actual disarming of these kinds of probes is heavy-weight and may not
be safely done from within the confines of some generic
interrupt-disabled atomic probe handler.  For the perfctr case, we may
just directly poke at registers.

Programmatic control of swaths of probes is an interesting problem
though.


- FChE

RE: Proposed systemtap access to perfmon hardware

[Stone, Joshua I]

William Cohen wrote:
> As far as when the performance counters are set up and torn down it
> seems like it would be most reasonable to set them up before the first
> probe begin action and tear them down after the last probe end action.

I agree

> This would mean for sampling would need to have it stop sampling if
> don't want any additional samples while doing the probe end action.

This should be gated for you by using the session_state.  Begin probes
only run during STAP_SESSION_STARTING, normal probes (including permon
sampling) should only run during STAP_SESSION_RUNNING, and end probes
only run during STAP_SESSION_STOPPING.  If a probe is entered during a
state that doesn't match what it expects, it just returns with out
taking any action.


Josh

Re: Proposed systemtap access to perfmon hardware

[William Cohen]

Maynard Johnson wrote:
> Stone, Joshua I wrote:
> 
>> Maynard Johnson wrote:
>>  
>>
>>> William Cohen wrote:
>>>   
>>>
>>>> The individual start and stop operations would be allowed.
>>>>     
>>>
>>> This is not so good.  Besides the fact that it may be difficult (or
>>> impossible) to do, I don't see it being all that useful.  But then,
>>> I'm a tool developer, not a performance analyst, so I could be
>>> missing the point.
>>>   
>>
>>
>> Enabling start & stop lets you narrow the context that you want to
>> measure.  Perfmon can only give you thread level virtualization of the
>> counters.  With start & stop I can, for example, start the counters when
>> I enter sys_open and stop when I return.  Now if I want I can get a
>> microbenchmark of IPC for the sys_open call (and its callees).

The place that I see the start and stop being most useful is for the 
sampling. Start the when some event occurs and stop the sampling when 
another event occurs to get a statistic picture of what is going on in a 
certain region. It would be possible have a flag in the sample routine 
to turn on and off recording the sample. However, this would mean the 
sampling would start counting when the sampling is turned on.

For the interval measurements it may be possible to leave the counter 
counting. This would avoid messing with the performance counter state. 
In the perfmon_start_counter() mark status as running and accumulate 
count in running state. In perfmon_stop_counter() mark status as stops 
and accumulate the count in the stopped state. This could be implemented 
for the global version. It might be a bit more complicated for the per 
process version there is state information for each context and I am not 
sure about whether the additional information managing the counter 
software state could fit in the context information for a thread.

>>
>> But this also opens up possibilities for more obscure "contexts" -
>> perhaps I want to start counting when a network packet is received and
>> stop when it is delivered to the thread.  Any probepoint you can do
>> today can become a start/stop point for the counters.
>>  
>>
> Yes, I can certainly see this benefit.  It gives you PAPI-level control 
> without having to modify source code.  My concern, however, was that if 
> you have multiple counters configured, then individual control of them 
> presents an extra level of difficulty.  But, as I've been thinking about 
> this a bit more, I think this could be done if you can guarantee that 
> the operation is not preempted or interrupted.  Then, the PMU can be 
> disabled, reloaded with any changes, and then re-enabled.  Then, any 
> counters that had been running before the operation -- and that were not 
> changed by the operation -- will be reloaded with their previous count 
> and continue running from where they left off.
> 
> -Maynard
> 
>>
>> Josh

Have the probe specifying whether the performance counter is in the 
running or stopped state when setup is a good idea.

As far as when the performance counters are set up and torn down it 
seems like it would be most reasonable to set them up before the first 
probe begin action and tear them down after the last probe end action. 
This would mean for sampling would need to have it stop sampling if 
don't want any additional samples while doing the probe end action.

-Will

Re: Proposed systemtap access to perfmon hardware

[Maynard Johnson]

Stone, Joshua I wrote:

>Maynard Johnson wrote:
>  
>
>>William Cohen wrote:
>>    
>>
>>>The individual start and stop operations would be allowed.
>>>      
>>>
>>This is not so good.  Besides the fact that it may be difficult (or
>>impossible) to do, I don't see it being all that useful.  But then,
>>I'm a tool developer, not a performance analyst, so I could be
>>missing the point.
>>    
>>
>
>Enabling start & stop lets you narrow the context that you want to
>measure.  Perfmon can only give you thread level virtualization of the
>counters.  With start & stop I can, for example, start the counters when
>I enter sys_open and stop when I return.  Now if I want I can get a
>microbenchmark of IPC for the sys_open call (and its callees).
>
>But this also opens up possibilities for more obscure "contexts" -
>perhaps I want to start counting when a network packet is received and
>stop when it is delivered to the thread.  Any probepoint you can do
>today can become a start/stop point for the counters.
>  
>
Yes, I can certainly see this benefit.  It gives you PAPI-level control 
without having to modify source code.  My concern, however, was that if 
you have multiple counters configured, then individual control of them 
presents an extra level of difficulty.  But, as I've been thinking about 
this a bit more, I think this could be done if you can guarantee that 
the operation is not preempted or interrupted.  Then, the PMU can be 
disabled, reloaded with any changes, and then re-enabled.  Then, any 
counters that had been running before the operation -- and that were not 
changed by the operation -- will be reloaded with their previous count 
and continue running from where they left off.

-Maynard

>
>Josh
>  
>

RE: Proposed systemtap access to perfmon hardware

[Stone, Joshua I]

Maynard Johnson wrote:
> William Cohen wrote:
>> The individual start and stop operations would be allowed.
> This is not so good.  Besides the fact that it may be difficult (or
> impossible) to do, I don't see it being all that useful.  But then,
> I'm a tool developer, not a performance analyst, so I could be
> missing the point.

Enabling start & stop lets you narrow the context that you want to
measure.  Perfmon can only give you thread level virtualization of the
counters.  With start & stop I can, for example, start the counters when
I enter sys_open and stop when I return.  Now if I want I can get a
microbenchmark of IPC for the sys_open call (and its callees).

But this also opens up possibilities for more obscure "contexts" -
perhaps I want to start counting when a network packet is received and
stop when it is delivered to the thread.  Any probepoint you can do
today can become a start/stop point for the counters.


Josh

RE: Proposed systemtap access to perfmon hardware

[Stone, Joshua I]

fche@redhat.com wrote:
> joshua wrote:
>>  [...]
>>> If they are started by default, where exactly are they running?
>>> Beginning of begin probe? End of begin probe?
>> 
>> I think the perfmon setup needs to happen before all begin probes, so
>> that the handle can be accessed within begin probes.  [...]
> 
> How important would that be?  At this time, we don't provide any
> ordering guarantees amongst begin/end probes.  If a begin probe would
> have to manipulate the handle, why not put that right into the
> perfctr.*.setup probe directly?

Perhaps it isn't that important - I didn't consider putting the work in
the ".setup" probe.  This is fine as long as the setup probe has the
guarantee that the counters haven't started yet (regardless of its
position in the begin stage)

It might be nice to have a similar shutdown/cleanup/whatever probe that
guarantees that the counters have been stopped.  This is superfluous if
we guarantee that the counters will be stopped before running any end
probes, but I haven't seen discussion on that point yet.  We should also
define when the handle be invalidated - this could be at the conclusion
of the cleanup probe, if it exists, or after all end probes have
completed.


Josh

Re: Proposed systemtap access to perfmon hardware

[Maynard Johnson]

William Cohen wrote:
> Maynard Johnson wrote:
> 
>> William Cohen wrote:
>>
>>> [snip]
>>>
>>> perfmon_create_context:long ()
>>>
>>> The perfmon_create_context command sets up the performance monitoring
>>> hardware for the allocated contexts and starts the counters running.
>>> If successful, the function will return zero. If the operation is
>>> unsuccessful because an error code will be returned. This function
>>> should only be used in probe begin. (FIXME list error code returned.)
>>>  
>>>
>> I'm confused about the relationship between this function and 
>> perfmon_start_counter, since starting the counters is mentioned in 
>> both.  Could you explain at what point this function is invoked and 
>> what the purpose of the context is?  I'm not real familiar with the 
>> perfmon2 interface, but just on the face of it, your context doesn't 
>> seem like a one-to-one fit with the way contexts are used in 
>> perfmon2.  In perfmon2, a context is created first, which is then 
>> passed in to the calls for setting up events, thereby associating 
>> those events with the context. Then 'start' uses the context to set up 
>> the PMU for all requested events and begin the counting.
> 
> 
> Yes, perfmon2 has a contexts that sets all the performance monitoring 
> hardware registers. The perfmon2 start and stop control the entire context.
> 
> Based on the feedback from earlier proposal email, revised to using 
> something like:
> 
> probe perfmon.event("blah") ...
> 
> All the probes using the perfmon hardware would be collected together 
> for the perfmon_create_context. 
This is good.
> The individual start and stop operations would be allowed. 
This is not so good.  Besides the fact that it may be difficult (or 
impossible) to do, I don't see it being all that useful.  But then, I'm 
a tool developer, not a performance analyst, so I could be missing the 
point.

 > It is and open question what the counters default are;
> do they start running by default or have to be explicitly started. If 
> they are started by default, where exactly are they running? Beginning 
> of begin probe? End of begin probe?
> 
>>>
>>> [snip]
>>>
>>> perfmon_start_counter:long (event_handle:long)
>>>
>>> The event_handle passed in indicates which counter to start. The value
>>> is returned as a 64-bit long of the current counter value.  The return
>>> value is undefined for an invalid event_handle.
>>>  
>>>
>> I think individually starting counters is problematic at a couple 
>> different levels.  On some architectures (like PowerPC64), you don't 
>> have fine-grained control over each counter.  Also, one usually wants 
>> all counters to begin counting at the same time.  Maybe I'm 
>> misinterpreting what the intention of this function is.
> 
> 
> I was thinking there are cases where one would want to start and stop 
> individual sampling and interval counting. Yes, starting and stoping 
> counters on some architectures can be a problem.  I was thinking if 
> cheating and not actually starting and stopping the counters, but rather 
> turning on and off the bits that enabling counting in user and kernel 
> space. Do this by finding which bits to twiddle in the control register. 
Unfortunately, this isn't possible for ppc64.  The control bits you 
mention (for user/kernel domain) are used for all counters, so there's 
no fine-grained control there.  There are PMCxSEL bits for setting up 
each counter for what you want it to count (including "count nothing"), 
but changing these on the fly (i.e., without disabling the PMU) may not 
have the desired effect.  The documentation states that you should first 
disable the PMU before you change these bits, but it doesn't say what 
would happen if you didn't disable.

-Maynard
> However, maybe this won't work for ppc64. I will have to review the 
> ppc64 hardware manual to see that this scheme would work.
> 
>>> [snip]
>>>
>>
>>> EVENT SPECIFICATION
>>>
>>> The performance monitoring events are specified in strings. The
>>> information at the very least include the event name being monitored
>>>  
>>>
>> Will, you allude to this in a later posting, but I'll reiterate here.  
>> Should the event name be the native event name for the arch?  Or some 
>> generic name that is mapped to a native name by some mechanism?  Or 
>> either (as in PAPI)?
> 
> 
> libpfm has some generic names for cycle counts. I expect that events 
> will be both generic names and architecture specific. This will be a 
> lookup in libpfm.
> 
>>> by the counter.  Additional information would include a event mask to
>>> specify subevents, whether to count in kernel or user space, whether
>>> to keep track of counts on a per thread or per CPU basis, and the
>>> interval for the sampling.
>>>
>>> (FIXME more detail on the string compositions)
>>>
>>>
>>> SYSTEMTAP PERFORMANCE HARDWARE ACCESS IMPLEMENTATION
>>>
>>> The SystemTap access performance monitoring hardware is planned to be
>>> built on the perfmon2 kernel support. The perfom2 provides reservation
>>> and access to the performance monitoring hardware on ia64, i386, and
>>> PowerPC processors. The perfmon2 support is not yet in the upstream
>>> kernels, but patches are available.
>>>  
>>>
>> As a proof of concept, I agree that this is the best route.  
>> Reinventing the wheel would be useless.  Maybe building this prototype 
>> might help with refining the perfmon2 interface.
> 
> 
> I have been working on patching oprofile so that it uses the perfmon2 
> interface. The work is being done on an amd64 machine. This should allow 
> some examination of the mechanisms for setting up the events and 
> sampling. It should be portable to perfmon2 for i386, ppc64, and ia64. I 
> will make the patches available for comment.
> 
> Next step would be to protoype similar opertation for systemtap.
> 
> I am trying to avoid reinventing the wheel. I am also very concerned 
> that raw access of the performance monitoring hardware will further 
> increase the chances of multiple device drivers stepping on each other 
> without knowing about it.
> 
> -Will

Re: Proposed systemtap access to perfmon hardware

[Frank Ch. Eigler]

maynardj wrote:

> [...]

Several aspects of the script interface to the performance counters
was changed later in this thread: please check that too.

> >The performance monitoring events are specified in strings. The
> >information at the very least include the event name being monitored
> [...]
> Should the event name be the native event name for the arch?  Or some
> generic name that is mapped to a native name by some mechanism?  Or
> either (as in PAPI)?

It may be sufficient to use systemtap's general abstraction mechanisms
to map between generic and native event names, in much the same way as
the system-call tapset defines generic names ("syscall.read") in terms
of native functions ("kernel.function(...)").  This aliasing widget
may need some extension in order to deal with parameters or partial
matches; we'll know after someone constructs an informal
generic<->native event name dictionary.

- FChE

Re: Proposed systemtap access to perfmon hardware

[William Cohen]

Maynard Johnson wrote:
> William Cohen wrote:
> 
>> [snip]
>>
>> perfmon_create_context:long ()
>>
>> The perfmon_create_context command sets up the performance monitoring
>> hardware for the allocated contexts and starts the counters running.
>> If successful, the function will return zero. If the operation is
>> unsuccessful because an error code will be returned. This function
>> should only be used in probe begin. (FIXME list error code returned.)
>>  
>>
> I'm confused about the relationship between this function and 
> perfmon_start_counter, since starting the counters is mentioned in 
> both.  Could you explain at what point this function is invoked and what 
> the purpose of the context is?  I'm not real familiar with the perfmon2 
> interface, but just on the face of it, your context doesn't seem like a 
> one-to-one fit with the way contexts are used in perfmon2.  In perfmon2, 
> a context is created first, which is then passed in to the calls for 
> setting up events, thereby associating those events with the context. 
> Then 'start' uses the context to set up the PMU for all requested events 
> and begin the counting.

Yes, perfmon2 has a contexts that sets all the performance monitoring 
hardware registers. The perfmon2 start and stop control the entire context.

Based on the feedback from earlier proposal email, revised to using 
something like:

probe perfmon.event("blah") ...

All the probes using the perfmon hardware would be collected together 
for the perfmon_create_context. The individual start and stop operations 
would be allowed. It is and open question what the counters default are; 
do they start running by default or have to be explicitly started. If 
they are started by default, where exactly are they running? Beginning 
of begin probe? End of begin probe?

>>
>> [snip]
>>
>> perfmon_start_counter:long (event_handle:long)
>>
>> The event_handle passed in indicates which counter to start. The value
>> is returned as a 64-bit long of the current counter value.  The return
>> value is undefined for an invalid event_handle.
>>  
>>
> I think individually starting counters is problematic at a couple 
> different levels.  On some architectures (like PowerPC64), you don't 
> have fine-grained control over each counter.  Also, one usually wants 
> all counters to begin counting at the same time.  Maybe I'm 
> misinterpreting what the intention of this function is.

I was thinking there are cases where one would want to start and stop 
individual sampling and interval counting. Yes, starting and stoping 
counters on some architectures can be a problem.  I was thinking if 
cheating and not actually starting and stopping the counters, but rather 
turning on and off the bits that enabling counting in user and kernel 
space. Do this by finding which bits to twiddle in the control register. 
However, maybe this won't work for ppc64. I will have to review the 
ppc64 hardware manual to see that this scheme would work.

>> [snip]
>>
> 
>> EVENT SPECIFICATION
>>
>> The performance monitoring events are specified in strings. The
>> information at the very least include the event name being monitored
>>  
>>
> Will, you allude to this in a later posting, but I'll reiterate here.  
> Should the event name be the native event name for the arch?  Or some 
> generic name that is mapped to a native name by some mechanism?  Or 
> either (as in PAPI)?

libpfm has some generic names for cycle counts. I expect that events 
will be both generic names and architecture specific. This will be a 
lookup in libpfm.

>> by the counter.  Additional information would include a event mask to
>> specify subevents, whether to count in kernel or user space, whether
>> to keep track of counts on a per thread or per CPU basis, and the
>> interval for the sampling.
>>
>> (FIXME more detail on the string compositions)
>>
>>
>> SYSTEMTAP PERFORMANCE HARDWARE ACCESS IMPLEMENTATION
>>
>> The SystemTap access performance monitoring hardware is planned to be
>> built on the perfmon2 kernel support. The perfom2 provides reservation
>> and access to the performance monitoring hardware on ia64, i386, and
>> PowerPC processors. The perfmon2 support is not yet in the upstream
>> kernels, but patches are available.
>>  
>>
> As a proof of concept, I agree that this is the best route.  Reinventing 
> the wheel would be useless.  Maybe building this prototype might help 
> with refining the perfmon2 interface.

I have been working on patching oprofile so that it uses the perfmon2 
interface. The work is being done on an amd64 machine. This should allow 
some examination of the mechanisms for setting up the events and 
sampling. It should be portable to perfmon2 for i386, ppc64, and ia64. I 
will make the patches available for comment.

Next step would be to protoype similar opertation for systemtap.

I am trying to avoid reinventing the wheel. I am also very concerned 
that raw access of the performance monitoring hardware will further 
increase the chances of multiple device drivers stepping on each other 
without knowing about it.

-Will

Re: Proposed systemtap access to perfmon hardware

[Maynard Johnson]

William Cohen wrote:

> [snip]
>
>perfmon_create_context:long ()
>
>The perfmon_create_context command sets up the performance monitoring
>hardware for the allocated contexts and starts the counters running.
>If successful, the function will return zero. If the operation is
>unsuccessful because an error code will be returned. This function
>should only be used in probe begin. (FIXME list error code returned.)
>  
>
I'm confused about the relationship between this function and 
perfmon_start_counter, since starting the counters is mentioned in 
both.  Could you explain at what point this function is invoked and what 
the purpose of the context is?  I'm not real familiar with the perfmon2 
interface, but just on the face of it, your context doesn't seem like a 
one-to-one fit with the way contexts are used in perfmon2.  In perfmon2, 
a context is created first, which is then passed in to the calls for 
setting up events, thereby associating those events with the context. 
Then 'start' uses the context to set up the PMU for all requested events 
and begin the counting.

>
>[snip]
>
>perfmon_start_counter:long (event_handle:long)
>
>The event_handle passed in indicates which counter to start. The value
>is returned as a 64-bit long of the current counter value.  The return
>value is undefined for an invalid event_handle.
>  
>
I think individually starting counters is problematic at a couple 
different levels.  On some architectures (like PowerPC64), you don't 
have fine-grained control over each counter.  Also, one usually wants 
all counters to begin counting at the same time.  Maybe I'm 
misinterpreting what the intention of this function is.

>[snip]
>

>EVENT SPECIFICATION
>
>The performance monitoring events are specified in strings. The
>information at the very least include the event name being monitored
>  
>
Will, you allude to this in a later posting, but I'll reiterate here.  
Should the event name be the native event name for the arch?  Or some 
generic name that is mapped to a native name by some mechanism?  Or 
either (as in PAPI)?

>by the counter.  Additional information would include a event mask to
>specify subevents, whether to count in kernel or user space, whether
>to keep track of counts on a per thread or per CPU basis, and the
>interval for the sampling.
>
>(FIXME more detail on the string compositions)
>
>
>SYSTEMTAP PERFORMANCE HARDWARE ACCESS IMPLEMENTATION
>
>The SystemTap access performance monitoring hardware is planned to be
>built on the perfmon2 kernel support. The perfom2 provides reservation
>and access to the performance monitoring hardware on ia64, i386, and
>PowerPC processors. The perfmon2 support is not yet in the upstream
>kernels, but patches are available.
>  
>
As a proof of concept, I agree that this is the best route.  Reinventing 
the wheel would be useless.  Maybe building this prototype might help 
with refining the perfmon2 interface.

>  
>
Regards,
-Maynard

Re: Proposed systemtap access to perfmon hardware

[Frank Ch. Eigler]

Hi -

wcohen wrote:
> [...]
> > [...]
> > Is that sufficiently detailed to begin an implementation?
>
> Pretty close. The one thing that isn't answered is the division of
> the labor for the sampling probes, onetime setup vs sample handler.
> [...]

In other words, the issue is the desire to control sampling-event-type
counters, not just the free-running counters.  In this case, one might
use both ".setup" and ".sample" probes for the same SPEC:

# probe perfctr.SPEC.setup { h = $handle }
# probe perfctr.SPEC.sample(1234) { /* like timer.profile */ }
# probe ANY { ... perfctr_{start,stop,query} (h) ... }

- FChE

Re: Proposed systemtap access to perfmon hardware

[William Cohen]

Frank Ch. Eigler wrote:
> wcohen wrote:
> 
> 
>>To try to get a feel on how the performance monitoring hardware
>>support would work in SystemTap I wrote some simple examples. 
> 
> 
> Nice work.  To flesh out the operational model (and please correct me
> if I'm wrong): the way this stuff would all work is:
> 
> - The systemtap translator would be linked with libpfm from perfmon2.
>   (libpfm license is friendly.)

The libpfm  library license is an MIT license, so it should be 
compatible with the systemtap licensing.

> - This library would be used at translation time to map perfmon.* probe
>   point specifications to PMC register descriptions (pfmlib_output_param_t).
>   (This will require telling the system the exact target cpu type for
>   cross-instrumentation.)

Yes, this complicates the cross kernel (build instrumentation on one 
system and run instrument on another). Different processors 
architectures could be used on each. Some performance monitoring systems 
such as PAPI has mappings for some generic names. This might help in 
some cases. However, there are some differences in computer architecture 
that just do not translate to the generic models

> - These descriptions would be emitted into the C code, for actual
>   installation during module initialization.  For our first cut, since
>   there appears to exist no kernel-side management API at the moment,
>   the C code would directly manipulate the PMC registers.  (This means
>   no coexistence for oprofile or other concurrent perfctr probing.
>   C'est la vie.)

Would prefer to reuse to other software to access the performance 
monitoring hardware. Don't want to generate yet another different piece 
of software that uses the performance monitoring hardware. We want 
64-bit values, but a number of the counters are much smaller than that 
(32-bit). On the pentium 4 the access to the performance counters is 
complicated and would prefer not reinventing the code to access the 
performance counters. This mechanism will only work with the global 
setup like sampling per thread would be unsupported. Also need to 
translate between the name and the event number the table in OProfile 
and perfmon are getting pretty large to keep all that information and 
catch any inabilities to map events to a register.

One advantage of generating the C code would be that it would work with 
existing RHEL4 kernel.

> - The "sample" type perfmon probes would map to the same kind of
>   dispatch/callback as the current "timer.profile": the probe handler
>   should have valid pt_regs available.

Yes, the pt_regs will be available to the sample type probe.

> - The free-running type perfmon probes, probably named
>   "perfctr.SPEC.setup" or ".start" or ".begin" would map to a one-time
>   initialization that passes a token (PMC counter number?)  to the
>   handler.  Other probe handlers can then query/manipulate the
>   free-running counter using that number via the start/stop/query
>   functions.
 >
> Is that sufficiently detailed to begin an implementation?

Pretty close. The one thing that isn't answered is the division of the 
labor for the sampling probes, onetime setup vs sample handler. Want to 
have some handle set in a global variable for the probe, but do not want 
to execute that everytime that the sample is collected. For the 
free-running probes it is pretty clear to handle the samples.

>>[...] print ("ipc is %d.%d \n", ipc/factor, ipc % factor);
> 
> 
> (An aside: we should have a more compact notation for this.  We won't
> support floating point numbers, but integers can be commonly scaled
> like this.  Maybe printf("%.Nf", value), where N implies a
> power-of-ten scaling factor, and printf("%*f", value, scale) for
> general factors.)

Yes, some scaling mechanism would be nice in some cases. The chances of 
having IPC around the value of one were pretty likely, so I put in the 
scaling to give a better picture of what is going on.

-Will

Re: Proposed systemtap access to perfmon hardware

[Frank Ch. Eigler]

wcohen wrote:

> To try to get a feel on how the performance monitoring hardware
> support would work in SystemTap I wrote some simple examples. 

Nice work.  To flesh out the operational model (and please correct me
if I'm wrong): the way this stuff would all work is:

- The systemtap translator would be linked with libpfm from perfmon2.
  (libpfm license is friendly.)

- This library would be used at translation time to map perfmon.* probe
  point specifications to PMC register descriptions (pfmlib_output_param_t).
  (This will require telling the system the exact target cpu type for
  cross-instrumentation.)

- These descriptions would be emitted into the C code, for actual
  installation during module initialization.  For our first cut, since
  there appears to exist no kernel-side management API at the moment,
  the C code would directly manipulate the PMC registers.  (This means
  no coexistence for oprofile or other concurrent perfctr probing.
  C'est la vie.)

- The "sample" type perfmon probes would map to the same kind of
  dispatch/callback as the current "timer.profile": the probe handler
  should have valid pt_regs available.

- The free-running type perfmon probes, probably named
  "perfctr.SPEC.setup" or ".start" or ".begin" would map to a one-time
  initialization that passes a token (PMC counter number?)  to the
  handler.  Other probe handlers can then query/manipulate the
  free-running counter using that number via the start/stop/query
  functions.

Is that sufficiently detailed to begin an implementation?


> [...] print ("ipc is %d.%d \n", ipc/factor, ipc % factor);

(An aside: we should have a more compact notation for this.  We won't
support floating point numbers, but integers can be commonly scaled
like this.  Maybe printf("%.Nf", value), where N implies a
power-of-ten scaling factor, and printf("%*f", value, scale) for
general factors.)


- FChE

Re: Proposed systemtap access to perfmon hardware

[Bill Rugolsky Jr.]

On Fri, Mar 17, 2006 at 11:20:46AM -0500, William Cohen wrote:
> To try to get a feel on how the performance monitoring hardware support 
> would work in SystemTap I wrote some simple examples. Below are examples 
> for computing IPC, average cycle count, and sampling within a function. 
> The IPC and average cycle count function need a bit of rework to work 
> for a SMP machines.
> 
> Let me know if there are comments or questions on the examples.
 
A non-statistical application for systemtap perfctrs:

In light of a suggestion made by Alan Cox yesterday,

   http://lkml.org/lkml/2006/3/16/118

I'm hacking up some additions to Ingo Molnar's latency tracing patch
that will track retired insns as well as clock cycles, so I can see
where the processor is stalled on I/O, SMM traps, etc., within a given
execution path.

For the particular case that triggered the aforementioned thread, I know
the precise offending instruction, but in the general case a search
for the culprit might be necessary, and doing this with systemtap would
be convenient.

Regards,

	Bill Rugolsky

Re: Proposed systemtap access to perfmon hardware

[William Cohen]

To try to get a feel on how the performance monitoring hardware support 
would work in SystemTap I wrote some simple examples. Below are examples 
for computing IPC, average cycle count, and sampling within a function. 
The IPC and average cycle count function need a bit of rework to work 
for a SMP machines.

Let me know if there are comments or questions on the examples.

-Will

COMPUTING IPC

global cycles_h
global instr_retired_h
probe perfmon.event("cycles") {cycles_h = $handle;}
probe perfmon.event("intr_retired") {instr_retired_h = $handle;}

probe begin {print ("start probe");}
probe end
{
	factor=100;
	ipc = (factor*perfmon_get_counter(intr_retired_h))/
	    perfmon_get_counter(cycles_h);
	print ("ipc is %d.%d \n", ipc/factor, ipc % factor);
}




DETERMINING AVERAGE CYCLE COUNT FOR FUNCTION (AND CHILDREN)

global cycles_h

probe perfmon.event("cycles") {
       cycles_h = $handle;
       perfmon_stop_counter(cycles_h);
}

global count

probe kernel.function("blah"){
       ++count;
       perfmon_start_counter(cycles_h);
}

probe kernel.function.return("blah"){
       perfmon_stop_counter(cycles_h);
}

probe begin {print ("start probe");}
probe end
{
	total_cycles=perfmon_stop_counter(cycles_h);
	print ("average count in blah %d\n", total_cycles/count);
}


SAMPLING WITHIN A FUNCTION (AND CHILDREN)

global cycles_h
global where_am_i

probe perfmon.event("cycles").sample(100000) {
       cycles_h = $handle;
       # record where sample occured
       where_am_i[instruction_pointer()]++;
}

global count

probe kernel.function("blah"){
       ++count;
       perfmon_start_counter(cycles_h);
}

probe kernel.function("blah").return{
       perfmon_stop_counter(cycles_h);
}

probe begin
{
       # turn off the sampling
       perfmon_stop_counter(cycles_h);
       print("start probe");
}

probe end
{
	#write out the where_am_i entries
	print("address\tcount\n");
	foreach ([+ip] in where_am_i) {
		print("0x%x\t%d\n", ip, where_am_i[ip]);
	}
}

Re: Proposed systemtap access to perfmon hardware

[Frank Ch. Eigler]

wcohen wrote:

> [...]  I have written up material describing how I would think that
> systemtap could use the performance monitoring hardware. [...]

It would be helpful to see a hypothetical script that would use the
proposed API.

Anyway, beyond such script language design issues, the hard part has
been the provision of *some* kernel-side API in terms of which this
stuff can be implemented.  How is that going?


> [...]
> perfmon_allocate_counter:long (event_spec:string)
> perfmon_free_counter:long (event_handle:long)
> perfmon_create_context:long ()
> probe kernel.perfmon.sample(event_handle:long) {/*body*/}

These sound rather like suitable lower level functions that the
translator could use under the covers, and not functions that are
wisely exposed at the script level.

Specifically, I would rather expose each particular event_spec source
as a first-class probe point construct:

# probe perfmon.sample("event_spec") { /* body */ }

This would entail calls to such alloc/free/create functions being
emitted in the probe (un)registration boilerplate.  "event_spec"
could perhaps be expanded into several probe point components, and
result in a periodic run of the handler much like timer.ms(N):

# probe perfmon.counter("tlbmiss").cpu(0).sample(1000) { /* ... */ }


That seems to leave just free-running counter operations:

> perfmon_get_counter:long (event_handle:long)
> perfmon_start_counter:long (event_handle:long)
> perfmon_stop_counter:long (event_handle:long)

One possible way to cast these into the language, and yet retain
automatic initialization/cleanup, might be this:

# probe perfmon.counter("tlbmiss").cpu(0).run { h = $handle }
# probe ANY { perfmon_{get,start,stop}_counter (h) }
# global h

What this would do is to have that perfmon.* probe handler run just
once (during initialization), supplying the script with the
system-assigned handle for this counter.  Then another probe (though
probably not a "begin" one) can use that handle value to manipulate
the counter.


- FChE

Proposed systemtap access to perfmon hardware

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

I have written up material describing how I would think that systemtap 
could use the performance monitoring hardware. It is a work in progress, 
but I would appreciate people's comments on it.

-Will

[-- Attachment #2: stapperfmon.txt --]
[-- Type: text/plain, Size: 5928 bytes --]

Systemtap Performance Monitoring Hardware Support Proposal

March 15, 2006

Most modern processors have performance monitoring hardware that can
count event such as processor clock cycles, memory references, cache
misses, branches, and branch mispredictions.  The hardware counts can
be used directly to guage the cost of operations or the counts can be
used to trigger sampling to find out where these operations occur in
code.  SystemTap should have the ability to uses this performance
monitoring hardware to indicate what the underlying causes of the
performance problems are.



SYSTEMTAP PERFORMANCE MONITORING API


perfmon_allocate_counter:long (event_spec:string)

All the perfmon_allocate_counter() calls must be in the probe begin
(removing this restrictions will be considered later). A string as
specified in the EVENT SPECIFICATION section describes the event
performance counter configuration. If the configuration is sucessful a
even_handle in the form of a non-zero 64-bit value will be returned. A
zero value indicates that there was a problem with the counter
allocation. This event_handle will be used by other functions to
uniquely identify the counter being used. The counters are not set up
or running until the perfmon_create_context is performed.


perfmon_free_counter:long (event_handle:long)

All perfmon_free_counter() calls must be in the probe end (removing
this restrictions will be considered later). The function returns the
event_handle for a successful free operation and zero for an
unsuccessful operation.


perfmon_create_context:long ()

The perfmon_create_context command sets up the performance monitoring
hardware for the allocated contexts and starts the counters running.
If successful, the function will return zero. If the operation is
unsuccessful because an error code will be returned. This function
should only be used in probe begin. (FIXME list error code returned.)


perfmon_get_counter:long (event_handle:long)

The event_handle passed in indicates which counter to read. The value
is returned as a 64-bit long of the current counter value; the counter
could be either running or stopped.  The return value is undefined for
an invalid event_handle.


perfmon_start_counter:long (event_handle:long)

The event_handle passed in indicates which counter to start. The value
is returned as a 64-bit long of the current counter value.  The return
value is undefined for an invalid event_handle.


perfmon_stop_counter:long (event_handle:long)

The event_handle passed in indicates which counter to stop. The value
is returned as a 64-bit long of the current counter value.  The return
value is undefined for an invalid event_handle.


perfmon_handle_to_string:string (event_handle:long)

The perfmon_handle_to_string operation returns the string used by the
perfmon_allocate_counter to generate the handle.


probe kernel.perfmon.sample(event_handle:long) {/*body*/}

The kernel.perfmon.sample probe indicates the action to implement when
the counter specified by event_handle overflows. This could be
triggered at anytime, so the context information is limited to the
same data available for an asynchronous timer probe.

The event_handle is a global variable in the instrumentation
script. Multiple probes for a particular global variable is allowed.


EVENT SPECIFICATION

The performance monitoring events are specified in strings. The
information at the very least include the event name being monitored
by the counter.  Additional information would include a event mask to
specify subevents, whether to count in kernel or user space, whether
to keep track of counts on a per thread or per CPU basis, and the
interval for the sampling.

(FIXME more detail on the string compositions)


SYSTEMTAP PERFORMANCE HARDWARE ACCESS IMPLEMENTATION

The SystemTap access performance monitoring hardware is planned to be
built on the perfmon2 kernel support. The perfom2 provides reservation
and access to the performance monitoring hardware on ia64, i386, and
PowerPC processors. The perfmon2 support is not yet in the upstream
kernels, but patches are available.

Outline where things are done.

	In Translator:
	   group all probe kernel.perfmon.sample() together

	In perfmon tapset:
	   perfmon_allocate_counter()
	   perfmon_free_counter()
	   perfmon_create_context()
	   perfmon_get_counter()
	   perfmon_start_counter()
	   perfmon_stop_counter()
	   perfmon_handle_to_string()


	On startup (probe begin):
	   if perfmon.sample used, register perfmon custom buffer mechanism
	   The following steps will need some work done in userspace (libpfm):
	   -translate each of the perfmon_allocate_counter into perfmon config
	   -set up the perfmon contexts (either per processor or per pid)
	   -activate the perfmon contexts

	On shutdown (probe end):
	   The following steps will need some work done in userspace (libpfm):
	   -destroy the perfmon contexts
	   -if perfmon.sample used, unregister perfmon custom buffer mechanism

FIXME more details on the proposed implementation.



SYSTEMTAP PERFMON ISSUES

-There are numerous constraints on event setup. It is possible to
 request a configuration that cannot be set up in the performance
 monitoring hardware.

-This mechanism does not provide access to other related information
 provided by the performance monitoring hardware, e.g. the performance
 monitoring registers storing the data address tha caused a cache miss
 on ia64.

-The perfmon clones the context for new threads that have the perfmon
 context set up, but we probably do not want to attach to each
 existing thread and set up the context on it. That is going to be
 relatively expensive.

-Perfmon can either do global or per thread monitoring, but they
 cannot be mixed.


REFERENCES


Stephane Eranian, The perfmon2 interface specification
HP Laboratories, HPL-2004-200(R.1), February 7, 2005.
http://www.hpl.hp.com/techreports/2004/HPL-2004-200R1.html