RE: Meeting minutes 20060727

RE: Meeting minutes 20060727

[Stone, Joshua I]

On Thursday, August 03, 2006 12:43 PM, Jose R. Santos wrote:
> Frank Ch. Eigler wrote:
>> "Jose R. Santos" <jrs@us.ibm.com> writes:
>> 
>>> [...]  One of the things we are currently working on is to have an
>>> alternative to gettimeofday.  [...]
>> 
>> Josh Stone's kprobes-safe rendition gettimeofday has been in the
>> systemtap runtime for some time now.  Is that not sufficient for
>> your purposes? 
>> 
> 
> The goal here is to improve performance.  Gui Jian has timed
> _stp_gettimeofday_us() using LKET and it give a bigger overhead than
> do_gettimeofday().  By doing this in user space we can obtain better
> performance by avoiding disabling preemption or calling seqlock like
> _stp_gettimeofday does.

If you have suggestions for improving _stp_gettimeofday, I'm all ears.
I'm never of the mind that the  existing code is the perfect solution.

But then, if you're doing something from userspace, I guess I don't
fully understand your usage model, or how that will be faster...


Josh

Re: Meeting minutes 20060727

[Jose R. Santos]

Stone, Joshua I wrote:
> On Thursday, August 03, 2006 12:43 PM, Jose R. Santos wrote:
> > Frank Ch. Eigler wrote:
> >> "Jose R. Santos" <jrs@us.ibm.com> writes:
> >> 
> >>> [...]  One of the things we are currently working on is to have an
> >>> alternative to gettimeofday.  [...]
> >> 
> >> Josh Stone's kprobes-safe rendition gettimeofday has been in the
> >> systemtap runtime for some time now.  Is that not sufficient for
> >> your purposes? 
> >> 
> > 
> > The goal here is to improve performance.  Gui Jian has timed
> > _stp_gettimeofday_us() using LKET and it give a bigger overhead than
> > do_gettimeofday().  By doing this in user space we can obtain better
> > performance by avoiding disabling preemption or calling seqlock like
> > _stp_gettimeofday does.
>
> If you have suggestions for improving _stp_gettimeofday, I'm all ears.
> I'm never of the mind that the  existing code is the perfect solution.
>
> But then, if you're doing something from userspace, I guess I don't
> fully understand your usage model, or how that will be faster...
>
>   

Is not that calculating something similar to gettimeofday in user space 
will be faster that doing it with in the kernel.  My point is that 
printing raw cycles will have less of an impact than the calculations 
needed for gettimeofday.  Now, the algorithm used to do the calculations 
in user space maybe 10 times slower that the one on the kernel, but at 
that point I don't really care what the overhead of the algorithm is 
because that overhead is not reflected at all on the trace data.

I'm not saying that the the performance of _stp_gettimeofday is 
horrible, but since trace data needs to be processed in user space 
anyway, we can reduce some of the overhead cause while gathering data by 
offloading these calculations at a later time. 

-JRS

Re: Meeting minutes 20060727

[Frank Ch. Eigler]

Hi -

On Thu, Aug 03, 2006 at 03:49:42PM -0500, Jose R. Santos wrote:

> [...]  My point is that printing raw cycles will have less of an
> impact than the calculations needed for gettimeofday.  [...]

Nothing is keeping you from tracing the value of get_cycles() instead
of a higher level figure.

- FChE

Re: Meeting minutes 20060727

[Vara Prasad]

Stone, Joshua I wrote:

>On Thursday, August 03, 2006 12:43 PM, Jose R. Santos wrote:
>  
>
>>Frank Ch. Eigler wrote:
>>    
>>
>>>"Jose R. Santos" <jrs@us.ibm.com> writes:
>>>
>>>      
>>>
>>>>[...]  One of the things we are currently working on is to have an
>>>>alternative to gettimeofday.  [...]
>>>>        
>>>>
>>>Josh Stone's kprobes-safe rendition gettimeofday has been in the
>>>systemtap runtime for some time now.  Is that not sufficient for
>>>your purposes? 
>>>
>>>      
>>>
>>The goal here is to improve performance.  Gui Jian has timed
>>_stp_gettimeofday_us() using LKET and it give a bigger overhead than
>>do_gettimeofday().  By doing this in user space we can obtain better
>>performance by avoiding disabling preemption or calling seqlock like
>>_stp_gettimeofday does.
>>    
>>
>
>If you have suggestions for improving _stp_gettimeofday, I'm all ears.
>I'm never of the mind that the  existing code is the perfect solution.
>
>But then, if you're doing something from userspace, I guess I don't
>fully understand your usage model, or how that will be faster...
>
>
>Josh
>  
>

During one of the BOF's in OLS the discussion of do_gettimeofday() 
overhead came, everyone agreed it is slow. Someone suggested that Linux 
now has more efficient interface to get the timestamp, my notes seems to 
lack details, does anyone remember what was the alternative suggestion. 
I vaguely recollect that either real time guys or the high resolution 
timer folks have introduced an interface which is platform dependent but 
more efficient. If anyone can recollect details of this please post it.

bye,
Vara Prasad

Re: Meeting minutes 20060727

[Frank Ch. Eigler]

Hi -

On Thu, Aug 03, 2006 at 05:22:46PM -0700, Vara Prasad wrote:
> During one of the BOF's in OLS the discussion of do_gettimeofday() 
> overhead came, everyone agreed it is slow.

For what it's worth, systemtap hasn't used do_gettimeofday proper in
over a month, but for reasons of safety rather than performance.

> Someone suggested that Linux now has more efficient interface to get
> the timestamp, my notes seems to lack details, does anyone remember
> what was the alternative suggestion.  [...]

I only heard of the hrtimer API in more recent kernels.

- FChE

Re: Meeting minutes 20060727

[Jose R. Santos]

Frank Ch. Eigler wrote:
> Hi -
>
> On Thu, Aug 03, 2006 at 03:49:42PM -0500, Jose R. Santos wrote:
>
> > [...]  My point is that printing raw cycles will have less of an
> > impact than the calculations needed for gettimeofday.  [...]
>
> Nothing is keeping you from tracing the value of get_cycles() instead
> of a higher level figure.
>
> - FChE
>   
Actually, in the email that was directed at Masami-san I stated two 
scenarios that prevent us just using get_cycles().  One is processors 
that support frequency scaling and the second one is virtulized 
environments like Xen or Power5 micro partitions.  This was something 
that Masami-san and myself were discussing at OLS, as one of his main 
concerns with LKET the performance impact of do_gettimeofday() as a 
timestamp mechanism.  The things that I wrote on my email to Masami-san 
were a continuation of those discussion we had at OLS in an effort to 
standardize the tracing mechanism.

-JRS

Re: Meeting minutes 20060727

[Masami Hiramatsu]

Hi Jose,

Jose R. Santos wrote:
> Actually, in the email that was directed at Masami-san I stated two
> scenarios that prevent us just using get_cycles().  One is processors
> that support frequency scaling and the second one is virtulized
> environments like Xen or Power5 micro partitions.  This was something
> that Masami-san and myself were discussing at OLS, as one of his main
> concerns with LKET the performance impact of do_gettimeofday() as a
> timestamp mechanism.  The things that I wrote on my email to Masami-san
> were a continuation of those discussion we had at OLS in an effort to
> standardize the tracing mechanism.

I think we can use get_cycles() even if the processor's frequency can be
changed, because we can re-calculate time by using the events which
records frequency changing. However, in the case of Xen, it would be more
difficult to record these kinds of events.

Thanks,

-- 
Masami HIRAMATSU
2nd Research Dept.
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Meeting minutes 20060727

[Roland McGrath]

sched_clock is probably a good thing to use.

Re: Meeting minutes 20060727

[Jose R. Santos]

Tim Bird wrote:
> Sorry for the late response.  I agree that LKST has some good
> post-processing programs.
>
> What do you propose to standardize between LKET and LKST?
> And what is the nature of the standardization?  (That is,
> what are you defining, and how would it be defined so that
> it would be re-usable between projects?)
>   

I think a good start would be for LKET to provide the same information 
that LKST provides with their hooks in order for their post-processing 
scripts to work with LKET.  The format does not have to be the same.  We 
could either get a converter program that transforms the LKET format 
over the something compatible with LKST or make LKST understand the LKET 
format.  As long as we can extract the same information with both tool, 
this should work.
> I think Hiramatsu-san said something about porting LKST
> probe points to SystemTap.  You mention the LKST post-processing
> tools.  I'm wondering how other projects can get involved in
> this standardization, if possible.
>   

Last time I talked with Hiramatsu-san he needed some approval in order 
to release the LKST source code.  I believe he is still working on it.  
One of the things that set LKET apart from LKST (and LTTng for that 
matter) is that the trace points are design to be modifiable so that the 
analysis is not limited by the tool.  Last time I check, LKST's format 
was a bit more restrictive in order to not compromise speed. Their are 
many advantages to all the different method for doing traces (the ones I 
am most familiar are with LKET, LKST and LTTng), but in my opinion, the 
real value of trace tools is not the trace mechanism it self but rather 
the post-processing capabilities than can be archived with the trace 
data.  I believe the best way to reach standardization between all the 
trace tools available is for all the tools to have a standard set of 
trace hooks.  This way we could allow all the post-processing tools to 
be usable between different trace utilities.


-JRS

Re: Meeting minutes 20060727

[Tim Bird]

Jose R. Santos wrote:
> After reviewing the code for both LTTng and LKST it seems like LKET
> shares more similarities with LKST than it does with LTTng (both in
> available trace hooks and general philosophy of the projects).  Another
> key advantage of the LKST projects is that the folks from Hitachi have a
> set of post-processing programs already available.  LTTv on the other
> hand does not seem to have that many useful post-processing plugin.
> 
> I would say that a good starting point would be to get LKET and LKST
> standardize as much as possible since I think they share the most in
> common.

Sorry for the late response.  I agree that LKST has some good
post-processing programs.

What do you propose to standardize between LKET and LKST?
And what is the nature of the standardization?  (That is,
what are you defining, and how would it be defined so that
it would be re-usable between projects?)

I think Hiramatsu-san said something about porting LKST
probe points to SystemTap.  You mention the LKST post-processing
tools.  I'm wondering how other projects can get involved in
this standardization, if possible.

Thanks,
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: Meeting minutes 20060727

[Gui,Jian]

Jose R. Santos writes:
> 
> [...] Gui Jian is currently working to implement this and his results
> are encouraging. [...]

Here are the results of different timing functions mentioned above.
I used a multi-thread(8 threads) app which calls getsid() in a loop
running on a 4-way ppc64 box.  The kernel is 2.6.17.7 and 
CONFIG_CPU_FREQ is set.

We tried to use different timing functions in LKET. The numbers are 
average microseconds for each getsid() in all cpus.

--------------
              (do_gettimeofday)  (get_cycles)  (_stp_gettimeofday_us)

original LKET:     4216         4045 (-4.05%)       4276 (+1.42%)

no data transfer:  3677         3526 (-4.10%)       3769 (+2.52%)

Re: Meeting minutes 20060727

[Jose R. Santos]

Frank Ch. Eigler wrote:
> "Jose R. Santos" <jrs@us.ibm.com> writes:
>
> > [...]  One of the things we are currently working on is to have an
> > alternative to gettimeofday.  [...]
>
> Josh Stone's kprobes-safe rendition gettimeofday has been in the
> systemtap runtime for some time now.  Is that not sufficient for
> your purposes?
>   

The goal here is to improve performance.  Gui Jian has timed 
_stp_gettimeofday_us() using LKET and it give a bigger overhead than 
do_gettimeofday().  By doing this in user space we can obtain better 
performance by avoiding disabling preemption or calling seqlock like 
_stp_gettimeofday does.

-JRS

Re: Meeting minutes 20060727

[Frank Ch. Eigler]

"Jose R. Santos" <jrs@us.ibm.com> writes:

> [...]  One of the things we are currently working on is to have an
> alternative to gettimeofday.  [...]

Josh Stone's kprobes-safe rendition gettimeofday has been in the
systemtap runtime for some time now.  Is that not sufficient for
your purposes?

- FChE

Re: Meeting minutes 20060727

[Jose R. Santos]

Masami Hiramatsu wrote:
>
> That's good to me.
> I had ported most of probing points of LKST to some tapsets against
> RHEL4U3. I'll release that code as soon as possible. It'll help you.
>   

I've looked at the probe points in LKST and it should be very easy to 
implement most of them using SystemTap.  Any code you can provide would 
help.
> LKST already has post-processing tools(Lkst Log Tools). "lkstla" is the
> core of these tools. Currently, this tool depends on LKST. But, I think
> its framework can be adaptable for many kind of traced data.
>
>   

We are currently focusing a lot on performance enhancements for LKET.  
One of the things we are currently working on is to have an alternative 
to gettimeofday.  In order to address processors that support frequency 
scaling, we plan to add an event hook every time there is a frequency 
change in order correctly tell time in user space.  Having the extra 
event hook is also useful for analysis.  Gui Jian is currently working 
to implement this and his results are encouraging.  We still plan to 
keep gettimeofday as an option for environments such as Xen or Power5 
micro-partitioning.

I know you were also playing around with performance enhancement to 
SystemTap for the purposes of tracing.  It would be good to see what you 
had in mind to see what thing we can come up with to serve both our 
tracing needs.

-JRS

Re: Meeting minutes 20060727

[Masami Hiramatsu]

Hi Jose,

Jose R. Santos wrote:
> Elena Zannoni wrote:
>> OLS action items: - static probes discussion
>> - standardization of tracing mechanism to happen on systemtap mailing
>> list
>>   
> Hi Folks,
> 
> After reviewing the code for both LTTng and LKST it seems like LKET
> shares more similarities with LKST than it does with LTTng (both in
> available trace hooks and general philosophy of the projects).  Another
> key advantage of the LKST projects is that the folks from Hitachi have a
> set of post-processing programs already available.  LTTv on the other
> hand does not seem to have that many useful post-processing plugin.
> 
> I would say that a good starting point would be to get LKET and LKST
> standardize as much as possible since I think they share the most in
> common.
> Comments?

That's good to me.
I had ported most of probing points of LKST to some tapsets against
RHEL4U3. I'll release that code as soon as possible. It'll help you.

LKST already has post-processing tools(Lkst Log Tools). "lkstla" is the
core of these tools. Currently, this tool depends on LKST. But, I think
its framework can be adaptable for many kind of traced data.

Thanks,

-- 
Masami HIRAMATSU
2nd Research Dept.
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Meeting minutes 20060727

[Jose R. Santos]

Elena Zannoni wrote:
> OLS action items: 
> - static probes discussion
> - standardization of tracing mechanism to happen on systemtap mailing list
>   
Hi Folks,

After reviewing the code for both LTTng and LKST it seems like LKET 
shares more similarities with LKST than it does with LTTng (both in 
available trace hooks and general philosophy of the projects).  Another 
key advantage of the LKST projects is that the folks from Hitachi have a 
set of post-processing programs already available.  LTTv on the other 
hand does not seem to have that many useful post-processing plugin.

I would say that a good starting point would be to get LKET and LKST 
standardize as much as possible since I think they share the most in 
common. 

Comments?

-JRS

Meeting minutes 20060727

[Elena Zannoni]

As you probably know, we have weekly phone meetings among
the various participants to the systemtap project.

I want to start posting the minutes so that we are a bit more
transparent about the development process, and what we are focusing
on.

Attendees:
RH: Elena, Will, David
Intel: Josh
IBM: Ananth, Hien, Prasanna, Dave, Mike
Hitachi: Satoshi

OLS action items: 
- static probes discussion
- standardization of tracing mechanism to happen on systemtap mailing list
- post weekly minutes to external mailing list


IBM: Check rawhide kernel for utrace ports. Verify that it's there and
that it is a base to start from for the userspace probes.

Will: discovered elfutils problem: make sure it doesn't occur on rhel4 u4.
Perfmon with valgrind.

David: worked on kernel patch that we need to handle modules and
saving stap probes for later. Discussed on internal rh kernel list.
remove duplicate functions and probe points bug: working on that.

Josh: Nothing to report.  Need to test the rawhide kernel with xen and
kprobes.

Satoshi: flight recorder. Masami has a demo, Will saw it. Masami
preparing demo for stap mailing list. Post sometimes this week or next
week.  Markup implementation: Masami has some suggestions he will post
in the next couple of weeks.

kprobes meeting: decided to postpone kprobe booster for preemptible
kernel. Very difficult to implement. Will be discussed on mailing
list.

Ananth: just got back from OLS, looking at user space
probes. Hopefully something about uprobes in the next 2 weeks.

Please somebody check that the s390 kprobes made into rawhide kernel: Dave.

Prasanna: working at watchpoint probes. Looking at utrace as well. Use
ptrace or a new set of uprobes interfaces.

Andy Whitcroft from IBM UK: started using systemtap, should we invite him
to a call. He has some feedback.

Rh bugzilla: 197220. 
http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=197220
Problems with creating kernel modules. Keep an eye on it.