From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 779 invoked by alias); 1 Feb 2008 22:15:09 -0000 Received: (qmail 765 invoked by uid 22791); 1 Feb 2008 22:15:08 -0000 X-Spam-Status: No, hits=-0.7 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: sourceware.org Received: from agminet01.oracle.com (HELO agminet01.oracle.com) (141.146.126.228) by sourceware.org (qpsmtpd/0.31) with ESMTP; Fri, 01 Feb 2008 22:14:37 +0000 Received: from rgmgw1.us.oracle.com (rgmgw1.us.oracle.com [138.1.186.110]) by agminet01.oracle.com (Switch-3.2.4/Switch-3.1.7) with ESMTP id m11MDsrM007017; Fri, 1 Feb 2008 16:13:55 -0600 Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rgmgw1.us.oracle.com (Switch-3.2.4/Switch-3.2.4) with ESMTP id m11MDqdc003632; Fri, 1 Feb 2008 15:13:52 -0700 Received: from c-76-19-28-43.hsd1.ma.comcast.net by acsmt359.oracle.com with ESMTP id 7169749501201903990; Fri, 01 Feb 2008 16:13:10 -0600 Message-ID: <47A397AD.9090602@oracle.com> Date: Fri, 01 Feb 2008 22:15:00 -0000 From: Elena Zannoni Organization: Oracle USA Inc. User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Andrew Cagney CC: systemtap@sourceware.org, frysk Subject: Re: my notes from the tracing workshop References: <47A34AA2.5070404@redhat.com> In-Reply-To: <47A34AA2.5070404@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: AAAAAQAAAAI= X-Brightmail-Tracker: AAAAAQAAAAI= X-Whitelist: TRUE X-Whitelist: TRUE X-IsSubscribed: yes Mailing-List: contact frysk-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: frysk-owner@sourceware.org X-SW-Source: 2008-q1/txt/msg00049.txt.bz2 Thanks for the notes, Andrew. Good summary. I posted mine here, including my slides: http://blogs.oracle.com/ezannoni/ elena Andrew Cagney wrote: > [The slides get published next week] > > > Overview > > The underlying goal of the workshop was to gather information on the > current state of tracing and monitoring technology, and identify areas > of potential research and development. The Canadian Government is > looking to significantly further research in this area; and is > preparing a report. > > Broadly the talks had an embedded bent, which isn't surprising given > its organizational origins in the telco industry. There was a wide > level of representation though with both large system, and deeply > embedded viewpoints being presented. > > > The Technology > > For most talks, the assumed approach was > > -> -> -> $LOG > > then on the host; or in user land: > > $LOG -> -> "DB" -> > > so I'll talk to that. > > > Probes > > That there were two technology camps (modified kernel, and dynamic > probes), with the majority in the former group. Interestingly, the > embedded players strongly indicated that deploying the modified kernel > was acceptable (even advantageous) - the systems were permanently > running in flight-recorder mode so they were in a better position to > do postmortem analysis. > > The exceptions were SystemTAP and SensorPoint (Wind River) (and on the > edge, frysk). Both SystemTAP and SensorPoint and the same basic > approaches. SensorPoint did have a djprobe like mechanism working, > and nested(?) probes (where you could specify the call chain required > to trigger the probe - it worked by watching the functions and not by > looking at backtraces); finally the ability to replace code on live > systems. > > > Finaly, the big and positive thing on probes was that the kernel > markers being accepted. Oracle(Elena) identified that a lacking > feature was being able to query the list of possible probe points -> > embedding markers in the code (and hopefully having them documented in > situ ????) will address this. On the other hand, I picked up a few > concerns (outside of presentations): who gets to back port this (if at > all); its an ABI, who gets to maintain it long term; and what happens > when someone refuses to accept markers in their code :-) > > > Filters > > This is where SystemTAP and SensorPoint stood out (I think :-). Both > have the ability to filter events before pushing them to the > recorder. Using SystemTAP on the kernel markers should be a wicked > combination. > > [Can I assume that, when there's a marked up kernel, SystemTAP inserts > jumps instead of traps? If fche had been giving the talk, it would > have been my question :-)] > > > Recorders and logs > > Zzzzz. > > > Converters > > The consistent approach was to implement some sort of converter that > could load random external file formats and load them into an internal > form. > > While there seemed to be a push to standardize on log-file format, I > got the impression that it was solving the wrong problem (and others > two). Size really did matter. > > > "DB" > > There was a strong consensus that the "internal" format of the log > data needed to be a fast light weight database; two vendors were using > sqlite for instance (TPTP the eclipse tool didn't but I suspect will > shortly). Wind River presented a discussion illustrating its advantages. > > There were suggestions, and it appears a strong degree of consensus, > of standardizing a database format, so that could be shared amongst > visualization tools. I think this, and the conversion tools will > gather traction. Something SystemTAP should monitor. > > > Visualization. > > Many visualization tools were presented (if I see another useless > full-screen snap-shot in a slide I'll scream), most built on eclipse, > but a few were not. While this is a very crowded market, there seems, > in mnsho, to still be a need for clear simple visualization tools > backed by a databse. > > The quote of the day, in describing eclipse, has to be "icon diarrhea". > > > A few of the Talks > > Me / Red Hat: SystemTAP / Frysk > (I got to do both talks). > What's the status of SystemTAP on the ARM? Ditto for Frysk. > > Robert Winsiewski / IBM: Performance analys and debugging at IBM > It was as much about IBM as a few other companies Robert had worked > for; it have a general history of logging challenges in a number of > companies. Strongly in favor of the marker approach; and set that as > a theme. Two notable ideas were non-locked logging (the in-memory log > file format handled synchronization using atomic instructions); and > sharing memory logs between user and system. > > Elena Zannoni / Oracle: Tracing at Oracle > Presented the challenges with using SystemTAP in a "binary only / > clean room" environment. > > Beth Tibbits / IBM: Eclipse Parallel Tools Platform > Underneath they are using a consolidating process that then, in turn, > talks to a distributed collection of gdb processes (makes you cry :-); > this basic approach is described in Bevin Brett's paper on making > ladebug HPC. There's work to generalize this, see > http://scalabletools.org/ > > Andrew McDermott / Wind River: Developing OS-agnostic visualization > tools. > Discussed the "DB" approach for managing all that data. > > Felix Burton / Wind River: Sensorpoint Technology > Wind Rivers rough equivalent to SystemTAP. Use "C" for the probes. > > > -- > > I was asked if SystemTAP is supported on arm (have e-mail address if > fche you want to contact them). > >