public inbox for
 help / color / mirror / Atom feed
From: Elena Zannoni <>
To: Andrew Cagney <>
Cc:, frysk <>
Subject: Re: my notes from the tracing workshop
Date: Fri, 01 Feb 2008 22:15:00 -0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

Thanks for the notes, Andrew. Good summary.
I posted mine here, including my slides:


Andrew Cagney wrote:
> [The slides get published next week]
> Overview
> The underlying goal of the workshop was to gather information on the 
> current state of tracing and monitoring technology, and identify areas 
> of potential research and development.  The Canadian Government is 
> looking to significantly further research in this area; and is 
> preparing a report.
> Broadly the talks had an embedded bent, which isn't surprising given 
> its organizational origins in the telco industry.  There was a wide 
> level of representation though with both large system, and deeply 
> embedded viewpoints being presented.
> The Technology
> For most talks, the assumed approach was
>    <probe> -> <filtering> -> <recorder> -> $LOG
> then on the host; or in user land:
>    $LOG -> <converter> -> "DB" -> <visualization>
> so I'll talk to that.
> Probes
> That there were two technology camps (modified kernel, and dynamic 
> probes), with the majority in the former group.  Interestingly, the 
> embedded players strongly indicated that deploying the modified kernel 
> was acceptable (even advantageous) - the systems were permanently 
> running in flight-recorder mode so they were in a better position to 
> do postmortem analysis.
> The exceptions were SystemTAP and SensorPoint (Wind River) (and on the 
> edge, frysk).  Both SystemTAP and SensorPoint and the same basic 
> approaches.  SensorPoint did have a djprobe like mechanism working, 
> and nested(?) probes (where you could specify the call chain required 
> to trigger the probe - it worked by watching the functions and not by 
> looking at backtraces); finally the ability to replace code on live 
> systems.
> Finaly, the big and positive thing on probes was that the kernel 
> markers being accepted.  Oracle(Elena) identified that a lacking 
> feature was being able to query the list of possible probe points -> 
> embedding markers in the code (and hopefully having them documented in 
> situ ????) will address this.  On the other hand, I picked up a few 
> concerns (outside of presentations): who gets to back port this (if at 
> all); its an ABI, who gets to maintain it long term; and what happens 
> when someone refuses to accept markers in their code :-)
> Filters
> This is where SystemTAP and SensorPoint stood out (I think :-).  Both 
> have the ability to filter events before pushing them to the 
> recorder.  Using SystemTAP on the kernel markers should be a wicked 
> combination.
> [Can I assume that, when there's a marked up kernel, SystemTAP inserts 
> jumps instead of traps?  If fche had been giving the talk, it would 
> have been my question :-)]
> Recorders and logs
> Zzzzz.
> Converters
> The consistent approach was to implement some sort of converter that 
> could load random external file formats and load them into an internal 
> form.
> While there seemed to be a push to standardize on log-file format, I 
> got the impression that it was solving the wrong problem (and others 
> two).  Size really did matter.
> "DB"
> There was a strong consensus that the "internal" format of the log 
> data needed to be a fast light weight database; two vendors were using 
> sqlite for instance (TPTP the eclipse tool didn't but I suspect will 
> shortly).  Wind River presented a discussion illustrating its advantages.
> There were suggestions, and it appears a strong degree of consensus, 
> of standardizing a database format, so that could be shared amongst 
> visualization tools.  I think this, and the conversion tools will 
> gather traction.  Something SystemTAP should monitor.
> Visualization.
> Many visualization tools were presented (if I see another useless 
> full-screen snap-shot in a slide I'll scream), most built on eclipse, 
> but a few were not.  While this is a very crowded market, there seems, 
> in mnsho, to still be a need for clear simple visualization tools 
> backed by a databse.
> The quote of the day, in describing eclipse, has to be "icon diarrhea".
> A few of the Talks
> Me / Red Hat: SystemTAP / Frysk
> (I got to do both talks).
> What's the status of SystemTAP on the ARM?  Ditto for Frysk.
> Robert Winsiewski / IBM: Performance analys and debugging at IBM
> It was as much about IBM as a few other companies Robert had worked 
> for; it have a general history of logging challenges in a number of 
> companies.  Strongly in favor of the marker approach; and set that as 
> a theme.  Two notable ideas were non-locked logging (the in-memory log 
> file format handled synchronization using atomic instructions); and 
> sharing memory logs between user and system.
> Elena Zannoni / Oracle: Tracing at Oracle
> Presented the challenges with using SystemTAP in a "binary only / 
> clean room" environment.
> Beth Tibbits / IBM: Eclipse Parallel Tools Platform
> Underneath they are using a consolidating process that then, in turn, 
> talks to a distributed collection of gdb processes (makes you cry :-); 
> this basic approach is described in Bevin Brett's paper on making 
> ladebug HPC.  There's work to generalize this, see 
> Andrew McDermott / Wind River: Developing OS-agnostic visualization 
> tools.
> Discussed the "DB" approach for managing all that data.
> Felix Burton / Wind River: Sensorpoint Technology
> Wind Rivers rough equivalent to SystemTAP.  Use "C" for the probes.
> -- 
> I was asked if SystemTAP is supported on arm (have e-mail address if 
> fche you want to contact them).

  reply	other threads:[~2008-02-01 22:15 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-01 16:37 Andrew Cagney
2008-02-01 22:15 ` Elena Zannoni [this message]
2008-02-05 20:37 ` William Cohen
2008-03-03 16:57   ` Andrew Cagney
     [not found] <47A34AA2.5070404__28393.9727153212$1201883893$gmane$>
2008-02-01 19:44 ` Frank Ch. Eigler
2008-02-05 19:02   ` Andrew Cagney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).