From: Andrew Cagney <email@example.com>
Cc: frysk <firstname.lastname@example.org>
Subject: my notes from the tracing workshop
Date: Fri, 01 Feb 2008 16:37:00 -0000 [thread overview]
Message-ID: <47A34AA2.email@example.com> (raw)
[The slides get published next week]
The underlying goal of the workshop was to gather information on the
current state of tracing and monitoring technology, and identify areas
of potential research and development. The Canadian Government is
looking to significantly further research in this area; and is preparing
Broadly the talks had an embedded bent, which isn't surprising given its
organizational origins in the telco industry. There was a wide level of
representation though with both large system, and deeply embedded
viewpoints being presented.
For most talks, the assumed approach was
<probe> -> <filtering> -> <recorder> -> $LOG
then on the host; or in user land:
$LOG -> <converter> -> "DB" -> <visualization>
so I'll talk to that.
That there were two technology camps (modified kernel, and dynamic
probes), with the majority in the former group. Interestingly, the
embedded players strongly indicated that deploying the modified kernel
was acceptable (even advantageous) - the systems were permanently
running in flight-recorder mode so they were in a better position to do
The exceptions were SystemTAP and SensorPoint (Wind River) (and on the
edge, frysk). Both SystemTAP and SensorPoint and the same basic
approaches. SensorPoint did have a djprobe like mechanism working, and
nested(?) probes (where you could specify the call chain required to
trigger the probe - it worked by watching the functions and not by
looking at backtraces); finally the ability to replace code on live systems.
Finaly, the big and positive thing on probes was that the kernel markers
being accepted. Oracle(Elena) identified that a lacking feature was
being able to query the list of possible probe points -> embedding
markers in the code (and hopefully having them documented in situ ????)
will address this. On the other hand, I picked up a few concerns
(outside of presentations): who gets to back port this (if at all); its
an ABI, who gets to maintain it long term; and what happens when someone
refuses to accept markers in their code :-)
This is where SystemTAP and SensorPoint stood out (I think :-). Both
have the ability to filter events before pushing them to the recorder.
Using SystemTAP on the kernel markers should be a wicked combination.
[Can I assume that, when there's a marked up kernel, SystemTAP inserts
jumps instead of traps? If fche had been giving the talk, it would have
been my question :-)]
Recorders and logs
The consistent approach was to implement some sort of converter that
could load random external file formats and load them into an internal form.
While there seemed to be a push to standardize on log-file format, I got
the impression that it was solving the wrong problem (and others two).
Size really did matter.
There was a strong consensus that the "internal" format of the log data
needed to be a fast light weight database; two vendors were using sqlite
for instance (TPTP the eclipse tool didn't but I suspect will shortly).
Wind River presented a discussion illustrating its advantages.
There were suggestions, and it appears a strong degree of consensus, of
standardizing a database format, so that could be shared amongst
visualization tools. I think this, and the conversion tools will gather
traction. Something SystemTAP should monitor.
Many visualization tools were presented (if I see another useless
full-screen snap-shot in a slide I'll scream), most built on eclipse,
but a few were not. While this is a very crowded market, there seems,
in mnsho, to still be a need for clear simple visualization tools backed
by a databse.
The quote of the day, in describing eclipse, has to be "icon diarrhea".
A few of the Talks
Me / Red Hat: SystemTAP / Frysk
(I got to do both talks).
What's the status of SystemTAP on the ARM? Ditto for Frysk.
Robert Winsiewski / IBM: Performance analys and debugging at IBM
It was as much about IBM as a few other companies Robert had worked for;
it have a general history of logging challenges in a number of
companies. Strongly in favor of the marker approach; and set that as a
theme. Two notable ideas were non-locked logging (the in-memory log
file format handled synchronization using atomic instructions); and
sharing memory logs between user and system.
Elena Zannoni / Oracle: Tracing at Oracle
Presented the challenges with using SystemTAP in a "binary only / clean
Beth Tibbits / IBM: Eclipse Parallel Tools Platform
Underneath they are using a consolidating process that then, in turn,
talks to a distributed collection of gdb processes (makes you cry :-);
this basic approach is described in Bevin Brett's paper on making
ladebug HPC. There's work to generalize this, see http://scalabletools.org/
Andrew McDermott / Wind River: Developing OS-agnostic visualization tools.
Discussed the "DB" approach for managing all that data.
Felix Burton / Wind River: Sensorpoint Technology
Wind Rivers rough equivalent to SystemTAP. Use "C" for the probes.
I was asked if SystemTAP is supported on arm (have e-mail address if
fche you want to contact them).
next reply other threads:[~2008-02-01 16:37 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-02-01 16:37 Andrew Cagney [this message]
2008-02-01 22:15 ` Elena Zannoni
2008-02-05 20:37 ` William Cohen
2008-03-03 16:57 ` Andrew Cagney
[not found] <47A34AA2.firstname.lastname@example.org>
2008-02-01 19:44 ` Frank Ch. Eigler
2008-02-05 19:02 ` Andrew Cagney
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).