Re: [pcp] suitability of PCP for event tracing

[Mark Goodwin <mgoodwin@redhat.com>]

On 09/01/2010 05:49 AM, Frank Ch. Eigler wrote:
> Hi -
>
> Thanks, Nathan, Ken, Greg, Mark, for clarifying the status quo and
> some of the history.
>
> We understand that the two problem domains are traditionally handled
> with the event-tracing -vs- stats-monitoring distinction.  We're trying
> to see where best to focus efforts to make some small steps to bridge
> the two, where plenty of compromises are possible.  We'd prefer to
> help build on an existing project with a nice community than to do new
> stuff.

yes certainly :)

> For the poll-based data gathering issue, a couple of approaches came up:
>
> (1) bypassing pmcd and generating an pmarchive file directly from
>      trace data This appears to imply continuing the archive-vs-live
>      dichotomy that makes it difficult for clients to process both
>      recent and current data seamlessly together.

one of the issues with the live vrs archive dichotomy is that live
data is always available (since you're requesting it explicitly from
a PMDA that is otherwise passive), whereas the archive data is not
available unless configured to be collected before-hand (see pmlogger).
There is too much data to collect everything all the time - it's too
impracticable and intrusive, so some form of filtering and/or aggregation
needs to be done (see pmlogsummary, and Greg's old project too).

> Since using such
>      files would probably also need a custom client, then we'd not be
>      using much of the pcp infrastructure, only as a passive data
>      encoding layer.  This may not be worthwhile.
>
> (2) protocol extensions for live-push on pmda and pmcd-client interfaces
>      This clearly larger effort is only worth undertaking with the
>      community's sympathy and assistance.  It might have some
>      interesting integration possibilities with the other tools,
>      espectially pmie (the inference engine).

yep - I suspect Ken and maybe Nathan would have further comments on this

>
> For the static-pmns issue, the possibility of dynamic instance
> domains, metric subspaces is probably sufficient, if the event
> parameters are limited to only 1-2 degrees of freedom.  (In contrast,
> imagine browsing a trace of NFS or kernel VFS operations; these have
> ~5 parameters.)

PCP instance domains are traditionally single dimensional, though there
are a few exceptions such as kernel.percpu.interrupts. It's easy enough
to split multi-dimensional data structures out into multiple metrics with
a common instance domain.

> For the scalar-payloads issue, the BLOB/STRING metric types are indeed
> available but are opaque to other tools, so don't compose well.  Would
> you consider one additional data type, something like a JSON[1]
> string?  It would be self-describing, with pmie and general processing
> opportunities, though those numbers would lack the PMDA_PMUNITS
> dimensioning.

this could work using string or binary blob data types in the
existing protocols - though there is a size limit. And one of
the blessed features of PCP is the client monitoring tools can
more or less monitor any metrics - so any solution here would
also need specially crafted client tools. Extensions to the perl
binding would probably work best, e.g. interface with perl-JASON-*

> For the filtering issue, pmStore() is an interesting possibility,
> allowing the PMDAs to bear the brunt.  OTOH, if pmcd evolved into a
> data-push-capable widget, it could serve as a filtering proxy,
> requiring separate API or interpretation of the pmStore data.

well pmcd is already data-push capable using the pmstore interface,
allowing clients to store values for certain metrics in some of
the PMDAs. Filtering and parsing is done by the PMDA itself and
pmcd just acts as a proxy passthru (kind of a back-channel to
the pull interface).

pmstore hasn't really been used in anger like this though - more just
for setting config & control options and the like. The same (or similar)
protocol has also been used for a data source to open a socket directly
to a PMDA and tie into the PMDA's select loop, rather than going via pmcd.

>
> For the web-based frontend issue, yeah, javascript+svg+etc. sounds
> most promising, especially if it can be made to speak the native wire
> protocol to pmdc.  This would seem to argue for a stateful
> archive-serving pmdc, or perhaps a archive-serving proxy, as in Greg's
> old project.

Time averaging, aggregation and filtering were all ambitious aims
of the project Greg's talking about - I wonder if that code could
ever be resurrected and open sourced? One abomination here was
that a PMDA could also be a client - and potentially query itself
for metrics(!)

> Is this sounding reasonable?
>

it's going to take a lot more discussion, but enthusiasm seems to
be on our side :)

Cheers
-- Mark Goodwin