Re: systemtap/pcp integration

[Nathan Scott <nathans@redhat.com>]

----- Original Message -----
> [...]
> Yes, but the systemtap side of this would be pretty trivial.  Printing
> JSON strings is easy.

Thats only a small part of the problem though (even just considering
the systemtap side of things), and there are many advantages to the MMV
format being so close to the representation the PMDA needs to export.

> >  (for ... reasons?)
> 
> Other messages in the thread pointed out some reasons

Those references were a little too vague for me to follow the intent.

> - ability to generalize to event traffic, not just sampled metrics

That's not really a reason to start over though - the MMV format is
versioned, it could easily be extended and all its many advantages
kept.  Or, a separate PMDA might make a better option there - there
is no reason one PMDA needs to service both these (very different)
tracing/sampling data models.

Anyway, do keep in mind the oft-mentioned cost of string conversions
(Ken mentioned on the last call, IIRC, David mentioned his concerns
originally, and I will say it again too) - every time we've profiled
pmdalinux, the CPU cost is always dominated by the string extraction/
conversion code for the procfs/sysfs files.

That will be orders of magnitude worse for event tracing, I imagine.

> - robustness, by avoiding fragile C code

Do you mean Davids systemtap code?  The MMV code in PCP has been around
for quite some time, and I know of a number of production environments
using it 24x7 on hundreds of machines, for many years - its not fragile
at all.  Stability in Davids code will come over time, if there really
is an issue there (?) but experience would suggest there is no inherent
flaw in the approach he's using.

> So the proposal is to think about a single general JSON PMDA that can
> be configured to bridge data from multiple JSON-emitting applications,
> including systemtap scripts.

Bridges like this have been implemented several times in the past and it
has never worked well.  The fundamental, unsolvable-in-the-general-case
problem is that we have well-defined metric semantics in PCP.  So these
bridges have to (attempt to) map from the usually-far-less-so,if-defined
-at-all semantics offered by the other tool, and come up with the highly
detailed PCP metric metadata, somehow.  To date custom PMDAs have always
done a better job - more simply-coded, and with domain-specific-code, as
designed for PCP PMDAs (we can still use JSON there of course, as in the
elasticsearch case).


FWIW, I chatted to someone last week who was actually using Zabbix in a
production environment - they were interested in exploring the opposite
direction to what you've described above.  IOW, they would like to have
a PCP component that talks Zabbix protocol (JSON-alike, IIRC) such that
they wouldn't have to install the Zabbix agent code everywhere.  They'd
like instead to rely on a PCP component (could pmwebd be taught to fill
this role, perhaps? dunno) giving them everything they need for all the
many machines they rollout, cookie-cutter style, from PCP.  Then, their
central Zabbix instance could continue doing their alerting, which they
like.  It sounded a little bit like the graphana/graphite exporting, and
I believe its also JSON-based, so just thought I'd mention it here.

cheers.

--
Nathan