From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 17192 invoked by alias); 24 Jul 2014 07:40:47 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 17184 invoked by uid 89); 24 Jul 2014 07:40:46 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-4.0 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mx3-phx2.redhat.com Received: from mx3-phx2.redhat.com (HELO mx3-phx2.redhat.com) (209.132.183.24) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 24 Jul 2014 07:40:44 +0000 Received: from zmail20.collab.prod.int.phx2.redhat.com (zmail20.collab.prod.int.phx2.redhat.com [10.5.83.23]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s6O7egcb005755; Thu, 24 Jul 2014 03:40:42 -0400 Date: Thu, 24 Jul 2014 07:40:00 -0000 From: Nathan Scott Reply-To: Nathan Scott To: "Frank Ch. Eigler" , David Smith Cc: pcp@oss.sgi.com, Systemtap List Message-ID: <686315886.17105377.1406187642603.JavaMail.zimbra@redhat.com> In-Reply-To: References: <53C83CB9.3020808@redhat.com> <861139755.14608867.1405992742567.JavaMail.zimbra@redhat.com> <53CE7A2E.6010306@redhat.com> <181878137.16180790.1406111368167.JavaMail.zimbra@redhat.com> Subject: Re: systemtap/pcp integration MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SW-Source: 2014-q3/txt/msg00078.txt.bz2 ----- Original Message ----- > [...] > Yes, but the systemtap side of this would be pretty trivial. Printing > JSON strings is easy. Thats only a small part of the problem though (even just considering the systemtap side of things), and there are many advantages to the MMV format being so close to the representation the PMDA needs to export. > > (for ... reasons?) > > Other messages in the thread pointed out some reasons Those references were a little too vague for me to follow the intent. > - ability to generalize to event traffic, not just sampled metrics That's not really a reason to start over though - the MMV format is versioned, it could easily be extended and all its many advantages kept. Or, a separate PMDA might make a better option there - there is no reason one PMDA needs to service both these (very different) tracing/sampling data models. Anyway, do keep in mind the oft-mentioned cost of string conversions (Ken mentioned on the last call, IIRC, David mentioned his concerns originally, and I will say it again too) - every time we've profiled pmdalinux, the CPU cost is always dominated by the string extraction/ conversion code for the procfs/sysfs files. That will be orders of magnitude worse for event tracing, I imagine. > - robustness, by avoiding fragile C code Do you mean Davids systemtap code? The MMV code in PCP has been around for quite some time, and I know of a number of production environments using it 24x7 on hundreds of machines, for many years - its not fragile at all. Stability in Davids code will come over time, if there really is an issue there (?) but experience would suggest there is no inherent flaw in the approach he's using. > So the proposal is to think about a single general JSON PMDA that can > be configured to bridge data from multiple JSON-emitting applications, > including systemtap scripts. Bridges like this have been implemented several times in the past and it has never worked well. The fundamental, unsolvable-in-the-general-case problem is that we have well-defined metric semantics in PCP. So these bridges have to (attempt to) map from the usually-far-less-so,if-defined -at-all semantics offered by the other tool, and come up with the highly detailed PCP metric metadata, somehow. To date custom PMDAs have always done a better job - more simply-coded, and with domain-specific-code, as designed for PCP PMDAs (we can still use JSON there of course, as in the elasticsearch case). FWIW, I chatted to someone last week who was actually using Zabbix in a production environment - they were interested in exploring the opposite direction to what you've described above. IOW, they would like to have a PCP component that talks Zabbix protocol (JSON-alike, IIRC) such that they wouldn't have to install the Zabbix agent code everywhere. They'd like instead to rely on a PCP component (could pmwebd be taught to fill this role, perhaps? dunno) giving them everything they need for all the many machines they rollout, cookie-cutter style, from PCP. Then, their central Zabbix instance could continue doing their alerting, which they like. It sounded a little bit like the graphana/graphite exporting, and I believe its also JSON-based, so just thought I'd mention it here. cheers. -- Nathan