JSON PMDA

JSON PMDA

[David Smith]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

Here's a pointer to the latest version of the JSON PMDA I'm working on.
Note that the JSON PMDA is now much more generic, and (in theory) should
support lots of JSON data sources, so I've renamed it from 'stap_json'
to just 'json'. I've used it with systemtap
(https://sourceware.org/systemtap/) and ceph (http://ceph.com/).

Here's how the JSON PMDA works. When adding a new JSON data source, you
write a JSON "metadata" file that maps a metric to the JSON that
contains the metric data via a JSON pointers. Here's an excerpt from a
ceph metadata file:

====
{
  "data-exec": "/usr/bin/ceph --admin-daemon
/var/run/ceph/ceph-osd.0.asok perf
dump",
  "metrics": [
    {
      "name": "WBThrottle.bytes_dirtied",
      "pointer": "/WBThrottle/bytes_dirtied",
      "type": "integer"
    },
    {
      "name": "WBThrottle.bytes_wb",
      "pointer": "/WBThrottle/bytes_wb",
      "type": "integer"
    },
    ... more metrics here ...
   ]
}
====

In this case the optional 'data-exec' item tells the PMDA what command
to run to get the JSON data, then the 'metrics' item describes the data.

The JSON PMDA is in the 'dsmith/dev' branch of pcpfans.git. You can see
it here:

<https://www.sourceware.org/git/gitweb.cgi?p=pcpfans.git;a=tree;f=src/pmdas/json;h=d62facaf3f1b0972ba783852b59e4e4e9eb5cc04;hb=refs/heads/dsmith/dev>

As I mentioned earlier, I've used the JSON PMDA to provide metrics from
2 different data sources:

1) Ceph support. For ceph, the json pmda directory also contains a
python script called 'generate_ceph_metadata.py' that when given a ceph
admin socket name, will produce an associated metadata file. Then the
JSON PMDA config file would need to be pointed at the directory
containing the metadata file. Then pcp will use the JSON PMDA to be able
to provide ceph metrics.

2) Systemtap support. For systemtap, I've written a JSON tapset that
provides scripts with the ability to export JSON data. The tapset is
checked into the dsmith/json systemtap git branch or can be seen here
(with its associated macros file):

<https://www.sourceware.org/git/gitweb.cgi?p=systemtap.git;a=blob;f=tapset/linux/json.stp;h=b54a83aabb6ec362c946a5811be35def5f349957;hb=refs/heads/dsmith/json>
<https://www.sourceware.org/git/gitweb.cgi?p=systemtap.git;a=blob;f=tapset/linux/json.stpm;h=6f35a6e187f5ce67cd6a62f22a9d6b6623f56bc6;hb=refs/heads/dsmith/json>

I've also attached 2 sample systemtap scripts:

- net_xmit_json.stp: A script written by Will Cohen to monitor network
transmit counts and latency

- netfilter_summary_json.stp: I took an existing systemtap sample script
(netfilter_summary.stp) and converted it to output JSON instead.


What's next? I could certainly use any suggestions on the JSON PMDA.
Here's a short to-do list:

- Add support to the PMDA for floating point values (right now any
floating point ceph values are skipped).
- Debug an issue with the PMDA where the command to get the ceph data is
run too often.
- Try this with other JSON data sources. Suggestions welcome.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

[-- Attachment #2: netfilter_summary_json.stp --]
[-- Type: text/plain, Size: 1075 bytes --]

#! /usr/bin/env stap

global packets

// Set up the metrics
probe begin
{
  // Set the prefix to be used instead of the module name (optional).
  json_set_prefix("netfilter")

  // Add the metrics
  json_add_array("netfilter_data",
		 "Network data indexed by source and destination addresses.")
  json_add_array_numeric_metric("netfilter_data", "packets",
				"Number of packets transferred.", "")
  json_add_array_numeric_metric("netfilter_data", "bytes","Bytes transferred.",
				"bytes")
}

probe json_data
{
  @json_output_data_start

  foreach ([saddr, daddr] in packets-) {
    index = sprintf("%15s --> %15s", saddr, daddr)
    @json_output_array_numeric_value("netfilter_data", index, "packets",
				     @count(packets[saddr,daddr]))
    @json_output_array_numeric_value("netfilter_data", index, "bytes",
				     @sum(packets[saddr,daddr]))
  }
  @json_output_data_end
}

probe netfilter.ipv4.pre_routing {
      // Using aggregates avoids contention from packets being sent in
      // parallel from different processors:
      packets[saddr, daddr] <<< length
}

[-- Attachment #3: net_xmit_json.stp --]
[-- Type: text/plain, Size: 1822 bytes --]

// This script tracks time between packet queue and xmit.
// The information is provided to userspace via procfs and are read
// using the JSON PCP PMDA.

global net_devices
global read_count

probe json_data
{
  @json_output_data_start

  foreach (dev in net_devices) {
    if (@count(skb_queue_t[dev])) {
      @json_output_array_numeric_value("net_xmit_data", dev, "xmit_count",
				       @sum(skb_queue_t[dev]))
      @json_output_array_numeric_value("net_xmit_data", dev, "xmit_latency",
				       @count(skb_queue_t[dev]))
    }
    else {
      @json_output_array_numeric_value("net_xmit_data", dev, "xmit_count", 0)
      @json_output_array_numeric_value("net_xmit_data", dev, "xmit_latency", 0)
    }
  }

  @json_output_data_end
}

// Set up the metrics
probe begin
{
  // fallback instance device "eth0" if none specified
  if (argc == 0) {
    argv[1] = "eth0"
    argc = 1
  }
  
  // remember all the network devices
  for (i = 1; i <= argc; i++) {
    dev = argv[i]
    net_devices[dev] = i - 1
  }

  // Set the prefix to be used instead of the module name.
  json_set_prefix("net_xmit")

  // Add the metrics
  json_add_array("net_xmit_data", "Network transmit data indexed by ethernet device")
  json_add_array_numeric_metric("net_xmit_data", "xmit_count", "number of packets for xmit device", "")
  json_add_array_numeric_metric("net_xmit_data", "xmit_latency", "sum of latency for xmit device", "")
}

// probes to track the information

global skb_queue_start_t, skb_queue_t

probe kernel.trace("net_dev_queue") {
  skb_queue_start_t[$skb] = gettimeofday_ns();
}

probe kernel.trace("net_dev_start_xmit"), kernel.trace("net_dev_xmit") {
  t = gettimeofday_ns();
  st = skb_queue_start_t[$skb]
  if (st){
    skb_queue_t[kernel_string($dev->name)] <<< t - st
    delete skb_queue_start_t[$skb]
  }
}

Re: JSON PMDA

[Frank Ch. Eigler]

dsmith wrote:


> Here's a pointer to the latest version of the JSON PMDA I'm working on. 
> [...]

Looks great!

> I've also attached 2 sample systemtap scripts:
>
> - net_xmit_json.stp: A script written by Will Cohen to monitor network
> transmit counts and latency
>
> - netfilter_summary_json.stp: I took an existing systemtap sample script
> (netfilter_summary.stp) and converted it to output JSON instead.

May I recommend also taking a peek at some of the other stap examples,
for consideration of json variantification?  For example, a script
collecting backtrace samples/profiles, I/O traces (kind of like the
last-100-frees glibc sample, but for some kernel tracepoints?), could
be neat, as it is hard to achieve in pcp without stap.
 

> [...]
> - Debug an issue with the PMDA where the command to get the ceph data is
> run too often.

(Do you need to run __load_[all_]json_data before an actual fetch for
metrics with the affected prefixes?  Loading/refreshing metadata is
bound to be less costly than loading actual data.)


> - Try this with other JSON data sources. Suggestions welcome.

How about ovirt?  It may need another python metadata-generator script
(to pass along hostname / authentication / vm-enumeration into the
data-exec command)
http://www.ovirt.org/REST-Api 
https://bugzilla.redhat.com/show_bug.cgi?id=1182275

How about jboss/wildfly?  Same need re. metadata-generation.
https://docs.jboss.org/author/display/WFLY8/The+HTTP+management+API?_sscc=t


- FChE

Re: JSON PMDA

[David Smith]

On 03/08/2015 04:25 PM, Frank Ch. Eigler wrote:
> dsmith wrote:
> 
>> Here's a pointer to the latest version of the JSON PMDA I'm working on. 
>> [...]
> 
> Looks great!

Thanks.

>> I've also attached 2 sample systemtap scripts:
>>
>> - net_xmit_json.stp: A script written by Will Cohen to monitor network
>> transmit counts and latency
>>
>> - netfilter_summary_json.stp: I took an existing systemtap sample script
>> (netfilter_summary.stp) and converted it to output JSON instead.
> 
> May I recommend also taking a peek at some of the other stap examples,
> for consideration of json variantification?  For example, a script
> collecting backtrace samples/profiles, I/O traces (kind of like the
> last-100-frees glibc sample, but for some kernel tracepoints?), could
> be neat, as it is hard to achieve in pcp without stap.

I'll look into it.

>> [...]
>> - Debug an issue with the PMDA where the command to get the ceph data is
>> run too often.
> 
> (Do you need to run __load_[all_]json_data before an actual fetch for
> metrics with the affected prefixes?  Loading/refreshing metadata is
> bound to be less costly than loading actual data.)

The __refresh_metrics callback is supposed to help fetch performance by
getting called once, then a bunch of fetch operations will follow. The
theory is that you do all your data collection in the __refresh_metrics
callback, then the fetches just query the data you've already collected.

I think what is happening is that __refresh_metrics is getting called
before *every* fetch, although I can't swear to that yet.

In addition the code could certainly get smarter, and not refresh every
JSON data source when only one is being queried.

>> - Try this with other JSON data sources. Suggestions welcome.
> 
> How about ovirt?  It may need another python metadata-generator script
> (to pass along hostname / authentication / vm-enumeration into the
> data-exec command)
> http://www.ovirt.org/REST-Api 
> https://bugzilla.redhat.com/show_bug.cgi?id=1182275
> 
> How about jboss/wildfly?  Same need re. metadata-generation.
> https://docs.jboss.org/author/display/WFLY8/The+HTTP+management+API?_sscc=t

I'll look into these.

Thanks for looking this over.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)