Re: [pcp] sketch of possible web pmapi

[Paul Smith <psmith@aconex.com>]

This is great.  This looks RESTful, but not sure if that was the intent.  Would be worth following the REST pattern fully.  Also, I think JSON is a better text output format and now much easily parseable by both humans AND code too.  A really good example of this architecture in place is ElasticSearch (see https://github.com/elasticsearch/elasticsearch)

http://en.wikipedia.org/wiki/Representational_State_Transfer#RESTful_web_services

Some more comments inline:

On 23/03/2011, at 4:53 AM, Frank Ch. Eigler wrote:

> Hi -
> 
> I've finally jotted down some rough ideas about how a hypothetical web
> api might look for pcp.  It exposes only a subset of the PMAPI (not
> sure whether pmns* should be added) in the form of http get operations
> that even a plain web browser can use.  Let me know if you agree this
> is worth prototyping.
> 
> 
> * model:
> 
>               XMLRPC
>  web client    -+-> apache / mod_proxy (reverse-proxying) -+
>  (javascript)   +-> (or directly)                         -+-> pmxmlrpc 
> 
>           PMAPI
>  pmxmlrpc -+-> network pmcd
>            +-> network pmproxy -> pmcd
>            +-> local pmda
> 
> * general idea:
> 
>  - Exposing meaty subset of PMAPI
>  - A plain web browser speaking http should be able to
>     - navigate metric namespace
>     - view metric metadata
>     - view metric values, including enqueued events
>     - GET requests produce XML results, with a minimal linked
>       XSLT->HTML renderer
>  - A plain wget should also work
> 
> * API outline
> 
>  - create a context
> 
>       /context?local
>       /context?hostname=HOSTNAME
>       /context?archivefile=FILENAME
>                       ?poll-timeout=SECONDS
>       /context/CONTEXT/copy
> 

Following REST pattern (see wiki pattern above), this should be a POST.  The 'get' bits after the query string I believe are still valid (although traditionally odd for POST, most people still do that ).

To add more clarity here to make sure I'm following things correctly, I'm presuming the above Create Context action would require a '?...&contextName=foo' style to name it, so that my below examples have some, err, context..


>  - list/describe metrics
> 
>       /context/CONTEXT/metric?glob=GLOB
>       /context/CONTEXT/metric?pmids=ID1,ID2,ID3

Following the REST pattern again, and stealing liberally from the way ElasticSearch does this, I would suggest:

/context/{contextname}/_metric?glob=...

as the pattern.  (You have "CONTEXT" here where I have "{contextname}" implying a template pattern which I'm sure is the same but just being explicit about the pattern. 

eg using the 'foo' context created above:

/context/foo/_metric?pmids=....

also, if it makes sense, the ElasticSearch way of performing actions across objects then maybe this works too:

/context/_metric?pmids=...

Where the query on the _metric action is across _all_ defined Context names.  Other action verbs like _fetch (see below) could then theoretically be applied across multiple contexts:

/context/foo,bar,othercontextnumber3/_metric?pmids=....


> 
>  - fetch metric values
> 
>       /context/CONTEXT/value?pmids=ID1,ID2,ID3
>       /context/CONTEXT/value?pmids=ID1,ID2,ID3&time=TIME
> 

What about:

/context/{contextname}/_fetch?....


>  - list/describe/modify instance profile
> 
>       /context/CONTEXT/domain?ids=ID1,ID2
>       /context/CONTEXT/domain?profile=add&ids=ID1,ID2
>       /context/CONTEXT/domain?profile=delete&ids=ID1,ID2
> 



> * operation outline:
> 
>  1) browse http://pmwebapisrv:8234/, authenticate, get html form to select
>     connection context: local (or) hostname=HOSTNAME
>                               (or) archivefile=FILENAME,
>     optional poll-timeout=NUMBER.  Each per-context request
>     restarts the timeout.  If it expires, the web proxy will disconnect
>     from the PCPD/PMDA / pmDestroyContext().
> 

I think the poll-timeout should be a client value, not part of the protocol..?  Did you mean that?  If you're meaning a timeout on the web-proxy pulling values from the PMDAs then that should be a configuration value on that side, no the REST or XMLRP protocol layer?


>  2) submit GET /context? form to return an XML document denoting the
>     creation of a context:
> 
>     <? ... ?>
>     <?xml-stylesheet ... href="context.xsl"?>
>     <pcp:context name="CONTEXTNAME" expires="TIMESTAMP"/>
> 

JSON for the win:

{
  context: { name: "foo", expires="2011-04-23 10:00"}
}


>     The context.xsl stylesheet renders this XML to HTML, noting how long
>     the context will remain open without a refresh, and listing navigation
>     options such as:
> 
>     /context/CONTEXTNAME/copy   -> to generate a new dup context; back to 2)
>     a form /context/CONTEXTNAME/metric?glob=GLOB ; and
>     a form /context/CONTEXTNAME/metric?id=PMID,PMID,PMID ; to jump to metric
>     metadata
> 

A well described REST architecture will allow most REST clients to self navigate these days.

>  3) the list of metric metadata is another XML document, from pmLookupDesc()
>     <? ... ?>
>     <?xml-stylesheet ... href="metrics.xsl"?>
>     <pcp:metrics expires="TIMESTAMP">
>     <pcp:metric name="METRICNAME" id="PMID">
>          <pcp:type number="NUM" name="TYPENAME"/>
>          <pcp:indom number="NUM" name="INDOMNAME"/>
>          <pcp:counter/> or <pcp:instant/> or <pcp:discrete/>
>          <pcp:units space="NUM" time="NUM" count="NUM"
>                     scalespace="STRING" scaletime="STRING" scalecount="NUM"/>
>     </pcp:metric>  ...
>     </pcp:metrics>
> 

JSON example with metric array:

{
  metrics: [
    { type:"NUM", metricType: "counter", name:"mycountername", indom: { number: 123, name:"indomname" },
    { type:"NUM", metricType: "instant", name:"myinstantname", indom: { number: 124, name:"indomname2" }
     ....
      
  ]
}

> 


I'm no fan of XSLT, I would think raw pretty printed JSON delivered by the server (cheap) is just inherently readable by a human compared with XML, and is VERY easily manipulated into a fully fledged UI (not simply xslt-> basic html stuff).

Ben Birch (cc'd) from Aconex write an ElasticSearch UI client based on this, (see [1]) and the JSON->UI is easily done and could be applicable example of how a lightweight JavaScript pcp 'browser' could work.  Getting ahead of myself here, a 'pcp-head' JavaScript UI client like Ben's ElasticSearch one could include graphs etc too... Perhaps a nice cross platform pmchart replacement... *ducks*.

cheers,

Paul Smith
[1] ElasticSearch-head : https://github.com/mobz/elasticsearch-head