sketch of possible web pmapi

sketch of possible web pmapi

[Frank Ch. Eigler]

Hi -

I've finally jotted down some rough ideas about how a hypothetical web
api might look for pcp.  It exposes only a subset of the PMAPI (not
sure whether pmns* should be added) in the form of http get operations
that even a plain web browser can use.  Let me know if you agree this
is worth prototyping.


* model:

               XMLRPC
  web client    -+-> apache / mod_proxy (reverse-proxying) -+
  (javascript)   +-> (or directly)                         -+-> pmxmlrpc 

           PMAPI
  pmxmlrpc -+-> network pmcd
            +-> network pmproxy -> pmcd
            +-> local pmda

* general idea:

  - Exposing meaty subset of PMAPI
  - A plain web browser speaking http should be able to
     - navigate metric namespace
     - view metric metadata
     - view metric values, including enqueued events
     - GET requests produce XML results, with a minimal linked
       XSLT->HTML renderer
  - A plain wget should also work

* API outline

  - create a context

       /context?local
       /context?hostname=HOSTNAME
       /context?archivefile=FILENAME
                       ?poll-timeout=SECONDS
       /context/CONTEXT/copy

  - list/describe metrics

       /context/CONTEXT/metric?glob=GLOB
       /context/CONTEXT/metric?pmids=ID1,ID2,ID3

  - fetch metric values

       /context/CONTEXT/value?pmids=ID1,ID2,ID3
       /context/CONTEXT/value?pmids=ID1,ID2,ID3&time=TIME

  - list/describe/modify instance profile

       /context/CONTEXT/domain?ids=ID1,ID2
       /context/CONTEXT/domain?profile=add&ids=ID1,ID2
       /context/CONTEXT/domain?profile=delete&ids=ID1,ID2

* operation outline:

  1) browse http://pmwebapisrv:8234/, authenticate, get html form to select
     connection context: local (or) hostname=HOSTNAME
                               (or) archivefile=FILENAME,
     optional poll-timeout=NUMBER.  Each per-context request
     restarts the timeout.  If it expires, the web proxy will disconnect
     from the PCPD/PMDA / pmDestroyContext().

  2) submit GET /context? form to return an XML document denoting the
     creation of a context:

     <? ... ?>
     <?xml-stylesheet ... href="context.xsl"?>
     <pcp:context name="CONTEXTNAME" expires="TIMESTAMP"/>

     The context.xsl stylesheet renders this XML to HTML, noting how long
     the context will remain open without a refresh, and listing navigation
     options such as:

     /context/CONTEXTNAME/copy   -> to generate a new dup context; back to 2)
     a form /context/CONTEXTNAME/metric?glob=GLOB ; and
     a form /context/CONTEXTNAME/metric?id=PMID,PMID,PMID ; to jump to metric
     metadata

  3) the list of metric metadata is another XML document, from pmLookupDesc()
     <? ... ?>
     <?xml-stylesheet ... href="metrics.xsl"?>
     <pcp:metrics expires="TIMESTAMP">
     <pcp:metric name="METRICNAME" id="PMID">
          <pcp:type number="NUM" name="TYPENAME"/>
          <pcp:indom number="NUM" name="INDOMNAME"/>
          <pcp:counter/> or <pcp:instant/> or <pcp:discrete/>
          <pcp:units space="NUM" time="NUM" count="NUM"
                     scalespace="STRING" scaletime="STRING" scalecount="NUM"/>
     </pcp:metric>  ...
     </pcp:metrics>

     which is to render to a lovely HTML table, describing the metric
     metadata, plus providing links to metric-value and domain-instance
     views below:

  4) the list of metric values is another XML document served by
     /context/CONTEXT/value?pmids=ID1,ID2,ID3 [&time=TIME]

     <? ... ?>
     <?xml-stylesheet ... href="values.xsl"?>
     <pcp:values time="TIMESTAMP" expires="TIMESTAMP">
     <pcp:value pmid="PMID">
                 <pcp:result inst="NUM">9999</pcp:value> # if printable?
         #or     <pcp:result inst="NUM" encoding="base64Binary">base64-text
                            </pcp:value>
         #or     <pcp:event inst="NUM" time="TIMESTAMP">
                            <pcp:result ...><pcp:result ...> ...
                            </pcp:event>
     </pcp:value>
     </pcp:values>

     The xslt widget would render the values into a tabular form, with
     a refresh link.  This is the one that a web application or
     browser would want to fetch frequently.

  5) domain instance metadata from pmGetInDom() would be served by
     /context/CONTEXT/domain?ids=ID1,ID2,ID3 :

     resulting in:
     <? ... ?>
     <?xml-stylesheet ... href="domains.xsl"?>
     <pcp:domains expires="TIMESTAMP">
     <pcp:domain name="NAME" id="ID">
                 <pcp:domain-instance name="NAME" id="ID"/>
                 ...
     </pcp:domain> ...
     </pcp:domains>

     This would be rendered into HTML that allows invidual domain-instances to 
     be pmAddProfile/pmDelProfile'd to the CONTEXT, via 

     /context/CONTEXT/domain?profile=add&ids=ID1,ID2,ID3
     /context/CONTEXT/domain?profile=add&ids=all
     /context/CONTEXT/domain?profile=delete&ids=ID1,ID2,ID3
     /context/CONTEXT/domain?profile=delete&ids=all

     which would return the same sort of XML as above, after adjusting
     the context.  XXX: is there a way to query PMAPI as to the
     current pmAddProfile active instances?

Re: [pcp] sketch of possible web pmapi

[Paul Smith]

This is great.  This looks RESTful, but not sure if that was the intent.  Would be worth following the REST pattern fully.  Also, I think JSON is a better text output format and now much easily parseable by both humans AND code too.  A really good example of this architecture in place is ElasticSearch (see https://github.com/elasticsearch/elasticsearch)

http://en.wikipedia.org/wiki/Representational_State_Transfer#RESTful_web_services

Some more comments inline:

On 23/03/2011, at 4:53 AM, Frank Ch. Eigler wrote:

> Hi -
> 
> I've finally jotted down some rough ideas about how a hypothetical web
> api might look for pcp.  It exposes only a subset of the PMAPI (not
> sure whether pmns* should be added) in the form of http get operations
> that even a plain web browser can use.  Let me know if you agree this
> is worth prototyping.
> 
> 
> * model:
> 
>               XMLRPC
>  web client    -+-> apache / mod_proxy (reverse-proxying) -+
>  (javascript)   +-> (or directly)                         -+-> pmxmlrpc 
> 
>           PMAPI
>  pmxmlrpc -+-> network pmcd
>            +-> network pmproxy -> pmcd
>            +-> local pmda
> 
> * general idea:
> 
>  - Exposing meaty subset of PMAPI
>  - A plain web browser speaking http should be able to
>     - navigate metric namespace
>     - view metric metadata
>     - view metric values, including enqueued events
>     - GET requests produce XML results, with a minimal linked
>       XSLT->HTML renderer
>  - A plain wget should also work
> 
> * API outline
> 
>  - create a context
> 
>       /context?local
>       /context?hostname=HOSTNAME
>       /context?archivefile=FILENAME
>                       ?poll-timeout=SECONDS
>       /context/CONTEXT/copy
> 

Following REST pattern (see wiki pattern above), this should be a POST.  The 'get' bits after the query string I believe are still valid (although traditionally odd for POST, most people still do that ).

To add more clarity here to make sure I'm following things correctly, I'm presuming the above Create Context action would require a '?...&contextName=foo' style to name it, so that my below examples have some, err, context..


>  - list/describe metrics
> 
>       /context/CONTEXT/metric?glob=GLOB
>       /context/CONTEXT/metric?pmids=ID1,ID2,ID3

Following the REST pattern again, and stealing liberally from the way ElasticSearch does this, I would suggest:

/context/{contextname}/_metric?glob=...

as the pattern.  (You have "CONTEXT" here where I have "{contextname}" implying a template pattern which I'm sure is the same but just being explicit about the pattern. 

eg using the 'foo' context created above:

/context/foo/_metric?pmids=....

also, if it makes sense, the ElasticSearch way of performing actions across objects then maybe this works too:

/context/_metric?pmids=...

Where the query on the _metric action is across _all_ defined Context names.  Other action verbs like _fetch (see below) could then theoretically be applied across multiple contexts:

/context/foo,bar,othercontextnumber3/_metric?pmids=....


> 
>  - fetch metric values
> 
>       /context/CONTEXT/value?pmids=ID1,ID2,ID3
>       /context/CONTEXT/value?pmids=ID1,ID2,ID3&time=TIME
> 

What about:

/context/{contextname}/_fetch?....


>  - list/describe/modify instance profile
> 
>       /context/CONTEXT/domain?ids=ID1,ID2
>       /context/CONTEXT/domain?profile=add&ids=ID1,ID2
>       /context/CONTEXT/domain?profile=delete&ids=ID1,ID2
> 



> * operation outline:
> 
>  1) browse http://pmwebapisrv:8234/, authenticate, get html form to select
>     connection context: local (or) hostname=HOSTNAME
>                               (or) archivefile=FILENAME,
>     optional poll-timeout=NUMBER.  Each per-context request
>     restarts the timeout.  If it expires, the web proxy will disconnect
>     from the PCPD/PMDA / pmDestroyContext().
> 

I think the poll-timeout should be a client value, not part of the protocol..?  Did you mean that?  If you're meaning a timeout on the web-proxy pulling values from the PMDAs then that should be a configuration value on that side, no the REST or XMLRP protocol layer?


>  2) submit GET /context? form to return an XML document denoting the
>     creation of a context:
> 
>     <? ... ?>
>     <?xml-stylesheet ... href="context.xsl"?>
>     <pcp:context name="CONTEXTNAME" expires="TIMESTAMP"/>
> 

JSON for the win:

{
  context: { name: "foo", expires="2011-04-23 10:00"}
}


>     The context.xsl stylesheet renders this XML to HTML, noting how long
>     the context will remain open without a refresh, and listing navigation
>     options such as:
> 
>     /context/CONTEXTNAME/copy   -> to generate a new dup context; back to 2)
>     a form /context/CONTEXTNAME/metric?glob=GLOB ; and
>     a form /context/CONTEXTNAME/metric?id=PMID,PMID,PMID ; to jump to metric
>     metadata
> 

A well described REST architecture will allow most REST clients to self navigate these days.

>  3) the list of metric metadata is another XML document, from pmLookupDesc()
>     <? ... ?>
>     <?xml-stylesheet ... href="metrics.xsl"?>
>     <pcp:metrics expires="TIMESTAMP">
>     <pcp:metric name="METRICNAME" id="PMID">
>          <pcp:type number="NUM" name="TYPENAME"/>
>          <pcp:indom number="NUM" name="INDOMNAME"/>
>          <pcp:counter/> or <pcp:instant/> or <pcp:discrete/>
>          <pcp:units space="NUM" time="NUM" count="NUM"
>                     scalespace="STRING" scaletime="STRING" scalecount="NUM"/>
>     </pcp:metric>  ...
>     </pcp:metrics>
> 

JSON example with metric array:

{
  metrics: [
    { type:"NUM", metricType: "counter", name:"mycountername", indom: { number: 123, name:"indomname" },
    { type:"NUM", metricType: "instant", name:"myinstantname", indom: { number: 124, name:"indomname2" }
     ....
      
  ]
}

> 


I'm no fan of XSLT, I would think raw pretty printed JSON delivered by the server (cheap) is just inherently readable by a human compared with XML, and is VERY easily manipulated into a fully fledged UI (not simply xslt-> basic html stuff).

Ben Birch (cc'd) from Aconex write an ElasticSearch UI client based on this, (see [1]) and the JSON->UI is easily done and could be applicable example of how a lightweight JavaScript pcp 'browser' could work.  Getting ahead of myself here, a 'pcp-head' JavaScript UI client like Ben's ElasticSearch one could include graphs etc too... Perhaps a nice cross platform pmchart replacement... *ducks*.

cheers,

Paul Smith
[1] ElasticSearch-head : https://github.com/mobz/elasticsearch-head

Re: [pcp] sketch of possible web pmapi

[Frank Ch. Eigler]

Hi, Paul -


> This is great.  This looks RESTful, but not sure if that was the
> intent.  Would be worth following the REST pattern fully.

I read about REST, but only with web-app newbie eyes.

> Also, I think JSON is a better text output format and now much
> easily parseable by both humans AND code too.  [...]

My main interest in using XML for the raw output is the ability to
easily associate it with an XSLT stylesheet to render the data into
HTML.  I believe that it would be useful to be able to perform basic
navigation over the PCP data with nothing other than a plain old web
browser or spider, even without javascript.

JSON could be an alternative output format, selected with some URL
parameter.  OTOH then HTML could be the same thing.


> >       /context?local
> >       /context?hostname=HOSTNAME
> >       /context?archivefile=FILENAME
> >                       ?poll-timeout=SECONDS
> >       /context/CONTEXT/copy
> > 

> Following REST pattern (see wiki pattern above), this should be a
> POST.  The 'get' bits after the query string I believe are still
> valid (although traditionally odd for POST, most people still do
> that ).

(I have no informed opinion on this.)

> To add more clarity here to make sure I'm following things
> correctly, I'm presuming the above Create Context action would
> require a '?...&contextName=foo' [...]

Actually, I was thinking that the server process would assign an
opaque context name, since it's the one in position to make a unique
one.


> >  - list/describe metrics
> > 
> >       /context/CONTEXT/metric?glob=GLOB
> >       /context/CONTEXT/metric?pmids=ID1,ID2,ID3

> Following the REST pattern again, and stealing liberally from the
> way ElasticSearch does this, I would suggest:
>
> /context/{contextname}/_metric?glob=...

Are you only proposing to prefix the listing-type operations with an
underscore, or is there something else?


> /context/_metric?pmids=...
>
> Where the query on the _metric action is across _all_ defined
> Context names.  Other action verbs like _fetch (see below) could
> then theoretically be applied across multiple contexts:

I've been cosidering separate contexts as belonging to independent
concurrent users, so I'd prefer not to expose the "all contexts"
object at all.

> /context/foo,bar,othercontextnumber3/_metric?pmids=....

This could make sense; is there any PCP precedent or need for this?


> >       /context/CONTEXT/value?pmids=ID1,ID2,ID3
> >       /context/CONTEXT/value?pmids=ID1,ID2,ID3&time=TIME
> 
> What about:
> 
> /context/{contextname}/_fetch?....

Sure, no strong opinion.


> [...]
> I think the poll-timeout should be a client value, not part of the
> protocol..?  Did you mean that?

I've been thinking of it as a statement that the client intends to
poll the PCP system no less frequently than the negotiated
poll-timeout value.  The main purpose of this is to manage the
lifecycles of the underlying pmNewContext / pmDestroyContext
calls, and specifically the enqueued trace-events.

> If you're meaning a timeout on the web-proxy pulling values from the
> PMDAs then that should be a configuration value on that side, no the
> REST or XMLRP protocol layer?

I'm not quite sure what you mean.


> [...] Getting ahead of myself here, a 'pcp-head' JavaScript UI
> client like Ben's ElasticSearch one could include graphs etc
> too... Perhaps a nice cross platform pmchart replacement... *ducks*.

This is what we have in mind as a potential eventual result.

- FChE

Re: [pcp] sketch of possible web pmapi

[Paul Smith]

> 
> My main interest in using XML for the raw output is the ability to
> easily associate it with an XSLT stylesheet to render the data into
> HTML.  I believe that it would be useful to be able to perform basic
> navigation over the PCP data with nothing other than a plain old web
> browser or spider, even without javascript.
> 

I'm not sure of the use case of viewing HTML these days with a browser that doesn't have javascript accessible though.. ?    

> JSON could be an alternative output format, selected with some URL
> parameter.  OTOH then HTML could be the same thing.
> 


yes, different formatters could be selected, although that adds complexity.  XSLT is just.. well... a fairly 'dead' platform these days.  Sure, still used, just if something is being done new here, I would think worth considering what the state of play of the browsers support for things?

Rendering a HTML table for example when the JSON is actually pretty readable by a human already (certainly compared with XML).

One thing I don't like about embedding XSLT details in the XML output is that it is still coupling the presentation layer to the data delivery layer here.  It would make it simple for browser to auto-display things, but it also fixes the browser to one XSLT based on the output defined there.  I would have thought much better to design the REST or whatever mechanism so that it is purely a cross platform data delivery platform that both simple browsers and other application clients can interact with.    Honestly, I wonder whether the PMDAs themselves shouldn't expose REST direct ??   (that's a way bigger change though obviously, nice to have a translation layer here bridging the pmda's like pmproxy does).


> 
>>>      /context?local
>>>      /context?hostname=HOSTNAME
>>>      /context?archivefile=FILENAME
>>>                      ?poll-timeout=SECONDS
>>>      /context/CONTEXT/copy
>>> 
> 
>> Following REST pattern (see wiki pattern above), this should be a
>> POST.  The 'get' bits after the query string I believe are still
>> valid (although traditionally odd for POST, most people still do
>> that ).
> 
> (I have no informed opinion on this.)
> 
>> To add more clarity here to make sure I'm following things
>> correctly, I'm presuming the above Create Context action would
>> require a '?...&contextName=foo' [...]
> 
> Actually, I was thinking that the server process would assign an
> opaque context name, since it's the one in position to make a unique
> one.
> 

so basically the contextName is a unique session handle between the browser client and the RPC bridge?

> 
>>> - list/describe metrics
>>> 
>>>      /context/CONTEXT/metric?glob=GLOB
>>>      /context/CONTEXT/metric?pmids=ID1,ID2,ID3
> 
>> Following the REST pattern again, and stealing liberally from the
>> way ElasticSearch does this, I would suggest:
>> 
>> /context/{contextname}/_metric?glob=...
> 
> Are you only proposing to prefix the listing-type operations with an
> underscore, or is there something else?
> 

yes, it's a RESTful pattern here seperating the objects/collections from the operation one is performing.   It's purely for readability.  This way it's unambiguous that:

/context/foo/metrics

means 'metrics' is a child object of 'foo', whereas

/contex/foo/_metrics

is a message 'please ask the metrics operation to query the 'foo' collection for these values'.


> 
>> /context/_metric?pmids=...
>> 
>> Where the query on the _metric action is across _all_ defined
>> Context names.  Other action verbs like _fetch (see below) could
>> then theoretically be applied across multiple contexts:
> 
> I've been cosidering separate contexts as belonging to independent
> concurrent users, so I'd prefer not to expose the "all contexts"
> object at all.
> 
>> /context/foo,bar,othercontextnumber3/_metric?pmids=....
> 
> This could make sense; is there any PCP precedent or need for this?
> 

probably not... there may be other cases in the protocol where a collection of <whatevers> makes sense, I had just borrowed this as a potentially example, wasn't sure if it were valid.  Just putting out there in some cases the URL protocol may like to have collections of objects as part of the objects.


> 
> I've been thinking of it as a statement that the client intends to
> poll the PCP system no less frequently than the negotiated
> poll-timeout value.  The main purpose of this is to manage the
> lifecycles of the underlying pmNewContext / pmDestroyContext
> calls, and specifically the enqueued trace-events.
> 
>> If you're meaning a timeout on the web-proxy pulling values from the
>> PMDAs then that should be a configuration value on that side, no the
>> REST or XMLRP protocol layer?
> 
> I'm not quite sure what you mean.
> 


Rereading your original post I see the creation of the context object is giving this setting as an indication of the polling frequency so this XML RPC proxy/bridge thing can make informed choices on when it's ok to automatically shut down it's internal collection to the PMDA's etc. That's probably valid.  I guess having an implicit default value configured on the RPC bridge if the client chooses to not provide one for safety?

Paul

Re: sketch of possible web pmapi

[Nathan Scott]

----- Original Message -----
> Hi -
> 
> I've finally jotted down some rough ideas about how a hypothetical web
> api might look for pcp. It exposes only a subset of the PMAPI (not
> sure whether pmns* should be added) in the form of http get operations
> that even a plain web browser can use. Let me know if you agree this
> is worth prototyping.

For sure, there's definitely a need.

> which would return the same sort of XML as above, after adjusting
> the context. XXX: is there a way to query PMAPI as to the
> current pmAddProfile active instances?

Nope, not today ... would need a protocol extension, I think.

cheers.

-- 
Nathan

Re: [pcp] sketch of possible web pmapi

[Frank Ch. Eigler]

Hi, Paul -

> I'm not sure of the use case of viewing HTML these days with a
> browser that doesn't have javascript accessible though.. ?

It's not just the browser, but other basic html consumers like web
spiders.  Plus on browsers too, security worries often mean lockdown
of javascript.  Plus it's a possible milestone for development: to get
something browseable & barely usable, before having to hack on
javascript.

> yes, different formatters could be selected, although that adds
> complexity.  XSLT is just.. well... a fairly 'dead' platform these
> days.  Sure, still used, just if something is being done new here, I
> would think worth considering what the state of play of the browsers
> support for things?

Yeah.

> Rendering a HTML table for example when the JSON is actually pretty
> readable by a human already (certainly compared with XML).

Yes, but a human-readable piece of text isn't clickable in the way
an html table with <A>'s in 

> One thing I don't like about embedding XSLT details in the XML
> output is that it is still coupling the presentation layer to the
> data delivery layer here.

I see what you mean, but if we are to generate something
html-renderable at some point, we'd have to provide a default
presentation layer somehow.

> I would have thought much better to design the REST or whatever
> mechanism so that it is purely a cross platform data delivery
> platform that both simple browsers and other application clients can
> interact with.

I figured that a more sophisticated use than the dumb html browser
would be able to suck out the exact raw xml data by simply skipping
the xslt transform.  It's this dual-use aspect that seems to make xml
attractive here.  With JSON, we get reasonably easy human eyeballing,
and machine parsing, but not the simple html case.  (And with the html
rendering, human eyeballing of the raw message may not be that
important anyway.)  I don't know what the best answer is though.


> [...]  so basically the contextName is a unique session handle
> between the browser client and the RPC bridge?

Yes.

> [...]  I guess having an implicit default value [for the
> poll-timeout configured on the RPC bridge if the client chooses to
> not provide one for safety?

Certainly, good idea.

- FChE

Re: [pcp] sketch of possible web pmapi

[Paul Smith]

On 24/03/2011, at 1:08 PM, Frank Ch. Eigler wrote:

> Hi, Paul -
> 
>> I'm not sure of the use case of viewing HTML these days with a
>> browser that doesn't have javascript accessible though.. ?
> 
> It's not just the browser, but other basic html consumers like web
> spiders.  Plus on browsers too, security worries often mean lockdown
> of javascript.  Plus it's a possible milestone for development: to get
> something browseable & barely usable, before having to hack on
> javascript.
> 

Not 100% sure what the web-spider use case is about..?  Could you describe that more?  With regards to disabling javascript on browsers in security concious environments would render many useful tools for production-like activities that would be rendered difficult there.  

A very basic javascript pattern to click->drill into is exceedingly simple though?

>> yes, different formatters could be selected, although that adds
>> complexity.  XSLT is just.. well... a fairly 'dead' platform these
>> days.  Sure, still used, just if something is being done new here, I
>> would think worth considering what the state of play of the browsers
>> support for things?
> 
> Yeah.
> 
>> Rendering a HTML table for example when the JSON is actually pretty
>> readable by a human already (certainly compared with XML).
> 
> Yes, but a human-readable piece of text isn't clickable in the way
> an html table with <A>'s in 

can be done with css/javascript in the same time or less than XSLT I think.  

> 
>> One thing I don't like about embedding XSLT details in the XML
>> output is that it is still coupling the presentation layer to the
>> data delivery layer here.
> 
> I see what you mean, but if we are to generate something
> html-renderable at some point, we'd have to provide a default
> presentation layer somehow.
> 

I agree they can't be done in isolation, I just do think they should be separated concerns.  Ben how much effort was the ES clickable JSON stuff?  Is there a basic example you can show us from elasticsearch-head?

>> I would have thought much better to design the REST or whatever
>> mechanism so that it is purely a cross platform data delivery
>> platform that both simple browsers and other application clients can
>> interact with.
> 
> I figured that a more sophisticated use than the dumb html browser
> would be able to suck out the exact raw xml data by simply skipping
> the xslt transform.  It's this dual-use aspect that seems to make xml
> attractive here.  With JSON, we get reasonably easy human eyeballing,
> and machine parsing, but not the simple html case.  (And with the html
> rendering, human eyeballing of the raw message may not be that
> important anyway.)  I don't know what the best answer is though.
> 

But you still have to develop the XSLT transform.  That's not difficult sure, but then it's probably the same as the css/javascript 'viewer'.  I'm totally happy to volunteer my time on that, actually I'm more likely to volunteer Ben's time on that.. :)

At the end of the day though this is driven by the community and what people want.  I'm just proposing JSON as a nice lightweight simple output that humans and code both like, together with choosing something that has good mainstream support now compared with something that is rapidly withering away as a common technology of use.

There's a lot more power and potential in the JSON/JavaScript for a 'client' application of this REST/whatever API.  Charts, drill throughs, tabular views with frameworks like jQuery/ExtJS etc.

Paul

Re: [pcp] sketch of possible web pmapi

[Ben Birch]

Hi All,

  So as Paul has indicated, I'm the developer of a elasticsearch-head, which is an admin interface built on top of a RESTful api serving json. In the past I have worked with an xml -> xslt -> website build.

Sorry for top posting, but I wanted to give an overview opinion rather than address individual points.

For something like the pmapi, really a RESTful interface is the way to go. If you accept this, you really don't want a RESTful api serving xslt (or even serving something that looks good in a browser), it conflates a pure data interface with presentational features that most consumers of your api are not interested in.

Now whether you serve XML or JSON is largely a decision based on who your likely consumers are. There does seem to be a swing away from xml to json, but I'm a ui engineer - so I might be biased. Certainly languages like java and python have json support that is as good as xml. JSON - which is a subset of a programming language - is more suitable for use IN programming languages (explicit support for boolean/numbers/arrays/maps) - where as xml is really a generalised document markup langauge. However if your primary goal is a static website, and you have xslt skills it might be a good choice!

Luckily this is not a choice you must make, as there are frameworks that help you write RESTful interfaces which will allow the consumer to choose XML or JSON (literally by appending ?format=xml or ?format=json ).

If you are worried about serving a basic website out of the box, you might consider starting two services
eg http://localhost:9100 is the website
   http://localhost:9101 is the rest api ?

Regarding js development: most front end engineer are going to be looking for a json api, since json can be directly pushed into rich interface frameworks. Most api consumers are likely to be java/phython/etc apps who can generally use xml or json interchangably, or web apps which have javascript available.
 
----

For an example of the kind of code I'm writing to display tablular data (the actual table templates are in a kind of third party library)

onDataReady: function(metadata) {
    this.metadata = metadata;
    this.dataSource = new es.QueryDataSourceInterface( { metadata: metadata, query: this.query } );
    this.resultTable = new es.ui.Table( {
      onHeaderClick: this._changeSort_handler,
      store: this.dataSource 
    } );
    this.resultTable.appendTo( this.el.find("> .browser-table") );
    this.updateResults();
}

not sure if this is illustrative though. As far as xslt vs javascript being easier to program - it's really not a fair comparison because all you can do with xslt is produce html, where as for the same effort javascript gives you an interactive web app. (actually I found xslt always gave me headaches... )


Cheers,

-Ben



----- Original Message -----
From: "Paul Smith" <psmith@aconex.com>
To: "Frank Ch. Eigler" <fche@redhat.com>
Cc: pcp@oss.sgi.com, systemtap@sourceware.org, "Ben Birch" <bbirch@aconex.com>
Sent: Thursday, 24 March, 2011 1:23:29 PM
Subject: Re: [pcp] sketch of possible web pmapi


> On 24/03/2011, at 1:08 PM, Frank Ch. Eigler wrote:
>
>> Hi, Paul -
>> 
>>> I'm not sure of the use case of viewing HTML these days with a
>>> browser that doesn't have javascript accessible though.. ?
>> 
>> It's not just the browser, but other basic html consumers like web
>> spiders.  Plus on browsers too, security worries often mean lockdown
>> of javascript.  Plus it's a possible milestone for development: to get
>> something browseable & barely usable, before having to hack on
>> javascript.
>> 
>
> Not 100% sure what the web-spider use case is about..?  Could you describe that more?  With regards to disabling javascript on browsers >  > in security concious environments would render many useful tools for production-like activities that would be rendered difficult there.  
>
> A very basic javascript pattern to click->drill into is exceedingly simple though?
>
>>> yes, different formatters could be selected, although that adds
>>> complexity.  XSLT is just.. well... a fairly 'dead' platform these
>>> days.  Sure, still used, just if something is being done new here, I
>>> would think worth considering what the state of play of the browsers
>>> support for things?
>> 
>> Yeah.
>> 
>>> Rendering a HTML table for example when the JSON is actually pretty
>>> readable by a human already (certainly compared with XML).
>> 
>> Yes, but a human-readable piece of text isn't clickable in the way
>> an html table with <A>'s in 
>
> can be done with css/javascript in the same time or less than XSLT I think.  
>
>> 
>>> One thing I don't like about embedding XSLT details in the XML
>>> output is that it is still coupling the presentation layer to the
>>> data delivery layer here.
>> 
>> I see what you mean, but if we are to generate something
>> html-renderable at some point, we'd have to provide a default
>> presentation layer somehow.
>> 
>
> I agree they can't be done in isolation, I just do think they should be separated concerns.  Ben how much effort was the ES clickable JSON stuff?  Is there a basic example you can show us from elasticsearch-head?
>
>>> I would have thought much better to design the REST or whatever
>>> mechanism so that it is purely a cross platform data delivery
>>> platform that both simple browsers and other application clients can
>>> interact with.
>> 
>> I figured that a more sophisticated use than the dumb html browser
>> would be able to suck out the exact raw xml data by simply skipping
>> the xslt transform.  It's this dual-use aspect that seems to make xml
>> attractive here.  With JSON, we get reasonably easy human eyeballing,
>> and machine parsing, but not the simple html case.  (And with the html
>> rendering, human eyeballing of the raw message may not be that
>> important anyway.)  I don't know what the best answer is though.
>> 
>
> But you still have to develop the XSLT transform.  That's not difficult sure, but then it's probably the same as the css/javascript
> 'viewer'.  I'm totally happy to volunteer my time on that, actually I'm more likely to volunteer Ben's time on that.. :)
>
> At the end of the day though this is driven by the community and what people want.  I'm just proposing JSON as a nice lightweight simple > output that humans and code both like, together with choosing something that has good mainstream support now compared with something that is rapidly withering away as a common technology of use.
>
> There's a lot more power and potential in the JSON/JavaScript for a 'client' application of this REST/whatever API.  Charts, drill throughs, tabular views with frameworks like jQuery/ExtJS etc.

> Paul

Re: [pcp] sketch of possible web pmapi

[Frank Ch. Eigler]

Hi - 

On Thu, Mar 24, 2011 at 05:11:05PM +1100, Ben Birch wrote:

> [...]  For something like the pmapi, really a RESTful interface is
> the way to go. If you accept this, you really don't want a RESTful
> api serving xslt [...]  Regarding js development: most front end
> engineer are going to be looking for a json api [...]

OK guys, thanks, I'm convinced.  I'll start hacking together a
JSON-emitting widget.

- FChE