initial suggestions re. compiler server (bug #6565)

initial suggestions re. compiler server (bug #6565)

[Frank Ch. Eigler]

Hi, Dave -

(Background: we're prototyping a compile server split of stap, to
allow a central network server to perform arbitrary script translation
and compilation for a machine without local debuginfo nor compiler.
It'll be useful later for unprivileged user-space probing too.  Some
prototype code is in bug #6565.)


Here are some suggestions about the prototype code.

* The code might as well get checked into git and be developed there;
  it does not need to be make-install'd or systemtap.spec'd until
  later.

* The scripts are not #! /bin/sh but rather technically #! /bin/bash.

* The wire protocol can probably do everything based on two
  unidirectional bits of communication: one client->server tar.gz
  file to encode the arguments, given tapsets, the script; and one
  server->client tar.gz file to encode the results (.ko/.c files,
  stdout, stderr).

* It should be testable now even without networks via dejagnu making
  two named pipes ("mknod p FOO.1; mknod p FOO.2") and then starting
  the server & client scripts with redirected stdout/stdin.

* The networking aspect of it need not go through xinetd.  If we can't
  find a suitable shell-script-invokable tool to create/attach-to
  listening sockets, then perhaps a few-liner perl script could do the
  job.  That same perl script can use avahi-publish-service to push
  the actual (random) listening port number into mdns, so that 65000
  number need not be hardcoded anywhere.  Clients would find the
  server by abstract type (not _http._tcp but something like
  _systemtap._tcp) and name, and particularly with some cleverly
  chosen TXT description that should identify the server's assigned
  cross-compilation target.

* The client is not using the right avahi search entry points AFAIK.
  The avahi-discovery gui can pick up the full address/txt bits, and
  those are what the client will need to choose its favorite server.
  There must be some other command-line tool for that.

- FChE

Re: initial suggestions re. compiler server (bug #6565)

[Dave Brolley]

Thanks for your comments!

Frank Ch. Eigler wrote:
> Hi, Dave -
>
> (Background: we're prototyping a compile server split of stap, to
> allow a central network server to perform arbitrary script translation
> and compilation for a machine without local debuginfo nor compiler.
> It'll be useful later for unprivileged user-space probing too.  Some
> prototype code is in bug #6565.)
>
>
> Here are some suggestions about the prototype code.
>
> * The code might as well get checked into git and be developed there;
>   it does not need to be make-install'd or systemtap.spec'd until
>   later.
>   
ok
> * The scripts are not #! /bin/sh but rather technically #! /bin/bash.
>   
ok
> * The wire protocol can probably do everything based on two
>   unidirectional bits of communication: one client->server tar.gz
>   file to encode the arguments, given tapsets, the script; and one
>   server->client tar.gz file to encode the results (.ko/.c files,
>   stdout, stderr).
>   
Thats' essentially what happens now. The extra messages are needed right 
now because of the rsync hack. The
client needs to know where to write files to and where to read them from 
on the server's host.
> * It should be testable now even without networks via dejagnu making
>   two named pipes ("mknod p FOO.1; mknod p FOO.2") and then starting
>   the server & client scripts with redirected stdout/stdin.
>   
I'll look into this.
> * The networking aspect of it need not go through xinetd.  If we can't
>   find a suitable shell-script-invokable tool to create/attach-to
>   listening sockets, then perhaps a few-liner perl script could do the
>   job.  That same perl script can use avahi-publish-service to push
>   the actual (random) listening port number into mdns, so that 65000
>   number need not be hardcoded anywhere.  Clients would find the
>   server by abstract type (not _http._tcp but something like
>   _systemtap._tcp) and name, and particularly with some cleverly
>   chosen TXT description that should identify the server's assigned
>   cross-compilation target.
>
> * The client is not using the right avahi search entry points AFAIK.
>   The avahi-discovery gui can pick up the full address/txt bits, and
>   those are what the client will need to choose its favorite server.
>   There must be some other command-line tool for that.
>   
Yes. I discovered some of this earlier this week, hence my comment about 
fixing the hard coded port number.
> - FChE
>   
Thanks again for you ideas!

Dave

Re: initial suggestions re. compiler server (bug #6565)

[Roland McGrath]

> * The networking aspect of it need not go through xinetd.  If we can't
>   find a suitable shell-script-invokable tool to create/attach-to
>   listening sockets, then perhaps a few-liner perl script could do the

nc does this, and stunnel if one wanted a trivially SSLified version.
There are perl modules that make this trivial too, so just hacker's taste.

Re: initial suggestions re. compiler server (bug #6565)

[Ananth N Mavinakayanahalli]

On Thu, May 29, 2008 at 01:40:25PM -0400, Frank Ch. Eigler wrote:

Hi Frank,
 
> (Background: we're prototyping a compile server split of stap, to
> allow a central network server to perform arbitrary script translation
> and compilation for a machine without local debuginfo nor compiler.
> It'll be useful later for unprivileged user-space probing too.  Some
> prototype code is in bug #6565.)

Does this entail distro install changes too?

Currently SystemTap is installed only if "Development" is chosen. With
the server split (in fact, with pre-built modules and staprun support),
one technically doesn't need to have development libraries and/or
kernel-devel, etc., packages on the machine without debuginfo/compiler.

It'd therefore be neat if SystemTap gets installed by default as part of
the base/default install.

Ananth

Re: initial suggestions re. compiler server (bug #6565)

[Dave Brolley]

Frank Ch. Eigler wrote:
> Hi, Dave -
>
> Here are some suggestions about the prototype code.
>
> * The code might as well get checked into git and be developed there;
>   it does not need to be make-install'd or systemtap.spec'd until
>   later.
>   
Done
> * The scripts are not #! /bin/sh but rather technically #! /bin/bash.
>   
Done
> * The networking aspect of it need not go through xinetd.
I'm looking into this. The scripts should probably not be dependent upon 
any particular supporting network service. For the record, I do like 
xinetd because it does a lot to prevent attacks (denial of service, etc) 
and can serve multiple clients simultaneously.
>   Clients would find the
>   server by abstract type (not _http._tcp but something like
>   _systemtap._tcp) and name, and particularly with some cleverly
>   chosen TXT description that should identify the server's assigned
>   cross-compilation target.
>   
Done
> * The client is not using the right avahi search entry points AFAIK.
>   The avahi-discovery gui can pick up the full address/txt bits, and
>   those are what the client will need to choose its favorite server.
>   There must be some other command-line tool for that.
>   
Yes, there was a -r option for the avahi-browse command. I'm using that 
now. All of the above has been committed and pushed.

Dave