security for systemtap compiler server

security for systemtap compiler server

[Frank Ch. Eigler]

Hi -



- FChE

security for systemtap compiler server

[Frank Ch. Eigler]

Hi -

(... oops, as I was saying ...)

Dave Brolley's prototype work for a networked systemtap compile server
is intended to serve two purposes.  

First, it aims to reduce the installation footprint on a given
deployment machine, so that systemtap users can defer work to a single
centralized compiler / kernel-devel / debuginfo toolchain on the
network rather than on every machine that needs probing.

Second, it is part of enabling unprivileged users to run systemtap
scripts that are severely restricted (no kernel probes; only probes on
one's own processes; that sort of thing).  (Allowing an unprivileged
user to build his own kernel modules via gcc etc., opens up too many
possibilities for subversion of the constraints.)


Both these parts are tantamount to the local machine's sysadmin
declaring that she trusts a particular remote service to build correct
systemtap modules, and is willing to permit running them (via the
local setuid "staprun" launcher).  The remote service must of course
distrust any data sent by the client, and emit code to verify anything
relevant (uid / stapdev group membership, versions/buildids).

How to identify the trusted remote service?  One natural way is via a
keypair as in ssh or gpg: have the compile server sign its output
module with a private key; let the (setuid) staprun check the
signature against a sysadmin-approved list in a file like
/etc/systemtap/authorized_keys.  (openssl is probably the right
toolkit for this stuff.)

At the minimum, this should prevent an ordinary unauthorized user from
just starting up his own compile server and fool a system into running
its modules.  An insecure network would benefit from encrypted data
transfer between client & server (as in stunnel).  I haven't thought
much about other threats.


Is all this making sense so far?


- FChE

RE: security for systemtap compiler server

[Stone, Joshua I]

Frank Ch. Eigler wrote:
> Second, it is part of enabling unprivileged users to run systemtap
> scripts that are severely restricted (no kernel probes; only probes on
> one's own processes; that sort of thing).

We can easily classify probe points in this way, but we'll have to be
careful about which tapset functions may be called.  Perhaps we can
bless them with a new keyword, similar to how "pure" works now.

Also, printf %b may have to be disallowed, unless we can find a way to
certify the pointer origin.

> Is all this making sense so far?

The rest that you said sounds good to me...

Josh

Re: security for systemtap compiler server

[Masami Hiramatsu]

Hi,

Frank Ch. Eigler wrote:
> Hi -
> 
> (... oops, as I was saying ...)
> 
> Dave Brolley's prototype work for a networked systemtap compile server
> is intended to serve two purposes.  
> 
> First, it aims to reduce the installation footprint on a given
> deployment machine, so that systemtap users can defer work to a single
> centralized compiler / kernel-devel / debuginfo toolchain on the
> network rather than on every machine that needs probing.

Sure.

> Second, it is part of enabling unprivileged users to run systemtap
> scripts that are severely restricted (no kernel probes; only probes on
> one's own processes; that sort of thing).  (Allowing an unprivileged
> user to build his own kernel modules via gcc etc., opens up too many
> possibilities for subversion of the constraints.)

I think this is also the issue of systemtap itself, not only
compiler server, because it is a run-time privilege issue.

> Both these parts are tantamount to the local machine's sysadmin
> declaring that she trusts a particular remote service to build correct
> systemtap modules, and is willing to permit running them (via the
> local setuid "staprun" launcher).  The remote service must of course
> distrust any data sent by the client, and emit code to verify anything
> relevant (uid / stapdev group membership, versions/buildids).
> 
> How to identify the trusted remote service?  One natural way is via a
> keypair as in ssh or gpg: have the compile server sign its output
> module with a private key; let the (setuid) staprun check the
> signature against a sysadmin-approved list in a file like
> /etc/systemtap/authorized_keys.  (openssl is probably the right
> toolkit for this stuff.)

IMHO, ssh is better approach, because it becomes the basic function
for remote access now, so we may not need to setup something special
on the server and the client.

I also think if we can use ssh to get compiled module from server,
we don't need special server program (this means we don't need to open
any port for server program). User authentication is also easy.
If you need more sophisticated system, ssh also supports '-s'(subsystem)
option like sftp.(I think it is enough to call a shellscript which
gets parameters, and returns module or result.)


Thanks,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: security for systemtap compiler server

[Frank Ch. Eigler]

Masami Hiramatsu <mhiramat@redhat.com> writes:

> [...]
>> Second, it is part of enabling unprivileged users to run systemtap
>> scripts that are severely restricted (no kernel probes; only probes on
>> one's own processes; that sort of thing).  (Allowing an unprivileged
>> user to build his own kernel modules via gcc etc., opens up too many
>> possibilities for subversion of the constraints.)
>
> I think this is also the issue of systemtap itself, not only
> compiler server, because it is a run-time privilege issue.

It is *both*.  A sysadmin may want to permit an unprivileged user to
run systemtap scripts, but only if they are built by a trusted
toolchain (so that it will compile in the proper constraints).  Such a
user cannot be build a trustworthy .ko himself -- after all, there is
no proof that any old .ko was even created by systemtap.


> [...]  IMHO, ssh is better approach, because it becomes the basic
> function for remote access now, so we may not need to setup
> something special on the server and the client.

While ssh could conceivably operate as the wire transport layer, we
still need something above it to (a) bundle any tapsets requested by
the end-user via -Ipath/ flags; (b) invoke the toolchain in a
trustworthy (unmodifiable) manner; (c) cause the resulting module to
be reliably crypto-signed; (d) get back all the results - .c/.ko,
stdout/stderr, exit-rc.  A plain "ssh SERVER stap -p4 ..."  wouldn't
accomplish these.


- FChE

Re: security for systemtap compiler server

[Masami Hiramatsu]

Hi,

Frank Ch. Eigler wrote:
> Masami Hiramatsu <mhiramat@redhat.com> writes:
> 
>> [...]
>>> Second, it is part of enabling unprivileged users to run systemtap
>>> scripts that are severely restricted (no kernel probes; only probes on
>>> one's own processes; that sort of thing).  (Allowing an unprivileged
>>> user to build his own kernel modules via gcc etc., opens up too many
>>> possibilities for subversion of the constraints.)
>> I think this is also the issue of systemtap itself, not only
>> compiler server, because it is a run-time privilege issue.
> 
> It is *both*.  A sysadmin may want to permit an unprivileged user to
> run systemtap scripts, but only if they are built by a trusted
> toolchain (so that it will compile in the proper constraints).  Such a
> user cannot be build a trustworthy .ko himself -- after all, there is
> no proof that any old .ko was even created by systemtap.

Indeed, there is no proof.
Hmm, if staprun needs to check whether .ko is genuine or not,
there may be same problem when using pre-compiled module.
In my thought, we might have to introduce 2-way authentication;
 - user/group based authentication for stap/staprun.
 - crypt-sign based authentication for staprun.

>> [...]  IMHO, ssh is better approach, because it becomes the basic
>> function for remote access now, so we may not need to setup
>> something special on the server and the client.
> 
> While ssh could conceivably operate as the wire transport layer, we
> still need something above it to (a) bundle any tapsets requested by
> the end-user via -Ipath/ flags; (b) invoke the toolchain in a
> trustworthy (unmodifiable) manner; (c) cause the resulting module to
> be reliably crypto-signed; (d) get back all the results - .c/.ko,
> stdout/stderr, exit-rc.  A plain "ssh SERVER stap -p4 ..."  wouldn't
> accomplish these.

I think some of them can be accomplished by ssh with some scripts
(not plain ssh :-)). But, yeah, (b) is hard, especially, over network...

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: security for systemtap compiler server

[anithra]

Hi Frank,

Frank Ch. Eigler wrote:
> Masami Hiramatsu <mhiramat@redhat.com> writes:
>
>   
>> [...]
>>     
>>> Second, it is part of enabling unprivileged users to run systemtap
>>> scripts that are severely restricted (no kernel probes; only probes on
>>> one's own processes; that sort of thing).  (Allowing an unprivileged
>>> user to build his own kernel modules via gcc etc., opens up too many
>>> possibilities for subversion of the constraints.)
>>>       
>> I think this is also the issue of systemtap itself, not only
>> compiler server, because it is a run-time privilege issue.
>>     
>
> It is *both*.  A sysadmin may want to permit an unprivileged user to
> run systemtap scripts, but only if they are built by a trusted
> toolchain (so that it will compile in the proper constraints).  Such a
> user cannot be build a trustworthy .ko himself -- after all, there is
> no proof that any old .ko was even created by systemtap.
>
>
>   
>> [...]  IMHO, ssh is better approach, because it becomes the basic
>> function for remote access now, so we may not need to setup
>> something special on the server and the client.
>>     
>
> While ssh could conceivably operate as the wire transport layer, we
> still need something above it to (a) bundle any tapsets requested by
> the end-user via -Ipath/ flags; (b) invoke the toolchain in a
> trustworthy (unmodifiable) manner; (c) cause the resulting module to
> be reliably crypto-signed; (d) get back all the results - .c/.ko,
> stdout/stderr, exit-rc.  A plain "ssh SERVER stap -p4 ..."  wouldn't
> accomplish these.
>
>
> - FChE
>   
 From what i understand the security requirements for the 
compiler-server and the stapgui-server(with remote-execution& 
compilation) are similiar, in which case is there anything else that 
prevents us from extending the compiler-server for remote-execution?.

Regards,
Anithra.

Re: security for systemtap compiler server

[Frank Ch. Eigler]

anithra wrote:

> [...] From what i understand the security requirements for the
> compiler-server and the stapgui-server (with remote-execution&
> compilation) are similiar [...]

They are really complementary and composable rather than similar.
Remote execution needs to authenticate the user (client), whereas
remote compilation needs to authenticate the compiler (server).

- FChE

Re: security for systemtap compiler server

[Dave Brolley]

Frank Ch. Eigler wrote:
> anithra wrote:
>
>   
>> [...] From what i understand the security requirements for the
>> compiler-server and the stapgui-server (with remote-execution&
>> compilation) are similiar [...]
>>     
>
> They are really complementary and composable rather than similar.
> Remote execution needs to authenticate the user (client), whereas
> remote compilation needs to authenticate the compiler (server).
>
>   
I tend to agree with Frank on this point. The combination of these 
separately valuable capabilities would accomplish what you need.

Dave

Re: security for systemtap compiler server

[Roland McGrath]

Btw, I gather that NSS is supposed to be the preferred and one true toolkit
for all crypto things now.  But I don't really know anything about it and
surely our needs are normal enough that the security folks with clue will
quickly direct us to exactly what to use in each implementation language
environment.

Two intertwined angles related to security come to mind off hand.

First is securing user from a bogus remote compile server, as well as the
system.  That is, only sending user data (i.e. the script) on a private
channel authenticated as the trusted server.

Second is "live" (session) security vs "batch" security.  Your descriptions
implied some things about this, but didn't make them explicit.

The user's data security is only about the live session security.  You want
to be sure your channel is private and is to a trusted server before you
send it any user data at all.  TLS/SSL alone does this, if used in the
appropriate way (client verifies expected server keys).  I don't know it's
easy to do that with stunnel, but it's surely easy enough with the
facilities to be an SSL client in whatever other environment fancier than
sh (perl, python, C, etc).

I think Frank implied "batch" security, i.e. signed module nuggets a la
gpg-signed files/messages.  I presume this is necessary too, because it
enables so many things we probably want: normal user files for module
caching, minimal privileged staprun separate from the app that talks to the
compile server.

So the "trust vectors" I see are:

1. Normal user trusts session to remote compile server with user's own data,
   that it will complete/fail properly, not produce a gargantuan result, etc.
   (Here "user" means unprivileged user process running stap-client/stap,
   unprivileged apps of same euid that invoked it, person who asked for that.)

2. (reciprocal) server trusts session from remote user to be permitted,
    vs DoS, unavoidable leaks of server config details, etc.

3. Local privileged staprun process trusts artifact based on crypto
   signature and local config of authorized module signing keys.
   i.e. signed module nugget supplied by untrusted/unprivileged user


#2 says paranoid admin might want to have various forms of access control
(ranging from user SSL key ACLs to nothing but "you can firewall it")

If #3 were going to be the only thing ever authenticating a server, then it
could be cobbled together in a scripty way with gpg quite easily.  

For #1, probably anyone who does not care about authenticity also does not
really care about privacy.  Simple cobbling together with stunnel gives us
privacy, but might not give us authenticity, and if it did certainly adds
to the key management headache.  So off hand I suspect that stunnel alone
is not worth bothering with.

If both ends are already shell scripts, then we can cobble something
demo-ready using gpg in about three minutes (and then spend the rest of the
month figuring out what exactly to cobble for the server public key
management on the client).  But for a long-run "real" plan, I tend to think
we'll need enough of something more real in dept #1 that I wouldn't bother
with a gpg hack.

If there will be authentication of a server's key in an SSL session for #1
and also authentication of a server's signed nugget in a minimal local
privilged program for #3, then it sure pays to have the key management
unified.  I'm pretty sure that the proper toolkits like NSS (and its
cousins and wrappers in various languages) make it pretty easy to do simple
batch-style signing and verifying using the same keys that the SSL/TLS
layer uses.


Thanks,
Roland