Network Security for the Systemtap Client/Server

Network Security for the Systemtap Client/Server

[Dave Brolley]

[-- Attachment #1: Type: text/plain, Size: 38 bytes --]

Hi,

See the attached file...


Dave


[-- Attachment #2: client-server-security.txt --]
[-- Type: text/plain, Size: 9833 bytes --]

Here is a proposal for security for the systemtap client and server. Please
poke holes in it and correct me where I am mistaken or where I have missed
something. As always, ideas, suggestions, corrections and concerns are
encouraged!

Toolkit
-------
Network Security Services (NSS), http://www.mozilla.org/projects/security/nss,
provides command line tools as well as an API for providing SSL services
such as encrypted network connections, client/server authentication and
for object signing.

openSSL, http://www.openssl.org, also provides a toolkit and API for these
services.

It would appear that the library/tools of choice for cryptography, going
forward on Fedora and RHEL is Network Security Services (NSS). This is mainly
due to superior certification history and completeness. See

http://fedoraproject.org/wiki/CryptoConsolidation
http://fedoraproject.org/wiki/CryptoConsolidationEval

Wire Level Security
-------------------
Wire level security is necessary to prevent spoofing (impersonation) of
systemtap servers as well as eavesdropping on the data stream
between the client and server (man in the middle).

Authentication of clients is also possible although may not be necessary.
The systemtap server is simply compiling the provided scripts and is not
returning any information which could not be obtained using the resources
used to perform the compilation. These resources (kernel version and
debuginfo, systemtap itself) are already widely available.
**** Comments? concerns? suggestions? ***

SSL network connections, as provided by NSS, provide this level of security.

I have identified two possibilities for implementation:

1) Use stunnel to provide SSL connections between the existing stap-client and
   stap-server(d) scripts. stunnel is easy to use and would require few, if any,
   changes to the existing scripts. On the downside, stunnel is currently based
   on openSSL and uses the .pem file format for certificates and keys, which
   differs from the password protected key/certificate databases used by NSS.
   stunnel has been identified as a tool to be converted to use NSS

     http://fedoraproject.org/wiki/CryptoConsolidation#Packages_to_NSS-enable

   but there is no indication of when or if this will happen.

2) Use the NSS C language API for SSL from within stap to communicate with
   the server. The stap-client script would still create the request as a
   directory tree as it does now and would still package the request into an
   archive (currently uses tar). It would then call stap, passing it the name of
   the archive. stap would then send it to the server and receive the result
   archive using an NSS SSL connection. The stap-client would then unpack the
   archive and process it as it does today.

   This approach has the advantages that it uses NSS immediately. This means we
   need not rely on migration of stunnel to NSS actually ocurring. We can also
   use the same client/server certificates for both signing and for secure
   connections.

   **** Is it more secure to use separate certificates for each? ****

3) Something else? Suggestions?

Module Signing
--------------
It is desirable that the client be able to verify that the data returned from
the server has not been tampered with. This can be accomplished by digitally
signing the data. Once again NSS provides facilities for signing and
verifying arbitrary data.

The NSS tool 'signtool' is able to sign a directory tree using a certificate
and keypair which it can also generate. Certificates and key pairs can also
be generated using the NSS tool 'certutil'. signtool signs the directory and
compresses it (using the zip utility) into a single file called a .jar file.
The .jar file can then be verified also using signtool.

Since the stap server's response to the client is a directory tree, it can be
signed in its entirety. The resulting .jar file could then be sent back to the
client, allowing the client to ensure that the entire response has not been
tampered with.

Administration
--------------
Here is how a sysadmin would administer a trusted systemap server. All
commands below must be executed as the userid which will run the server.

1) The sysadmin creates an NSS certificate/key database using the commands

     touch <passwordfile>
     chmod 600 <passwordfile>
     echo "password" > <passwordfile>
     mkdir -p <dirname>
     certutil -N -d <dirname> -f <passwordfile>

   This creates an empty certificate/key database in the directory named by
   <dirname>. The password needed to access the private key(s) in the database
   is obtained from <passwordfile>, which is a plain text file, so care must
   be taken to prevent unauthorized access to this file. Use of a password
   file will allow the systemtap server to access the private key(s) in the
   database without being prompted.

2) The sysadmin then creates a certificate and its associated public/private
   key pair and adds them to the database for the server using

     signtool -G <certname> -d <dirname> -f <passwordfile>

   where <certname> is a nickname used to refer to the certificate/keys in the
   database. The sysadmin will be prompted for all information needed to
   create the certificate.

   The certificate created is a self-signed certificate (since it was not
   issued by a certificate authority) and is also a signing certificate (it
   can be used by NSS tools to sign arbitrary data). Because it is self-signed,
   each client must use a local copy in order to verify an SSL connection
   with a server or the signed data (see below).

3) The 'signtool -G' command also creates the file x509.cacert which has
   the MIME-type application/x-x509-ca-cert. This file can be installed
   in the certificate/key database of each client in order to allow each
   client to establish secure connections with the server and to verify the
   integrity of the server response. This is done on each client host by
   copying the file to the client host and then:

     mkdir -p <dirname>
     certutil -A -n <certname> -d <dirname> -i x509.cacert -t "P,P,P"

   This creates an NSS certificate/key database in <dirname> and adds a copy
   of the server's certificate and pulic key. The certificates of multiple
   servers can be added to the database. No password is required, since the
   database contains only the certificates and public keys of each server.
   This database must be accessible by all users who will be running the
   client on the client host. In addition to, or instead of a common
   certificate/key database, each user could have their own.

Implementation
--------------
Once the certificate databases have been set up on the client server, they would
be used as follows:

1) The client would use NSS services to establish a secure SSL connection with
   the server. During the SSL handshake, the client would verify the certificate
   presented by the server against its own local copy. This is necessary since
   the server's certificate is self-signed and thus can not be verified on its
   own (i.e. it is not signed by a recognized certificate authority).

2) The server would sign its response tree using its own certificate and
   private key as follows:

     signtool -Z <response-tree>.jar -d <dirname> -k <certname> -f <passwordfile> <response-tree>

   This generates the signed archive '<reponse-tree>.jar' which would be sent
   back to the client over the secure connection.

3) The client would verify the response using its own copy of the server's
   certificate and public key as follows:

     signtool -d <dirname> -v <response-tree>.jar

   Note that no password is required since only the server's certificate and
   public key are being used.

4) Once verified, the client can unpack the response using the unzip utility.

Notes
-----
o The use of self-signed certificates requires that each client maintain
  a database of the certificates and public keys of each server to be used. It
  is possible for a sysadmin to set up an internal certificate authority and to
  use the certificate and private key of that authority to sign the certificates
  of each server. The clients would then need only the certificate and public
  key of the internal certificate authority in order to verify any server on the
  network.

  signtool has the ability to verify such certificate authority chains.

  A combination of authorized servers and ad-hoc (self-signed) servers
  is also possible. The verification process would then be to attempt
  verification against the certificate authority certificate and then against
  local copies of ad-hoc server certificates.

o Although stunnel uses a different method of storing certificates and keys,
  it is possible to construct the required .pem files using the output of
  signtool and certutil. It is also possible to construct certificate authority
  chains for use by stunnel. Initially it is likely that stunnel will be used
  for secure SSL connections bewteen the client and server.

o The above model does not address authentication of clients by servers. At
  this point, I do not believe it is necessary (problems and concerns please!).
  However, client authentication is an optional part of the SSL handshake
  and the creation and maintenance of client certificates and keys is
  analogous to those of the server.

o One of the goals of the Fedora Crypto Consolidation Project

    http://fedoraproject.org/wiki/CryptoConsolidation

  is that all applications on a system access a common certification/key
  database. Using NSS, this will be possible once such a thing exists. However,
  this would lead to trusting systemtap servers on hosts which have been
  potentially entered into the database (i.e. trusted) for other reasons
  (e.g. SSL-enabled mail server), which may not be desirable.

Re: Network Security for the Systemtap Client/Server

[Frank Ch. Eigler]

Hi, Dave -

Thanks for your writeup.


> [...]
> Wire Level Security
> -------------------
> Wire level security is necessary to prevent spoofing (impersonation) of
> systemtap servers as well as eavesdropping on the data stream
> between the client and server (man in the middle).

The first of those threats is not so bad, considering module signing.
The second is not that bad.  But a third, modification of the script
en route to the server, is bad, and justifies efforts to secure the
wire.


> Authentication of clients is also possible although may not be
> necessary.  [...]

Agreed.


> [...]
> 2) Use the NSS C language API for SSL from within stap to communicate with
>    the server. The stap-client script would still create the request as a
>    directory tree [...] It would then call stap, passing it the name of
>    the archive. stap would then send it to the server [...]

It seems clumsy to have to use stap just so that it can do
stunnel-like data transfer.  Is there a perl binding to NSS, so that a
rewritten stap-client could use it.  Maybe nss_compat_ossl....


Or, really, we can buck the fedora trend briefly and go with plain
openssl (with stunnel and/or perl bindings) until NSS becomes
similarly functional.



> [...]
> Module Signing
> --------------
> It is desirable that the client be able to verify that the data returned from
> the server has not been tampered with. This can be accomplished by digitally
> signing the data. Once again NSS provides facilities for signing and
> verifying arbitrary data.

An extra complication here is that module signing is mainly there to
help enable future systemtap use by unprivileged users.  The trusted
server would impose extra constraints and return a probe module that
compiles in any additional checks needed.

The signature on such a module needs to be carried through right
through to staprun - beyond stap-client - since staprun is the setuid
piece that must ultimately decide whether to load/run a proposed
module or not.  It needs to perform the signature verification to
assure the local system(admin) that the possibly unprivileged user who
ran stap-client did not mess with the resulting module.

(Doing all this via the kernel module-signing code in some kernels
would not be appropriate, according to its author.)


> [...]
> Administration
> --------------
> Here is how a sysadmin would administer a trusted systemap server. All
> commands below must be executed as the userid which will run the server.
> [...]

All that must be automatable to death.  The wire protocol part's user
interface should be no clumsier than, say, svn talking to a https:
server.


The module signature verification is a whole separate mechanism.
There, the sysadmin-approved list of module-signing public keys need
to be installed into a special place ($etcdir/systemtap/server-keys),
which staprun would validate against.


- FCHE

Re: Network Security for the Systemtap Client/Server

[Dave Brolley]

Frank Ch. Eigler wrote:
>   
>> [...]
>> Wire Level Security
>> -------------------
>> Wire level security is necessary to prevent spoofing (impersonation) of
>> systemtap servers as well as eavesdropping on the data stream
>> between the client and server (man in the middle).
>>     
>
> The first of those threats is not so bad, considering module signing.
> The second is not that bad.  But a third, modification of the script
> en route to the server, is bad, and justifies efforts to secure the
> wire.
>   
If I understand correctly, the only way to ensure that the script has 
not been modified on route is to have the client sign it with its own 
certificate and private key. This can be easily done using the same 
techniques used by the server to sign its response. This, in my opinion 
is a separate issue from the two issues mentioned above.
>
>> [...]
>> 2) Use the NSS C language API for SSL from within stap to communicate with
>>    the server. The stap-client script would still create the request as a
>>    directory tree [...] It would then call stap, passing it the name of
>>    the archive. stap would then send it to the server [...]
>>     
>
> It seems clumsy to have to use stap just so that it can do
> stunnel-like data transfer.  Is there a perl binding to NSS, so that a
> rewritten stap-client could use it.  Maybe nss_compat_ossl....
>
>
> Or, really, we can buck the fedora trend briefly and go with plain
> openssl (with stunnel and/or perl bindings) until NSS becomes
> similarly functional.
>   
Agreed. I mentioned this in the notes at the end of the proposal.
>> [...]
>> Module Signing
>> --------------
>> It is desirable that the client be able to verify that the data returned from
>> the server has not been tampered with. This can be accomplished by digitally
>> signing the data. Once again NSS provides facilities for signing and
>> verifying arbitrary data.
>>     
>
> An extra complication here is that module signing is mainly there to
> help enable future systemtap use by unprivileged users.  The trusted
> server would impose extra constraints and return a probe module that
> compiles in any additional checks needed.
>
> The signature on such a module needs to be carried through right
> through to staprun - beyond stap-client - since staprun is the setuid
> piece that must ultimately decide whether to load/run a proposed
> module or not.  It needs to perform the signature verification to
> assure the local system(admin) that the possibly unprivileged user who
> ran stap-client did not mess with the resulting module.
>
> (Doing all this via the kernel module-signing code in some kernels
> would not be appropriate, according to its author.)
>   
With appropriate constraints applied by the server to ensure that an 
unprivileged client does not gain permission to load/run a maliciously 
designed module, I agree. Note that stap-client makes use of more than 
just the returned module. Output from stap on the server side is also 
used and this is why I'm proposing that the entire server response be 
signed. stap-client needs to know that no part of the server response 
has been tampered with.

Some possibilities for verification of the module by staprun on the 
client side are:

1) Separately sign the module within the signed server response, which 
seems a bit redundant to me given that the entire server response is 
already signed by the server and verified by the client.

2) Have staprun verify the entire server response again. Again some 
redundancy, but perhaps necessary given that staprun can not afford to 
trust that it was called by a legitimate client.

3) Something else?
>
>   
>> [...]
>> Administration
>> --------------
>> Here is how a sysadmin would administer a trusted systemap server. All
>> commands below must be executed as the userid which will run the server.
>> [...]
>>     
>
> All that must be automatable to death.  The wire protocol part's user
> interface should be no clumsier than, say, svn talking to a https:
> server.
>   
Right. Bear in mind that these are sysadmin tasks performed once for 
each client/server on the network. These tasks are analogous to using 
ssh-keygen to generate key pairs for ssh and adding the public 
identities to each machine one wants to access.
>
> The module signature verification is a whole separate mechanism.
> There, the sysadmin-approved list of module-signing public keys need
> to be installed into a special place ($etcdir/systemtap/server-keys),
> which staprun would validate against.
>
>   
Yes. I suggest that 'make' and 'make install' generate and install key 
pairs and certificates for development environments and that installing 
the stap-server rpm do the same for production environments. We should 
also create some helper scripts to help sysadmins add the public keys 
and certificates of their trusted servers to client machines properly.

Dave

> - FCHE
>   

Re: Network Security for the Systemtap Client/Server

[Frank Ch. Eigler]

Hi -

On Wed, Nov 05, 2008 at 04:53:25PM -0500, Dave Brolley wrote:

> >>Wire Level Security
> [...]
> If I understand correctly, the only way to ensure that the script has 
> not been modified on route is to have the client sign it with its own 
> certificate and private key. This can be easily done using the same 
> techniques used by the server to sign its response. [...]

That could well be an overkill.  Standard wire-level security like
TLS/SSL, without extra explicit signatures, should be sufficient for
protection against a hostile network.


> [...]
> Note that stap-client makes use of more than just the returned
> module. Output from stap on the server side is also used and this is
> why I'm proposing that the entire server response be
> signed. stap-client needs to know that no part of the server
> response has been tampered with.

That's a good point, but that's adequately addressed by wire-level
security.  It may help to consider the wire-level stuff orthogonal -
as indeed it is since we can theoretically connect stap-client and
stap-server with a local unix pipe.


> Some possibilities for verification of the module by staprun on the 
> client side are:
> 
> 1) Separately sign the module within the signed server response, which 
> seems a bit redundant to me given that the entire server response is 
> already signed by the server and verified by the client. [...]

Yes, but the client (stap-client) cannot be trusted by staprun.
staprun need only care that the final module is built correctly.


> >[...]
> >All that must be automatable to death.  The wire protocol part's user
> >interface should be no clumsier than, say, svn talking to a https:
> >server.
>
> Right. Bear in mind that these are sysadmin tasks performed once for 
> each client/server on the network. These tasks are analogous to using 
> ssh-keygen to generate key pairs for ssh and adding the public 
> identities to each machine one wants to access.

I don't see a need yet for client-side *authentication* that might
necessitate signing keys there, so that leaves only the ssh-keygen
part.

So, for module-signing purposes, sshd's host key is analogous to
stap-server's signing key, and ssh's $HOME/.ssh/known_keys (treated
more like authorized_keys) is analogous to the staprun's approved keys
list.


- FChE

Re: Network Security for the Systemtap Client/Server

[Dave Brolley]

Frank Ch. Eigler wrote:
> Hi -
>
> On Wed, Nov 05, 2008 at 04:53:25PM -0500, Dave Brolley wrote:
>
>   
>>>> Wire Level Security
>>>>         
>> [...]
>> If I understand correctly, the only way to ensure that the script has 
>> not been modified on route is to have the client sign it with its own 
>> certificate and private key. This can be easily done using the same 
>> techniques used by the server to sign its response. [...]
>>     
>
> That could well be an overkill.  Standard wire-level security like
> TLS/SSL, without extra explicit signatures, should be sufficient for
> protection against a hostile network.
>   
I'll let you make the call on sufficiency. However while a SSL/TLS 
connection provides server authentication and encryption, I still don't 
believe that it alone protects against tampering. Search for "tampering" 
in the following page:

https://developer.mozilla.org/en/Introduction_to_Public-Key_Cryptography#Internet_Security_Issues
>
>   
>> [...]
>> Note that stap-client makes use of more than just the returned
>> module. Output from stap on the server side is also used and this is
>> why I'm proposing that the entire server response be
>> signed. stap-client needs to know that no part of the server
>> response has been tampered with.
>>     
>
> That's a good point, but that's adequately addressed by wire-level
> security.  It may help to consider the wire-level stuff orthogonal -
> as indeed it is since we can theoretically connect stap-client and
> stap-server with a local unix pipe.
>   
Once again, I believe that if we're worried about tampering, then 
signing is the way to protect against it.
>
>   
>> Some possibilities for verification of the module by staprun on the 
>> client side are:
>>
>> 1) Separately sign the module within the signed server response, which 
>> seems a bit redundant to me given that the entire server response is 
>> already signed by the server and verified by the client. [...]
>>     
>
> Yes, but the client (stap-client) cannot be trusted by staprun.
> staprun need only care that the final module is built correctly.
>   
So are you preferring the option above over option 2 (staprun 
re-verifies the entire response)?
>>> [...]
>>> All that must be automatable to death.  The wire protocol part's user
>>> interface should be no clumsier than, say, svn talking to a https:
>>> server.
>>>       
>> Right. Bear in mind that these are sysadmin tasks performed once for 
>> each client/server on the network. These tasks are analogous to using 
>> ssh-keygen to generate key pairs for ssh and adding the public 
>> identities to each machine one wants to access.
>>     
>
> I don't see a need yet for client-side *authentication* that might
> necessitate signing keys there, so that leaves only the ssh-keygen
> part.
>
> So, for module-signing purposes, sshd's host key is analogous to
> stap-server's signing key, and ssh's $HOME/.ssh/known_keys (treated
> more like authorized_keys) is analogous to the staprun's approved keys
> list.
>   
We're in agreement here.

Dave

Re: Network Security for the Systemtap Client/Server

[Frank Ch. Eigler]

Hi -

On Thu, Nov 06, 2008 at 01:33:06PM -0500, Dave Brolley wrote:
> [...]
> >>[...]
> >>If I understand correctly, the only way to ensure that the script has 
> >>not been modified on route is to have the client sign it with its own 
> >>certificate and private key. [...]
> >
> >That could well be an overkill.  Standard wire-level security like
> >TLS/SSL, without extra explicit signatures, should be sufficient for
> >protection against a hostile network.
> >  
> I'll let you make the call on sufficiency. However while a SSL/TLS 
> connection provides server authentication and encryption, I still don't 
> believe that it alone protects against tampering. Search for "tampering" 
> in the following page:
> 
> https://developer.mozilla.org/en/Introduction_to_Public-Key_Cryptography#Internet_Security_Issues

The overall SSL/TLS protocol (via encryption and other stuff) does just that:

# Once the server has been authenticated, the client and server use
# techniques of symmetric-key encryption, which is very fast, to
# encrypt all the information they exchange for the remainder of the
# session and to detect any tampering that may have occurred.


> >Yes, but the client (stap-client) cannot be trusted by staprun.
> >staprun need only care that the final module is built correctly.
>
> So are you preferring the option above over option 2 (staprun 
> re-verifies the entire response)?

No, I'm suggesting that stap-client need not verify the response at
all, assuming that wire-level security was in place.


- FChE

Re: Network Security for the Systemtap Client/Server

[Dave Brolley]

Frank Ch. Eigler wrote:
> # Once the server has been authenticated, the client and server use
> # techniques of symmetric-key encryption, which is very fast, to
> # encrypt all the information they exchange for the remainder of the
> # session and to detect any tampering that may have occurred.
>
>   
I didn't see the assertion of tamper protection here when I read it the 
first time. It would seem to me that any kind encryption alone can not 
provide protection against tampering. However, perhaps the assertion is 
that only the encryption is weakened at this point and that other 
techniques, such as signing all or part of the data, continue to be 
employed.
>>
>> So are you preferring the option above over option 2 (staprun 
>> re-verifies the entire response)?
>>     
>
> No, I'm suggesting that stap-client need not verify the response at
> all, assuming that wire-level security was in place.
>   
Assuming that what you quoted above is correct, then I now agree.

Dave

Re: Network Security for the Systemtap Client/Server

[Frank Ch. Eigler]

Hi -

On Fri, Nov 07, 2008 at 11:37:03AM -0500, Dave Brolley wrote:
> ># Once the server has been authenticated, the client and server use
> ># techniques of symmetric-key encryption, which is very fast, to
> ># encrypt all the information they exchange for the remainder of the
> ># session and to detect any tampering that may have occurred.
>
> I didn't see the assertion of tamper protection here when I read it the 
> first time. It would seem to me that any kind encryption alone can not 
> provide protection against tampering. However, perhaps the assertion is 
> that only the encryption is weakened at this point and that other 
> techniques, such as signing all or part of the data, continue to be 
> employed.

In SSL and similar wire-level security protocols, simple encryption is
only one part of the work.  There are checksums, authentication codes,
and other gunk being passed back and forth, so that the channel can be
deemed "secure" - meaning more than just "confidential" but also
"tamper-proof" and some other properties.

- FChE

Re: Network Security for the Systemtap Client/Server

[Roland McGrath]

> The signature on such a module needs to be carried through right
> through to staprun - beyond stap-client - since staprun is the setuid
> piece that must ultimately decide whether to load/run a proposed
> module or not.  It needs to perform the signature verification to
> assure the local system(admin) that the possibly unprivileged user who
> ran stap-client did not mess with the resulting module.
> 
> (Doing all this via the kernel module-signing code in some kernels
> would not be appropriate, according to its author.)

However, carrying a similar signature in a similar fashion could work just
fine.  That is, at build time, add a new ELF section to the .ko file
containing the signature bits.  Then, check this signature against the .ko
file's contents before loading the module.  Our check can be in staprun,
rather than inside the kernel itself as the old modsign scheme does it.

Both the signing phase and the checking phase are pretty easy to implement,
either crudely with scripts around objcopy or fairly cleanly via libelf and
the NSS libraries.


Thanks,
Roland

Re: Network Security for the Systemtap Client/Server

[Roland McGrath]

> Authentication of clients is also possible although may not be necessary.
> The systemtap server is simply compiling the provided scripts and is not
> returning any information which could not be obtained using the resources
> used to perform the compilation. These resources (kernel version and
> debuginfo, systemtap itself) are already widely available.
> **** Comments? concerns? suggestions? ***

One can imagine a compiling server having available local private kernel
builds whose internal layout details someone's policy might consider
security-sensitive.  Leave the policy on authentication requirements to the
local admins to choose.  Just make out-of-the-box defaults be easy to use
without infrastructure set-up even when that means less stringent security
policies than a concerned admin could configure.

Also, keep in mind the long-run need to mesh with sophisticated key
management infrastructure a particular installation might have and want to
use it (be required by local security regulations) for everything crypto.
I'm not suggesting you dwell on this especially while just getting
something going for casual set-up and use.  Probably any implementation
that can be scripted around somehow using NSS command line tools will fully
meet that requirement down the line without much trying.  But keep it in my
in the approaches taken and how accessible the system is to configuration
of how keys are acquired and used, etc.


Thanks,
Roland