Need some security advice for systemtap

Need some security advice for systemtap

[David Smith]

I need some security development help (and this might be the wrong list 
- if so, please point me in the right direction).  I'm one of the 
systemtap developers.  You can see <http://sourceware.org/systemtap/> 
for details, but a 2 second overview is that systemtap allows users to 
write a script that probes points in the kernel.  systemtap takes the 
script, converts it into C, compiles the C into a kernel module, inserts 
the kernel module, and displays any output from the compiled script. 
When the script finishes, we remove the kernel module.

One of the complaints we get from users is that we require root access 
(using sudo) to install/remove the kernel module.  Large enterprise 
customers typically don't give out sudo access to all admins.  So, they 
would like a way to designate certain scripts/modules as "blessed", and 
allow admins/developers/etc. without root access to run those "blessed" 
scripts/modules.

Some basic ideas about how we can allow users without sudo access to run 
"blessed" scripts/modules can be seen at 
<http://sources.redhat.com/bugzilla/show_bug.cgi?id=4523>,

So, I'm looking for thoughts, criticisms, pointers, etc. to do this in a 
  manner that won't allow a system to be easily compromised.  We're in 
the fairly early stages of this idea, and I'm looking for direction 
before heading down the wrong road.

Thanks for the help.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[Tomasz Chmielewski]

David Smith schrieb:

(...)

> Some basic ideas about how we can allow users without sudo access to run 
> "blessed" scripts/modules can be seen at 
> <http://sources.redhat.com/bugzilla/show_bug.cgi?id=4523>,
> 
> So, I'm looking for thoughts, criticisms, pointers, etc. to do this in a 
>  manner that won't allow a system to be easily compromised.  We're in 
> the fairly early stages of this idea, and I'm looking for direction 
> before heading down the wrong road.

Am I right? Is it security based on md5sum?

I'm not sure how easy would it be to "produce" two kernel modules having 
the same MD5 checksum - but before you continue, you might want to read 
a short article called "Attacking Hash Functions by Poisoned Messages":

http://www.cits.rub.de/MD5Collisions/


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Need some security advice for systemtap

[Frank Ch. Eigler]

Tomasz Chmielewski <mangoo@wpkg.org> writes:

> Am I right? Is it security based on md5sum? [...]
> http://www.cits.rub.de/MD5Collisions/

Indeed.

md5 was just a "for sake of argument" possibility.  We can go with a
different or even multiple hashes.  Or, if hashes are deemed
untrustworthy in some particular installation,
/etc/systemtap/authorized_probes could contain copies of the probe
modules in their entirety rather than their hashes.

- FChE

Re: Need some security advice for systemtap

[David Smith]

Tomasz Chmielewski wrote:
> David Smith schrieb:
> 
> (...)
> 
>> Some basic ideas about how we can allow users without sudo access to 
>> run "blessed" scripts/modules can be seen at 
>> <http://sources.redhat.com/bugzilla/show_bug.cgi?id=4523>,
>>
>> So, I'm looking for thoughts, criticisms, pointers, etc. to do this in 
>> a  manner that won't allow a system to be easily compromised.  We're 
>> in the fairly early stages of this idea, and I'm looking for direction 
>> before heading down the wrong road.
> 
> Am I right? Is it security based on md5sum?

That was the basic idea.  It would be easy enough to substitute a better 
hash function - I'm guessing one of the shaXXXsum would be more robust.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[grundy]

I think a good way to handle it would be to have a configuration file
like /etc/sudoers and setuid root stap (or staprun). The access control
would then be built into systemtap. 

Here are my ideas of what would make a "good" set of controls:

 - level of tap script they can run, e.g. guru mode code or not
 - sections of the kernel they can access (maybe this is
   better represented as what tapsets may they use)
 - how much overhead are they allowed to put on the system
 - are they allowed to look at data for other user's processes
 - are they allowed to reference line #'s or direct memory addrs

I think that it would probably mean having a list created at compile
time that indicates what things the resulting modules do. staprun would
then create a permissions "mask" for the user and compare it to what is
in the module (yeah, ummm, magic happens here?) or maybe when the module
loads it looks at it's internal list and the permissions of the calling
user (passed in by staprun) and decides if it will run or not.

Thanks
Mike

Re: Need some security advice for systemtap

[David Smith]

grundy wrote:
> I think a good way to handle it would be to have a configuration file
> like /etc/sudoers and setuid root stap (or staprun). The access control
> would then be built into systemtap. 
> 
> Here are my ideas of what would make a "good" set of controls:
> 
>  - level of tap script they can run, e.g. guru mode code or not
>  - sections of the kernel they can access (maybe this is
>    better represented as what tapsets may they use)
>  - how much overhead are they allowed to put on the system
>  - are they allowed to look at data for other user's processes
>  - are they allowed to reference line #'s or direct memory addrs

That sounds nice, but I'm worried about implementing such a feature 
correctly, on at least two levels.  First, you assume that systemtap can 
correctly characterize the effects a script will have on the system. 
Then you want to add an ACL system into systemtap based on those effects.

One advantage the proposed system has is that there *is* a human in the 
loop, a root user who will (hopefully) look at a script and check it out 
before "blessing" it.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[Pavel Kankovsky]

On Mon, 4 Jun 2007, David Smith wrote:

> One of the complaints we get from users is that we require root access
> (using sudo) to install/remove the kernel module.  Large enterprise
> customers typically don't give out sudo access to all admins.  So, they
> would like a way to designate certain scripts/modules as "blessed", and
> allow admins/developers/etc. without root access to run those "blessed"
> scripts/modules.

The easiest way to designate a certain script as "blessed" (and quite
difficult to goof it up) is to let a trusted user take the source of the
script, check it, make a kernel module, and install it into a designated
directory on the target system (or systems), perhaps under
"/lib/modules/$version".

This is similar to keeping a list of allowed hashes as proposed on
Bugzilla but simpler. You need the compiled code to compute the hash
anyway.

If you are afraid of allowing "untrusted admins" run "sudo modprobe
the_probe" (but not afraid of allowing to run your own setuid root
program) you can create something like "sustaprun" that will make it
possible to load blessed modules (from the designated directory) only.

You might also exploit module autoloading (something like adding "alias
binfmt-1234 the_probe" to modprobe.conf and making an attempt to execute a
file starting with 1234) but it would make it difficult to grant different
users the ability to load different sets of modules and to remove modules.


On Tue, 5 Jun 2007, grundy wrote:

> I think a good way to handle it would be to have a configuration file
> like /etc/sudoers and setuid root stap (or staprun). The access control
> would then be built into systemtap. 

It is an attractive idea but I am afraid it will turn out to be a sort of
conceptual minefield when you start working on it.

> Here are my ideas of what would make a "good" set of controls:
> 
>  - level of tap script they can run, e.g. guru mode code or not

Guru mode means the ability to run arbitrary C code in the kernel space.
It is easier to give them the root password.

>  - sections of the kernel they can access (maybe this is
>    better represented as what tapsets may they use)

I think the history has taught us it is much better to express policy
rules in the terms of elementary operations than in the terms of
procedures, modules or whole programs. The latter choice turns the
procedures etc. into trusted components--correct enforcement of policy
relies on their correct behaviour, and this is a very bad thing.

Unfortunately, any block of embedded C code is a black box (and it 
has to power to turn anything in its vicinity into black boxes).

One reasonable approach might be to form a group of trusted tapsets
allowed to use embedded C code (and perhaps other kinds of privileged
operations) even without guru mode. They would form a kind of "TCB". The
policy would provide a set of trusted tapsets (and perhaps additional
restrictions) and the script and any untrusted tapsets it uses would have
to obey it.

>  - how much overhead are they allowed to put on the system

This is not a property of an individual script. A million of tiny 
lightweight scripts can kill the system as well as (and perhaps even 
better than) a single heavyweight script.

>  - are they allowed to look at data for other user's processes

You'd have to deal with the fact that many useful predefined functions
(kernel_string() just to name one example) appear to take arbitrary
numbers, interpret them as pointers, and fetch a value at that address,
making it possible to read all kernel memory.


--Pavel Kankovsky aka Peak  [ Boycott Microsoft--http://www.vcnet.com/bms ]
"Resistance is futile. Open your source code and prepare for assimilation."

Re: Need some security advice for systemtap

[David Smith]

Thanks for the response.  See stuff below.

Pavel Kankovsky wrote:
> On Mon, 4 Jun 2007, David Smith wrote:
> 
>> One of the complaints we get from users is that we require root access
>> (using sudo) to install/remove the kernel module.  Large enterprise
>> customers typically don't give out sudo access to all admins.  So, they
>> would like a way to designate certain scripts/modules as "blessed", and
>> allow admins/developers/etc. without root access to run those "blessed"
>> scripts/modules.
> 
> The easiest way to designate a certain script as "blessed" (and quite
> difficult to goof it up) is to let a trusted user take the source of the
> script, check it, make a kernel module, and install it into a designated
> directory on the target system (or systems), perhaps under
> "/lib/modules/$version".

That's my current thinking.  Seems easier from a user's point of view 
and less fragile from a security point of view.

> If you are afraid of allowing "untrusted admins" run "sudo modprobe
> the_probe" (but not afraid of allowing to run your own setuid root
> program) you can create something like "sustaprun" that will make it
> possible to load blessed modules (from the designated directory) only.

That's also my current thinking.  I'm trying to prototype this now.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[David Smith]

David Smith wrote:
> Thanks for the response.  See stuff below.
> 
> Pavel Kankovsky wrote:
>> On Mon, 4 Jun 2007, David Smith wrote:
>>
>>> One of the complaints we get from users is that we require root access
>>> (using sudo) to install/remove the kernel module.  Large enterprise
>>> customers typically don't give out sudo access to all admins.  So, they
>>> would like a way to designate certain scripts/modules as "blessed", and
>>> allow admins/developers/etc. without root access to run those "blessed"
>>> scripts/modules.
>>
>> The easiest way to designate a certain script as "blessed" (and quite
>> difficult to goof it up) is to let a trusted user take the source of the
>> script, check it, make a kernel module, and install it into a designated
>> directory on the target system (or systems), perhaps under
>> "/lib/modules/$version".
> 
> That's my current thinking.  Seems easier from a user's point of view 
> and less fragile from a security point of view.
> 
>> If you are afraid of allowing "untrusted admins" run "sudo modprobe
>> the_probe" (but not afraid of allowing to run your own setuid root
>> program) you can create something like "sustaprun" that will make it
>> possible to load blessed modules (from the designated directory) only.
> 
> That's also my current thinking.  I'm trying to prototype this now.

After playing around with this a bit, here are my thoughts/plans.  We've 
basically got 2 problems:

(1) we want _certain_ non-privileged users
(2) to be able to run _certain_ compiled scripts

Solving both problems would look like this:

(A) A sysadmin would compile systemtap tap scripts into kernel modules 
and store the module in something like 
/etc/systemtap/authorized_modules/$kernel_version/foo.ko

(B) The sysadmin would add anyone needing to be able to run those 
modules to the new 'systemtap' group.

(C) We'll have a new program, staprun.auth, which would be owned by 
root, group systemtap, and file permission would be 04110.  Here's what 
a 'ls -l' would look like on it:

---s--x--- 1 root systemtap {size} {date} /usr/bin/staprun.auth

This would mean that only people in the systemtap group could run 
staprun.auth and it would give them root privileges.  staprun.auth would 
have similar arguments as staprun (more on that in a sec).  staprun.auth 
would make sure the module is coming from 
/etc/systemtap/authorized_modules/$kernel_version and then just exec 
staprun.

(D) staprun.auth will need to disallow certain staprun.auth command-line 
arguments, such as:

- "-c CMD" (Command CMD will be run and the staprun program will exit 
when CMD does).  We can't allow this since CMD would be run as root 
("staprun.auth -c xterm module" would be a quick way to a root shell).

- "-O FILE" (Send output to FILE).  Since FILE gets created as root, 
that could be used by an user seeking to overwrite a file he couldn't 
normally.


 From a users point of view, it would look something like this:

(as root, installing a new module)
# stap -k -p4 -m stap_foo foo.stp
Keeping temporary directory "/tmp/stapygupi1"
# cp /tmp/stapygupil/stap_foo.ko 
/etc/systemtap/authorized_modules/`uname -r`/stap_foo.ko

(as non-root user, running the new module)
$ groups
users systemtap
(ah, good, this user is in the 'systemtap' group, which means he should 
be able to run staprun.auth)
$ staprun.auth stap_foo.ko
(staprun.auth will make sure /etc/systemtap/authorized_modules/`uname 
-r`/stap_foo.ko exists, then exec staprun with that module)

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[Frank Ch. Eigler]

David Smith <dsmith@redhat.com> writes:

> [...]
> Solving both problems would look like this:
> 
> (A) A sysadmin would compile systemtap tap scripts into kernel modules
> and store the module in something like
> /etc/systemtap/authorized_modules/$kernel_version/foo.ko

The suggestion of using /lib/modules itself is a great one.

> (B) The sysadmin would add anyone needing to be able to run those
> modules to the new 'systemtap' group.

Good idea.

> (C) We'll have a new program, staprun.auth, which would be owned by
> root, group systemtap, and file permission would be 04110.  Here's
> what a 'ls -l' would look like on it:
> 
> ---s--x--- 1 root systemtap {size} {date} /usr/bin/staprun.auth

OK.

> [...]
> (D) staprun.auth will need to disallow certain staprun.auth
> command-line arguments, such as:
> - "-c CMD" [...]
> - "-O FILE" [...]

Actually, it doesn't.  A setuid program can drop its privileges after
performing the root-only operations (module loading), and invoke the
rest of the normal commands as the real userid.

> [...]
> $ staprun.auth stap_foo.ko
> (staprun.auth will make sure /etc/systemtap/authorized_modules/`uname
> -r`/stap_foo.ko exists, then exec staprun with that module)
> [...]

It would be better to have staprun.auth perform the module loading and
setup parts of current staprun, and defer the unprivileged work to
ordinary staprun.

Now, if only we could automate the sudo vs. setuid mechanisms...
Maybe even SYSTEMTAP_DIR=/lib/modules/`uname -r`/systemtap?

- FChE

Re: Need some security advice for systemtap

[David Smith]

Frank Ch. Eigler wrote:
> David Smith <dsmith@redhat.com> writes:
> 
>> [...]
>> Solving both problems would look like this:
>>
>> (A) A sysadmin would compile systemtap tap scripts into kernel modules
>> and store the module in something like
>> /etc/systemtap/authorized_modules/$kernel_version/foo.ko
> 
> The suggestion of using /lib/modules itself is a great one.

I'm OK with that.  From later in your email it looks like you are 
shooting for /lib/modules/`uname -r`/systemtap, which seems reasonable.

  >> [...]
>> (D) staprun.auth will need to disallow certain staprun.auth
>> command-line arguments, such as:
>> - "-c CMD" [...]
>> - "-O FILE" [...]
> 
> Actually, it doesn't.  A setuid program can drop its privileges after
> performing the root-only operations (module loading), and invoke the
> rest of the normal commands as the real userid.

Hmm.


I was trying to duplicate as little of staprun as possible - just parse 
arguments, make sure the module is in the correct place, then exec 
staprun for all the real processing.  I was trying to make staprun_auth 
a very thin wrapper around staprun.

With your idea I don't see a way around duplicating all of staprun (not 
actual code duplication, but compiling all of staprun into staprun_auth).

Perhaps there is a merged approach.  Keep staprun_auth a thin wrapper 
around staprun, but change staprun to raise and lower privileges as 
needed when inserting/removing modules, setting up relayfs, etc.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[Pavel Kankovsky]

On Mon, 11 Jun 2007, David Smith wrote:

> (D) staprun.auth will need to disallow certain staprun.auth command-line 
> arguments, such as:

Plus:

- "-u USERNAME" (assuming you allow -c, otherwise it has no effect;
  on the other hand, its support in staprun can help to to make it 
  possible to use -c via staprun.auth)

- "-t PID" (or "-x PID") (you need to disallow this because it is
  virtually impossible to avoid race conditions when you check whether
  a user is allowed to mess with a certain running process; alternatively,
  you can allow it and make it a requirement for blessed scripts that
  authorized users can attach them to any running process without
  compromising the security of the system)


On 11 Jun 2007, Frank Ch. Eigler wrote:

> Actually, it doesn't.  A setuid program can drop its privileges after
> performing the root-only operations (module loading), and invoke the
> rest of the normal commands as the real userid.

Staprun has to keep root privileges to be able to unload the kernel module
when it finishes. Moreover, the mere possesion of an open fd for the
control channel seems to be dangerous enough to make staprun de facto
running under euid 0 as long as it keeps the fd open (correct me if I am 
wrong).

BTW1: staprun should close the file descriptors it uses internally 
(control fd, relayfs fds) when it executes the target program given by -c.

BTW2: Let's suppose start_cmd() creates a process running under an
unprivileged user. I think it can be killed (by the unprivileged user)  
before it gets SIGUSR1 and the system might recycle its pid. Therefore
kill() in STP_START branch of stp_main_loop() is unsafe.


On Mon, 11 Jun 2007, David Smith wrote:

> Perhaps there is a merged approach.  Keep staprun_auth a thin wrapper 
> around staprun, but change staprun to raise and lower privileges as 
> needed when inserting/removing modules, setting up relayfs, etc.

This might work but be very careful when you do it while multiple threads 
are running.


--Pavel Kankovsky aka Peak  [ Boycott Microsoft--http://www.vcnet.com/bms ]
"Resistance is futile. Open your source code and prepare for assimilation."

Re: Need some security advice for systemtap

[David Smith]

Pavel Kankovsky wrote:
> On Mon, 11 Jun 2007, David Smith wrote:
> 
>> (D) staprun.auth will need to disallow certain staprun.auth command-line 
>> arguments, such as:
> 
> Plus:
> 
> - "-u USERNAME" (assuming you allow -c, otherwise it has no effect;
>   on the other hand, its support in staprun can help to to make it 
>   possible to use -c via staprun.auth)
> 
> - "-t PID" (or "-x PID") (you need to disallow this because it is
>   virtually impossible to avoid race conditions when you check whether
>   a user is allowed to mess with a certain running process; alternatively,
>   you can allow it and make it a requirement for blessed scripts that
>   authorized users can attach them to any running process without
>   compromising the security of the system)

I knew about '-u USERNAME'.  We'll probably allow '-x PID' for 
authorized users because if they can look at the entire system there 
isn't any point in not allowing them to look at a single process.

> On 11 Jun 2007, Frank Ch. Eigler wrote:
> 
>> Actually, it doesn't.  A setuid program can drop its privileges after
>> performing the root-only operations (module loading), and invoke the
>> rest of the normal commands as the real userid.
> 
> Staprun has to keep root privileges to be able to unload the kernel module
> when it finishes. Moreover, the mere possesion of an open fd for the
> control channel seems to be dangerous enough to make staprun de facto
> running under euid 0 as long as it keeps the fd open (correct me if I am 
> wrong).

Yep, you are right.

> BTW1: staprun should close the file descriptors it uses internally 
> (control fd, relayfs fds) when it executes the target program given by -c.

Good idea, I'll work on that.

> BTW2: Let's suppose start_cmd() creates a process running under an
> unprivileged user. I think it can be killed (by the unprivileged user)  
> before it gets SIGUSR1 and the system might recycle its pid. Therefore
> kill() in STP_START branch of stp_main_loop() is unsafe.

Hmm.  Got any ideas on how to fix this?

> On Mon, 11 Jun 2007, David Smith wrote:
> 
>> Perhaps there is a merged approach.  Keep staprun_auth a thin wrapper 
>> around staprun, but change staprun to raise and lower privileges as 
>> needed when inserting/removing modules, setting up relayfs, etc.
> 
> This might work but be very careful when you do it while multiple threads 
> are running.

I believe I see what you mean here - if one thread raises privilegs 
while another thread performs a security-sensitive operation, we've got 
a problem.

Thanks for your time.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[Martin Hunt]

On Mon, 2007-06-18 at 14:44 -0500, David Smith wrote:
> Pavel Kankovsky wrote:
> > On Mon, 11 Jun 2007, David Smith wrote:
> > 
> >> (D) staprun.auth will need to disallow certain staprun.auth command-line 
> >> arguments, such as:
> > 
> > Plus:
> > 
> > - "-u USERNAME" (assuming you allow -c, otherwise it has no effect;
> >   on the other hand, its support in staprun can help to to make it 
> >   possible to use -c via staprun.auth)

The only reason for "-u" is so we can currently do things like 
>stap -c "make all" systrace.stp
and have "make" execute as a normal user rather than root (because stap
does "sudo staprun").  

If staprun is not run by root, the setreuid() call will just fail if
someone tries to misuse it.

> > - "-t PID" (or "-x PID") (you need to disallow this because it is
> >   virtually impossible to avoid race conditions when you check whether
> >   a user is allowed to mess with a certain running process; alternatively,
> >   you can allow it and make it a requirement for blessed scripts that
> >   authorized users can attach them to any running process without
> >   compromising the security of the system)

"-x pid "doesn't actually do anything except cause target() to return
pid. So disabling it would be pointless. It exists as a convenient way
to pass a value to scripts that wish to filter based on pid. 

> > BTW1: staprun should close the file descriptors it uses internally 
> > (control fd, relayfs fds) when it executes the target program given by -c.
> 
> Good idea, I'll work on that.
> 
> > BTW2: Let's suppose start_cmd() creates a process running under an
> > unprivileged user. I think it can be killed (by the unprivileged user)  
> > before it gets SIGUSR1 and the system might recycle its pid. Therefore
> > kill() in STP_START branch of stp_main_loop() is unsafe.
> 
> Hmm.  Got any ideas on how to fix this?

So, while the module loads its probes, we kill the start_cmd() process
and create enough new processes to recycle the pid? Then staprun sends
either SIGKILL or SIGUSR1 to the wrong process? Theoretically, if we set
tens of thousands of probes, we would have a few milliseconds to do
this. 

Now that I've caught up on this discussion (sorry) I see we are simply
talking about the blessed module approach to security.  So the only
issue here is how to load/unload the module, right?  It seems like you
have some good ideas. Why are we worrying about staprun?  Load the
module with your blessed loader, then staprun runs as a normal user. The
only problem I see is unloading the module automatically. 

BTW, you can kind of try this now. 

Run with sudo or root:
>sudo /sbin/insmod close.ko
>sudo chown hunt.hunt /sys/kernel/debug/systemtap/close/*

Now run staprun as a normal user
>staprun -A close
cpufreq-applet: close(17) = 0
cpufreq-applet: close(17) = 0
hald-addon-stor: close(4) = 0
cpufreq-applet: close(17) = 0
cpufreq-applet: close(17) = 0
crond: close(5) = 0
crond: close(5) = 0
gpm: close(0) = 0
cpufreq-applet: close(17) = 0
cpufreq-applet: close(17) = 0
DONE
ERROR: Removing 'close': Operation not permitted
ERROR: couldn't rmmod probe module close.

--

Hmmm.  So lets say we create stap_load owned by 
root, group systemtap, and file permission would be 04110.

stap_load is a C program that does something like
if (root)
	insmod(modname)
else if modname is blessed && user is valid
	insmod(modname)
endif

stap_unload would be similar or it could be the same program with
an argument to indicate load/unload.

Then rename staprun to staprun.bin and create a script, staprun:
if stap_load modname
	staprun.bin modname ...
	stap_unload modname
endif

Are there problems with this approach?

Martin

Re: Need some security advice for systemtap

[David Smith]

Martin Hunt wrote:
> On Mon, 2007-06-18 at 14:44 -0500, David Smith wrote:
>> Pavel Kankovsky wrote:
>>> On Mon, 11 Jun 2007, David Smith wrote:
>>
>>> BTW2: Let's suppose start_cmd() creates a process running under an
>>> unprivileged user. I think it can be killed (by the unprivileged user)  
>>> before it gets SIGUSR1 and the system might recycle its pid. Therefore
>>> kill() in STP_START branch of stp_main_loop() is unsafe.
>> Hmm.  Got any ideas on how to fix this?
> 
> So, while the module loads its probes, we kill the start_cmd() process
> and create enough new processes to recycle the pid? Then staprun sends
> either SIGKILL or SIGUSR1 to the wrong process? Theoretically, if we set
> tens of thousands of probes, we would have a few milliseconds to do
> this. 

I do agree it isn't a likely occurrence, but if it is possible to fix we 
ought to look at it.

> Now that I've caught up on this discussion (sorry) I see we are simply
> talking about the blessed module approach to security.  So the only
> issue here is how to load/unload the module, right?  It seems like you
> have some good ideas. Why are we worrying about staprun?  Load the
> module with your blessed loader, then staprun runs as a normal user. The
> only problem I see is unloading the module automatically. 

The problem with the blessed loader approach is that:

(a) it duplicates code with regular staprun (although perhaps not much)
(b) unloading the module isn't possible

So, Frank and I have been trying to think through other ideas.

> BTW, you can kind of try this now. 
> 
> Run with sudo or root:
>> sudo /sbin/insmod close.ko
>> sudo chown hunt.hunt /sys/kernel/debug/systemtap/close/*

Actually that doesn't work if /sys/kernel/debug hasn't been mounted before.

> Now run staprun as a normal user
>> staprun -A close
> cpufreq-applet: close(17) = 0
> cpufreq-applet: close(17) = 0
> hald-addon-stor: close(4) = 0
> cpufreq-applet: close(17) = 0
> cpufreq-applet: close(17) = 0
> crond: close(5) = 0
> crond: close(5) = 0
> gpm: close(0) = 0
> cpufreq-applet: close(17) = 0
> cpufreq-applet: close(17) = 0
> DONE
> ERROR: Removing 'close': Operation not permitted
> ERROR: couldn't rmmod probe module close.
> 
> --
> 
> Hmmm.  So lets say we create stap_load owned by 
> root, group systemtap, and file permission would be 04110.
> 
> stap_load is a C program that does something like
> if (root)
> 	insmod(modname)
> else if modname is blessed && user is valid
> 	insmod(modname)
> endif
> 
> stap_unload would be similar or it could be the same program with
> an argument to indicate load/unload.
> 
> Then rename staprun to staprun.bin and create a script, staprun:
> if stap_load modname
> 	staprun.bin modname ...
> 	stap_unload modname
> endif
> 
> Are there problems with this approach?

Having two different setuid programs isn't the best plan.  One possible 
alternative could be:

stap_load (setuid) inserts module, then forks and execs (as a user) 
"staprun -A".  stap_load then waits for "staprun -A" to finish.  After 
staprun finishes, stap_load removes the module.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Need some security advice for systemtap

[Stone, Joshua I]

David Smith wrote:
> Martin Hunt wrote:
>> On Mon, 2007-06-18 at 14:44 -0500, David Smith wrote:
>>> Pavel Kankovsky wrote:
>>>> On Mon, 11 Jun 2007, David Smith wrote:
>>>
>>>> BTW2: Let's suppose start_cmd() creates a process running under an
>>>> unprivileged user. I think it can be killed (by the unprivileged 
>>>> user)  before it gets SIGUSR1 and the system might recycle its pid. 
>>>> Therefore
>>>> kill() in STP_START branch of stp_main_loop() is unsafe.
>>> Hmm.  Got any ideas on how to fix this?
>>
>> So, while the module loads its probes, we kill the start_cmd() process
>> and create enough new processes to recycle the pid? Then staprun sends
>> either SIGKILL or SIGUSR1 to the wrong process? Theoretically, if we set
>> tens of thousands of probes, we would have a few milliseconds to do
>> this. 
> 
> I do agree it isn't a likely occurrence, but if it is possible to fix we 
> ought to look at it.

This should be manageable.  When a child process exits, it sends a 
SIGCHLD and sits as a zombie until the parent has wait()ed for it.  As 
long as it's a zombie, the pid won't be recycled.

We just need to notice in our sig handler that the start_cmd process 
died, and make sure we don't try to kill the pid after that.

Josh

Re: Need some security advice for systemtap

[Pavel Kankovsky]

On Mon, 18 Jun 2007, David Smith wrote:

> > This might work but be very careful when you do it while multiple threads 
> > are running.
> 
> I believe I see what you mean here - if one thread raises privilegs 
> while another thread performs a security-sensitive operation, we've got 
> a problem.

Exactly.


On Mon, 18 Jun 2007, Martin Hunt wrote:

> "-x pid "doesn't actually do anything except cause target() to return
> pid. So disabling it would be pointless. It exists as a convenient way
> to pass a value to scripts that wish to filter based on pid. 

It depends on the script. I can imagine a script that is safe as long as
the pid is yours but becomes dangerous when you can give it other pids.
E.g. a strace-like script peeking into data being read and written.

You need to establish a policy: either the blessed must never leak
potentially sensitive data (and the strace-like script is not eligible to
be blessed) or -x must be disabled.

> So, while the module loads its probes, we kill the start_cmd() process
> and create enough new processes to recycle the pid? Then staprun sends
> either SIGKILL or SIGUSR1 to the wrong process? Theoretically, if we set
> tens of thousands of probes, we would have a few milliseconds to do
> this. 

Unless the parent process is stopped at the right moment...

> Run with sudo or root:
> >sudo /sbin/insmod close.ko
> >sudo chown hunt.hunt /sys/kernel/debug/systemtap/close/*

The idea of granting access to systemtap control channel to mortals
gives me the creeps and I won't feel better unless you can prove it is not 
possible to cause any harm when untrusted data is written to the 
control channel. What about STP_SYMBOLS?


On Tue, 19 Jun 2007, Stone, Joshua I wrote:

> This should be manageable.  When a child process exits, it sends a 
> SIGCHLD and sits as a zombie until the parent has wait()ed for it.  As 
> long as it's a zombie, the pid won't be recycled.
> 
> We just need to notice in our sig handler that the start_cmd process 
> died, and make sure we don't try to kill the pid after that.

There's still a race condition there:

1. the parent process checks the flag, the child is still alive
   and flag is reset, ok, let's kill the child
2. the child dies spontaneously, SIGCHLD handler sets a flag
   and reaps the child's zombie
3. the parent send a signal...oops!

It might help to block SIGCHLD temporarily. Or to take waitpid() away from
the signal handler and reap zombies synchronously in the main loop.

BTW: Is it always safe to call send_request() in a signal handler?
And fprintf()? (See <http://lcamtuf.coredump.cx/signals.txt>)


--Pavel Kankovsky aka Peak  [ Boycott Microsoft--http://www.vcnet.com/bms ]
"Resistance is futile. Open your source code and prepare for assimilation."