kprobes on by default in 2.6.20.1 kernel.org kernels

kprobes on by default in 2.6.20.1 kernel.org kernels

[Nathan DeBardeleben]

I just wanted to probe (har har) to see if you guys knew why kprobes is 
set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel I 
got from kernel.org.  I don't consider this a great move and am a little 
worried about it.  For one thing, I know now we'll actively make certain 
it's off in all future kernels we build that are intended for production 
machines.

I grabbed the latest kernel to do some kprobes testing, went to 
configure it to turn probing on and was very surprised to find it on by 
default.

What's the process that the kernel maintainers go through to determine 
which options are on by default anyway?

Thanks!

-- 
-- Nathan
Correspondence
---------------------------------------------------------------------
Nathan DeBardeleben, Ph.D.
Los Alamos National Laboratory
Parallel Tools Team
High Performance Computing Environments (HPC-4)
phone: 505-667-3428
email: ndebard@lanl.gov
---------------------------------------------------------------------

Re: kprobes on by default in 2.6.20.1 kernel.org kernels

[Ananth N Mavinakayanahalli]

On Wed, Feb 28, 2007 at 04:24:14PM -0700, Nathan DeBardeleben wrote:
> I just wanted to probe (har har) to see if you guys knew why kprobes is 
> set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel I 
> got from kernel.org.  I don't consider this a great move and am a little 
> worried about it.  For one thing, I know now we'll actively make certain 
> it's off in all future kernels we build that are intended for production 
> machines.
> 
> I grabbed the latest kernel to do some kprobes testing, went to 
> configure it to turn probing on and was very surprised to find it on by 
> default.

Kprobes has been enabled by default since 2.6.19 on i386. x86_64 has had
it on by default for a long while now.

> What's the process that the kernel maintainers go through to determine 
> which options are on by default anyway?

I am not aware of any process as such, but its mostly a decision taken
based on whether or not the feature is useful or not for the general
populace.

Ananth

Re: kprobes on by default in 2.6.20.1 kernel.org kernels

[Keshavamurthy, Anil S]

On Wed, Feb 28, 2007 at 04:24:14PM -0700, Nathan DeBardeleben wrote:
> I just wanted to probe (har har) to see if you guys knew why kprobes is 
> set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel I 
> got from kernel.org.  I don't consider this a great move and am a little 
> worried about it.  For one thing, I know now we'll actively make certain 
> it's off in all future kernels we build that are intended for production 
> machines.

Can you please explain your cause for worry in detail. In what way it 
is causing problems. Hopefully we can address your concerns.

> 
> I grabbed the latest kernel to do some kprobes testing, went to 
> configure it to turn probing on and was very surprised to find it on by 
> default.
As a matter of fact, KPROBE is enabled by several major OSD like
Red Hat and SuSE on their enterprise versions and are in
the market since begining/middle of last year.

> 
> What's the process that the kernel maintainers go through to determine 
> which options are on by default anyway?

AFAIK, their is no formal process(depends on developer community). 
And if you are building a kernel for production machine then I assume 
you just don;t depend on default config which can turn on lots of stuff 
intended for testing and experimental purposes.

-Anil Keshavamurthy

Re: kprobes on by default in 2.6.20.1 kernel.org kernels

[Nathan DeBardeleben]

You know, it's not really me that has a problem with it.  I subscribe to 
the camp of: If it requires root to run a probe, then someone who has 
root can already do some pretty nefarious things to your system.  
However, due the fact that with kprobes you can slip in a probe, and 
then even hide that probe from being reported, I think some people I 
work with are scared that it leaves them slightly more exposed / vulnerable.

I'd love to be able to convince them to let me run probes on production 
systems but at this time they're relegated to test machines.  Honestly 
I'd like to be able to add to my repertoire a good argument to allow 
kprobes on by default on these machines but when you're dealing with 
security plans and documents and whatnot a new technology like this is 
easily deemed scary, unknown, and just relegated to the "security risk" 
column.

-- Nathan
Correspondence
---------------------------------------------------------------------
Nathan DeBardeleben, Ph.D.
Los Alamos National Laboratory
Parallel Tools Team
High Performance Computing Environments (HPC-4)
phone: 505-667-3428
email: ndebard@lanl.gov
---------------------------------------------------------------------



Keshavamurthy, Anil S wrote:
> On Wed, Feb 28, 2007 at 04:24:14PM -0700, Nathan DeBardeleben wrote:
>   
>> I just wanted to probe (har har) to see if you guys knew why kprobes is 
>> set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel I 
>> got from kernel.org.  I don't consider this a great move and am a little 
>> worried about it.  For one thing, I know now we'll actively make certain 
>> it's off in all future kernels we build that are intended for production 
>> machines.
>>     
>
> Can you please explain your cause for worry in detail. In what way it 
> is causing problems. Hopefully we can address your concerns.
>
>   
>> I grabbed the latest kernel to do some kprobes testing, went to 
>> configure it to turn probing on and was very surprised to find it on by 
>> default.
>>     
> As a matter of fact, KPROBE is enabled by several major OSD like
> Red Hat and SuSE on their enterprise versions and are in
> the market since begining/middle of last year.
>
>   
>> What's the process that the kernel maintainers go through to determine 
>> which options are on by default anyway?
>>     
>
> AFAIK, their is no formal process(depends on developer community). 
> And if you are building a kernel for production machine then I assume 
> you just don;t depend on default config which can turn on lots of stuff 
> intended for testing and experimental purposes.
>
> -Anil Keshavamurthy
>
>   

RE: kprobes on by default in 2.6.20.1 kernel.org kernels

[Keshavamurthy, Anil S]

Ha..If that is you concern, then I think you are lucky. We recently
addressed this issues where 
in one can list all the active probes in the system by accessing the
file /sys/kernel/debug/kprobes/list. 

And the output look like this
=====================
llm40:~/a # cat /sys/kernel/debug/kprobes/list
c0169ae3  r  sys_read+0x0
c0169ae3  k  sys_read+0x0
c01694c8  k  vfs_write+0x0
c0167d20  r  sys_open+0x0
f8e658a6  k  reiserfs_delete_inode+0x0  reiserfs
c0120f4a  k  do_fork+0x0
c0120f4a  j  do_fork+0x0
c0169b4a  r  sys_write+0x0
c0169b4a  k  sys_write+0x0
c0169622  r  vfs_read+0x0
====================

Will this help?

Thanks,

-Anil Keshavamurthy
Open Source Technology Center/SSG
Intel Corp.
(w) 503-712-4476
email: anil.s.keshavamurthy@intel.com

-----Original Message-----
From: Nathan DeBardeleben [mailto:ndebard@lanl.gov] 
Sent: Thursday, March 01, 2007 3:56 PM
To: Keshavamurthy, Anil S
Cc: systemtap@sources.redhat.com
Subject: Re: kprobes on by default in 2.6.20.1 kernel.org kernels

You know, it's not really me that has a problem with it.  I subscribe to

the camp of: If it requires root to run a probe, then someone who has 
root can already do some pretty nefarious things to your system.  
However, due the fact that with kprobes you can slip in a probe, and 
then even hide that probe from being reported, I think some people I 
work with are scared that it leaves them slightly more exposed /
vulnerable.

I'd love to be able to convince them to let me run probes on production 
systems but at this time they're relegated to test machines.  Honestly 
I'd like to be able to add to my repertoire a good argument to allow 
kprobes on by default on these machines but when you're dealing with 
security plans and documents and whatnot a new technology like this is 
easily deemed scary, unknown, and just relegated to the "security risk" 
column.

-- Nathan
Correspondence
---------------------------------------------------------------------
Nathan DeBardeleben, Ph.D.
Los Alamos National Laboratory
Parallel Tools Team
High Performance Computing Environments (HPC-4)
phone: 505-667-3428
email: ndebard@lanl.gov
---------------------------------------------------------------------



Keshavamurthy, Anil S wrote:
> On Wed, Feb 28, 2007 at 04:24:14PM -0700, Nathan DeBardeleben wrote:
>   
>> I just wanted to probe (har har) to see if you guys knew why kprobes
is 
>> set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel I

>> got from kernel.org.  I don't consider this a great move and am a
little 
>> worried about it.  For one thing, I know now we'll actively make
certain 
>> it's off in all future kernels we build that are intended for
production 
>> machines.
>>     
>
> Can you please explain your cause for worry in detail. In what way it 
> is causing problems. Hopefully we can address your concerns.
>
>   
>> I grabbed the latest kernel to do some kprobes testing, went to 
>> configure it to turn probing on and was very surprised to find it on
by 
>> default.
>>     
> As a matter of fact, KPROBE is enabled by several major OSD like
> Red Hat and SuSE on their enterprise versions and are in
> the market since begining/middle of last year.
>
>   
>> What's the process that the kernel maintainers go through to
determine 
>> which options are on by default anyway?
>>     
>
> AFAIK, their is no formal process(depends on developer community). 
> And if you are building a kernel for production machine then I assume 
> you just don;t depend on default config which can turn on lots of
stuff 
> intended for testing and experimental purposes.
>
> -Anil Keshavamurthy
>
>   

Re: kprobes on by default in 2.6.20.1 kernel.org kernels

[Nathan DeBardeleben]

I wasn't aware of that one.  However, since one is dynamically editing 
the kernel couldn't one probe the function that prints this out and 
claim there were no probes, even though you had some nice and nasty 
probes inserted?  Or have you guys declared that code as unprobable or 
something like that?

-- Nathan
Correspondence
---------------------------------------------------------------------
Nathan DeBardeleben, Ph.D.
Los Alamos National Laboratory
Parallel Tools Team
High Performance Computing Environments (HPC-4)
phone: 505-667-3428
email: ndebard@lanl.gov
---------------------------------------------------------------------



Keshavamurthy, Anil S wrote:
> Ha..If that is you concern, then I think you are lucky. We recently
> addressed this issues where 
> in one can list all the active probes in the system by accessing the
> file /sys/kernel/debug/kprobes/list. 
>
> And the output look like this
> =====================
> llm40:~/a # cat /sys/kernel/debug/kprobes/list
> c0169ae3  r  sys_read+0x0
> c0169ae3  k  sys_read+0x0
> c01694c8  k  vfs_write+0x0
> c0167d20  r  sys_open+0x0
> f8e658a6  k  reiserfs_delete_inode+0x0  reiserfs
> c0120f4a  k  do_fork+0x0
> c0120f4a  j  do_fork+0x0
> c0169b4a  r  sys_write+0x0
> c0169b4a  k  sys_write+0x0
> c0169622  r  vfs_read+0x0
> ====================
>
> Will this help?
>
> Thanks,
>
> -Anil Keshavamurthy
> Open Source Technology Center/SSG
> Intel Corp.
> (w) 503-712-4476
> email: anil.s.keshavamurthy@intel.com
>
> -----Original Message-----
> From: Nathan DeBardeleben [mailto:ndebard@lanl.gov] 
> Sent: Thursday, March 01, 2007 3:56 PM
> To: Keshavamurthy, Anil S
> Cc: systemtap@sources.redhat.com
> Subject: Re: kprobes on by default in 2.6.20.1 kernel.org kernels
>
> You know, it's not really me that has a problem with it.  I subscribe to
>
> the camp of: If it requires root to run a probe, then someone who has 
> root can already do some pretty nefarious things to your system.  
> However, due the fact that with kprobes you can slip in a probe, and 
> then even hide that probe from being reported, I think some people I 
> work with are scared that it leaves them slightly more exposed /
> vulnerable.
>
> I'd love to be able to convince them to let me run probes on production 
> systems but at this time they're relegated to test machines.  Honestly 
> I'd like to be able to add to my repertoire a good argument to allow 
> kprobes on by default on these machines but when you're dealing with 
> security plans and documents and whatnot a new technology like this is 
> easily deemed scary, unknown, and just relegated to the "security risk" 
> column.
>
> -- Nathan
> Correspondence
> ---------------------------------------------------------------------
> Nathan DeBardeleben, Ph.D.
> Los Alamos National Laboratory
> Parallel Tools Team
> High Performance Computing Environments (HPC-4)
> phone: 505-667-3428
> email: ndebard@lanl.gov
> ---------------------------------------------------------------------
>
>
>
> Keshavamurthy, Anil S wrote:
>   
>> On Wed, Feb 28, 2007 at 04:24:14PM -0700, Nathan DeBardeleben wrote:
>>   
>>     
>>> I just wanted to probe (har har) to see if you guys knew why kprobes
>>>       
> is 
>   
>>> set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel I
>>>       
>
>   
>>> got from kernel.org.  I don't consider this a great move and am a
>>>       
> little 
>   
>>> worried about it.  For one thing, I know now we'll actively make
>>>       
> certain 
>   
>>> it's off in all future kernels we build that are intended for
>>>       
> production 
>   
>>> machines.
>>>     
>>>       
>> Can you please explain your cause for worry in detail. In what way it 
>> is causing problems. Hopefully we can address your concerns.
>>
>>   
>>     
>>> I grabbed the latest kernel to do some kprobes testing, went to 
>>> configure it to turn probing on and was very surprised to find it on
>>>       
> by 
>   
>>> default.
>>>     
>>>       
>> As a matter of fact, KPROBE is enabled by several major OSD like
>> Red Hat and SuSE on their enterprise versions and are in
>> the market since begining/middle of last year.
>>
>>   
>>     
>>> What's the process that the kernel maintainers go through to
>>>       
> determine 
>   
>>> which options are on by default anyway?
>>>     
>>>       
>> AFAIK, their is no formal process(depends on developer community). 
>> And if you are building a kernel for production machine then I assume 
>> you just don;t depend on default config which can turn on lots of
>>     
> stuff 
>   
>> intended for testing and experimental purposes.
>>
>> -Anil Keshavamurthy
>>
>>   
>>     
>
>   

RE: kprobes on by default in 2.6.20.1 kernel.org kernels

[Keshavamurthy, Anil S]

 Good call. Never thought about this. Just checked the code and found
that we are 
doing the right thing i.e no probes are allowed on this code path :)

Anyway our goal is to make Kprobes production worthy and any feedback 
from the real world environment are very welcome and highly appreciated.


Thanks,
Anil  Keshavamurthy

-----Original Message-----
From: Nathan DeBardeleben [mailto:ndebard@lanl.gov] 
Sent: Thursday, March 01, 2007 4:10 PM
To: Keshavamurthy, Anil S
Cc: systemtap@sources.redhat.com
Subject: Re: kprobes on by default in 2.6.20.1 kernel.org kernels

I wasn't aware of that one.  However, since one is dynamically editing 
the kernel couldn't one probe the function that prints this out and 
claim there were no probes, even though you had some nice and nasty 
probes inserted?  Or have you guys declared that code as unprobable or 
something like that?

-- Nathan
Correspondence
---------------------------------------------------------------------
Nathan DeBardeleben, Ph.D.
Los Alamos National Laboratory
Parallel Tools Team
High Performance Computing Environments (HPC-4)
phone: 505-667-3428
email: ndebard@lanl.gov
---------------------------------------------------------------------



Keshavamurthy, Anil S wrote:
> Ha..If that is you concern, then I think you are lucky. We recently
> addressed this issues where 
> in one can list all the active probes in the system by accessing the
> file /sys/kernel/debug/kprobes/list. 
>
> And the output look like this
> =====================
> llm40:~/a # cat /sys/kernel/debug/kprobes/list
> c0169ae3  r  sys_read+0x0
> c0169ae3  k  sys_read+0x0
> c01694c8  k  vfs_write+0x0
> c0167d20  r  sys_open+0x0
> f8e658a6  k  reiserfs_delete_inode+0x0  reiserfs
> c0120f4a  k  do_fork+0x0
> c0120f4a  j  do_fork+0x0
> c0169b4a  r  sys_write+0x0
> c0169b4a  k  sys_write+0x0
> c0169622  r  vfs_read+0x0
> ====================
>
> Will this help?
>
> Thanks,
>
> -Anil Keshavamurthy
> Open Source Technology Center/SSG
> Intel Corp.
> (w) 503-712-4476
> email: anil.s.keshavamurthy@intel.com
>
> -----Original Message-----
> From: Nathan DeBardeleben [mailto:ndebard@lanl.gov] 
> Sent: Thursday, March 01, 2007 3:56 PM
> To: Keshavamurthy, Anil S
> Cc: systemtap@sources.redhat.com
> Subject: Re: kprobes on by default in 2.6.20.1 kernel.org kernels
>
> You know, it's not really me that has a problem with it.  I subscribe
to
>
> the camp of: If it requires root to run a probe, then someone who has 
> root can already do some pretty nefarious things to your system.  
> However, due the fact that with kprobes you can slip in a probe, and 
> then even hide that probe from being reported, I think some people I 
> work with are scared that it leaves them slightly more exposed /
> vulnerable.
>
> I'd love to be able to convince them to let me run probes on
production 
> systems but at this time they're relegated to test machines.  Honestly

> I'd like to be able to add to my repertoire a good argument to allow 
> kprobes on by default on these machines but when you're dealing with 
> security plans and documents and whatnot a new technology like this is

> easily deemed scary, unknown, and just relegated to the "security
risk" 
> column.
>
> -- Nathan
> Correspondence
> ---------------------------------------------------------------------
> Nathan DeBardeleben, Ph.D.
> Los Alamos National Laboratory
> Parallel Tools Team
> High Performance Computing Environments (HPC-4)
> phone: 505-667-3428
> email: ndebard@lanl.gov
> ---------------------------------------------------------------------
>
>
>
> Keshavamurthy, Anil S wrote:
>   
>> On Wed, Feb 28, 2007 at 04:24:14PM -0700, Nathan DeBardeleben wrote:
>>   
>>     
>>> I just wanted to probe (har har) to see if you guys knew why kprobes
>>>       
> is 
>   
>>> set to "Y" (in the kernel) by default on the latest 2.6.20.1 kernel
I
>>>       
>
>   
>>> got from kernel.org.  I don't consider this a great move and am a
>>>       
> little 
>   
>>> worried about it.  For one thing, I know now we'll actively make
>>>       
> certain 
>   
>>> it's off in all future kernels we build that are intended for
>>>       
> production 
>   
>>> machines.
>>>     
>>>       
>> Can you please explain your cause for worry in detail. In what way it

>> is causing problems. Hopefully we can address your concerns.
>>
>>   
>>     
>>> I grabbed the latest kernel to do some kprobes testing, went to 
>>> configure it to turn probing on and was very surprised to find it on
>>>       
> by 
>   
>>> default.
>>>     
>>>       
>> As a matter of fact, KPROBE is enabled by several major OSD like
>> Red Hat and SuSE on their enterprise versions and are in
>> the market since begining/middle of last year.
>>
>>   
>>     
>>> What's the process that the kernel maintainers go through to
>>>       
> determine 
>   
>>> which options are on by default anyway?
>>>     
>>>       
>> AFAIK, their is no formal process(depends on developer community). 
>> And if you are building a kernel for production machine then I assume

>> you just don;t depend on default config which can turn on lots of
>>     
> stuff 
>   
>> intended for testing and experimental purposes.
>>
>> -Anil Keshavamurthy
>>
>>   
>>     
>
>   

Re: kprobes on by default in 2.6.20.1 kernel.org kernels

[Frank Ch. Eigler]

Nathan DeBardeleben <ndebard@lanl.gov> writes:

> [...]  machines.  Honestly I'd like to be able to add to my
> repertoire a good argument to allow kprobes on by default on these
> machines but when you're dealing with security plans and documents
> and whatnot a new technology like this is easily deemed scary,
> unknown, and just relegated to the "security risk" column.

Just as keeping module loading activated, /proc/*mem available, and
probably a dozen other facilities, calling kprobes a security risk
just takes an expediently simple view of the thing.  It must be
remembered that all of these exist because of the benefits they bring,
whether functional, administrative, operational - they tend to be
sufficiently useful to outweigh their hypothetical abuse potential.

In this way, kprobes is really no worse than the others.  If your
threat scenario involves software that actively hides itself, then you
have to lock many other things down.  If you worry about a lesser
complexity of attacks, then the various self-policing mechanisms
present (the __kprobe-marked non-probable sections, root access
required, systemtap's auditing printk at startup, ...) are probably
sufficient.

- FChE