have we been sucked by suckit?

have we been sucked by suckit?

[Joe Buck]

Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
verified that gcc.gnu.org / sources.redhat.com has not been rooted?
And if not, has its kernel been upgraded?

Re: have we been sucked by suckit?

[law]

In message <20031203152137.A10206@synopsys.com>, Joe Buck writes:
 >Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
 >verified that gcc.gnu.org / sources.redhat.com has not been rooted?
 >And if not, has its kernel been upgraded?
We're already working to get a 2hr window where our sysadmins can take the
box down, verify the system hasn't been rooted and make sure it's got
the appropriate kernel fix.

I'd be surprised if it isn't all taken care of by the end of this week.

jeff

Re: have we been sucked by suckit?

[Joe Buck]

On Wed, Dec 03, 2003 at 05:08:17PM -0700, law@redhat.com wrote:
> In message <20031203152137.A10206@synopsys.com>, Joe Buck writes:
>  >Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
>  >verified that gcc.gnu.org / sources.redhat.com has not been rooted?
>  >And if not, has its kernel been upgraded?
> We're already working to get a 2hr window where our sysadmins can take the
> box down, verify the system hasn't been rooted and make sure it's got
> the appropriate kernel fix.

There's a suckit detection tool that you could try running before you take
the box down.  See

http://tsd.student.utwente.nl/skdetect/

Of course, this only detects one particular rootkit, but it seems to be
the one that was used at Debian and savannah.

Re: have we been sucked by suckit?

[Christopher Faylor]

On Wed, Dec 03, 2003 at 04:18:25PM -0800, Joe Buck wrote:
>On Wed, Dec 03, 2003 at 05:08:17PM -0700, law@redhat.com wrote:
>>In message <20031203152137.A10206@synopsys.com>, Joe Buck writes:
>>>Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
>>>verified that gcc.gnu.org / sources.redhat.com has not been rooted?
>>>And if not, has its kernel been upgraded?
>>We're already working to get a 2hr window where our sysadmins can take
>>the box down, verify the system hasn't been rooted and make sure it's
>>got the appropriate kernel fix.
>
>There's a suckit detection tool that you could try running before you
>take the box down.  See
>
>http://tsd.student.utwente.nl/skdetect/
>
>Of course, this only detects one particular rootkit, but it seems to be
>the one that was used at Debian and savannah.

Thanks for the pointer.  This tool indicates that we're clean for
whatever that's worth.

cgf

Re: have we been sucked by suckit?

[Matthew Galgoci]

I am going to check it by hand this evening, booted from rescue media.

I also have a hardened kernel to install on it that raises the bar on 
exports through /dev/mem, which is how sukkit is installed.

On Wed, 3 Dec 2003, Joe Buck wrote:

> Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
> verified that gcc.gnu.org / sources.redhat.com has not been rooted?
> And if not, has its kernel been upgraded?
> 
> 
> 

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: have we been sucked by suckit?

[Christopher Faylor]

On Wed, Dec 03, 2003 at 08:29:16PM -0500, Matthew Galgoci wrote:
>I am going to check it by hand this evening, booted from rescue media.
>
>I also have a hardened kernel to install on it that raises the bar on
>exports through /dev/mem, which is how sukkit is installed.

And, eventually I'll turn off module support in the kernel, which is
another potential hole.

Long term plans are to use SElinux.  That would be cool even for the
non-security aspects.

cgf

Re: have we been sucked by suckit?

[Matthew Galgoci]

Btw I am running late unfortunately :\

I will be at the colo some time this evening (soon I hope) and get
the checking done as soon as possible.

On Wed, 3 Dec 2003, Christopher Faylor wrote:

> On Wed, Dec 03, 2003 at 08:29:16PM -0500, Matthew Galgoci wrote:
> >I am going to check it by hand this evening, booted from rescue media.
> >
> >I also have a hardened kernel to install on it that raises the bar on
> >exports through /dev/mem, which is how sukkit is installed.
> 
> And, eventually I'll turn off module support in the kernel, which is
> another potential hole.
> 
> Long term plans are to use SElinux.  That would be cool even for the
> non-security aspects.
> 
> cgf
> 

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: have we been sucked by suckit?

[Matthew Galgoci]

Ok, I am en route as soon as I send this mail. ETA of 45 minutes barring
me actually finding anything wrong with sources.

On Wed, 3 Dec 2003, Matthew Galgoci wrote:

> 
> Btw I am running late unfortunately :\
> 
> I will be at the colo some time this evening (soon I hope) and get
> the checking done as soon as possible.
> 
> On Wed, 3 Dec 2003, Christopher Faylor wrote:
> 
> > On Wed, Dec 03, 2003 at 08:29:16PM -0500, Matthew Galgoci wrote:
> > >I am going to check it by hand this evening, booted from rescue media.
> > >
> > >I also have a hardened kernel to install on it that raises the bar on
> > >exports through /dev/mem, which is how sukkit is installed.
> > 
> > And, eventually I'll turn off module support in the kernel, which is
> > another potential hole.
> > 
> > Long term plans are to use SElinux.  That would be cool even for the
> > non-security aspects.
> > 
> > cgf
> > 
> 
> 

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: have we been sucked by suckit?

[Zack Weinberg]

Christopher Faylor <cgf@redhat.com> writes:

> On Wed, Dec 03, 2003 at 08:29:16PM -0500, Matthew Galgoci wrote:
>>I am going to check it by hand this evening, booted from rescue media.
>>
>>I also have a hardened kernel to install on it that raises the bar on
>>exports through /dev/mem, which is how sukkit is installed.
>
> And, eventually I'll turn off module support in the kernel, which is
> another potential hole.
>
> Long term plans are to use SElinux.  That would be cool even for the
> non-security aspects.

The machine never runs X11, so knocking CAP_SYS_RAWIO out of the
capability bounding set might be a good move.

More draconian: rearrange the filesystem such that / and /usr
partitions can be read-only, then take out CAP_MKNOD and
CAP_SYS_ADMIN.  This is likely to cause a lot more trouble than it's
worth, but it does prevent mucking with most of the important user
space files.

zw

Re: have we been sucked by suckit?

[Phil Edwards]

On Wed, Dec 03, 2003 at 06:50:34PM -0800, Zack Weinberg wrote:
> 
> More draconian: rearrange the filesystem such that / and /usr
> partitions can be read-only, then take out CAP_MKNOD and
> CAP_SYS_ADMIN.  This is likely to cause a lot more trouble than it's
> worth, but it does prevent mucking with most of the important user
> space files.

Setting the immutable attribute on files in /etc (if not already
on a read-only partition) can also be useful.

-- 
As my wings crash through the ocean
I was loved before I lost my honor
Remember me in glory, not on fire eternally
  - Soil & Eclipse

Re: have we been sucked by suckit?

[law]

In message <20031204010855.GB14702@redhat.com>, Christopher Faylor writes:
 >On Wed, Dec 03, 2003 at 04:18:25PM -0800, Joe Buck wrote:
 >>On Wed, Dec 03, 2003 at 05:08:17PM -0700, law@redhat.com wrote:
 >>>In message <20031203152137.A10206@synopsys.com>, Joe Buck writes:
 >>>>Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
 >>>>verified that gcc.gnu.org / sources.redhat.com has not been rooted?
 >>>>And if not, has its kernel been upgraded?
 >>>We're already working to get a 2hr window where our sysadmins can take
 >>>the box down, verify the system hasn't been rooted and make sure it's
 >>>got the appropriate kernel fix.
 >>
 >>There's a suckit detection tool that you could try running before you
 >>take the box down.  See
 >>
 >>http://tsd.student.utwente.nl/skdetect/
 >>
 >>Of course, this only detects one particular rootkit, but it seems to be
 >>the one that was used at Debian and savannah.
 >
 >Thanks for the pointer.  This tool indicates that we're clean for
 >whatever that's worth.
I believe Matthew was going to boot off rescue media and use that for
additional verification.

jeff

Re: have we been sucked by suckit?

[Matthew Galgoci]

On Wed, 3 Dec 2003 law@redhat.com wrote:

> In message <20031204010855.GB14702@redhat.com>, Christopher Faylor writes:
>  >On Wed, Dec 03, 2003 at 04:18:25PM -0800, Joe Buck wrote:
>  >>On Wed, Dec 03, 2003 at 05:08:17PM -0700, law@redhat.com wrote:
>  >>>In message <20031203152137.A10206@synopsys.com>, Joe Buck writes:
>  >>>>Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
>  >>>>verified that gcc.gnu.org / sources.redhat.com has not been rooted?
>  >>>>And if not, has its kernel been upgraded?
>  >>>We're already working to get a 2hr window where our sysadmins can take
>  >>>the box down, verify the system hasn't been rooted and make sure it's
>  >>>got the appropriate kernel fix.
>  >>
>  >>There's a suckit detection tool that you could try running before you
>  >>take the box down.  See
>  >>
>  >>http://tsd.student.utwente.nl/skdetect/
>  >>
>  >>Of course, this only detects one particular rootkit, but it seems to be
>  >>the one that was used at Debian and savannah.
>  >
>  >Thanks for the pointer.  This tool indicates that we're clean for
>  >whatever that's worth.
> I believe Matthew was going to boot off rescue media and use that for
> additional verification.

I don't know how skdetect works, but suckit overrides the kernel interrupt 
vector with its own interrupt vector with fallbacks to the original
interrupt vector. This lets suckit hide itself with impunity.

Any detection kit that claims to be able to 100% detect a kernel level root
kit is snake oil. You need to boot from rescue media and walk the filesystem
looking for telltale signs of compromise.

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: have we been sucked by suckit?

[Matthew Galgoci]

> The machine never runs X11, so knocking CAP_SYS_RAWIO out of the
> capability bounding set might be a good move.

I'd sooner rip out /dev/mem, /dev/kmem, and all the vm86 support.
 
> More draconian: rearrange the filesystem such that / and /usr
> partitions can be read-only, then take out CAP_MKNOD and
> CAP_SYS_ADMIN.  This is likely to cause a lot more trouble than it's
> worth, but it does prevent mucking with most of the important user
> space files.

SELinux will let us restict things nicely. I don't think the draconian approach
above will help terribly.

I'm angling for some selinux training and I hope to make selinux on sources
a reality in the next 6 to 8 months.

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: have we been sucked by suckit?

[Christopher Faylor]

On Thu, Dec 04, 2003 at 12:44:17AM -0500, Matthew Galgoci wrote:
>On Wed, 3 Dec 2003 law@redhat.com wrote:
>
>> In message <20031204010855.GB14702@redhat.com>, Christopher Faylor writes:
>>  >On Wed, Dec 03, 2003 at 04:18:25PM -0800, Joe Buck wrote:
>>  >>On Wed, Dec 03, 2003 at 05:08:17PM -0700, law@redhat.com wrote:
>>  >>>In message <20031203152137.A10206@synopsys.com>, Joe Buck writes:
>>  >>>>Given the attacks on debian, gentoo, and savannah.gnu.org, has anyone
>>  >>>>verified that gcc.gnu.org / sources.redhat.com has not been rooted?
>>  >>>>And if not, has its kernel been upgraded?
>>  >>>We're already working to get a 2hr window where our sysadmins can take
>>  >>>the box down, verify the system hasn't been rooted and make sure it's
>>  >>>got the appropriate kernel fix.
>>  >>
>>  >>There's a suckit detection tool that you could try running before you
>>  >>take the box down.  See
>>  >>
>>  >>http://tsd.student.utwente.nl/skdetect/
>>  >>
>>  >>Of course, this only detects one particular rootkit, but it seems to be
>>  >>the one that was used at Debian and savannah.
>>  >
>>  >Thanks for the pointer.  This tool indicates that we're clean for
>>  >whatever that's worth.
>
>> I believe Matthew was going to boot off rescue media and use that for
>> additional verification.
>
>I don't know how skdetect works, but suckit overrides the kernel interrupt 
>vector with its own interrupt vector with fallbacks to the original
>interrupt vector. This lets suckit hide itself with impunity.
>
>Any detection kit that claims to be able to 100% detect a kernel level root
>kit is snake oil. You need to boot from rescue media and walk the filesystem
>looking for telltale signs of compromise.

In case this is all considered an education for me, let me point you to
the:

"for whatever that's worth..."

in the original email.  This was intended to convey that I understood
that skdetect obviously wasn't a perfect tool.  I wasn't suggesting that
the positive results from running the tool were an indication that
everything was A-OK and that nothing further needed to be done on the
system.

cgf

Re: have we been sucked by suckit?

[Zack Weinberg]

Matthew Galgoci <mgalgoci@redhat.com> writes:

>> The machine never runs X11, so knocking CAP_SYS_RAWIO out of the
>> capability bounding set might be a good move.
>
> I'd sooner rip out /dev/mem, /dev/kmem, and all the vm86 support.

I don't think that does anything about iopl(); but clearing
CAP_SYS_RAWIO does.

Deleting /dev/mem and /dev/kmem is a good idea, but an attacker with
root can just mknod them right back unless CAP_SYS_MKNOD is cleared.

> SELinux will let us restict things nicely. I don't think the
> draconian approach above will help terribly.
>
> I'm angling for some selinux training and I hope to make selinux on sources
> a reality in the next 6 to 8 months.

cool.

zw

Re: have we been sucked by suckit?

[Matthew Galgoci]

On Thu, 4 Dec 2003, Zack Weinberg wrote:

> Matthew Galgoci <mgalgoci@redhat.com> writes:
> 
> >> The machine never runs X11, so knocking CAP_SYS_RAWIO out of the
> >> capability bounding set might be a good move.
> >
> > I'd sooner rip out /dev/mem, /dev/kmem, and all the vm86 support.
> 
> I don't think that does anything about iopl(); but clearing
> CAP_SYS_RAWIO does.
> 
> Deleting /dev/mem and /dev/kmem is a good idea, but an attacker with
> root can just mknod them right back unless CAP_SYS_MKNOD is cleared.

I meant remove the kernel drivers, not the device files.
 
> > SELinux will let us restict things nicely. I don't think the
> > draconian approach above will help terribly.
> >
> > I'm angling for some selinux training and I hope to make selinux on sources
> > a reality in the next 6 to 8 months.
> 
> cool.
> 
> zw
> 

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: have we been sucked by suckit?

[Christopher Faylor]

On Thu, Dec 04, 2003 at 12:39:47AM -0500, Matthew Galgoci wrote:
>On Wed, 3 Dec 2003 law@redhat.com wrote:
>>In message >><Pine.LNX.4.44.0312032115580.6431-100000@lacrosse.corp.redhat.com>,
>>Matthew Galgoci writes:
>>>
>>>Btw I am running late unfortunately :\
>>>
>>>I will be at the colo some time this evening (soon I hope) and get the
>>>checking done as soon as possible.
>>No problem.  It's just good to have more help watching out for the
>>system.
>
>I've finished a cursory exam of the system from rescue media and
>nothing seems amiss.  No sign of sk files or other common rootkits.
>
>I wasn't able to use the kernel that has /dev/mem restricted because I
>didn't have all the iptables modules I needed compiled into it, so I
>used the one that Chris Faylor installed.  The one with the /dev/mem
>restriction was also entirely monolithic with modules disabled.  The
>only wart was the fact that I needed a couple more iptables modules
>compiled into it.

That's why I didn't create a monolithic kernel myself.  I wasn't sure if I'd
gotten all of the iptables modules.  It was going to be a stepped process.

>I'll probably work with chris to schedule another reboot once I have a
>totally monolithic kernel with module support entirely disabled and
>/dev/mem sufficiently neutered.  I will of course send the patch and
>config files to this list so there is no 'lore' about how the kernel is
>built.

Please just send the patch to me personally, or, if you prefer, I'll get
the patch myself and rebuild the kernel.  I don't see any reason to
bother this list with this info.  Only a few people here have the
ability to build a kernel.  It's just spam to everyone else.

FWIW, the kernels are being built in a standard location on
sources.redhat.com.

We should be communicating about sensitive issues like this in another
forum anyway.  Discussing system security holes on an open, archived
mailing list isn't a good idea.

cgf

Re: have we been sucked by suckit?

[Matthew Galgoci]

On Wed, 3 Dec 2003 law@redhat.com wrote:

> In message <Pine.LNX.4.44.0312032115580.6431-100000@lacrosse.corp.redhat.com>, 
> Matthew Galgoci writes:
>  >
>  >Btw I am running late unfortunately :\
>  >
>  >I will be at the colo some time this evening (soon I hope) and get
>  >the checking done as soon as possible.
> No problem.  It's just good to have more help watching out for the system.
> 
> jeff

I've finished a cursory exam of the system from rescue media and nothing
seems amiss. No sign of sk files or other common rootkits.

I wasn't able to use the kernel that has /dev/mem restricted because I didn't
have all the iptables modules I needed compiled into it, so I used the one
that Chris Faylor installed. The one with the /dev/mem restriction was also
entirely monolithic with modules disabled. The only wart was the fact that
I needed a couple more iptables modules compiled into it.

I'll probably work with chris to schedule another reboot once I have a totally
monolithic kernel with module support entirely disabled and /dev/mem 
sufficiently neutered. I will of course send the patch and config files to
this list so there is no 'lore' about how the kernel is built.

-- 
Matthew Galgoci
System Administrator
Red Hat, Inc
919.754.3700 x44155