we seem to be alive

we seem to be alive

[Christopher Faylor]

We're back online.

As most of you know, we suffered a pretty severe hardware outage on
Thursday.  The best theory right now is that a bad hard drive in the
RAID array indirectly caused a data corruption problem.  The problem was
really due to a problem with old firmware (which mgalgoci has since
updated) which we think caused corruption when the replacement disk was
brought online.

We've experimented to make sure that the new firmware does not duplicate
this problem and, as far as we can tell, we are ok from now on, so this
problem should not reoccur.

Restoral information:

The CVS repository was restored to its state about ~2 hours before the
system was brought down on Thursday at around 1PM EST (18:00 GMT).  The
other volumes were restored from backups that were less than 24 hours
old.

After the CVS volume was restored, Ian Taylor added any missing checkins
to the gcc repository.  Other repositories reflect the last backup state.
So, it is possible that some repositories may be in an odd state now
with the data on a user's disk appearing to be newer than what is in
CVS.

So we did lose some data.  It may be noticeable in the web pages or it
is possible that we lost some subscription information so that someone
who subscribed to a mailing list may have to resubscribe.  If something
was transferred to ftp, it may have to be transferred again.

htdig is down and may be down and out.  There is an ominous internal
error now if you attempt to search.  I'll fix that tomorrow (unless
someone beats me to it).  The fate of htdig is still in question,
however.  It hasn't been running right lately, no one wants to
maintain it, and it may not be the best search solution.

I hope that Angela and Ian will respond to this message with any
information that I missed.

Kudos:

Matt Galgoci was the man onsite who got everything working after the
hard drive and subsequent RAID firmware problems.  We'd be totally dead
in the water if we didn't have someone like him available to help out.

The free software community owes a huge debt of thanks to Angela Thomas
for 1) backing up the system so regularly and so reliably and 2)
spending countless hours in the last several days transferring the
backups, commiserating on the best way to get the system up and running,
and generally doing whatever it took to get the system up.

Ian Taylor also provided his usual services, making sure that
qmail was working ok and providing general guiding advice.

And a BIG thanks to Daniel Berlin.  His knowledge of RAID, LVM,
mysql, and just general technical expertise were invaluable.  He
stopped us from panicking when the system came back up with
what appeared to be missing logical volume information by
providing us with the right commands to do to restore things
to a sane state.

And now I'm going to sleep.

cgf

Re: we seem to be alive

[Angela Marie Thomas]

> I hope that Angela and Ian will respond to this message with any
> information that I missed.

I updated the gcc.gnu.org and sources.redhat.com home pages with
a very short note that the system suffered hardware failure and
was restored from backups.  I also rotated out the old news for
sources.redhat.com and made the news format in sources.redhat.com's
index.html the same as gcc.gnu.org to make it more readable and
easier to copy into news.html.  I'm sure gcc folks will forgive me
giving myself write privs temporarily to update the web pages.  O:-)

The links for /usr/bonsaitools and /usr/lib/sendmail were missing
which caused some failures for bugzilla.  Since they didn't seem
to come from an RPM, I restored them.

It looks like the bzip2 on the system is busted.  It won't extract
my root backup archive even though the md5sum is OK.  The archive
tested OK with bunzip2 -tvv on 2 other systems (same md5sum).
I copied over a bzip2 from my machine to a tmp directory, ran it
and it seemed to work OK.  This is very odd since there didn't
seem to be any differences between the bzip2 on the system and
the one in the backups.  I'm too tired to figure it out tonight.
Maybe it will go away in the morning.

Once I got the backup version of /usr, I diffed against what's
installed.  There are several things missing.  Here are some of
the interesting ones:
  /usr/bin/cpan (which I think means the installed perl is older)
  /usr/bin/cron
  ksymoops (definitely want this back)
  postgresql
  ruby
  perl-Parse-Yapp
  /usr/src/linux*
  openssl
  kernel-doc
There are also some things that are just different.  The ones that
stick out are cvsup and rsync.  Nothing else stuck out as needing
fixing tonight.

I'm resuming the regular backups tonight (04:13 Pacific) and Chris
turned on the rdiff-backup backups as well.

There were fileattr problems for some folks.  After talking,
Chris and I decided to brute force clear all the fileattr files.
I zeroed them out and made sure they were all owned by anoncvs.
Trivia of the Night: There's only one directory in the repository
that has a space in its name.  O:-)

While fixing the fileattrs, we noticed that the eclipse project was
accidentally created in the wrong location.  Instead of being in
/sourceware/projects with a link in /cvs, the directory is in /cvs.
Besides being incorrect, it also means it won't get the added backups
the other projects get.  Can whomever created it please fix it?

I cleaned up htdig and moved things we wanted to save to
/pool/crash-050203 so H-P can do his thing.  Should be able to nuke
all the temporary files tomorrow.  There may still be a few non-ml
files from the www lost+found we want to try to recover.

> Kudos:

Here here.

But let's not forget our lead singer.  Kudos to you, too, for sticking
through it, migraine and all.  And to The Other Angela for losing her
hubby for a few days because of this.

And to my sweetie, Geoff Keating, for burning the DVDs for me while
I worked with the rest of the team to get the beast up and running.
Not like he had anything better to do, though.  O:-)

> And now I'm going to sleep.

Me, too.

--Angela

Re: we seem to be alive

[Joseph S. Myers]

On Sun, 6 Feb 2005, Christopher Faylor wrote:

> So we did lose some data.  It may be noticeable in the web pages or it
> is possible that we lost some subscription information so that someone
> who subscribed to a mailing list may have to resubscribe.  If something
> was transferred to ftp, it may have to be transferred again.

Are personal list mailboxes wanted to sort out gaps in the mailing list 
archives, or is everything for that in the qmail archives?  (Regarding the 
gcc list, for example, the ezmlm message numbers continue straight on from 
before the outage without going backwards, but web archive message numbers 
from http://gcc.gnu.org/ml/gcc/2005-02/msg00079.html to 
http://gcc.gnu.org/ml/gcc/2005-02/msg00098.html are used twice in the 
index (i.e. links to older messages in the archive index actually point to 
newer ones); it seems there are messages from some time on Wednesday until 
the system went down to recover; and I haven't examined the mbox archives.

Thanks to everyone for fixing the system!
-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: we seem to be alive

[Hans-Peter Nilsson]

On Sun, 6 Feb 2005, Christopher Faylor wrote:
> htdig is down and may be down and out.  There is an ominous internal
> error now if you attempt to search.

Only because I removed the (rebuildable) huge databases to make
room for temporary storage for the (manual) restore process.

>  I'll fix that tomorrow (unless
> someone beats me to it).

Please don't, I'll fix it.  Save for the ml archives possibly
being wrong and the integrity of bzip2 being questioned that is.

>  The fate of htdig is still in question,
> however.  It hasn't been running right lately, no one wants to
> maintain it, and it may not be the best search solution.

The biggest problems (as mentioned on IRC):

- Long update times (a replacement *really should* handle this
  better.  I have my doubts; they must be able to index *files*
  as htdig does, rather than through web accesses to achieve this.)
  File access slowness due to malfunctioning hardware didn't
  help, I guess.

- 2G limit.  A replacement *must* handle this.

There's 5% left to the 2G limit.

> Kudos:

To all you involved!

brgds, H-P

Re: we seem to be alive

[Christopher Faylor]

On Sun, Feb 06, 2005 at 08:02:23AM -0500, Hans-Peter Nilsson wrote:
>On Sun, 6 Feb 2005, Christopher Faylor wrote:
>> htdig is down and may be down and out.  There is an ominous internal
>> error now if you attempt to search.
>
>Only because I removed the (rebuildable) huge databases to make
>room for temporary storage for the (manual) restore process.
>
>>  I'll fix that tomorrow (unless
>> someone beats me to it).
>
>Please don't, I'll fix it.  Save for the ml archives possibly
>being wrong and the integrity of bzip2 being questioned that is.

Ok.

I don't understand the bzip2 problem.  I bzip'ed a lot of files
in the last few days, downloaded them to my machine, and extracted
them without problem.

cgf

Re: we seem to be alive

[Christopher Faylor]

On Sun, Feb 06, 2005 at 04:41:43AM -0800, Angela Marie Thomas wrote:
>> I hope that Angela and Ian will respond to this message with any
>> information that I missed.
>
>I updated the gcc.gnu.org and sources.redhat.com home pages with
>a very short note that the system suffered hardware failure and
>was restored from backups.  I also rotated out the old news for
>sources.redhat.com and made the news format in sources.redhat.com's
>index.html the same as gcc.gnu.org to make it more readable and
>easier to copy into news.html.  I'm sure gcc folks will forgive me
>giving myself write privs temporarily to update the web pages.  O:-)
>
>The links for /usr/bonsaitools and /usr/lib/sendmail were missing
>which caused some failures for bugzilla.  Since they didn't seem
>to come from an RPM, I restored them.
>
>It looks like the bzip2 on the system is busted.  It won't extract
>my root backup archive even though the md5sum is OK.  The archive
>tested OK with bunzip2 -tvv on 2 other systems (same md5sum).
>I copied over a bzip2 from my machine to a tmp directory, ran it
>and it seemed to work OK.  This is very odd since there didn't
>seem to be any differences between the bzip2 on the system and
>the one in the backups.  I'm too tired to figure it out tonight.
>Maybe it will go away in the morning.
>
>Once I got the backup version of /usr, I diffed against what's
>installed.  There are several things missing.  Here are some of
>the interesting ones:
>  /usr/bin/cpan (which I think means the installed perl is older)

No.  We just don't need this.
>  /usr/bin/cron

Hmm.  This is not a real program.  It's a symptom of being hacked, I believe.
That's not a good sign.

>  ksymoops (definitely want this back)

I just didn't install this.  Since we've never used it

>  postgresql

Or this.

>  ruby
>  perl-Parse-Yapp

Or these.

>  /usr/src/linux*

Or these.  We aren't using the kernel sources anymore.

>  openssl

?  openssl is there.

>  kernel-doc

I've put this back.

>There are also some things that are just different.  The ones that
>stick out are cvsup and rsync.  Nothing else stuck out as needing
>fixing tonight.

I installed newer versions of both cvsup and rsync.

I actually tediously went through the list of rpm packages and installed
anything that was missing.  I also took the opportunity to not install
packages like 'ruby' which didn't seem to be required.

cgf

Re: we seem to be alive

[Hans-Peter Nilsson]

On Sun, 6 Feb 2005, Christopher Faylor wrote:
> I don't understand the bzip2 problem.  I bzip'ed a lot of files
> in the last few days, downloaded them to my machine, and extracted
> them without problem.

The way I read Angela's notes, it seemed the problem exposed was
the other way round, with unbzipping rather than bzipping.

Hm, something's still fishy, related or not.  I got a SEGV from
the newly compiled htdig.  Could have been a bug exposed just by
using --verbose (-v), but then again...

brgds, H-P

Re: we seem to be alive

[Christopher Faylor]

On Sun, Feb 06, 2005 at 12:19:41PM -0500, Hans-Peter Nilsson wrote:
>On Sun, 6 Feb 2005, Christopher Faylor wrote:
>> I don't understand the bzip2 problem.  I bzip'ed a lot of files
>> in the last few days, downloaded them to my machine, and extracted
>> them without problem.
>
>The way I read Angela's notes, it seemed the problem exposed was
>the other way round, with unbzipping rather than bzipping.

Right.  That's what I meant by "extracted them".

cgf

Re: we seem to be alive

[Hans-Peter Nilsson]

On Sun, 6 Feb 2005, Christopher Faylor wrote:
> On Sun, Feb 06, 2005 at 12:19:41PM -0500, Hans-Peter Nilsson wrote:
> >On Sun, 6 Feb 2005, Christopher Faylor wrote:
> >> I don't understand the bzip2 problem.  I bzip'ed a lot of files
> >> in the last few days, downloaded them to my machine, and extracted
> >> them without problem.
> >
> >The way I read Angela's notes, it seemed the problem exposed was
> >the other way round, with unbzipping rather than bzipping.
>
> Right.  That's what I meant by "extracted them".

But not "extracted them on sourceware, using the sourceware
installation of b{un}zip2", which seems the key?

brgds, H-P

Re: we seem to be alive

[Christopher Faylor]

On Sun, Feb 06, 2005 at 12:28:43PM -0500, Hans-Peter Nilsson wrote:
>On Sun, 6 Feb 2005, Christopher Faylor wrote:
>> On Sun, Feb 06, 2005 at 12:19:41PM -0500, Hans-Peter Nilsson wrote:
>> >On Sun, 6 Feb 2005, Christopher Faylor wrote:
>> >> I don't understand the bzip2 problem.  I bzip'ed a lot of files
>> >> in the last few days, downloaded them to my machine, and extracted
>> >> them without problem.
>> >
>> >The way I read Angela's notes, it seemed the problem exposed was
>> >the other way round, with unbzipping rather than bzipping.
>>
>> Right.  That's what I meant by "extracted them".
>
>But not "extracted them on sourceware, using the sourceware
>installation of b{un}zip2", which seems the key?

I did both, actually.

cgf

Mailing list archives (Re: we seem to be alive)

[Jason Molenda]

On Sun, Feb 06, 2005 at 12:44:39PM +0000, Joseph S. Myers wrote:

> Are personal list mailboxes wanted to sort out gaps in the mailing list 
> archives, or is everything for that in the qmail archives?  (Regarding the 
> gcc list, for example, the ezmlm message numbers continue straight on from 
> before the outage without going backwards, but web archive message numbers 
> from http://gcc.gnu.org/ml/gcc/2005-02/msg00079.html to 
> http://gcc.gnu.org/ml/gcc/2005-02/msg00098.html are used twice in the 
> index (i.e. links to older messages in the archive index actually point to 
> newer ones); it seems there are messages from some time on Wednesday until 
> the system went down to recover; and I haven't examined the mbox archives.

(I've been completely uninvolved in the system recovery so don't
take anything I say as particularly authoritative)

It looks like the ezmlm archives are correct (read: I looked at one
list and it looks OK therefore all lists are OK :).  Fixing the web
archives from the ezmlm archives is going to be messy because the
ordering of the messages is different.  Unfortunately the ftp mbox
archives are of little help because they too are in a different
order than either the ezmlm or the web archives.

I'll probably have to do another thing where I create brand new
"2005-02n" directories that are populated ezmlm-order and abandon
the current "2005-02" directories so that any links into archvies
depending on the order of the current 2005-02 messages aren't broken.
It's been years since I've had to do this, so I'll have to think
about it a bit.  And unfortunately I'm obligated to go to a coworker's
wife's birthday party in a little bit so I'm out for the afternoon.
But I'll dig into this Sunday evening PST (Monday morning UTC) and
figure something out.

(in case anyone is wondering, we have raw one-file-per-message
archives in the /www partition that *are* kept in the same order
as the mhonarc archives for specifically this purpose -- if those
are in place, a corrupt mhonarc archvie can be regenerated easily
and in the correct order.  Unfortunately for such a total system
failure these don't help us out much.)


J

Re: we seem to be alive

[Angela Marie Thomas]

> >  /usr/bin/cpan (which I think means the installed perl is older)
> 
> No.  We just don't need this.

On my FC2 system, it's part of perl-5.8.3 which is why I thought it
indicated that the new system had an older perl.

> >  /usr/bin/cron
> 
> Hmm.  This is not a real program.  It's a symptom of being hacked, I believe.
> That's not a good sign.

Suck

> >  ksymoops (definitely want this back)
> 
> I just didn't install this.  Since we've never used it

OK.  Normally it's useful but I guess we've not been having the kernel
oops as a problem.

> >  openssl
> 
> ?  openssl is there.

Oops, sorry.  The RPM is there but the certs are missing.

  Only in root/usr/share/ssl: .rnd
  Only in root/usr/share/ssl/certs: ftpd-rsa.pem
  Only in root/usr/share/ssl/certs: imapd.pem
  Only in root/usr/share/ssl/certs: ipop3d.pem
  Only in root/usr/share/ssl/certs: slapd.pem
  Only in root/usr/share/ssl/private: ftpd-rsa-key.pem

> I actually tediously went through the list of rpm packages and installed
> anything that was missing.  I also took the opportunity to not install
> packages like 'ruby' which didn't seem to be required.

I didn't doubt that.  I just wasn't sure how much of /usr was missing
due to not being in an RPM since a few links were missing.

--Angela

Re: we seem to be alive

[Christopher Faylor]

On Sun, Feb 06, 2005 at 12:48:27PM -0800, Angela Marie Thomas wrote:
>> >  /usr/bin/cpan (which I think means the installed perl is older)
>> 
>> No.  We just don't need this.
>
>On my FC2 system, it's part of perl-5.8.3 which is why I thought it
>indicated that the new system had an older perl.


>> >  /usr/bin/cron
>> 
>> Hmm.  This is not a real program.  It's a symptom of being hacked, I believe.
>> That's not a good sign.
>
>Suck

Do you still have this file somewhere where it can be inspected?  I'd like to
see how recent it is.

I deleted one of these from my system last year (after that wonderful
awful sinking feeling) so that's why I was aware of the file.  Although,
OTOH, maybe I'm wrong and this is valid.


>>>ksymoops (definitely want this back)
>>
>>I just didn't install this.  Since we've never used it
>
>OK.  Normally it's useful but I guess we've not been having the kernel
>oops as a problem.

I looked around for it to recreate it but couldn't find the right rpm to
install after a very cursory search.

>> >  openssl
>> 
>> ?  openssl is there.
>
>Oops, sorry.  The RPM is there but the certs are missing.
>
>  Only in root/usr/share/ssl: .rnd
>  Only in root/usr/share/ssl/certs: ftpd-rsa.pem
>  Only in root/usr/share/ssl/certs: imapd.pem
>  Only in root/usr/share/ssl/certs: ipop3d.pem
>  Only in root/usr/share/ssl/certs: slapd.pem
>  Only in root/usr/share/ssl/private: ftpd-rsa-key.pem

Ah, right.  I noticed that some of the certs and GPG keys weren't
installed.  I figured it would become obvious eventually if they were
required.

>> I actually tediously went through the list of rpm packages and installed
>> anything that was missing.  I also took the opportunity to not install
>> packages like 'ruby' which didn't seem to be required.
>
>I didn't doubt that.  I just wasn't sure how much of /usr was missing
>due to not being in an RPM since a few links were missing.

I checked most directories but I forgot to check /usr/lib.  I did make
sure that /usr/games was fully intact, FWIW.

cgf

Re: we seem to be alive

[Angela Marie Thomas]

> >> >  /usr/bin/cron
> >> 
> >> Hmm.  This is not a real program.  It's a symptom of being hacked, I belie
ve.
> >> That's not a good sign.
> >
> >Suck
> 
> Do you still have this file somewhere where it can be inspected?  I'd like to
> see how recent it is.
> 
> I deleted one of these from my system last year (after that wonderful
> awful sinking feeling) so that's why I was aware of the file.  Although,
> OTOH, maybe I'm wrong and this is valid.

Yup.  I'll extract it from the archive.  I'll let you know when that's done.

> Ah, right.  I noticed that some of the certs and GPG keys weren't
> installed.  I figured it would become obvious eventually if they were
> required.

I have those as well if we need them back.

--Angela

Re: several messages

[Gerald Pfeifer]

On Sun, 6 Feb 2005, Christopher Faylor wrote:
> And now I'm going to sleep.

Thanks to you and everyone else involved for bringing the machine up
again!

On Sun, 6 Feb 2005, Angela Marie Thomas wrote:
> I also rotated out the old news for sources.redhat.com and made the news 
> format in sources.redhat.com's index.html the same as gcc.gnu.org to 
> make it more readable and easier to copy into news.html.  I'm sure gcc 
> folks will forgive me giving myself write privs temporarily to update 
> the web pages.  O:-)

I'll discuss this with Gerald. ;-)

Gerald

Re: Mailing list archives (Re: we seem to be alive)

[Jason Molenda]

> It's been years since I've had to do this, so I'll have to think
> about it a bit.  And unfortunately I'm obligated to go to a coworker's
> wife's birthday party in a little bit so I'm out for the afternoon.
> But I'll dig into this Sunday evening PST (Monday morning UTC) and
> figure something out.

Uh, cough cough, I didn't get time to do it and it's getting really
late here. :/  It looks like Chris has the archives functioning
sans the notes that got dropped -- I'll try to get the missing mails
integrated tomorrow night after jazzercise.

J-the-huge-slacker

Re: Mailing list archives (Re: we seem to be alive)

[Joseph S. Myers]

On Mon, 7 Feb 2005, Jason Molenda wrote:

> Uh, cough cough, I didn't get time to do it and it's getting really
> late here. :/  It looks like Chris has the archives functioning
> sans the notes that got dropped -- I'll try to get the missing mails

(For the lists he listed, e.g. the gcc-testresults index still has 
duplication from 00090 to 00092 and no doubt needs the same treatment.)

MHonarc is supposed to detect duplicates using Message-ID, so in principle 
feeding in more of the qmail archives than is strictly necessary (to give 
some room for the archives being in different order) should suffice to get 
the missing messages included.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)