hard disk recovered?

hard disk recovered?

[Christopher Faylor]

I think that I've recovered all of the hard disk info from the previously
mounted SCSI disk and inserted the information into the new IDE drive.
Any corruption in things like GNATS should be rectified, too -- at least
with regard to the info on the old disk.

That means that any corruption since our last "backup" which was
sometime around 2001-03-19 at 19:00 GMT and the crash, which was
sometime on 2001-03-28 is probably unretrievable.

As I stated before, we are reviewing our backup policies.  Somebody
really dropped the ball on this one (me being one of the somebody's).
We're going to be tightening our backup procedures ASAP.

Anyway, if you still see any hard disk problems, please send them
to overseers@redhat.com.

Thanks,
cgf

Re: hard disk recovered?

[Hans-Peter Nilsson]

On Sat, 7 Apr 2001, Christopher Faylor wrote:
> Anyway, if you still see any hard disk problems, please send them
> to overseers@redhat.com.

Errors as reported in
<URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00058.html > are
still there.  In fact, it's so bad that I haven't even been able to get
the GCC search up since then due to a mix of disk-full errors and the
"access beyond end of device" errors.

In an attempt to limp along with the now-to-small htdig partition, I've
re-enabled bzip2 of db.wordlist as mentioned before.

Anyway, thanks to you and Jeff Law for looking into the disk issues.

brgds, H-P

Re: hard disk recovered?

[Christopher Faylor]

On Sat, Apr 07, 2001 at 10:23:03PM -0400, Hans-Peter Nilsson wrote:
>On Sat, 7 Apr 2001, Christopher Faylor wrote:
>> Anyway, if you still see any hard disk problems, please send them
>> to overseers@redhat.com.
>
>Errors as reported in
><URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00058.html > are
>still there.  In fact, it's so bad that I haven't even been able to get
>the GCC search up since then due to a mix of disk-full errors and the
>"access beyond end of device" errors.

Yes.  I didn't do anything to fix those, unfortunately.  I was just
focusing on the IDE problems.  I should have made that clear.

I was going to bring down the htdig partition for one more fruitless
attempt at fixing this problem but it looks like htdig is running on it
now.

If I wanted to just stop everything htdig related for a while, do I have
to do more than just umount the partition?  I'm thinking about backing
up and mkfs'ing the partition again to see if that solves things.

>In an attempt to limp along with the now-to-small htdig partition, I've
>re-enabled bzip2 of db.wordlist as mentioned before.
>
>Anyway, thanks to you and Jeff Law for looking into the disk issues.

It sure is frustrating not being in front of the machine.  I'm thinking
of just flying out to Sunnyvale and camping out in the computer room
for a week or so.

cgf

Re: hard disk recovered?

[Hans-Peter Nilsson]

On Sat, 7 Apr 2001, Christopher Faylor wrote:
> I was going to bring down the htdig partition for one more fruitless
> attempt at fixing this problem

I stumbled on "/sbin/badblocks" which may or may not be useful; I've never
run it.

> but it looks like htdig is running on it
> now.

Yep.  Please wait a little while, it's in the word-sorting phase.  See
"~htdigid/gcc/update-log.failed.2001-04-07d".  It might just succeed this
time; fingers crossed.  It's soon over anyway...

> If I wanted to just stop everything htdig related for a while, do I have
> to do more than just umount the partition?  I'm thinking about backing
> up and mkfs'ing the partition again to see if that solves things.

See <URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00001.html >,
last paragraph.

> It sure is frustrating not being in front of the machine.  I'm thinking
> of just flying out to Sunnyvale and camping out in the computer room
> for a week or so.

:-)

brgds, H-P

Re: hard disk recovered?

[Christopher Faylor]

On Sat, Apr 07, 2001 at 10:55:16PM -0400, Hans-Peter Nilsson wrote:
>On Sat, 7 Apr 2001, Christopher Faylor wrote:
>> I was going to bring down the htdig partition for one more fruitless
>> attempt at fixing this problem
>
>I stumbled on "/sbin/badblocks" which may or may not be useful; I've never
>run it.

Hmm.  fsck should run this automatically if you give it the -c option,
or at least that is how I'm interpreting what I've read.  I guess I'll
try that next.  Thanks for the pointer.

>> but it looks like htdig is running on it
>> now.
>
>Yep.  Please wait a little while, it's in the word-sorting phase.  See
>"~htdigid/gcc/update-log.failed.2001-04-07d".  It might just succeed this
>time; fingers crossed.  It's soon over anyway...
>
>> If I wanted to just stop everything htdig related for a while, do I have
>> to do more than just umount the partition?  I'm thinking about backing
>> up and mkfs'ing the partition again to see if that solves things.
>
>See <URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00001.html >,
>last paragraph.

Thanks for the reference.  Sorry that I didn't recall reading this before.
I'm watching the process now.  I hope that the fact that the seconds are
ticking and there are no apparent errors in /var/log/messages is a good
sign.

Once this completes, I'll umount the partition and run fsck again.  I
just upgraded e2fsck on sourceware.  Maybe the new version will find
something that previously was missed.

cgf

Re: hard disk recovered?

[Hans-Peter Nilsson]

On Sat, 7 Apr 2001, Christopher Faylor wrote:
> I'm watching the process now.  I hope that the fact that the seconds are
> ticking and there are no apparent errors in /var/log/messages is a good
> sign.

Yes!  There's now a working index for GCC!  Finally.  Now there's just a
15min delay until the bzip2 completes...

brgds, H-P

Re: hard disk recovered?

[law]

  In message < 20010407214932.A18032@redhat.com >you write:
  > I think that I've recovered all of the hard disk info from the
  > previously mounted SCSI disk and inserted the information into the
  > new IDE drive.  Any corruption in things like GNATS should be
  > rectified, too -- at least with regard to the info on the old disk.
Thanks.  I *knew* there was something I needed to do before heading
out to Flordia for a couple days on the beach to decompress.  Thanks
for finding the right partitions and such on sda.

The plans for my next trip to Sunnyvale are to get the kernel reved
up to the current 2.2.x version to pick up the ide big disk geometry
fix, then enable the hda4 partition.
jeff

htdig crontab on sourceware

[Jason Molenda]

Hi Hans-Peter,

37 2 * * * /bin/sh /sourceware/htdig/gcc/bin/htupdate.sh
37 7 * * * /bin/sh /sourceware/libre/infra/bin/htupdate-sourceware.sh

The sourceware htdig run is causing some load problems.  The actual
htdig process didn't get started until 10:30am, and the system is
hella busy at that point.  sourceware is currently running a load
average of over ten; this just isn't a good time to be running
htdig.  (there isn't any one culprit - there are a lot of people
doing anoncvs updates, web accesses, sending e-mail, etc, nad htdig
is just one more busy process on top of all that.)

How about we movie it to 20.00 localtime?  By 8pm at night the
system load is very low and a little indexing won't cause significant
problems.  We could push the gcc htupdate back to 3.30 localtime
if you think they might run concurrently and eat up all the disk
space.

Jason

Re: htdig crontab on sourceware

[Hans-Peter Nilsson]

On Tue, 17 Apr 2001, Jason Molenda wrote:

> The sourceware htdig run is causing some load problems.

> How about we movie it to 20.00 localtime?

No problem at all; the time most convenient for the rest of the system
is the best.

> We could push the gcc htupdate back to 3.30 localtime
> if you think they might run concurrently and eat up all the disk
> space.

Whatever time you think is best, though these two updates should stay at
least 5h apart.  As reported, due to *very frequent* disk subsystem
failures, most of the time indexing fails and must be restarted.  That's
why it takes so long and part of the reason why it eats all the CPU.

brgds, H-P

Re: htdig crontab on sourceware

[Christopher Faylor]

On Tue, Apr 17, 2001 at 04:28:30PM -0400, Hans-Peter Nilsson wrote:
>On Tue, 17 Apr 2001, Jason Molenda wrote:
>
>> The sourceware htdig run is causing some load problems.
>
>> How about we movie it to 20.00 localtime?
>
>No problem at all; the time most convenient for the rest of the system
>is the best.
>
>> We could push the gcc htupdate back to 3.30 localtime
>> if you think they might run concurrently and eat up all the disk
>> space.
>
>Whatever time you think is best, though these two updates should stay at
>least 5h apart.  As reported, due to *very frequent* disk subsystem
>failures, most of the time indexing fails and must be restarted.  That's
>why it takes so long and part of the reason why it eats all the CPU.

Yeah, this partition is consistently hosed for some reason.

I ran badblocks on it a while ago and there was no obvious problem.
The OS has been upgraded but I wonder if this is still some sort of
kernel bug.

cgf

Re: htdig crontab on sourceware

[Jason Molenda]

On Tue, Apr 17, 2001 at 01:33:58PM -0700, Jason Molenda wrote:

> OK, I changed the crontab to
> 
> 37 2 * * * /bin/sh /sourceware/htdig/gcc/bin/htupdate.sh
> 37 19 * * * /bin/sh /sourceware/libre/infra/bin/htupdate-sourceware.sh
> 
> 
> This gives us 7 hours between the two runs, and the gcc update has five
> hours before the load on sourceware starts to increase.


Looks like it worked correctly.  The 7pm sourceware htdig process was 
running htmerge when I looked at it around 10pm; the gcc htdig process
was completely finished when I just checked now (8:20am).

J

Re: hard disk recovered?

[Christopher Faylor]

On Sat, Apr 07, 2001 at 10:23:03PM -0400, Hans-Peter Nilsson wrote:
>On Sat, 7 Apr 2001, Christopher Faylor wrote:
>> Anyway, if you still see any hard disk problems, please send them
>> to overseers@redhat.com.
>
>Errors as reported in
><URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00058.html > are
>still there.  In fact, it's so bad that I haven't even been able to get
>the GCC search up since then due to a mix of disk-full errors and the
>"access beyond end of device" errors.

Yes.  I didn't do anything to fix those, unfortunately.  I was just
focusing on the IDE problems.  I should have made that clear.

I was going to bring down the htdig partition for one more fruitless
attempt at fixing this problem but it looks like htdig is running on it
now.

If I wanted to just stop everything htdig related for a while, do I have
to do more than just umount the partition?  I'm thinking about backing
up and mkfs'ing the partition again to see if that solves things.

>In an attempt to limp along with the now-to-small htdig partition, I've
>re-enabled bzip2 of db.wordlist as mentioned before.
>
>Anyway, thanks to you and Jeff Law for looking into the disk issues.

It sure is frustrating not being in front of the machine.  I'm thinking
of just flying out to Sunnyvale and camping out in the computer room
for a week or so.

cgf

Re: hard disk recovered?

[Hans-Peter Nilsson]

On Sat, 7 Apr 2001, Christopher Faylor wrote:
> I was going to bring down the htdig partition for one more fruitless
> attempt at fixing this problem

I stumbled on "/sbin/badblocks" which may or may not be useful; I've never
run it.

> but it looks like htdig is running on it
> now.

Yep.  Please wait a little while, it's in the word-sorting phase.  See
"~htdigid/gcc/update-log.failed.2001-04-07d".  It might just succeed this
time; fingers crossed.  It's soon over anyway...

> If I wanted to just stop everything htdig related for a while, do I have
> to do more than just umount the partition?  I'm thinking about backing
> up and mkfs'ing the partition again to see if that solves things.

See <URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00001.html >,
last paragraph.

> It sure is frustrating not being in front of the machine.  I'm thinking
> of just flying out to Sunnyvale and camping out in the computer room
> for a week or so.

:-)

brgds, H-P

hard disk recovered?

[Christopher Faylor]

I think that I've recovered all of the hard disk info from the previously
mounted SCSI disk and inserted the information into the new IDE drive.
Any corruption in things like GNATS should be rectified, too -- at least
with regard to the info on the old disk.

That means that any corruption since our last "backup" which was
sometime around 2001-03-19 at 19:00 GMT and the crash, which was
sometime on 2001-03-28 is probably unretrievable.

As I stated before, we are reviewing our backup policies.  Somebody
really dropped the ball on this one (me being one of the somebody's).
We're going to be tightening our backup procedures ASAP.

Anyway, if you still see any hard disk problems, please send them
to overseers@redhat.com.

Thanks,
cgf

Re: hard disk recovered?

[Hans-Peter Nilsson]

On Sat, 7 Apr 2001, Christopher Faylor wrote:
> Anyway, if you still see any hard disk problems, please send them
> to overseers@redhat.com.

Errors as reported in
<URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00058.html > are
still there.  In fact, it's so bad that I haven't even been able to get
the GCC search up since then due to a mix of disk-full errors and the
"access beyond end of device" errors.

In an attempt to limp along with the now-to-small htdig partition, I've
re-enabled bzip2 of db.wordlist as mentioned before.

Anyway, thanks to you and Jeff Law for looking into the disk issues.

brgds, H-P

Re: hard disk recovered?

[Christopher Faylor]

On Sat, Apr 07, 2001 at 10:55:16PM -0400, Hans-Peter Nilsson wrote:
>On Sat, 7 Apr 2001, Christopher Faylor wrote:
>> I was going to bring down the htdig partition for one more fruitless
>> attempt at fixing this problem
>
>I stumbled on "/sbin/badblocks" which may or may not be useful; I've never
>run it.

Hmm.  fsck should run this automatically if you give it the -c option,
or at least that is how I'm interpreting what I've read.  I guess I'll
try that next.  Thanks for the pointer.

>> but it looks like htdig is running on it
>> now.
>
>Yep.  Please wait a little while, it's in the word-sorting phase.  See
>"~htdigid/gcc/update-log.failed.2001-04-07d".  It might just succeed this
>time; fingers crossed.  It's soon over anyway...
>
>> If I wanted to just stop everything htdig related for a while, do I have
>> to do more than just umount the partition?  I'm thinking about backing
>> up and mkfs'ing the partition again to see if that solves things.
>
>See <URL: http://sources.redhat.com/ml/overseers/2001-q2/msg00001.html >,
>last paragraph.

Thanks for the reference.  Sorry that I didn't recall reading this before.
I'm watching the process now.  I hope that the fact that the seconds are
ticking and there are no apparent errors in /var/log/messages is a good
sign.

Once this completes, I'll umount the partition and run fsck again.  I
just upgraded e2fsck on sourceware.  Maybe the new version will find
something that previously was missed.

cgf

Re: hard disk recovered?

[Hans-Peter Nilsson]

On Sat, 7 Apr 2001, Christopher Faylor wrote:
> I'm watching the process now.  I hope that the fact that the seconds are
> ticking and there are no apparent errors in /var/log/messages is a good
> sign.

Yes!  There's now a working index for GCC!  Finally.  Now there's just a
15min delay until the bzip2 completes...

brgds, H-P

Re: hard disk recovered?

[Angela Marie Thomas]

>It sure is frustrating not being in front of the machine.  I'm thinking
>of just flying out to Sunnyvale and camping out in the computer room
>for a week or so.

I'm looking at getting a PC Weasel (www.realweasel.com) for it so folks
can get to the console from anywhere in the world.  It's not like we
try to run X or something.  Maybe if we promote them they'll cut us a
deal on the card.

They don't have pricing on their site so I've sent in a request.  The
web page is nearly enough to buy one on general principles.  Ya just
gotta love the customer testimonials being listed under "Asses Saved".
Under construction unfortunately.  I can't wait to see the stories.  O:-)

--Angela

Re: hard disk recovered?

[Angela Marie Thomas]

>It sure is frustrating not being in front of the machine.  I'm thinking
>of just flying out to Sunnyvale and camping out in the computer room
>for a week or so.

I'm looking at getting a PC Weasel (www.realweasel.com) for it so folks
can get to the console from anywhere in the world.  It's not like we
try to run X or something.  Maybe if we promote them they'll cut us a
deal on the card.

They don't have pricing on their site so I've sent in a request.  The
web page is nearly enough to buy one on general principles.  Ya just
gotta love the customer testimonials being listed under "Asses Saved".
Under construction unfortunately.  I can't wait to see the stories.  O:-)

--Angela