DB2 problem...: PANIC: No space left on device

DB2 problem...: PANIC: No space left on device

[Hans-Peter Nilsson]

I saw the subject line in both gcc and sourceware htdig logs for Sunday.

I'm taking care of short-term actions now, but I'm on the last line of
defense; bzip2 of db.wordlist between runs.  Win: bzip2 disk compression
of 2*~600M.  Loss: time for bzip2 of 2*~600M.

In other words: can there please be some more disk for htdig use?

It might be that all that is needed (at least short- to middle-term) is
that someone with sufficient powers make a directory writable by htdigid
in /sourceware/snapshot-tmp so the "sort" invocations could use that for
its temp files.  Though the proposed move of either the gcc or sourceware
htdig parts to a partition of its own would be a bit better.

brgds, H-P

DB2 problem...: PANIC: No space left on device

[Hans-Peter Nilsson]

I saw the subject line in both gcc and sourceware htdig logs for Sunday.

I'm taking care of short-term actions now, but I'm on the last line of
defense; bzip2 of db.wordlist between runs.  Win: bzip2 disk compression
of 2*~600M.  Loss: time for bzip2 of 2*~600M.

In other words: can there please be some more disk for htdig use?

It might be that all that is needed (at least short- to middle-term) is
that someone with sufficient powers make a directory writable by htdigid
in /sourceware/snapshot-tmp so the "sort" invocations could use that for
its temp files.  Though the proposed move of either the gcc or sourceware
htdig parts to a partition of its own would be a bit better.

brgds, H-P

Re: DB2 problem...: PANIC: No space left on device

[Jason Molenda]

On Sun, Mar 25, 2001 at 11:42:58PM -0500, Hans-Peter Nilsson wrote:
> 
> In other words: can there please be some more disk for htdig use?
> 

It seems that /dev/hda4 is the partition that we were talking about
mounting on /gcc/htdig.  Is there some reason not to mount this?
If it's available and writable by user htdig, I'm sure either H-P
or I will find the time to do the rest.

Jason

Re: DB2 problem...: PANIC: No space left on device

[Jason Molenda]

On Sun, Mar 25, 2001 at 11:42:58PM -0500, Hans-Peter Nilsson wrote:
> 
> In other words: can there please be some more disk for htdig use?
> 

It seems that /dev/hda4 is the partition that we were talking about
mounting on /gcc/htdig.  Is there some reason not to mount this?
If it's available and writable by user htdig, I'm sure either H-P
or I will find the time to do the rest.

Jason

Re: DB2 problem...: PANIC: No space left on device

[Hans-Peter Nilsson]

On Mon, 26 Mar 2001, Jason Molenda wrote:

> On Sun, Mar 25, 2001 at 11:42:58PM -0500, Hans-Peter Nilsson wrote:
> >
> > In other words: can there please be some more disk for htdig use?
> >
>
> It seems that /dev/hda4 is the partition that we were talking about
> mounting on /gcc/htdig.  Is there some reason not to mount this?
> If it's available and writable by user htdig, I'm sure either H-P
> or I will find the time to do the rest.

Yes.

Sourceware is now spending >2 hours compressing and decompressing files
*only* and still won't do better than an incomplete search database.
That seems unnecessarily bad.

If there's a root person listening, but who do not at this time want to
get /dev/hda4 working (or similar thingy) please consider doing something
like the following:

 mkdir /sourceware/snapshot-tmp/htdig-tmp
 chown htdigid:htdig /sourceware/snapshot-tmp/htdig-tmp

Then there would be room for the temporary-space-consuming sort part, and
the CPU and disk-I/O-consuming compressing-for-space kludge could be
removed, and I would dare to enable indexing of more than 200k of each
index.html and the sun would shine again.

Thanks.

brgds, H-P

Re: DB2 problem...: PANIC: No space left on device

[Hans-Peter Nilsson]

On Mon, 26 Mar 2001, Jason Molenda wrote:

> On Sun, Mar 25, 2001 at 11:42:58PM -0500, Hans-Peter Nilsson wrote:
> >
> > In other words: can there please be some more disk for htdig use?
> >
>
> It seems that /dev/hda4 is the partition that we were talking about
> mounting on /gcc/htdig.  Is there some reason not to mount this?
> If it's available and writable by user htdig, I'm sure either H-P
> or I will find the time to do the rest.

Yes.

Sourceware is now spending >2 hours compressing and decompressing files
*only* and still won't do better than an incomplete search database.
That seems unnecessarily bad.

If there's a root person listening, but who do not at this time want to
get /dev/hda4 working (or similar thingy) please consider doing something
like the following:

 mkdir /sourceware/snapshot-tmp/htdig-tmp
 chown htdigid:htdig /sourceware/snapshot-tmp/htdig-tmp

Then there would be room for the temporary-space-consuming sort part, and
the CPU and disk-I/O-consuming compressing-for-space kludge could be
removed, and I would dare to enable indexing of more than 200k of each
index.html and the sun would shine again.

Thanks.

brgds, H-P

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 06:28:58PM -0500, Hans-Peter Nilsson wrote:
>On Mon, 26 Mar 2001, Jason Molenda wrote:
>
>> On Sun, Mar 25, 2001 at 11:42:58PM -0500, Hans-Peter Nilsson wrote:
>> >
>> > In other words: can there please be some more disk for htdig use?
>> >
>>
>> It seems that /dev/hda4 is the partition that we were talking about
>> mounting on /gcc/htdig.  Is there some reason not to mount this?
>> If it's available and writable by user htdig, I'm sure either H-P
>> or I will find the time to do the rest.
>
>Yes.
>
>Sourceware is now spending >2 hours compressing and decompressing files
>*only* and still won't do better than an incomplete search database.
>That seems unnecessarily bad.
>
>If there's a root person listening, but who do not at this time want to
>get /dev/hda4 working (or similar thingy) please consider doing something
>like the following:
>
> mkdir /sourceware/snapshot-tmp/htdig-tmp
> chown htdigid:htdig /sourceware/snapshot-tmp/htdig-tmp

I've done this.

I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
really need that much space?  Will it help to have that much space or will
the above change be sufficient?

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 06:28:58PM -0500, Hans-Peter Nilsson wrote:
>On Mon, 26 Mar 2001, Jason Molenda wrote:
>
>> On Sun, Mar 25, 2001 at 11:42:58PM -0500, Hans-Peter Nilsson wrote:
>> >
>> > In other words: can there please be some more disk for htdig use?
>> >
>>
>> It seems that /dev/hda4 is the partition that we were talking about
>> mounting on /gcc/htdig.  Is there some reason not to mount this?
>> If it's available and writable by user htdig, I'm sure either H-P
>> or I will find the time to do the rest.
>
>Yes.
>
>Sourceware is now spending >2 hours compressing and decompressing files
>*only* and still won't do better than an incomplete search database.
>That seems unnecessarily bad.
>
>If there's a root person listening, but who do not at this time want to
>get /dev/hda4 working (or similar thingy) please consider doing something
>like the following:
>
> mkdir /sourceware/snapshot-tmp/htdig-tmp
> chown htdigid:htdig /sourceware/snapshot-tmp/htdig-tmp

I've done this.

I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
really need that much space?  Will it help to have that much space or will
the above change be sufficient?

cgf

Re: DB2 problem...: PANIC: No space left on device

[Jason Molenda]

On Wed, Mar 28, 2001 at 07:08:40PM -0500, Christopher Faylor wrote:

> I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
> really need that much space?  Will it help to have that much space or will
> the above change be sufficient?

	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00480.html

	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00482.html

/sourceware/htdig has 6GB, which is waaaay too small for both sourceware and
gcc databases.  6GB will be too small for _one_ of them within the next year.
15GB may not be necessary for the gcc DB, but I'd give it at least 10GB so
this doesn't need to be fiddled with again any time soon.

Jason

Re: DB2 problem...: PANIC: No space left on device

[Jason Molenda]

On Wed, Mar 28, 2001 at 07:08:40PM -0500, Christopher Faylor wrote:

> I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
> really need that much space?  Will it help to have that much space or will
> the above change be sufficient?

	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00480.html

	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00482.html

/sourceware/htdig has 6GB, which is waaaay too small for both sourceware and
gcc databases.  6GB will be too small for _one_ of them within the next year.
15GB may not be necessary for the gcc DB, but I'd give it at least 10GB so
this doesn't need to be fiddled with again any time soon.

Jason

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 04:23:02PM -0800, Jason Molenda wrote:
>On Wed, Mar 28, 2001 at 07:08:40PM -0500, Christopher Faylor wrote:
>
>> I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
>> really need that much space?  Will it help to have that much space or will
>> the above change be sufficient?
>
>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00480.html
>
>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00482.html
>
>/sourceware/htdig has 6GB, which is waaaay too small for both sourceware and
>gcc databases.  6GB will be too small for _one_ of them within the next year.
>15GB may not be necessary for the gcc DB, but I'd give it at least 10GB so
>this doesn't need to be fiddled with again any time soon.

Ok.  I remember now.  I've started the transfer to the larger partition.  I
hope to be done shortly.

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 04:23:02PM -0800, Jason Molenda wrote:
>On Wed, Mar 28, 2001 at 07:08:40PM -0500, Christopher Faylor wrote:
>
>> I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
>> really need that much space?  Will it help to have that much space or will
>> the above change be sufficient?
>
>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00480.html
>
>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00482.html
>
>/sourceware/htdig has 6GB, which is waaaay too small for both sourceware and
>gcc databases.  6GB will be too small for _one_ of them within the next year.
>15GB may not be necessary for the gcc DB, but I'd give it at least 10GB so
>this doesn't need to be fiddled with again any time soon.

Ok.  I remember now.  I've started the transfer to the larger partition.  I
hope to be done shortly.

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 07:40:54PM -0500, Christopher Faylor wrote:
>On Wed, Mar 28, 2001 at 04:23:02PM -0800, Jason Molenda wrote:
>>On Wed, Mar 28, 2001 at 07:08:40PM -0500, Christopher Faylor wrote:
>>
>>> I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
>>> really need that much space?  Will it help to have that much space or will
>>> the above change be sufficient?
>>
>>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00480.html
>>
>>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00482.html
>>
>>/sourceware/htdig has 6GB, which is waaaay too small for both sourceware and
>>gcc databases.  6GB will be too small for _one_ of them within the next year.
>>15GB may not be necessary for the gcc DB, but I'd give it at least 10GB so
>>this doesn't need to be fiddled with again any time soon.
>
>Ok.  I remember now.  I've started the transfer to the larger partition.  I
>hope to be done shortly.

It's done.

One thing that I noticed is that it doesn't look like we're using the maxium
throughput settings for this drive.  Would it make sense to twiddle some of
the parameters with hdparm or is that just too dangerous?

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 07:40:54PM -0500, Christopher Faylor wrote:
>On Wed, Mar 28, 2001 at 04:23:02PM -0800, Jason Molenda wrote:
>>On Wed, Mar 28, 2001 at 07:08:40PM -0500, Christopher Faylor wrote:
>>
>>> I don't remember discussing /dev/hda4.  It is a 15GB partition.  Do we
>>> really need that much space?  Will it help to have that much space or will
>>> the above change be sufficient?
>>
>>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00480.html
>>
>>	http://sourceware.cygnus.com/ml/overseers/2001-q1/msg00482.html
>>
>>/sourceware/htdig has 6GB, which is waaaay too small for both sourceware and
>>gcc databases.  6GB will be too small for _one_ of them within the next year.
>>15GB may not be necessary for the gcc DB, but I'd give it at least 10GB so
>>this doesn't need to be fiddled with again any time soon.
>
>Ok.  I remember now.  I've started the transfer to the larger partition.  I
>hope to be done shortly.

It's done.

One thing that I noticed is that it doesn't look like we're using the maxium
throughput settings for this drive.  Would it make sense to twiddle some of
the parameters with hdparm or is that just too dangerous?

cgf

Re: DB2 problem...: PANIC: No space left on device

[Jason Molenda]

Is this stuff OK from /var/log/messages?

Mar 28 17:30:16 sourceware kernel: EXT2-fs error (device ide0(3,4)): ext2_check_inodes_bitmap: Wrong free inodes count in super block, stored = 4254237, counted = 3937993 
Mar 28 17:31:11 sourceware kernel: EXT2-fs warning: mounting fs with errors, running e2fsck is recommended 



On Wed, Mar 28, 2001 at 08:34:04PM -0500, Christopher Faylor wrote:

> 
> One thing that I noticed is that it doesn't look like we're using the maxium
> throughput settings for this drive.  Would it make sense to twiddle some of
> the parameters with hdparm or is that just too dangerous?

According to the specs,

	http://www.westerndigital.com/products/drives/wd400bb.html

this drive supports the Ultra ATA/100 interface, so enabling DMA
transfers (what I assume you're talking about) should be fine.
It's a modern drive and should handle all those kinds of little
tweaks correctly.

J

Re: DB2 problem...: PANIC: No space left on device

[Jason Molenda]

Is this stuff OK from /var/log/messages?

Mar 28 17:30:16 sourceware kernel: EXT2-fs error (device ide0(3,4)): ext2_check_inodes_bitmap: Wrong free inodes count in super block, stored = 4254237, counted = 3937993 
Mar 28 17:31:11 sourceware kernel: EXT2-fs warning: mounting fs with errors, running e2fsck is recommended 



On Wed, Mar 28, 2001 at 08:34:04PM -0500, Christopher Faylor wrote:

> 
> One thing that I noticed is that it doesn't look like we're using the maxium
> throughput settings for this drive.  Would it make sense to twiddle some of
> the parameters with hdparm or is that just too dangerous?

According to the specs,

	http://www.westerndigital.com/products/drives/wd400bb.html

this drive supports the Ultra ATA/100 interface, so enabling DMA
transfers (what I assume you're talking about) should be fine.
It's a modern drive and should handle all those kinds of little
tweaks correctly.

J

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 05:40:53PM -0800, Jason Molenda wrote:
>
>Is this stuff OK from /var/log/messages?
>
>Mar 28 17:30:16 sourceware kernel: EXT2-fs error (device ide0(3,4)): ext2_check_inodes_bitmap: Wrong free inodes count in super block, stored = 4254237, counted = 3937993 
>Mar 28 17:31:11 sourceware kernel: EXT2-fs warning: mounting fs with errors, running e2fsck is recommended 

Uh oh.  It doesn't look like it.  Damn.  I should have thought to check
the log.

I'm fscking now.  It'll probably take forever.

>On Wed, Mar 28, 2001 at 08:34:04PM -0500, Christopher Faylor wrote:
>> One thing that I noticed is that it doesn't look like we're using the maxium
>> throughput settings for this drive.  Would it make sense to twiddle some of
>> the parameters with hdparm or is that just too dangerous?
>
>According to the specs,
>
>	http://www.westerndigital.com/products/drives/wd400bb.html
>
>this drive supports the Ultra ATA/100 interface, so enabling DMA
>transfers (what I assume you're talking about) should be fine.
>It's a modern drive and should handle all those kinds of little
>tweaks correctly.

Yes.  DMA transfers and multi-block I/O is what I was referring to.
hdparm actually lists the supported DMA transfer types.  It's using DMA
mode now but I don't think it is using the fastest mode available.

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 05:40:53PM -0800, Jason Molenda wrote:
>
>Is this stuff OK from /var/log/messages?
>
>Mar 28 17:30:16 sourceware kernel: EXT2-fs error (device ide0(3,4)): ext2_check_inodes_bitmap: Wrong free inodes count in super block, stored = 4254237, counted = 3937993 
>Mar 28 17:31:11 sourceware kernel: EXT2-fs warning: mounting fs with errors, running e2fsck is recommended 

Uh oh.  It doesn't look like it.  Damn.  I should have thought to check
the log.

I'm fscking now.  It'll probably take forever.

>On Wed, Mar 28, 2001 at 08:34:04PM -0500, Christopher Faylor wrote:
>> One thing that I noticed is that it doesn't look like we're using the maxium
>> throughput settings for this drive.  Would it make sense to twiddle some of
>> the parameters with hdparm or is that just too dangerous?
>
>According to the specs,
>
>	http://www.westerndigital.com/products/drives/wd400bb.html
>
>this drive supports the Ultra ATA/100 interface, so enabling DMA
>transfers (what I assume you're talking about) should be fine.
>It's a modern drive and should handle all those kinds of little
>tweaks correctly.

Yes.  DMA transfers and multi-block I/O is what I was referring to.
hdparm actually lists the supported DMA transfer types.  It's using DMA
mode now but I don't think it is using the fastest mode available.

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 08:55:19PM -0500, Christopher Faylor wrote:
>On Wed, Mar 28, 2001 at 05:40:53PM -0800, Jason Molenda wrote:
>>
>>Is this stuff OK from /var/log/messages?
>>
>>Mar 28 17:30:16 sourceware kernel: EXT2-fs error (device ide0(3,4)): ext2_check_inodes_bitmap: Wrong free inodes count in super block, stored = 4254237, counted = 3937993 
>>Mar 28 17:31:11 sourceware kernel: EXT2-fs warning: mounting fs with errors, running e2fsck is recommended 
>
>Uh oh.  It doesn't look like it.  Damn.  I should have thought to check
>the log.
>
>I'm fscking now.  It'll probably take forever.

Uh oh.  I don't like the looks of this fsck.  I think that something is
not right with this disk.  I recently saw similar breakage with a large disk
and an older linux kernel.

I've stopped the fsck.  I hope I haven't damaged anything on any other
partition.  Gulp.

cgf

Re: DB2 problem...: PANIC: No space left on device

[Christopher Faylor]

On Wed, Mar 28, 2001 at 08:55:19PM -0500, Christopher Faylor wrote:
>On Wed, Mar 28, 2001 at 05:40:53PM -0800, Jason Molenda wrote:
>>
>>Is this stuff OK from /var/log/messages?
>>
>>Mar 28 17:30:16 sourceware kernel: EXT2-fs error (device ide0(3,4)): ext2_check_inodes_bitmap: Wrong free inodes count in super block, stored = 4254237, counted = 3937993 
>>Mar 28 17:31:11 sourceware kernel: EXT2-fs warning: mounting fs with errors, running e2fsck is recommended 
>
>Uh oh.  It doesn't look like it.  Damn.  I should have thought to check
>the log.
>
>I'm fscking now.  It'll probably take forever.

Uh oh.  I don't like the looks of this fsck.  I think that something is
not right with this disk.  I recently saw similar breakage with a large disk
and an older linux kernel.

I've stopped the fsck.  I hope I haven't damaged anything on any other
partition.  Gulp.

cgf

Re: DB2 problem...: PANIC: No space left on device

[Frank Ch. Eigler]

Hi -

On Wed, Mar 28, 2001 at 09:03:02PM -0500, Christopher Faylor wrote:
: [...]
: Uh oh.  I don't like the looks of this fsck.  I think that something is
: not right with this disk.  I recently saw similar breakage with a large disk
: and an older linux kernel.
: [...]

Another note of concern is that turning on aggressive IDE 
performance parameters is also risky with older kernels.
Keep DMA etc. off.

- FChE

Re: DB2 problem...: PANIC: No space left on device

[Frank Ch. Eigler]

Hi -

On Wed, Mar 28, 2001 at 09:03:02PM -0500, Christopher Faylor wrote:
: [...]
: Uh oh.  I don't like the looks of this fsck.  I think that something is
: not right with this disk.  I recently saw similar breakage with a large disk
: and an older linux kernel.
: [...]

Another note of concern is that turning on aggressive IDE 
performance parameters is also risky with older kernels.
Keep DMA etc. off.

- FChE
-- 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE6wppnVZbdDOm/ZT0RAnX+AJ0RaX0KrhAtROPoBQPEOyjGSqiM/QCfR35C
MiGKFMkXEJpv/+gigPCSGCM=
=zECE
-----END PGP SIGNATURE-----

Re: DB2 problem...: PANIC: No space left on device

[Hans-Peter Nilsson]

On Wed, 28 Mar 2001, Christopher Faylor wrote:
> On Wed, Mar 28, 2001 at 06:28:58PM -0500, Hans-Peter Nilsson wrote:
> > mkdir /sourceware/snapshot-tmp/htdig-tmp
> > chown htdigid:htdig /sourceware/snapshot-tmp/htdig-tmp
>
> I've done this.

Thanks!

I switched to using that space rather than /sourceware/www/htdig-tmp/
(but thanks, Jason!) as I'd rather not have htdig play in the WWW
partition.

I've started re-indexing with the 1e6 document limit that I committed
*just* before sourceware went down.  (I've been away since then.)

brgds, H-P

Re: DB2 problem...: PANIC: No space left on device

[Hans-Peter Nilsson]

On Wed, 28 Mar 2001, Christopher Faylor wrote:
> On Wed, Mar 28, 2001 at 06:28:58PM -0500, Hans-Peter Nilsson wrote:
> > mkdir /sourceware/snapshot-tmp/htdig-tmp
> > chown htdigid:htdig /sourceware/snapshot-tmp/htdig-tmp
>
> I've done this.

Thanks!

I switched to using that space rather than /sourceware/www/htdig-tmp/
(but thanks, Jason!) as I'd rather not have htdig play in the WWW
partition.

I've started re-indexing with the 1e6 document limit that I committed
*just* before sourceware went down.  (I've been away since then.)

brgds, H-P