Re: www out of space

Re: www out of space

[Christopher Faylor]

On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>/dev/vg1/lvol6        23980436  22742628         0 100% /sourceware/www

ARGH.  It had a little less than a gigabyte yesterday.

I wonder if the fsck reset the free space.

cgf

Re: www out of space

[Christopher Faylor]

On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>/dev/vg1/lvol6        23980436  22742628         0 100% /sourceware/www
>
>ARGH.  It had a little less than a gigabyte yesterday.
>
>I wonder if the fsck reset the free space.

Nope.  It looks like something gobbled up 700MB in 21 hours.

cgf

Re: www out of space

[Daniel Berlin]

On Apr 9, 2004, at 5:24 PM, Christopher Faylor wrote:

> On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>> On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>> /dev/vg1/lvol6        23980436  22742628         0 100% 
>>> /sourceware/www
>>
>> ARGH.  It had a little less than a gigabyte yesterday.
>>
>> I wonder if the fsck reset the free space.
>
> Nope.  It looks like something gobbled up 700MB in 21 hours.
>

It wasn't bugzilla, i checked for that :)
Logs is 1.1 gig. Is that normal?

> cgf

Re: www out of space

[Daniel Berlin]

On Apr 9, 2004, at 5:42 PM, Christopher Faylor wrote:

> On Fri, Apr 09, 2004 at 05:35:07PM -0400, Daniel Berlin wrote:
>>
>> On Apr 9, 2004, at 5:24 PM, Christopher Faylor wrote:
>>
>>> On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>>>> On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>>>> /dev/vg1/lvol6        23980436  22742628         0 100%
>>>>> /sourceware/www
>>>>
>>>> ARGH.  It had a little less than a gigabyte yesterday.
>>>>
>>>> I wonder if the fsck reset the free space.
>>>
>>> Nope.  It looks like something gobbled up 700MB in 21 hours.
>>
>> It wasn't bugzilla, i checked for that :)
>> Logs is 1.1 gig. Is that normal?
>
> Last weeks logs were ~100MB compressed, so I'd say no, it isn't normal.
>

it seemed a bit large.
Are you rotating based on date or size?

-rw-r--r--    1 apache   apache   352521027 Apr  9 21:47 
gcc-combined_log
-rw-r--r--    1 apache   apache   26661181 Apr  9 21:47 
ecos-combined_log
-rw-r--r--    1 apache   apache   456554238 Apr  9 21:47 
cygwin-combined_log
-rw-r--r--    1 apache   apache   324403715 Apr  9 21:47 
sourceware-combined_log


> cgf

Re: www out of space

[Christopher Faylor]

On Fri, Apr 09, 2004 at 05:24:53PM -0400, Christopher Faylor wrote:
>On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>>On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>>/dev/vg1/lvol6        23980436  22742628         0 100% /sourceware/www
>>
>>ARGH.  It had a little less than a gigabyte yesterday.
>>
>>I wonder if the fsck reset the free space.
>
>Nope.  It looks like something gobbled up 700MB in 21 hours.

I added another 1.7G to /sourceware/www.

(Sorry, Daniel, I killed two of your bash processes accessing that area
since you didn't seem to be doing anything there and I needed to umount
and fsck it.)

I don't know if fche is still planning on doing something with /dev/sdc2
but, if not, we should probably move at least the logs directory to that
partition to reduce the likelihood of this happening again.

I also blocked the !@(*&( idiots who caused this to happen.  I thought
I'd previously blocked the robots responsible for this kind of thing
at the httpd level but obviously I hadn't succeeded.

Jason, are you responsible for the HTTP_USER_AGENT RewriteEngine stuff
in httpd.conf?  Could you see what I'm doing wrong with the "MS Search
4.0 Robot" detection?

cgf

Re: www out of space

[Daniel Berlin]

Christopher Faylor wrote:

>On Fri, Apr 09, 2004 at 05:24:53PM -0400, Christopher Faylor wrote:
>  
>
>>On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>>    
>>
>>>On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>>      
>>>
>>>>/dev/vg1/lvol6        23980436  22742628         0 100% /sourceware/www
>>>>        
>>>>
>>>ARGH.  It had a little less than a gigabyte yesterday.
>>>
>>>I wonder if the fsck reset the free space.
>>>      
>>>
>>Nope.  It looks like something gobbled up 700MB in 21 hours.
>>    
>>
>
>I added another 1.7G to /sourceware/www.
>
>(Sorry, Daniel, I killed two of your bash processes accessing that area
>since you didn't seem to be doing anything there and I needed to umount
>and fsck it.)
>
>  
>

Oh, sorry.
I thought i had logged out before i walked away.
I wasn't doing anything.
Sorry about that.

Re: www out of space

[Angela Marie Thomas]

> Nope.  It looks like something gobbled up 700MB in 21 hours.

I don't see anything in the backup logs to show an obvious culprit.
I ran an rsync -n to see what changed between the backups and nowish
and also don't see anything obvious.

--Angela

Re: www out of space

[Christopher Faylor]

On Fri, Apr 09, 2004 at 05:42:52PM -0400, Christopher Faylor wrote:
>On Fri, Apr 09, 2004 at 05:35:07PM -0400, Daniel Berlin wrote:
>>
>>On Apr 9, 2004, at 5:24 PM, Christopher Faylor wrote:
>>
>>>On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>>>>On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>>>>/dev/vg1/lvol6        23980436  22742628         0 100% 
>>>>>/sourceware/www
>>>>
>>>>ARGH.  It had a little less than a gigabyte yesterday.
>>>>
>>>>I wonder if the fsck reset the free space.
>>>
>>>Nope.  It looks like something gobbled up 700MB in 21 hours.
>>
>>It wasn't bugzilla, i checked for that :)
>>Logs is 1.1 gig. Is that normal?
>
>Last weeks logs were ~100MB compressed, so I'd say no, it isn't normal.

We were hit hard by at least to spammers scanning for email.

I'm blocking them now.

cgf

Re: www out of space

[Christopher Faylor]

On Fri, Apr 09, 2004 at 05:35:07PM -0400, Daniel Berlin wrote:
>
>On Apr 9, 2004, at 5:24 PM, Christopher Faylor wrote:
>
>>On Fri, Apr 09, 2004 at 05:17:20PM -0400, Christopher Faylor wrote:
>>>On Fri, Apr 09, 2004 at 05:10:47PM -0400, Daniel Berlin wrote:
>>>>/dev/vg1/lvol6        23980436  22742628         0 100% 
>>>>/sourceware/www
>>>
>>>ARGH.  It had a little less than a gigabyte yesterday.
>>>
>>>I wonder if the fsck reset the free space.
>>
>>Nope.  It looks like something gobbled up 700MB in 21 hours.
>
>It wasn't bugzilla, i checked for that :)
>Logs is 1.1 gig. Is that normal?

Last weeks logs were ~100MB compressed, so I'd say no, it isn't normal.

cgf

Re: www out of space

[Frank Ch. Eigler]

Hi -

> [...]
> I don't know if fche is still planning on doing something with /dev/sdc2
> but, if not, we should probably move at least the logs directory to that
> partition to reduce the likelihood of this happening again. [...]

I was not going to interfere with /dev/sdc2 aka /pool until its current
users (especially mysql) clear it.  However, at the LVM level, there is 
already about 18GB more unallocated room on vg1 which could be added to
the www filesystem, which can apparently be resized online.

- FChE

Re: www out of space

[Christopher Faylor]

On Sat, Apr 10, 2004 at 07:15:36AM -0400, Frank Ch. Eigler wrote:
>>I don't know if fche is still planning on doing something with
>>/dev/sdc2 but, if not, we should probably move at least the logs
>>directory to that partition to reduce the likelihood of this happening
>>again.  [...]
>
>I was not going to interfere with /dev/sdc2 aka /pool until its current
>users (especially mysql) clear it.  However, at the LVM level, there is
>already about 18GB more unallocated room on vg1 which could be added to
>the www filesystem, which can apparently be resized online.

Yes, I noticed this yesterday after I finished resizing the www
partition.  I wish I had noticed it before the resize.  For some reason,
I thought there was a lot less extra space left for this type of thing.

cgf