/www is full - wtf?

/www is full - wtf?

[Jason Molenda]

/sourceware/www is 100% full, I've removed all the httpd error logs
from this year until I can figure out what's going on.  Is anything
doing something that could cause this?  It could result in serious
problems with the mail archives - very bad mess.  Let me know right
away.  I'll send a follow-up as soon as I figure out what's going on.

J

Re: /www is full - wtf?

[Jason Molenda]

I hate to leave this unfinished, but unfortunately today was a very
long day -  I have to get to sleep, I'm really tired.

I removed some more stuff (but I won't tell what ;-) and there are
40MB free on /sourceware/www.  That wouldn't get us through the
night normally, but I've taken the unusual step of turning off all
httpd logging - no errors, no accesses, no nothing.  The log files
would have most certainly filled the disk again before morning.

I don't know if there's any single thing we can blame for the disk
filling, it may have been an inevitable outcome.  Appended are the
nightly df outputs for the last couple weeks - remember that log
files are compressed on Saturday morning PST, so disk space generally
gets really tight through Friday night, then frees up a lot.  We're
about 24 hours from the next log file roll-over/compress right now.  The
other up-and-down pertubations might be due to the cygwin tar file
browser cache thing, but that's just a guess.

Chris - 190MB is for aaaspam's web archiving.  I assume this can
be removed (and maybe even web archiving disabled), but I didn't want
to do this without speaking to you first.

I'm running some big du's that will show a better outline of where
the disk space is going, but they won't finish for a long time.
I'll summarize the results when I wake up.  I don't think we're
going to find anything easy to remove; we can buy a bit more time
by deleting the remaining web logs from this year (cgf already
removed the ancient ones after we talked about the disk space issue
a while ago)

Filesystem                         Used     Available Use% Mounted on
2002-12-22:/dev/hda3               14173693   1103328  93% /sourceware/www
2002-12-23:/dev/hda3               14325875    951146  94% /sourceware/www
2002-12-24:/dev/hda3               14411788    865233  94% /sourceware/www
2002-12-25:/dev/hda3               14490697    786324  95% /sourceware/www
2002-12-26:/dev/hda3               14594025    682996  96% /sourceware/www
2002-12-27:/dev/hda3               14665972    611049  96% /sourceware/www
2002-12-28:/dev/hda3               14764141    512880  97% /sourceware/www
2002-12-29:/dev/hda3               14633478    643543  96% /sourceware/www
2002-12-30:/dev/hda3               14366878    910143  94% /sourceware/www
2002-12-31:/dev/hda3               14516228    760793  95% /sourceware/www
2003-01-01:/dev/hda3               14856199    420822  97% /sourceware/www
2003-01-02:/dev/hda3               14600696    676325  96% /sourceware/www
2003-01-03:/dev/hda3               14718334    558687  96% /sourceware/www
2003-01-04:/dev/hda3               14869114    407907  97% /sourceware/www
2003-01-05:/dev/hda3               14621795    655226  96% /sourceware/www
2003-01-06:/dev/hda3               14366183    910838  94% /sourceware/www
2003-01-07:/dev/hda3               14603694    673327  96% /sourceware/www
2003-01-08:/dev/hda3               15072241    204780  99% /sourceware/www
2003-01-09:/dev/hda3               15037289    239732  98% /sourceware/www
2003-01-10:/dev/hda3               15277026         0 100% /sourceware/www


J

On Fri, Jan 10, 2003 at 12:19:05AM -0800, Jason Molenda wrote:
> /sourceware/www is 100% full, I've removed all the httpd error logs
> from this year until I can figure out what's going on.  Is anything
> doing something that could cause this?  It could result in serious
> problems with the mail archives - very bad mess.  Let me know right
> away.  I'll send a follow-up as soon as I figure out what's going on.
> 
> J

Re: /www is full - wtf?

[Christopher Faylor]

On Fri, Jan 10, 2003 at 01:05:48AM -0800, Jason Molenda wrote:
>I hate to leave this unfinished, but unfortunately today was a very
>long day -  I have to get to sleep, I'm really tired.
>
>I removed some more stuff (but I won't tell what ;-) and there are
>40MB free on /sourceware/www.  That wouldn't get us through the
>night normally, but I've taken the unusual step of turning off all
>httpd logging - no errors, no accesses, no nothing.  The log files
>would have most certainly filled the disk again before morning.
>
>I don't know if there's any single thing we can blame for the disk
>filling, it may have been an inevitable outcome.  Appended are the
>nightly df outputs for the last couple weeks - remember that log
>files are compressed on Saturday morning PST, so disk space generally
>gets really tight through Friday night, then frees up a lot.  We're
>about 24 hours from the next log file roll-over/compress right now.  The
>other up-and-down pertubations might be due to the cygwin tar file
>browser cache thing, but that's just a guess.
>
>Chris - 190MB is for aaaspam's web archiving.  I assume this can
>be removed (and maybe even web archiving disabled), but I didn't want
>to do this without speaking to you first.

I'd rather not remove the web archiving entirely but we can get rid
of all but the last couple of months.  I like to inspect the archives
for trends from time to time.  And, very occasionally, there is a
confused non-English-speaker asking for help there.  :-(

Hmm.  We had this problem with the qmail partition a while ago, too.
Maybe I'd better bump up the sizes of all of the partitions on the new
machine.

cgf

Re: /www is full - wtf?

[Benjamin Kosnik]

On Fri, 10 Jan 2003 10:49:07 -0500
Christopher Faylor <cgf@redhat.com> wrote:

>..the new machine.

Hehe. You said "new machine."

-benjamin