Re: /sourceware/www is full

Re: /sourceware/www is full

[Christopher Faylor]

On Thu, Mar 18, 2004 at 05:01:01PM -0800, Jason Molenda wrote:
>On Thu, Mar 18, 2004 at 07:59:17PM -0500, Ian Lance Taylor wrote:
>> /sourceware/www is full.  This is breaking bugzilla, and probably
>> causing numerous other problems.
>> 
>> I'm going to try to find out what the problem is.
>
>
>Most likely log files.  They grow throughout the week,
>scrolling/compressing on Saturday.
>
>Remove some old ones.

I'm moving some now.

cgf

Re: /sourceware/www is full (/www/gcc/htdocs/onlinedocs?)

[Christopher Faylor]

On Thu, Mar 18, 2004 at 08:06:32PM -0500, Christopher Faylor wrote:
>On Thu, Mar 18, 2004 at 05:01:01PM -0800, Jason Molenda wrote:
>>On Thu, Mar 18, 2004 at 07:59:17PM -0500, Ian Lance Taylor wrote:
>>> /sourceware/www is full.  This is breaking bugzilla, and probably
>>> causing numerous other problems.
>>> 
>>> I'm going to try to find out what the problem is.
>>
>>
>>Most likely log files.  They grow throughout the week,
>>scrolling/compressing on Saturday.
>>
>>Remove some old ones.
>
>I'm moving some now.

One thing I notice taking up space is in /www/gcc/htdocs/onlinedocs.
There seems to be compressed and uncompressed versions of files in this
directory.  Is it necessary to have both?

cgf

Re: /sourceware/www is full (/www/gcc/htdocs/onlinedocs?)

[Jason Molenda]

On Thu, Mar 18, 2004 at 08:12:13PM -0500, Christopher Faylor wrote:

> One thing I notice taking up space is in /www/gcc/htdocs/onlinedocs.
> There seems to be compressed and uncompressed versions of files in this
> directory.  Is it necessary to have both?


With the old mod_gzip, the .gz version would be sent instead of
the .html version resulting in a huge bandwidth savings.  When the
system was on a T1, it was definitely worth it.  Now I gather
bandwidth isn't such a problem for sourceware so it's less compelling.
It would still be appreciated by those on the other ends of slow
net connections, of coruse.


(but then again, I think we dropped this module when we went to RH
RPM based apache server.  In which case they are probably completely
unused.)

J

Re: /sourceware/www is full (/www/gcc/htdocs/onlinedocs?)

[Joseph S. Myers]

On Thu, 18 Mar 2004, Christopher Faylor wrote:

> One thing I notice taking up space is in /www/gcc/htdocs/onlinedocs.
> There seems to be compressed and uncompressed versions of files in this
> directory.  Is it necessary to have both?

The compressed HTML versions were originally for transparently sending to
browsers supporting them.  (It is clearly perceptible that old mailing
list index pages no longer get sent transparently compressed as they used
to.)  At one point the PostScript versions were linked to uncompressed -
with the intent of getting the compressed versions sent transparently -
but this was changed to link to the compressed versions following a
discussion of the difficulties of getting HTTP headers that work well for
all browsers <http://gcc.gnu.org/ml/gcc/2001-08/msg01435.html>.  The
tarballs have never been linked to uncompressed.  The same mechanism is
used to generate the compressed versions of all these different files
without discriminating about whether particular files need them.

-- 
Joseph S. Myers
jsm@polyomino.org.uk

Re: /sourceware/www is full

[Ian Lance Taylor]

Christopher Faylor <cgf@redhat.com> writes:

> >> /sourceware/www is full.  This is breaking bugzilla, and probably
> >> causing numerous other problems.
> >> 
> >> I'm going to try to find out what the problem is.
> >
> >
> >Most likely log files.  They grow throughout the week,
> >scrolling/compressing on Saturday.
> >
> >Remove some old ones.
> 
> I'm moving some now.

That fixed the immediate problem.  Thanks.

We seem to be skating pretty close to the edge, though.  The disk
partition is 23.9G.  Before Chris cleaned out the log files, the log
directory held 1.7G.  So even if we discard all the log files, we only
have 7% of the disk free.  And the mailing list archives keep getting
bigger, and bigger, and bigger....

While I too wonder about all the files in gcc/htdocs, the whole
directory only takes up 797M, or just 3% of the partition.

The biggest mailing list archive is gcc-patches, at 1.9G.  And the
biggest month in gcc-patches is last January, with February not too
far behind.

Some mailing list sizes:

469048  ./sourceware/ml/binutils
481740  ./sourceware/ml/xsl-list
516788  ./sourceware/ml/gdb-patches
645836  ./gcc/ml/gcc-cvs
822428  ./sourceware/ml/subversion-dev
974144  ./gcc/ml/gcc-prs
1238412 ./gcc/ml/gcc-testresults
1315064 ./sourceware/ml/cygwin
1323496 ./gcc/ml/gcc
1573776 ./gcc/ml/gcc-bugs
1933996 ./gcc/ml/gcc-patches

I think we may want to consider the somewhat radical step of moving
some of the old mailing list archival stuff off to a different server.
I assume that people don't look up old mailing list data very often.
It might be too much of a pain to get searching to work right, though.

Ian

Re: /sourceware/www is full

[Joseph S. Myers]

On Fri, 18 Mar 2004, Ian Lance Taylor wrote:

> I think we may want to consider the somewhat radical step of moving
> some of the old mailing list archival stuff off to a different server.
> I assume that people don't look up old mailing list data very often.
> It might be too much of a pain to get searching to work right, though.

I just found it a pain some hours ago that list archives from 1998 had
dead URLs to the egcs list archives originally on www.cygnus.com and it
was necessary to look at messages with similar numbers in the current
version of the archives to work out what message a dead link was referring
to.  (archive.org didn't have the old version of the archives to provide a
shortcut.)  It's routinely useful to recall some message or discussion
from one of the egcs/GCC lists at any time since 1997 and then search and
bring up that particular message, and anything breaking URLs or hindering
searching is a pain in that regard.

(The logs should give real statistics about the number of accesses to each
month's archives.)

-- 
Joseph S. Myers
jsm@polyomino.org.uk

Re: /sourceware/www is full

[Christopher Faylor]

On Thu, Mar 18, 2004 at 08:47:48PM -0500, Ian Lance Taylor wrote:
>Christopher Faylor <cgf@redhat.com> writes:
>
>> >> /sourceware/www is full.  This is breaking bugzilla, and probably
>> >> causing numerous other problems.
>> >> 
>> >> I'm going to try to find out what the problem is.
>> >
>> >
>> >Most likely log files.  They grow throughout the week,
>> >scrolling/compressing on Saturday.
>> >
>> >Remove some old ones.
>> 
>> I'm moving some now.
>
>That fixed the immediate problem.  Thanks.
>
>We seem to be skating pretty close to the edge, though.  The disk
>partition is 23.9G.  Before Chris cleaned out the log files, the log
>directory held 1.7G.  So even if we discard all the log files, we only
>have 7% of the disk free.  And the mailing list archives keep getting
>bigger, and bigger, and bigger....

I can grow the partition at some point.  I'd been planning on doing that
but missed that the partition was so close to being full.  I guess we
should plan on some downtime this weekend.

cgf

Re: /sourceware/www is full

[Ian Lance Taylor]

"Joseph S. Myers" <jsm@polyomino.org.uk> writes:

> On Fri, 18 Mar 2004, Ian Lance Taylor wrote:
> 
> > I think we may want to consider the somewhat radical step of moving
> > some of the old mailing list archival stuff off to a different server.
> > I assume that people don't look up old mailing list data very often.
> > It might be too much of a pain to get searching to work right, though.
> 
> I just found it a pain some hours ago that list archives from 1998 had
> dead URLs to the egcs list archives originally on www.cygnus.com and it
> was necessary to look at messages with similar numbers in the current
> version of the archives to work out what message a dead link was referring
> to.  (archive.org didn't have the old version of the archives to provide a
> shortcut.)  It's routinely useful to recall some message or discussion
> from one of the egcs/GCC lists at any time since 1997 and then search and
> bring up that particular message, and anything breaking URLs or hindering
> searching is a pain in that regard.

Searching is a problem which would have to be addressed, but URL
redirection is trivial.  We can't do useful URL redirection for
cygnus.com references, because, sadly and foolishly, Red Hat let the
domain name go.  But we can easily redirect URLs among domain names
which we control.

> (The logs should give real statistics about the number of accesses to each
> month's archives.)

As a quick check, the current gcc-combined_log shows 1,298,388
references to mailing lists, of which 204,710 are to messages in years
beginning with 19.  That is 15%.  Actually more than I would have
guessed.  If we don't count 1999, we are left with 68,464 references,
or 5%.  Of course, gcc messages before 1999 take up a total 353M, or
all of 1.5% of the space on the partition.  Hmmm.  Maybe not such a
worthwhile idea.

Ian

Re: /sourceware/www is full

[Jason Molenda]

On Thu, Mar 18, 2004 at 05:01:50PM -0800, Jason Molenda wrote:

> Incidentally, there will probably be some mailing list archives
> hosed by this.  I'll be home from karate class around 10:30pm or
> so and will deal with the fallout then.


Yeah, suck, lists with potential problems:

gcc-patches
java-patches
gcc
gccadmin
gcc-help
gcc-cvs
gcc-testresults
libstdc++

gnats-devel
libc-alpha
docbook-apps
cygwin-apps
libc-hacker
gdb
cygwin
overseers
cygwin-xfree
binutils
gdb-patches
glibc-cvs


Sigh.  The raw text archives (from which it's easy to regen the web
archives) are also corrupt in the one or two cases I looked at, so 
it's going to take some careful by-hand reconstruction of the raw
text archives and then regenerate the web archives.

I'll do the gdb-patches gcc-patches java-patches lists tonight
because those are the most likely to have references to particular
messages, but I can't stay up late enough to finish all of them.
I'll do a few more tomorrow night but I'll only have an hour or
so there again -- and Saturday might be wonky as well.  I might
not finally finish this until late Saturday night.

J

Re: /sourceware/www is full

[Jonathan Larmour]

Christopher Faylor wrote:
> 
> I can grow the partition at some point.  I'd been planning on doing that
> but missed that the partition was so close to being full.  I guess we
> should plan on some downtime this weekend.

OOI I know that when I start running low on disk space on my Red Hat Linux 
system, there's something that whinges at me (well, root) via mail every 
hour. Perhaps it's disabled/not installed? [.....] Ah yes it's an RPM 
called diskcheck and it isn't installed on sourceware.

Jifl
-- 
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
 >>>>> Visit us in booth 2527 at the Embedded Systems Conference 2004 <<<<<
March 30 - April 1, San Francisco http://www.esconline.com/electronicaUSA/
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine