GCC snapshot generation and machine load

GCC snapshot generation and machine load

[Zack Weinberg]

Over the past several days gcc.gnu.org/sources.redhat.com has been
under unusually heavy load, e.g. no fewer than 20 processes in D state
at any given time.  All CVS operations take orders of magnitude longer
than they should.

It's still not clear what's causing the problem - it may well be a
hardware fault, e.g. a failed disk in the RAID array - but for right
now I want to talk about not making things worse.  Monday morning,
generating the GCC snapshot drove the load up into the 50-60 range for
hours.  I would like to suggest two things:

1) Let's disable GCC snapshot generation entirely until the problem is
   resolved.

2) Let's think about moving the snapshots to a lower-load time of
   week.  Monday morning in the USA is probably the worst possible
   time to run them.  I don't know what Wednesday is like.  Off the
   top of my head it seems likely that running them both in succession
   on Sunday afternoon would probably be the best choice.

Thoughts?

zw

Re: GCC snapshot generation and machine load

[Gerald Pfeifer]

On Wed, 7 Apr 2004, Zack Weinberg wrote:
> It's still not clear what's causing the problem - it may well be a
> hardware fault, e.g. a failed disk in the RAID array - but for right
> now I want to talk about not making things worse.  Monday morning,
> generating the GCC snapshot drove the load up into the 50-60 range for
> hours.  I would like to suggest two things:
>
> 1) Let's disable GCC snapshot generation entirely until the problem is
>    resolved.

Unfortunately, we missed the start of today's snapshot, but machine load
is lower than on Monday (in the mid 20s), so I suggest to go ahead and not
abort that snapshot.

> 2) Let's think about moving the snapshots to a lower-load time of
>    week.  Monday morning in the USA is probably the worst possible
>    time to run them.  I don't know what Wednesday is like.  Off the
>    top of my head it seems likely that running them both in succession
>    on Sunday afternoon would probably be the best choice.

I've been thinking about moving snapshots after reading the threads on
overseers yesterday, especially the one on Monday (3.3 branch). Having
all snapshots on the same day might cause our FTP load to explode and
mirror sites not being able to get at the snapshots in time, though.


How about moving the snapshot(s) from release branches, which are more
expensive due to the way CVS operates, to Sunday, and keeping the HEAD
snapshot a few days apart, perhaps moving it from Wednesday to Thursday?

What would be the best time of the day to run a snapshot on Sunday and
on a weekday, respectively?

Gerald

Re: GCC snapshot generation and machine load

[Jason Molenda]

On Wed, Apr 07, 2004 at 07:57:49PM +0200, Gerald Pfeifer wrote:

> What would be the best time of the day to run a snapshot on Sunday and
> on a weekday, respectively?

If you have a few minutes, it's probably easiest to look at the load
#'s and pick a time yourself.  Grab /var/log/monitors/2004-03.tar.bz2
and breeze over the data -- if you aggregate the individual samples
a little bit, it is pretty clear when the system is heavily loaded
and when the system is idle.  Timestamps are UTC+0 (like most everything
on the system).

J

Re: GCC snapshot generation and machine load

[Gerald Pfeifer]

On Wed, 7 Apr 2004, Jason Molenda wrote:
>> What would be the best time of the day to run a snapshot on Sunday and
>> on a weekday, respectively?
> If you have a few minutes, it's probably easiest to look at the load
> #'s and pick a time yourself.  Grab /var/log/monitors/2004-03.tar.bz2
> and breeze over the data

I've been working on that now.  Wednesday, 22:30 seems to be a good time
for GCC 3.3 snapshots, and I'll also try to find a slot for 3.4 and 3.5
so these three have a distance of two days (as not to overload FTP).


BTW, who changed permissions of /sourceware/snapshot-tmp so that non-root
cannot create directories any longer?  This broke GCC snapshots for the
last two, three weeks, and I was too busy to debug that until now. :-(

I have now created
  drwxrwsr-x    2 gccadmin gcc          4096 May  2 11:19 gcc
in /sourceware/snapshot-tmp and will adjust the GCC snapshot script
accordingly.

Gerald
-- 
Gerald Pfeifer (Jerry)   gerald@pfeifer.com   http://www.pfeifer.com/gerald/

Re: GCC snapshot generation and machine load

[Christopher Faylor]

On Sun, May 02, 2004 at 01:32:44PM +0200, Gerald Pfeifer wrote:
>On Wed, 7 Apr 2004, Jason Molenda wrote:
>>>What would be the best time of the day to run a snapshot on Sunday and
>>>on a weekday, respectively?
>>If you have a few minutes, it's probably easiest to look at the load
>>#'s and pick a time yourself.  Grab /var/log/monitors/2004-03.tar.bz2
>>and breeze over the data
>
>I've been working on that now.  Wednesday, 22:30 seems to be a good
>time for GCC 3.3 snapshots, and I'll also try to find a slot for 3.4
>and 3.5 so these three have a distance of two days (as not to overload
>FTP).
>
>
>BTW, who changed permissions of /sourceware/snapshot-tmp so that
>non-root cannot create directories any longer?  This broke GCC
>snapshots for the last two, three weeks, and I was too busy to debug
>that until now.  :-(

Huh?  AFAIK, the ownership has been like that for months.

This isn't some analogue of tmp.  You are supposed to be using your
own directory in that area, as you have done.  You are also not supposed
to be storing anything permanent in here, i.e., please don't let this
directory become an analogue of the gcc ftp area that grows without
bounds.

Since you are working on the snapshots, I don't suppose you have fixed
the problem of ancient gcc snapshots not disappearing eventually?

cgf

Re: GCC snapshot generation and machine load

[Gerald Pfeifer]

On Sun, 2 May 2004, Christopher Faylor wrote:
>> BTW, who changed permissions of /sourceware/snapshot-tmp so that
>> non-root cannot create directories any longer?  This broke GCC
>> snapshots for the last two, three weeks, and I was too busy to debug
>> that until now.  :-(
> Huh?  AFAIK, the ownership has been like that for months.

I don't think so. It seems this was changed between 14 Apr 2004 and
18 Apr 2004, at least according to
  http://gcc.gnu.org/ml/gccadmin/2004-q2/msg00032.html
and
  http://gcc.gnu.org/ml/gccadmin/2004-q2/msg00041.html

> This isn't some analogue of tmp.  You are supposed to be using your own
> directory in that area, as you have done.  You are also not supposed to
> be storing anything permanent in here, i.e., please don't let this
> directory become an analogue of the gcc ftp area that grows without
> bounds.

Definitely not!  The GCC release/snapshot script nicely cleans up, unless
I'm manually running it in debug mode.

> Since you are working on the snapshots, I don't suppose you have fixed
> the problem of ancient gcc snapshots not disappearing eventually?

Not yet. :-(  In the last 12 months I've spent a couple of working days
on the gcc_release script, overall, and I really hope that's it for now.


If someone could help with a small Perl/sh/whatever script, I think I
have an idea for a simple algorithm on which GCC snapshots to remove.

Gerald