src crippled, cvs inaccessible.

src crippled, cvs inaccessible.

[Dave Korn]

    Hi overseers!

  I've been getting almost nothing but 

cvs [diff aborted]: reading from server: Software caused connection abort

and

cvs [diff aborted]: end of file from server (consult above messages if any)

messages trying to access the cvs repository on src for about four hours
now.  I know that this is the sort of thing you expect to see when it's
heavily loaded, but it seems to have been a lot worse and going on for
longer than usual.

  Could someone with login access run a quick 'top' and make sure there
isn't some stuck process hogging all the cpu (or similar)?  TIA!

    cheers, 
      DaveK
-- 
Can't think of a witty .sigline today....

Re: src crippled, cvs inaccessible.

[Ian Lance Taylor]

"Dave Korn" <dave.korn@artimi.com> writes:

>   I've been getting almost nothing but 
> 
> cvs [diff aborted]: reading from server: Software caused connection abort
> 
> and
> 
> cvs [diff aborted]: end of file from server (consult above messages if any)
> 
> messages trying to access the cvs repository on src for about four hours
> now.  I know that this is the sort of thing you expect to see when it's
> heavily loaded, but it seems to have been a lot worse and going on for
> longer than usual.
> 
>   Could someone with login access run a quick 'top' and make sure there
> isn't some stuck process hogging all the cpu (or similar)?  TIA!

The load average is high today, but I'm not seeing anything
extraordinary, at least not yet.  At the moment it's hovering around
20.

Ian

RE: src crippled, cvs inaccessible.

[Dave Korn]

----Original Message----
>From: Ian Lance Taylor
>Sent: 05 April 2005 16:15


> The load average is high today, but I'm not seeing anything
> extraordinary, at least not yet.  At the moment it's hovering around
> 20.


  Thanks.  As it happens, I just finally managed to get in for a quick diff,
after a hundred-and-some retries....  ouch!


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: src crippled, cvs inaccessible.

[Jeffrey A Law]

On Tue, 2005-04-05 at 15:59 +0100, Dave Korn wrote:
>     Hi overseers!
> 
>   I've been getting almost nothing but 
> 
> cvs [diff aborted]: reading from server: Software caused connection abort
> 
> and
> 
> cvs [diff aborted]: end of file from server (consult above messages if any)
> 
> messages trying to access the cvs repository on src for about four hours
> now.  I know that this is the sort of thing you expect to see when it's
> heavily loaded, but it seems to have been a lot worse and going on for
> longer than usual.
> 
>   Could someone with login access run a quick 'top' and make sure there
> isn't some stuck process hogging all the cpu (or similar)?  TIA!
The machine seems to be running "OK" -- we're hitting the disks pretty
hard, which is causing us to spend a fair amount of time in disk wait.

The net result is we have a load average of ~20 due to all the processes
sitting in disk wait.  If you're using anoncvs, the connection refused
messages are probably due to the high load average.

jeff

RE: src crippled, cvs inaccessible.

[Dave Korn]

----Original Message----
>From: Jeffrey A Law
>Sent: 05 April 2005 16:17

>> now.  I know that this is the sort of thing you expect to see when it's
>> heavily loaded, but it seems to have been a lot worse and going on for
>> longer than usual. 
>> 
>>   Could someone with login access run a quick 'top' and make sure there
>> isn't some stuck process hogging all the cpu (or similar)?  TIA!
> The machine seems to be running "OK" -- we're hitting the disks pretty
> hard, which is causing us to spend a fair amount of time in disk wait.
> 
> The net result is we have a load average of ~20 due to all the processes
> sitting in disk wait.  If you're using anoncvs, the connection refused
> messages are probably due to the high load average.
> 
> jeff


  20LA, yeesh!  Heh, well I guess that explains it.... thanks for checking
Jeff!


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: src crippled, cvs inaccessible.

[Zack Weinberg]

Jeffrey A Law <law@redhat.com> writes:

> The net result is we have a load average of ~20 due to all the
> processes sitting in disk wait.  If you're using anoncvs, the
> connection refused messages are probably due to the high load
> average.

Lots and lots of disk wait was a symptom of the RAID having degraded
due to a disk failure, the last time...

zw

RE: src crippled, cvs inaccessible.

[Dave Korn]

----Original Message----
>From: Zack Weinberg
>Sent: 05 April 2005 17:55

> Jeffrey A Law <law@redhat.com> writes:
> 
>> The net result is we have a load average of ~20 due to all the
>> processes sitting in disk wait.  If you're using anoncvs, the
>> connection refused messages are probably due to the high load
>> average.
> 
> Lots and lots of disk wait was a symptom of the RAID having degraded
> due to a disk failure, the last time...
> 
> zw


  Oops.  Maybe someone had better take a look at the status in whatever
drive-monitoring utility comes with the RAID array?  It's still going on and
it has been continuing for over six hours now and that is *unusual* and
therefore scary.....


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....

Re: src crippled, cvs inaccessible.

[Ian Lance Taylor]

"Dave Korn" <dave.korn@artimi.com> writes:

> >> The net result is we have a load average of ~20 due to all the
> >> processes sitting in disk wait.  If you're using anoncvs, the
> >> connection refused messages are probably due to the high load
> >> average.
> > 
> > Lots and lots of disk wait was a symptom of the RAID having degraded
> > due to a disk failure, the last time...
> > 
> > zw
> 
> 
>   Oops.  Maybe someone had better take a look at the status in whatever
> drive-monitoring utility comes with the RAID array?  It's still going on and
> it has been continuing for over six hours now and that is *unusual* and
> therefore scary.....

We did check the RAID.

The current load is not really unusual, unfortunately.  There is new
hardware in the queue somewhere at Red Hat, but I don't have any more
information on it.

Ian

Re: src crippled, cvs inaccessible.

[Jonathan Larmour]

Dave Korn wrote:
>     Hi overseers!
> 
>   I've been getting almost nothing but 
> 
> cvs [diff aborted]: reading from server: Software caused connection abort
> 
> and
> 
> cvs [diff aborted]: end of file from server (consult above messages if any)
> 
> messages trying to access the cvs repository on src for about four hours
> now.  I know that this is the sort of thing you expect to see when it's
> heavily loaded, but it seems to have been a lot worse and going on for
> longer than usual.
> 
>   Could someone with login access run a quick 'top' and make sure there
> isn't some stuck process hogging all the cpu (or similar)?  TIA!

The processes don't look stuck. There's just a lot of them! Multiple 
rsyncs, many CVS clients, a couple of FTP downloads, on top of the usual 
continuous mail and web server load. It looks very much disk bound right 
now. There's no single culprit, but CVS processes seem to be the majority.

There are about 19 CVS server processes running right now. I guess the 
West coast has woken up and people start the day with a cvs update.

Jifl
-- 
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

RE: src crippled, cvs inaccessible.

[Dave Korn]

----Original Message----
>From: Jonathan Larmour
>Sent: 05 April 2005 16:24

> There are about 19 CVS server processes running right now. I guess the
> West coast has woken up and people start the day with a cvs update.
> 
> Jifl


  <g>  Yeh, I like to myself... I first noticed it about four-and-a-bit
hours ago, at which time I thought not a huge amount of the US folks would
be up-and-at-work on it yet, so I wondered if some overnight job had got
stuck, but as you say, it's just very very busy.



    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....