Re: network overload

Re: network overload

[Ulrich Drepper]

Jason Molenda <jason@molenda.com> writes:

> For releases, the best thing is to provide tarballs in both .gz
> and .bz2 format so those who are capable of using the .bz2 can grab
> it

What I do is put up both files but only give the complete URL for the
.bz2 file in the announcement.  Something like this

	foo-X.Y.Z.tar.bz2	(also .gz)

This urges people a bit in one direction.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: network overload

[Jonathan Larmour]

Jim Kingdon wrote:
> 
> Sourceware's T1 seems rather congested (at least I can't find other
> reasons for the slowness I'm seeing), and has been for the last hour
> or few.  Is this typical for this time of day?  I don't remember
> seeing it before.  I looked through some logs and didn't find evidence
> that we were being slashdotted (a bunch of hits all on the same page)
> or that kind of thing, but I'm not completely sure I know all the
> places to look.

Perhaps it would be good to build up some network traffic analysis tools
which you could use to show the current stats for connections to each server
port. Recommendations?

One snag would be that unless it sniffed ftp control packets it wouldn't
catch PASV FTP traffic. I'm not sure if tools out there can do that, or how
common passive FTP is anyway.

Jifl
-- 
Red Hat, 35 Cambridge Place, Cambridge, UK. CB2 1NS  Tel: +44 (1223) 728762
"Plan to be spontaneous tomorrow."  ||  These opinions are all my own fault

Re: network overload

[Jim Kingdon]

OK, here's the state of play:

* Regarding whether we are saturating the T1, I tried doing some
  calculations and mostly got confused.  To some extent we need better
  tools to answer this well but maybe someone can help me get at least
  rough answers with the data we have.

* Marc Rovner tells me there are some problems with the NIC and that
  he'd like to put in two NIC's in the sourceware box (one to the
  sourceware T1 and one to the Cygnus network, it sounded like,
  although I didn't follow all the details of how the network is set
  up).  I'm planning on giving him the go-ahead on this, unless
  someone wants to suggest otherwise.

* I reduced the number of FTP users from 60 to 30 and this helped
  quite noticeably.  It solved the immediate crisis.

* Before posting I ruled out generic internet problems to my
  satisfaction, e.g. I did a number of traceroutes and other tests
  (e.g. log into rtl.cygnus.com, and from there into sourceware - and
  see good performance).

* Regarding bzip2, well, if you want a personal preference I'm more in
  the anti-bz2 camp.  However, with a sourceware admin hat on I have
  to point out both sides: bz2 is about 10-20% smaller than gzip, and
  that means that if we use gzip, then 10-20% of the people will need
  to use mirrors instead.  The pluses of gzip, of course, are wider
  deployment and less memory needed to unpack.

Re: network overload

[Jason Molenda]

On Wed, Mar 01, 2000 at 04:31:20PM -0700, Jeffrey A Law wrote:

> As
> I believe was mentioned FTP access to the GCC snapshots is the single
> biggest user of bandwidth on the T1.

I don't know if it's snapshots or releases off-hand, but as long
as we're talking about this, how about putting GCC snapshots up in
.bz2 format exclusively?  I remember when I looked at this a year
ago, it would be a not-insignificant savings if we forced people
to get their GCC snapshots in .bz2 fmt.

FWIW GDB and binutils snapshots have been available only in .bz2
format for about a year.  Beyond a little initial whining, people
have adjusted and dealt with it.  The next version of GNU tar will
include a single-letter option ("-I" IIRC) to unbzip2 archives so
you won't need to pipe things, but I have no idea when it will
finally be released.

J

network overload

[Jim Kingdon]

Sourceware's T1 seems rather congested (at least I can't find other
reasons for the slowness I'm seeing), and has been for the last hour
or few.  Is this typical for this time of day?  I don't remember
seeing it before.  I looked through some logs and didn't find evidence
that we were being slashdotted (a bunch of hits all on the same page)
or that kind of thing, but I'm not completely sure I know all the
places to look.

If I look through http://sourceware.cygnus.com/sourceware/bandwidth/ I
do see that those numbers increased from roughly 50-60 Gbyte/week in
December, to about 80Gbyte/week now.  I don't know if we were close
enough to the edge for this trend to explain the 200 ms ping times
(and 5-10% packet loss) I'm seeing.

Anyone able to help explain what is going on?  If we are really maxxed
out and need more bandwidth (somehow), that might be, but I want to
make sure I understand the situation before I assume that.

[root@sourceware log]# traceroute www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 204.71.202.160
traceroute to www.yahoo.com (204.71.202.160), 30 hops max, 38 byte packets
 1  gw-cygnus3-sv (205.180.83.66)  3.223 ms  1.865 ms  1.730 ms
 2  s11-0-0-22.paloalto-cr20.bbnplanet.net (4.1.120.85)  251.907 ms
227.308 ms  227.308 ms
 . . .
[root@sourceware log]# tail -f /sourceware/ftp/log/xferlog
 . . .
[root@sourceware log]# tail -f /www/logs/sourceware-access_log
 . . .
[root@sourceware log]# ps auxww | grep anoncvs | wc -l
      6
[root@sourceware log]# 

Re: network overload

[Geoffrey Noer]

On Wed, Mar 01, 2000, Jeffrey A Law wrote:
>
> Round trip times have certainly climbed over the last couple months; one
> thing that is easy to try is to throttle the number of FTP connections.  As
> I believe was mentioned FTP access to the GCC snapshots is the single
> biggest user of bandwidth on the T1.

Of course, our upcoming new Cygwin Net release may change things.  :-)

-- 
Geoffrey Noer				Email: noer@cygnus.com
Field Applications Engineer		http://www.cygnus.com/
Cygnus Solutions, a Red Hat company	http://www.redhat.com/

Re: network overload

[Jason Molenda]

On Wed, Mar 01, 2000 at 03:59:05PM -0500, Jim Kingdon wrote:
> Sourceware's T1 seems rather congested (at least I can't find other
> reasons for the slowness I'm seeing), and has been for the last hour
> or few.  Is this typical for this time of day?  

The webalizer graphs show the most accesses are morning time PST,
although the entire workday US time is heavy.  (morning time PST
includes the evening in the Europe as well as most of the US starting
out the day).  The lowest time, IIRC, is mid-evening PST when the
people are doing things at home on the west coast but Asia hasn't
really gotten started surfing.

The biggest network user is ftp traffic.  In fact, half of our
entire http+ftp bandwidth each week is GCC downloads.  If you want
to reduce the net load, the easiest thing to do is reduce the number
of simultaneous ftp connections.  It's set in
/sourceware/ftp/etc/ftpaccess, which is under RCS control.

If you look at the weekly bandwidth usage #s and look at the RCS log
for this file, you'll find a pretty close correlation between when 
the # of ftp connections and the total bandwidth usage.

I probably bumped it up once sourceware was on its own T1 (figuring
that it wouldn't cause problems for cygnus users any longer so what
the hell), but it may be time to bump it back down again.


Jason

Re: network overload

[Andrew Cagney]

Excerpts from mail: 1-Mar-100 Re: network overload Jason
Molenda@molenda.co (838*)


> FWIW GDB and binutils snapshots have been available only in .bz2
> format for about a year.  Beyond a little initial whining, people
> have adjusted and dealt with it.  The next version of GNU tar will
> include a single-letter option ("-I" IIRC) to unbzip2 archives so
> you won't need to pipe things, but I have no idea when it will
> finally be released.

I know.  I'll go and fix the gdb snaps.
	Andrew

Re: network overload

[Jason Molenda]

On Thu, Mar 02, 2000 at 01:18:43AM -0500, Jim Kingdon wrote:

> * Regarding bzip2, well, if you want a personal preference I'm more in
>   the anti-bz2 camp.  However, with a sourceware admin hat on I have
>   to point out both sides: bz2 is about 10-20% smaller than gzip, and
>   that means that if we use gzip, then 10-20% of the people will need
>   to use mirrors instead.  The pluses of gzip, of course, are wider
>   deployment and less memory needed to unpack.


I think the fact that we're talking about developer snapshots is
important.  Developers can be expected to have a slightly beefier
environment than the general public, and it is not too onerous for
(say) a GCC contributor to download bzip2 and compile it.  Or
download one of the prebuilt binaries for just about any platform
you're likely to see.

For releases, the best thing is to provide tarballs in both .gz
and .bz2 format so those who are capable of using the .bz2 can grab
it (and people in countries that pay their telecom costs by the
minute will inevitably do so).

Well, that's my opinion, for whatever it's worth. :)

Jason

Re: network overload

[Jeffrey A Law]

  In message < 200003012059.PAA07906@devserv.devel.redhat.com >you write:
  > Sourceware's T1 seems rather congested (at least I can't find other
  > reasons for the slowness I'm seeing), and has been for the last hour
  > or few.  Is this typical for this time of day?  I don't remember
  > seeing it before.  I looked through some logs and didn't find evidence
  > that we were being slashdotted (a bunch of hits all on the same page)
  > or that kind of thing, but I'm not completely sure I know all the
  > places to look.
  > 
  > If I look through http://sourceware.cygnus.com/sourceware/bandwidth/ I
  > do see that those numbers increased from roughly 50-60 Gbyte/week in
  > December, to about 80Gbyte/week now.  I don't know if we were close
  > enough to the edge for this trend to explain the 200 ms ping times
  > (and 5-10% packet loss) I'm seeing.
  > 
  > Anyone able to help explain what is going on?  If we are really maxxed
  > out and need more bandwidth (somehow), that might be, but I want to
  > make sure I understand the situation before I assume that.
Round trip times have certainly climbed over the last couple months; one
thing that is easy to try is to throttle the number of FTP connections.  As
I believe was mentioned FTP access to the GCC snapshots is the single
biggest user of bandwidth on the T1.

jeff

Re: network overload

[Andrew Cagney]

Jim,

I don't know where your comming from (physcially :-) but from here,
yesterday, the US was dropping random packets and showing general
congestion.

	Andrew