[Fwd: ftp probs]

[Fwd: ftp probs]

[Stan Shebs]

Are there any plans to solve this?  When I go to test that the packages
really are downloadable, I can never get on either... -s

-------- Original Message --------
Subject: ftp probs
Date: Tue, 20 Jun 2000 22:13:34 +0000
From: oli donald <mcai8od2@ist4.co.umist.ac.uk>
To: shebs@shebs.cnchost.com

Hi

I'm desperate to get hold of the 7.3.3 release of Mac X-Conq, but I'm 
having major problems. I've logged on to Sourceare for the last three 
nights, but I get a maximum download speed of about 500 bytes/sec, 
with the connection dropping for minutes at a time. That's way below 
normal for me (usually 4k+), and being in the UK I have to pay by the 
minute! :) Plus, the FTP sites too busy most of the time. Is there 
anywhere else I can get the latest version? I tried the mirrors but 
couldn't find anything recent :(

Anyways cheers for your time and a great bit of software :)

Oli Donald
Widosm

Re: [Fwd: ftp probs]

[Chris Faylor]

On Tue, Jun 20, 2000 at 10:13:58PM -0700, Stan Shebs wrote:
>Are there any plans to solve this?  When I go to test that the packages
>really are downloadable, I can never get on either... -s

Do I remember ftp stats somewhere?  It would be nice to see what's eating
up the bandwidth (I fear it's cygwin...)

cgf

>-------- Original Message --------
>Subject: ftp probs
>Date: Tue, 20 Jun 2000 22:13:34 +0000
>From: oli donald <mcai8od2@ist4.co.umist.ac.uk>
>To: shebs@shebs.cnchost.com
>
>Hi
>
>I'm desperate to get hold of the 7.3.3 release of Mac X-Conq, but I'm 
>having major problems. I've logged on to Sourceare for the last three 
>nights, but I get a maximum download speed of about 500 bytes/sec, 
>with the connection dropping for minutes at a time. That's way below 
>normal for me (usually 4k+), and being in the UK I have to pay by the 
>minute! :) Plus, the FTP sites too busy most of the time. Is there 
>anywhere else I can get the latest version? I tried the mirrors but 
>couldn't find anything recent :(
>
>Anyways cheers for your time and a great bit of software :)
>
>Oli Donald
>Widosm

Re: [Fwd: ftp probs]

[Jim Kingdon]

> Are there any plans to solve this?  When I go to test that the packages
> really are downloadable, I can never get on either... -s

People need to use mirrors (I'm trying to get sourceforge signed up as
a mirror but it might need to wait a few weeks until I'm working there
- I tried to send email, didn't get a response, and haven't pressed
the point).

As for mirrors being out of date, does the cygwin mirror polling
script test for up-to-date-ness?  Someone(TM) should look into what it
would take to use that script site-wide (to test for mirrors being up
even if it can't deal with staleness).

> I get a maximum download speed of about 500 bytes/sec, 
> with the connection dropping for minutes at a time.

I've been noticing this too (slow times from traceroute and web too).
It probably means that we need to restrict the number of anonymous FTP
users even further, although I haven't looked at the statistics lately.

The underlying issue, of course, is that sourceware's T1 is maxxed out
(see graphs and numbers at
http://sourceware.cygnus.com/sourceware/net.html ).

As I've said before (in some contexts in a half-joking manner), I
suspect VA might be willing to help somehow (with the sourceforge
mirror being the simplest way - I've already discussed that part with
them).  People who are planning to stay at Red Hat beyond 1 Jul will
need to think about what kinds of offers of help would be acceptable
to Red Hat, and what other solutions are possible.

I think I asked whether Apple had any interest in setting up a mirror.
Not justified based on xconq, I'm sad to say, but for GCC and
what-not.  Maybe that particular one isn't a good idea, but if
sourceware is really going to be a collaboration between various
companies, we need to figure out how to make that happen in a
financial sense.

I don't know, maybe just asking the right person (sysadmin? Kim?) in
Red Hat (with a case about how Red Hat customers rely on the site) is
all it would take.  But the nice thing about leaving is that I get to
seek another sucker for duties like that one :-).

Re: [Fwd: ftp probs]

[Jim Kingdon]

> Do I remember ftp stats somewhere?

They are at
http://sourceware.cygnus.com/sourceware/net.html
but that page might be a bit slow to load because

(wait for it)

we are out of bandwidth ;-).

> It would be nice to see what's eating up the bandwidth (I fear it's
> cygwin...)

Yup, Cygwin is 35Gbyte/week and GCC+GDB are another 35Gbyte/week.
Everything else is pretty much insignificant (maybe 10-15Gbyte/week
for everything combined).

Re: [Fwd: ftp probs]

[Chris Faylor]

On Wed, Jun 21, 2000 at 01:29:23AM -0400, Jim Kingdon wrote:
>> Are there any plans to solve this?  When I go to test that the packages
>> really are downloadable, I can never get on either... -s
>
>People need to use mirrors (I'm trying to get sourceforge signed up as
>a mirror but it might need to wait a few weeks until I'm working there
>- I tried to send email, didn't get a response, and haven't pressed
>the point).
>
>As for mirrors being out of date, does the cygwin mirror polling
>script test for up-to-date-ness?  Someone(TM) should look into what it
>would take to use that script site-wide (to test for mirrors being up
>even if it can't deal with staleness).

Yes, I believe so.  DJ is refining this all of the time.  I've cc'ed him.

>> I get a maximum download speed of about 500 bytes/sec, 
>> with the connection dropping for minutes at a time.
>
>I've been noticing this too (slow times from traceroute and web too).
>It probably means that we need to restrict the number of anonymous FTP
>users even further, although I haven't looked at the statistics lately.
>
>The underlying issue, of course, is that sourceware's T1 is maxxed out
>(see graphs and numbers at
> http://sourceware.cygnus.com/sourceware/net.html ).

Ah.  Thanks.  I couldn't remember that URL.

It looks like cygwin is taking up quite a bit of the bandwidth.  I
wonder if there is a rogue mirror site out there.

We changed the layout of one of the download directories a couple
of months ago and changed one thing from a symbolic link to an
actual directory.  That screwed up a couple of mirrors.  I wonder
if they are now just repeatedly downloading the 'latest' directory.

Can we get any more detail about what is being downloaded and by whom?

>As I've said before (in some contexts in a half-joking manner), I
>suspect VA might be willing to help somehow (with the sourceforge
>mirror being the simplest way - I've already discussed that part with
>them).  People who are planning to stay at Red Hat beyond 1 Jul will
>need to think about what kinds of offers of help would be acceptable
>to Red Hat, and what other solutions are possible.

A sourceforge mirror would be cool.  As long as they don't ask us
to reciprocate.  :-)

cgf

Re: [Fwd: ftp probs]

[Jason Molenda]

On Wed, Jun 21, 2000 at 01:38:21AM -0400, Chris Faylor wrote:

> Can we get any more detail about what is being downloaded and by whom?

Grope around /www/logs/ftp<something> - it's pretty clear.

The final field is a c or an i - it tells you whether the d/l was 
complete or incomplete.  Useful for a variety of purposes.

I probably have a script in /www/bin which can parse this file and
provide simple summaries  (I'm only on-line for a second right now),
but it's nothing more sophisticated than an awk grep sort type thing.


There are only three solutions.  One, allow only mirror sites to 
access gcc and cygwin.  Two, add a second T1 to sourceware - IanE set
up the Sourceware line so that additional T1s could be added in the
future, anticipating increased demand.  Three, colocate sourceware at
an ISP and pay by-the-byte for the traffic.

Or four, stop the cygwin project and remove all the files. :-)

The bandwidth usage could be reduced slightly - 10% ? - if someone
went around auditing usage.  You know, making periodic snapshots
available only in .bz2 format, adding compressed versions of _all_
web pages so that they'll be downloaded faster, staging mail archives
in a temp directory instead of an ftp directory, that kind of thing.
It's a lot of work and you'd have to know the system really well.
The other options are better.

Jason

Re: [Fwd: ftp probs]

[Jason Molenda]

(real quick ... )

On Tue, Jun 20, 2000 at 11:56:56PM -0700, Jason Molenda wrote:

> There are only three solutions.  [...]


OK, four/five - move gcc/cygwin to sourceforge or gnu.org.

Six (I forgot about this) - set up multiple IP#s on sourceware.
1 for sourceware, 1 for gcc.gnu.org, 1 for cygwin.cygnus.com.
I believe wuftpd or proftpd can support virtual hosts when multiple
IP#s are present, so you can require someone to go to ftp://gcc.gnu.org
to get access to /pub/gcc and ftp://cygwin.cygnus.com to get access
to /pub/cygwin.  Allow only mirror sites to have access to those two
'ftp sites', and ease/drop restrictions on the sourceware 'ftp site'.

The problem is entirely with gcc/cygwin - reduce the ftp load of those
two (both of which have extensive mirroring networks) and the bandwidth
problems evaporate.

FWIW I occasionally read some mailing list archives via the web on
sourceware, and I can also attest that the bandwidth has gone to
crap, especially of late.


Jason

Re: [Fwd: ftp probs]

[DJ Delorie]

> Yes, I believe so.  DJ is refining this all of the time.  I've cc'ed him.

/www/sourceware/htdocs/cygwin/check-mirrors is the script.  About half
our mirrors fail the test each day.

> It looks like cygwin is taking up quite a bit of the bandwidth.  I
> wonder if there is a rogue mirror site out there.

I'll see if I can figure it out.

Re: [Fwd: ftp probs]

[DJ Delorie]

> Allow only mirror sites to have access to those two 'ftp sites'

If we can get a better handle on our ftp mirrors (i.e. registered
contacts for all mirrors, cancel all rogues) I'd go for this solution.
For cygwin, it's just a matter of taking sourceware off our web page
and then the setup program won't offer it as a choice.

Re: [Fwd: ftp probs]

[DJ Delorie]

> Grope around /www/logs/ftp<something> - it's pretty clear.

21% of our ftp traffic is *snapshots* (about 2Gb/day).  If each group
did snapshots half as often, it would reduce our ftp load by 10%.

Our top 30 FTP clients for the 10th-17th attached.  Note that
Yggdrasil does full mirrors of sourceware, but I've had complaints
that it has a 10 user maximum!  It also doesn't mirror right;
"lastest" is still a symlink.  I'll send them a note.

         224,412,100 wh4036.stw.uni-rostock.de
         227,370,945 archive.threeweb.ad.jp
         235,302,855 nephtys.lip6.fr
         263,295,217 ccftp.kpprc.edu.tw
         297,557,102 210.77.38.126  (in cnc.ac.cn somewhere)
         373,046,918 ftp.jaist.ac.jp
         377,356,047 sysg.kek.jp
         398,887,710 203-165-64-57.sugnm1.kt.home.ne.jp
         419,318,464 netserv2.chg.ru
         484,886,139 kepler.mirror.ac.uk
         493,323,342 swallow
         493,933,460 prx1.commerzbank.com
         545,111,124 u6117.seaple.icc.ne.jp
         553,389,538 203-165-64-79.sugnm1.kt.home.ne.jp
         554,374,570 altay.adm.deu.edu.tr
         597,926,849 swiss1.fw-bc.sony.com
         608,625,473 ftpsv1.u-aizu.ac.jp
         626,512,801 Venus.buyingsources.com
         676,789,845 compton.mirror.ac.uk
         729,746,782 pine.ict.nsk.su
         763,177,919 206.153.39.108  (in racsa.co.cr somewhere)
         839,094,832 polar.lerc.nasa.gov
         879,603,531 202.181.240.186  (in cubix.net somewhere)
         923,427,993 eufrat.pvv.ntnu.no
         931,281,969 wizard.freesoftware.com
         952,943,148 ftp1.sinica.edu.tw
         959,735,851 ftp.unina.it
         965,826,867 nic.funet.fi
       1,371,216,531 freya.yggdrasil.com
       2,402,662,730 206.153.39.110  (in racsa.co.cr somewhere)

Re: [Fwd: ftp probs]

[Chris Faylor]

On Wed, Jun 21, 2000 at 10:54:05AM -0400, DJ Delorie wrote:
>> Grope around /www/logs/ftp<something> - it's pretty clear.
>
>21% of our ftp traffic is *snapshots* (about 2Gb/day).  If each group
>did snapshots half as often, it would reduce our ftp load by 10%.

Yow.  I wonder if it is time to use bzip on our snapshots and in our
latest directory, then.

cgf

Re: [Fwd: ftp probs]

[Gerald Pfeifer]

On Wed, 21 Jun 2000, Jason Molenda wrote:
> The problem is entirely with gcc/cygwin - reduce the ftp load of those
> two (both of which have extensive mirroring networks) and the bandwidth
> problems evaporate.

I'm also working on this problem, from the GCC side of things:

 o From our web pages we no longer refer to ftp://gcc.gnu.org when it
   comes to snapshots, but only list our mirror sites.

 o Shortly I will do the same for our releases.

Jeff, currently the "new snapshot is now available" mails directly
specify "egcs.cygnus.com:/pub/egcs/snapshots/2000-06-19". Would you
mind changing that to " http://gcc.gnu.org/snapshots.html "?

As far as I understood DJ's analysis, this alone has the potential
to save at least 10% (possibly more) of overall traffic.

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: [Fwd: ftp probs]

[DJ Delorie]

> Yow.  I wonder if it is time to use bzip on our snapshots and in our
> latest directory, then.

I can work on snapshots, although I'll have to adjust the CGIs on
sourceware as well.

As for latest, we'll have to update setup first, *then* latest.  And
bootstrap, too.

Re: [Fwd: ftp probs]

[DJ Delorie]

> As far as I understood DJ's analysis, this alone has the potential
> to save at least 10% (possibly more) of overall traffic.

That depends on how much of the snapshot traffic is from users vs
mirror sites (all mirrors download snapshots every day), and what
percent of the snapshot traffic is gcc vs binutils, gdb, cygwin, etc.
Cutting the frequency (or size) of snapshots is the only way to cut
down the traffic from mirror sites.

However, pointing away from sourceware's ftp will help.

Re: [Fwd: ftp probs]

[Jeffrey A Law]

  In message < Pine.GSO.4.21.0006211627060.16472-100000@nunki.dbai.tuwien.ac.at >
you write:
  > On Wed, 21 Jun 2000, Jason Molenda wrote:
  > > The problem is entirely with gcc/cygwin - reduce the ftp load of those
  > > two (both of which have extensive mirroring networks) and the bandwidth
  > > problems evaporate.
  > 
  > I'm also working on this problem, from the GCC side of things:
  > 
  >  o From our web pages we no longer refer to ftp://gcc.gnu.org when it
  >    comes to snapshots, but only list our mirror sites.
  > 
  >  o Shortly I will do the same for our releases.
  > 
  > Jeff, currently the "new snapshot is now available" mails directly
  > specify "egcs.cygnus.com:/pub/egcs/snapshots/2000-06-19". Would you
  > mind changing that to " http://gcc.gnu.org/snapshots.html "?
  > 
  > As far as I understood DJ's analysis, this alone has the potential
  > to save at least 10% (possibly more) of overall traffic.
I don't mind changing it; however, we need to make it clear to folks not
using web browswers how to get snapshots too.  ie, we need to note at
least one site that can be used via raw ftp.

I don't see how we can do that and list a mirror since the snapshot process
won't know when mirrors have sucked down the latest snapshot.

Suggestions?

jeff

Re: [Fwd: ftp probs]

[DJ Delorie]

> I don't see how we can do that and list a mirror since the snapshot process
> won't know when mirrors have sucked down the latest snapshot.

What cygwin does is mention the URL for the mirror sites first, and
then says "or sourceware's heavily overloaded ftp server [here]".
That gives them the option but makes it clear that they should use the
mirrors if possible.

DJGPP's mail says "will be on the mirrors within the next day or so"
but then there is no other way to get DJGPP so they have to wait.

In the cygwin group we've taken the position that people needing
snapshots faster than that should use [incremental] cvs anyway.  I
just changed cygwin's snapshot process to run every three days instead
of every day, and we've been reducing the number of stored snapshots
as well.  I'll also start switching from gzip to bzip.

Re: [Fwd: ftp probs]

[Jeffrey A Law]

  In message < 20000621011314.B13594@cygnus.com >you write:
  > On Tue, Jun 20, 2000 at 10:13:58PM -0700, Stan Shebs wrote:
  > >Are there any plans to solve this?  When I go to test that the packages
  > >really are downloadable, I can never get on either... -s
  > 
  > Do I remember ftp stats somewhere?  It would be nice to see what's eating
  > up the bandwidth (I fear it's cygwin...)
I get a summary each week from one of Jason's scripts.  The big hogs (in
order, highest to lowest) are cygwin, gcc, gdb.  After that, nobody matters.

I don't have the report handy, but if anyone is interested I can forward it
when I get it this weekend.

jeff

Re: [Fwd: ftp probs]

[Gerald Pfeifer]

On Wed, 21 Jun 2000, Jeffrey A Law wrote:
>> Jeff, currently the "new snapshot is now available" mails directly
>> specify "egcs.cygnus.com:/pub/egcs/snapshots/2000-06-19". Would you
>> mind changing that to " http://gcc.gnu.org/snapshots.html "?
> I don't mind changing it; however, we need to make it clear to folks not
> using web browswers how to get snapshots too.  ie, we need to note at
> least one site that can be used via raw ftp.
> 
> I don't see how we can do that and list a mirror since the snapshot process
> won't know when mirrors have sucked down the latest snapshot.
> 
> Suggestions?

I see three possibilities:

 1. Delaying the announcement by several hours

    This probably is useful in any case, as many folks tend to download
    something immediately once it has been announced; without such a delay
    they definitely will use the main site.

 2. I believe these snapshots are done from cron, so we can (and I'm
    willing to take care of that -- just tell me the day and time) tell
    the folks at ftp.freesoftware.com when to synchronize.

 3. We can check whether the folks there can mirror every hour. If it's
    done properly, a mirror run when there has been no change should be
    cheap, essentially a dir list.

Combining 1. and either one of 2. and 3. seems to make sense. To me at
least! ;-)

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/
Have a look at http://petition.eurolinux.org !

[Fwd: ftp probs]

[Stan Shebs]

Are there any plans to solve this?  When I go to test that the packages
really are downloadable, I can never get on either... -s

-------- Original Message --------
Subject: ftp probs
Date: Tue, 20 Jun 2000 22:13:34 +0000
From: oli donald <mcai8od2@ist4.co.umist.ac.uk>
To: shebs@shebs.cnchost.com

Hi

I'm desperate to get hold of the 7.3.3 release of Mac X-Conq, but I'm 
having major problems. I've logged on to Sourceare for the last three 
nights, but I get a maximum download speed of about 500 bytes/sec, 
with the connection dropping for minutes at a time. That's way below 
normal for me (usually 4k+), and being in the UK I have to pay by the 
minute! :) Plus, the FTP sites too busy most of the time. Is there 
anywhere else I can get the latest version? I tried the mirrors but 
couldn't find anything recent :(

Anyways cheers for your time and a great bit of software :)

Oli Donald
Widosm

Re: [Fwd: ftp probs]

[Jeffrey A Law]

  In message < 20000621011314.B13594@cygnus.com >you write:
  > On Tue, Jun 20, 2000 at 10:13:58PM -0700, Stan Shebs wrote:
  > >Are there any plans to solve this?  When I go to test that the packages
  > >really are downloadable, I can never get on either... -s
  > 
  > Do I remember ftp stats somewhere?  It would be nice to see what's eating
  > up the bandwidth (I fear it's cygwin...)
I get a summary each week from one of Jason's scripts.  The big hogs (in
order, highest to lowest) are cygwin, gcc, gdb.  After that, nobody matters.

I don't have the report handy, but if anyone is interested I can forward it
when I get it this weekend.

jeff

Re: [Fwd: ftp probs]

[DJ Delorie]

> I don't see how we can do that and list a mirror since the snapshot process
> won't know when mirrors have sucked down the latest snapshot.

What cygwin does is mention the URL for the mirror sites first, and
then says "or sourceware's heavily overloaded ftp server [here]".
That gives them the option but makes it clear that they should use the
mirrors if possible.

DJGPP's mail says "will be on the mirrors within the next day or so"
but then there is no other way to get DJGPP so they have to wait.

In the cygwin group we've taken the position that people needing
snapshots faster than that should use [incremental] cvs anyway.  I
just changed cygwin's snapshot process to run every three days instead
of every day, and we've been reducing the number of stored snapshots
as well.  I'll also start switching from gzip to bzip.

Re: [Fwd: ftp probs]

[DJ Delorie]

> Allow only mirror sites to have access to those two 'ftp sites'

If we can get a better handle on our ftp mirrors (i.e. registered
contacts for all mirrors, cancel all rogues) I'd go for this solution.
For cygwin, it's just a matter of taking sourceware off our web page
and then the setup program won't offer it as a choice.

Re: [Fwd: ftp probs]

[Jason Molenda]

(real quick ... )

On Tue, Jun 20, 2000 at 11:56:56PM -0700, Jason Molenda wrote:

> There are only three solutions.  [...]


OK, four/five - move gcc/cygwin to sourceforge or gnu.org.

Six (I forgot about this) - set up multiple IP#s on sourceware.
1 for sourceware, 1 for gcc.gnu.org, 1 for cygwin.cygnus.com.
I believe wuftpd or proftpd can support virtual hosts when multiple
IP#s are present, so you can require someone to go to ftp://gcc.gnu.org
to get access to /pub/gcc and ftp://cygwin.cygnus.com to get access
to /pub/cygwin.  Allow only mirror sites to have access to those two
'ftp sites', and ease/drop restrictions on the sourceware 'ftp site'.

The problem is entirely with gcc/cygwin - reduce the ftp load of those
two (both of which have extensive mirroring networks) and the bandwidth
problems evaporate.

FWIW I occasionally read some mailing list archives via the web on
sourceware, and I can also attest that the bandwidth has gone to
crap, especially of late.


Jason

Re: [Fwd: ftp probs]

[Jim Kingdon]

> Are there any plans to solve this?  When I go to test that the packages
> really are downloadable, I can never get on either... -s

People need to use mirrors (I'm trying to get sourceforge signed up as
a mirror but it might need to wait a few weeks until I'm working there
- I tried to send email, didn't get a response, and haven't pressed
the point).

As for mirrors being out of date, does the cygwin mirror polling
script test for up-to-date-ness?  Someone(TM) should look into what it
would take to use that script site-wide (to test for mirrors being up
even if it can't deal with staleness).

> I get a maximum download speed of about 500 bytes/sec, 
> with the connection dropping for minutes at a time.

I've been noticing this too (slow times from traceroute and web too).
It probably means that we need to restrict the number of anonymous FTP
users even further, although I haven't looked at the statistics lately.

The underlying issue, of course, is that sourceware's T1 is maxxed out
(see graphs and numbers at
http://sourceware.cygnus.com/sourceware/net.html ).

As I've said before (in some contexts in a half-joking manner), I
suspect VA might be willing to help somehow (with the sourceforge
mirror being the simplest way - I've already discussed that part with
them).  People who are planning to stay at Red Hat beyond 1 Jul will
need to think about what kinds of offers of help would be acceptable
to Red Hat, and what other solutions are possible.

I think I asked whether Apple had any interest in setting up a mirror.
Not justified based on xconq, I'm sad to say, but for GCC and
what-not.  Maybe that particular one isn't a good idea, but if
sourceware is really going to be a collaboration between various
companies, we need to figure out how to make that happen in a
financial sense.

I don't know, maybe just asking the right person (sysadmin? Kim?) in
Red Hat (with a case about how Red Hat customers rely on the site) is
all it would take.  But the nice thing about leaving is that I get to
seek another sucker for duties like that one :-).

Re: [Fwd: ftp probs]

[Jeffrey A Law]

  In message < Pine.GSO.4.21.0006211627060.16472-100000@nunki.dbai.tuwien.ac.at >
you write:
  > On Wed, 21 Jun 2000, Jason Molenda wrote:
  > > The problem is entirely with gcc/cygwin - reduce the ftp load of those
  > > two (both of which have extensive mirroring networks) and the bandwidth
  > > problems evaporate.
  > 
  > I'm also working on this problem, from the GCC side of things:
  > 
  >  o From our web pages we no longer refer to ftp://gcc.gnu.org when it
  >    comes to snapshots, but only list our mirror sites.
  > 
  >  o Shortly I will do the same for our releases.
  > 
  > Jeff, currently the "new snapshot is now available" mails directly
  > specify "egcs.cygnus.com:/pub/egcs/snapshots/2000-06-19". Would you
  > mind changing that to " http://gcc.gnu.org/snapshots.html "?
  > 
  > As far as I understood DJ's analysis, this alone has the potential
  > to save at least 10% (possibly more) of overall traffic.
I don't mind changing it; however, we need to make it clear to folks not
using web browswers how to get snapshots too.  ie, we need to note at
least one site that can be used via raw ftp.

I don't see how we can do that and list a mirror since the snapshot process
won't know when mirrors have sucked down the latest snapshot.

Suggestions?

jeff

Re: [Fwd: ftp probs]

[DJ Delorie]

> As far as I understood DJ's analysis, this alone has the potential
> to save at least 10% (possibly more) of overall traffic.

That depends on how much of the snapshot traffic is from users vs
mirror sites (all mirrors download snapshots every day), and what
percent of the snapshot traffic is gcc vs binutils, gdb, cygwin, etc.
Cutting the frequency (or size) of snapshots is the only way to cut
down the traffic from mirror sites.

However, pointing away from sourceware's ftp will help.

Re: [Fwd: ftp probs]

[DJ Delorie]

> Grope around /www/logs/ftp<something> - it's pretty clear.

21% of our ftp traffic is *snapshots* (about 2Gb/day).  If each group
did snapshots half as often, it would reduce our ftp load by 10%.

Our top 30 FTP clients for the 10th-17th attached.  Note that
Yggdrasil does full mirrors of sourceware, but I've had complaints
that it has a 10 user maximum!  It also doesn't mirror right;
"lastest" is still a symlink.  I'll send them a note.

         224,412,100 wh4036.stw.uni-rostock.de
         227,370,945 archive.threeweb.ad.jp
         235,302,855 nephtys.lip6.fr
         263,295,217 ccftp.kpprc.edu.tw
         297,557,102 210.77.38.126  (in cnc.ac.cn somewhere)
         373,046,918 ftp.jaist.ac.jp
         377,356,047 sysg.kek.jp
         398,887,710 203-165-64-57.sugnm1.kt.home.ne.jp
         419,318,464 netserv2.chg.ru
         484,886,139 kepler.mirror.ac.uk
         493,323,342 swallow
         493,933,460 prx1.commerzbank.com
         545,111,124 u6117.seaple.icc.ne.jp
         553,389,538 203-165-64-79.sugnm1.kt.home.ne.jp
         554,374,570 altay.adm.deu.edu.tr
         597,926,849 swiss1.fw-bc.sony.com
         608,625,473 ftpsv1.u-aizu.ac.jp
         626,512,801 Venus.buyingsources.com
         676,789,845 compton.mirror.ac.uk
         729,746,782 pine.ict.nsk.su
         763,177,919 206.153.39.108  (in racsa.co.cr somewhere)
         839,094,832 polar.lerc.nasa.gov
         879,603,531 202.181.240.186  (in cubix.net somewhere)
         923,427,993 eufrat.pvv.ntnu.no
         931,281,969 wizard.freesoftware.com
         952,943,148 ftp1.sinica.edu.tw
         959,735,851 ftp.unina.it
         965,826,867 nic.funet.fi
       1,371,216,531 freya.yggdrasil.com
       2,402,662,730 206.153.39.110  (in racsa.co.cr somewhere)

Re: [Fwd: ftp probs]

[Jason Molenda]

On Wed, Jun 21, 2000 at 01:38:21AM -0400, Chris Faylor wrote:

> Can we get any more detail about what is being downloaded and by whom?

Grope around /www/logs/ftp<something> - it's pretty clear.

The final field is a c or an i - it tells you whether the d/l was 
complete or incomplete.  Useful for a variety of purposes.

I probably have a script in /www/bin which can parse this file and
provide simple summaries  (I'm only on-line for a second right now),
but it's nothing more sophisticated than an awk grep sort type thing.


There are only three solutions.  One, allow only mirror sites to 
access gcc and cygwin.  Two, add a second T1 to sourceware - IanE set
up the Sourceware line so that additional T1s could be added in the
future, anticipating increased demand.  Three, colocate sourceware at
an ISP and pay by-the-byte for the traffic.

Or four, stop the cygwin project and remove all the files. :-)

The bandwidth usage could be reduced slightly - 10% ? - if someone
went around auditing usage.  You know, making periodic snapshots
available only in .bz2 format, adding compressed versions of _all_
web pages so that they'll be downloaded faster, staging mail archives
in a temp directory instead of an ftp directory, that kind of thing.
It's a lot of work and you'd have to know the system really well.
The other options are better.

Jason

Re: [Fwd: ftp probs]

[Chris Faylor]

On Wed, Jun 21, 2000 at 01:29:23AM -0400, Jim Kingdon wrote:
>> Are there any plans to solve this?  When I go to test that the packages
>> really are downloadable, I can never get on either... -s
>
>People need to use mirrors (I'm trying to get sourceforge signed up as
>a mirror but it might need to wait a few weeks until I'm working there
>- I tried to send email, didn't get a response, and haven't pressed
>the point).
>
>As for mirrors being out of date, does the cygwin mirror polling
>script test for up-to-date-ness?  Someone(TM) should look into what it
>would take to use that script site-wide (to test for mirrors being up
>even if it can't deal with staleness).

Yes, I believe so.  DJ is refining this all of the time.  I've cc'ed him.

>> I get a maximum download speed of about 500 bytes/sec, 
>> with the connection dropping for minutes at a time.
>
>I've been noticing this too (slow times from traceroute and web too).
>It probably means that we need to restrict the number of anonymous FTP
>users even further, although I haven't looked at the statistics lately.
>
>The underlying issue, of course, is that sourceware's T1 is maxxed out
>(see graphs and numbers at
> http://sourceware.cygnus.com/sourceware/net.html ).

Ah.  Thanks.  I couldn't remember that URL.

It looks like cygwin is taking up quite a bit of the bandwidth.  I
wonder if there is a rogue mirror site out there.

We changed the layout of one of the download directories a couple
of months ago and changed one thing from a symbolic link to an
actual directory.  That screwed up a couple of mirrors.  I wonder
if they are now just repeatedly downloading the 'latest' directory.

Can we get any more detail about what is being downloaded and by whom?

>As I've said before (in some contexts in a half-joking manner), I
>suspect VA might be willing to help somehow (with the sourceforge
>mirror being the simplest way - I've already discussed that part with
>them).  People who are planning to stay at Red Hat beyond 1 Jul will
>need to think about what kinds of offers of help would be acceptable
>to Red Hat, and what other solutions are possible.

A sourceforge mirror would be cool.  As long as they don't ask us
to reciprocate.  :-)

cgf

Re: [Fwd: ftp probs]

[Chris Faylor]

On Wed, Jun 21, 2000 at 10:54:05AM -0400, DJ Delorie wrote:
>> Grope around /www/logs/ftp<something> - it's pretty clear.
>
>21% of our ftp traffic is *snapshots* (about 2Gb/day).  If each group
>did snapshots half as often, it would reduce our ftp load by 10%.

Yow.  I wonder if it is time to use bzip on our snapshots and in our
latest directory, then.

cgf

Re: [Fwd: ftp probs]

[Gerald Pfeifer]

On Wed, 21 Jun 2000, Jason Molenda wrote:
> The problem is entirely with gcc/cygwin - reduce the ftp load of those
> two (both of which have extensive mirroring networks) and the bandwidth
> problems evaporate.

I'm also working on this problem, from the GCC side of things:

 o From our web pages we no longer refer to ftp://gcc.gnu.org when it
   comes to snapshots, but only list our mirror sites.

 o Shortly I will do the same for our releases.

Jeff, currently the "new snapshot is now available" mails directly
specify "egcs.cygnus.com:/pub/egcs/snapshots/2000-06-19". Would you
mind changing that to " http://gcc.gnu.org/snapshots.html "?

As far as I understood DJ's analysis, this alone has the potential
to save at least 10% (possibly more) of overall traffic.

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: [Fwd: ftp probs]

[DJ Delorie]

> Yes, I believe so.  DJ is refining this all of the time.  I've cc'ed him.

/www/sourceware/htdocs/cygwin/check-mirrors is the script.  About half
our mirrors fail the test each day.

> It looks like cygwin is taking up quite a bit of the bandwidth.  I
> wonder if there is a rogue mirror site out there.

I'll see if I can figure it out.

Re: [Fwd: ftp probs]

[Chris Faylor]

On Tue, Jun 20, 2000 at 10:13:58PM -0700, Stan Shebs wrote:
>Are there any plans to solve this?  When I go to test that the packages
>really are downloadable, I can never get on either... -s

Do I remember ftp stats somewhere?  It would be nice to see what's eating
up the bandwidth (I fear it's cygwin...)

cgf

>-------- Original Message --------
>Subject: ftp probs
>Date: Tue, 20 Jun 2000 22:13:34 +0000
>From: oli donald <mcai8od2@ist4.co.umist.ac.uk>
>To: shebs@shebs.cnchost.com
>
>Hi
>
>I'm desperate to get hold of the 7.3.3 release of Mac X-Conq, but I'm 
>having major problems. I've logged on to Sourceare for the last three 
>nights, but I get a maximum download speed of about 500 bytes/sec, 
>with the connection dropping for minutes at a time. That's way below 
>normal for me (usually 4k+), and being in the UK I have to pay by the 
>minute! :) Plus, the FTP sites too busy most of the time. Is there 
>anywhere else I can get the latest version? I tried the mirrors but 
>couldn't find anything recent :(
>
>Anyways cheers for your time and a great bit of software :)
>
>Oli Donald
>Widosm

Re: [Fwd: ftp probs]

[DJ Delorie]

> Yow.  I wonder if it is time to use bzip on our snapshots and in our
> latest directory, then.

I can work on snapshots, although I'll have to adjust the CGIs on
sourceware as well.

As for latest, we'll have to update setup first, *then* latest.  And
bootstrap, too.

Re: [Fwd: ftp probs]

[Gerald Pfeifer]

On Wed, 21 Jun 2000, Jeffrey A Law wrote:
>> Jeff, currently the "new snapshot is now available" mails directly
>> specify "egcs.cygnus.com:/pub/egcs/snapshots/2000-06-19". Would you
>> mind changing that to " http://gcc.gnu.org/snapshots.html "?
> I don't mind changing it; however, we need to make it clear to folks not
> using web browswers how to get snapshots too.  ie, we need to note at
> least one site that can be used via raw ftp.
> 
> I don't see how we can do that and list a mirror since the snapshot process
> won't know when mirrors have sucked down the latest snapshot.
> 
> Suggestions?

I see three possibilities:

 1. Delaying the announcement by several hours

    This probably is useful in any case, as many folks tend to download
    something immediately once it has been announced; without such a delay
    they definitely will use the main site.

 2. I believe these snapshots are done from cron, so we can (and I'm
    willing to take care of that -- just tell me the day and time) tell
    the folks at ftp.freesoftware.com when to synchronize.

 3. We can check whether the folks there can mirror every hour. If it's
    done properly, a mirror run when there has been no change should be
    cheap, essentially a dir list.

Combining 1. and either one of 2. and 3. seems to make sense. To me at
least! ;-)

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/
Have a look at http://petition.eurolinux.org !

Re: [Fwd: ftp probs]

[Jim Kingdon]

> Do I remember ftp stats somewhere?

They are at
http://sourceware.cygnus.com/sourceware/net.html
but that page might be a bit slow to load because

(wait for it)

we are out of bandwidth ;-).

> It would be nice to see what's eating up the bandwidth (I fear it's
> cygwin...)

Yup, Cygwin is 35Gbyte/week and GCC+GDB are another 35Gbyte/week.
Everything else is pretty much insignificant (maybe 10-15Gbyte/week
for everything combined).