splitting sourceware into pieces?

splitting sourceware into pieces?

[Phil Edwards]

Not the physical machine, but rather its functionality.  (Y'all can resume
breathing now.  :-)

During the middle of the day, mail seems to be taking upwards of half an
hour to get processed by the lists; CVS connections can be dog slow for
anyone who has less than an OC-3, etc, etc, you all have heard this before.
How about delgating email to one machine and everything else to another?
Or maybe FTP and web traffic to one machine, and everything else to another.
You get the picture; it doesn't need to all be on one system.

Maybe this is a completely pointless idea considering the upcoming co-loc.


Phil

-- 
pedwards at disaster dot jaj dot com  |  pme at sources dot redhat dot com
devphil at several other less interesting addresses in various dot domains
The gods do not protect fools.  Fools are protected by more capable fools.

Re: splitting sourceware into pieces?

[Andrew Cagney]

Phil Edwards wrote:
> 
> Not the physical machine, but rather its functionality.  (Y'all can resume
> breathing now.  :-)

Er, you mean functionality dedicated to specific physical machines.

> During the middle of the day, mail seems to be taking upwards of half an
> hour to get processed by the lists; CVS connections can be dog slow for
> anyone who has less than an OC-3, etc, etc, you all have heard this before.
> How about delgating email to one machine and everything else to another?
> Or maybe FTP and web traffic to one machine, and everything else to another.
> You get the picture; it doesn't need to all be on one system.

While I might have a strong preference for one function, one physical
machine (if someone hoses mail, they don't hose CVS, or my cron jobs
don't hose web functionality ...) I do feel like I'm just doing a ``me
too''.  I don't see my self being able to contribute to this.

I suspect Jason would have the most imput.

	Andrew

Re: splitting sourceware into pieces?

[Jason Molenda]

On Wed, Feb 21, 2001 at 02:21:47PM -0500, Phil Edwards wrote:

> During the middle of the day, mail seems to be taking upwards of half an
> hour to get processed by the lists; CVS connections can be dog slow for
> anyone who has less than an OC-3, etc, etc, you all have heard this before.


I haven't monitored the system's performance in a long time, but
I would suspect (read - I'd be surprised if I'm wrong) that the
bottleneck is entirely in the T1 to the system.

The oft-mentioned co location would address this problem.  I gather
a new system would be constructed at that time as well.  The disks
won't get much faster (drive technology hasn't advanced much in
the past two years, and I'd put in the fastest ~$1k drives I could
find back then), but the CPU will be much faster (it has a 450MHz
P-III right now).  I suppose memory might get a bit bigger, but it
has 768MB right now.  

I kind of doubt we're CPU starved very often, though.

If the network is the bottleneck, then separating the functionality
across multiple hosts doesn't gain much.

If there was a bonanza of extra CPU, a few more tricks could be
used, like maybe using a loadable apache module which would gzip
_all_ the content on the site on the fly.  Right now there is a
module which will send gzipped versions of content if they exist,
but they have to be pre-generated, and it can't interoperate with
server side includes.  This change would decrease the response time
for people browsing the web archives of mailing lists by quite a
bit I think.


> You get the picture; it doesn't need to all be on one system.

It doesn't need to be, but it makes a lot of things a whole lot
easier when it is.  On the good side, all the services on sourceware
are tightly integrated so we can have mailing list archives right in
the ftp dir with no problems.  On the bad side, it makes maintenance
a lot harder because anyone wanting to change things has to figure out
how all the hair sticks together.


If it were necessary to stick to the T1 and people were unhappy
with system performance, I'd attack the ftp traffic side first.
It's something like half the traffic, and most of it is gzipped
(not even bzip2'ed!!) gcc snapshots and the like.  I'd at least
get these compressed in .bz2 instead of .gz, and I'd probably make
the snapshots etc available only to mirror sites and require users
to get them from the mirrors.

I haven't looked over the ftp logs (/www/logs/ftp*), but I'm sure
some examination of what content is being downloaded the most would
suggest some obvious alternatives for reducing the load on the T1.

Jason

Re: splitting sourceware into pieces?

[Jason Molenda]

On Wed, Feb 21, 2001 at 01:06:10PM -0800, Jason Molenda wrote:

> 
> I kind of doubt we're CPU starved very often, though.
> 

I just logged into the system, and there actually is about 0% idle
on the CPU with just a few processes executing concurrently.  I
still don't think this is where the real bottleneck is, but it does
mean that getting a second CPU or upgrading to a faster CPU could
help things a bit.

Jason

Re: splitting sourceware into pieces?

[Phil Edwards]

On Wed, Feb 21, 2001 at 01:06:10PM -0800, Jason Molenda wrote:
> If the network is the bottleneck, then separating the functionality
> across multiple hosts doesn't gain much.

True, of course.  I had a list of "things to mention in my note to
overseers," and the idea of splitting the tasks up amongst multiple boxes
would only help if they aren't sharing the same piece of cable, the same
hub, etc.  Guess which of the items didn't get a check mark after I wrote
the message.


> If it were necessary to stick to the T1 and people were unhappy
> with system performance, I'd attack the ftp traffic side first.
[...]
> and I'd probably make
> the snapshots etc available only to mirror sites and require users
> to get them from the mirrors.

That sounds like a good idea even after the box is moved to the co-loc.


Phil

-- 
pedwards at disaster dot jaj dot com  |  pme at sources dot redhat dot com
devphil at several other less interesting addresses in various dot domains
The gods do not protect fools.  Fools are protected by more capable fools.

Re: splitting sourceware into pieces?

[Jason Molenda]

On Wed, Feb 21, 2001 at 03:48:57PM -0500, Andrew Cagney wrote:

> 
> While I might have a strong preference for one function, one physical
> machine (if someone hoses mail, they don't hose CVS, or my cron jobs
> don't hose web functionality ...) I do feel like I'm just doing a ``me
> too''.  I don't see my self being able to contribute to this.
> 
> I suspect Jason would have the most input.


Well, I have opinions anyway. :-)  My opinions are just that - the
current maintainers should (naturally) feel free to do as they see
fit.

I'm serious about the additional features we're able to do by having
everything on one host - it creeps into lots of places.  Tom's
modifications log_accum which will notice a mention of a PR in a
cvs commit msg (e.g. "Fixes gdb/30") and cause the commit notice
to be appended to the PR if it exists.  That'd be tough to do
without the cvs server and the gnats database on the same host.
I mentioned the mailing list mbox archives being put in the ftp
server - a little bit harder to do if the mail server and the ftp
server are separate.  Or the link on the mailing list archive index
pages which says like

  March 2000  [<a href="">click here for mbox bzip2'ed version (408k)</a>]
  Feb 2000  [<a href="">click here for mbox bzip2'ed version (350k)</a>]

That'd be tougher to do if the web server and the ftp server weren't
on the same host.  Web archiving of the mail notes is a case where
having the mail server and the web server on the same box simplifies
things.  The project web pages in cvs is easy because the web server
is on the same box as the cvs server -- if the gdb home page was
maintained on one system but served from another, you've got some
additional stuff to work out to make it all happen.  It probably
wouldn't be as real-time-slick as the current setup.

These are just off the top of my head.  I love the compartmentalization
style of security as much as the next person, but you gain a lot if 
you're willing to give that up.

Jason

PS-  Another thing which could be considered -- ezmlm has facilities
for multiple hosts distributing outgoing mail notes.  Y'all could
saturate the Cygnus T1 in addition to the Sourceware T1 is you put
an ezmlm slave server on that side of the net and foisted some of
the deliveries off to that.  I've never looked in depth as this side
of ezmlm's capabilities, but I know they exist.

Re: splitting sourceware into pieces?

[Jeffrey A Law]

  In message < 20010221142147.A20931@disaster.jaj.com >you write:
  > Not the physical machine, but rather its functionality.  (Y'all can resume
  > breathing now.  :-)
  > 
  > During the middle of the day, mail seems to be taking upwards of half an
  > hour to get processed by the lists; CVS connections can be dog slow for
  > anyone who has less than an OC-3, etc, etc, you all have heard this before.
  > How about delgating email to one machine and everything else to another?
  > Or maybe FTP and web traffic to one machine, and everything else to another
  > .
  > You get the picture; it doesn't need to all be on one system.
  > 
  > Maybe this is a completely pointless idea considering the upcoming co-loc.
We're mostly concerned with getting ready for moving to the co-loc;
as part of that move we're purchasing a new machine, roughly 2X the
performance of the current box which should help a lot.

jeff

Re: splitting sourceware into pieces?

[Phil Edwards]

On Wed, Feb 21, 2001 at 02:28:23PM -0800, Jason Molenda wrote:
> Tom's
> modifications log_accum which will notice a mention of a PR in a
> cvs commit msg (e.g. "Fixes gdb/30") and cause the commit notice
> to be appended to the PR if it exists.

*boggle*

I never knew that.  Wow.  Most cool.  Thanks, Tom!

I've seen people writing those notes but just thought it was "only" good
style, not actually feeding into GNATS.


Phil

-- 
pedwards at disaster dot jaj dot com  |  pme at sources dot redhat dot com
devphil at several other less interesting addresses in various dot domains
The gods do not protect fools.  Fools are protected by more capable fools.

Re: splitting sourceware into pieces?

[Jason Molenda]

On Wed, Feb 21, 2001 at 05:53:23PM -0500, Phil Edwards wrote:
> On Wed, Feb 21, 2001 at 02:28:23PM -0800, Jason Molenda wrote:
> > Tom's
> > modifications log_accum which will notice a mention of a PR in a
> > cvs commit msg (e.g. "Fixes gdb/30") and cause the commit notice
> > to be appended to the PR if it exists.
> 
> *boggle*
> 
> I never knew that.  Wow.  Most cool.  Thanks, Tom!
> 
> I've seen people writing those notes but just thought it was "only" good
> style, not actually feeding into GNATS.


I think you need to add the -G command line opt to your log_accum
command line in CVSROOT/loginfo for it to be enabled.  cf

http://sourceware.cygnus.com/cgi-bin/cvsweb.cgi/CVSROOT/loginfo?rev=1.7&content-type=text/x-cvsweb-markup&cvsroot=java


The -C cmd line option adds URLs to the bottom of cvs commit messages
which will show you the changes for that commit (except for newly added
files or just-deleted files).  Tom added that one too.  In fact, together
these options make a nice cheap integration of the bug tracking and the
revision control systems.  You commit a change to foo.c which fixes gcc/35,
the cvs commit message is appended to gcc/35 along with the URLs to get
the patch via cvsweb.  

J

Re: splitting sourceware into pieces?

[Jeffrey A Law]

  In message < 20010221130610.A11627@shell17.ba.best.com >you write:
  > I haven't monitored the system's performance in a long time, but
  > I would suspect (read - I'd be surprised if I'm wrong) that the
  > bottleneck is entirely in the T1 to the system.
Most likely.  Though I have seen CPU utilization increase a lot over
the last year.  More CPU power isn't going to hurt :-)


  > The oft-mentioned co location would address this problem.
Yes.

  > I gather a new system would be constructed at that time as well.
Yes.  We need a rack-mountable box for the coloc.  I've already got
the new box roughly spec'd out.    Figure it's 2X bigger in just about
every category except physical size :-)

jeff