* sourceware hardware transition plan, part 1/N
@ 2012-09-07 21:47 Frank Ch. Eigler
2012-09-07 22:47 ` Joseph S. Myers
` (2 more replies)
0 siblings, 3 replies; 23+ messages in thread
From: Frank Ch. Eigler @ 2012-09-07 21:47 UTC (permalink / raw)
To: Sourceware Overseers
Hi -
As many of you know, two IBM x3650 servers have been donated by Red
Hat to the sourceware project. They've been waiting for our time to
configure them and migrate the content from the old machines over.
The migration will be non-trivial. We're moving from RHEL4->RHEL6. I
would like to move from unusual/obsolete services to mainstream ones
if at all possible, for simplicity of administration, and to reduce
exposure to vulnerabilities. I would like to retire several old
services (leaving behind a read-only web mirror if appropriate).
As a first step, here is a draft of the overall "before & after".
Please let me know what parts may be incomplete or controversial.
After this is settled, I'll put forward a task list as to how to get
there.
old: new:
hardware:
server[1]+server[23] server[45]
no redundancy warm standby, ~identical servers
dell 2850 ibm x3650 M3
500GB raid5 2TB raid6
16GB RAM 24GB RAM
4cpu*Xeon 3GHz 2cpu*4core*2thread 2.4GHz Xeon E5620
OS:
rhel4 i686 rhel6 x6-64
iptables <same>
vcs:
svn <same>
git <same>
cvs <same>
cvsupd <retire>
misc:
djbdns bind
proftpd <same>
ntpd <same>
rsync <same>
mysql <same>
gnats <retire>
rssh <same>
mail:
qmail sendmail
ezmlm mailman
mhonarc mailman
spamassassin <same>, focus on inbound mail
clamav <same>
mlcheckd <retire>
httpd:
httpd 2.0 httpd 2.2 + https
multi-root (gcc/sware/ecos/+) <same>
faq-o-matic mediawiki
moinmoin mediawiki
cvsweb viewvc
gitweb <same>
mnogosearch <same>
bugzilla <same>
retired services <html snapshot preserved>
content cvs git
projects:
root users <same>
project files <same>
users <same>
cron jobs <same>
gcc, sourceware, infra <same>
gcc supybot <same>
cygwin upset <same>
- FChE
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: sourceware hardware transition plan, part 1/N 2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler @ 2012-09-07 22:47 ` Joseph S. Myers 2012-09-08 12:23 ` Frank Ch. Eigler 2012-09-07 23:55 ` Jonathan Larmour 2012-09-07 23:59 ` Christopher Faylor 2 siblings, 1 reply; 23+ messages in thread From: Joseph S. Myers @ 2012-09-07 22:47 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: Sourceware Overseers On Fri, 7 Sep 2012, Frank Ch. Eigler wrote: > ezmlm mailman > mhonarc mailman mailman's built-in archiver is pretty hopeless at URL stability (it may well also lack other features such as the linking to a raw text version of each message that we've put into our mhonarc configuration). I advise continuing to do web archives through much the same mhonarc configuration as at present and hiding any built-in mailman archives as far as possible to avoid people posting references to their unstable URLs. Is it possible to get mailman to assign unique sequence numbers to each message to a list like ezmlm does (whether in the envelope return address or in a header) so you can readily tell what order messages were supposed to be in and whether there are any gaps in the messages you have received, or that appear in the list archives? That's the feature of ezmlm I find most useful (other than that, mailman certainly has various advantages). -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-07 22:47 ` Joseph S. Myers @ 2012-09-08 12:23 ` Frank Ch. Eigler 2012-09-08 13:19 ` Joseph S. Myers 2012-10-28 22:44 ` Gerald Pfeifer 0 siblings, 2 replies; 23+ messages in thread From: Frank Ch. Eigler @ 2012-09-08 12:23 UTC (permalink / raw) To: Sourceware Overseers Hi - On Fri, Sep 07, 2012 at 10:47:24PM +0000, Joseph S. Myers wrote: > [...] > > ezmlm mailman > > mhonarc mailman > > mailman's built-in archiver is pretty hopeless at URL stability (it may > well also lack other features such as the linking to a raw text version of > each message that we've put into our mhonarc configuration). I advise > continuing to do web archives through much the same mhonarc configuration > as at present [...] I haven't heard of any concerns about mailman URL stability before. The installations I know of (@redhat.com, @fedorahosted.org) seem to be fine. > Is it possible to get mailman [...] I don't know. Reading more about it now, it's not as full-featured as I thought it was. Note that our mhonarc installation has been a source of problems too, e.g. recurrent XSS bugs in the get-raw-message CGIs. On Sat, Sep 08, 2012 at 12:55:30AM +0100, Jonathan Larmour wrote: > The main thing I want to comment on is that this should be as phased a > changeover as possible. [...] Given the extent to which the various sourceware services are interdependent, this may not be possible. > Care is also needed given the level of coordination with individual projects > which will be required. Most obviously for mail and mail archives, but also > the wikis. Projects need to not just be given advanced warning, but actually > say when they think they will be ready for switchovers (with new web pages > written, users updated, etc.), and assuming the timescales are within reason, > schedule things around that. It would be better if the transition were not drawn out or too synchronized across projects. I was thinking something like a half-day's outage, bringing the essential services online first (version control and mailing lists), others (e.g. wikis) as the conversion tools / rsync finish. > [...] > > 16GB RAM 24GB RAM > > Given the change from i686 to x86-64, as well as moving to much more > memory-hungry services (bind, mailman, etc.), will this RAM increment be > enough? [...] Yeah, that is a little small. Luckily, the new servers have 18 DIMM slots, 6 used each, with DDR3 PC3-10600 1333MHz ECC Registered cards, e.g. http://www.memoryamerica.com/m393b5170fh0-ch9.html, so we can fairly cheaply double/triple it. > > mail: > > qmail sendmail > > I'm a bit surprised about wanting to use sendmail. [...] > I would have thought postfix or exim more appropriate [...] I have about the same amount of familiarity with them all; they could each the job. > One thing missing off your list is backups. The problem with warm > standby servers is that they allow problems to be replicated quickly > and efficiently :-). Obviously I'm more concerned about system or > human mess-ups than hardware failure here. Right. One possibility is to slice up the larger storage into LVM areas that serve as periodic snapshots of the root / project partitions. Then the machines would serve as warm spares for each other, plus anti-fat-finger backups. Off-site backups would be as per the status quo. > I assume Angela is still doing backups, [...] (So am I, periodically.) On Fri, Sep 07, 2012 at 07:58:45PM -0400, Christopher Faylor wrote: > [...] > I am probably stepping into a religious debate but wouldn't it make more > sense to use postfix here? [...] > (And as strange as qmail is, I have had a lot more trouble tweaking > sendmail and postfix than I ever did with qmail) [...] OK. > You missed qpsmtpd, btw. I don't know if that can be made to work with > the above or not. AFAIK yes, if it's worth keeping (as opposed to putting spam filtering directly into the normal MTA paths.) > > spamassassin <same>, focus on inbound mail > > "inbound mail"? As opposed to...? Outbound. E.g. frequent current complaints about our bugzilla installation relate to the amount of time taken by spam checking for outgoing messages. > > clamav <same> > > mlcheckd <retire> > > See above. mlcheckd is how we block the disclaimer email. And, it > works as a secondary line of spam defense for when spamassassin fails. > I also use it to block off-topic email in the cygwin list. > I'm sure that could still be done with mailman but it will take research. OK. > >httpd: > > httpd 2.0 httpd 2.2 + https > > Maybe we could tweak the httpd log file formats. (What do you see wrong with ours?) > > multi-root (gcc/sware/ecos/+) <same> > > faq-o-matic mediawiki > > moinmoin mediawiki > > cvsweb viewvc > > I thought we were already using viewvc. Yet /cgi-bin/cvsweb.cgi is still installed. > > gitweb <same> > > mnogosearch <same> > > bugzilla <same> > > retired services <html snapshot preserved> > > content cvs git > > "content cvs"? Does that mean replace automatic updating of web pages > with git? As far as possible, yeah; or even make the htdocs hierarchies directly into git repositories. > [...] > > cron jobs <same> > > Maybe we could retire the httpd cron jobs. Does anyone besides me read > those weekly summaries? [...] I glance at them. - FChE ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-08 12:23 ` Frank Ch. Eigler @ 2012-09-08 13:19 ` Joseph S. Myers 2012-10-28 22:44 ` Gerald Pfeifer 1 sibling, 0 replies; 23+ messages in thread From: Joseph S. Myers @ 2012-09-08 13:19 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: Sourceware Overseers On Sat, 8 Sep 2012, Frank Ch. Eigler wrote: > > mailman's built-in archiver is pretty hopeless at URL stability (it may > > well also lack other features such as the linking to a raw text version of > > each message that we've put into our mhonarc configuration). I advise > > continuing to do web archives through much the same mhonarc configuration > > as at present [...] > > I haven't heard of any concerns about mailman URL stability before. > The installations I know of (@redhat.com, @fedorahosted.org) seem to > be fine. There's a long discussion starting at <http://lists.wikimedia.org/pipermail/wikitech-l/2012-August/062364.html> (at present, that URL itself may of course not be stable). In essence, any sort of archive regeneration is liable to break URLs throughout the entire period of the archives (whereas with mhonarc archives, if a regeneration is needed then we've left the old files alone and just regenerated the relevant month into a new directory such as 2012-09n, and other months are completely untouched). > I don't know. Reading more about it now, it's not as full-featured as > I thought it was. Note that our mhonarc installation has been a > source of problems too, e.g. recurrent XSS bugs in the get-raw-message > CGIs. I'd suggest ameliorating such XSS issues through using separate domains for more services (with of course redirects for existing URLs) so that the list archives are not same-origin as Bugzilla, for example. I think a separate domain for Bugzilla attachments is now considered a good idea as well. This is of course something that can be done incrementally after the move. get-raw-message is extremely useful functionality when you wish to extract a patch from the archives, and I'm not aware of mailman having such functionality. -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-08 12:23 ` Frank Ch. Eigler 2012-09-08 13:19 ` Joseph S. Myers @ 2012-10-28 22:44 ` Gerald Pfeifer 2012-11-03 17:31 ` Daniel Berlin 1 sibling, 1 reply; 23+ messages in thread From: Gerald Pfeifer @ 2012-10-28 22:44 UTC (permalink / raw) To: overseers On Sat, 8 Sep 2012, Frank Ch. Eigler wrote: > On Fri, Sep 07, 2012 at 10:47:24PM +0000, Joseph S. Myers wrote: >>> ezmlm mailman >>> mhonarc mailman >> mailman's built-in archiver is pretty hopeless at URL stability (it may >> well also lack other features such as the linking to a raw text version of >> each message that we've put into our mhonarc configuration). I advise >> continuing to do web archives through much the same mhonarc configuration >> as at present [...] > I haven't heard of any concerns about mailman URL stability before. I have not heard either way, but from the perspective of continuity it would be nice to keep using mhonarc. Is this a big issue in terms of maintenance/security? (I know about the XSS bugs we had to address, so I understand there is some.) >>> content cvs git >> "content cvs"? Does that mean replace automatic updating of web pages >> with git? > As far as possible, yeah; or even make the htdocs hierarchies directly > into git repositories. I strongly suggest we use the some source control tool that the respective projects use for the web pages as well. When GCC was migrated from CVS to SVN, Daniel did not have the time/motivation to also migrate the web pages and these are still stuck with CVS which is just not helpful. I always wanted to do this myself as time permits to learn what is necessary for such a conversion and see this as the next step. Can't be rocket science. ;-) In fact, if any of you knows how to bring the GCC wwwdocs CVS module into GCC SVN _once_, I'd be happy to take it from there. That is really the part I do not feel comfortable with. The rest, I am sure I can pull off. :-) Gerald ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-10-28 22:44 ` Gerald Pfeifer @ 2012-11-03 17:31 ` Daniel Berlin 0 siblings, 0 replies; 23+ messages in thread From: Daniel Berlin @ 2012-11-03 17:31 UTC (permalink / raw) To: Gerald Pfeifer; +Cc: overseers On Sun, Oct 28, 2012 at 6:39 PM, Gerald Pfeifer <gerald@pfeifer.com> wrote: > On Sat, 8 Sep 2012, Frank Ch. Eigler wrote: >> On Fri, Sep 07, 2012 at 10:47:24PM +0000, Joseph S. Myers wrote: >>>> ezmlm mailman >>>> mhonarc mailman >>> mailman's built-in archiver is pretty hopeless at URL stability (it may >>> well also lack other features such as the linking to a raw text version of >>> each message that we've put into our mhonarc configuration). I advise >>> continuing to do web archives through much the same mhonarc configuration >>> as at present [...] >> I haven't heard of any concerns about mailman URL stability before. > > I have not heard either way, but from the perspective of continuity > it would be nice to keep using mhonarc. Is this a big issue in terms > of maintenance/security? (I know about the XSS bugs we had to address, > so I understand there is some.) > >>>> content cvs git >>> "content cvs"? Does that mean replace automatic updating of web pages >>> with git? >> As far as possible, yeah; or even make the htdocs hierarchies directly >> into git repositories. > > I strongly suggest we use the some source control tool that the > respective projects use for the web pages as well. > > When GCC was migrated from CVS to SVN, Daniel did not have the > time/motivation to also migrate the web pages and these are still > stuck with CVS which is just not helpful. I always wanted to do > this myself as time permits to learn what is necessary for such > a conversion and see this as the next step. > > Can't be rocket science. ;-) In fact, if any of you knows how to > bring the GCC wwwdocs CVS module into GCC SVN _once_, I'd be happy > to take it from there. That is really the part I do not feel > comfortable with. The rest, I am sure I can pull off. :-) Nowadays, cvs2svn or something similar should "just work" for this purpose. Heck, you could probably just use svnsync. > > Gerald ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler 2012-09-07 22:47 ` Joseph S. Myers @ 2012-09-07 23:55 ` Jonathan Larmour 2012-09-10 17:32 ` Angela Marie Thomas 2012-09-07 23:59 ` Christopher Faylor 2 siblings, 1 reply; 23+ messages in thread From: Jonathan Larmour @ 2012-09-07 23:55 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: Sourceware Overseers On 07/09/12 22:46, Frank Ch. Eigler wrote: > > As a first step, here is a draft of the overall "before & after". > Please let me know what parts may be incomplete or controversial. > After this is settled, I'll put forward a task list as to how to get > there. The main thing I want to comment on is that this should be as phased a changeover as possible. I suspect that iptables DNAT using --to-destination (or some upstream router equivalent) will be invaluable in order to transition carefully and mostly reversibly if needed. Care is also needed given the level of coordination with individual projects which will be required. Most obviously for mail and mail archives, but also the wikis. Projects need to not just be given advanced warning, but actually say when they think they will be ready for switchovers (with new web pages written, users updated, etc.), and assuming the timescales are within reason, schedule things around that. > old: new: > > hardware: > server[1]+server[23] server[45] > no redundancy warm standby, ~identical servers > dell 2850 ibm x3650 M3 > 500GB raid5 2TB raid6 > 16GB RAM 24GB RAM Given the change from i686 to x86-64, as well as moving to much more memory-hungry services (bind, mailman, etc.), will this RAM increment be enough? At the moment 12GB of that 16GB is used for page cache, but we are probably quite reliant on that. Essentially, will the increase from the new services and x86-64 be greater than 8GB? Personally, I think it will slightly reduce the available page cache, but not enough to be a problem; and I assume there is capacity for expansion. I thought I should ask the question though. > mail: > qmail sendmail I'm a bit surprised about wanting to use sendmail. It's track record has been so poor; obviously it's been bodged and bodged again so there are no vulnerabilities at the moment, but that's not the same as designed-in security. And as for its configuration, M4 only gets you so far and I'm happy to let the neurons in my brain that remember sendmail configuration wither and die. I would have thought postfix or exim more appropriate on a system where both security and performance are vital. Allegedly postfix has a better security record than exim, but I'm not intending to start an advocacy thread about which is best; just non-advocacy of sendmail :-). One thing missing off your list is backups. The problem with warm standby servers is that they allow problems to be replicated quickly and efficiently :-). Obviously I'm more concerned about system or human mess-ups than hardware failure here. I assume Angela is still doing backups, and if so, either transition will need to be co-ordinated with her, or some other means of incremental backup provided. During the transition period, both old and new systems will want to be backed up. Jifl ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-07 23:55 ` Jonathan Larmour @ 2012-09-10 17:32 ` Angela Marie Thomas 2012-09-10 17:57 ` Frank Ch. Eigler 0 siblings, 1 reply; 23+ messages in thread From: Angela Marie Thomas @ 2012-09-10 17:32 UTC (permalink / raw) To: Jonathan Larmour; +Cc: Frank Ch. Eigler, Sourceware Overseers On Sep 7, 2012, at 4:55 PM, Jonathan Larmour <jifl@jifvik.org> wrote: > One thing missing off your list is backups. The problem with warm standby > servers is that they allow problems to be replicated quickly and efficiently > :-). Obviously I'm more concerned about system or human mess-ups than hardware > failure here. I assume Angela is still doing backups, and if so, either > transition will need to be co-ordinated with her, or some other means of > incremental backup provided. During the transition period, both old and new > systems will want to be backed up. I still do full system backups daily except for htdig and some temporary data. I assume offsite backups are still desired just to be on the safe side. I agree if there's a phased transition both will need to be backed up until the old server is fully decommissioned. Is the hardware already racked with console access? If not, when will that happen? Knowing the earliest we could start helps with planning backup strategies and capacity increases. --Angela ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-10 17:32 ` Angela Marie Thomas @ 2012-09-10 17:57 ` Frank Ch. Eigler 2012-09-10 19:01 ` Christopher Faylor 0 siblings, 1 reply; 23+ messages in thread From: Frank Ch. Eigler @ 2012-09-10 17:57 UTC (permalink / raw) To: Angela Marie Thomas; +Cc: Jonathan Larmour, Sourceware Overseers Hi, Angela - > [...] Is the hardware already racked with console access? [...] Yes and no. The /root/.ssh/authorized_keys have been copied over so y'all may *look around* via ssh root@server[45].sourceware.org. The virtual console is on the RH internal network. - FChE ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-10 17:57 ` Frank Ch. Eigler @ 2012-09-10 19:01 ` Christopher Faylor 2012-09-10 20:18 ` Angela Marie Thomas 0 siblings, 1 reply; 23+ messages in thread From: Christopher Faylor @ 2012-09-10 19:01 UTC (permalink / raw) To: Frank Ch. Eigler, Sourceware Overseers, Angela Marie Thomas, Jonathan Larmour On Mon, Sep 10, 2012 at 01:56:59PM -0400, Frank Ch. Eigler wrote: >Hi, Angela - > >> [...] Is the hardware already racked with console access? [...] > >Yes and no. The /root/.ssh/authorized_keys have been copied over so >y'all may *look around* via ssh root@server[45].sourceware.org. The >virtual console is on the RH internal network. But that's nothing new... Should we start rsyncing data over? It seems like we'll want a full backup of old sourceware there eventually. cgf ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-10 19:01 ` Christopher Faylor @ 2012-09-10 20:18 ` Angela Marie Thomas 2012-09-12 15:29 ` Frank Ch. Eigler 0 siblings, 1 reply; 23+ messages in thread From: Angela Marie Thomas @ 2012-09-10 20:18 UTC (permalink / raw) To: Christopher Faylor Cc: Frank Ch. Eigler, Sourceware Overseers, Jonathan Larmour On Sep 10, 2012, at 12:00 PM, Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote: > On Mon, Sep 10, 2012 at 01:56:59PM -0400, Frank Ch. Eigler wrote: >> Hi, Angela - >> >>> [...] Is the hardware already racked with console access? [...] >> >> Yes and no. The /root/.ssh/authorized_keys have been copied over so >> y'all may *look around* via ssh root@server[45].sourceware.org. The >> virtual console is on the RH internal network. > > But that's nothing new... Indeed. My query was more along the lines of figuring out timelines. If the partitioning is as desired and the base OS is installed, that's a significant milestone. I see that server[45] are configured with a single data partition /srv. Is it likely that it will be a mirror of sourceware's current /export/u0 or do folks anticipate some changes in the directory layout? > Should we start rsyncing data over? It seems like we'll want a full > backup of old sourceware there eventually. Given that the new servers have considerably more capacity, it makes sense to store a complete snapshot of current sourceware there. It's stating the obvious, but we'd want to avoid anything that changes a lot as that tends to make the final copy take more time rather than less time. That's one reason I don't backup htdig. Regenerating it is easier than backing it up. --Angela ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-10 20:18 ` Angela Marie Thomas @ 2012-09-12 15:29 ` Frank Ch. Eigler 2012-09-12 15:37 ` Christopher Faylor 0 siblings, 1 reply; 23+ messages in thread From: Frank Ch. Eigler @ 2012-09-12 15:29 UTC (permalink / raw) To: Angela Marie Thomas; +Cc: Sourceware Overseers Hi - > Indeed. My query was more along the lines of figuring out timelines. > If the partitioning is as desired and the base OS is installed, that's a > significant milestone. > I see that server[45] are configured with a single data partition /srv. Is > it likely that it will be a mirror of sourceware's current /export/u0 or do > folks anticipate some changes in the directory layout? I've removed the large /srv and created three smaller equal /sourceware[123] partitions. The idea is that one of these would be a temporary sourceware rsync destination; one would be the real new-sourceware filesystem like /export/u0 was; and one would be a local backup (normally unmounted). Once the transition is complete, two of them could be local backups (of different generations, say monthly and weekly local rsync's). > Given that the new servers have considerably more capacity, it makes > sense to store a complete snapshot of current sourceware there. > [...] Right. One concern though is to avoid cross-authentication on the machines as much as possible (i.e., storage of general-use ssh private keys or equivalent), so as to prevent one from being used to break into another some day. - FChE ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 15:29 ` Frank Ch. Eigler @ 2012-09-12 15:37 ` Christopher Faylor 2012-09-12 15:51 ` Christopher Faylor ` (2 more replies) 0 siblings, 3 replies; 23+ messages in thread From: Christopher Faylor @ 2012-09-12 15:37 UTC (permalink / raw) To: Frank Ch. Eigler, Sourceware Overseers, Angela Marie Thomas On Wed, Sep 12, 2012 at 11:28:59AM -0400, Frank Ch. Eigler wrote: >Hi - > >> Indeed. My query was more along the lines of figuring out timelines. >> If the partitioning is as desired and the base OS is installed, that's a >> significant milestone. > >> I see that server[45] are configured with a single data partition /srv. Is >> it likely that it will be a mirror of sourceware's current /export/u0 or do >> folks anticipate some changes in the directory layout? > >I've removed the large /srv and created three smaller equal >/sourceware[123] partitions. The idea is that one of these would be a >temporary sourceware rsync destination; one would be the real >new-sourceware filesystem like /export/u0 was; and one would be a >local backup (normally unmounted). Once the transition is complete, >two of them could be local backups (of different generations, say >monthly and weekly local rsync's). Ok, how do we get started? I can take some time off from work to focus on this if needed. We have days available for working on stuff like this. Should we rsync stuff over and then start working on a ezmlm->mailman translation? We'll also likely need to do a lot of work to keep URLs correct. Do we REALLY want to move from ezmlm? Isn't there an rpm-based package out there somewhere that we could use? cgf ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 15:37 ` Christopher Faylor @ 2012-09-12 15:51 ` Christopher Faylor 2012-09-12 15:59 ` Joseph S. Myers 2012-09-12 16:09 ` Frank Ch. Eigler 2 siblings, 0 replies; 23+ messages in thread From: Christopher Faylor @ 2012-09-12 15:51 UTC (permalink / raw) To: Frank Ch. Eigler, Sourceware Overseers, Angela Marie Thomas On Wed, Sep 12, 2012 at 11:37:24AM -0400, Christopher Faylor wrote: >Do we REALLY want to move from ezmlm? Isn't there an rpm-based package >out there somewhere that we could use? Sorry. That's just crazy talk. There is no officially supported ezmlm package. Nevermind. cgf ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 15:37 ` Christopher Faylor 2012-09-12 15:51 ` Christopher Faylor @ 2012-09-12 15:59 ` Joseph S. Myers 2012-09-12 16:20 ` Jeff Law 2012-09-12 16:09 ` Frank Ch. Eigler 2 siblings, 1 reply; 23+ messages in thread From: Joseph S. Myers @ 2012-09-12 15:59 UTC (permalink / raw) To: Sourceware Overseers; +Cc: Frank Ch. Eigler, Angela Marie Thomas On Wed, 12 Sep 2012, Christopher Faylor wrote: > Do we REALLY want to move from ezmlm? I don't see any strong need to move from ezmlm (and qmail, which is pretty closely tied to it); it may not have a pretty web interface, but it does work fine for what's done with it on sourceware, and I think ezmlm and qmail generally have a stronger security record than many more modern packages for the same purposes. I'd also think that each service for which we change the software while changing the hardware adds an extra complication to the hardware move, and so an extra delay. (I would however certainly be glad for the wikis to move to MediaWiki; the existing wiki software has had various problems including accidental deletion of attachments by recursive wget with no good way to undelete them. But does such a move actually need to be combined with the hardware move and OS upgrade, or could it just as well be done separately once the new hardware and OS are in used with the existing choice of software?) -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 15:59 ` Joseph S. Myers @ 2012-09-12 16:20 ` Jeff Law 2012-09-12 16:29 ` Joseph S. Myers 0 siblings, 1 reply; 23+ messages in thread From: Jeff Law @ 2012-09-12 16:20 UTC (permalink / raw) To: Joseph S. Myers Cc: Sourceware Overseers, Frank Ch. Eigler, Angela Marie Thomas On 09/12/2012 09:59 AM, Joseph S. Myers wrote: >> Do we REALLY want to move from ezmlm? > > I don't see any strong need to move from ezmlm (and qmail, which is pretty > closely tied to it); ezmlm is definitely tied to qmail. IIRC the primary reasons for qmail/ezmlm were performance, security, good archiving and allowing users to self-help as much as possible. I'd think anything which does those things well would be sufficient; there's certainly some benefit in using more standardized components which would need to be balanced against the conversion cost. Personally I'd tend to look to get the hardware move done first, then start replacing software components as desired. But I haven't done any admin work on sourceware in 10+ years, so I'll leave the final decisions to those actually doing the work. jeff ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 16:20 ` Jeff Law @ 2012-09-12 16:29 ` Joseph S. Myers 0 siblings, 0 replies; 23+ messages in thread From: Joseph S. Myers @ 2012-09-12 16:29 UTC (permalink / raw) To: Jeff Law; +Cc: Sourceware Overseers, Frank Ch. Eigler, Angela Marie Thomas On Wed, 12 Sep 2012, Jeff Law wrote: > Personally I'd tend to look to get the hardware move done first, then start > replacing software components as desired. But I haven't done any admin work That makes sense to me as well. The old hardware is a problem - the present system tends to be heavily overloaded. The old OS on the old hardware is a problem - various packages are too old to work well with current versions of some non-OS software such as Bugzilla. For the most part, the local choices of software components to use for various purposes are not problems in the same way. -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 15:37 ` Christopher Faylor 2012-09-12 15:51 ` Christopher Faylor 2012-09-12 15:59 ` Joseph S. Myers @ 2012-09-12 16:09 ` Frank Ch. Eigler 2012-09-12 17:09 ` Christopher Faylor 2 siblings, 1 reply; 23+ messages in thread From: Frank Ch. Eigler @ 2012-09-12 16:09 UTC (permalink / raw) To: Sourceware Overseers Hi - On Wed, Sep 12, 2012 at 11:37:24AM -0400, Christopher Faylor wrote: > [...] > Ok, how do we get started? I can take some time off from work to focus > on this if needed. We have days available for working on stuff like > this. I think the key will be mechanizing & logging our work, so that a trial migration can be done on our own time, with a later refresh at the actual cutover. So, first step would seem to be a rerunnable rsync from server1:/ ->server4:/sourceware3, and (to start exercising warm-backup capabilities) from server4:/sourceware[123]->server5:/sourceware[123]. I suggest rsyncing the whole machine (not just /export/u0) to capture /etc and /usr/local and the databases and everything. Chris, want to give those a try? Next step could be creation of a script on server4 that takes /sourceware3 and populates the migrated forms of /sourceware1 and various symlinks, etc. on the server, to the extent easily automated. I'd exclude things like iptables, passwd, httpd configs, which I imagine we'll have to hand-migrate. > Should we rsync stuff over and then start working on a ezmlm->mailman > translation? We'll also likely need to do a lot of work to keep URLs > correct. (My original thought was that the existing mhonarc archives would be preserved, so existing URLs would stay valid; just new archives would go to a new http url hierarchy.) > Do we REALLY want to move from ezmlm? Isn't there an rpm-based package > out there somewhere that we could use? Well, dunno. ezmlm is not the worst thing about old sourceware; if others are content, we could keep it awhile longer. The mediawiki transition could also be done later, on-line. - FChE ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 16:09 ` Frank Ch. Eigler @ 2012-09-12 17:09 ` Christopher Faylor 2012-09-12 20:48 ` Joseph S. Myers 0 siblings, 1 reply; 23+ messages in thread From: Christopher Faylor @ 2012-09-12 17:09 UTC (permalink / raw) To: Frank Ch. Eigler, Sourceware Overseers On Wed, Sep 12, 2012 at 12:09:35PM -0400, Frank Ch. Eigler wrote: >> Should we rsync stuff over and then start working on a ezmlm->mailman >> translation? We'll also likely need to do a lot of work to keep URLs >> correct. > >(My original thought was that the existing mhonarc archives would be >preserved, so existing URLs would stay valid; just new archives would >go to a new http url hierarchy.) Ah, that would be easier. So we just have an "old" archive which preserves the old ml and a "new" one. So, would it make sense to populate the new archive with old content too just for consistency? We'd have duplicate content but it seems like it would be less confusing to end users. >> Do we REALLY want to move from ezmlm? Isn't there an rpm-based package >> out there somewhere that we could use? > >Well, dunno. ezmlm is not the worst thing about old sourceware; if >others are content, we could keep it awhile longer. The mediawiki >transition could also be done later, on-line. I chatted with Ian about this on irc. The problem with ezmlm and qmail is that they are not maintained and, so, are unaware of new standards like DKIM. If we think we will have to at some point then I think we should pull the band-aid off now. Or, alternatively, commit to supporting both packages ourselves. cgf ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 17:09 ` Christopher Faylor @ 2012-09-12 20:48 ` Joseph S. Myers 2012-09-13 3:18 ` Christopher Faylor 0 siblings, 1 reply; 23+ messages in thread From: Joseph S. Myers @ 2012-09-12 20:48 UTC (permalink / raw) To: Sourceware Overseers; +Cc: Frank Ch. Eigler On Wed, 12 Sep 2012, Christopher Faylor wrote: > On Wed, Sep 12, 2012 at 12:09:35PM -0400, Frank Ch. Eigler wrote: > >> Should we rsync stuff over and then start working on a ezmlm->mailman > >> translation? We'll also likely need to do a lot of work to keep URLs > >> correct. > > > >(My original thought was that the existing mhonarc archives would be > >preserved, so existing URLs would stay valid; just new archives would > >go to a new http url hierarchy.) > > Ah, that would be easier. So we just have an "old" archive which > preserves the old ml and a "new" one. > > So, would it make sense to populate the new archive with old content too > just for consistency? We'd have duplicate content but it seems like it > would be less confusing to end users. It would be less confusing, avoid URL instability if we ever again in future wish to remove a message from the archives and preserve useful features such as raw versions of messages just to keep using mhonarc with the same URL structure as now, and make any mailman/pipermail URLs redirect to the /ml/ mhonarc ones so noone ever accesses a message via them. As far as I know mhonarc is maintained. > I chatted with Ian about this on irc. The problem with ezmlm and qmail > is that they are not maintained and, so, are unaware of new standards > like DKIM. Does ezmlm actually do any of the sort of changes to messages that would break DKIM? -- Joseph S. Myers joseph@codesourcery.com ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-12 20:48 ` Joseph S. Myers @ 2012-09-13 3:18 ` Christopher Faylor 2012-09-13 3:59 ` Ian Lance Taylor 0 siblings, 1 reply; 23+ messages in thread From: Christopher Faylor @ 2012-09-13 3:18 UTC (permalink / raw) To: Frank Ch. Eigler, Sourceware Overseers, Joseph S. Myers On Wed, Sep 12, 2012 at 08:48:22PM +0000, Joseph S. Myers wrote: >On Wed, 12 Sep 2012, Christopher Faylor wrote: > >> On Wed, Sep 12, 2012 at 12:09:35PM -0400, Frank Ch. Eigler wrote: >> >> Should we rsync stuff over and then start working on a ezmlm->mailman >> >> translation? We'll also likely need to do a lot of work to keep URLs >> >> correct. >> > >> >(My original thought was that the existing mhonarc archives would be >> >preserved, so existing URLs would stay valid; just new archives would >> >go to a new http url hierarchy.) >> >> Ah, that would be easier. So we just have an "old" archive which >> preserves the old ml and a "new" one. >> >> So, would it make sense to populate the new archive with old content too >> just for consistency? We'd have duplicate content but it seems like it >> would be less confusing to end users. > >It would be less confusing, avoid URL instability if we ever again in >future wish to remove a message from the archives and preserve useful >features such as raw versions of messages just to keep using mhonarc with >the same URL structure as now, and make any mailman/pipermail URLs >redirect to the /ml/ mhonarc ones so noone ever accesses a message via >them. As far as I know mhonarc is maintained. > >> I chatted with Ian about this on irc. The problem with ezmlm and qmail >> is that they are not maintained and, so, are unaware of new standards >> like DKIM. > >Does ezmlm actually do any of the sort of changes to messages that would >break DKIM? Ian had to add patches to qmail (or maybe it was ezmlm) for DKIM recently because mail from sourceware was being treated as spam. So that's actually one of the examples of what it means for these tools to be unsupported. cgf ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-13 3:18 ` Christopher Faylor @ 2012-09-13 3:59 ` Ian Lance Taylor 0 siblings, 0 replies; 23+ messages in thread From: Ian Lance Taylor @ 2012-09-13 3:59 UTC (permalink / raw) To: Sourceware Overseers Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> writes: >>Does ezmlm actually do any of the sort of changes to messages that would >>break DKIM? > > Ian had to add patches to qmail (or maybe it was ezmlm) for DKIM > recently because mail from sourceware was being treated as spam. So > that's actually one of the examples of what it means for these tools to > be unsupported. Right, I had to change qmail to make it add DKIM signatures to outgoing e-mail. You can see the signature in the headers of this very message. In fact we add both a DKIM and a DomainKey signature. If we stick with qmail, then in the future we will have to patch it to add IPv6 support. In general I like qmail and ezmlm quite a bit, but I'm not aware of any coherent support for them. Ian ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N 2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler 2012-09-07 22:47 ` Joseph S. Myers 2012-09-07 23:55 ` Jonathan Larmour @ 2012-09-07 23:59 ` Christopher Faylor 2 siblings, 0 replies; 23+ messages in thread From: Christopher Faylor @ 2012-09-07 23:59 UTC (permalink / raw) To: Frank Ch. Eigler, Sourceware Overseers On Fri, Sep 07, 2012 at 05:46:32PM -0400, Frank Ch. Eigler wrote: >Hi - > >As many of you know, two IBM x3650 servers have been donated by Red >Hat to the sourceware project. They've been waiting for our time to >configure them and migrate the content from the old machines over. > >The migration will be non-trivial. We're moving from RHEL4->RHEL6. I >would like to move from unusual/obsolete services to mainstream ones >if at all possible, for simplicity of administration, and to reduce >exposure to vulnerabilities. I would like to retire several old >services (leaving behind a read-only web mirror if appropriate). > >As a first step, here is a draft of the overall "before & after". >Please let me know what parts may be incomplete or controversial. >After this is settled, I'll put forward a task list as to how to get >there. > > old: new: > >hardware: > server[1]+server[23] server[45] > no redundancy warm standby, ~identical servers > dell 2850 ibm x3650 M3 > 500GB raid5 2TB raid6 > 16GB RAM 24GB RAM > 4cpu*Xeon 3GHz 2cpu*4core*2thread 2.4GHz Xeon E5620 >OS: > rhel4 i686 rhel6 x6-64 > iptables <same> >vcs: > svn <same> > git <same> > cvs <same> > cvsupd <retire> >misc: > djbdns bind > proftpd <same> > ntpd <same> > rsync <same> > mysql <same> > gnats <retire> > rssh <same> >mail: > qmail sendmail > ezmlm mailman > mhonarc mailman I am probably stepping into a religious debate but wouldn't it make more sense to use postfix here? I haven't maintained a sendmail system in years but I am familiar with postfix. (And as strange as qmail is, I have had a lot more trouble tweaking sendmail and postfix than I ever did with qmail) I have maintained a mailman system before but getting things like the extra spam blocking that is currently plugged into ezmlm will probably take some time. You missed qpsmtpd, btw. I don't know if that can be made to work with the above or not. > spamassassin <same>, focus on inbound mail "inbound mail"? As opposed to...? > clamav <same> > mlcheckd <retire> See above. mlcheckd is how we block the disclaimer email. And, it works as a secondary line of spam defense for when spamassassin fails. I also use it to block off-topic email in the cygwin list. I'm sure that could still be done with mailman but it will take research. >httpd: > httpd 2.0 httpd 2.2 + https Maybe we could tweak the httpd log file formats. > multi-root (gcc/sware/ecos/+) <same> > faq-o-matic mediawiki > moinmoin mediawiki > cvsweb viewvc I thought we were already using viewvc. > gitweb <same> > mnogosearch <same> > bugzilla <same> > retired services <html snapshot preserved> > content cvs git "content cvs"? Does that mean replace automatic updating of web pages with git? >projects: > root users <same> > project files <same> > users <same> > cron jobs <same> Maybe we could retire the httpd cron jobs. Does anyone besides me read those weekly summaries? > cygwin upset <same> That's a given if cron jobs are the same. cgf ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2012-10-28 23:45 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler 2012-09-07 22:47 ` Joseph S. Myers 2012-09-08 12:23 ` Frank Ch. Eigler 2012-09-08 13:19 ` Joseph S. Myers 2012-10-28 22:44 ` Gerald Pfeifer 2012-11-03 17:31 ` Daniel Berlin 2012-09-07 23:55 ` Jonathan Larmour 2012-09-10 17:32 ` Angela Marie Thomas 2012-09-10 17:57 ` Frank Ch. Eigler 2012-09-10 19:01 ` Christopher Faylor 2012-09-10 20:18 ` Angela Marie Thomas 2012-09-12 15:29 ` Frank Ch. Eigler 2012-09-12 15:37 ` Christopher Faylor 2012-09-12 15:51 ` Christopher Faylor 2012-09-12 15:59 ` Joseph S. Myers 2012-09-12 16:20 ` Jeff Law 2012-09-12 16:29 ` Joseph S. Myers 2012-09-12 16:09 ` Frank Ch. Eigler 2012-09-12 17:09 ` Christopher Faylor 2012-09-12 20:48 ` Joseph S. Myers 2012-09-13 3:18 ` Christopher Faylor 2012-09-13 3:59 ` Ian Lance Taylor 2012-09-07 23:59 ` Christopher Faylor
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).