* sourceware hardware transition plan, part 1/N
@ 2012-09-07 21:47 Frank Ch. Eigler
2012-09-07 22:47 ` Joseph S. Myers
` (2 more replies)
0 siblings, 3 replies; 23+ messages in thread
From: Frank Ch. Eigler @ 2012-09-07 21:47 UTC (permalink / raw)
To: Sourceware Overseers
Hi -
As many of you know, two IBM x3650 servers have been donated by Red
Hat to the sourceware project. They've been waiting for our time to
configure them and migrate the content from the old machines over.
The migration will be non-trivial. We're moving from RHEL4->RHEL6. I
would like to move from unusual/obsolete services to mainstream ones
if at all possible, for simplicity of administration, and to reduce
exposure to vulnerabilities. I would like to retire several old
services (leaving behind a read-only web mirror if appropriate).
As a first step, here is a draft of the overall "before & after".
Please let me know what parts may be incomplete or controversial.
After this is settled, I'll put forward a task list as to how to get
there.
old: new:
hardware:
server[1]+server[23] server[45]
no redundancy warm standby, ~identical servers
dell 2850 ibm x3650 M3
500GB raid5 2TB raid6
16GB RAM 24GB RAM
4cpu*Xeon 3GHz 2cpu*4core*2thread 2.4GHz Xeon E5620
OS:
rhel4 i686 rhel6 x6-64
iptables <same>
vcs:
svn <same>
git <same>
cvs <same>
cvsupd <retire>
misc:
djbdns bind
proftpd <same>
ntpd <same>
rsync <same>
mysql <same>
gnats <retire>
rssh <same>
mail:
qmail sendmail
ezmlm mailman
mhonarc mailman
spamassassin <same>, focus on inbound mail
clamav <same>
mlcheckd <retire>
httpd:
httpd 2.0 httpd 2.2 + https
multi-root (gcc/sware/ecos/+) <same>
faq-o-matic mediawiki
moinmoin mediawiki
cvsweb viewvc
gitweb <same>
mnogosearch <same>
bugzilla <same>
retired services <html snapshot preserved>
content cvs git
projects:
root users <same>
project files <same>
users <same>
cron jobs <same>
gcc, sourceware, infra <same>
gcc supybot <same>
cygwin upset <same>
- FChE
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler
@ 2012-09-07 22:47 ` Joseph S. Myers
2012-09-08 12:23 ` Frank Ch. Eigler
2012-09-07 23:55 ` Jonathan Larmour
2012-09-07 23:59 ` Christopher Faylor
2 siblings, 1 reply; 23+ messages in thread
From: Joseph S. Myers @ 2012-09-07 22:47 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: Sourceware Overseers
On Fri, 7 Sep 2012, Frank Ch. Eigler wrote:
> ezmlm mailman
> mhonarc mailman
mailman's built-in archiver is pretty hopeless at URL stability (it may
well also lack other features such as the linking to a raw text version of
each message that we've put into our mhonarc configuration). I advise
continuing to do web archives through much the same mhonarc configuration
as at present and hiding any built-in mailman archives as far as possible
to avoid people posting references to their unstable URLs.
Is it possible to get mailman to assign unique sequence numbers to each
message to a list like ezmlm does (whether in the envelope return address
or in a header) so you can readily tell what order messages were supposed
to be in and whether there are any gaps in the messages you have received,
or that appear in the list archives? That's the feature of ezmlm I find
most useful (other than that, mailman certainly has various advantages).
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler
2012-09-07 22:47 ` Joseph S. Myers
@ 2012-09-07 23:55 ` Jonathan Larmour
2012-09-10 17:32 ` Angela Marie Thomas
2012-09-07 23:59 ` Christopher Faylor
2 siblings, 1 reply; 23+ messages in thread
From: Jonathan Larmour @ 2012-09-07 23:55 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: Sourceware Overseers
On 07/09/12 22:46, Frank Ch. Eigler wrote:
>
> As a first step, here is a draft of the overall "before & after".
> Please let me know what parts may be incomplete or controversial.
> After this is settled, I'll put forward a task list as to how to get
> there.
The main thing I want to comment on is that this should be as phased a
changeover as possible. I suspect that iptables DNAT using --to-destination
(or some upstream router equivalent) will be invaluable in order to transition
carefully and mostly reversibly if needed.
Care is also needed given the level of coordination with individual projects
which will be required. Most obviously for mail and mail archives, but also
the wikis. Projects need to not just be given advanced warning, but actually
say when they think they will be ready for switchovers (with new web pages
written, users updated, etc.), and assuming the timescales are within reason,
schedule things around that.
> old: new:
>
> hardware:
> server[1]+server[23] server[45]
> no redundancy warm standby, ~identical servers
> dell 2850 ibm x3650 M3
> 500GB raid5 2TB raid6
> 16GB RAM 24GB RAM
Given the change from i686 to x86-64, as well as moving to much more
memory-hungry services (bind, mailman, etc.), will this RAM increment be
enough? At the moment 12GB of that 16GB is used for page cache, but we are
probably quite reliant on that. Essentially, will the increase from the new
services and x86-64 be greater than 8GB?
Personally, I think it will slightly reduce the available page cache, but not
enough to be a problem; and I assume there is capacity for expansion. I
thought I should ask the question though.
> mail:
> qmail sendmail
I'm a bit surprised about wanting to use sendmail. It's track record has been
so poor; obviously it's been bodged and bodged again so there are no
vulnerabilities at the moment, but that's not the same as designed-in
security. And as for its configuration, M4 only gets you so far and I'm happy
to let the neurons in my brain that remember sendmail configuration wither and
die.
I would have thought postfix or exim more appropriate on a system where both
security and performance are vital. Allegedly postfix has a better security
record than exim, but I'm not intending to start an advocacy thread about
which is best; just non-advocacy of sendmail :-).
One thing missing off your list is backups. The problem with warm standby
servers is that they allow problems to be replicated quickly and efficiently
:-). Obviously I'm more concerned about system or human mess-ups than hardware
failure here. I assume Angela is still doing backups, and if so, either
transition will need to be co-ordinated with her, or some other means of
incremental backup provided. During the transition period, both old and new
systems will want to be backed up.
Jifl
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler
2012-09-07 22:47 ` Joseph S. Myers
2012-09-07 23:55 ` Jonathan Larmour
@ 2012-09-07 23:59 ` Christopher Faylor
2 siblings, 0 replies; 23+ messages in thread
From: Christopher Faylor @ 2012-09-07 23:59 UTC (permalink / raw)
To: Frank Ch. Eigler, Sourceware Overseers
On Fri, Sep 07, 2012 at 05:46:32PM -0400, Frank Ch. Eigler wrote:
>Hi -
>
>As many of you know, two IBM x3650 servers have been donated by Red
>Hat to the sourceware project. They've been waiting for our time to
>configure them and migrate the content from the old machines over.
>
>The migration will be non-trivial. We're moving from RHEL4->RHEL6. I
>would like to move from unusual/obsolete services to mainstream ones
>if at all possible, for simplicity of administration, and to reduce
>exposure to vulnerabilities. I would like to retire several old
>services (leaving behind a read-only web mirror if appropriate).
>
>As a first step, here is a draft of the overall "before & after".
>Please let me know what parts may be incomplete or controversial.
>After this is settled, I'll put forward a task list as to how to get
>there.
>
> old: new:
>
>hardware:
> server[1]+server[23] server[45]
> no redundancy warm standby, ~identical servers
> dell 2850 ibm x3650 M3
> 500GB raid5 2TB raid6
> 16GB RAM 24GB RAM
> 4cpu*Xeon 3GHz 2cpu*4core*2thread 2.4GHz Xeon E5620
>OS:
> rhel4 i686 rhel6 x6-64
> iptables <same>
>vcs:
> svn <same>
> git <same>
> cvs <same>
> cvsupd <retire>
>misc:
> djbdns bind
> proftpd <same>
> ntpd <same>
> rsync <same>
> mysql <same>
> gnats <retire>
> rssh <same>
>mail:
> qmail sendmail
> ezmlm mailman
> mhonarc mailman
I am probably stepping into a religious debate but wouldn't it make more
sense to use postfix here? I haven't maintained a sendmail system in
years but I am familiar with postfix.
(And as strange as qmail is, I have had a lot more trouble tweaking
sendmail and postfix than I ever did with qmail)
I have maintained a mailman system before but getting things like
the extra spam blocking that is currently plugged into ezmlm will
probably take some time.
You missed qpsmtpd, btw. I don't know if that can be made to work with
the above or not.
> spamassassin <same>, focus on inbound mail
"inbound mail"? As opposed to...?
> clamav <same>
> mlcheckd <retire>
See above. mlcheckd is how we block the disclaimer email. And, it
works as a secondary line of spam defense for when spamassassin fails.
I also use it to block off-topic email in the cygwin list.
I'm sure that could still be done with mailman but it will take research.
>httpd:
> httpd 2.0 httpd 2.2 + https
Maybe we could tweak the httpd log file formats.
> multi-root (gcc/sware/ecos/+) <same>
> faq-o-matic mediawiki
> moinmoin mediawiki
> cvsweb viewvc
I thought we were already using viewvc.
> gitweb <same>
> mnogosearch <same>
> bugzilla <same>
> retired services <html snapshot preserved>
> content cvs git
"content cvs"? Does that mean replace automatic updating of web pages
with git?
>projects:
> root users <same>
> project files <same>
> users <same>
> cron jobs <same>
Maybe we could retire the httpd cron jobs. Does anyone besides me read
those weekly summaries?
> cygwin upset <same>
That's a given if cron jobs are the same.
cgf
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-07 22:47 ` Joseph S. Myers
@ 2012-09-08 12:23 ` Frank Ch. Eigler
2012-09-08 13:19 ` Joseph S. Myers
2012-10-28 22:44 ` Gerald Pfeifer
0 siblings, 2 replies; 23+ messages in thread
From: Frank Ch. Eigler @ 2012-09-08 12:23 UTC (permalink / raw)
To: Sourceware Overseers
Hi -
On Fri, Sep 07, 2012 at 10:47:24PM +0000, Joseph S. Myers wrote:
> [...]
> > ezmlm mailman
> > mhonarc mailman
>
> mailman's built-in archiver is pretty hopeless at URL stability (it may
> well also lack other features such as the linking to a raw text version of
> each message that we've put into our mhonarc configuration). I advise
> continuing to do web archives through much the same mhonarc configuration
> as at present [...]
I haven't heard of any concerns about mailman URL stability before.
The installations I know of (@redhat.com, @fedorahosted.org) seem to
be fine.
> Is it possible to get mailman [...]
I don't know. Reading more about it now, it's not as full-featured as
I thought it was. Note that our mhonarc installation has been a
source of problems too, e.g. recurrent XSS bugs in the get-raw-message
CGIs.
On Sat, Sep 08, 2012 at 12:55:30AM +0100, Jonathan Larmour wrote:
> The main thing I want to comment on is that this should be as phased a
> changeover as possible. [...]
Given the extent to which the various sourceware services are
interdependent, this may not be possible.
> Care is also needed given the level of coordination with individual projects
> which will be required. Most obviously for mail and mail archives, but also
> the wikis. Projects need to not just be given advanced warning, but actually
> say when they think they will be ready for switchovers (with new web pages
> written, users updated, etc.), and assuming the timescales are within reason,
> schedule things around that.
It would be better if the transition were not drawn out or too
synchronized across projects. I was thinking something like a
half-day's outage, bringing the essential services online first
(version control and mailing lists), others (e.g. wikis) as the
conversion tools / rsync finish.
> [...]
> > 16GB RAM 24GB RAM
>
> Given the change from i686 to x86-64, as well as moving to much more
> memory-hungry services (bind, mailman, etc.), will this RAM increment be
> enough? [...]
Yeah, that is a little small. Luckily, the new servers have 18 DIMM
slots, 6 used each, with DDR3 PC3-10600 1333MHz ECC Registered cards,
e.g. http://www.memoryamerica.com/m393b5170fh0-ch9.html, so we can
fairly cheaply double/triple it.
> > mail:
> > qmail sendmail
>
> I'm a bit surprised about wanting to use sendmail. [...]
> I would have thought postfix or exim more appropriate [...]
I have about the same amount of familiarity with them all;
they could each the job.
> One thing missing off your list is backups. The problem with warm
> standby servers is that they allow problems to be replicated quickly
> and efficiently :-). Obviously I'm more concerned about system or
> human mess-ups than hardware failure here.
Right. One possibility is to slice up the larger storage into LVM
areas that serve as periodic snapshots of the root / project
partitions. Then the machines would serve as warm spares for each
other, plus anti-fat-finger backups. Off-site backups would be as
per the status quo.
> I assume Angela is still doing backups, [...]
(So am I, periodically.)
On Fri, Sep 07, 2012 at 07:58:45PM -0400, Christopher Faylor wrote:
> [...]
> I am probably stepping into a religious debate but wouldn't it make more
> sense to use postfix here? [...]
> (And as strange as qmail is, I have had a lot more trouble tweaking
> sendmail and postfix than I ever did with qmail) [...]
OK.
> You missed qpsmtpd, btw. I don't know if that can be made to work with
> the above or not.
AFAIK yes, if it's worth keeping (as opposed to putting spam filtering
directly into the normal MTA paths.)
> > spamassassin <same>, focus on inbound mail
>
> "inbound mail"? As opposed to...?
Outbound. E.g. frequent current complaints about our bugzilla
installation relate to the amount of time taken by spam checking for
outgoing messages.
> > clamav <same>
> > mlcheckd <retire>
>
> See above. mlcheckd is how we block the disclaimer email. And, it
> works as a secondary line of spam defense for when spamassassin fails.
> I also use it to block off-topic email in the cygwin list.
> I'm sure that could still be done with mailman but it will take research.
OK.
> >httpd:
> > httpd 2.0 httpd 2.2 + https
>
> Maybe we could tweak the httpd log file formats.
(What do you see wrong with ours?)
> > multi-root (gcc/sware/ecos/+) <same>
> > faq-o-matic mediawiki
> > moinmoin mediawiki
> > cvsweb viewvc
>
> I thought we were already using viewvc.
Yet /cgi-bin/cvsweb.cgi is still installed.
> > gitweb <same>
> > mnogosearch <same>
> > bugzilla <same>
> > retired services <html snapshot preserved>
> > content cvs git
>
> "content cvs"? Does that mean replace automatic updating of web pages
> with git?
As far as possible, yeah; or even make the htdocs hierarchies directly
into git repositories.
> [...]
> > cron jobs <same>
>
> Maybe we could retire the httpd cron jobs. Does anyone besides me read
> those weekly summaries? [...]
I glance at them.
- FChE
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-08 12:23 ` Frank Ch. Eigler
@ 2012-09-08 13:19 ` Joseph S. Myers
2012-10-28 22:44 ` Gerald Pfeifer
1 sibling, 0 replies; 23+ messages in thread
From: Joseph S. Myers @ 2012-09-08 13:19 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: Sourceware Overseers
On Sat, 8 Sep 2012, Frank Ch. Eigler wrote:
> > mailman's built-in archiver is pretty hopeless at URL stability (it may
> > well also lack other features such as the linking to a raw text version of
> > each message that we've put into our mhonarc configuration). I advise
> > continuing to do web archives through much the same mhonarc configuration
> > as at present [...]
>
> I haven't heard of any concerns about mailman URL stability before.
> The installations I know of (@redhat.com, @fedorahosted.org) seem to
> be fine.
There's a long discussion starting at
<http://lists.wikimedia.org/pipermail/wikitech-l/2012-August/062364.html>
(at present, that URL itself may of course not be stable). In essence,
any sort of archive regeneration is liable to break URLs throughout the
entire period of the archives (whereas with mhonarc archives, if a
regeneration is needed then we've left the old files alone and just
regenerated the relevant month into a new directory such as 2012-09n, and
other months are completely untouched).
> I don't know. Reading more about it now, it's not as full-featured as
> I thought it was. Note that our mhonarc installation has been a
> source of problems too, e.g. recurrent XSS bugs in the get-raw-message
> CGIs.
I'd suggest ameliorating such XSS issues through using separate domains
for more services (with of course redirects for existing URLs) so that the
list archives are not same-origin as Bugzilla, for example. I think a
separate domain for Bugzilla attachments is now considered a good idea as
well. This is of course something that can be done incrementally after
the move.
get-raw-message is extremely useful functionality when you wish to extract
a patch from the archives, and I'm not aware of mailman having such
functionality.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-07 23:55 ` Jonathan Larmour
@ 2012-09-10 17:32 ` Angela Marie Thomas
2012-09-10 17:57 ` Frank Ch. Eigler
0 siblings, 1 reply; 23+ messages in thread
From: Angela Marie Thomas @ 2012-09-10 17:32 UTC (permalink / raw)
To: Jonathan Larmour; +Cc: Frank Ch. Eigler, Sourceware Overseers
On Sep 7, 2012, at 4:55 PM, Jonathan Larmour <jifl@jifvik.org> wrote:
> One thing missing off your list is backups. The problem with warm standby
> servers is that they allow problems to be replicated quickly and efficiently
> :-). Obviously I'm more concerned about system or human mess-ups than hardware
> failure here. I assume Angela is still doing backups, and if so, either
> transition will need to be co-ordinated with her, or some other means of
> incremental backup provided. During the transition period, both old and new
> systems will want to be backed up.
I still do full system backups daily except for htdig and some temporary data.
I assume offsite backups are still desired just to be on the safe side. I agree
if there's a phased transition both will need to be backed up until the old server
is fully decommissioned.
Is the hardware already racked with console access? If not, when will that
happen? Knowing the earliest we could start helps with planning backup
strategies and capacity increases.
--Angela
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-10 17:32 ` Angela Marie Thomas
@ 2012-09-10 17:57 ` Frank Ch. Eigler
2012-09-10 19:01 ` Christopher Faylor
0 siblings, 1 reply; 23+ messages in thread
From: Frank Ch. Eigler @ 2012-09-10 17:57 UTC (permalink / raw)
To: Angela Marie Thomas; +Cc: Jonathan Larmour, Sourceware Overseers
Hi, Angela -
> [...] Is the hardware already racked with console access? [...]
Yes and no. The /root/.ssh/authorized_keys have been copied over so
y'all may *look around* via ssh root@server[45].sourceware.org. The
virtual console is on the RH internal network.
- FChE
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-10 17:57 ` Frank Ch. Eigler
@ 2012-09-10 19:01 ` Christopher Faylor
2012-09-10 20:18 ` Angela Marie Thomas
0 siblings, 1 reply; 23+ messages in thread
From: Christopher Faylor @ 2012-09-10 19:01 UTC (permalink / raw)
To: Frank Ch. Eigler, Sourceware Overseers, Angela Marie Thomas,
Jonathan Larmour
On Mon, Sep 10, 2012 at 01:56:59PM -0400, Frank Ch. Eigler wrote:
>Hi, Angela -
>
>> [...] Is the hardware already racked with console access? [...]
>
>Yes and no. The /root/.ssh/authorized_keys have been copied over so
>y'all may *look around* via ssh root@server[45].sourceware.org. The
>virtual console is on the RH internal network.
But that's nothing new...
Should we start rsyncing data over? It seems like we'll want a full
backup of old sourceware there eventually.
cgf
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-10 19:01 ` Christopher Faylor
@ 2012-09-10 20:18 ` Angela Marie Thomas
2012-09-12 15:29 ` Frank Ch. Eigler
0 siblings, 1 reply; 23+ messages in thread
From: Angela Marie Thomas @ 2012-09-10 20:18 UTC (permalink / raw)
To: Christopher Faylor
Cc: Frank Ch. Eigler, Sourceware Overseers, Jonathan Larmour
On Sep 10, 2012, at 12:00 PM, Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org> wrote:
> On Mon, Sep 10, 2012 at 01:56:59PM -0400, Frank Ch. Eigler wrote:
>> Hi, Angela -
>>
>>> [...] Is the hardware already racked with console access? [...]
>>
>> Yes and no. The /root/.ssh/authorized_keys have been copied over so
>> y'all may *look around* via ssh root@server[45].sourceware.org. The
>> virtual console is on the RH internal network.
>
> But that's nothing new...
Indeed. My query was more along the lines of figuring out timelines.
If the partitioning is as desired and the base OS is installed, that's a
significant milestone.
I see that server[45] are configured with a single data partition /srv. Is
it likely that it will be a mirror of sourceware's current /export/u0 or do
folks anticipate some changes in the directory layout?
> Should we start rsyncing data over? It seems like we'll want a full
> backup of old sourceware there eventually.
Given that the new servers have considerably more capacity, it makes sense
to store a complete snapshot of current sourceware there. It's stating the
obvious, but we'd want to avoid anything that changes a lot as that tends to
make the final copy take more time rather than less time. That's one reason
I don't backup htdig. Regenerating it is easier than backing it up.
--Angela
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-10 20:18 ` Angela Marie Thomas
@ 2012-09-12 15:29 ` Frank Ch. Eigler
2012-09-12 15:37 ` Christopher Faylor
0 siblings, 1 reply; 23+ messages in thread
From: Frank Ch. Eigler @ 2012-09-12 15:29 UTC (permalink / raw)
To: Angela Marie Thomas; +Cc: Sourceware Overseers
Hi -
> Indeed. My query was more along the lines of figuring out timelines.
> If the partitioning is as desired and the base OS is installed, that's a
> significant milestone.
> I see that server[45] are configured with a single data partition /srv. Is
> it likely that it will be a mirror of sourceware's current /export/u0 or do
> folks anticipate some changes in the directory layout?
I've removed the large /srv and created three smaller equal
/sourceware[123] partitions. The idea is that one of these would be a
temporary sourceware rsync destination; one would be the real
new-sourceware filesystem like /export/u0 was; and one would be a
local backup (normally unmounted). Once the transition is complete,
two of them could be local backups (of different generations, say
monthly and weekly local rsync's).
> Given that the new servers have considerably more capacity, it makes
> sense to store a complete snapshot of current sourceware there.
> [...]
Right. One concern though is to avoid cross-authentication on the
machines as much as possible (i.e., storage of general-use ssh private
keys or equivalent), so as to prevent one from being used to break
into another some day.
- FChE
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 15:29 ` Frank Ch. Eigler
@ 2012-09-12 15:37 ` Christopher Faylor
2012-09-12 15:51 ` Christopher Faylor
` (2 more replies)
0 siblings, 3 replies; 23+ messages in thread
From: Christopher Faylor @ 2012-09-12 15:37 UTC (permalink / raw)
To: Frank Ch. Eigler, Sourceware Overseers, Angela Marie Thomas
On Wed, Sep 12, 2012 at 11:28:59AM -0400, Frank Ch. Eigler wrote:
>Hi -
>
>> Indeed. My query was more along the lines of figuring out timelines.
>> If the partitioning is as desired and the base OS is installed, that's a
>> significant milestone.
>
>> I see that server[45] are configured with a single data partition /srv. Is
>> it likely that it will be a mirror of sourceware's current /export/u0 or do
>> folks anticipate some changes in the directory layout?
>
>I've removed the large /srv and created three smaller equal
>/sourceware[123] partitions. The idea is that one of these would be a
>temporary sourceware rsync destination; one would be the real
>new-sourceware filesystem like /export/u0 was; and one would be a
>local backup (normally unmounted). Once the transition is complete,
>two of them could be local backups (of different generations, say
>monthly and weekly local rsync's).
Ok, how do we get started? I can take some time off from work to focus
on this if needed. We have days available for working on stuff like
this.
Should we rsync stuff over and then start working on a ezmlm->mailman
translation? We'll also likely need to do a lot of work to keep URLs
correct.
Do we REALLY want to move from ezmlm? Isn't there an rpm-based package
out there somewhere that we could use?
cgf
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 15:37 ` Christopher Faylor
@ 2012-09-12 15:51 ` Christopher Faylor
2012-09-12 15:59 ` Joseph S. Myers
2012-09-12 16:09 ` Frank Ch. Eigler
2 siblings, 0 replies; 23+ messages in thread
From: Christopher Faylor @ 2012-09-12 15:51 UTC (permalink / raw)
To: Frank Ch. Eigler, Sourceware Overseers, Angela Marie Thomas
On Wed, Sep 12, 2012 at 11:37:24AM -0400, Christopher Faylor wrote:
>Do we REALLY want to move from ezmlm? Isn't there an rpm-based package
>out there somewhere that we could use?
Sorry. That's just crazy talk. There is no officially supported ezmlm
package.
Nevermind.
cgf
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 15:37 ` Christopher Faylor
2012-09-12 15:51 ` Christopher Faylor
@ 2012-09-12 15:59 ` Joseph S. Myers
2012-09-12 16:20 ` Jeff Law
2012-09-12 16:09 ` Frank Ch. Eigler
2 siblings, 1 reply; 23+ messages in thread
From: Joseph S. Myers @ 2012-09-12 15:59 UTC (permalink / raw)
To: Sourceware Overseers; +Cc: Frank Ch. Eigler, Angela Marie Thomas
On Wed, 12 Sep 2012, Christopher Faylor wrote:
> Do we REALLY want to move from ezmlm?
I don't see any strong need to move from ezmlm (and qmail, which is pretty
closely tied to it); it may not have a pretty web interface, but it does
work fine for what's done with it on sourceware, and I think ezmlm and
qmail generally have a stronger security record than many more modern
packages for the same purposes. I'd also think that each service for
which we change the software while changing the hardware adds an extra
complication to the hardware move, and so an extra delay.
(I would however certainly be glad for the wikis to move to MediaWiki; the
existing wiki software has had various problems including accidental
deletion of attachments by recursive wget with no good way to undelete
them. But does such a move actually need to be combined with the hardware
move and OS upgrade, or could it just as well be done separately once the
new hardware and OS are in used with the existing choice of software?)
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 15:37 ` Christopher Faylor
2012-09-12 15:51 ` Christopher Faylor
2012-09-12 15:59 ` Joseph S. Myers
@ 2012-09-12 16:09 ` Frank Ch. Eigler
2012-09-12 17:09 ` Christopher Faylor
2 siblings, 1 reply; 23+ messages in thread
From: Frank Ch. Eigler @ 2012-09-12 16:09 UTC (permalink / raw)
To: Sourceware Overseers
Hi -
On Wed, Sep 12, 2012 at 11:37:24AM -0400, Christopher Faylor wrote:
> [...]
> Ok, how do we get started? I can take some time off from work to focus
> on this if needed. We have days available for working on stuff like
> this.
I think the key will be mechanizing & logging our work, so that a
trial migration can be done on our own time, with a later refresh at
the actual cutover.
So, first step would seem to be a rerunnable rsync from server1:/
->server4:/sourceware3, and (to start exercising warm-backup
capabilities) from server4:/sourceware[123]->server5:/sourceware[123].
I suggest rsyncing the whole machine (not just /export/u0) to capture
/etc and /usr/local and the databases and everything.
Chris, want to give those a try?
Next step could be creation of a script on server4 that takes
/sourceware3 and populates the migrated forms of /sourceware1 and
various symlinks, etc. on the server, to the extent easily automated.
I'd exclude things like iptables, passwd, httpd configs, which I
imagine we'll have to hand-migrate.
> Should we rsync stuff over and then start working on a ezmlm->mailman
> translation? We'll also likely need to do a lot of work to keep URLs
> correct.
(My original thought was that the existing mhonarc archives would be
preserved, so existing URLs would stay valid; just new archives would
go to a new http url hierarchy.)
> Do we REALLY want to move from ezmlm? Isn't there an rpm-based package
> out there somewhere that we could use?
Well, dunno. ezmlm is not the worst thing about old sourceware; if
others are content, we could keep it awhile longer. The mediawiki
transition could also be done later, on-line.
- FChE
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 15:59 ` Joseph S. Myers
@ 2012-09-12 16:20 ` Jeff Law
2012-09-12 16:29 ` Joseph S. Myers
0 siblings, 1 reply; 23+ messages in thread
From: Jeff Law @ 2012-09-12 16:20 UTC (permalink / raw)
To: Joseph S. Myers
Cc: Sourceware Overseers, Frank Ch. Eigler, Angela Marie Thomas
On 09/12/2012 09:59 AM, Joseph S. Myers wrote:
>> Do we REALLY want to move from ezmlm?
>
> I don't see any strong need to move from ezmlm (and qmail, which is pretty
> closely tied to it);
ezmlm is definitely tied to qmail.
IIRC the primary reasons for qmail/ezmlm were performance, security,
good archiving and allowing users to self-help as much as possible.
I'd think anything which does those things well would be sufficient;
there's certainly some benefit in using more standardized components
which would need to be balanced against the conversion cost.
Personally I'd tend to look to get the hardware move done first, then
start replacing software components as desired. But I haven't done any
admin work on sourceware in 10+ years, so I'll leave the final decisions
to those actually doing the work.
jeff
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 16:20 ` Jeff Law
@ 2012-09-12 16:29 ` Joseph S. Myers
0 siblings, 0 replies; 23+ messages in thread
From: Joseph S. Myers @ 2012-09-12 16:29 UTC (permalink / raw)
To: Jeff Law; +Cc: Sourceware Overseers, Frank Ch. Eigler, Angela Marie Thomas
On Wed, 12 Sep 2012, Jeff Law wrote:
> Personally I'd tend to look to get the hardware move done first, then start
> replacing software components as desired. But I haven't done any admin work
That makes sense to me as well.
The old hardware is a problem - the present system tends to be heavily
overloaded.
The old OS on the old hardware is a problem - various packages are too old
to work well with current versions of some non-OS software such as
Bugzilla.
For the most part, the local choices of software components to use for
various purposes are not problems in the same way.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 16:09 ` Frank Ch. Eigler
@ 2012-09-12 17:09 ` Christopher Faylor
2012-09-12 20:48 ` Joseph S. Myers
0 siblings, 1 reply; 23+ messages in thread
From: Christopher Faylor @ 2012-09-12 17:09 UTC (permalink / raw)
To: Frank Ch. Eigler, Sourceware Overseers
On Wed, Sep 12, 2012 at 12:09:35PM -0400, Frank Ch. Eigler wrote:
>> Should we rsync stuff over and then start working on a ezmlm->mailman
>> translation? We'll also likely need to do a lot of work to keep URLs
>> correct.
>
>(My original thought was that the existing mhonarc archives would be
>preserved, so existing URLs would stay valid; just new archives would
>go to a new http url hierarchy.)
Ah, that would be easier. So we just have an "old" archive which
preserves the old ml and a "new" one.
So, would it make sense to populate the new archive with old content too
just for consistency? We'd have duplicate content but it seems like it
would be less confusing to end users.
>> Do we REALLY want to move from ezmlm? Isn't there an rpm-based package
>> out there somewhere that we could use?
>
>Well, dunno. ezmlm is not the worst thing about old sourceware; if
>others are content, we could keep it awhile longer. The mediawiki
>transition could also be done later, on-line.
I chatted with Ian about this on irc. The problem with ezmlm and qmail
is that they are not maintained and, so, are unaware of new standards
like DKIM.
If we think we will have to at some point then I think we should pull
the band-aid off now. Or, alternatively, commit to supporting both
packages ourselves.
cgf
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 17:09 ` Christopher Faylor
@ 2012-09-12 20:48 ` Joseph S. Myers
2012-09-13 3:18 ` Christopher Faylor
0 siblings, 1 reply; 23+ messages in thread
From: Joseph S. Myers @ 2012-09-12 20:48 UTC (permalink / raw)
To: Sourceware Overseers; +Cc: Frank Ch. Eigler
On Wed, 12 Sep 2012, Christopher Faylor wrote:
> On Wed, Sep 12, 2012 at 12:09:35PM -0400, Frank Ch. Eigler wrote:
> >> Should we rsync stuff over and then start working on a ezmlm->mailman
> >> translation? We'll also likely need to do a lot of work to keep URLs
> >> correct.
> >
> >(My original thought was that the existing mhonarc archives would be
> >preserved, so existing URLs would stay valid; just new archives would
> >go to a new http url hierarchy.)
>
> Ah, that would be easier. So we just have an "old" archive which
> preserves the old ml and a "new" one.
>
> So, would it make sense to populate the new archive with old content too
> just for consistency? We'd have duplicate content but it seems like it
> would be less confusing to end users.
It would be less confusing, avoid URL instability if we ever again in
future wish to remove a message from the archives and preserve useful
features such as raw versions of messages just to keep using mhonarc with
the same URL structure as now, and make any mailman/pipermail URLs
redirect to the /ml/ mhonarc ones so noone ever accesses a message via
them. As far as I know mhonarc is maintained.
> I chatted with Ian about this on irc. The problem with ezmlm and qmail
> is that they are not maintained and, so, are unaware of new standards
> like DKIM.
Does ezmlm actually do any of the sort of changes to messages that would
break DKIM?
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-12 20:48 ` Joseph S. Myers
@ 2012-09-13 3:18 ` Christopher Faylor
2012-09-13 3:59 ` Ian Lance Taylor
0 siblings, 1 reply; 23+ messages in thread
From: Christopher Faylor @ 2012-09-13 3:18 UTC (permalink / raw)
To: Frank Ch. Eigler, Sourceware Overseers, Joseph S. Myers
On Wed, Sep 12, 2012 at 08:48:22PM +0000, Joseph S. Myers wrote:
>On Wed, 12 Sep 2012, Christopher Faylor wrote:
>
>> On Wed, Sep 12, 2012 at 12:09:35PM -0400, Frank Ch. Eigler wrote:
>> >> Should we rsync stuff over and then start working on a ezmlm->mailman
>> >> translation? We'll also likely need to do a lot of work to keep URLs
>> >> correct.
>> >
>> >(My original thought was that the existing mhonarc archives would be
>> >preserved, so existing URLs would stay valid; just new archives would
>> >go to a new http url hierarchy.)
>>
>> Ah, that would be easier. So we just have an "old" archive which
>> preserves the old ml and a "new" one.
>>
>> So, would it make sense to populate the new archive with old content too
>> just for consistency? We'd have duplicate content but it seems like it
>> would be less confusing to end users.
>
>It would be less confusing, avoid URL instability if we ever again in
>future wish to remove a message from the archives and preserve useful
>features such as raw versions of messages just to keep using mhonarc with
>the same URL structure as now, and make any mailman/pipermail URLs
>redirect to the /ml/ mhonarc ones so noone ever accesses a message via
>them. As far as I know mhonarc is maintained.
>
>> I chatted with Ian about this on irc. The problem with ezmlm and qmail
>> is that they are not maintained and, so, are unaware of new standards
>> like DKIM.
>
>Does ezmlm actually do any of the sort of changes to messages that would
>break DKIM?
Ian had to add patches to qmail (or maybe it was ezmlm) for DKIM
recently because mail from sourceware was being treated as spam. So
that's actually one of the examples of what it means for these tools to
be unsupported.
cgf
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-13 3:18 ` Christopher Faylor
@ 2012-09-13 3:59 ` Ian Lance Taylor
0 siblings, 0 replies; 23+ messages in thread
From: Ian Lance Taylor @ 2012-09-13 3:59 UTC (permalink / raw)
To: Sourceware Overseers
Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org>
writes:
>>Does ezmlm actually do any of the sort of changes to messages that would
>>break DKIM?
>
> Ian had to add patches to qmail (or maybe it was ezmlm) for DKIM
> recently because mail from sourceware was being treated as spam. So
> that's actually one of the examples of what it means for these tools to
> be unsupported.
Right, I had to change qmail to make it add DKIM signatures to outgoing
e-mail. You can see the signature in the headers of this very message.
In fact we add both a DKIM and a DomainKey signature.
If we stick with qmail, then in the future we will have to patch it to
add IPv6 support.
In general I like qmail and ezmlm quite a bit, but I'm not aware of any
coherent support for them.
Ian
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-09-08 12:23 ` Frank Ch. Eigler
2012-09-08 13:19 ` Joseph S. Myers
@ 2012-10-28 22:44 ` Gerald Pfeifer
2012-11-03 17:31 ` Daniel Berlin
1 sibling, 1 reply; 23+ messages in thread
From: Gerald Pfeifer @ 2012-10-28 22:44 UTC (permalink / raw)
To: overseers
On Sat, 8 Sep 2012, Frank Ch. Eigler wrote:
> On Fri, Sep 07, 2012 at 10:47:24PM +0000, Joseph S. Myers wrote:
>>> ezmlm mailman
>>> mhonarc mailman
>> mailman's built-in archiver is pretty hopeless at URL stability (it may
>> well also lack other features such as the linking to a raw text version of
>> each message that we've put into our mhonarc configuration). I advise
>> continuing to do web archives through much the same mhonarc configuration
>> as at present [...]
> I haven't heard of any concerns about mailman URL stability before.
I have not heard either way, but from the perspective of continuity
it would be nice to keep using mhonarc. Is this a big issue in terms
of maintenance/security? (I know about the XSS bugs we had to address,
so I understand there is some.)
>>> content cvs git
>> "content cvs"? Does that mean replace automatic updating of web pages
>> with git?
> As far as possible, yeah; or even make the htdocs hierarchies directly
> into git repositories.
I strongly suggest we use the some source control tool that the
respective projects use for the web pages as well.
When GCC was migrated from CVS to SVN, Daniel did not have the
time/motivation to also migrate the web pages and these are still
stuck with CVS which is just not helpful. I always wanted to do
this myself as time permits to learn what is necessary for such
a conversion and see this as the next step.
Can't be rocket science. ;-) In fact, if any of you knows how to
bring the GCC wwwdocs CVS module into GCC SVN _once_, I'd be happy
to take it from there. That is really the part I do not feel
comfortable with. The rest, I am sure I can pull off. :-)
Gerald
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: sourceware hardware transition plan, part 1/N
2012-10-28 22:44 ` Gerald Pfeifer
@ 2012-11-03 17:31 ` Daniel Berlin
0 siblings, 0 replies; 23+ messages in thread
From: Daniel Berlin @ 2012-11-03 17:31 UTC (permalink / raw)
To: Gerald Pfeifer; +Cc: overseers
On Sun, Oct 28, 2012 at 6:39 PM, Gerald Pfeifer <gerald@pfeifer.com> wrote:
> On Sat, 8 Sep 2012, Frank Ch. Eigler wrote:
>> On Fri, Sep 07, 2012 at 10:47:24PM +0000, Joseph S. Myers wrote:
>>>> ezmlm mailman
>>>> mhonarc mailman
>>> mailman's built-in archiver is pretty hopeless at URL stability (it may
>>> well also lack other features such as the linking to a raw text version of
>>> each message that we've put into our mhonarc configuration). I advise
>>> continuing to do web archives through much the same mhonarc configuration
>>> as at present [...]
>> I haven't heard of any concerns about mailman URL stability before.
>
> I have not heard either way, but from the perspective of continuity
> it would be nice to keep using mhonarc. Is this a big issue in terms
> of maintenance/security? (I know about the XSS bugs we had to address,
> so I understand there is some.)
>
>>>> content cvs git
>>> "content cvs"? Does that mean replace automatic updating of web pages
>>> with git?
>> As far as possible, yeah; or even make the htdocs hierarchies directly
>> into git repositories.
>
> I strongly suggest we use the some source control tool that the
> respective projects use for the web pages as well.
>
> When GCC was migrated from CVS to SVN, Daniel did not have the
> time/motivation to also migrate the web pages and these are still
> stuck with CVS which is just not helpful. I always wanted to do
> this myself as time permits to learn what is necessary for such
> a conversion and see this as the next step.
>
> Can't be rocket science. ;-) In fact, if any of you knows how to
> bring the GCC wwwdocs CVS module into GCC SVN _once_, I'd be happy
> to take it from there. That is really the part I do not feel
> comfortable with. The rest, I am sure I can pull off. :-)
Nowadays, cvs2svn or something similar should "just work" for this purpose.
Heck, you could probably just use svnsync.
>
> Gerald
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2012-10-28 23:45 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-09-07 21:47 sourceware hardware transition plan, part 1/N Frank Ch. Eigler
2012-09-07 22:47 ` Joseph S. Myers
2012-09-08 12:23 ` Frank Ch. Eigler
2012-09-08 13:19 ` Joseph S. Myers
2012-10-28 22:44 ` Gerald Pfeifer
2012-11-03 17:31 ` Daniel Berlin
2012-09-07 23:55 ` Jonathan Larmour
2012-09-10 17:32 ` Angela Marie Thomas
2012-09-10 17:57 ` Frank Ch. Eigler
2012-09-10 19:01 ` Christopher Faylor
2012-09-10 20:18 ` Angela Marie Thomas
2012-09-12 15:29 ` Frank Ch. Eigler
2012-09-12 15:37 ` Christopher Faylor
2012-09-12 15:51 ` Christopher Faylor
2012-09-12 15:59 ` Joseph S. Myers
2012-09-12 16:20 ` Jeff Law
2012-09-12 16:29 ` Joseph S. Myers
2012-09-12 16:09 ` Frank Ch. Eigler
2012-09-12 17:09 ` Christopher Faylor
2012-09-12 20:48 ` Joseph S. Myers
2012-09-13 3:18 ` Christopher Faylor
2012-09-13 3:59 ` Ian Lance Taylor
2012-09-07 23:59 ` Christopher Faylor
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).