Required host migration (Sourceware)

Required host migration (Sourceware)

[Jonathan Pickard]

Hello,

You or your group manages one or more servers located in Red Hat's PHX1
datacenter:

sourceware
sourceware1

If you are not the correct owner for the listed system(s) please respond
and let me know. If you know who the correct owner is, please include
that as well.

We are retrofitting PHX1 and your host(s) will need to be moved to our
PHX2 datacenter sometime in the next few months. Additionally, steps
will need to be taken by you or your group to make sure your hosts and
services remain accessible.

Work required:
* reconfigure networking on systems (IP address/netmask/gateway will change)
* reconfiguration of services on system (if necessary, due to new IPs)
* updates to DNS (due to new IPs)

We will take care of physically migrating the server for you, as well as
(re)configuring any available remote management access, such as serial,
DRACs, RSAs, iLO's, etc. Once remote management access is available, you
can perform the needed network/daemon reconfigurations using the new
network information we will provide. We will be available to assist if
any problems relating to the needed reconfigurations come up.

Note that the physical move will result in downtime for the hosts.
Allowing time for unracking, transportation, and re-racking, you can
anticipate up to 48 hours of downtime. Specific dates and times are not
yet available, but will be communicated out as soon as they are determined.

With that said, your options for proceeding are:
1) move to PHX2 (migrate)
2) have us return the systems to you if you own them (leave)
3) have us power off and store the systems (decommission)

Please respond to this message to acknowledge ownership of the listed
system(s), inform us which listed option you will proceed with, and
state that you or someone form your group will be available to assist in
the work required for migration.

Thanks,
Jonathan Pickard
Principal Systems Administrator
Corporate Infrastructure Services
Red Hat, Inc.

Re: Required host migration (Sourceware)

[Frank Ch. Eigler]

Hi -


On Tue, Sep 29, 2009 at 12:26:38PM -0700, Jonathan Pickard wrote:
> You or your group manages one or more servers located in Red Hat's PHX1
> datacenter:
> 
> sourceware
> sourceware1

Actually there are three machines, with DNS names sourceware.org,
server2.sourceware.org, server3.sourceware.org.


> [...]
> Work required:
> * reconfigure networking on systems (IP address/netmask/gateway will change)
> * reconfiguration of services on system (if necessary, due to new IPs)
> * updates to DNS (due to new IPs)

Can the new IP addresses be provided in advance?  Would it be possible
to configure the machines in PHX1 with (temporary) dual-homing to
actually exercise the new addresses, so as to give us a transition
period?


> We will take care of physically migrating the server for you, 

This could be an opportune (for us) time to get a helpful upgrade for
the main old machine.  None of our guys are near PHX, so would it be
possible to coordinate having you guys add in new memory DIMMs during
the downtime?


> as well as (re)configuring any available remote management access,
> such as serial, DRACs, RSAs, iLO's, etc.  Once remote management
> access is available, you can perform the needed network/daemon
> reconfigurations [...]

The machines currently have a somewhat incestuous interconnection for
their serial consoles, which may preclude trivial remote-access
bootstrapping.  I believe <mgalgoci> has another separate console
access method that he would have to share with us.


> Note that the physical move will result in downtime for the hosts.
> Allowing time for unracking, transportation, and re-racking, you can
> anticipate up to 48 hours of downtime. Specific dates and times are
> not yet available, but will be communicated out as soon as they are
> determined.

The longer notice, the better.  These machines are hubs of several
well-known free software projects, so a large community is being
impacted.


- FChE

Re: Required host migration (Sourceware)

[Matthew Galgoci]

> >> Work required:
> >> * reconfigure networking on systems (IP address/netmask/gateway will change)
> >> * reconfiguration of services on system (if necessary, due to new IPs)
> >> * updates to DNS (due to new IPs)
> >
> > Can the new IP addresses be provided in advance?  Would it be possible
> > to configure the machines in PHX1 with (temporary) dual-homing to
> > actually exercise the new addresses, so as to give us a transition
> > period?
>
> We will know the new network details in advance of the move, but I don't
> think its possible to do the dual-homing. I've cc:ed mgalgoci who can
> provide a definitive answer.

Setting up the new netblock and then moving it would be somewhat of a pain.
I would rather see us expend our effort on the new setup and suggest we take
one of the lighter used servers and migrate it before the other two and setup
some preliminary services on it. Make sense?

> I've not physically inspected the serial setup, but I believe we have a
> cyclades hooked up for remote access. mgalgoci, do you know specific
> details if we are doing something non-standard?

The current sourceware setup is somewhat non-standard as far as serial
ports and RAC controllers go. In the new environment we will be able to
provide a dedicated cyclades just for sourceware use and empower the
sourceware folks to self service.

-- 
Matthew Galgoci
Network Operations
Red Hat, Inc
919.754.3700 x43155

Re: Required host migration (Sourceware)

[Christopher Faylor]

On Thu, Oct 01, 2009 at 03:05:48PM -0400, Matthew Galgoci wrote:
>> >> Work required:
>> >> * reconfigure networking on systems (IP address/netmask/gateway will change)
>> >> * reconfiguration of services on system (if necessary, due to new IPs)
>> >> * updates to DNS (due to new IPs)
>> >
>> > Can the new IP addresses be provided in advance?  Would it be possible
>> > to configure the machines in PHX1 with (temporary) dual-homing to
>> > actually exercise the new addresses, so as to give us a transition
>> > period?
>>
>> We will know the new network details in advance of the move, but I don't
>> think its possible to do the dual-homing. I've cc:ed mgalgoci who can
>> provide a definitive answer.
>
>Setting up the new netblock and then moving it would be somewhat of a pain.
>I would rather see us expend our effort on the new setup and suggest we take
>one of the lighter used servers and migrate it before the other two and setup
>some preliminary services on it. Make sense?

Actually, no, I don't think so.  The other two servers are actually very
underused.  Frank might correct me but I think we probably wouldn't
notice if they went down right now.

What would be great is if, in the process of this move, we could get what
we'd originally hoped for when sourceware was upgraded a few years ago -
two identical (or nearly identical) machines.  We could use one for the
main system and one for backup, allowing us to hot swap between the two
if needed.

If we could have one of the systems set up at the new site ahead of time
we wouldn't even need 48 hours down time.

cgf

Re: Required host migration (Sourceware)

[Frank Ch. Eigler]

Hi -

cgf wrote:

> mgalgoci wrote:
> > I would rather see us expend our effort on the new setup and suggest we take
> > one of the lighter used servers and migrate it before the other two and setup
> > some preliminary services on it. Make sense?

Unfortunately sourceware is not set up to be easily segmented into
partial services.  server2 and server3 are too small to even store all
the data.


> [...]  The other two servers are actually very underused.  Frank
> might correct me but I think we probably wouldn't notice if they
> went down right now.

Re. #3, right, it contains periodic partial backups of #1.
Re. #2, no, it houses the mysql instance that serves the search engine.


> What would be great is if, in the process of this move, we could get
> what we'd originally hoped for when sourceware was upgraded a few
> years ago [...]

(I fear that may be beyond even the considerable powers of this
particular crew...  unless some other group abandons PHX1 hardware...)


- FChE

Re: Required host migration (Sourceware)

[Matthew Galgoci]

> Date: Thu, 1 Oct 2009 15:20:43 -0400
> From: Frank Ch. Eigler <fche@redhat.com>
> To: Eric Brown <ebrown@redhat.com>, overseers@sourceware.org,
>     Jonathan Pickard <jpickard@redhat.com>,
>     Matthew Galgoci <mgalgoci@redhat.com>, ian@airs.com
> Subject: Re: Required host migration (Sourceware)
>
> Hi -
>
> cgf wrote:
>
> > mgalgoci wrote:
> > > I would rather see us expend our effort on the new setup and suggest we take
> > > one of the lighter used servers and migrate it before the other two and setup
> > > some preliminary services on it. Make sense?
>
> Unfortunately sourceware is not set up to be easily segmented into
> partial services.  server2 and server3 are too small to even store all
> the data.

Be that as it may, we can at least stand up so that folks can test
connectivity as mentioned earlier. On that server a "sorry" page could
be setup indicating that sourceware is in the process of being moved.

-- 
Matthew Galgoci
Network Operations
Red Hat, Inc
919.754.3700 x43155

Re: Required host migration (Sourceware)

[Jonathan Pickard]

On 10/01/2009 02:10 PM, Matthew Galgoci wrote:
>> Date: Thu, 1 Oct 2009 15:20:43 -0400
>> From: Frank Ch. Eigler <fche@redhat.com>
>> To: Eric Brown <ebrown@redhat.com>, overseers@sourceware.org,
>>     Jonathan Pickard <jpickard@redhat.com>,
>>     Matthew Galgoci <mgalgoci@redhat.com>, ian@airs.com
>> Subject: Re: Required host migration (Sourceware)
>>
>> Hi -
>>
>> cgf wrote:
>>
>>> mgalgoci wrote:
>>>> I would rather see us expend our effort on the new setup and suggest we take
>>>> one of the lighter used servers and migrate it before the other two and setup
>>>> some preliminary services on it. Make sense?
>>
>> Unfortunately sourceware is not set up to be easily segmented into
>> partial services.  server2 and server3 are too small to even store all
>> the data.
> 
> Be that as it may, we can at least stand up so that folks can test
> connectivity as mentioned earlier. On that server a "sorry" page could
> be setup indicating that sourceware is in the process of being moved.
> 

All,

We have identified the weekend of December 12 as the migration period
for these servers. Based on the original e-mail and responses, I believe
the choice is between two options:

1) move to PHX2 (migrate)
2) order new hardware for PHX2 (phased migration)

For 1) - Red Hat IT will move the machines and bring them back up in
PHX2, reconfiguring their network details if wanted and providing remote
access (serial and/or KVM along with remote power management).

For 2) - Red Hat IT will install any new hardware and provide remote
access (serial and/or KVM along with remote power management) and you
will migrate your services from PHX to PHX2.

Please advise on how you would like to proceed.

Thanks,
Jonathan

Re: Required host migration (Sourceware)

[Christopher Faylor]

On Thu, Nov 05, 2009 at 03:37:14PM -0700, Jonathan Pickard wrote:
>On 10/01/2009 02:10 PM, Matthew Galgoci wrote:
>>> Date: Thu, 1 Oct 2009 15:20:43 -0400
>>> From: Frank Ch. Eigler <fche@redhat.com>
>>> To: Eric Brown <ebrown@redhat.com>, overseers@sourceware.org,
>>>     Jonathan Pickard <jpickard@redhat.com>,
>>>     Matthew Galgoci <mgalgoci@redhat.com>, ian@airs.com
>>> Subject: Re: Required host migration (Sourceware)
>>>
>>> Hi -
>>>
>>> cgf wrote:
>>>
>>>> mgalgoci wrote:
>>>>> I would rather see us expend our effort on the new setup and suggest we take
>>>>> one of the lighter used servers and migrate it before the other two and setup
>>>>> some preliminary services on it. Make sense?
>>>
>>> Unfortunately sourceware is not set up to be easily segmented into
>>> partial services.  server2 and server3 are too small to even store all
>>> the data.
>> 
>> Be that as it may, we can at least stand up so that folks can test
>> connectivity as mentioned earlier. On that server a "sorry" page could
>> be setup indicating that sourceware is in the process of being moved.
>> 
>
>All,
>
>We have identified the weekend of December 12 as the migration period
>for these servers. Based on the original e-mail and responses, I believe
>the choice is between two options:
>
>1) move to PHX2 (migrate)
>2) order new hardware for PHX2 (phased migration)
>
>For 1) - Red Hat IT will move the machines and bring them back up in
>PHX2, reconfiguring their network details if wanted and providing remote
>access (serial and/or KVM along with remote power management).
>
>For 2) - Red Hat IT will install any new hardware and provide remote
>access (serial and/or KVM along with remote power management) and you
>will migrate your services from PHX to PHX2.
>
>Please advise on how you would like to proceed.

I would very much prefer 2.

cgf

Re: Required host migration (Sourceware)

[Frank Ch. Eigler]

Hi -

On Thu, Nov 05, 2009 at 05:57:21PM -0500, Christopher Faylor wrote:
> [...]
> >[...]
> >1) move to PHX2 (migrate)
> >2) order new hardware for PHX2 (phased migration)
> >
> >For 1) - Red Hat IT will move the machines and bring them back up in
> >PHX2, reconfiguring their network details if wanted and providing remote
> >access (serial and/or KVM along with remote power management).
> >
> >For 2) - Red Hat IT will install any new hardware and provide remote
> >access (serial and/or KVM along with remote power management) and you
> >will migrate your services from PHX to PHX2.

> I would very much prefer 2.

I don't think IT was offering to procure a new machine for us, just to
install one if we can get hold of same.  (And as we discussed before,
since our server2/server3 machines were purchased with inadequate
resources, they can't bear the load.)

So by default, this would mean option #1.

(As I offered before, I can arrange to have a RAM upgrade package sent
to RH IS to install, along with option #1.)

- FChE

Re: Required host migration (Sourceware)

[Jonathan Pickard]

On 11/06/2009 09:09 AM, Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Nov 05, 2009 at 05:57:21PM -0500, Christopher Faylor wrote:
>> [...]
>>> [...]
>>> 1) move to PHX2 (migrate)
>>> 2) order new hardware for PHX2 (phased migration)
>>>
>>> For 1) - Red Hat IT will move the machines and bring them back up in
>>> PHX2, reconfiguring their network details if wanted and providing remote
>>> access (serial and/or KVM along with remote power management).
>>>
>>> For 2) - Red Hat IT will install any new hardware and provide remote
>>> access (serial and/or KVM along with remote power management) and you
>>> will migrate your services from PHX to PHX2.
> 
>> I would very much prefer 2.
> 
> I don't think IT was offering to procure a new machine for us, just to
> install one if we can get hold of same.  (And as we discussed before,
> since our server2/server3 machines were purchased with inadequate
> resources, they can't bear the load.)
> 
> So by default, this would mean option #1.
> 
> (As I offered before, I can arrange to have a RAM upgrade package sent
> to RH IS to install, along with option #1.)
> 
> - FChE

Frank,

You can order the extra RAM at any time and have it shipped to PHX2:

Red Hat, C/O Internap
Attn: Jonathan Pickard (Sourceware)
2121 South Price Road
Chandler, AZ 85248, USA

Provide the tracking # once you have it. Once it is ordered send me
instructions on how much should go into what machines and I will make
sure it is taken care of during the migration window.

Regards,
Jonathan

Re: Required host migration (Sourceware)

[Jonathan Pickard]

On 11/06/2009 02:52 PM, Jonathan Pickard wrote:
> On 11/06/2009 09:09 AM, Frank Ch. Eigler wrote:
>> Hi -
>>
>> On Thu, Nov 05, 2009 at 05:57:21PM -0500, Christopher Faylor wrote:
>>> [...]
>>>> [...]
>>>> 1) move to PHX2 (migrate)
>>>> 2) order new hardware for PHX2 (phased migration)
>>>>
>>>> For 1) - Red Hat IT will move the machines and bring them back up in
>>>> PHX2, reconfiguring their network details if wanted and providing remote
>>>> access (serial and/or KVM along with remote power management).
>>>>
>>>> For 2) - Red Hat IT will install any new hardware and provide remote
>>>> access (serial and/or KVM along with remote power management) and you
>>>> will migrate your services from PHX to PHX2.
>>
>>> I would very much prefer 2.
>>
>> I don't think IT was offering to procure a new machine for us, just to
>> install one if we can get hold of same.  (And as we discussed before,
>> since our server2/server3 machines were purchased with inadequate
>> resources, they can't bear the load.)
>>
>> So by default, this would mean option #1.
>>
>> (As I offered before, I can arrange to have a RAM upgrade package sent
>> to RH IS to install, along with option #1.)
>>
>> - FChE
> 
> Frank,
> 
> You can order the extra RAM at any time and have it shipped to PHX2:
> 
> Red Hat, C/O Internap
> Attn: Jonathan Pickard (Sourceware)
> 2121 South Price Road
> Chandler, AZ 85248, USA
> 
> Provide the tracking # once you have it. Once it is ordered send me
> instructions on how much should go into what machines and I will make
> sure it is taken care of during the migration window.
> 
> Regards,
> Jonathan

Hello,

I see that the extra RAM has been ordered.

An additional reminder that the physical migrations will happen on
12/12. You should send out any needed outage notices and make any
additional preparations ASAP. You can expect up to 48 hours of downtime
on 12/12 and 12/13. I will send more details out to you early next week.

Regards,
Jonathan