Red Hat PDU upgrades (Mesa, AZ datacenter)

Red Hat PDU upgrades (Mesa, AZ datacenter)

[Jonathan Pickard]

Hello,

We will soon be conducting an upgrade to the PDUs that provide power  
to the servers that you are hosting with Red Hat in our Mesa, AZ  
(PHX) facility. To minimize service disruption, I am coordinating the  
work with the various host owners. You are listed as a contact for  
the following machines:
sourceware
sourceware1
sourceware2

Your affected machines have dual power supplies, and the work being  
planned is staged to take advantage of this through the following steps:

* Install new PDU (new-a)
* Removing servers power from old primary PDU (old-a)
* Connecting servers power to new primary PDU (new-a)
* Repeat process with secondary PDUs (old-a and old-b)

If your hosts are properly configured and the hardware is in working  
order, there should be no downtime. However, it may be at the point  
of power swap that a problem is realized. To minimize that risk, we  
are working with the host owners to schedule a date and time that  
will lessen the impact of a downed host or hosts.

Please respond to me as soon as possible with the following:
* Confirmation that you have working remote management access to the  
host, in order to bring it back online if power is disrupted  
(something such as remote serial access through a Cyclades, a Dell  
DRAC, IBM RASII, or the like).
* A range of dates and times, before November 21, that you prefer the  
outage be scheduled for.

Once I have this information I will work with the various owners to  
coordinate a finalized date and time for the work.

Thanks,
Jonathan Pickard
Corporate Infrastructure Services
Red Hat, Inc

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Frank Ch. Eigler]

Hi -

On Mon, Nov 10, 2008 at 11:45:55AM -0700, Jonathan Pickard wrote:
> [...]
> We will soon be conducting an upgrade to the PDUs that provide power  
> to the servers that you are hosting with Red Hat in our Mesa, AZ  
> (PHX) facility. To minimize service disruption, I am coordinating the  
> work with the various host owners. You are listed as a contact for  
> the following machines:
> sourceware
> sourceware1
> sourceware2
> [...]

I believe mgalgoci is the only one with control over the remote power
management hardware for these machines, should they become necessary.


- FChE

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Matthew Galgoci]

> On Mon, Nov 10, 2008 at 11:45:55AM -0700, Jonathan Pickard wrote:
> > [...]
> > We will soon be conducting an upgrade to the PDUs that provide power
> > to the servers that you are hosting with Red Hat in our Mesa, AZ
> > (PHX) facility. To minimize service disruption, I am coordinating the
> > work with the various host owners. You are listed as a contact for
> > the following machines:
> > sourceware
> > sourceware1
> > sourceware2
> > [...]
>
> I believe mgalgoci is the only one with control over the remote power
> management hardware for these machines, should they become necessary.

Correct but you should have serial console from either the secondary or
tertiary box as well. That doesn't help you if they're all down though.

-- 
Matthew Galgoci
Network Operations
Red Hat, Inc
919.754.3700 x44155

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Christopher Faylor]

On Mon, Nov 10, 2008 at 06:13:54PM -0500, Matthew Galgoci wrote:
>> On Mon, Nov 10, 2008 at 11:45:55AM -0700, Jonathan Pickard wrote:
>> > [...]
>> > We will soon be conducting an upgrade to the PDUs that provide power
>> > to the servers that you are hosting with Red Hat in our Mesa, AZ
>> > (PHX) facility. To minimize service disruption, I am coordinating the
>> > work with the various host owners. You are listed as a contact for
>> > the following machines:
>> > sourceware
>> > sourceware1
>> > sourceware2
>> > [...]
>>
>> I believe mgalgoci is the only one with control over the remote power
>> management hardware for these machines, should they become necessary.
>
>Correct but you should have serial console from either the secondary or
>tertiary box as well. That doesn't help you if they're all down though.

We do have this but, as you say...

cgf

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Matthew Galgoci]

> Date: Mon, 10 Nov 2008 19:49:02 -0500
> From: Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org>
> To: Sourceware Overseers <overseers@sources.redhat.com>,
>     Frank Ch. Eigler <fche@redhat.com>,
>     Jonathan Pickard <jpickard@redhat.com>,
>     Matthew Galgoci <mgalgoci@redhat.com>
> Subject: Re: Red Hat PDU upgrades (Mesa, AZ datacenter)
>
> On Mon, Nov 10, 2008 at 06:13:54PM -0500, Matthew Galgoci wrote:
> >> On Mon, Nov 10, 2008 at 11:45:55AM -0700, Jonathan Pickard wrote:
> >> > [...]
> >> > We will soon be conducting an upgrade to the PDUs that provide power
> >> > to the servers that you are hosting with Red Hat in our Mesa, AZ
> >> > (PHX) facility. To minimize service disruption, I am coordinating the
> >> > work with the various host owners. You are listed as a contact for
> >> > the following machines:
> >> > sourceware
> >> > sourceware1
> >> > sourceware2
> >> > [...]
> >>
> >> I believe mgalgoci is the only one with control over the remote power
> >> management hardware for these machines, should they become necessary.
> >
> >Correct but you should have serial console from either the secondary or
> >tertiary box as well. That doesn't help you if they're all down though.
>
> We do have this but, as you say...

Jonathan will be on site, and I'll be available remotely to assist with
any cleanup....

-- 
Matthew Galgoci
Network Operations
Red Hat, Inc
919.754.3700 x44155

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Christopher Faylor]

On Tue, Nov 11, 2008 at 10:28:45AM -0500, Matthew Galgoci wrote:
>> Date: Mon, 10 Nov 2008 19:49:02 -0500
>> From: Christopher Faylor <cgf-use-the-mailinglist-please@sourceware.org>
>> To: Sourceware Overseers <overseers@sources.redhat.com>,
>>     Frank Ch. Eigler <fche@redhat.com>,
>>     Jonathan Pickard <jpickard@redhat.com>,
>>     Matthew Galgoci <mgalgoci@redhat.com>
>> Subject: Re: Red Hat PDU upgrades (Mesa, AZ datacenter)
>>
>> On Mon, Nov 10, 2008 at 06:13:54PM -0500, Matthew Galgoci wrote:
>> >> On Mon, Nov 10, 2008 at 11:45:55AM -0700, Jonathan Pickard wrote:
>> >> > [...]
>> >> > We will soon be conducting an upgrade to the PDUs that provide power
>> >> > to the servers that you are hosting with Red Hat in our Mesa, AZ
>> >> > (PHX) facility. To minimize service disruption, I am coordinating the
>> >> > work with the various host owners. You are listed as a contact for
>> >> > the following machines:
>> >> > sourceware
>> >> > sourceware1
>> >> > sourceware2
>> >> > [...]
>> >>
>> >> I believe mgalgoci is the only one with control over the remote power
>> >> management hardware for these machines, should they become necessary.
>> >
>> >Correct but you should have serial console from either the secondary or
>> >tertiary box as well. That doesn't help you if they're all down though.
>>
>> We do have this but, as you say...
>
>Jonathan will be on site, and I'll be available remotely to assist with
>any cleanup....

Ok, in that case, I'd say late at night (10PM PST) or weekends would be
fine for this.  Just let us know what is convenient for you.

Overseers feel free to disagree if I'm wrong here.

cgf

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Jonathan Pickard]

Hello,

The PDU upgrades have been scheduled for Thursday, November 20, 2008  
at 6pm MST (5pm PST, 8pm EST). I will follow-up once the work is  
completed or if issues arise.

Regards,
Jonathan

On Nov 10, 2008, at 11:45 AM, Jonathan Pickard wrote:

> Hello,
>
> We will soon be conducting an upgrade to the PDUs that provide  
> power to the servers that you are hosting with Red Hat in our Mesa,  
> AZ (PHX) facility. To minimize service disruption, I am  
> coordinating the work with the various host owners. You are listed  
> as a contact for the following machines:
> sourceware
> sourceware1
> sourceware2
>
> Your affected machines have dual power supplies, and the work being  
> planned is staged to take advantage of this through the following  
> steps:
>
> * Install new PDU (new-a)
> * Removing servers power from old primary PDU (old-a)
> * Connecting servers power to new primary PDU (new-a)
> * Repeat process with secondary PDUs (old-a and old-b)
>
> If your hosts are properly configured and the hardware is in  
> working order, there should be no downtime. However, it may be at  
> the point of power swap that a problem is realized. To minimize  
> that risk, we are working with the host owners to schedule a date  
> and time that will lessen the impact of a downed host or hosts.
>
> Please respond to me as soon as possible with the following:
> * Confirmation that you have working remote management access to  
> the host, in order to bring it back online if power is disrupted  
> (something such as remote serial access through a Cyclades, a Dell  
> DRAC, IBM RASII, or the like).
> * A range of dates and times, before November 21, that you prefer  
> the outage be scheduled for.
>
> Once I have this information I will work with the various owners to  
> coordinate a finalized date and time for the work.
>
> Thanks,
> Jonathan Pickard
> Corporate Infrastructure Services
> Red Hat, Inc

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Jonathan Pickard]

[Reminder that this maintenance will take place tomorrow.]

Hello,

The PDU upgrades have been scheduled for Thursday, November 20, 2008  
at 6pm MST (5pm PST, 8pm EST). I will follow-up once the work is  
completed or if issues arise.

Regards,
Jonathan

On Nov 10, 2008, at 11:45 AM, Jonathan Pickard wrote:

> Hello,
>
> We will soon be conducting an upgrade to the PDUs that provide  
> power to the servers that you are hosting with Red Hat in our Mesa,  
> AZ (PHX) facility. To minimize service disruption, I am  
> coordinating the work with the various host owners. You are listed  
> as a contact for the following machines:
> sourceware
> sourceware1
> sourceware2
>
> Your affected machines have dual power supplies, and the work being  
> planned is staged to take advantage of this through the following  
> steps:
>
> * Install new PDU (new-a)
> * Removing servers power from old primary PDU (old-a)
> * Connecting servers power to new primary PDU (new-a)
> * Repeat process with secondary PDUs (old-a and old-b)
>
> If your hosts are properly configured and the hardware is in  
> working order, there should be no downtime. However, it may be at  
> the point of power swap that a problem is realized. To minimize  
> that risk, we are working with the host owners to schedule a date  
> and time that will lessen the impact of a downed host or hosts.
>
> Please respond to me as soon as possible with the following:
> * Confirmation that you have working remote management access to  
> the host, in order to bring it back online if power is disrupted  
> (something such as remote serial access through a Cyclades, a Dell  
> DRAC, IBM RASII, or the like).
> * A range of dates and times, before November 21, that you prefer  
> the outage be scheduled for.
>
> Once I have this information I will work with the various owners to  
> coordinate a finalized date and time for the work.
>
> Thanks,
> Jonathan Pickard
> Corporate Infrastructure Services
> Red Hat, Inc

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Christopher Faylor]

On Wed, Nov 19, 2008 at 09:18:00AM -0700, Jonathan Pickard wrote:
> [Reminder that this maintenance will take place tomorrow.]
>
> Hello,
>
> The PDU upgrades have been scheduled for Thursday, November 20, 2008 at 6pm 
> MST (5pm PST, 8pm EST). I will follow-up once the work is completed or if 
> issues arise.

Thanks for the reminder.

cgf

Re: Red Hat PDU upgrades (Mesa, AZ datacenter)

[Jonathan Pickard]

All,

The maintenance finished without major incident.

Regards,
Jonathan

On Nov 19, 2008, at 9:18 AM, Jonathan Pickard wrote:

> [Reminder that this maintenance will take place tomorrow.]
>
> Hello,
>
> The PDU upgrades have been scheduled for Thursday, November 20,  
> 2008 at 6pm MST (5pm PST, 8pm EST). I will follow-up once the work  
> is completed or if issues arise.
>
> Regards,
> Jonathan
>
> On Nov 10, 2008, at 11:45 AM, Jonathan Pickard wrote:
>
>> Hello,
>>
>> We will soon be conducting an upgrade to the PDUs that provide  
>> power to the servers that you are hosting with Red Hat in our  
>> Mesa, AZ (PHX) facility. To minimize service disruption, I am  
>> coordinating the work with the various host owners. You are listed  
>> as a contact for the following machines:
>> sourceware
>> sourceware1
>> sourceware2
>>
>> Your affected machines have dual power supplies, and the work  
>> being planned is staged to take advantage of this through the  
>> following steps:
>>
>> * Install new PDU (new-a)
>> * Removing servers power from old primary PDU (old-a)
>> * Connecting servers power to new primary PDU (new-a)
>> * Repeat process with secondary PDUs (old-a and old-b)
>>
>> If your hosts are properly configured and the hardware is in  
>> working order, there should be no downtime. However, it may be at  
>> the point of power swap that a problem is realized. To minimize  
>> that risk, we are working with the host owners to schedule a date  
>> and time that will lessen the impact of a downed host or hosts.
>>
>> Please respond to me as soon as possible with the following:
>> * Confirmation that you have working remote management access to  
>> the host, in order to bring it back online if power is disrupted  
>> (something such as remote serial access through a Cyclades, a Dell  
>> DRAC, IBM RASII, or the like).
>> * A range of dates and times, before November 21, that you prefer  
>> the outage be scheduled for.
>>
>> Once I have this information I will work with the various owners  
>> to coordinate a finalized date and time for the work.
>>
>> Thanks,
>> Jonathan Pickard
>> Corporate Infrastructure Services
>> Red Hat, Inc
>