down time

down time

[Matthew Galgoci]

I need to schedule some downtime for sourceware.

Sourceware will need to be re-ip'd, and upgraded to 4GB of ram.
The reason for the re-ip is that the machine is coming back to my 
machine room and will live on a dedicated hosting network. The ram 
upgrade is courtesy of Red Hat.

I would like to do this preferably today if nobody objects too much, or
sometime this weekend. I was thinking that if I do it in the middle of the
afternoon folks could do a long lunch.

I don't anticipate the downtime will be more than 2 hrs.

Aye? Nay?

-- 
Matthew Galgoci
System Administrator and Sr. Manager of Ruminants
Red Hat, Inc
919.754.3700 x44155

Re: down time

[Ian Lance Taylor]

Matthew Galgoci <mgalgoci@redhat.com> writes:

> I need to schedule some downtime for sourceware.
> 
> Sourceware will need to be re-ip'd, and upgraded to 4GB of ram.
> The reason for the re-ip is that the machine is coming back to my 
> machine room and will live on a dedicated hosting network. The ram 
> upgrade is courtesy of Red Hat.
> 
> I would like to do this preferably today if nobody objects too much, or
> sometime this weekend. I was thinking that if I do it in the middle of the
> afternoon folks could do a long lunch.
> 
> I don't anticipate the downtime will be more than 2 hrs.
> 
> Aye? Nay?

Please let me know the downtime/uptime when you commit to it, so that
I can update the sourceware.org IP address at the same time.

I'll drop the sourceware.org TTL right now.

I'm sure there is some reason for the short notice.  At the risk of
pointing out the obvious, in the future it would be good to have more
notice.  I do appreciate the work that you are doing.

Ian

Re: down time

[Matthew Galgoci]

On 7 May 2004, Ian Lance Taylor wrote:

> Matthew Galgoci <mgalgoci@redhat.com> writes:
> 
> > I need to schedule some downtime for sourceware.
> > 
> > Sourceware will need to be re-ip'd, and upgraded to 4GB of ram.
> > The reason for the re-ip is that the machine is coming back to my 
> > machine room and will live on a dedicated hosting network. The ram 
> > upgrade is courtesy of Red Hat.
> > 
> > I would like to do this preferably today if nobody objects too much, or
> > sometime this weekend. I was thinking that if I do it in the middle of the
> > afternoon folks could do a long lunch.
> > 
> > I don't anticipate the downtime will be more than 2 hrs.
> > 
> > Aye? Nay?
> 
> Please let me know the downtime/uptime when you commit to it, so that
> I can update the sourceware.org IP address at the same time.
> 
> I'll drop the sourceware.org TTL right now.
> 
> I'm sure there is some reason for the short notice.  At the risk of
> pointing out the obvious, in the future it would be good to have more
> notice.  I do appreciate the work that you are doing.

I can commit to 1pm EDT today if nobody objects since I will be moving most of
gnome.org and rpm.org at that time. The new ip will be 12.107.209.250

Again, I am looking for a big fat "a-ok do it" or "no dear $deity don't"

-- 
Matthew Galgoci
System Administrator and Sr. Manager of Ruminants
Red Hat, Inc
919.754.3700 x44155

Re: down time

[Ian Lance Taylor]

Matthew Galgoci <mgalgoci@redhat.com> writes:

> Sourceware will need to be re-ip'd, and upgraded to 4GB of ram.
> The reason for the re-ip is that the machine is coming back to my 
> machine room and will live on a dedicated hosting network. The ram 
> upgrade is courtesy of Red Hat.

By the way, a couple of files which need to change for the new IP
address:

/sourceware/djbdns/axfrdns/env/IP
/sourceware/djbdns/tinydns/env/IP

Ian

Re: down time

[Matthew Galgoci]

> > 
> > Please let me know the downtime/uptime when you commit to it, so that
> > I can update the sourceware.org IP address at the same time.
> > 
> > I'll drop the sourceware.org TTL right now.
> > 
> > I'm sure there is some reason for the short notice.  At the risk of
> > pointing out the obvious, in the future it would be good to have more
> > notice.  I do appreciate the work that you are doing.
> 
> I can commit to 1pm EDT today if nobody objects since I will be moving most of
> gnome.org and rpm.org at that time. The new ip will be 12.107.209.250
> 
> Again, I am looking for a big fat "a-ok do it" or "no dear $deity don't"

Ok, 1pm it is. I spoke to cgf via phone and he's ok though the notive is
extremely short, for which I apologize profusely.

Ian, Chris (cgf), please update your dns accordingly once you notive sourceware
has been taken down. I'm on my way to the colo now.

Cgf has my cell phone number.

-- 
Matthew Galgoci
System Administrator and Sr. Manager of Ruminants
Red Hat, Inc
919.754.3700 x44155

Re: down time

[Christopher Faylor]

On Fri, May 07, 2004 at 12:00:47PM -0400, Matthew Galgoci wrote:
>On 7 May 2004, Ian Lance Taylor wrote:
>
>> Matthew Galgoci <mgalgoci@redhat.com> writes:
>> 
>> > I need to schedule some downtime for sourceware.
>> > 
>> > Sourceware will need to be re-ip'd, and upgraded to 4GB of ram.
>> > The reason for the re-ip is that the machine is coming back to my 
>> > machine room and will live on a dedicated hosting network. The ram 
>> > upgrade is courtesy of Red Hat.
>> > 
>> > I would like to do this preferably today if nobody objects too much, or
>> > sometime this weekend. I was thinking that if I do it in the middle of the
>> > afternoon folks could do a long lunch.
>> > 
>> > I don't anticipate the downtime will be more than 2 hrs.
>> > 
>> > Aye? Nay?
>> 
>> Please let me know the downtime/uptime when you commit to it, so that
>> I can update the sourceware.org IP address at the same time.
>> 
>> I'll drop the sourceware.org TTL right now.
>> 
>> I'm sure there is some reason for the short notice.  At the risk of
>> pointing out the obvious, in the future it would be good to have more
>> notice.  I do appreciate the work that you are doing.
>
>I can commit to 1pm EDT today if nobody objects since I will be moving most of
>gnome.org and rpm.org at that time. The new ip will be 12.107.209.250
>
>Again, I am looking for a big fat "a-ok do it" or "no dear $deity don't"

I echo Ian's sentiment but if you think it's a priority to do this, then
I say lets go for it.  I wish we had a little more buy-in from the gcc
and gdb components here, though.

cgf

Re: down time

[Angela Marie Thomas]

> I echo Ian's sentiment but if you think it's a priority to do this, then
> I say lets go for it.  I wish we had a little more buy-in from the gcc
> and gdb components here, though.
> 
> cgf

STOP THE PRESSES!!!!!!!!!!!!!!!!!!

Just woke up, sorry.  It's not even 10am on the west coast ya know.

Who's talked to the FSF about gcc.gnu.org?  Traditionally this has taken
multiple days, not multiple hours, notice.

--Angela

Re: down time

[Christopher Faylor]

On Fri, May 07, 2004 at 10:10:27AM -0700, Angela Marie Thomas wrote:
>>I echo Ian's sentiment but if you think it's a priority to do this,
>>then I say lets go for it.  I wish we had a little more buy-in from the
>>gcc and gdb components here, though.
>
>STOP THE PRESSES!!!!!!!!!!!!!!!!!!
>
>Just woke up, sorry.  It's not even 10am on the west coast ya know.
>
>Who's talked to the FSF about gcc.gnu.org?  Traditionally this has
>taken multiple days, not multiple hours, notice.

Matt says that he has that covered.  IIRC, he knows one of the sysadmins
there personally.

cgf

Re: down time

[Angela Marie Thomas]

Now that my quick message went out, I can send a longer one.

First, I find the lack of notice completely unacceptable.
Not everyone is located on the east coast.  We require *at least*
24 hours notice, probably more, so we can notify all the necessary
people in all the different time zones about the change.  We've
gone through IP changes several times now and each has resulted
in extended down time due to lack of communication or forgetting
to change some obscure configuration file.  Let's learn from our
past mistakes.

You obviously knew this was going to happen some time in the future.
Maybe I missed it, but I didn't see any discussion of this on the
sourceware sysadmin mailing list or the overseers list.  There are
a number of things that need to be done when we change the
IP address.  Because we know this has been an issue in the past,
we need a plan up front so we can identify everything that needs
to change and who should be notified.

Not only is the IP address changing, but the machine is physically
moving as well.  This sort of thing must be coordinated with me so
we can ensure the backups are as fresh as possible relative to when
the machine is taken down.  We don't want to lose a day or more of
work just because we were too stupid to coordinate times.

The timing sucks.  10am Pacific on a work day with no notice.
There's a pretty big chunk of gcc folks in this timezone who will
not be very happy to find out they can't do anything and don't
know why.  It would suck the same way for other timezones.  A move
like this should probably happen on a Sat or a calculated very low
usage time.  Sat is good because it's a weekend day everywhere.
Sun starts pushing into the beginning of the week in AU.

And since I feel it bears repeating, I'll note again that it is
completely an utterly unacceptable to send out such short notice
*especially* when the work will begin BEFORE A LOT OF FOLKS ARE
EVEN AWAKE!

--Angela

Re: down time

[Matthew Galgoci]

On Fri, 7 May 2004, Angela Marie Thomas wrote:

> 
> Now that my quick message went out, I can send a longer one.
> 
> First, I find the lack of notice completely unacceptable.
> Not everyone is located on the east coast.  We require *at least*
> 24 hours notice, probably more, so we can notify all the necessary
> people in all the different time zones about the change.  We've
> gone through IP changes several times now and each has resulted
> in extended down time due to lack of communication or forgetting
> to change some obscure configuration file.  Let's learn from our
> past mistakes.

The colo was in danger of shutting down.

I know it sucks. I was working on slowly migrating stuff out of that
colo and then apparently out of the blue I was told that I would lose
physical access within 48hrs. That was yesterday. 

I had to pull all of my gear from that colo presence in 48hrs time.
 
> Not only is the IP address changing, but the machine is physically
> moving as well.  This sort of thing must be coordinated with me so
> we can ensure the backups are as fresh as possible relative to when
> the machine is taken down.  We don't want to lose a day or more of
> work just because we were too stupid to coordinate times.

I am sorry, I accept full responsibility for what happened. Do not
blame Red Hat or anyone else. I made a judgement call after consulting
with chris and frank.
 
> And since I feel it bears repeating, I'll note again that it is
> completely an utterly unacceptable to send out such short notice
> *especially* when the work will begin BEFORE A LOT OF FOLKS ARE
> EVEN AWAKE!

To be honest I now believe that the morning is the best time to do work,
though advance notice is key :)

-- 
Matthew Galgoci
System Administrator and Sr. Manager of Ruminants
Red Hat, Inc
919.754.3700 x44155

Re: down time

[Matthew Galgoci]

On Fri, 7 May 2004, Angela Marie Thomas wrote:

> 
> > I echo Ian's sentiment but if you think it's a priority to do this, then
> > I say lets go for it.  I wish we had a little more buy-in from the gcc
> > and gdb components here, though.
> > 
> > cgf
> 
> STOP THE PRESSES!!!!!!!!!!!!!!!!!!
> 
> Just woke up, sorry.  It's not even 10am on the west coast ya know.
> 
> Who's talked to the FSF about gcc.gnu.org?  Traditionally this has taken
> multiple days, not multiple hours, notice.
> 
> --Angela

I know one of their admins. This can be taken care of quickly.

-- 
Matthew Galgoci
System Administrator and Sr. Manager of Ruminants
Red Hat, Inc
919.754.3700 x44155

Re: down time

[Phil Edwards]

On Fri, May 07, 2004 at 04:20:41PM -0400, Matthew Galgoci wrote:
> I know it sucks. I was working on slowly migrating stuff out of that
> colo and then apparently out of the blue I was told that I would lose
> physical access within 48hrs. That was yesterday. 
> 
> I had to pull all of my gear from that colo presence in 48hrs time.

So the list should have had 46 or 47 hours notice.  Not 1 hour.

The reason this sucks for me -- and I'm even in the same time zone as you
and the box -- is that the TTL was still set at its normal high level until
the 1 hour notice.  If it had been reduced at the 48 hours mark, then the
/new/ TTL would have finished propogating by the time the IP address changed.
I.e., the way these changes are supposed to be done.

(Maybe that should be an automatic reaction to even a rumour of configuration
change:  drop the TTL to something low.  If it's a false alarm, well,
no serious harm done.)

Now we have people who are going to be stuck with the old address until
Sunday.


-- 
The spirit is willing, but the flesh is spongy and bruised.  - Zapp

Re: down time

[Jason Molenda]

On Fri, May 07, 2004 at 05:03:22PM -0400, Phil Edwards wrote:

> 
> Now we have people who are going to be stuck with the old address until
> Sunday.


Hm.  Here at Apple sources.redhat.com maps to the new addy;
sourceware.org and gcc.gnu.org to the old.   And ssh'ing in to
sources.redhat.com gets me the happy-happy message

Address 12.107.209.250 maps to sourceware.org, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT!

Hi-larious.

The compiler group guys were complaining because they couldn't get
to their stuff.  I told them to use sources.redhat.com for cvs
access for the moment, but there's no way to give them access to
the mailing list web archives until the IP# updates percolate.  
(I gather some of them read the development lists via the web).

J

PS- My ISP account has all the new IP#'s already.

Re: down time

[Ian Lance Taylor]

Jason Molenda <jason-swarelist@molenda.com> writes:

> Hm.  Here at Apple sources.redhat.com maps to the new addy;
> sourceware.org and gcc.gnu.org to the old.   And ssh'ing in to
> sources.redhat.com gets me the happy-happy message

sourceware.org should resolve to the correct address by 3pm tomorrow
at the latest.  Or of course you could restart your DNS cache.

Ian