Looking for service/server operators

Looking for service/server operators

[Karsten Wade]

[-- Attachment #1.1: Type: text/plain, Size: 568 bytes --]

Hi:

I'm wondering who is responsible for what happens at sourceware.org?

I was alerted that projects had migrated from fedorahosted.org, and I'm
curious what all is hosted here now?

I've also heard the hardware is seriously aging, and I'd like to work
with someone to make sure that this service is taken care of. I'm
working on a budget request to replace hardware based on best guesses
with Stephen Smoogen.

Best regards,

- Karsten
-- 
Karsten Wade
Community Infrastructure Team
Open Source and Standards, @redhatopen
@quaid gpg: AD0E0C41


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Looking for service/server operators

[Frank Ch. Eigler]

Hi -

> I'm wondering who is responsible for what happens at sourceware.org?

Hey there, I guess.

> I was alerted that projects had migrated from fedorahosted.org, and
> I'm curious what all is hosted here now?

We recently welcomed elfutils, lvm2 (back), publican, plus all the
older GNU etc. stuff.

> I've also heard the hardware is seriously aging, and I'd like to
> work with someone to make sure that this service is taken care
> of. [...]

The hardware is around 5 years old, but is running reasonably well.
(16-core, E5620, 72GB-RAM, 3T disk).  It may not be worthwhile to get
an only sligthly-better new box, as migration downtimes are a hassle.
But, not to look a gift horse in the mouth, what did you have in mind?


- FChE

Re: Looking for service/server operators

[Karsten Wade]

[-- Attachment #1.1: Type: text/plain, Size: 2041 bytes --]

On 03/16/2017 12:40 PM, Frank Ch. Eigler wrote:
> Hi -

Hi! Glad to find you.

>> I'm wondering who is responsible for what happens at sourceware.org?
> 
> Hey there, I guess.
> 
>> I was alerted that projects had migrated from fedorahosted.org, and
>> I'm curious what all is hosted here now?
> 
> We recently welcomed elfutils, lvm2 (back), publican, plus all the
> older GNU etc. stuff.
> 
>> I've also heard the hardware is seriously aging, and I'd like to
>> work with someone to make sure that this service is taken care
>> of. [...]
> 
> The hardware is around 5 years old, but is running reasonably well.
> (16-core, E5620, 72GB-RAM, 3T disk).  It may not be worthwhile to get
> an only sligthly-better new box, as migration downtimes are a hassle.
> But, not to look a gift horse in the mouth, what did you have in mind?

Well, it's not much of a gift yet as Eng is having to cut overall CapEx
requests by quite a bit.

Smooge had identified that this hardware might be in trouble soon or at
least was aging beyond support. I've currently got a request in for
$28,000 that would cover two Dell servers. Hoping that doesn't get cut
much more, but it could be chopped smaller.

Alternately, would we be able to extend the warranty on that hardware?
Agreed that five years old is not that bad at this point with the type
of hardware it likely is.

Sometime this coming quarter it would be great to have a conversation
with those of you who are running Sourceware.org to learn more about how
you do things, what the future might look like, etc. Our team has been
collaborating with other community infra sysadmins, for helping with
best practices, finding ways to collaborate, etc. We've got a handful of
projects we are providing a range of service for from bare metal to
basic colo:

https://mojo.redhat.com/groups/community-infrastructure-cage

Best regards,

- Karsten
-- 
Karsten Wade
Community Infrastructure Team
Open Source and Standards, @redhatopen
@quaid gpg: AD0E0C41


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Looking for service/server operators

[Karsten Wade]

[-- Attachment #1.1: Type: text/plain, Size: 2854 bytes --]

Hi Frank et al:

Since we last addressed this, I had put in the request for $14K for a
new host and got overall approval. However, CapEx for all of PnT Eng was
over budget by quite a bit, and we've since formed a council that is
reviewing each approved item for relative prioritization on everything
from new initiatives to keeping the lights on.

If this host is fine as is, I would recommend that we return the $14K to
the pool for this year.

If there are warranty extensions needed, I might be able to fund that
out of OSAS OpEx budget.

Another option is to look at shared hosting. For example, the OSAS
ComInfra team provides managed and unmanaged VMs out of the Raeigh data
center.

https://mojo.redhat.com/groups/community-infrastructure-cage

If we had a problem with the current hardware, and probably just as a
backup situation, we could likely provision a VM to bring service back
up. That might be a good way to keep things going for this year while we
look at a plan for the future.

Ultimately my goal is to normalize how we sponsor community infra at Red
Hat so that things don't fall between the cracks, live under desks, get
lost in the hardware lifecycle, etc. Some of this gets easier with
shared infra. For requests of this size I'm happy to run them through
the OSAS budget, happy to help provide and maintain services, and/or
simply appear in a budget request to help justify where and why we need
dedicated hosting, etc.

We are clearly inspired by what sourecware.org has been doing over the
years, and regardless of this current budget discussion I would like to
find a way for us all to be in better contact in the future. It might be
that we can provide some sysadmin support, VM backups, alternative
hosting, etc.

Let me know what you all thing.

Best regards,

- Karsten



On 03/16/2017 12:40 PM, Frank Ch. Eigler wrote:
> Hi -
> 
>> I'm wondering who is responsible for what happens at sourceware.org?
> 
> Hey there, I guess.
> 
>> I was alerted that projects had migrated from fedorahosted.org, and
>> I'm curious what all is hosted here now?
> 
> We recently welcomed elfutils, lvm2 (back), publican, plus all the
> older GNU etc. stuff.
> 
>> I've also heard the hardware is seriously aging, and I'd like to
>> work with someone to make sure that this service is taken care
>> of. [...]
> 
> The hardware is around 5 years old, but is running reasonably well.
> (16-core, E5620, 72GB-RAM, 3T disk).  It may not be worthwhile to get
> an only sligthly-better new box, as migration downtimes are a hassle.
> But, not to look a gift horse in the mouth, what did you have in mind?
> 
> 
> - FChE
> 

-- 
Karsten Wade
Community Infrastructure Team : https://osci.io
Red Hat Open Source and Standards (OSAS) : @redhatopen
@quaid gpg: AD0E0C41


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Looking for service/server operators

[Christopher Faylor]

On Tue, Jul 25, 2017 at 01:41:15PM -0700, Karsten Wade wrote:
>Hi Frank et al:
>
>Since we last addressed this, I had put in the request for $14K for a
>new host and got overall approval. However, CapEx for all of PnT Eng was
>over budget by quite a bit, and we've since formed a council that is
>reviewing each approved item for relative prioritization on everything
>from new initiatives to keeping the lights on.

Is there any chance that we could get the bad battery on the raid
replaced on sourceware?  We have errors showing up in the logs from
time to time.

Re: Looking for service/server operators

[Frank Ch. Eigler]

Hi, Karsten -

> Since we last addressed this, I had put in the request for $14K [...]
> If there are warranty extensions needed, I might be able to fund that
> out of OSAS OpEx budget.

We're some way out of warranty (on both our twin servers), so not sure
it's retroactively possible.  That'd be great though.


> Another option is to look at shared hosting. For example, the OSAS
> ComInfra team provides managed and unmanaged VMs [...]

The size of sourceware (esp. its RAM/cpu/disk load) suggest that you'd
need very large VMs to carry it.  We're happy with bare metal, though
periodically some upgrading is nice.  For example, a couple of years
ago we bumped up physical RAM by a factor of 6, which made a big
difference.

The next resource we seem to be short of is disk iops.  Would y'll be
able to help fund something like this pciexpress ssd card:
https://www.newegg.com/Product/Product.aspx?item=0D9-000G-00071 ?


> If we had a problem with the current hardware, and probably just as
> a backup situation, we could likely provision a VM to bring service
> back up. [...]

We have a warm-backup server that's a near-twin of the mainline one,
so this concern is not too pressing.


> Ultimately my goal is to normalize how we sponsor community infra at
> Red Hat [...]

Sounds good.  Our capital/expense needs have generally been modest and
on-site physical support needs pretty rare too.  But checking in
regularly, sharing knowledge of how the other groups are getting by,
would all be great.


- FChE

Re: Looking for service/server operators

[Karsten Wade]

[-- Attachment #1.1: Type: text/plain, Size: 1858 bytes --]

Frank:

I've included you on a thread with the other parties involved so we can
get a decision on bare metal vs. VM by this Friday.

That VM would be provided by RDO Cloud on an ongoing basis. Our team
would help create and broker the initial relationship, then get out of
the way once you all and RDO Cloud ops team are properly connected.

Some further comments below:

On 07/26/2017 09:49 AM, Frank Ch. Eigler wrote:
> The next resource we seem to be short of is disk iops.  Would y'll be
> able to help fund something like this pciexpress ssd card:
> https://www.newegg.com/Product/Product.aspx?item=0D9-000G-00071 ?

I would be happy to get that for you from OSAS, we some have budget
specifically for such things.

The questions are:

* Is it safe to do an upgrade with existing hardware?
* Should we wait for the new VM/server before proceeding with an
upgrade? But then why bother?
  - One point is that it may still be a few months before something is
operational to back up or replace the current server, where we can get
that part delivered very soon and installed in the system fairly
quickly, but are you all ready on the operations side for that?

Thanks for working with us and having patience as we try to 'help'. ;-)
Part of what our team is doing is finding all the community
infrastructure that has built up organically over the years and make
sure it has ongoing love and attention from the various groups at Red
Hat who care ... once they know about and have something plugged into
their systems. So plugging into the systems is what OSAS Community Infra
team is busy doing all the time and happy to continue helping here.

Best regards,

- Karsten
-- 
Karsten Wade
Community Infrastructure Team : https://osci.io
Red Hat Open Source and Standards (OSAS) : @redhatopen
@quaid gpg: AD0E0C41


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]