Choice of distribution for new sourceware.org server?

Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

Overseers,

May I ask why CentOS 8 was chosen as the distribution choice for
the new sourceware.org servers?

Is there any reason that RHEL 8 was not chosen?

Alternatively Fedora Server on a faster update cycle?

Was there any requirement or discussion around which distribution
to choose for the new server?

I can see RHEL 8 serving as a better platform than CentOS 8, particularly
if Red Hat donates the licenses.

Likewise using Fedora Server would keep all of our services running on
the latest distribution with access to new server-side tooling for CI.

Thank you for your time!

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Christopher Faylor]

On Wed, Feb 05, 2020 at 08:05:01PM -0500, Carlos O'Donell wrote:
>May I ask why CentOS 8 was chosen as the distribution choice for
>the new sourceware.org servers?
>
>Is there any reason that RHEL 8 was not chosen?
>
>Alternatively Fedora Server on a faster update cycle?
>
>Was there any requirement or discussion around which distribution
>to choose for the new server?
>
>I can see RHEL 8 serving as a better platform than CentOS 8, particularly
>if Red Hat donates the licenses.
>
>Likewise using Fedora Server would keep all of our services running on
>the latest distribution with access to new server-side tooling for CI.
>
>Thank you for your time!

I have suggested Fedora in the past and would be happy to be using it.

I don't know what the difference would be between CentOS 8 and RHEL 8
though.

cgf

Re: Choice of distribution for new sourceware.org server?

[Frank Ch. Eigler]

Hi -

> I have suggested Fedora in the past and would be happy to be using it.

Having to upgrade apprx. every 8 months is a PITA though, with so much
activity and non-distro software on the machine.

> I don't know what the difference would be between CentOS 8 and RHEL 8
> though.

I'm not aware of -much-.  Maybe timelier security patches.  I'll look
into upgrading centos->rhel online, but OTOH it doesn't seem that
necessary either.

- FChE

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/5/20 8:30 PM, Christopher Faylor wrote:
> On Wed, Feb 05, 2020 at 08:05:01PM -0500, Carlos O'Donell wrote:
>> May I ask why CentOS 8 was chosen as the distribution choice for
>> the new sourceware.org servers?
>>
>> Is there any reason that RHEL 8 was not chosen?
>>
>> Alternatively Fedora Server on a faster update cycle?
>>
>> Was there any requirement or discussion around which distribution
>> to choose for the new server?
>>
>> I can see RHEL 8 serving as a better platform than CentOS 8, particularly
>> if Red Hat donates the licenses.
>>
>> Likewise using Fedora Server would keep all of our services running on
>> the latest distribution with access to new server-side tooling for CI.
>>
>> Thank you for your time!
> 
> I have suggested Fedora in the past and would be happy to be using it.

I would also be happy to see us use Fedora.
 
> I don't know what the difference would be between CentOS 8 and RHEL 8
> though.

For us?

* Directly faster security updates since CentOS 8 lags RHEL.
* Access to ELS if we want to keep the server alive longer.
* Support for things we don't have the time to fix or are outside our scope (kernel bugs).
  - Usually here we can rely on internal access to Red Hat experts.

Do these things matter to us?

Otherwise dog-fooding Fedora Server would give us the latest packages
for services that sourceware could offer, but we'd have to update once
a year. It would force us to actively track the software we're using and
look at well supported alternatives e.g. postfix + mailman/public inbox vs. 
qmail + ezmlm.

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/5/20 8:34 PM, Frank Ch. Eigler wrote:
> Hi -
> 
>> I have suggested Fedora in the past and would be happy to be using it.
> 
> Having to upgrade apprx. every 8 months is a PITA though, with so much
> activity and non-distro software on the machine.

Sorry, I don't follow, could you expand on this a bit?

How much non-distro software is there on sourceware?

When you say "activity" do you mean that users would be impacted once
every 8 months for a small window of downtime?

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Frank Ch. Eigler]

Hi -


> > Having to upgrade apprx. every 8 months is a PITA though, with so much
> > activity and non-distro software on the machine.
> 
> Sorry, I don't follow, could you expand on this a bit?

> How much non-distro software is there on sourceware?

Well, quite a bit.

> When you say "activity" do you mean that users would be impacted
> once every 8 months for a small window of downtime?

Probably, if nothing goes wrong.  And if there's an incompatible
rebase of python or php or whatever, then we (?) have to fix the
thing.  There is a downside to upgrade churn, in terms of our
manpower.

Note that with rhel8/centos8, we'll have a competent container
platform.  Heck, the new machine is big enough to carry a few VMs too,
so if some future service were to require newer rawhide-y prereqs, we
can accomomdate that even with a stable base.

- FChE

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/5/20 9:02 PM, Frank Ch. Eigler wrote:
> Hi -
> 
> 
>>> Having to upgrade apprx. every 8 months is a PITA though, with so much
>>> activity and non-distro software on the machine.
>>
>> Sorry, I don't follow, could you expand on this a bit?
> 
>> How much non-distro software is there on sourceware?
> 
> Well, quite a bit.

If non-distro softare impacts our maintenance costs, should we be evaluating
the list of non-distro software for value vs. cost?

>> When you say "activity" do you mean that users would be impacted
>> once every 8 months for a small window of downtime?
> 
> Probably, if nothing goes wrong.  And if there's an incompatible
> rebase of python or php or whatever, then we (?) have to fix the
> thing.  There is a downside to upgrade churn, in terms of our
> manpower.

Best practice is to have a production and development server to
avoid these kinds of problems and shake out such changes.

Do we have 2 servers now?

> Note that with rhel8/centos8, we'll have a competent container
> platform.  Heck, the new machine is big enough to carry a few VMs too,
> so if some future service were to require newer rawhide-y prereqs, we
> can accomomdate that even with a stable base.

I agree it would be great to run containers or VMs on sourcware for
specific services.

In which case RHEL8 with ELS as an option is even more valuable?

In summary:
- Why not stick with RHEL8 to get ELS and avoid CentOS 8 early phase out.
- Stick with RHEL8 as long as we can to avoid costly maintenance upgrades.
- Use VMs and containers to isolate new systems to provide services that
  need newer userspace or kernels.

Does RHEL8 seem like the better choice here with ELS?

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Christopher Faylor]

On Wed, Feb 05, 2020 at 08:45:28PM -0500, Carlos O'Donell wrote:
>Otherwise dog-fooding Fedora Server would give us the latest packages
>for services that sourceware could offer, but we'd have to update once
>a year.

We talk about that every time we go through a system or OS upgrade.
I've maintained a postfix/mailman system in the past so maybe those bits
would swap back in.  I don't really relish the thought of trying to
convert the ezmlm archives to something else but maybe it isn't a big
deal.

>It would force us to actively track the software we're using and look
>at well supported alternatives e.g.  postfix + mailman/public inbox vs.
>qmail + ezmlm.

I use Rawhide on some of my systems and rarely have had a problem but I
can hear the screams now.

Upgrading to the next version only requires minimal downtime though.
So, I'd still be happy to be using it.  I assume we can get the same
level of input from Red Hat if we have kernel issues.

This is the third or fourth time I've moved to a new version of
qmail/ezmlm and I never have a good feeling about it since the available
rpms always seem a little odd.  They have dependencies on things that I
don't really understand like "vpopmail".  We could roll our own like
we did when sourceware was new but that means being diligent about
security issues.

I think I've just convinced myself to look into using postfix and
mailman.  I'll try to do that in the next few days.

cgf

Re: Choice of distribution for new sourceware.org server?

[Frank Ch. Eigler]

Hi -

> >> How much non-distro software is there on sourceware?
> > Well, quite a bit.
> 
> If non-distro softare impacts our maintenance costs, should we be evaluating
> the list of non-distro software for value vs. cost?

We evaluate regularly.  It's not just software that substitutes for
distro bits (there's only a bit of that).  Remember things like
multiple wikis, bugzilla.


> >> When you say "activity" do you mean that users would be impacted
> >> once every 8 months for a small window of downtime?
> > 
> > Probably, if nothing goes wrong.  And if there's an incompatible
> > rebase of python or php or whatever, then we (?) have to fix the
> > thing.  There is a downside to upgrade churn, in terms of our
> > manpower.
> 
> Best practice is to have a production and development server to
> avoid these kinds of problems and shake out such changes.
> Do we have 2 servers now?

rawhide level upgrade churn has to bring tangible value that is in
excess of the costs (to all the parties, not just users).


> In which case RHEL8 with ELS as an option is even more valuable?

> In summary:
> - Why not stick with RHEL8 to get ELS and avoid CentOS 8 early phase out.

Already said I'll look into moving from centos8 to rhel8.  centos
lifespan does not look materially different from rhel.


- FChE

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/6/20 1:05 AM, Christopher Faylor wrote:
> On Wed, Feb 05, 2020 at 08:45:28PM -0500, Carlos O'Donell wrote:
>> Otherwise dog-fooding Fedora Server would give us the latest packages
>> for services that sourceware could offer, but we'd have to update once
>> a year.
> 
> We talk about that every time we go through a system or OS upgrade.
> I've maintained a postfix/mailman system in the past so maybe those bits
> would swap back in.  I don't really relish the thought of trying to
> convert the ezmlm archives to something else but maybe it isn't a big
> deal.
> 
>> It would force us to actively track the software we're using and look
>> at well supported alternatives e.g.  postfix + mailman/public inbox vs.
>> qmail + ezmlm.
> 
> I use Rawhide on some of my systems and rarely have had a problem but I
> can hear the screams now.
> 
> Upgrading to the next version only requires minimal downtime though.
> So, I'd still be happy to be using it.  I assume we can get the same
> level of input from Red Hat if we have kernel issues.
> 
> This is the third or fourth time I've moved to a new version of
> qmail/ezmlm and I never have a good feeling about it since the available
> rpms always seem a little odd.  They have dependencies on things that I
> don't really understand like "vpopmail".  We could roll our own like
> we did when sourceware was new but that means being diligent about
> security issues.
> 
> I think I've just convinced myself to look into using postfix and
> mailman.  I'll try to do that in the next few days.

I think switching  to postfix would really be to our benefit, both
from an upstream support, and long-term perspective.

The archives will be an interesting challenge, but I think we can
preserve them statically.

Have you seen public inbox? It's amazingly fast and the kernel is
using it now. There is even a libc-alpha archive:
https://public-inbox.org/libc-alpha/

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/6/20 9:49 AM, Frank Ch. Eigler wrote:
> Hi -
> 
>>>> How much non-distro software is there on sourceware?
>>> Well, quite a bit.
>>
>> If non-distro softare impacts our maintenance costs, should we be evaluating
>> the list of non-distro software for value vs. cost?
> 
> We evaluate regularly.  It's not just software that substitutes for
> distro bits (there's only a bit of that).  Remember things like
> multiple wikis, bugzilla.

Awesome. Do you need any help with regular evaluations? Is there a meeting
for that? Or is this tactically driven when something can't be upgraded?

I'm surprised the wiki is non-distro software. I recently setup a dokuwiki
backed by git, and it was all distro-software with just configuration data.

Moinmoin is basically dead upstream.

Dokuwiki is actively maintained and there are migration paths from moinmoin
to dokuwiki.

Also if we did a git-backed wiki we could use the same git access mechanisms
for the wiki and not maintain distinct lists of users!

And I would switch to gitolite following the kernel's lead here for serving
git, and we could do that with relatively little effort (pickup all the pub
keys people have, populate gitolite keysdir and copy the raw git repos).

This would also allow us to restrict shell access to only those that ask for
it and thus limit our security exposure (best practice).

Then we'd only have one process for registration, hand over your pub key,
and you get immediate access to git, including the wikis now stored in git.
 
>>>> When you say "activity" do you mean that users would be impacted
>>>> once every 8 months for a small window of downtime?
>>>
>>> Probably, if nothing goes wrong.  And if there's an incompatible
>>> rebase of python or php or whatever, then we (?) have to fix the
>>> thing.  There is a downside to upgrade churn, in terms of our
>>> manpower.
>>
>> Best practice is to have a production and development server to
>> avoid these kinds of problems and shake out such changes.
>> Do we have 2 servers now?
> 
> rawhide level upgrade churn has to bring tangible value that is in
> excess of the costs (to all the parties, not just users).

I agree completely. I don't suggest we run Rawhide at all.
 
>> In which case RHEL8 with ELS as an option is even more valuable?
> 
>> In summary:
>> - Why not stick with RHEL8 to get ELS and avoid CentOS 8 early phase out.
> 
> Already said I'll look into moving from centos8 to rhel8.  centos
> lifespan does not look materially different from rhel.

CentOS 8 is EOL when RHEL 8 ELS starts. RHEL 8's lifespan is 10 years,
with RHEL 8 ELS adding another 4-5 years of support. That's almost a
50% increase in lifespan to schedule and upgrade. That looks materially
different to me and impactful for managing the upgrade.

CentOS 8 forces you to keep upgrading through the changing y-stream.
Some y-stream upgrades for AppStream may be rebases of pakcages
that break your deployment.

With ELS you can camp out on RHEL 8.4 *without* upgrading to
RHEL 8.5/8.6/8.7, and then upgrade to RHEL 8.8. It gives you the
flexibility to camp out on an ELS release and schedule around
the upgrade.

Please review the following details:
https://access.redhat.com/support/policy/updates/errata

We would have to ask Red Hat for licenses.

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Lukas Berk]

Hi,

Carlos O'Donell <carlos@redhat.com> writes:
> On 2/6/20 1:05 AM, Christopher Faylor wrote:
[...]
>> I think I've just convinced myself to look into using postfix and
>> mailman.  I'll try to do that in the next few days.
>
> I think switching  to postfix would really be to our benefit, both
> from an upstream support, and long-term perspective.
>
> The archives will be an interesting challenge, but I think we can
> preserve them statically.
>
> Have you seen public inbox? It's amazingly fast and the kernel is
> using it now. There is even a libc-alpha archive:
> https://public-inbox.org/libc-alpha/

Weren't we just discussing (in a parallel thread) the maintenance cost
and additional overhead of using out-of-distro bits and pieces?  We
should be careful and use the same evaluation when proposing to introduce
further new pieces.

Cheers,
Lukas

Re: Choice of distribution for new sourceware.org server?

[Frank Ch. Eigler]

Hi -

> Awesome. Do you need any help with regular evaluations? Is there a meeting
> for that? Or is this tactically driven when something can't be upgraded?

We evaluate when people want to evaluate, as we are doing right now.

> I'm surprised the wiki is non-distro software. I recently setup a dokuwiki
> backed by git, and it was all distro-software with just configuration data.

What's a "dokuwiki"?  Yum list doesn't know it.  We use ikiwiki (a
git-backed wiki system) for one of the wikis on sourceware, and I like
it a lot.  ikiwiki is not a RHEL product either.


> Moinmoin is basically dead upstream.
> Dokuwiki is actively maintained and there are migration paths from moinmoin
> to dokuwiki.
> And I would switch to gitolite [...]

The way that we can make use of ideas like this is for someone to go
beyond suggest them to actually try things out, consult with the
respective user base, install, popularize, transition, watch over it
all, and ideally help shut it down when that time comes.  IOW, there
is no shortage of other tools we -could- run.  What is short is
interest & commitment in doing all that work.


> This would also allow us to restrict shell access to only those that
> ask for it and thus limit our security exposure (best practice).

Our shell access is already restricted.


> [RHEL8 vs CentOS]

Luckily, we have until 2029 to work through the details either way.


- FChE

Re: Choice of distribution for new sourceware.org server?

[Christopher Faylor]

On Thu, Feb 06, 2020 at 09:50:20AM -0500, Carlos O'Donell wrote:
>On 2/6/20 1:05 AM, Christopher Faylor wrote:
>> I think I've just convinced myself to look into using postfix and
>> mailman.  I'll try to do that in the next few days.
>
>I think switching  to postfix would really be to our benefit, both
>from an upstream support, and long-term perspective.

I'm very familiar with and very lukewarm about Postfix but it's what
everyone uses so...

>The archives will be an interesting challenge, but I think we can
>preserve them statically.

IMO, for support purposes it will be important to have a unified archive
interface.

cgf

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/6/20 10:10 AM, Lukas Berk wrote:
> Hi,
> 
> Carlos O'Donell <carlos@redhat.com> writes:
>> On 2/6/20 1:05 AM, Christopher Faylor wrote:
> [...]
>>> I think I've just convinced myself to look into using postfix and
>>> mailman.  I'll try to do that in the next few days.
>>
>> I think switching  to postfix would really be to our benefit, both
>> from an upstream support, and long-term perspective.
>>
>> The archives will be an interesting challenge, but I think we can
>> preserve them statically.
>>
>> Have you seen public inbox? It's amazingly fast and the kernel is
>> using it now. There is even a libc-alpha archive:
>> https://public-inbox.org/libc-alpha/
> 
> Weren't we just discussing (in a parallel thread) the maintenance cost
> and additional overhead of using out-of-distro bits and pieces?  We
> should be careful and use the same evaluation when proposing to introduce
> further new pieces.

True! Welcome to the review committee! ;-)

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/6/20 10:19 AM, Frank Ch. Eigler wrote:
> Hi -
> 
>> Awesome. Do you need any help with regular evaluations? Is there a meeting
>> for that? Or is this tactically driven when something can't be upgraded?
> 
> We evaluate when people want to evaluate, as we are doing right now.

Understood.

>> I'm surprised the wiki is non-distro software. I recently setup a dokuwiki
>> backed by git, and it was all distro-software with just configuration data.
> 
> What's a "dokuwiki"?  Yum list doesn't know it.  We use ikiwiki (a
> git-backed wiki system) for one of the wikis on sourceware, and I like
> it a lot.  ikiwiki is not a RHEL product either.

It's in Fedora, I had not checked for availability in RHEL 8 AppStream or EPEL.

dnf search dokuwiki
Last metadata expiration check: 1:36:11 ago on Thu 06 Feb 2020 08:46:10 AM EST.
=========================== Name Exactly Matched: dokuwiki ===========================
dokuwiki.noarch : Standards compliant simple to use wiki
========================== Name & Summary Matched: dokuwiki ==========================
dokuwiki-selinux.noarch : SELinux support for dokuwiki

Dokuwiki is an actively maintained wiki with plugins and good internationalization.
https://www.dokuwiki.org/dokuwiki

ikiwiki doesn't seem as feature rich as dokuwiki, nor does it seem to have as active
a development community, or wide usage? The plugins for dokuwiki are quite useful
and the community has support for git, markdown, captchas, etc.
 
>> Moinmoin is basically dead upstream.
>> Dokuwiki is actively maintained and there are migration paths from moinmoin
>> to dokuwiki.
>> And I would switch to gitolite [...]
> 
> The way that we can make use of ideas like this is for someone to go
> beyond suggest them to actually try things out, consult with the
> respective user base, install, popularize, transition, watch over it
> all, and ideally help shut it down when that time comes.  IOW, there
> is no shortage of other tools we -could- run.  What is short is
> interest & commitment in doing all that work.

Understood.

>> This would also allow us to restrict shell access to only those that
>> ask for it and thus limit our security exposure (best practice).
> 
> Our shell access is already restricted.

Can non-administrators login to the system via ssh? Why?
 
>> [RHEL8 vs CentOS]
> 
> Luckily, we have until 2029 to work through the details either way.

I don't follow. Could you expand on this please?

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Christopher Faylor]

On Thu, Feb 06, 2020 at 11:36:33AM -0500, Carlos O'Donell wrote:
>On 2/6/20 10:19 AM, Frank Ch. Eigler wrote:
>>> This would also allow us to restrict shell access to only those that
>>> ask for it and thus limit our security exposure (best practice).
>> 
>> Our shell access is already restricted.
>
>Can non-administrators login to the system via ssh? Why?

non-administrators can't login to the system.  That's what Frank means
by "Our shell access is already restricted".

cgf

Re: Choice of distribution for new sourceware.org server?

[Frank Ch. Eigler]

Hi -


> > Our shell access is already restricted.
> 
> Can non-administrators login to the system via ssh? Why?

Many people only with repo commit privileges log on via ssh to a
restricted shell that lets them do only a few specific things.  ssh
access has been required because of multiple version control systems
that accept implicit (logged on uid) authentication only, and to do
other account management chores (change email forwarding or ssh
pubkey).


> >> [RHEL8 vs CentOS]
> > 
> > Luckily, we have until 2029 to work through the details either way.
> 
> I don't follow. Could you expand on this please?

Assuming online conversion to rhel8 is possible, we don't have to do
it before 2029, in order to get the extended-life-support coverage.


- FChE

Re: Choice of distribution for new sourceware.org server?

[Carlos O'Donell]

On 2/6/20 10:20 AM, Christopher Faylor wrote:
> On Thu, Feb 06, 2020 at 09:50:20AM -0500, Carlos O'Donell wrote:
>> On 2/6/20 1:05 AM, Christopher Faylor wrote:
>>> I think I've just convinced myself to look into using postfix and
>>> mailman.  I'll try to do that in the next few days.
>>
>> I think switching  to postfix would really be to our benefit, both
>>from an upstream support, and long-term perspective.
> 
> I'm very familiar with and very lukewarm about Postfix but it's what
> everyone uses so...

I have no strong opinions. I agree that everyone uses it. Postfix also
has good support for modern SPF/DKIM/DMARC/SRS etc because others have
run into this already. It didn't seem that straight forward to do all
of this with qmail (except with some hackish scripts).

-- 
Cheers,
Carlos.

Re: Choice of distribution for new sourceware.org server?

[Joseph Myers]

On Thu, 6 Feb 2020, Christopher Faylor wrote:

> We talk about that every time we go through a system or OS upgrade.
> I've maintained a postfix/mailman system in the past so maybe those bits
> would swap back in.  I don't really relish the thought of trying to
> convert the ezmlm archives to something else but maybe it isn't a big
> deal.

mailman has its own issues.  mailman2 is written in Python 2 which is EOL 
(though <https://pagure.io/fesco/issue/2312> has someone saying they're 
working on a Python 3 port).  mailman3 is a complete rewrite and its web 
interface (subscription management and archives) has a huge pile of web 
application dependencies that are hard to ensure long term security 
support for on a stable distribution.

(mailman2's default archiver has its own issues - its email address 
munging renders patches to Texinfo documentation pretty unreadable, for 
example - but I don't think it's hard to link mailman, either 2 or 3, up 
to other archive software, such as the existing MHonArc setup.)

-- 
Joseph S. Myers
joseph@codesourcery.com