Requiring Linux 3.2, again

Requiring Linux 3.2, again

[Joseph Myers]

When we discussed moving to Linux 3.2 as the minimum kernel version 
requirement for glibc over a year ago, concerns were expressed about how 
this would affect some containers 
<https://sourceware.org/ml/libc-alpha/2016-02/threads.html#00173> and we 
only had consensus for a change for architectures other than x86 / x86_64.

Now that more than a year has passed and 2.6.32 has been EOL for a year 
more, do people still care about running distributions from late 2017 or 
later on such old kernels, or can we now move to a 3.2 minimum globally?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Requiring Linux 3.2, again

[Florian Weimer]

* Joseph Myers:

> When we discussed moving to Linux 3.2 as the minimum kernel version 
> requirement for glibc over a year ago, concerns were expressed about how 
> this would affect some containers 
> <https://sourceware.org/ml/libc-alpha/2016-02/threads.html#00173> and we 
> only had consensus for a change for architectures other than x86 / x86_64.
>
> Now that more than a year has passed and 2.6.32 has been EOL for a year 
> more, do people still care about running distributions from late 2017 or 
> later on such old kernels, or can we now move to a 3.2 minimum globally?

Parallels/OpenVZ seems to have a 3.10-based kernel now:

| * RHEL7 (3.10+) kernel. 

<https://docs.openvz.org/openvz_readme.webhelp/_what_8217_s_new.html>

I don't know how far this release has propagated to service providers.
But I don't think this should fix glibc support at 2.6.32 anymore for
i386/x86_64.

Re: Requiring Linux 3.2, again

[Arnd Bergmann]

On Wed, May 3, 2017 at 1:26 PM, Joseph Myers <joseph@codesourcery.com> wrote:
> When we discussed moving to Linux 3.2 as the minimum kernel version
> requirement for glibc over a year ago, concerns were expressed about how
> this would affect some containers
> <https://sourceware.org/ml/libc-alpha/2016-02/threads.html#00173> and we
> only had consensus for a change for architectures other than x86 / x86_64.
>
> Now that more than a year has passed and 2.6.32 has been EOL for a year
> more, do people still care about running distributions from late 2017 or
> later on such old kernels, or can we now move to a 3.2 minimum globally?

I looked again at my list of distro packages from
https://sourceware.org/ml/libc-alpha/2016-02/msg00284.html

* Fedora is unchanged, passing --enable-kernel=2.6.32, matching the kernel
  in the RHEL6 release, EOL date for RHEL6 is 11/30/2020.

* Debian is unchanged, requiring 2.6.32 on x86 (32 and 64 bit) but 3.2
elsewhere.
  2.6.32 was used in Debian 6 (Squeeze) which is now EOL.

* Ubuntu now requires 3.2 on all architectures, changed from last year.

* OpenSUSE is unchanged, requiring 3.0 for compatibility with SLES11 kernels,
  SLES11 support ends 31 Mar 2019 or 31 Mar 2022 (for extended support)

* Arch Linux is unchanged, still using --enable-kernel=2.6.32

* Gentoo is unchanged, not passing --enable-kernel= on any architecture.

* Further, I found that "OpenVZ-legacy (RHEL6 based)" uses a 2.6.32 kernel
  and has a projected EOL date of Nov 2019. This was the latest release
  a year ago, but OpenVZ 7 was released in July 2016 and uses a 3.10 kernel.

Both SLES11 and RHEL6 support LXC containers, which like OpenVZ are in
theory able to run the latest x86 distros except for Ubuntu because of
the glibc kernel version check. OpenSUSE (and presumably the next
SLES release based on it) will run in LXC on SLES11 but not on RHEL6.

      Arnd

Re: Requiring Linux 3.2, again

[Carlos O'Donell]

On 05/03/2017 07:26 AM, Joseph Myers wrote:
> When we discussed moving to Linux 3.2 as the minimum kernel version 
> requirement for glibc over a year ago, concerns were expressed about how 
> this would affect some containers 
> <https://sourceware.org/ml/libc-alpha/2016-02/threads.html#00173> and we 
> only had consensus for a change for architectures other than x86 / x86_64.
> 
> Now that more than a year has passed and 2.6.32 has been EOL for a year 
> more, do people still care about running distributions from late 2017 or 
> later on such old kernels, or can we now move to a 3.2 minimum globally?
 
We're fine going to linux 3.2.

TLDR;

At a high level I worry about container technology (regardless of the
choice of framework around namespaces) and the pressure we apply to users
by moving glibc to require a newer kernel.

Every time we move glibc to require a newer linux kernel that creates an
inflection point where users have to upgrade their container host to run
the newer distribution containers.

I don't have any good data about what this looks like and the Parallels/OpenVZ
is a singular data point, and not very conclusive in my opinion, but we took
a conservative approach and waited, and that's probably right.

Looking at the OpenShift Origin server as another example, it already requires
RHEL 7.3 (linux 3.10), so I see no problem there. Similarly Project Atomic also
has as a requirement either CentOS7 or Fedora25, both of which are linux 3.10
or newer. The technology is moving quickly on the container host front.

What about the next step though? And the next? Moving beyond linux 3.10 would
mean that any future RHEL which is rebased to the newest glibc would not be
able to easily run in a container on RHEL 7 using a RHEL 7 kernel. 
Taking note that RHEL7 has support until 2024 at a minimum.

Do we harm the adoption of new glibc versions because of container requirements?

This isn't a new problem, and I've been educating people as much as I can about
glibc's minimum kernel requirements. Perhaps we'll succeed at embedding in the
common knowledge that you need a newer host kernel than the containers guests
expect and just plan for that to be the case that gives you the least problems
and the best compatibility overall (short of running with an exact match of
expectations).

-- 
Cheers,
Carlos.

Re: Requiring Linux 3.2, again

[Joseph Myers]

On Thu, 4 May 2017, Carlos O'Donell wrote:

> What about the next step though? And the next? Moving beyond linux 3.10 would
> mean that any future RHEL which is rebased to the newest glibc would not be
> able to easily run in a container on RHEL 7 using a RHEL 7 kernel. 
> Taking note that RHEL7 has support until 2024 at a minimum.

My view is that we base things on the oldest supported kernel version 
listed at https://www.kernel.org/category/releases.html - remembering that 
distributions supporting a distribution kernel long term may choose to 
contribute to maintaining the upstream kernel for the community, and this 
may mean some of the kernels listed there in fact live on with other 
maintainers beyond the dates currently shown there.

Based on the dates shown there, we might consider moving from a 3.2 
minimum to a 3.16 minimum for glibc 2.28 (Aug 2018) or 2.29 (Feb 2019).  
But actually it doesn't look like there's any difference between 3.2 and 
3.16 for features glibc cares about on x86 / x86_64 (whereas there *are* 
cleanups possible by moving from 2.6.32 to 3.2), so there would be very 
little cost to keeping the version back on some architectures (while 
moving to require 3.16 on other architectures) if that makes sense at the 
time.  And of course if 3.10, say, is in fact still supported when 3.2 
ceases to be, that rather than 3.16 would be the version to consider.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Requiring Linux 3.2, again

[Joseph Myers]

On Thu, 4 May 2017, Carlos O'Donell wrote:

> At a high level I worry about container technology (regardless of the
> choice of framework around namespaces) and the pressure we apply to users
> by moving glibc to require a newer kernel.
> 
> Every time we move glibc to require a newer linux kernel that creates an
> inflection point where users have to upgrade their container host to run
> the newer distribution containers.

Even without glibc requiring a newer kernel, it's entirely plausible that 
some userspace software on new distributions will in fact be relying on 
new kernel features and so fail to work on containers with old host 
kernels unless those features have been backported.  E.g., if it uses 
getrandom (Linux 3.17) and doesn't have a runtime fallback for it being 
unavailable (only maybe e.g. a configure-time fallback for being linked 
with old glibc without the function).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Requiring Linux 3.2, again

[Carlos O'Donell]

On 05/04/2017 07:47 AM, Joseph Myers wrote:
> On Thu, 4 May 2017, Carlos O'Donell wrote:
> 
>> At a high level I worry about container technology (regardless of the
>> choice of framework around namespaces) and the pressure we apply to users
>> by moving glibc to require a newer kernel.
>>
>> Every time we move glibc to require a newer linux kernel that creates an
>> inflection point where users have to upgrade their container host to run
>> the newer distribution containers.
> 
> Even without glibc requiring a newer kernel, it's entirely plausible that 
> some userspace software on new distributions will in fact be relying on 
> new kernel features and so fail to work on containers with old host 
> kernels unless those features have been backported.  E.g., if it uses 
> getrandom (Linux 3.17) and doesn't have a runtime fallback for it being 
> unavailable (only maybe e.g. a configure-time fallback for being linked 
> with old glibc without the function).
 
Yes, but in that case it's just *one* application which fails at runtime,
and the application author is considered at fault.

If glibc causes the container to even boot before allowing applications to
run, then we get blamed, and the politics of the situation are different.

What stops us from simply allowing individual interfaces to fail according
to the results of the syscalls they make? Or in some cases just calling
abort if the syscall is critical for operation of glibc? This would move
glibc out of the path of blame, and make it the fault of the developer?

Assume your application only used memset. With a bumped up minimum kernel
version we would no longer run that application on an older host. Shouldn't
we just allow it to run?

Now that the minimum kernel version is moving forward smoothly and we all
like the cleanups. Perhaps it's time to question the "FATAL: too old" warning
and remove it?

-- 
Cheers,
Carlos.

Re: Requiring Linux 3.2, again

[Carlos O'Donell]

On 05/04/2017 07:24 AM, Joseph Myers wrote:
> On Thu, 4 May 2017, Carlos O'Donell wrote:
> 
>> What about the next step though? And the next? Moving beyond linux 3.10 would
>> mean that any future RHEL which is rebased to the newest glibc would not be
>> able to easily run in a container on RHEL 7 using a RHEL 7 kernel. 
>> Taking note that RHEL7 has support until 2024 at a minimum.
> 
> My view is that we base things on the oldest supported kernel version 
> listed at https://www.kernel.org/category/releases.html - remembering that 
> distributions supporting a distribution kernel long term may choose to 
> contribute to maintaining the upstream kernel for the community, and this 
> may mean some of the kernels listed there in fact live on with other 
> maintainers beyond the dates currently shown there.

That's a very good argument, and a similar argument can be made for FSF
supported versions of GCC and I can't argue against that.
 
> Based on the dates shown there, we might consider moving from a 3.2 
> minimum to a 3.16 minimum for glibc 2.28 (Aug 2018) or 2.29 (Feb 2019).  
> But actually it doesn't look like there's any difference between 3.2 and 
> 3.16 for features glibc cares about on x86 / x86_64 (whereas there *are* 
> cleanups possible by moving from 2.6.32 to 3.2), so there would be very 
> little cost to keeping the version back on some architectures (while 
> moving to require 3.16 on other architectures) if that makes sense at the 
> time.  And of course if 3.10, say, is in fact still supported when 3.2 
> ceases to be, that rather than 3.16 would be the version to consider.

I think I've outlined another argument in my downstream email. That the
abort we issue for being on too old a kernel may be too catastropic here
and that there might be a middle ground where such applications could work
and we don't let them.

-- 
Cheers,
Carlos.

Re: Requiring Linux 3.2, again

[Joseph Myers]

On Thu, 4 May 2017, Carlos O'Donell wrote:

> Now that the minimum kernel version is moving forward smoothly and we all
> like the cleanups. Perhaps it's time to question the "FATAL: too old" warning
> and remove it?

I think that would be a reasonable approach (so --enable-kernel affects 
the notes in binaries indicating the required kernel version, and disables 
runtime fallback code, but doesn't stop programs running).  There might be 
a risk of security issues from code that doesn't expect or allow for 
affected interfaces to fail, however.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Requiring Linux 3.2, again

[Joseph Myers]

On Thu, 4 May 2017, Joseph Myers wrote:

> I think that would be a reasonable approach (so --enable-kernel affects 
> the notes in binaries indicating the required kernel version, and disables 
> runtime fallback code, but doesn't stop programs running).  There might be 

 ... which would incidentally mean, as regards what programs work, that 
all --enable-kernel versions 3.1 and above are exactly the same on x86_64, 
and all from 3.0 to 4.2 are the same on i386, since no features with 
--enable-kernel conditionals on those architectures were added in those 
ranges (there's code using newer syscalls such as getrandom, but not with 
any --enable-kernel fallbacks).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Requiring Linux 3.2, again

[Joseph Myers]

On Thu, 4 May 2017, Carlos O'Donell wrote:

> > Based on the dates shown there, we might consider moving from a 3.2 
> > minimum to a 3.16 minimum for glibc 2.28 (Aug 2018) or 2.29 (Feb 2019).  
> > But actually it doesn't look like there's any difference between 3.2 and 
> > 3.16 for features glibc cares about on x86 / x86_64 (whereas there *are* 
> > cleanups possible by moving from 2.6.32 to 3.2), so there would be very 
> > little cost to keeping the version back on some architectures (while 
> > moving to require 3.16 on other architectures) if that makes sense at the 
> > time.  And of course if 3.10, say, is in fact still supported when 3.2 
> > ceases to be, that rather than 3.16 would be the version to consider.
> 
> I think I've outlined another argument in my downstream email. That the
> abort we issue for being on too old a kernel may be too catastropic here
> and that there might be a middle ground where such applications could work
> and we don't let them.

Empirically, userspace QEMU, told to fake a newer kernel version than is 
actually running on the host, works quite well for running e.g. GCC tests 
(though some libstdc++ threading tests fall over, since threading doesn't 
work very well with userspace QEMU).  I don't know to what extent it does 
any emulation of syscalls or just converts them to the corresponding host 
syscalls, but if it converts to (possibly missing) host syscalls that 
doesn't seem fatal for those tests.

(I have wondered how much of the glibc testsuite could usefully run in 
userspace QEMU (and thus whether build-many-glibcs.py could support a mode 
running a subset of execution tests on userspace QEMU for architectures 
with the right support, as something between just running compilation 
tests as at present, and running the complete testsuite which would 
require booting an actual OS on the desired architecture).)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Requiring Linux 3.2, again

[Carlos O'Donell]

On 05/04/2017 01:00 PM, Joseph Myers wrote:
> On Thu, 4 May 2017, Carlos O'Donell wrote:
> 
>> Now that the minimum kernel version is moving forward smoothly and we all
>> like the cleanups. Perhaps it's time to question the "FATAL: too old" warning
>> and remove it?
> 
> I think that would be a reasonable approach (so --enable-kernel affects 
> the notes in binaries indicating the required kernel version, and disables 
> runtime fallback code, but doesn't stop programs running).  There might be 
> a risk of security issues from code that doesn't expect or allow for 
> affected interfaces to fail, however.
 
Correct, the NT_GNU_ABI_TAG would clearly specify that the application has
a compatibility requirement with the newer kernel, even if this is a loose
bound.

Then, rather than being prescriptive, the dynamic loader allows the 
application to run, and the user is responsible for ensuring they only use
those interfaces present in the kernel they are running on.

From a security perspective I think we can have __glibc_unlikely paths
that abort if critical syscalls fail, and those paths will be on cold
branches, optimized far away from the hot path. We would still obviously
have a check on the hot path for failure.

Making this decision to remove the fatal abort will change some of our 
day-to-day practice in this regard, but I think it's a change we need
to make to better support the more flexible interaction between container 
guests and the host kernel.

I'll do some testing and propose the change.

-- 
Cheers,
Carlos.

Re: Requiring Linux 3.2, again

[Andreas Schwab]

On Mai 04 2017, Joseph Myers <joseph@codesourcery.com> wrote:

> (I have wondered how much of the glibc testsuite could usefully run in 
> userspace QEMU (and thus whether build-many-glibcs.py could support a mode 
> running a subset of execution tests on userspace QEMU for architectures 
> with the right support, as something between just running compilation 
> tests as at present, and running the complete testsuite which would 
> require booting an actual OS on the desired architecture).)

See
https://build.opensuse.org/package/live_build_log/home:Andreas_Schwab:glibc/glibc-testsuite/a/armv6l
for current results.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Requiring Linux 3.2, again

[James Cloos]

>>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:

FW> Parallels/OpenVZ seems to have a 3.10-based kernel now:

FW> | * RHEL7 (3.10+) kernel. 

FW> <https://docs.openvz.org/openvz_readme.webhelp/_what_8217_s_new.html>

FW> I don't know how far this release has propagated to service providers.
FW> But I don't think this should fix glibc support at 2.6.32 anymore for
FW> i386/x86_64.

Mostly it has not propagated at all.  Sollus does not support it.

Openvz providers are expected to remain on openvz6 for some time to come.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6