Intel microcode update and glibc HLE

Intel microcode update and glibc HLE

[Siddhesh Poyarekar]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Hi,

The microcode_ctl package recently updated itself to include the
latest intel microcode update that disables HLE.  This is resulting in
unbootable Haswell systems on Fedora[1] and elsewhere.  The problem
seems to be that the microcode update is applied a bit late during
every boot.  Due to this, the kernel has stale CPU capabilities and
systemd sees HLE enabled before the microcode update is applied.
Later, HLE is disabled and the next pthread operation in systemd dies
with a SIGILL.

The ideal solution for this would probably be to apply the microcode
update as early as possible during boot, but things get complicated
with suspend-resume or hibernate, so it's not very simple.  In Fedora
we're now rebuilding glibc with elision disabled again since keeping
it enabled is useless and is currently harmful.

Siddhesh

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1146967

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: Intel microcode update and glibc HLE

[Adhemerval Zanella]

On 26-09-2014 15:09, Siddhesh Poyarekar wrote:
> Hi,
>
> The microcode_ctl package recently updated itself to include the
> latest intel microcode update that disables HLE.  This is resulting in
> unbootable Haswell systems on Fedora[1] and elsewhere.  The problem
> seems to be that the microcode update is applied a bit late during
> every boot.  Due to this, the kernel has stale CPU capabilities and
> systemd sees HLE enabled before the microcode update is applied.
> Later, HLE is disabled and the next pthread operation in systemd dies
> with a SIGILL.
>
> The ideal solution for this would probably be to apply the microcode
> update as early as possible during boot, but things get complicated
> with suspend-resume or hibernate, so it's not very simple.  In Fedora
> we're now rebuilding glibc with elision disabled again since keeping
> it enabled is useless and is currently harmful.
>
> Siddhesh
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1146967

I am about to ping about my patches to add support of TLE on POWER8, and
with this recent issue with HLE on Intel lead to an idea of creating
a patch to enabled TLE conditionally using an environment variable.  The
idea is to build GLIBC with TLE enabled, add the environment variable
and let architecture work based on its value.

I know this maybe not the best solution in long term if we aim to enable
TLE as default (we will need to either deprecate the env. var or change its
semantics), but my idea is to let developers a way to evaluate either if
TLE is safe and if it indeed gives the workload a performance boost.

Comments?

Re: Intel microcode update and glibc HLE

[Josh Boyer]

On Fri, Sep 26, 2014 at 2:09 PM, Siddhesh Poyarekar <siddhesh@redhat.com> wrote:
> Hi,
>
> The microcode_ctl package recently updated itself to include the
> latest intel microcode update that disables HLE.  This is resulting in
> unbootable Haswell systems on Fedora[1] and elsewhere.  The problem
> seems to be that the microcode update is applied a bit late during
> every boot.  Due to this, the kernel has stale CPU capabilities and
> systemd sees HLE enabled before the microcode update is applied.
> Later, HLE is disabled and the next pthread operation in systemd dies
> with a SIGILL.

I was under the impression that the glibc implementation was looking
at the cpuid registers directly for the HLE/RTM support.  If it was
looking at cpuflags then the kernel could have probably hidden this
with a quirk.

> The ideal solution for this would probably be to apply the microcode
> update as early as possible during boot, but things get complicated
> with suspend-resume or hibernate, so it's not very simple.  In Fedora
> we're now rebuilding glibc with elision disabled again since keeping
> it enabled is useless and is currently harmful.

I'll be updating the Fedora kernel and dracut to do early microcode
loading today.  Kyle McMartin looked at the suspend/hibernate-resume
paths and things should work fine with early microcode loading.

Disabling this in glibc on x86 seems fine since nothing will be
capable of using it until newer CPUs are released, but it wasn't
strictly necessary.

josh

Re: Intel microcode update and glibc HLE

[Siddhesh Poyarekar]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

On Fri, Sep 26, 2014 at 02:28:13PM -0400, Josh Boyer wrote:
> I was under the impression that the glibc implementation was looking
> at the cpuid registers directly for the HLE/RTM support.  If it was
> looking at cpuflags then the kernel could have probably hidden this
> with a quirk.

It looks at the cpuid registers directly.

> I'll be updating the Fedora kernel and dracut to do early microcode
> loading today.  Kyle McMartin looked at the suspend/hibernate-resume
> paths and things should work fine with early microcode loading.

That's great.

> Disabling this in glibc on x86 seems fine since nothing will be
> capable of using it until newer CPUs are released, but it wasn't
> strictly necessary.

I wasn't aware of the plan to do the early microcode loading, which is
why we went with disabling HLE.  It's really just a flip of a
configure flag and rebuild, so we can even revert it and test once the
kernel bits are in.

Siddhesh

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: Intel microcode update and glibc HLE

[Siddhesh Poyarekar]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On Fri, Sep 26, 2014 at 03:19:34PM -0300, Adhemerval Zanella wrote:
> I am about to ping about my patches to add support of TLE on POWER8, and
> with this recent issue with HLE on Intel lead to an idea of creating
> a patch to enabled TLE conditionally using an environment variable.  The
> idea is to build GLIBC with TLE enabled, add the environment variable
> and let architecture work based on its value.
> 
> I know this maybe not the best solution in long term if we aim to enable
> TLE as default (we will need to either deprecate the env. var or change its
> semantics), but my idea is to let developers a way to evaluate either if
> TLE is safe and if it indeed gives the workload a performance boost.

We've thrown around the idea of using tunables (yes, that abstract
vapourware that we've mentioned before and never written code for.
some day...) for things like elision, so your idea matches with that
in principle.

Siddhesh

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: Intel microcode update and glibc HLE

[Carlos O'Donell]

On 09/26/2014 02:54 PM, Siddhesh Poyarekar wrote:
>> I'll be updating the Fedora kernel and dracut to do early microcode
>> loading today.  Kyle McMartin looked at the suspend/hibernate-resume
>> paths and things should work fine with early microcode loading.
> 
> That's great.

I agree, that is great news.
 
>> Disabling this in glibc on x86 seems fine since nothing will be
>> capable of using it until newer CPUs are released, but it wasn't
>> strictly necessary.
> 
> I wasn't aware of the plan to do the early microcode loading, which is
> why we went with disabling HLE.  It's really just a flip of a
> configure flag and rebuild, so we can even revert it and test once the
> kernel bits are in.

It turns out to be more than just a configure flag.

Andi's recent rwlock changes use TSX unconditionally.

I've raised this issue with Andi to see if we can put it all under the
same configure flag.

Similarly for ppc64 and s390 I think I'll make the flag do this:

--enable-lock-elision=yes (enable for all machines)
--enable-lock-elisoin=x86_64,ppc64,s390x (enable for these machines)
--enable-lock-elision-no (disable for all machines)

Cheers,
Carlos.

Re: Intel microcode update and glibc HLE

[Carlos O'Donell]

On 09/27/2014 01:10 AM, Carlos O'Donell wrote:
> Andi's recent rwlock changes use TSX unconditionally.

To be clear I meant "use TSX unconditionally if present."

c.

Re: Intel microcode update and glibc HLE

[Siddhesh Poyarekar]

On 27 September 2014 10:40, Carlos O'Donell <carlos@redhat.com> wrote:
> It turns out to be more than just a configure flag.
>
> Andi's recent rwlock changes use TSX unconditionally.

I would reckon that it's wrong to do so. Not using
--enable-lock-elision should disable all elision code.

> I've raised this issue with Andi to see if we can put it all under the
> same configure flag.
>
> Similarly for ppc64 and s390 I think I'll make the flag do this:
>
> --enable-lock-elision=yes (enable for all machines)
> --enable-lock-elisoin=x86_64,ppc64,s390x (enable for these machines)

Why do you need this?  Wouldn't it be sufficient to do this in
distribution spec files?

Siddhesh
-- 
http://siddhesh.in

Re: Intel microcode update and glibc HLE

[Roland McGrath]

> On 27 September 2014 10:40, Carlos O'Donell <carlos@redhat.com> wrote:
> > It turns out to be more than just a configure flag.
> >
> > Andi's recent rwlock changes use TSX unconditionally.
> 
> I would reckon that it's wrong to do so. Not using
> --enable-lock-elision should disable all elision code.

Emphatically agreed.

> > I've raised this issue with Andi to see if we can put it all under the
> > same configure flag.
> >
> > Similarly for ppc64 and s390 I think I'll make the flag do this:
> >
> > --enable-lock-elision=yes (enable for all machines)
> > --enable-lock-elisoin=x86_64,ppc64,s390x (enable for these machines)
> 
> Why do you need this?  Wouldn't it be sufficient to do this in
> distribution spec files?

I agree.  There's no precedent for that sort of value for enable options.
It's just saving people conditionally choosing that configure arguments
they actually want.  They have scripting of various sorts for that.

Re: Intel microcode update and glibc HLE

[Carlos O'Donell]

On 09/27/2014 01:14 AM, Siddhesh Poyarekar wrote:
> On 27 September 2014 10:40, Carlos O'Donell <carlos@redhat.com> wrote:
>> It turns out to be more than just a configure flag.
>>
>> Andi's recent rwlock changes use TSX unconditionally.
> 
> I would reckon that it's wrong to do so. Not using
> --enable-lock-elision should disable all elision code.

I agree.

>> Similarly for ppc64 and s390 I think I'll make the flag do this:
>>
>> --enable-lock-elision=yes (enable for all machines)
>> --enable-lock-elisoin=x86_64,ppc64,s390x (enable for these machines)
> 
> Why do you need this?  Wouldn't it be sufficient to do this in
> distribution spec files?

We don't need it. We can indeed push it into the distribution
to handle there. However, the remaining problem is that upstream
will want a sensible default per-machine. I guess we can look at
refactoring this such that the machines can influence the default.

For example as the Intel elision code is tested more it might
default to being enabled by default sooner, while the ppc64 and s390x
would not.

Cheers,
Carlos.

Re: Intel microcode update and glibc HLE

[Andreas Schwab]

"Carlos O'Donell" <carlos@redhat.com> writes:

> --enable-lock-elision-no (disable for all machines)

--disable-lock-elision

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Intel microcode update and glibc HLE

[Carlos O'Donell]

On 09/27/2014 03:37 AM, Andreas Schwab wrote:
> "Carlos O'Donell" <carlos@redhat.com> writes:
> 
>> --enable-lock-elision-no (disable for all machines)
> 
> --disable-lock-elision

Sure :-)

Cheers,
Carlos.