Shadow stack enabling from dynamic loader v/s kernel on exec

Shadow stack enabling from dynamic loader v/s kernel on exec

[Deepak Gupta]

I don't want to divert focus from patch specific comments on shadow
stack patches that're being
discussed in the mailing list. And that's starting this separate
thread about enabling the shadow
stack in the dynamic loader v/s kernel. I've put relevant folks in
"To" and "kernel" and "libc" in CC.

We've beaten this record many times but I think this is the first time
I am getting into weeds.

Per this https://lore.kernel.org/all/20220130211838.8382-1-rick.p.edgecombe@intel.com/,
all
the binaries that were marked with shadow stack are ready to break as
soon as new kernel enables
shadow stack by default based on ELF bit. And thus the reason to let
it be decided up in user
mode and making the kernel oblivious about this decision making during exec.

It looks like it was done because libc changes landed in userspace
binaries from major distros before
the kernel changes could be merged and kernel-user interface changed
as kernel changes matured.

Such an issue doesn't exist for non-x86 (at least risc-v because
that's what I am focussing on). And
have been wondering that if doing below would be a better choice:

- On `exec`, the kernel looks at the ELF bit and sets up a shadow stack
- Dynamic loader (or statically built binary) starts life with shadow stack
- Dynamic loader can disable shadow stack if it wants if it sees some
compat issues
  [This last step of figuring out compat issues, anyways dynamic
loader is performing today]

This has many advantages
- dynamic loaders (and static binary) are protected from loader
specific ROP attack in a small window
- stack and shadow stack are always balanced

One disadvantage I can see is that enabling the shadow stack is split
but I really don't see it as a big
disadvantage. Please note that enable/disable/get status prctls can
still exist. And thus user space still
has all the enabling / disabling control with itself depending on
configuration.

Larger question and opinion / input that I am seeking here is that
(from kernel / libc community)

"Was there any other reason other than supporting ELF binaries that
went ahead of kernel changes that
led to decision of delegating of shadow stack enabling in dynamic loader"

If there are other complications that can happen due to kernel
enabling of shadow stack based on ELF bits,
I would like to know about them.


-Deepak

Re: Shadow stack enabling from dynamic loader v/s kernel on exec

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

On Wed, Nov 22, 2023 at 04:19:51PM -0800, Deepak Gupta wrote:

> "Was there any other reason other than supporting ELF binaries that
> went ahead of kernel changes that
> led to decision of delegating of shadow stack enabling in dynamic loader"

> If there are other complications that can happen due to kernel
> enabling of shadow stack based on ELF bits,
> I would like to know about them.

This wouldn't play nicely with security policies that prevent disabling
the shadow stacks - it would be fine with the prctl() based locking but
something imposed externally with seccomp or similar would be there from
process start.

I'll also note that for arm64 BTI where we're less concerned with
compatibility (since the protection is per page we only need to make
sure that each ELF image is BTI enabled when we map it, we don't need to
worry about any further code that might be mapped/loaded) we only
enforce BTI for the dynamic loader, we still leave it to the dynamic
loader to remap the main executable as BTI.  The architecture
maintainers have a strong preference for delegating as much as possible
to userspace in order to reduce the potential for being locked into an
unwanted policy or having difficulty in working around breakage.  The
issues on x86 are an example of the sort of situation people are worried
about seeing in future.  I personally would be OK with directly
interpreting the ELF markings there but it wasn't the consensus.

On arm64 there would be the potential for disrupting some limited and
theoretical use cases where GCS is enabled even though some libraries do
not support it, we don't allow GCS to be reenabled for a thread after it
has been disabled in order to avoid dealing with the issues around
reinitiating the GCS for something that's a corner case.  x86 does allow
reenabling so wouldn't have that issue.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Shadow stack enabling from dynamic loader v/s kernel on exec

[Edgecombe, Rick P]

On Sat, 2023-11-25 at 11:35 +0000, Mark Brown wrote:
> On Wed, Nov 22, 2023 at 04:19:51PM -0800, Deepak Gupta wrote:
> 
> > "Was there any other reason other than supporting ELF binaries that
> > went ahead of kernel changes that
> > led to decision of delegating of shadow stack enabling in dynamic
> > loader"
> 
> > If there are other complications that can happen due to kernel
> > enabling of shadow stack based on ELF bits,
> > I would like to know about them.
> 
> This wouldn't play nicely with security policies that prevent
> disabling
> the shadow stacks - it would be fine with the prctl() based locking
> but
> something imposed externally with seccomp or similar would be there
> from
> process start.

IIRC we discussed this at some point. Yea, the downside is that exec
doesn't know if all the linked libraries will have shadow stack. So for
compatibility, it would need to leave the "disable" prctl() available
for the loader to use.

On x86 there was discussion of having a WRSS elf bit. (I mention it
towards the similar features on the other archs) But this bit would
work differently. Instead of enabling it if ALL of the DSOs supported
shadow stack, WRSS would be enabled if ANY of the DSOs needed WRSS.
Exec-time enabling would have trouble knowing what to do about this. It
could leave it unlocked to pick later, but this doesn't seem great in a
security focused runtime environment. So a seccomp mode starts to seem
like a separate enabling mode with it's own rules, in which case it
could be left for the future.

I don't think doing exec based enabling will impact the app developers
expectations, because it should be confined to the loader. So it's fine
either way from my perspective.

> 
> I'll also note that for arm64 BTI where we're less concerned with
> compatibility (since the protection is per page we only need to make
> sure that each ELF image is BTI enabled when we map it, we don't need
> to
> worry about any further code that might be mapped/loaded) we only
> enforce BTI for the dynamic loader, we still leave it to the dynamic
> loader to remap the main executable as BTI.  The architecture
> maintainers have a strong preference for delegating as much as
> possible
> to userspace in order to reduce the potential for being locked into
> an
> unwanted policy or having difficulty in working around breakage.  The
> issues on x86 are an example of the sort of situation people are
> worried
> about seeing in future.  I personally would be OK with directly
> interpreting the ELF markings there but it wasn't the consensus.
> 
> On arm64 there would be the potential for disrupting some limited and
> theoretical use cases where GCS is enabled even though some libraries
> do
> not support it

On x86 we see this case already in testing. Why do you think it is
theoretical?

Re: Shadow stack enabling from dynamic loader v/s kernel on exec

[Edgecombe, Rick P]

On Wed, 2023-11-22 at 16:19 -0800, Deepak Gupta wrote:
> I don't want to divert focus from patch specific comments on shadow
> stack patches that're being
> discussed in the mailing list. And that's starting this separate
> thread about enabling the shadow
> stack in the dynamic loader v/s kernel. I've put relevant folks in
> "To" and "kernel" and "libc" in CC.

Thanks. As we look at adding some final glibc support, I've wondered if
there might be enough topics to warrant an occasional meeting to
discuss stuff like this. I'd also like to discuss the shadow stack life
cycle issues (uncontext, etc), alt shadow stacks and all of the
compatibility last mile problems. Towards the goal of avoiding
unnecessary divergence on app developer expectations.

> This has many advantages
> - dynamic loaders (and static binary) are protected from loader
> specific ROP attack in a small window

Loaders can call the prctl() as the first step, so the loader is
protected. The x86 glibc patches did this at one point. So the prctl
supports enabling at pretty close to either point, "exec time" or later
in the loader process. Enabling before the first CALL (or unwound to
that point) leaves the shadow stack's balanced.

I think the main disadvantage are:
 - *Maybe* it requires duplication for the seccomp use case
 - Requires mapping, then unmapping shadow stack for cases of
incompatible DSO or disable via TUNABLE

It is probably worth noting, the old elf bit based enabling would
enable shadow stack if the *loader* DSO had the elf bit set. Then the
loader would check the elf bit of the execing binary, and disable
shadow stack if not supported. This means with shadow stack enabled
kernel and glibc, all legacy apps would be subjected to a map and unmap
of the shadow stack. It probably isn't the biggest deal, but it's nice
to avoid.

Re: Shadow stack enabling from dynamic loader v/s kernel on exec

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On Mon, Nov 27, 2023 at 05:41:16PM +0000, Edgecombe, Rick P wrote:
> On Wed, 2023-11-22 at 16:19 -0800, Deepak Gupta wrote:

> Thanks. As we look at adding some final glibc support, I've wondered if
> there might be enough topics to warrant an occasional meeting to
> discuss stuff like this. I'd also like to discuss the shadow stack life
> cycle issues (uncontext, etc), alt shadow stacks and all of the
> compatibility last mile problems. Towards the goal of avoiding
> unnecessary divergence on app developer expectations.

It'd definitely be good to ensure we're coordinating well, I'd be
willing to give it a go.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Shadow stack enabling from dynamic loader v/s kernel on exec

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Mon, Nov 27, 2023 at 05:32:24PM +0000, Edgecombe, Rick P wrote:

> security focused runtime environment. So a seccomp mode starts to seem
> like a separate enabling mode with it's own rules, in which case it
> could be left for the future.

My understanding is that seccomp is generic enough to allow you to write
filters without any specific shadow stack support, though I'm not sure
about handling for arch_prctl() so perhaps it's harder on x86.

> I don't think doing exec based enabling will impact the app developers
> expectations, because it should be confined to the loader. So it's fine
> either way from my perspective.

Yes, I don't think there's an issue for apps either way - it's more an
issue for people doing system level security.

> > On arm64 there would be the potential for disrupting some limited and
> > theoretical use cases where GCS is enabled even though some libraries
> > do
> > not support it

> On x86 we see this case already in testing. Why do you think it is
> theoretical?

Right, it's fairly easy to add something not flagged as GCS compatible
at runtime through dlopen().  I think I was thinking of the case where
the program is not marked as supporting GCS but enables GCS usage
selectively at runtime, it wouldn't be able to do that for the main
thread since we don't allow reenabling.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Shadow stack enabling from dynamic loader v/s kernel on exec

[Deepak Gupta]

On Tue, Nov 28, 2023 at 12:51:44PM +0000, Mark Brown wrote:
>On Mon, Nov 27, 2023 at 05:41:16PM +0000, Edgecombe, Rick P wrote:
>> On Wed, 2023-11-22 at 16:19 -0800, Deepak Gupta wrote:
>
>> Thanks. As we look at adding some final glibc support, I've wondered if
>> there might be enough topics to warrant an occasional meeting to
>> discuss stuff like this. I'd also like to discuss the shadow stack life
>> cycle issues (uncontext, etc), alt shadow stacks and all of the
>> compatibility last mile problems. Towards the goal of avoiding
>> unnecessary divergence on app developer expectations.
>
>It'd definitely be good to ensure we're coordinating well, I'd be
>willing to give it a go.

Same here.