[PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Mark Brown]

Deployments of BTI on arm64 have run into issues interacting with
systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
linked executables the kernel will only handle architecture specific
properties like BTI for the interpreter, the expectation is that the
interpreter will then handle any properties on the main executable.
For BTI this means remapping the executable segments PROT_EXEC |
PROT_BTI.

This interacts poorly with MemoryDenyWriteExecute since that is
implemented using a seccomp filter which prevents setting PROT_EXEC on
already mapped memory and lacks the context to be able to detect that
memory is already mapped with PROT_EXEC.  This series resolves this by
handling the BTI property for both the interpreter and the main
executable.

This does mean that we may get more code with BTI enabled if running on
a system without BTI support in the dynamic linker, this is expected to
be a safe configuration and testing seems to confirm that. It also
reduces the flexibility userspace has to disable BTI but it is expected
that for cases where there are problems which require BTI to be disabled
it is more likely that it will need to be disabled on a system level.

v8:
 - Rebase onto v5.17-rc1.
v7:
 - Rebase onto v5.16-rc1.
v6:
 - Rebase onto v5.15-rc1.
v5:
 - Rebase onto v5.14-rc2.
 - Tweak changelog on patch 1.
 - Use the helper for interpreter/executable flag in elf.h as well.
v4:
 - Rebase onto v5.14-rc1.
v3:
 - Fix passing of properties for parsing by the main executable.
 - Drop has_interp from arch_parse_elf_property().
 - Coding style tweaks.
v2:
 - Add a patch dropping has_interp from arch_adjust_elf_prot()
 - Fix bisection issue with static executables on arm64 in the first
   patch.

Mark Brown (4):
  elf: Allow architectures to parse properties on the main executable
  arm64: Enable BTI for main executable as well as the interpreter
  elf: Remove has_interp property from arch_adjust_elf_prot()
  elf: Remove has_interp property from arch_parse_elf_property()

 arch/arm64/include/asm/elf.h | 14 ++++++++++++--
 arch/arm64/kernel/process.c  | 16 +++-------------
 fs/binfmt_elf.c              | 29 ++++++++++++++++++++---------
 include/linux/elf.h          |  8 +++++---
 4 files changed, 40 insertions(+), 27 deletions(-)


base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07
-- 
2.30.2

[PATCH v8 1/4] elf: Allow architectures to parse properties on the main executable

[Mark Brown]

Currently the ELF code only attempts to parse properties on the image
that will start execution, either the interpreter or for statically linked
executables the main executable. The expectation is that any property
handling for the main executable will be done by the interpreter. This is
a bit inconsistent since we do map the executable and is causing problems
for the arm64 BTI support when used in conjunction with systemd's use of
seccomp to implement MemoryDenyWriteExecute which stops the dynamic linker
adjusting the permissions of executable segments.

Allow architectures to handle properties for both the dynamic linker and
main executable, adjusting arch_parse_elf_properties() to have a new
flag is_interp flag as with arch_elf_adjust_prot() and calling it for
both the main executable and any intepreter.

The user of this code, arm64, is adapted to ensure that there is no
functional change.

Signed-off-by: Mark Brown <broonie@kernel.org>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/elf.h |  3 ++-
 fs/binfmt_elf.c              | 27 +++++++++++++++++++--------
 include/linux/elf.h          |  4 +++-
 3 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index 97932fbf973d..5cc002376abe 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -259,6 +259,7 @@ struct arch_elf_state {
 
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
+					  bool has_interp, bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	/* No known properties for AArch32 yet */
@@ -271,7 +272,7 @@ static inline int arch_parse_elf_property(u32 type, const void *data,
 		if (datasz != sizeof(*p))
 			return -ENOEXEC;
 
-		if (system_supports_bti() &&
+		if (system_supports_bti() && has_interp == is_interp &&
 		    (*p & GNU_PROPERTY_AARCH64_FEATURE_1_BTI))
 			arch->flags |= ARM64_ELF_BTI;
 	}
diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index 605017eb9349..eabd7e0f6592 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -716,8 +716,9 @@ static unsigned long load_elf_interp(struct elfhdr *interp_elf_ex,
  */
 
 static int parse_elf_property(const char *data, size_t *off, size_t datasz,
-			      struct arch_elf_state *arch,
-			      bool have_prev_type, u32 *prev_type)
+			      struct arch_elf_state *arch, bool has_interp,
+			      bool is_interp, bool have_prev_type,
+			      u32 *prev_type)
 {
 	size_t o, step;
 	const struct gnu_property *pr;
@@ -751,7 +752,8 @@ static int parse_elf_property(const char *data, size_t *off, size_t datasz,
 	*prev_type = pr->pr_type;
 
 	ret = arch_parse_elf_property(pr->pr_type, data + o,
-				      pr->pr_datasz, ELF_COMPAT, arch);
+				      pr->pr_datasz, ELF_COMPAT,
+				      has_interp, is_interp, arch);
 	if (ret)
 		return ret;
 
@@ -764,6 +766,7 @@ static int parse_elf_property(const char *data, size_t *off, size_t datasz,
 #define NOTE_NAME_SZ (sizeof(GNU_PROPERTY_TYPE_0_NAME))
 
 static int parse_elf_properties(struct file *f, const struct elf_phdr *phdr,
+				bool has_interp, bool is_interp,
 				struct arch_elf_state *arch)
 {
 	union {
@@ -813,7 +816,8 @@ static int parse_elf_properties(struct file *f, const struct elf_phdr *phdr,
 	have_prev_type = false;
 	do {
 		ret = parse_elf_property(note.data, &off, datasz, arch,
-					 have_prev_type, &prev_type);
+					 has_interp, is_interp, have_prev_type,
+					 &prev_type);
 		have_prev_type = true;
 	} while (!ret);
 
@@ -828,6 +832,7 @@ static int load_elf_binary(struct linux_binprm *bprm)
 	unsigned long error;
 	struct elf_phdr *elf_ppnt, *elf_phdata, *interp_elf_phdata = NULL;
 	struct elf_phdr *elf_property_phdata = NULL;
+	struct elf_phdr *interp_elf_property_phdata = NULL;
 	unsigned long elf_bss, elf_brk;
 	int bss_prot = 0;
 	int retval, i;
@@ -963,12 +968,11 @@ static int load_elf_binary(struct linux_binprm *bprm)
 			goto out_free_dentry;
 
 		/* Pass PT_LOPROC..PT_HIPROC headers to arch code */
-		elf_property_phdata = NULL;
 		elf_ppnt = interp_elf_phdata;
 		for (i = 0; i < interp_elf_ex->e_phnum; i++, elf_ppnt++)
 			switch (elf_ppnt->p_type) {
 			case PT_GNU_PROPERTY:
-				elf_property_phdata = elf_ppnt;
+				interp_elf_property_phdata = elf_ppnt;
 				break;
 
 			case PT_LOPROC ... PT_HIPROC:
@@ -979,10 +983,17 @@ static int load_elf_binary(struct linux_binprm *bprm)
 					goto out_free_dentry;
 				break;
 			}
+
+		retval = parse_elf_properties(interpreter,
+					      interp_elf_property_phdata,
+					      true, true, &arch_state);
+		if (retval)
+			goto out_free_dentry;
+
 	}
 
-	retval = parse_elf_properties(interpreter ?: bprm->file,
-				      elf_property_phdata, &arch_state);
+	retval = parse_elf_properties(bprm->file, elf_property_phdata,
+				      interpreter, false, &arch_state);
 	if (retval)
 		goto out_free_dentry;
 
diff --git a/include/linux/elf.h b/include/linux/elf.h
index c9a46c4e183b..1c45ecf29147 100644
--- a/include/linux/elf.h
+++ b/include/linux/elf.h
@@ -88,13 +88,15 @@ struct arch_elf_state;
 #ifndef CONFIG_ARCH_USE_GNU_PROPERTY
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
+					  bool has_interp, bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	return 0;
 }
 #else
 extern int arch_parse_elf_property(u32 type, const void *data, size_t datasz,
-				   bool compat, struct arch_elf_state *arch);
+				   bool compat, bool has_interp, bool is_interp,
+				   struct arch_elf_state *arch);
 #endif
 
 #ifdef CONFIG_ARCH_HAVE_ELF_PROT
-- 
2.30.2

[PATCH v8 2/4] arm64: Enable BTI for main executable as well as the interpreter

[Mark Brown]

Currently for dynamically linked ELF executables we only enable BTI for
the interpreter, expecting the interpreter to do this for the main
executable. This is a bit inconsistent since we do map main executable and
is causing issues with systemd's MemoryDenyWriteExecute feature which is
implemented using a seccomp filter which prevents setting PROT_EXEC on
already mapped memory and lacks the context to be able to detect that
memory is already mapped with PROT_EXEC.

Resolve this by checking the BTI property for the main executable and
enabling BTI if it is present when doing the initial mapping. This does
mean that we may get more code with BTI enabled if running on a system
without BTI support in the dynamic linker, this is expected to be a safe
configuration and testing seems to confirm that. It also reduces the
flexibility userspace has to disable BTI but it is expected that for cases
where there are problems which require BTI to be disabled it is more likely
that it will need to be disabled on a system level.

Signed-off-by: Mark Brown <broonie@kernel.org>
Reviewed-by: Dave Martin <Dave.Martin@arm.com>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
---
 arch/arm64/include/asm/elf.h | 15 ++++++++++++---
 arch/arm64/kernel/process.c  | 14 ++------------
 2 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index 5cc002376abe..c4aa60db76a4 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -251,12 +251,21 @@ struct arch_elf_state {
 	int flags;
 };
 
-#define ARM64_ELF_BTI		(1 << 0)
+#define ARM64_ELF_INTERP_BTI		(1 << 0)
+#define ARM64_ELF_EXEC_BTI		(1 << 1)
 
 #define INIT_ARCH_ELF_STATE {			\
 	.flags = 0,				\
 }
 
+static inline int arm64_elf_bti_flag(bool is_interp)
+{
+	if (is_interp)
+		return ARM64_ELF_INTERP_BTI;
+	else
+		return ARM64_ELF_EXEC_BTI;
+}
+
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
 					  bool has_interp, bool is_interp,
@@ -272,9 +281,9 @@ static inline int arch_parse_elf_property(u32 type, const void *data,
 		if (datasz != sizeof(*p))
 			return -ENOEXEC;
 
-		if (system_supports_bti() && has_interp == is_interp &&
+		if (system_supports_bti() &&
 		    (*p & GNU_PROPERTY_AARCH64_FEATURE_1_BTI))
-			arch->flags |= ARM64_ELF_BTI;
+			arch->flags |= arm64_elf_bti_flag(is_interp);
 	}
 
 	return 0;
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 5369e649fa79..8157910a4181 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -704,18 +704,8 @@ core_initcall(tagged_addr_init);
 int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
 			 bool has_interp, bool is_interp)
 {
-	/*
-	 * For dynamically linked executables the interpreter is
-	 * responsible for setting PROT_BTI on everything except
-	 * itself.
-	 */
-	if (is_interp != has_interp)
-		return prot;
-
-	if (!(state->flags & ARM64_ELF_BTI))
-		return prot;
-
-	if (prot & PROT_EXEC)
+	if ((prot & PROT_EXEC) &&
+	    (state->flags & arm64_elf_bti_flag(is_interp)))
 		prot |= PROT_BTI;
 
 	return prot;
-- 
2.30.2

[PATCH v8 3/4] elf: Remove has_interp property from arch_adjust_elf_prot()

[Mark Brown]

Since we have added an is_interp flag to arch_parse_elf_property() we can
drop the has_interp flag from arch_elf_adjust_prot(), the only user was
the arm64 code which no longer needs it and any future users will be able
to use arch_parse_elf_properties() to determine if an interpreter is in
use.

Signed-off-by: Mark Brown <broonie@kernel.org>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-By: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/process.c | 2 +-
 fs/binfmt_elf.c             | 2 +-
 include/linux/elf.h         | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 8157910a4181..8017d963be3f 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -702,7 +702,7 @@ core_initcall(tagged_addr_init);
 
 #ifdef CONFIG_BINFMT_ELF
 int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
-			 bool has_interp, bool is_interp)
+			 bool is_interp)
 {
 	if ((prot & PROT_EXEC) &&
 	    (state->flags & arm64_elf_bti_flag(is_interp)))
diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index eabd7e0f6592..c1620358ab77 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -580,7 +580,7 @@ static inline int make_prot(u32 p_flags, struct arch_elf_state *arch_state,
 	if (p_flags & PF_X)
 		prot |= PROT_EXEC;
 
-	return arch_elf_adjust_prot(prot, arch_state, has_interp, is_interp);
+	return arch_elf_adjust_prot(prot, arch_state, is_interp);
 }
 
 /* This is much more generalized than the library routine read function,
diff --git a/include/linux/elf.h b/include/linux/elf.h
index 1c45ecf29147..d8392531899d 100644
--- a/include/linux/elf.h
+++ b/include/linux/elf.h
@@ -101,11 +101,11 @@ extern int arch_parse_elf_property(u32 type, const void *data, size_t datasz,
 
 #ifdef CONFIG_ARCH_HAVE_ELF_PROT
 int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state,
-			 bool has_interp, bool is_interp);
+			 bool is_interp);
 #else
 static inline int arch_elf_adjust_prot(int prot,
 				       const struct arch_elf_state *state,
-				       bool has_interp, bool is_interp)
+				       bool is_interp)
 {
 	return prot;
 }
-- 
2.30.2

[PATCH v8 4/4] elf: Remove has_interp property from arch_parse_elf_property()

[Mark Brown]

Since all current users of arch_parse_elf_property() are able to treat the
interpreter and main executable orthogonaly the has_interp argument is now
redundant so remove it.

Signed-off-by: Mark Brown <broonie@kernel.org>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-By: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/elf.h | 2 +-
 fs/binfmt_elf.c              | 2 +-
 include/linux/elf.h          | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index c4aa60db76a4..8009e8f07f1e 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -268,7 +268,7 @@ static inline int arm64_elf_bti_flag(bool is_interp)
 
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
-					  bool has_interp, bool is_interp,
+					  bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	/* No known properties for AArch32 yet */
diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index c1620358ab77..caedfdcead61 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -753,7 +753,7 @@ static int parse_elf_property(const char *data, size_t *off, size_t datasz,
 
 	ret = arch_parse_elf_property(pr->pr_type, data + o,
 				      pr->pr_datasz, ELF_COMPAT,
-				      has_interp, is_interp, arch);
+				      is_interp, arch);
 	if (ret)
 		return ret;
 
diff --git a/include/linux/elf.h b/include/linux/elf.h
index d8392531899d..cdb080d4b34a 100644
--- a/include/linux/elf.h
+++ b/include/linux/elf.h
@@ -88,14 +88,14 @@ struct arch_elf_state;
 #ifndef CONFIG_ARCH_USE_GNU_PROPERTY
 static inline int arch_parse_elf_property(u32 type, const void *data,
 					  size_t datasz, bool compat,
-					  bool has_interp, bool is_interp,
+					  bool is_interp,
 					  struct arch_elf_state *arch)
 {
 	return 0;
 }
 #else
 extern int arch_parse_elf_property(u32 type, const void *data, size_t datasz,
-				   bool compat, bool has_interp, bool is_interp,
+				   bool compat, bool is_interp,
 				   struct arch_elf_state *arch);
 #endif
 
-- 
2.30.2

Re: [PATCH v8 1/4] elf: Allow architectures to parse properties on the main executable

[Catalin Marinas]

On Mon, Jan 24, 2022 at 03:07:01PM +0000, Mark Brown wrote:
> Currently the ELF code only attempts to parse properties on the image
> that will start execution, either the interpreter or for statically linked
> executables the main executable. The expectation is that any property
> handling for the main executable will be done by the interpreter. This is
> a bit inconsistent since we do map the executable and is causing problems
> for the arm64 BTI support when used in conjunction with systemd's use of
> seccomp to implement MemoryDenyWriteExecute which stops the dynamic linker
> adjusting the permissions of executable segments.
> 
> Allow architectures to handle properties for both the dynamic linker and
> main executable, adjusting arch_parse_elf_properties() to have a new
> flag is_interp flag as with arch_elf_adjust_prot() and calling it for
> both the main executable and any intepreter.
> 
> The user of this code, arm64, is adapted to ensure that there is no
> functional change.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Tested-by: Jeremy Linton <jeremy.linton@arm.com>
> Reviewed-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH v8 2/4] arm64: Enable BTI for main executable as well as the interpreter

[Catalin Marinas]

On Mon, Jan 24, 2022 at 03:07:02PM +0000, Mark Brown wrote:
> Currently for dynamically linked ELF executables we only enable BTI for
> the interpreter, expecting the interpreter to do this for the main
> executable. This is a bit inconsistent since we do map main executable and
> is causing issues with systemd's MemoryDenyWriteExecute feature which is
> implemented using a seccomp filter which prevents setting PROT_EXEC on
> already mapped memory and lacks the context to be able to detect that
> memory is already mapped with PROT_EXEC.
> 
> Resolve this by checking the BTI property for the main executable and
> enabling BTI if it is present when doing the initial mapping. This does
> mean that we may get more code with BTI enabled if running on a system
> without BTI support in the dynamic linker, this is expected to be a safe
> configuration and testing seems to confirm that.

Alternatively we could only enable BTI on the main executable if the
interpreter is also compiled with the BTI attributes. But it still feels
a bit inconsistent as a BTI-capable interpreter doesn't necessarily mean
it had to be compiled with BTI support.

> It also reduces the
> flexibility userspace has to disable BTI but it is expected that for cases
> where there are problems which require BTI to be disabled it is more likely
> that it will need to be disabled on a system level.

That's my expectation as well. I'm not aware of any plans to make a
dynamic loader choose BTI based on some environment variable (not sure
about Android zygote, I haven't heard anything either).

> Signed-off-by: Mark Brown <broonie@kernel.org>
> Reviewed-by: Dave Martin <Dave.Martin@arm.com>
> Tested-by: Jeremy Linton <jeremy.linton@arm.com>

For this patch:

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH v8 3/4] elf: Remove has_interp property from arch_adjust_elf_prot()

[Catalin Marinas]

On Mon, Jan 24, 2022 at 03:07:03PM +0000, Mark Brown wrote:
> Since we have added an is_interp flag to arch_parse_elf_property() we can
> drop the has_interp flag from arch_elf_adjust_prot(), the only user was
> the arm64 code which no longer needs it and any future users will be able
> to use arch_parse_elf_properties() to determine if an interpreter is in
> use.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Tested-by: Jeremy Linton <jeremy.linton@arm.com>
> Reviewed-By: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH v8 4/4] elf: Remove has_interp property from arch_parse_elf_property()

[Catalin Marinas]

On Mon, Jan 24, 2022 at 03:07:04PM +0000, Mark Brown wrote:
> Since all current users of arch_parse_elf_property() are able to treat the
> interpreter and main executable orthogonaly the has_interp argument is now
> redundant so remove it.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Tested-by: Jeremy Linton <jeremy.linton@arm.com>
> Reviewed-By: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Will Deacon]

On Mon, Jan 24, 2022 at 03:07:00PM +0000, Mark Brown wrote:
> Deployments of BTI on arm64 have run into issues interacting with
> systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> linked executables the kernel will only handle architecture specific
> properties like BTI for the interpreter, the expectation is that the
> interpreter will then handle any properties on the main executable.
> For BTI this means remapping the executable segments PROT_EXEC |
> PROT_BTI.
> 
> This interacts poorly with MemoryDenyWriteExecute since that is
> implemented using a seccomp filter which prevents setting PROT_EXEC on
> already mapped memory and lacks the context to be able to detect that
> memory is already mapped with PROT_EXEC.  This series resolves this by
> handling the BTI property for both the interpreter and the main
> executable.

This appears to be a user-visible change which cannot be detected or
disabled from userspace. If there is code out there which does not work
when BTI is enabled, won't that now explode when the kernel enables it?
How are we supposed to handle such a regression?

Will

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Catalin Marinas]

On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:
> On Mon, Jan 24, 2022 at 03:07:00PM +0000, Mark Brown wrote:
> > Deployments of BTI on arm64 have run into issues interacting with
> > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > linked executables the kernel will only handle architecture specific
> > properties like BTI for the interpreter, the expectation is that the
> > interpreter will then handle any properties on the main executable.
> > For BTI this means remapping the executable segments PROT_EXEC |
> > PROT_BTI.
> > 
> > This interacts poorly with MemoryDenyWriteExecute since that is
> > implemented using a seccomp filter which prevents setting PROT_EXEC on
> > already mapped memory and lacks the context to be able to detect that
> > memory is already mapped with PROT_EXEC.  This series resolves this by
> > handling the BTI property for both the interpreter and the main
> > executable.
> 
> This appears to be a user-visible change which cannot be detected or
> disabled from userspace. If there is code out there which does not work
> when BTI is enabled, won't that now explode when the kernel enables it?
> How are we supposed to handle such a regression?

If this ever happens, the only workaround is to disable BTI on the
kernel command line. If we need a knob closer to user, we could add a
sysctl option (as we did for the tagged address ABI, though I doubt
people are even aware that exists). The dynamic loader doesn't do
anything smart when deciding to map objects with PROT_BTI (like env
variables), it simply relies on the ELF information.

I think that's very unlikely and feedback from Szabolcs in the past and
additional testing by Mark and Jeremy was that it should be fine. The
architecture allows interworking between BTI and non-BTI objects and on
distros with both BTI and MDWE enabled, this is already the case: the
main executable is mapped without PROT_BTI while the libraries will be
mapped with PROT_BTI. The new behaviour allows both to be mapped with
PROT_BTI, just as if MDWE was disabled.

I think the only difference would be with a BTI-unware dynamic loader
(e.g. older distro). Here the main executable, if compiled with BTI,
would be mapped as executable while the rest of the libraries are
non-BTI. The interworking should be fine but we can't test everything
since such BTI binaries would not normally be part of the distro.

If there are dodgy libraries out there that do tricks and branch into
the middle of a function in the main executable, they will fail with
this series but also fail if MDWE is disabled and the dynamic linker is
BTI-aware. So this hardly counts as a use-case.

For consistency, I think whoever does the initial mapping should also
set the correct attributes as we do for static binaries. If you think
another knob is needed other than the cmdline, I'm fine with it.

-- 
Catalin

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Szabolcs Nagy]

The 02/16/2022 13:34, Catalin Marinas via Libc-alpha wrote:
> On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:
> > This appears to be a user-visible change which cannot be detected or
> > disabled from userspace. If there is code out there which does not work
> > when BTI is enabled, won't that now explode when the kernel enables it?
> > How are we supposed to handle such a regression?
> 
> If this ever happens, the only workaround is to disable BTI on the
> kernel command line. If we need a knob closer to user, we could add a
> sysctl option (as we did for the tagged address ABI, though I doubt
> people are even aware that exists). The dynamic loader doesn't do
> anything smart when deciding to map objects with PROT_BTI (like env
> variables), it simply relies on the ELF information.
> 
> I think that's very unlikely and feedback from Szabolcs in the past and
> additional testing by Mark and Jeremy was that it should be fine. The
> architecture allows interworking between BTI and non-BTI objects and on
> distros with both BTI and MDWE enabled, this is already the case: the
> main executable is mapped without PROT_BTI while the libraries will be
> mapped with PROT_BTI. The new behaviour allows both to be mapped with
> PROT_BTI, just as if MDWE was disabled.
> 
> I think the only difference would be with a BTI-unware dynamic loader
> (e.g. older distro). Here the main executable, if compiled with BTI,
> would be mapped as executable while the rest of the libraries are
> non-BTI. The interworking should be fine but we can't test everything
> since such BTI binaries would not normally be part of the distro.

note: a bti marked exe has to be built against a bti enabled
runtime (so crt files etc are bti compatible) and any system
that executes such an exe must be newer and thus bti aware
(according to glibc abi compatibility rules).

so while it's possible that ld.so is not bti marked when an
exe is, the ld.so will be at least bti aware, which means it
will map all bti binaries (including the exe) with PROT_BTI.

so this doesn't really affect glibc based systems. and there
are not many other systems that can produce bti marked exes.
(i don't think this can cause problems on android either)

if we ever wanted to map bti marked binaries without PROT_BTI
and introduced a knob to do that in ld.so, then this change
would be problematic (we cannot easily remove PROT_BTI from
the exe), but we don't have such plans.

> 
> If there are dodgy libraries out there that do tricks and branch into
> the middle of a function in the main executable, they will fail with
> this series but also fail if MDWE is disabled and the dynamic linker is
> BTI-aware. So this hardly counts as a use-case.
> 
> For consistency, I think whoever does the initial mapping should also
> set the correct attributes as we do for static binaries. If you think
> another knob is needed other than the cmdline, I'm fine with it.
> 
> -- 
> Catalin

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Wed, Feb 16, 2022 at 04:49:54PM +0000, Szabolcs Nagy wrote:

> if we ever wanted to map bti marked binaries without PROT_BTI
> and introduced a knob to do that in ld.so, then this change
> would be problematic (we cannot easily remove PROT_BTI from
> the exe), but we don't have such plans.

In general or only in the case where MWDE is enabled (in which case it's
the same problem as exists today trying to enable BTI in the presence of
MWDE)?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Szabolcs Nagy]

The 02/16/2022 17:01, Mark Brown wrote:
> On Wed, Feb 16, 2022 at 04:49:54PM +0000, Szabolcs Nagy wrote:
> 
> > if we ever wanted to map bti marked binaries without PROT_BTI
> > and introduced a knob to do that in ld.so, then this change
> > would be problematic (we cannot easily remove PROT_BTI from
> > the exe), but we don't have such plans.
> 
> In general or only in the case where MWDE is enabled (in which case it's
> the same problem as exists today trying to enable BTI in the presence of
> MWDE)?

ah yes with mwde only.

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2268 bytes --]

On Wed, Feb 16, 2022 at 01:34:25PM +0000, Catalin Marinas wrote:
> On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:

> > This appears to be a user-visible change which cannot be detected or
> > disabled from userspace. If there is code out there which does not work
> > when BTI is enabled, won't that now explode when the kernel enables it?
> > How are we supposed to handle such a regression?

> If this ever happens, the only workaround is to disable BTI on the
> kernel command line. If we need a knob closer to user, we could add a
> sysctl option (as we did for the tagged address ABI, though I doubt
> people are even aware that exists). The dynamic loader doesn't do
> anything smart when deciding to map objects with PROT_BTI (like env
> variables), it simply relies on the ELF information.

The dynamic loader is the place where I'd expect to do any
per-executable workarounds, but currently that's not actually
implemented anywhere.  Someone could also make a tool to strip BTI
annotations from executables.

> I think the only difference would be with a BTI-unware dynamic loader
> (e.g. older distro). Here the main executable, if compiled with BTI,
> would be mapped as executable while the rest of the libraries are
> non-BTI. The interworking should be fine but we can't test everything
> since such BTI binaries would not normally be part of the distro.

> If there are dodgy libraries out there that do tricks and branch into
> the middle of a function in the main executable, they will fail with
> this series but also fail if MDWE is disabled and the dynamic linker is
> BTI-aware. So this hardly counts as a use-case.

I'm not aware of any issues we've run into which are due to interworking
between binaries rather than within a binary due to either miscompilation
or doing something in hand coded assembler that needs updating for BTI.
It doesn't mean it can't happen but it's hard to see what people might
be doing.

> For consistency, I think whoever does the initial mapping should also
> set the correct attributes as we do for static binaries. If you think
> another knob is needed other than the cmdline, I'm fine with it.

Might also be worth pointing out that we already map the vDSO with BTI
enabled if it's built with BTI.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Will Deacon]

On Wed, Feb 16, 2022 at 01:34:25PM +0000, Catalin Marinas wrote:
> On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:
> > On Mon, Jan 24, 2022 at 03:07:00PM +0000, Mark Brown wrote:
> > > Deployments of BTI on arm64 have run into issues interacting with
> > > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > > linked executables the kernel will only handle architecture specific
> > > properties like BTI for the interpreter, the expectation is that the
> > > interpreter will then handle any properties on the main executable.
> > > For BTI this means remapping the executable segments PROT_EXEC |
> > > PROT_BTI.
> > > 
> > > This interacts poorly with MemoryDenyWriteExecute since that is
> > > implemented using a seccomp filter which prevents setting PROT_EXEC on
> > > already mapped memory and lacks the context to be able to detect that
> > > memory is already mapped with PROT_EXEC.  This series resolves this by
> > > handling the BTI property for both the interpreter and the main
> > > executable.
> > 
> > This appears to be a user-visible change which cannot be detected or
> > disabled from userspace. If there is code out there which does not work
> > when BTI is enabled, won't that now explode when the kernel enables it?
> > How are we supposed to handle such a regression?
> 
> If this ever happens, the only workaround is to disable BTI on the
> kernel command line. If we need a knob closer to user, we could add a
> sysctl option (as we did for the tagged address ABI, though I doubt
> people are even aware that exists). The dynamic loader doesn't do
> anything smart when deciding to map objects with PROT_BTI (like env
> variables), it simply relies on the ELF information.
> 
> I think that's very unlikely and feedback from Szabolcs in the past and
> additional testing by Mark and Jeremy was that it should be fine. The
> architecture allows interworking between BTI and non-BTI objects and on
> distros with both BTI and MDWE enabled, this is already the case: the
> main executable is mapped without PROT_BTI while the libraries will be
> mapped with PROT_BTI. The new behaviour allows both to be mapped with
> PROT_BTI, just as if MDWE was disabled.
> 
> I think the only difference would be with a BTI-unware dynamic loader
> (e.g. older distro). Here the main executable, if compiled with BTI,
> would be mapped as executable while the rest of the libraries are
> non-BTI. The interworking should be fine but we can't test everything
> since such BTI binaries would not normally be part of the distro.
> 
> If there are dodgy libraries out there that do tricks and branch into
> the middle of a function in the main executable, they will fail with
> this series but also fail if MDWE is disabled and the dynamic linker is
> BTI-aware. So this hardly counts as a use-case.
> 
> For consistency, I think whoever does the initial mapping should also
> set the correct attributes as we do for static binaries. If you think
> another knob is needed other than the cmdline, I'm fine with it.

I still think this new behaviour should be opt-in, so adding a sysctl for
that would be my preference if we proceed with this approach.

Will

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

On Fri, Feb 25, 2022 at 01:53:51PM +0000, Will Deacon wrote:

> I still think this new behaviour should be opt-in, so adding a sysctl for
> that would be my preference if we proceed with this approach.

I'm happy to have a sysctl but I'd rather it be opt out rather than opt
in since it seems better to default to enabling the security feature
when there is a strong expectation that it would seem better to enable
it by default sine it's not expected to be disruptive and the sysctl is
more of a "what if there's a problem" thing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v8 0/4] arm64: Enable BTI for the executable as well as the interpreter

[Will Deacon]

On Fri, Feb 25, 2022 at 03:11:43PM +0000, Mark Brown wrote:
> On Fri, Feb 25, 2022 at 01:53:51PM +0000, Will Deacon wrote:
> 
> > I still think this new behaviour should be opt-in, so adding a sysctl for
> > that would be my preference if we proceed with this approach.
> 
> I'm happy to have a sysctl but I'd rather it be opt out rather than opt
> in since it seems better to default to enabling the security feature
> when there is a strong expectation that it would seem better to enable
> it by default sine it's not expected to be disruptive and the sysctl is
> more of a "what if there's a problem" thing.

I think new behaviour has to be opt-in, so that if somebody takes a new
kernel then we can guarantee it's not going to break them. Systemd can
enable this unconditionally if it wants to.

Will