[PATCH 01/16] aarch64: Add arm_streaming(_compatible) attributes

[Richard Sandiford <richard.sandiford@arm.com>]

This patch adds support for recognising the SME arm_streaming
and arm_streaming_compatible attributes.  These attributes
respectively describe whether the processor is definitely in
"streaming mode" (PSTATE.SM==1), whether the processor is
definitely not in streaming mode (PSTATE.SM==0), or whether
we don't know at compile time either way.

As far as the compiler is concerned, this effectively create three
ISA submodes: streaming mode enables things that are not available
in non-streaming mode, non-streaming mode enables things that not
available in streaming mode, and streaming-compatible mode has to stick
to the common subset.  This means that some instructions are conditional
on PSTATE.SM==1 and some are conditional on PSTATE.SM==0.

I wondered about recording the streaming state in a new variable.
However, the set of available instructions is also influenced by
PSTATE.ZA (added later), so I think it makes sense to view this
as an instance of a more general mechanism.  Also, keeping the
PSTATE.SM state in the same flag variable as the other ISA
features makes it possible to sum up the requirements of an
ACLE function in a single value.

The patch therefore adds a new set of feature flags called "ISA modes".
Unlike the other two sets of flags (optional features and architecture-
level features), these ISA modes are not controlled directly by
command-line parameters or "target" attributes.

arm_streaming and arm_streaming_compatible are function type attributes
rather than function declaration attributes.  This means that we need
to find somewhere to copy the type information across to a function's
target options.  The patch does this in aarch64_set_current_function.

We also need to record which ISA mode a callee expects/requires
to be active on entry.  (The same mode is then active on return.)
The patch extends the current UNSPEC_CALLEE_ABI cookie to include
this information, as well as the PCS variant that it recorded
previously.

gcc/
	* config/aarch64/aarch64-isa-modes.def: New file.
	* config/aarch64/aarch64.h: Include it in the feature enumerations.
	(AARCH64_FL_SM_STATE, AARCH64_FL_ISA_MODES): New constants.
	(AARCH64_FL_DEFAULT_ISA_MODE): Likewise.
	(AARCH64_ISA_MODE): New macro.
	(CUMULATIVE_ARGS): Add an isa_mode field.
	* config/aarch64/aarch64-protos.h (aarch64_gen_callee_cookie): Declare.
	(aarch64_tlsdesc_abi_id): Return an arm_pcs.
	* config/aarch64/aarch64.cc (attr_streaming_exclusions): New variable.
	(aarch64_attribute_table): Add arm_streaming and
	arm_streaming_compatible.
	(aarch64_fntype_sm_state, aarch64_fntype_isa_mode): New functions.
	(aarch64_fndecl_sm_state, aarch64_fndecl_isa_mode): Likewise.
	(aarch64_gen_callee_cookie, aarch64_callee_abi): Likewise.
	(aarch64_insn_callee_cookie, aarch64_insn_callee_abi): Use them.
	(aarch64_function_arg, aarch64_output_mi_thunk): Likewise.
	(aarch64_init_cumulative_args): Initialize the isa_mode field.
	(aarch64_override_options): Add the ISA mode to the feature set.
	(aarch64_temporary_target::copy_from_fndecl): Likewise.
	(aarch64_fndecl_options, aarch64_handle_attr_arch): Likewise.
	(aarch64_set_current_function): Maintain the correct ISA mode.
	(aarch64_tlsdesc_abi_id): Return an arm_pcs.
	(aarch64_comp_type_attributes): Handle arm_streaming and
	arm_streaming_compatible.
	* config/aarch64/aarch64.md (tlsdesc_small_<mode>): Use
	aarch64_gen_callee_cookie to get the ABI cookie.
	* config/aarch64/t-aarch64 (TM_H): Add all feature-related .def files.

gcc/testsuite/
	* gcc.target/aarch64/sme/aarch64-sme.exp: New harness.
	* gcc.target/aarch64/sme/streaming_mode_1.c: New test.
	* gcc.target/aarch64/sme/streaming_mode_2.c: Likewise.
	* gcc.target/aarch64/auto-init-1.c: Only expect the call insn
	to contain 1 (const_int 0), not 2.
---
 gcc/config/aarch64/aarch64-isa-modes.def      |  35 ++++
 gcc/config/aarch64/aarch64-protos.h           |   3 +-
 gcc/config/aarch64/aarch64.cc                 | 194 +++++++++++++++---
 gcc/config/aarch64/aarch64.h                  |  24 ++-
 gcc/config/aarch64/aarch64.md                 |   3 +-
 gcc/config/aarch64/t-aarch64                  |   5 +-
 .../gcc.target/aarch64/auto-init-1.c          |   3 +-
 .../gcc.target/aarch64/sme/aarch64-sme.exp    |  41 ++++
 .../gcc.target/aarch64/sme/streaming_mode_1.c | 106 ++++++++++
 .../gcc.target/aarch64/sme/streaming_mode_2.c |  25 +++
 10 files changed, 403 insertions(+), 36 deletions(-)
 create mode 100644 gcc/config/aarch64/aarch64-isa-modes.def
 create mode 100644 gcc/testsuite/gcc.target/aarch64/sme/aarch64-sme.exp
 create mode 100644 gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_1.c
 create mode 100644 gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_2.c

diff --git a/gcc/config/aarch64/aarch64-isa-modes.def b/gcc/config/aarch64/aarch64-isa-modes.def
new file mode 100644
index 00000000000..fba8eafbae1
--- /dev/null
+++ b/gcc/config/aarch64/aarch64-isa-modes.def
@@ -0,0 +1,35 @@
+/* Copyright (C) 2022 Free Software Foundation, Inc.
+
+   This file is part of GCC.
+
+   GCC is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published
+   by the Free Software Foundation; either version 3, or (at your
+   option) any later version.
+
+   GCC is distributed in the hope that it will be useful, but WITHOUT
+   ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+   or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public
+   License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with GCC; see the file COPYING3.  If not see
+   <http://www.gnu.org/licenses/>.  */
+
+/* This file defines a set of "ISA modes"; in other words, it defines
+   various bits of runtime state that control the set of available
+   instructions or that affect the semantics of instructions in some way.
+
+   Before using #include to read this file, define a macro:
+
+      DEF_AARCH64_ISA_MODE(NAME)
+
+  where NAME is the name of the mode.  */
+
+/* Indicates that PSTATE.SM is known to be 1 or 0 respectively.  These
+   modes are mutually exclusive.  If neither mode is active then the state
+   of PSTATE.SM is not known at compile time.  */
+DEF_AARCH64_ISA_MODE(SM_ON)
+DEF_AARCH64_ISA_MODE(SM_OFF)
+
+#undef DEF_AARCH64_ISA_MODE
diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h
index 866d68ad4d7..06b926b42d6 100644
--- a/gcc/config/aarch64/aarch64-protos.h
+++ b/gcc/config/aarch64/aarch64-protos.h
@@ -771,6 +771,7 @@ bool aarch64_const_vec_all_same_in_range_p (rtx, HOST_WIDE_INT,
 bool aarch64_constant_address_p (rtx);
 bool aarch64_emit_approx_div (rtx, rtx, rtx);
 bool aarch64_emit_approx_sqrt (rtx, rtx, bool);
+rtx aarch64_gen_callee_cookie (aarch64_feature_flags, arm_pcs);
 void aarch64_expand_call (rtx, rtx, rtx, bool);
 bool aarch64_expand_cpymem (rtx *);
 bool aarch64_expand_setmem (rtx *);
@@ -849,7 +850,7 @@ int aarch64_movk_shift (const wide_int_ref &, const wide_int_ref &);
 bool aarch64_use_return_insn_p (void);
 const char *aarch64_output_casesi (rtx *);
 
-unsigned int aarch64_tlsdesc_abi_id ();
+arm_pcs aarch64_tlsdesc_abi_id ();
 enum aarch64_symbol_type aarch64_classify_symbol (rtx, HOST_WIDE_INT);
 enum aarch64_symbol_type aarch64_classify_tls_symbol (rtx);
 enum reg_class aarch64_regno_regclass (unsigned);
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index a40ac6fd903..a2e910daddf 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -2731,6 +2731,16 @@ handle_aarch64_vector_pcs_attribute (tree *node, tree name, tree,
   gcc_unreachable ();
 }
 
+/* Mutually-exclusive function type attributes for controlling PSTATE.SM.  */
+static const struct attribute_spec::exclusions attr_streaming_exclusions[] =
+{
+  /* Attribute name     exclusion applies to:
+			function, type, variable */
+  { "arm_streaming", false, false, false },
+  { "arm_streaming_compatible", false, true, false },
+  { NULL, false, false, false }
+};
+
 /* Table of machine attributes.  */
 static const struct attribute_spec aarch64_attribute_table[] =
 {
@@ -2738,6 +2748,10 @@ static const struct attribute_spec aarch64_attribute_table[] =
        affects_type_identity, handler, exclude } */
   { "aarch64_vector_pcs", 0, 0, false, true,  true,  true,
 			  handle_aarch64_vector_pcs_attribute, NULL },
+  { "arm_streaming",	  0, 0, false, true,  true,  true,
+			  NULL, attr_streaming_exclusions },
+  { "arm_streaming_compatible", 0, 0, false, true,  true,  true,
+			  NULL, attr_streaming_exclusions },
   { "arm_sve_vector_bits", 1, 1, false, true,  false, true,
 			  aarch64_sve::handle_arm_sve_vector_bits_attribute,
 			  NULL },
@@ -4048,6 +4062,47 @@ aarch64_fntype_abi (const_tree fntype)
   return default_function_abi;
 }
 
+/* Return the state of PSTATE.SM on entry to functions of type FNTYPE.  */
+
+static aarch64_feature_flags
+aarch64_fntype_sm_state (const_tree fntype)
+{
+  if (lookup_attribute ("arm_streaming", TYPE_ATTRIBUTES (fntype)))
+    return AARCH64_FL_SM_ON;
+
+  if (lookup_attribute ("arm_streaming_compatible", TYPE_ATTRIBUTES (fntype)))
+    return 0;
+
+  return AARCH64_FL_SM_OFF;
+}
+
+/* Return the ISA mode on entry to functions of type FNTYPE.  */
+
+static aarch64_feature_flags
+aarch64_fntype_isa_mode (const_tree fntype)
+{
+  return aarch64_fntype_sm_state (fntype);
+}
+
+/* Return the state of PSTATE.SM when compiling the body of
+   function FNDECL.  This might be different from the state of
+   PSTATE.SM on entry.  */
+
+static aarch64_feature_flags
+aarch64_fndecl_sm_state (const_tree fndecl)
+{
+  return aarch64_fntype_sm_state (TREE_TYPE (fndecl));
+}
+
+/* Return the ISA mode that should be used to compile the body of
+   function FNDECL.  */
+
+static aarch64_feature_flags
+aarch64_fndecl_isa_mode (const_tree fndecl)
+{
+  return aarch64_fndecl_sm_state (fndecl);
+}
+
 /* Implement TARGET_COMPATIBLE_VECTOR_TYPES_P.  */
 
 static bool
@@ -4110,17 +4165,46 @@ aarch64_reg_save_mode (unsigned int regno)
   gcc_unreachable ();
 }
 
-/* Implement TARGET_INSN_CALLEE_ABI.  */
+/* Given the ISA mode on entry to a callee and the ABI of the callee,
+   return the CONST_INT that should be placed in an UNSPEC_CALLEE_ABI rtx.  */
 
-const predefined_function_abi &
-aarch64_insn_callee_abi (const rtx_insn *insn)
+rtx
+aarch64_gen_callee_cookie (aarch64_feature_flags isa_mode, arm_pcs pcs_variant)
+{
+  return gen_int_mode ((unsigned int) isa_mode
+		       | (unsigned int) pcs_variant << AARCH64_NUM_ISA_MODES,
+		       DImode);
+}
+
+/* COOKIE is a CONST_INT from an UNSPEC_CALLEE_ABI rtx.  Return the
+   callee's ABI.  */
+
+static const predefined_function_abi &
+aarch64_callee_abi (rtx cookie)
+{
+  return function_abis[UINTVAL (cookie) >> AARCH64_NUM_ISA_MODES];
+}
+
+/* INSN is a call instruction.  Return the CONST_INT stored in its
+   UNSPEC_CALLEE_ABI rtx.  */
+
+static rtx
+aarch64_insn_callee_cookie (const rtx_insn *insn)
 {
   rtx pat = PATTERN (insn);
   gcc_assert (GET_CODE (pat) == PARALLEL);
   rtx unspec = XVECEXP (pat, 0, 1);
   gcc_assert (GET_CODE (unspec) == UNSPEC
 	      && XINT (unspec, 1) == UNSPEC_CALLEE_ABI);
-  return function_abis[INTVAL (XVECEXP (unspec, 0, 0))];
+  return XVECEXP (unspec, 0, 0);
+}
+
+/* Implement TARGET_INSN_CALLEE_ABI.  */
+
+const predefined_function_abi &
+aarch64_insn_callee_abi (const rtx_insn *insn)
+{
+  return aarch64_callee_abi (aarch64_insn_callee_cookie (insn));
 }
 
 /* Implement TARGET_HARD_REGNO_CALL_PART_CLOBBERED.  The callee only saves
@@ -7861,7 +7945,7 @@ aarch64_function_arg (cumulative_args_t pcum_v, const function_arg_info &arg)
 	      || pcum->pcs_variant == ARM_PCS_SVE);
 
   if (arg.end_marker_p ())
-    return gen_int_mode (pcum->pcs_variant, DImode);
+    return aarch64_gen_callee_cookie (pcum->isa_mode, pcum->pcs_variant);
 
   aarch64_layout_arg (pcum_v, arg);
   return pcum->aapcs_reg;
@@ -7882,9 +7966,15 @@ aarch64_init_cumulative_args (CUMULATIVE_ARGS *pcum,
   pcum->aapcs_nextnvrn = 0;
   pcum->aapcs_nextnprn = 0;
   if (fntype)
-    pcum->pcs_variant = (arm_pcs) fntype_abi (fntype).id ();
+    {
+      pcum->pcs_variant = (arm_pcs) fntype_abi (fntype).id ();
+      pcum->isa_mode = aarch64_fntype_isa_mode (fntype);
+    }
   else
-    pcum->pcs_variant = ARM_PCS_AAPCS64;
+    {
+      pcum->pcs_variant = ARM_PCS_AAPCS64;
+      pcum->isa_mode = AARCH64_FL_DEFAULT_ISA_MODE;
+    }
   pcum->aapcs_reg = NULL_RTX;
   pcum->aapcs_arg_processed = false;
   pcum->aapcs_stack_words = 0;
@@ -10372,7 +10462,9 @@ aarch64_output_mi_thunk (FILE *file, tree thunk ATTRIBUTE_UNUSED,
     }
   funexp = XEXP (DECL_RTL (function), 0);
   funexp = gen_rtx_MEM (FUNCTION_MODE, funexp);
-  rtx callee_abi = gen_int_mode (fndecl_abi (function).id (), DImode);
+  auto isa_mode = aarch64_fntype_isa_mode (TREE_TYPE (function));
+  auto pcs_variant = arm_pcs (fndecl_abi (function).id ());
+  rtx callee_abi = aarch64_gen_callee_cookie (isa_mode, pcs_variant);
   insn = emit_call_insn (gen_sibcall (funexp, const0_rtx, callee_abi));
   SIBLING_CALL_P (insn) = 1;
 
@@ -18315,6 +18407,7 @@ aarch64_override_options (void)
   SUBTARGET_OVERRIDE_OPTIONS;
 #endif
 
+  auto isa_mode = AARCH64_FL_DEFAULT_ISA_MODE;
   if (cpu && arch)
     {
       /* If both -mcpu and -march are specified, warn if they are not
@@ -18327,25 +18420,25 @@ aarch64_override_options (void)
 	}
 
       selected_arch = arch->arch;
-      aarch64_set_asm_isa_flags (arch_isa);
+      aarch64_set_asm_isa_flags (arch_isa | isa_mode);
     }
   else if (cpu)
     {
       selected_arch = cpu->arch;
-      aarch64_set_asm_isa_flags (cpu_isa);
+      aarch64_set_asm_isa_flags (cpu_isa | isa_mode);
     }
   else if (arch)
     {
       cpu = &all_cores[arch->ident];
       selected_arch = arch->arch;
-      aarch64_set_asm_isa_flags (arch_isa);
+      aarch64_set_asm_isa_flags (arch_isa | isa_mode);
     }
   else
     {
       /* No -mcpu or -march specified, so use the default CPU.  */
       cpu = &all_cores[TARGET_CPU_DEFAULT];
       selected_arch = cpu->arch;
-      aarch64_set_asm_isa_flags (cpu->flags);
+      aarch64_set_asm_isa_flags (cpu->flags | isa_mode);
     }
 
   selected_tune = tune ? tune->ident : cpu->ident;
@@ -18518,6 +18611,21 @@ aarch64_save_restore_target_globals (tree new_tree)
     TREE_TARGET_GLOBALS (new_tree) = save_target_globals_default_opts ();
 }
 
+/* Return the target_option_node for FNDECL, or the current options
+   if FNDECL is null.  */
+
+static tree
+aarch64_fndecl_options (tree fndecl)
+{
+  if (!fndecl)
+    return target_option_current_node;
+
+  if (tree options = DECL_FUNCTION_SPECIFIC_TARGET (fndecl))
+    return options;
+
+  return target_option_default_node;
+}
+
 /* Implement TARGET_SET_CURRENT_FUNCTION.  Unpack the codegen decisions
    like tuning and ISA features from the DECL_FUNCTION_SPECIFIC_TARGET
    of the function, if such exists.  This function may be called multiple
@@ -18527,25 +18635,24 @@ aarch64_save_restore_target_globals (tree new_tree)
 static void
 aarch64_set_current_function (tree fndecl)
 {
-  if (!fndecl || fndecl == aarch64_previous_fndecl)
-    return;
+  tree old_tree = aarch64_fndecl_options (aarch64_previous_fndecl);
+  tree new_tree = aarch64_fndecl_options (fndecl);
 
-  tree old_tree = (aarch64_previous_fndecl
-		   ? DECL_FUNCTION_SPECIFIC_TARGET (aarch64_previous_fndecl)
-		   : NULL_TREE);
-
-  tree new_tree = DECL_FUNCTION_SPECIFIC_TARGET (fndecl);
-
-  /* If current function has no attributes but the previous one did,
-     use the default node.  */
-  if (!new_tree && old_tree)
-    new_tree = target_option_default_node;
+  auto new_isa_mode = (fndecl
+		       ? aarch64_fndecl_isa_mode (fndecl)
+		       : AARCH64_FL_DEFAULT_ISA_MODE);
+  auto isa_flags = TREE_TARGET_OPTION (new_tree)->x_aarch64_isa_flags;
 
   /* If nothing to do, return.  #pragma GCC reset or #pragma GCC pop to
      the default have been handled by aarch64_save_restore_target_globals from
      aarch64_pragma_target_parse.  */
-  if (old_tree == new_tree)
-    return;
+  if (old_tree == new_tree
+      && (!fndecl || aarch64_previous_fndecl)
+      && (isa_flags & AARCH64_FL_ISA_MODES) == new_isa_mode)
+    {
+      gcc_assert (AARCH64_ISA_MODE == new_isa_mode);
+      return;
+    }
 
   aarch64_previous_fndecl = fndecl;
 
@@ -18553,7 +18660,28 @@ aarch64_set_current_function (tree fndecl)
   cl_target_option_restore (&global_options, &global_options_set,
 			    TREE_TARGET_OPTION (new_tree));
 
+  /* The ISA mode can vary based on function type attributes and
+     function declaration attributes.  Make sure that the target
+     options correctly reflect these attributes.  */
+  if ((isa_flags & AARCH64_FL_ISA_MODES) != new_isa_mode)
+    {
+      auto base_flags = (aarch64_asm_isa_flags & ~AARCH64_FL_ISA_MODES);
+      aarch64_set_asm_isa_flags (base_flags | new_isa_mode);
+
+      aarch64_override_options_internal (&global_options);
+      new_tree = build_target_option_node (&global_options,
+					   &global_options_set);
+      DECL_FUNCTION_SPECIFIC_TARGET (fndecl) = new_tree;
+
+      tree new_optimize = build_optimization_node (&global_options,
+						   &global_options_set);
+      if (new_optimize != optimization_default_node)
+	DECL_FUNCTION_SPECIFIC_OPTIMIZATION (fndecl) = new_optimize;
+    }
+
   aarch64_save_restore_target_globals (new_tree);
+
+  gcc_assert (AARCH64_ISA_MODE == new_isa_mode);
 }
 
 /* Enum describing the various ways we can handle attributes.
@@ -18603,7 +18731,7 @@ aarch64_handle_attr_arch (const char *str)
     {
       gcc_assert (tmp_arch);
       selected_arch = tmp_arch->arch;
-      aarch64_set_asm_isa_flags (tmp_flags);
+      aarch64_set_asm_isa_flags (tmp_flags | AARCH64_ISA_MODE);
       return true;
     }
 
@@ -18644,7 +18772,7 @@ aarch64_handle_attr_cpu (const char *str)
       gcc_assert (tmp_cpu);
       selected_tune = tmp_cpu->ident;
       selected_arch = tmp_cpu->arch;
-      aarch64_set_asm_isa_flags (tmp_flags);
+      aarch64_set_asm_isa_flags (tmp_flags | AARCH64_ISA_MODE);
       return true;
     }
 
@@ -18744,7 +18872,7 @@ aarch64_handle_attr_isa_flags (char *str)
      features if the user wants to handpick specific features.  */
   if (strncmp ("+nothing", str, 8) == 0)
     {
-      isa_flags = 0;
+      isa_flags = AARCH64_ISA_MODE;
       str += 8;
     }
 
@@ -19237,7 +19365,7 @@ aarch64_can_inline_p (tree caller, tree callee)
 /* Return the ID of the TLDESC ABI, initializing the descriptor if hasn't
    been already.  */
 
-unsigned int
+arm_pcs
 aarch64_tlsdesc_abi_id ()
 {
   predefined_function_abi &tlsdesc_abi = function_abis[ARM_PCS_TLSDESC];
@@ -19251,7 +19379,7 @@ aarch64_tlsdesc_abi_id ()
 	SET_HARD_REG_BIT (full_reg_clobbers, regno);
       tlsdesc_abi.initialize (ARM_PCS_TLSDESC, full_reg_clobbers);
     }
-  return tlsdesc_abi.id ();
+  return ARM_PCS_TLSDESC;
 }
 
 /* Return true if SYMBOL_REF X binds locally.  */
@@ -26956,6 +27084,10 @@ aarch64_comp_type_attributes (const_tree type1, const_tree type2)
     return 0;
   if (!check_attr ("SVE sizeless type"))
     return 0;
+  if (!check_attr ("arm_streaming"))
+    return 0;
+  if (!check_attr ("arm_streaming_compatible"))
+    return 0;
   return 1;
 }
 
diff --git a/gcc/config/aarch64/aarch64.h b/gcc/config/aarch64/aarch64.h
index e60f9bce023..1ac37b902bf 100644
--- a/gcc/config/aarch64/aarch64.h
+++ b/gcc/config/aarch64/aarch64.h
@@ -157,10 +157,13 @@
 
 #ifndef USED_FOR_TARGET
 
-/* Define an enum of all features (architectures and extensions).  */
+/* Define an enum of all features (ISA modes, architectures and extensions).
+   The ISA modes must come first.  */
 enum class aarch64_feature : unsigned char {
+#define DEF_AARCH64_ISA_MODE(IDENT) IDENT,
 #define AARCH64_OPT_EXTENSION(A, IDENT, C, D, E, F) IDENT,
 #define AARCH64_ARCH(A, B, IDENT, D, E) IDENT,
+#include "aarch64-isa-modes.def"
 #include "aarch64-option-extensions.def"
 #include "aarch64-arches.def"
 };
@@ -169,16 +172,34 @@ enum class aarch64_feature : unsigned char {
 #define HANDLE(IDENT) \
   constexpr auto AARCH64_FL_##IDENT \
     = aarch64_feature_flags (1) << int (aarch64_feature::IDENT);
+#define DEF_AARCH64_ISA_MODE(IDENT) HANDLE (IDENT)
 #define AARCH64_OPT_EXTENSION(A, IDENT, C, D, E, F) HANDLE (IDENT)
 #define AARCH64_ARCH(A, B, IDENT, D, E) HANDLE (IDENT)
+#include "aarch64-isa-modes.def"
 #include "aarch64-option-extensions.def"
 #include "aarch64-arches.def"
 #undef HANDLE
 
+constexpr auto AARCH64_FL_SM_STATE = AARCH64_FL_SM_ON | AARCH64_FL_SM_OFF;
+
+constexpr unsigned int AARCH64_NUM_ISA_MODES = (0
+#define DEF_AARCH64_ISA_MODE(IDENT) + 1
+#include "aarch64-isa-modes.def"
+);
+
+/* The mask of all ISA modes.  */
+constexpr auto AARCH64_FL_ISA_MODES
+  = (aarch64_feature_flags (1) << AARCH64_NUM_ISA_MODES) - 1;
+
+/* The default ISA mode, for functions with no attributes that specify
+   something to the contrary.  */
+constexpr auto AARCH64_FL_DEFAULT_ISA_MODE = AARCH64_FL_SM_OFF;
+
 #endif
 
 /* Macros to test ISA flags.  */
 
+#define AARCH64_ISA_MODE           (aarch64_isa_flags & AARCH64_FL_ISA_MODES)
 #define AARCH64_ISA_CRC            (aarch64_isa_flags & AARCH64_FL_CRC)
 #define AARCH64_ISA_CRYPTO         (aarch64_isa_flags & AARCH64_FL_CRYPTO)
 #define AARCH64_ISA_FP             (aarch64_isa_flags & AARCH64_FL_FP)
@@ -895,6 +916,7 @@ enum arm_pcs
 typedef struct
 {
   enum arm_pcs pcs_variant;
+  aarch64_feature_flags isa_mode;
   int aapcs_arg_processed;	/* No need to lay out this argument again.  */
   int aapcs_ncrn;		/* Next Core register number.  */
   int aapcs_nextncrn;		/* Next next core register number.  */
diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
index ca2e618d9b9..cd6d5e5000c 100644
--- a/gcc/config/aarch64/aarch64.md
+++ b/gcc/config/aarch64/aarch64.md
@@ -7088,7 +7088,8 @@ (define_expand "tlsdesc_small_<mode>"
   {
     if (TARGET_SVE)
       {
-	rtx abi = gen_int_mode (aarch64_tlsdesc_abi_id (), DImode);
+	rtx abi = aarch64_gen_callee_cookie (AARCH64_ISA_MODE,
+					     aarch64_tlsdesc_abi_id ());
 	rtx_insn *call
 	  = emit_call_insn (gen_tlsdesc_small_sve_<mode> (operands[0], abi));
 	RTL_CONST_CALL_P (call) = 1;
diff --git a/gcc/config/aarch64/t-aarch64 b/gcc/config/aarch64/t-aarch64
index ba74abc0a43..47a753c5f1b 100644
--- a/gcc/config/aarch64/t-aarch64
+++ b/gcc/config/aarch64/t-aarch64
@@ -18,7 +18,10 @@
 #  along with GCC; see the file COPYING3.  If not see
 #  <http://www.gnu.org/licenses/>.
 
-TM_H += $(srcdir)/config/aarch64/aarch64-cores.def
+TM_H += $(srcdir)/config/aarch64/aarch64-cores.def \
+	$(srcdir)/config/aarch64/aarch64-isa-modes.def \
+	$(srcdir)/config/aarch64/aarch64-option-extensions.def \
+	$(srcdir)/config/aarch64/aarch64-arches.def
 OPTIONS_H_EXTRA += $(srcdir)/config/aarch64/aarch64-cores.def \
 		   $(srcdir)/config/aarch64/aarch64-arches.def \
 		   $(srcdir)/config/aarch64/aarch64-fusion-pairs.def \
diff --git a/gcc/testsuite/gcc.target/aarch64/auto-init-1.c b/gcc/testsuite/gcc.target/aarch64/auto-init-1.c
index 0fa470880bf..48c5bb6a45c 100644
--- a/gcc/testsuite/gcc.target/aarch64/auto-init-1.c
+++ b/gcc/testsuite/gcc.target/aarch64/auto-init-1.c
@@ -29,4 +29,5 @@ void foo()
   return;
 }
 
-/* { dg-final { scan-rtl-dump-times "const_int 0" 11 "expand" } } */
+/* Includes 1 for the call instruction and one for a nop.  */
+/* { dg-final { scan-rtl-dump-times "const_int 0" 10 "expand" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sme/aarch64-sme.exp b/gcc/testsuite/gcc.target/aarch64/sme/aarch64-sme.exp
new file mode 100644
index 00000000000..c542912e14a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sme/aarch64-sme.exp
@@ -0,0 +1,41 @@
+#  Specific regression driver for AArch64 SME.
+#  Copyright (C) 2009-2022 Free Software Foundation, Inc.
+#  Contributed by ARM Ltd.
+#
+#  This file is part of GCC.
+#
+#  GCC is free software; you can redistribute it and/or modify it
+#  under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; either version 3, or (at your option)
+#  any later version.
+#
+#  GCC is distributed in the hope that it will be useful, but
+#  WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#  General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with GCC; see the file COPYING3.  If not see
+#  <http://www.gnu.org/licenses/>.  */
+
+# GCC testsuite that uses the `dg.exp' driver.
+
+# Exit immediately if this isn't an AArch64 target.
+if {![istarget aarch64*-*-*] } then {
+    return
+}
+
+# Load support procs.
+load_lib gcc-dg.exp
+
+# Initialize `dg'.
+dg-init
+
+aarch64-with-arch-dg-options "" {
+    # Main loop.
+    dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/*.\[cCS\]]] \
+	"" ""
+}
+
+# All done.
+dg-finish
diff --git a/gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_1.c b/gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_1.c
new file mode 100644
index 00000000000..22d4a8bcc97
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_1.c
@@ -0,0 +1,106 @@
+// { dg-options "" }
+
+void __attribute__((arm_streaming_compatible)) sc_a ();
+void sc_a (); // { dg-error "conflicting types" }
+
+void sc_b ();
+void __attribute__((arm_streaming_compatible)) sc_b (); // { dg-error "conflicting types" }
+
+void __attribute__((arm_streaming_compatible)) sc_c ();
+void sc_c () {} // Inherits attribute from declaration (confusingly).
+
+void sc_d ();
+void __attribute__((arm_streaming_compatible)) sc_d () {} // { dg-error "conflicting types" }
+
+void __attribute__((arm_streaming_compatible)) sc_e () {}
+void sc_e (); // { dg-error "conflicting types" }
+
+void sc_f () {}
+void __attribute__((arm_streaming_compatible)) sc_f (); // { dg-error "conflicting types" }
+
+extern void (*sc_g) ();
+extern __attribute__((arm_streaming_compatible)) void (*sc_g) (); // { dg-error "conflicting types" }
+
+extern __attribute__((arm_streaming_compatible)) void (*sc_h) ();
+extern void (*sc_h) (); // { dg-error "conflicting types" }
+
+//----------------------------------------------------------------------------
+
+void __attribute__((arm_streaming)) s_a ();
+void s_a (); // { dg-error "conflicting types" }
+
+void s_b ();
+void __attribute__((arm_streaming)) s_b (); // { dg-error "conflicting types" }
+
+void __attribute__((arm_streaming)) s_c ();
+void s_c () {} // Inherits attribute from declaration (confusingly).
+
+void s_d ();
+void __attribute__((arm_streaming)) s_d () {} // { dg-error "conflicting types" }
+
+void __attribute__((arm_streaming)) s_e () {}
+void s_e (); // { dg-error "conflicting types" }
+
+void s_f () {}
+void __attribute__((arm_streaming)) s_f (); // { dg-error "conflicting types" }
+
+extern void (*s_g) ();
+extern __attribute__((arm_streaming)) void (*s_g) (); // { dg-error "conflicting types" }
+
+extern __attribute__((arm_streaming)) void (*s_h) ();
+extern void (*s_h) (); // { dg-error "conflicting types" }
+
+//----------------------------------------------------------------------------
+
+void __attribute__((arm_streaming)) mixed_a ();
+void __attribute__((arm_streaming_compatible)) mixed_a (); // { dg-error "conflicting types" }
+// { dg-warning "ignoring attribute" "" { target *-*-* } .-1 }
+
+void __attribute__((arm_streaming_compatible)) mixed_b ();
+void __attribute__((arm_streaming)) mixed_b (); // { dg-error "conflicting types" }
+// { dg-warning "ignoring attribute" "" { target *-*-* } .-1 }
+
+void __attribute__((arm_streaming)) mixed_c ();
+void __attribute__((arm_streaming_compatible)) mixed_c () {} // { dg-warning "ignoring attribute" }
+
+void __attribute__((arm_streaming_compatible)) mixed_d ();
+void __attribute__((arm_streaming)) mixed_d () {} // { dg-warning "ignoring attribute" }
+
+void __attribute__((arm_streaming)) mixed_e () {}
+void __attribute__((arm_streaming_compatible)) mixed_e (); // { dg-error "conflicting types" }
+// { dg-warning "ignoring attribute" "" { target *-*-* } .-1 }
+
+void __attribute__((arm_streaming_compatible)) mixed_f () {}
+void __attribute__((arm_streaming)) mixed_f (); // { dg-error "conflicting types" }
+// { dg-warning "ignoring attribute" "" { target *-*-* } .-1 }
+
+extern __attribute__((arm_streaming_compatible)) void (*mixed_g) ();
+extern __attribute__((arm_streaming)) void (*mixed_g) (); // { dg-error "conflicting types" }
+
+extern __attribute__((arm_streaming)) void (*mixed_h) ();
+extern __attribute__((arm_streaming_compatible)) void (*mixed_h) (); // { dg-error "conflicting types" }
+
+//----------------------------------------------------------------------------
+
+void __attribute__((arm_streaming, arm_streaming_compatible)) contradiction_1(); // { dg-warning "conflicts with attribute" }
+void __attribute__((arm_streaming_compatible, arm_streaming)) contradiction_2(); // { dg-warning "conflicts with attribute" }
+
+int __attribute__((arm_streaming_compatible)) int_attr; // { dg-warning "only applies to function types" }
+void *__attribute__((arm_streaming)) ptr_attr; // { dg-warning "only applies to function types" }
+
+typedef void __attribute__((arm_streaming)) s_callback ();
+typedef void __attribute__((arm_streaming_compatible)) sc_callback ();
+
+void (*__attribute__((arm_streaming)) s_callback_ptr) ();
+void (*__attribute__((arm_streaming_compatible)) sc_callback_ptr) ();
+
+typedef void __attribute__((arm_streaming, arm_streaming_compatible)) contradiction_callback_1 (); // { dg-warning "conflicts with attribute" }
+typedef void __attribute__((arm_streaming_compatible, arm_streaming)) contradiction_callback_2 (); // { dg-warning "conflicts with attribute" }
+
+void __attribute__((arm_streaming, arm_streaming_compatible)) (*contradiction_callback_ptr_1) (); // { dg-warning "conflicts with attribute" }
+void __attribute__((arm_streaming_compatible, arm_streaming)) (*contradiction_callback_ptr_2) (); // { dg-warning "conflicts with attribute" }
+
+struct s {
+  void __attribute__((arm_streaming, arm_streaming_compatible)) (*contradiction_callback_ptr_1) (); // { dg-warning "conflicts with attribute" }
+  void __attribute__((arm_streaming_compatible, arm_streaming)) (*contradiction_callback_ptr_2) (); // { dg-warning "conflicts with attribute" }
+};
diff --git a/gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_2.c b/gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_2.c
new file mode 100644
index 00000000000..448ddb5feb1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sme/streaming_mode_2.c
@@ -0,0 +1,25 @@
+// { dg-options "" }
+
+void __attribute__((arm_streaming_compatible)) sc_fn ();
+void __attribute__((arm_streaming)) s_fn ();
+void ns_fn ();
+
+__attribute__((arm_streaming_compatible)) void (*sc_fn_ptr) ();
+__attribute__((arm_streaming)) void (*s_fn_ptr) ();
+void (*ns_fn_ptr) ();
+
+void
+f ()
+{
+  sc_fn_ptr = sc_fn;
+  sc_fn_ptr = s_fn; // { dg-warning "incompatible pointer type" }
+  sc_fn_ptr = ns_fn; // { dg-warning "incompatible pointer type" }
+
+  s_fn_ptr = sc_fn; // { dg-warning "incompatible pointer type" }
+  s_fn_ptr = s_fn;
+  s_fn_ptr = ns_fn; // { dg-warning "incompatible pointer type" }
+
+  ns_fn_ptr = sc_fn; // { dg-warning "incompatible pointer type" }
+  ns_fn_ptr = s_fn; // { dg-warning "incompatible pointer type" }
+  ns_fn_ptr = ns_fn;
+}
-- 
2.25.1